Azure Reference Architecture

This section of the documentation contains Terraform configurations to deploy a Microsoft Azure-based architecture for LogScale. It leverages multiple Azure services including Azure Key Vault, Azure Kubernetes Service, and Azure Storage.

Overview

The logscale-azure terraform modules provide the following architecture choices:

Basic use cases:

  • Development, Testing
  • Smaller Search Teams
  • Minimal ingest processing

Ingress use cases:

  • Ingress tier in specified DMZ
  • Ingress resources not shared with Kubernetes system resources

Dedicated UI use cases:

  • Separated ingress tier
  • Separated UI tier
  • Ingest/Digest on same hosts

Advanced use cases:

  • Dedicated processing tier (ingestion) scaling separate from digest/storage
  • Dedicated UI tier for dashboards and search
  • Segmentation of system responsibilities and independent scaling

All architecture choices rely on the same underlying technologies:

  • Azure Infrastructure and Resource Groups - Note that the user will have 2 resource groups, 1 created by this terraform and the other associated to the managed Kubernetes services.
  • Azure Kubernetes Service (AKS)
  • Azure NAT Gateway - Allows egress data to pull images for kubernetes pods
  • Azure KeyVault:
    • Stores kube secrets for the environment
    • Stores Logscale encryption string
    • Stores Logscale single user password
    • Stores encryption key for AKS disk encryption set
  • Azure Load Balancer - Allow data ingest and UI access to the environment
  • Azure Storage Account - Object storage for Logscale data durability
  • Kubernetes Apps:
    • cert-manager: For automated provisioning of certificates in the environment
    • strimzi-operator: For provisioning Kafka broker nodes (kraft mode)
    • humio-operator: For provisioning of Logscale clusters in the environment
    • nginx-ingress: For connecting the Azure Load Balancer

By default:

  • Kubernetes API is public but has IP restrictions applied based on user-controlled variables to limit access.
  • Ingest endpoint is publicly available:
    • Port 80: Used for Let's Encrypt certificate signing - Access is global but only available temporarily during ACME challenge/response while the cert-manager response pod is alive.
    • Port 443: Used for ingest/UI to Logscale - Access is limited to user configured IP list variable
  • Storage account access is restricted to:
    • User provided IP ranges
    • Subnet(s) created for LogScale nodes
    • Azure Trusted Services, Logging, and Metrics
  • KeyVault access is restricted to:
    • User provided IP ranges
    • Azure Trusted Services
  • All AKS nodes are made available on required ports to AzureCloud IP ranges to allow for Kubernetes control plane operations
  • All AKS nodes are made accessible within the VNET

Important

Some settings are configurable and still pending testing. For example, it is a configuration option to make the Load Balancer and Kubernetes API internal-only but this has not been tested.

Networking is created as follows:

Purpose Architecture Created IP Range
Virtual Network all 172.16.0.0/16
Kubernetes System Nodes all 172.16.0.0/24
Kafka Nodes all 172.16.2.0/24
Bastion Nodes all 172.16.1.0/26
Logscale UI Nodes advanced 172.16.6.0/24
Nginx Ingress Nodes ingress, advanced 172.16.4.0/24
Logscale Ingest Nodes advanced 172.16.5.0/24
Logscale Digest Nodes all 172.16.3.0/24

Diagnostic (Audit) Logging

Logging can be enabled for supported resources to supported target destinations that must be created outside of this terraform. Currently supported resources:

  • Azure Key Vault
  • Azure Kubernetes Service
  • Azure Storage Account

Currently supported destinations:

  • Azure Event Hub
  • Azure Storage Account
  • Azure Log Analytics

Update your created .tfvars file to enable this feature. Example:

ini
enable_auditlogging_to_storage      = true
diag_logging_storage_account_id     = "/subscriptions/$mysubscription/resourceGroups/$myresourcegroup/providers/Microsoft.Storage/storageAccounts/$mystorageacct"

The following guidance box provides links to the main section of this documentation:

Architecture Types

This section describes the supported architecture types.

Requirements

This section describes prerequisites for deploying a Microsoft Azure reference architecture for LogScale.

Terraform Modules

This section describes Terraform modules for deploying the Azure reference architecture.

Build Process

This section describes the build process.