LogScale Cluster Deployment in Azure AKS

Overview :

This document is a guide to provision a self-hosted LogScale cluster on Azure Cloud using Azure AKS kubernetes, with Azure object store for event repositories.

The sequence of steps and associated tasks required to provision LogScale are split into sections for planning, deployment, and validation. Sections should be processed in top-down order to ensure that prerequisites are met as provisioning steps are executed.

Notes:

• This cluster deployment utilizes an independent Kafka service.
• This document assumes at least intermediate level knowledge of Azure Cloud.

Architectural Diagrams:

Deployment Overview:

Functional Overview:

Prerequisites :

• Falcon LogScale License Key
• LogScale Instance sizing guidelines:
  • Instance Sizing
  • Recommended Instance Sizing
• Azure Portal account with an active subscription
• Azure Storage account
  • Create a storage account
• Azure Blob Container (i.e. storage bucket)
  • Create a blob container
• Good understanding of Azure Resource groups
• Good understanding Kubernetes

Deployment:

1. Create an AKS cluster using any of the following methods:

• Azure CLI
• Azure Portal
• Terraform

Validate the cluster is up and running by checking kubernetes service status:

• kubectl get svc

2. Deploy Kafka/ZooKeeper to the AKS cluster using the strimzi-kafka-operator - Note: Default partition count is 24. For details, refer to the LogScale Digest Rules - Deploy the strimzi-kafka-operator (kafka/k8s orchestration facility - strimzi key features) - Execute following commands in K8s environment - helm repo add strimzi https://strimzi.io/charts/ - helm repo update - helm install strimzi-kafka strimzi/strimzi-kafka-operator

• Provision the Kafka+zookeeper cluster
  • Configure Kafka/zookeeper parameters (see sample kafka-zookeeper.yaml)
    • replicas
    • offsets.topic.replication.factor
    • transaction.state.log.replication.factor
    • transaction.state.log.min.isr
    • default.replication.factor
    • min.insync.replicas
    • Inter.broker.protocol.version
    • storage.size
  • Apply Kafka yaml

• Validate installation by checking pods and services

  • kubectl get pods
  • kubectl get svc

3. Deploy S3Proxy

• LogScale supports S3 compatible storage using S3Proxy service. Azure Blob Storage is one of them. For details, refer to the S3Proxy documentation.
• Provision S3Proxy node(s)
  • Configure parameters as required (see sample s3proxy.yaml) | Name | Value | | :-- | :-- | | S3PROXY_AUTHORIZATION | none | | JCLOUDS_PROVIDER | azureblob | | JCLOUDS_ENDPOINT | <Azure storage a/c endpoint> | | JCLOUDS_IDENTITY | <Azure storage a/c name> | | JCLOUDS_CREDENTIAL | <Azure storage a/c access key> | | JCLOUDS_AZYREBLOB_AUTH | azureKey | | LOG_LEVEL | debug |
  • Apply S3Proxy & S3Proxy service (see sample s3proxy-service.yaml)
• Verify that s3proxy pod and service are running.

4. Deploy LogScale - Note: This deployment uses NODE_ROLES="all". For details, refer to the NODE_ROLES documentation. - Deploy LogScale Operator (see setup_logscale.sh) - Obtain the latest stable version from LogScale Operator Releases. - Apply LogScale CRDs & Operator - Configure LogScale - Create a license key secret - kubectl create secret generic example-humiocluster-license --from-literal=data=<LogScale License> - Obtain the latest stable version from LogScale Releases. - In HumioCluster yaml, set the parameters as required (see sample logscale-cluster.yaml) - Set the parameters under spec - targetReplicationFactor - storagePartitionsCount - digestPartitionsCount - Set the Env. variables under spec.environmentVariables which are required to enable the S3 compatible storage | Name | Value | | :-- | :-- | | USING_EPHEMERAL_DISKS | true | | NODE_ROLES | all\|ingestonly\|httponly| | S3_STORAGE_ENDPOINT_BASE | <s3proxy endpoint> | | S3_STORAGE_ACCESSKEY | <Azure Storage a/c accesskey> | | S3_STORAGE_SECRETKEY | <Azure Storage a/c accesskey> | | LOCAL_STORAGE_PERCENTAGE | "80" | | S3_STORAGE_PATH_STYLE_ACCESS | true | | S3_STORAGE_IBM_COMPAT | true | | BUCKET_STORAGE_IGNORE_ETAG_UPLOAD | true | | BUCKET_STORAGE_IGNORE_ETAG_AFTER_UPLOAD | false | | BUCKET_STORAGE_SSE_COMPATIBLE | true | | S3_STORAGE_ENCRYPTION_KEY | off | | S3_STORAGE_BUCKET | "<blob container>" | | S3_ARCHIVING_PATH_STYLE_ACCESS | true | | S3_EXPORT_PATH_STYLE_ACCESS | true | | S3_STORAGE_PREFERRED_COPY_SOURCE | true

    More information on environment variables : [Falcon LogScale - Configuration Parameters](https://library.humio.com/falcon-logscale-self-hosted/envar.html)
  - Apply HumioCluster yaml
- Status of each container can be checked by running following command:
  - kubectl get pods
  - kubectl describe pod \<humio-cluster-name\>

5. Test and Verify

• Check availability of pods and services
  • kubectl get pods
  • kubectl get svc
• Test connectivity and availability of LogScale cluster forwarding the port locally
  • kubectl port-forward svc/example-humiocluster 8888:8080

6. Deploy Application gateway and enable AGIC - Deploy Application Gateway Ingress Controller Add-on to access LogScale App from the Internet - Create and configure Application Gateway - Enable the ingress controller add-on for a new AKS cluster

Useful Reference Links: - LogScale K8s Reference Architecture - K8s core concept for AKS

WE STOP BREACHES