Azure Bucket Storage

Azure bucket storage using the native file format of LogScale, allowing LogScale to fetch those files and search them efficiently if the local copies are lost or deleted. For more details on this topic in general, see the Bucket Storage page.

In the LogScale implementation of Azure blob storage encryption is LogScale-implemented and enforced, not relying on any support from the bucket provider. It is controlled by the configuration variable AZURE_STORAGE_ENCRYPTION_KEY, which is applied whenever its value is not set to off (because this value would disable the use of an encryption key).

Configure Azure Blob Storage

Note

Prior to configuring Azure blob storage, make sure you or the person configuring blob storage has the Storage account contributor role to be able to configure blob storage correctly and obtain all of the information needed to configure Azure blob storage in LogScale.

LogScale only supports authorization using the Account key-based authentication with Azure Role Based Access Control (RBAC). Each storage account to which LogScale needs to read, write, and delete to the account's blob storage must have the following three standard, built-in Azure permissions:

Azure storage account permissions

Figure 94. Azure storage account permissions


For detailed information about how to set up an Azure storage account, see Azure storage account setup.

For detailed information about configuring blob storage in your Azure storage account, see Create storage containers.

Once you have set up blob storage, you can retrieve the storage account name and account key to use when configuring storage in LogScale:

Azure storage account name and account key

Figure 95. Azure storage account name and account key


Enable Azure Storage in LogScale

Azure storage only supports Access Key based authentication for now. This means the environment variables AZURE_STORAGE_ACCOUNTNAME and AZURE_STORAGE_ACCOUNTKEY are required for authentication. These environment variables map to the Azure Storage account name and access key.

Read, write, and delete access is set in Azure on the storage container.

Important

If you rotate the key in Azure, you must update the AZURE_STORAGE_ACCOUNTKEY environment variable accordingly.

You must configure which storage bucket to use. These settings must be identical on all nodes across the entire cluster.

ini
AZURE_STORAGE_BUCKET=$BUCKET_NAME
AZURE_STORAGE_ENCRYPTION_KEY=$ENCRYPTION_SECRET
AZURE_STORAGE_OBJECT_KEY_PREFIX=/basefolder
USING_EPHEMERAL_DISKS=true

The first option here is to set the name of the bucket to use and the bucket region. The encryption key given with AZURE_STORAGE_ENCRYPTION_KEY can be any UTF-8 string. The suggested value is 64 or more random ASCII characters.

The AZURE_STORAGE_OBJECT_KEY_PREFIX is used to set the optional prefix for all object keys allows multiple LogScale clusters to use the same bucket. The prefix is unset by default.

Note

There is a performance penalty when using a non-empty prefix. LogScale recommends an unset prefix. If there are any ephemeral disks in the cluster, you must set the last option here to true.

Encryption of Azure Bucket Data

Azure enforces server-side encryption of stored data. Explicit configuration of an encryption key for storage of bucket data must be configured. Without explicit declaration of an encryption key, Azure bucket key encryption in LogScale is required due to the checksum reported by Azure being different to the checksum calculated by LogScale.

To switch off additional LogScale encryption in LogScale:

ini
AZURE_STORAGE_ENCRYPTION_KEY=off

This will allow data stored in Azure to be encrypted using the Azure configured key (see Protecting data with server-side encryption for more information).

Using a custom key is supported, and can be configured to encrypt the bucket data by setting the encryption key explicitly:

ini
AZURE_STORAGE_ENCRYPTION_KEY=customencryptionkey

This key will be used in addition to the SSE key used.

Switch to a Fresh Bucket

You can change the settings using the AZURE_STORAGE_BUCKET to point to a fresh bucket at any point in time. From that point, LogScale will write new files to that bucket while still reading from any previously-configured buckets. Existing files already written to any previous bucket will not get written to the new bucket. LogScale will continue to delete files from the old buckets that match the file names that LogScale would put there.