Azure Storage Configuration

To manage configuration settings for Cloudfuse for Azure storage, either blob or datalake, you have three options (in order of precedence):

1. Configuration file
2. Environment variables
3. CLI parameters

Using a configuration file is the preferred method, but the other methods might be useful in some circumstances. Only authentication can be setup with configuration files or environment variables.

Configuration file

Creating a configuration file is the preferred method to establish settings for Cloudfuse. When you've specified the settings you want in the configuration file, reference the configuration file when you use blobfuse2 mount or other commands.

Here's an example:

cloudfuse mount ./mount --config-file=./config.yaml

The Cloudfuse base configuration file contains a list of all settings and a brief explanation of each setting.

Use the sample file cache configuration file or the sample streaming configuration file to get started quickly by using some basic settings for each of those scenarios.

To connect to Azure, there are four authentication methods supports: Key, SAS, Managed Identity, or Service Principal Name. In the 'azstorage' section of your configuration file specify the required fields for your given authentication methods. Here is the given azstorage section that you would need to fill in for your authentication method.

azstorage:
# Required
  type: block|adls <type of storage account to be connected. Default - block>
  account-name: <name of the storage account>
  container: <name of the storage container to be mounted>
  endpoint: <storage account endpoint (example - https://account-name.blob.core.windows.net)>
  mode: key|sas|spn|msi <kind of authentication to be used>
  account-key: <storage account key>
  # OR
  sas: <storage account sas>
  # OR
  appid: <storage account app id / client id for MSI>
  resid: <storage account resource id for MSI>
  objid: <object id for MSI>
  # OR
  tenantid: <storage account tenant id for SPN>
  clientid: <storage account client id for SPN>
  clientsecret: <storage account client secret for SPN>
  oauth-token-path: <path to file containing the OAuth token>

Environment variables

Setting environment variables is another way to configure some Cloudfuse settings. The supported environment variables are useful for specifying the Azure Blob Storage container to access and the authorization method to use.

The followingenvironment variables are supported:

• General options
  • AZURE_STORAGE_ACCOUNT: Specifies the storage account to be connected.
  • AZURE_STORAGE_ACCOUNT_TYPE: Specifies the account type 'block' or 'adls'
  • AZURE_STORAGE_ACCOUNT_CONTAINER: Specifies the name of the container to be mounted
  • AZURE_STORAGE_BLOB_ENDPOINT: Specifies the blob endpoint to use. Defaults to *.blob.core.windows.net, but is useful for targeting storage emulators.
  • AZURE_STORAGE_AUTH_TYPE: Overrides the currently specified auth type. Case insensitive. Options: Key, SAS, MSI, SPN
• Account key auth:
  • AZURE_STORAGE_ACCESS_KEY: Specifies the storage account key to use for authentication.
• SAS token auth:
  • AZURE_STORAGE_SAS_TOKEN: Specifies the SAS token to use for authentication.
• Managed Identity auth:
  • AZURE_STORAGE_IDENTITY_CLIENT_ID: Only one of these three parameters are needed if multiple identities are present on the system.
  • AZURE_STORAGE_IDENTITY_OBJECT_ID: Only one of these three parameters are needed if multiple identities are present on the system.
  • AZURE_STORAGE_IDENTITY_RESOURCE_ID: Only one of these three parameters are needed if multiple identities are present on the system.
  • MSI_ENDPOINT: Specifies a custom managed identity endpoint, as IMDS may not be available under some scenarios. Uses the MSI_SECRET parameter as the Secret header.
  • MSI_SECRET: Specifies a custom secret for an alternate managed identity endpoint.
• Service Principal Name auth:
  • AZURE_STORAGE_SPN_CLIENT_ID: Specifies the client ID for your application registration
  • AZURE_STORAGE_SPN_TENANT_ID: Specifies the tenant ID for your application registration
  • AZURE_STORAGE_AAD_ENDPOINT: Specifies a custom AAD endpoint to authenticate against
  • AZURE_STORAGE_SPN_CLIENT_SECRET: Specifies the client secret for your application registration.
  • AZURE_STORAGE_AUTH_RESOURCE : Scope to be used while requesting for token.
• Proxy Server:
  • http_proxy: The proxy server address. Example: 10.1.22.4:8080.
  • https_proxy: The proxy server address when https is turned off forcing http. Example: 10.1.22.4:8080.

CLI parameters

You also can set configuration settings when you pass them as parameters of the Cloudfuse command set, such as by using the cloudfuse mount command. The mount command typically references a configuration file that contains all the settings. But you can use CLI parameters to override individual settings in the configuration file. In this example, the config.yaml configuration file is referenced, but the container to be mounted is overridden:

cloudfuse mount ./mount_dir --config-file=./config.yaml --container-name=blobfuse2b 

CLI parameters cannot be used for authentication with Azure storage. The following are the support parameters for Azure storage.

• Storage options
  • --container-name=<CONTAINER NAME>: The container to mount.
  • --cancel-list-on-mount-seconds=<TIMEOUT IN SECONDS>: Time for which list calls will be blocked after mount. (prevent billing charges on mounting)
  • --virtual-directory=true : Support virtual directories without existence of a special marker blob for block blob account (Azure only).
  • --subdirectory=<path> : Subdirectory to mount instead of entire container.
  • --disable-compression:false : Disable content encoding negotiation with server. If objects/blobs have 'content-encoding' set to 'gzip' then turn on this flag.
  • --use-adls=false : Specify configured storage account is HNS enabled or not. This must be turned on when HNS enabled account is mounted.