[BUG] Failures with Java security.policy when using repository-azure plugin managed identity

[c3-davidtran]

Describe the bug

I've noticed 2 bugs related to java security.policy :

Managed Identity authentication :
SInce Opensearch 2.15, we are now able to authenticate with Azure Managed identity when using repository-azure plugin.
As recommended by Azure, the privileged approach is to use workload identity with a federated Identity.
https://azure.github.io/azure-workload-identity/docs/introduction.html

This will generate a federated token when the pod starts, allowing authentication.
The token is located at path : /var/run/secrets/azure/tokens/azure-identity-token
So when enabling authentication using Managed identity, the plugin fails because it cannot read this file.
To solve it, we need to add a read permission on the java security.policy.
e.g

permission java.io.FilePermission "/var/run/secrets/azure/tokens/azure-identity-token", "read";

Related component

Plugins

To Reproduce

Deploy Azure wokload identity controller
Deploy an opensearch cluster :

• Make sure to use in opensearch.yaml : azure.client.default.token_credential_type: "managed_identity"
• Make sure pods authenticate using federated identity

Expected behavior

Pods java process read /var/run/secrets/azure/tokens/azure-identity-token

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[cwperks]

@c3-davidtran repository-azure is an OpenSearch plugin which has its own plugin-security.policy file. This would be the place to add an entry to grant permission to read the file for the plugin.

Would it be possible to test an added entry to that file? After modifying the plugin-security.policy you can create a local snapshot using ./gradlew assemble from the root directory of the plugin. The output of ./gradlew assemble would be a zip file in the build/distributions/ directory.

[reta]

@cwperks none of the OpenSearch plugins have access to the file system outside OPENSEARCH_PATH_CONFIG (and although technically possible, it is not recommended).

@c3-davidtran this is clearly an issue that was missed, they way we solved it for S3 plugin [1] is by:

1. having a setting for a token
2. having a symlink under OPENSEARCH_PATH_CONFIG

If you don’t want to configure AWS access and secret keys, modify the following opensearch.yml setting. Make sure the file is accessible by the repository-s3 plugin:
s3.client.default.identity_token_file: /usr/share/opensearch/plugins/repository-s3/token

If copying is not an option, you can create a symlink to the web identity token file in the ${OPENSEARCH_PATH_CONFIG} folder:
ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "${OPENSEARCH_PATH_CONFIG}/aws-web-identity-token-file"

You can reference the web identity token file in the following opensearch.yml setting by specifying the relative path that is resolved against ${OPENSEARCH_PATH_CONFIG}:
s3.client.default.identity_token_file: aws-web-identity-token-file

We would need similar approach for Azure.

[1] https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/

[peternied]

[Triage - attendees 1 2 3 4 5]
@c3-davidtran Thanks for creating this issue, what do you think about creating a pull request to fix this?

[cthtrifork]

@reta Can you expand upon your thoughts on solving it through a setting?

@chengwushi-netapp Did you not hit this issue while implementing #12559 ?

We are using kubernetes and the opensearch-operator. I suppose we can do the symlink in a custom Dockerfile for OpenSearch 🤔

[reta]

@reta Can you expand upon your thoughts on solving it through a setting?

Absolutely, thanks @cthtrifork . The idea is to introduce the setting like this:

    /** The identity token file for connecting to Azure. */
    static final Setting.AffixSetting<String> IDENTITY_TOKEN_FILE_SETTING = Setting.affixKeySetting(
        PREFIX,
        "identity_token_file",
        key -> SecureSetting.simpleString(key, Property.NodeScope)
    );

The lookup in default location (/var/run/secrets/azure/tokens/azure-identity-token) should be replaced by this setting (the corresponding Azure SDK configuration setting / system property is AZURE_FEDERATED_TOKEN_FILE). The /var/run/secrets/azure/tokens/azure-identity-token could be soft linked to ${OPENSEARCH_PATH_CONFIG}/azure-identity-token and the new setting could point to the link (the location is relative to ${OPENSEARCH_PATH_CONFIG}):

azure.default.identity_token_file: azure-identity-token

The plugin should be able to read the azure-identity-token without any security violations.

[chengwushi-netapp]

@chengwushi-netapp Did you not hit this issue while implementing #12559 ?

Hello @cthtrifork, thank you for the question. When I was working on the implementation, I did not encounter the issue you're referring to. This is because the scope of the original ticket, #12423, did not include Workload Identity. As such, I didn't delve into the workings of Workload Identity and might have overlooked that it could be set up with Managed Identity. Consequently, I never tested the setup of Workload Identity with the Managed Identity implemented in #12559.

[cthtrifork]

@chengwushi-netapp Thank you for clarifying. The landscape in azure for managed identities is quite confusing, as it has had many names and paradigms! Now i understand why you have not had the issue.

So are there any other approaches than creating a custom Dockerfile with a run ln -s #softlink instruction + adding the new setting? It seems like a hack.

It almost seems like updating https://github.com/opensearch-project/OpenSearch/blob/main/plugins/repository-azure/src/main/plugin-metadata/plugin-security.policy is a better approach.

What are people thoughts? @chengwushi-netapp @reta
@peternied any opinions on approaches?

[reta]

What are people thoughts? @chengwushi-netapp @reta\n@peternied any opinions on approaches?

Thanks @cthtrifork , patching the security policy is not recommended (it opens up yet another can of worms), soft link is the safest option I believe and is aligned with other plugins that support managed identity