Allow users to provide implicit managed identity to Azure Batch when pool identity is set to true

It turns out Fusion can automatically pick up a managed identity that is available, however in the first iteration we insisted users provide an explicit identity.

If we avoid setting FUSION_AZ_MSI_CLIENT_ID fusion will pick up this identity and authenticate to Azure Storage automatically, with no details shared externally.

I've overloaded the config item to allow users to set it to 'true', which will avoid setting FUSION_AZ_MSI_CLIENT_ID and enable Fusion to do this.

This isn't a great implementation, it should probably use a dedicated config item like AzManagedIdentityOpts does for Nextflow itself, but it's a POC that allows me to test the methods.

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>

Author: adamrtalbot

docs/reference/config.md

@@ -417,7 +417,7 @@ The following settings are available:
 `azure.batch.poolIdentityClientId`
 : :::{versionadded} 25.05.0-edge
   :::
-: Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) that is available on all Azure Batch node pools. This identity will be used for task-level authentication to Azure services. See {ref}`azure-managed-identities` for more details.
+: Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) that is available on all Azure Batch node pools. This identity will be used by Fusion to authenticate to Azure storage. If set to 'true', Fusion will use the first available managed identity.
 
 `azure.managedIdentity.clientId`
 : Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). See {ref}`azure-managed-identities` for more details. Defaults to environment variable `AZURE_MANAGED_IDENTITY_USER`.

plugins/nf-azure/src/main/nextflow/cloud/azure/fusion/AzFusionEnv.groovy

@@ -64,7 +64,13 @@ class AzFusionEnv implements FusionEnv {
         // If pool has a managed identity, ONLY add the MSI client ID
         // DO NOT add any SAS token or reference cfg.storage().sasToken
         if (managedIdentityId) {
-            result.FUSION_AZ_MSI_CLIENT_ID = managedIdentityId
+            // Fusion will try and pick up a managed identity that is available.
+            // We recommend explicitly setting the config item to the managed ID so you know which one is being used.
+            // However if set to 'true' it will use whichever is available.
+            // This can be helpful if the pools have different managed identities.
+            if (managedIdentityId != 'true') {
+                result.FUSION_AZ_MSI_CLIENT_ID = managedIdentityId
+            }
             // No SAS token is added or generated
             return result
         }

plugins/nf-azure/src/test/nextflow/cloud/azure/fusion/AzFusionEnvTest.groovy

@@ -243,4 +243,26 @@ class AzFusionEnvTest extends Specification {
         env.size() == 2  // Only account name and managed identity
     }
 
+    def 'should not provide explicit managed identity when pool identity is set to true'() {
+        given:
+        def NAME = 'myaccount'
+        Global.session = Mock(Session) {
+            getConfig() >> [azure: [
+                storage: [accountName: NAME],
+                batch: [poolIdentityClientId: 'true']
+            ]]
+        }
+
+        when:
+        def config = Mock(FusionConfig)
+        def fusionEnv = new AzFusionEnv()
+        def env = fusionEnv.getEnvironment('az', config)
+
+        then:
+        env.AZURE_STORAGE_ACCOUNT == NAME
+        !env.FUSION_AZ_MSI_CLIENT_ID
+        !env.AZURE_STORAGE_SAS_TOKEN  // SAS token should NOT be present
+        env.size() == 1  // Only account name
+    }
+
 }