Azure Monitor Alerts with Azure Functions, Event Hub, and Key Vault Integration

The goal is to be able to take Azure Monitor Alerts with Common Alert Schema Definitions and to be able to transform the alert.

We can use Azure Monitor Action Groups to point to our 'sender' Azure Function.

We'll use an Event Hub for downstream processing by an Azure Function. We'll want to ensure that both Azure Functions are able to use Key Vault References in order to point to appropriate settings for storage and the Event Hub itself.

We're going to also set up a test Virtual Machine so we can target it for alerting purposes from Azure Monitor. This could also later be used for checking on Event Hub and Azure function connectivity from within the VNET, but that would be a stretch goal.

We're going to test ip filtering on the sending function, as well as looking into use service endpoints for Event Hub, Key Vault, and Storage Account for the subnets hosting our Azure Functions. Since we're using Azure Functions with VNET Integration and IP Filtering, we can test with Azure Functions Premium Plan.

For now, we'll use the Azure Functions Core Tools to publish our Python Azure Functions, but this is something that can also be later handled with CI/CD (e.g. Azure DevOps Pipelines).

We would like to make sure that we can retrieve the secret from Key Vault, but also keep a copy so that we can avoid making additional calls to Key Vault which could potentially cause an issue with Key Vault rate limiting.

Of course, this will not be a cure-all, and each scenario / workload will need to be evaluated. This is merely an approach to keep in mind when working with Key Vault.

We're going to refer to this Getting Secrets in Key Vault post as guidance.

Links

These will describe some of the concepts that we're using in this scenario.

Key Vault rate limiting
Getting Secrets in KV from Azure Functions
Azure Functions Event Hub Bindings
Managed Identities with Azure Functions
Install Azure CLI
Azure Function App Config cli
Azure Functions Python Core Tools
Azure Event Hub cli
Azure Event Hub Connection String
Azure Event Hub Consumer Group
Azure Key Vault Logging
Service Bus Explorer
Generate Random Letters in Powershell
Azure Monitor Alerts
Azure Monitor Metric Alerts
Azure Monitor Common Alert Schema Definitions
Azure Monitor Action Groups
Azure Monitor Webhooks
Azure Functions Authentication Options
Azure Functions Deployment Options
Azure Functions Premium Plan
Azure Functions VNET Integration Options
Azure Functions IP Filtering
Azure Functions Core Tools
Azure Functions Extension
Retrieve an Azure Function Key
Azure Functions Create VNET
Azure App Service VNET Integration
Azure Virtual Network Service Endpoints
Azure Event Hub Service Endpoints
Azure Storage Account Service Endpoints
Azure Key Vault Service Endpoints
Azure Key Vault Network Security
Azure Key Vault Az CLI
Azure App Service with Key Vault References
Azure App Service Environment

Deploy Components

Clone this repo to pull down the bits.

Assuming we have a fresh Azure subscription, we can put together a test resource group along with Azure Event Hubs, Azure Functions, Azure Monitor, and a Key Vault.

Please ensure that we use Azure CLI and login to the Azure subscription.

az login
az account set -s <subscription id>

For reference we can use this deployment script from the Scenario's root directory to stand up the components for this exercise.

Deployment Script Notes

For convenience we can run the script, but we'll discuss some of the details here.

We can create a resource group and a virtual network with placeholder subnets. We'll also add service endpoints to each of the subnets for Event Hub, Storage, and Key Vault.

#create an rg
az group create -n $rg -l $location

#create VNET / subnets
az network vnet create -g $rg -n $vnetName --address-prefixes $vnetAddressPrefix --subnet-name $subnetName --subnet-prefixes $subnetAddressPrefix
$subnetID = $(az network vnet show -g $rg -n $vnetName --query "subnets[?name=='$subnetName'].id" -o tsv)

$vmSubnetObject = $(az network vnet subnet create -g $rg --vnet-name $vnetName -n $vmSubnetName --address-prefixes $vmSubnetAddressPrefix) | ConvertFrom-Json
$receiveSubnetObject = $(az network vnet subnet create -g $rg --vnet-name $vnetName -n $receiveSubnetName --address-prefixes $receiveSubnetAddressPrefix) | ConvertFrom-Json

az network vnet subnet update --resource-group $rg --vnet-name $vnetName --name $subnetName --service-endpoints "Microsoft.KeyVault", "Microsoft.Storage", "Microsoft.EventHub"
az network vnet subnet update --resource-group $rg --vnet-name $vnetName --name $vmSubnetName --service-endpoints "Microsoft.KeyVault", "Microsoft.Storage", "Microsoft.EventHub"
az network vnet subnet update --resource-group $rg --vnet-name $vnetName --name $receiveSubnetName --service-endpoints "Microsoft.KeyVault", "Microsoft.Storage", "Microsoft.EventHub"

We can now add a VM to the VM subnet. We'll later point Azure Monitor to target this VM for an Alert. This test VM can also be used to confirm VNET integration with the functions, as well as general VNET connectivity.

#This subnet is hosting a VM.
Write-Host "Creating a VM"
$defaultSubnetID = "/subscriptions/$subscriptionID/resourceGroups/$rg/providers/Microsoft.Network/virtualNetworks/$vnetName/subnets/$vmSubnetName"
#need to asssociate with a static ip address
az vm create --image $vmImage --admin-username $vmAdminUserName --admin-password $vmAdminPassword -l $location -g $rg -n $vmName --subnet $defaultSubnetID --public-ip-address-allocation $vmPublicIpAddressAllocation --size $vmSize

We can now add a storage account for the Azure Functions. This storage account must have blob containers for syncing Azure Functions. We'll also add in network rules to allow access to the subnets hosting our Azure Functions. We'll also set the default-action to deny so that only allowed networks can access the storage account.

# Create a storage account in the resource group.
az storage account create --name $saName --location $location --resource-group $rg --sku Standard_LRS --default-action deny
az storage account network-rule add -g $rg -n $saName --subnet $subnetID
az storage account network-rule add -g $rg -n $saName --subnet $receiveSubnetObject.id

We'll now stand up both the sender and receiver function apps. We'll use the App service plan as the location for the function app, and we'll need to ensure we're using the Azure Functions Premium Plan. Further, since we're using Python Azure Functions Python, we'll need to use a Linux host. We can add VNET Integration to point to the placeholder subnets created earlier, and also add in a Managed Identity for the function. This Managed Identity will be used for retrieving secrets from Key Vault.

#create app service plan and function for Sender
az appservice plan create --name $sendAspName --resource-group $rg --sku P2V2 --is-linux
$sendFaObject = $(az functionapp create --name $sendFaName --resource-group $rg --plan $sendAspName --storage-account $saName --os-type Linux --runtime python --runtime-version 3.7) | ConvertFrom-Json
az functionapp vnet-integration add -g $rg -n $sendFaName --vnet $vnetName --subnet $subnetName

az functionapp identity assign --name $sendFaName --resource-group $rg

#create app service plan and function for Receiver
az appservice plan create --name $receiveAspName --resource-group $rg --sku P2V2 --is-linux
$receiveFaObject = $(az functionapp create --name $receiveFaName --resource-group $rg --plan $receiveAspName --storage-account $saName --os-type Linux --runtime python --runtime-version 3.7) | ConvertFrom-Json
#need a separate subnet to avoid conflict with hosting another function app.
az functionapp vnet-integration add -g $rg -n $receiveFaName --vnet $vnetName --subnet $receiveSubnetName

az functionapp identity assign --name $receiveFaName --resource-group $rg

We'll now retrieve the service principal for the Managed Identity of the Azure Functions. We'll use this with a Key Vault policy.

#get service principal ID attached to function app
$sendSpID=$(az functionapp show --resource-group $rg --name $sendFaName --query identity.principalId --out tsv)

$receiveSpID=$(az functionapp show --resource-group $rg --name $receiveFaName --query identity.principalId --out tsv)

We'll create a Key Vault. We'll also add network rules to allow access to the placeholder subnets.

A workaround for Function Apps w/VNET Integration is that we'll want to also allow the possible outbound ip addresses to reach the Key Vault. We'll also later set the default-action to deny once we configure key vault secrets. Also, note that the possible outbound ip addresses can change. If these need to be further constrained to a single guaranteed IP address, we'll want to consider hosting the Azure Function in an Azure App Service Environment.

We can also add in a test secret, and then add a policy on Key Vault to allow access to the Service Principals (Managed Identities) for the Azure Functions.

#create a key vault
az keyvault create --name $kvName --resource-group $rg --location $location

az keyvault network-rule add -g $rg -n $kvName --vnet $vnetName --subnet $subnetName
az keyvault network-rule add -g $rg -n $kvName --vnet $vnetName --subnet $vmSubnetName
az keyvault network-rule add -g $rg -n $kvName --vnet $vnetName --subnet $receiveSubnetName

#combine outbound ip addresses from function apps.
$outboundIpAddresses = $sendFaObject.possibleOutboundIpAddresses.split(",") + $receiveFaObject.possibleOutboundIpAddresses.split(",") | Select-Object -Unique

##for kv also need to add possible outbound ip addresses for functions to reach this key vault
Write-Host "Add Outbound IP Addresses for Functions to reach KV"
$outboundIpAddresses.forEach({
    az keyvault network-rule add -g $rg -n $kvName --ip-address $_
})

#create a secret for Key Vault
az keyvault secret set --name $kvSecretName --value $kvSecretValue --description FunctionAppsecret  --vault-name $kvName

#grant access to key vault.  The Service Principal bound to the Function App needs rights to access the secret in Key vault.
az keyvault set-policy --name $kvName --resource-group $rg --object-id $sendSpID --secret-permissions get

az keyvault set-policy --name $kvName --resource-group $rg --object-id $receiveSpID --secret-permissions get

We can now bind a sample secret to the function app. This can be a test Key Vault Reference.

#get secret URI
$secretURI = $(az keyvault secret show --name $kvSecretName --vault-name $kvName --query id --output tsv)

#bind app setting / environment variable for Azure Function app to key vault secret URI.  Note the extra space for parsing the secret.
az functionapp config appsettings set --name $sendFaName --resource-group $rg --settings "$kvSecretName=@Microsoft.KeyVault(SecretUri=$secretURI) "

az functionapp config appsettings set --name $receiveFaName --resource-group $rg --settings "$kvSecretName=@Microsoft.KeyVault(SecretUri=$secretURI) "

Let's create our Event Hub Namespace and Event Hub. We'll also want to add the placeholder subnets as part of the network rules to have access to the Event Hub Namespace. We'll also want to create a test Azure Event Hub Consumer Group.

#create event hub
az eventhubs namespace create --resource-group $rg --name $ehNamespace --location $location

az eventhubs namespace network-rule add --action Allow -g $rg --namespace-name $ehNamespace --vnet $vnetName --subnet $subnetName
az eventhubs namespace network-rule add --action Allow -g $rg --namespace-name $ehNamespace --vnet $vnetName --subnet $receiveSubnetName
az eventhubs namespace network-rule add --action Allow -g $rg --namespace-name $ehNamespace --vnet $vnetName --subnet $vmSubnetName

az eventhubs eventhub create --resource-group $rg --namespace-name $ehNamespace --name $ehName --message-retention 4 --partition-count 15
az eventhubs eventhub consumer-group create --resource-group $rg --name $ehConsumerGroup --namespace-name $ehNamespace --eventhub-name $ehName

Let's start working through saving secrets for the Azure Functions to bind from Key Vault for Storage and Event Hubs.

For Event Hub, we can retrieve the connection information and store this in key vault as a secret. We can then bind the function apps to use the secret as a key vault reference.

#store event hub connection string in key vault, and bind key vault to function app settings for event hub trigger use
$ehcs = $(az eventhubs namespace authorization-rule keys list --resource-group $rg --namespace-name $ehNamespace --name RootManageSharedAccessKey --query primaryConnectionString --output tsv)
$kvEHSecretValue = "$ehcs;EntityPath=$ehName"

az keyvault secret set --name $kvEHSecretName --value $kvEHSecretValue --description FunctionAppsecret  --vault-name $kvName
$ehsecretURI = $(az keyvault secret show --name $kvEHSecretName --vault-name $kvName --query id --output tsv)

#bind app setting / environment variable for Azure Function app to key vault secret URI.  Note the extra space for parsing the secret.
az functionapp config appsettings set --name $sendFaName --resource-group $rg --settings "$faEventHubAppConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "

az functionapp config appsettings set --name $receiveFaName --resource-group $rg --settings "$faEventHubAppConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "

We can also store the storage account information for function apps, and use a key vault secret reference too.

#storage account for function app
$saKey = $(az storage account keys list -n $saName -g $rg --query [0].value --output tsv)
$kvSASecretValue = "DefaultEndpointsProtocol=https;AccountName=$saName;AccountKey=$saKey;EndpointSuffix=core.windows.net"

az keyvault secret set --name $kvSASecretName --value $kvSASecretValue --description FunctionAppsecret  --vault-name $kvName
$sasecretURI = $(az keyvault secret show --name $kvSASecretName --vault-name $kvName --query id --output tsv)

az functionapp config appsettings set --name $sendFaName --resource-group $rg --settings "$faStorageAccountConfig=@Microsoft.KeyVault(SecretUri=$sasecretURI) "

az functionapp config appsettings set --name $receiveFaName --resource-group $rg --settings "$faStorageAccountConfig=@Microsoft.KeyVault(SecretUri=$sasecretURI) "

For now, the send function uses broken out secrets for the event hub to form the connection string. We can also store broken out secrets and retrieve them from Key Vault for our Azure Function.

az keyvault secret set --name $kvEHHostNameSecretName --value $kvEHHostNameSecretValue --description FunctionAppsecret  --vault-name $kvName
$ehsecretURI = $(az keyvault secret show --name $kvEHHostNameSecretName --vault-name $kvName --query id --output tsv)

az functionapp config appsettings set --name $sendFaName --resource-group $rg --settings "$faEHHostNameConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "
az functionapp config appsettings set --name $receiveFaName --resource-group $rg --settings "$faEHHostNameConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "

az keyvault secret set --name $kvEHSasPolicyConfigName --value $kvEHSasSecretValue --description FunctionAppsecret  --vault-name $kvName
$ehsecretURI = $(az keyvault secret show --name $kvEHSasPolicyConfigName --vault-name $kvName --query id --output tsv)

az functionapp config appsettings set --name $sendFaName --resource-group $rg --settings "$faEHSasPolicyConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "
az functionapp config appsettings set --name $receiveFaName --resource-group $rg --settings "$faEHSasPolicyConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "

$kvEHSasKeySecretValue = $(az eventhubs namespace authorization-rule keys list --resource-group $rg --namespace-name $ehNamespace --name RootManageSharedAccessKey --query primaryKey --output tsv)
az keyvault secret set --name $kvEHSasKeyName --value $kvEHSasKeySecretValue --description FunctionAppsecret  --vault-name $kvName
$ehsecretURI = $(az keyvault secret show --name $kvEHSasKeyName --vault-name $kvName --query id --output tsv)

az functionapp config appsettings set --name $sendFaName --resource-group $rg --settings "$faEHSasKeyConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "
az functionapp config appsettings set --name $receiveFaName --resource-group $rg --settings "$faEHSasKeyConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "

$kvEHNameSecretValue = $ehName
az keyvault secret set --name $kvEhName --value $kvEHNameSecretValue --description FunctionAppsecret  --vault-name $kvName
$ehsecretURI = $(az keyvault secret show --name $kvEhName --vault-name $kvName --query id --output tsv)

az functionapp config appsettings set --name $sendFaName --resource-group $rg --settings "$faEHNameConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "
az functionapp config appsettings set --name $receiveFaName --resource-group $rg --settings "$faEHNameConfig=@Microsoft.KeyVault(SecretUri=$ehsecretURI) "

Once we are done managing secrets with Key Vault (from our deployment environment), we can now set the default-action on the key vault to deny.

We can also set default-action to deny on key vault creation. However, this means that we should also include the host IP for managing secrets.

az keyvault update -n $kvname -g $rg --default-action "deny"

We can now attempt to deploy the Azure Functions to the function app. While there are multiple Azure Functions Deployment Options, for simplicity, we'll use the Azure Functions Python Core Tools to deploy the function app. This will attempt to build the function remotely based off the local bits.

Note, for now this will attempt to publish the function twice in a row. If we end up publishing while the function app is still cycling, we may get 400 response codes. Wait for the function app to finish cycling, which can also manually restart in the Azure Portal for the function app. The remote build looks to have a timeout so the second call appears to push the function. This also assumes that the subfolder for each of the application holds the function itself (e.g. scenario/send-function and scenario/receive-function). Also be sure that the requirements.txt will have the appropriate dependencies or else the newly built version may not deploy correctly.

    Push-Location $sendFaFolder
    #might need to check if the publish was successful, or run twice?
    func azure functionapp publish $sendFaName
    func azure functionapp publish $sendFaName
    Pop-Location

    Write-Host "Publishing Receive Function"
    Push-Location $receiveFaFolder
    func azure functionapp publish $receiveFaName
    func azure functionapp publish $receiveFaName
    Pop-Location

Once we have the function app deployed, let's add IP Filtering. In this case, we'll want to restrict the send function app to allow calls from Azure Monitor Webhooks.

#add network restriction on function app for sender.
$priority = 10
$index = 0
$ruleName = "AzureMonitorWebhookRule"
$azMonIPs.ForEach({
    $ipToAdd = "$_/32"; #this needs to use a /32 to add for the rule
    az functionapp config access-restriction add -g $rg -n $sendFaName --rule-name "$ruleName-$index" --action Allow --ip-address $ipToAdd --priority $priority;
    $priority += 10;
    $index += 1;
    Write-Host "Added Rule for $_"
})

We can now set up an Action Group for Azure Monitor to point to the Send Azure Function. Since the Azure Function uses an HTTP trigger, we'll need to Retrieve an Azure Function Key to ensure that Azure Monitor can trigger the function.

In this scenario we're using the function host key to call the Azure Function. There are other Azure Functions Authentication Options], but we're using this approach for simplicity, while in real-world scenarios this would likely need to be evaluated further.

$faSender = $(az functionapp show -n $sendFaName -g $rg)
$faSenderObject = $faSender | ConvertFrom-Json
$faSenderId = $faSenderObject.id
$faSendKey = $(az rest --method post --uri "$faSenderId/host/default/listKeys?api-version=2018-11-01" --query functionKeys.default --output tsv)

$faHostName = $faSenderObject.defaultHostName
$faSendURI = "https://$faHostName/$sendURIPath" + "?code=$faSendKey"

$sendActionGroup = $(az monitor action-group create -g $rg -n $sendActionGroupName -a azurefunction $sendActionGroupReceiverName $faSenderObject.id $sendFuncName $faSendURI useCommonAlertSchema)
$sendActionGroupObject = $sendActionGroup | ConvertFrom-Json

Let's set up an Alert for Azure Storage Account, as well as the test VM. We'll tie the Azure Monitor Metric Alerts to the Action Group that will trigger the 'Send' Azure Function.

We're using simple / easy to meet conditions just to see that the alerts will fire. For a real-world scenario, we'd want to update the alert conditions to reflect proper alerting conditions.

$saId = $(az storage account show -n $saName -g $rg --query id -o tsv)
##simpler metric alert on storage account.
$saAlert = $(az monitor metrics alert create -n $saAlertName -g $rg --scopes $saId --evaluation-frequency 1m --window-size 1m --action $sendActionGroupObject.id --description "Storage Transactions" --condition "total transactions >= 0")

$saAlert

$vmId = $(az vm show -g $rg -n $vmName --query id -o tsv)
$vmAlert = $(az monitor metrics alert create -n $vmAlertName -g $rg --scopes $vmId --evaluation-frequency 1m --window-size 1m --action $sendActionGroupObject.id --description "Percentage CPU Used" --condition "avg Percentage CPU >= 0")

$vmAlert

We can then make sure to start and stop the test VM.

az vm stop --ids $vmId
#wait for a minute to start up the VM again.
sleep 60
az vm start --ids $vmId

sleep 600
az vm stop --ids $vmId

Debugging Locally

We're also going to assume that we can use VS Code with the Azure Functions extension installed too. We'll also want to clone this repo to pull in the sample functions.

Be sure to install the latest version of Azure Functions Core Tools.

Work Around for Azure Functions Core Tools for Chocolatey install path: Azure/azure-functions-core-tools#693 (comment).

Run choco uninstall azure-functions-core-tools

Download nupkg file from here (see the Download Link for nupkg for Azure Functions Core Tools).

Open the nupkg in Package Explorer and edit the tools\chocolateyinstall.ps1 script (change x86 to x64 in the package URL). Be sure to save the nupkg with the changes!
choco install nugetpackageexplorer
Run this command in the folder where edited nupkg file is, and be sure to ignore the checksums since we did not make an update to the checksum but instead edited the URL for the nupkg.
choco install azure-functions-core-tools -source . --ignore-checksums

Debugging the Send Function

Assuming we have a local.settings.json file associated with the function app, we can fill in the details and rely on the functions core tools runtime to debug locally:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=mysa;AccountKey=key123==;EndpointSuffix=core.windows.net",
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "EVENT_HUB_HOSTNAME": "myeh.servicebus.windows.net",
    "EVENT_HUB_SAS_POLICY": "RootManageSharedAccessKey",
    "EVENT_HUB_SAS_KEY": "ehkey123=",
    "EVENT_HUB_NAME": "alert-eh",
    "my_RootManageSharedAccessKey_EVENTHUB" : "Endpoint=sb://myeh.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=ehkey123=;EntityPath=alert-eh"
  }
}

Make sure that the VSCode tasks.json points to the appropriate function to debug.

...
{
            "type": "func",
            "command": "host start",
            "problemMatcher": "$func-watch",
            "isBackground": true,
            "dependsOn": "pipInstall",
            "options": {
                "cwd": "${workspaceFolder}/Scenarios\\az-mon-eh-kv-python\\az-mon-eh-send-kv-python"
            }
        },
        {
            "label": "pipInstall",
            "type": "shell",
            "osx": {
                "command": "${config:azureFunctions.pythonVenv}/bin/python -m pip install -r requirements.txt"
            },
            "windows": {
                "command": "${config:azureFunctions.pythonVenv}\\Scripts\\python -m pip install -r requirements.txt"
            },
            "linux": {
                "command": "${config:azureFunctions.pythonVenv}/bin/python -m pip install -r requirements.txt"
            },
            "problemMatcher": [],
            "options": {
                "cwd": "${workspaceFolder}/Scenarios\\az-mon-eh-kv-python\\az-mon-eh-send-kv-python"
            }
        }
...

We can set a break point in the Send Function, and then hit F5 in VSCode.

In order to trigger the function, we can use a POST call with a sample payload. We can then post to the locally running Send API.

body = Get-Content .\az-mon-eh-send-kv-python\sample-payload.json | ConvertTo-Json

invoke-webrequest -Method POST -uri http://localhost:7071/api/SendAlertHttpTrigger -body $body

If we want to send to the event hub we created as part of the deployment script, we'll want to be sure to add the debugging client's IP address in order to have access to Event Hub.

Debugging the Receive Function

We'll want to make sure the function.json has a placeholder to point to the local settings.

{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "type": "eventHubTrigger",
      "name": "event",
      "direction": "in",
      "eventHubName": "alert-eh",
      "connection": "my_RootManageSharedAccessKey_EVENTHUB",
      "cardinality": "many",
      "consumerGroup": "myconsumergroup"
    }
  ]
}

Note, when we deploy, the connection should point to RootManageSharedAccessKey_EVENTHUB. The "my_" prepended in the connection works for local debugging. For deployment, we tested without using the "my_" prepended in the connection setting.

Assuming we have a local.settings.json file associated with the function app, we can fill in the details and rely on the functions core tools runtime to debug locally:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=myeh;AccountKey=ehkey123==;EndpointSuffix=core.windows.net",
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "my_RootManageSharedAccessKey_EVENTHUB": "Endpoint=sb://myeh.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=ehkey123=;EntityPath=alert-eh"
  }
}

Make sure that the VSCode tasks.json points to the appropriate function to debug.

...
{
            "type": "func",
            "command": "host start",
            "problemMatcher": "$func-watch",
            "isBackground": true,
            "dependsOn": "pipInstall",
            "options": {
                "cwd": "${workspaceFolder}/Scenarios\\az-mon-eh-kv-python\\az-mon-eh-receive-kv-python"
            }
        },
        {
            "label": "pipInstall",
            "type": "shell",
            "osx": {
                "command": "${config:azureFunctions.pythonVenv}/bin/python -m pip install -r requirements.txt"
            },
            "windows": {
                "command": "${config:azureFunctions.pythonVenv}\\Scripts\\python -m pip install -r requirements.txt"
            },
            "linux": {
                "command": "${config:azureFunctions.pythonVenv}/bin/python -m pip install -r requirements.txt"
            },
            "problemMatcher": [],
            "options": {
                "cwd": "${workspaceFolder}/Scenarios\\az-mon-eh-kv-python\\az-mon-eh-receive-kv-python"
            }
        }
...

We can set a break point in the function, and then hit F5 in VSCode. We can also use Service Bus Explorer and pass in the Azure Event Hub Connection String to send an event to the event hub to trigger the function.

If we want to send to the event hub we created as part of the deployment script, we'll want to be sure to add the debugging client's IP address in order to have access to Event Hub.

Validate The Scenario

We can check the alerts have fired in the resource group's alerts in the Azure Portal.

We should be able to check that the action group which is linked to the Azure Monitor Metric Alerts will point to the 'Send' Azure function.

We should be able to see that the Send Function App had successful calls.

We should be able to see that the Send Function App was triggered, and be able to examine details for the function. This depends on what was logged in the function.

We should be able to check for the key vault references in the app settings. Click on Manage Application Settings. Then we can see that the Key Vault references should be working (assuming network access and permissions are granted).

We should be able to check VNET Integration in the Azure Function network settings. Click on Manage Application Settings.

We should be able to check the IP Filtering in the Azure Function network settings. Click on Manage Application Settings.

We did not update the scm IP Filtering in this case, but this could be further restricted for appropriate access.

We can check on the Event Hub next. We can see that an Event was sent to Event Hub.

We can check that Event Hub allows access to our subnets in the Firewall and Virtual Network settings.

We can check that Key Vault allows access to our subnets in the Firewall and Virtual Network settings.

In the case that Azure Functions cannot resolve the Key Vault Reference due to IP filtering, we can add the outbound IP addresses associated with the Azure Functions that need to reach Azure Key Vault as a workaround. With Premium Azure Functions, the outbound IP addresses can possibly change and are not dedicated to the Azure Function; should we want dedicated outbound IP addresses, we should look into hosting the Azure Function in an App Service Environment.

We can check that Storage Account allows access to our subnets in the Firewall and Virtual Network settings.

If the Azure Function doesn't have access to the storage account, then the function runtime will have an error starting. The subnet reference should be sufficient in this case.

We can check that Receiver Azure Function by clicking on Monitor. We can see prior triggers and whether they were successful.

We can check that Receiver Azure Function Key Vault References in the Function app Settings.

We should be able to check VNET Integration in the Azure Function network settings. Click on Manage Application Settings.

We did not set IP filtering on the receiver Azure Function as this will reach out to Event Hub (as an outbound request) instead of Event Hub sending an inbound request. We did not update the scm IP Filtering in this case, but this could be further restricted for appropriate access.

When we are satisified with the test, we can clean up with the following az cli command:

az group delete -n $rg

Additional Notes

While this is an example for how we can use VNET integration, IP Filtering, Event Hubs, Key Vault, and Azure Functions to work with Azure Monitor, this is simply a POC to show how this can work together. Of course, additional lockdown details should be explored for real-world scenarios.

We can look into redundancy / geo-replication if required. The base deployment should be evaluated to see if it's suitable or can be adjusted for a given scenario, and then the base stamp can be deployed for potential geo-replication scenarios.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README-az-mon-eh-kv-python.md

README-az-mon-eh-kv-python.md

Azure Monitor Alerts with Azure Functions, Event Hub, and Key Vault Integration

Links

Deploy Components

Deployment Script Notes

Debugging Locally

Debugging the Send Function

Debugging the Receive Function

Validate The Scenario

Additional Notes

Files

README-az-mon-eh-kv-python.md

Latest commit

History

README-az-mon-eh-kv-python.md

File metadata and controls

Azure Monitor Alerts with Azure Functions, Event Hub, and Key Vault Integration

Links

Deploy Components

Deployment Script Notes

Debugging Locally

Debugging the Send Function

Debugging the Receive Function

Validate The Scenario

Additional Notes