new-sl

src/aws/codebuild_s3_cpu_recipe.md

@@ -0,0 +1,10 @@
+# codebuild-s3
+
+## Prerequisites
+
+* A [Github Account](https://github.com/join)
+* An [AWS Account](https://portal.aws.amazon.com/billing/signup)
+
+## Recipes 
+
+- [codebuild + S3](./codebuild_s3_cpu.md) ~ automated binary build + run using CPU

src/aws/index.md

@@ -15,4 +15,3 @@
 
 - [CodePipeline + Jenkins Build](./jenkins-pipeline.md) ~ automated binary builds on EC2 with Jenkins + CodePipeline
 
-- [codebuild + S3](./codebuild_s3_cpu.md) ~ automated binary build + run using CPU

src/azure/azure-spot-runner/.github/workflow/create-image.yml

@@ -0,0 +1,31 @@
+on: workflow_dispatch
+
+jobs:
+  job1:
+    runs-on: ubuntu-latest
+    name: Create Custom Linux Image
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Login via Az module
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZURE_CREDENTIALS}}
+
+    - name: Build and Distribute Custom VM Image
+      uses: azure/build-vm-image@v0
+      with:
+        resource-group-name: '<RESOURCE_GROUP_NAME>'
+        location: '<RESOURCE_LOCATION>'
+        managed-identity: '<AZURE-IDENTITY>' 
+        source-os-type: 'linux'
+        vm-size: "Standard_D2s_v3"
+        source-image-type: 'platformimage'
+        source-image: Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest
+        customizer-source: ${{ github.workspace }}/workflow-artifacts
+        customizer-script: |
+          ls -l /tmp/
+          ls -l /tmp/workflow-artifacts
+          sh /tmp/workflow-artifacts/setup-image.sh
+

src/azure/azure-spot-runner/.github/workflow/runner.yml

@@ -0,0 +1,63 @@
+on: workflow_dispatch
+
+jobs:
+  job1:
+    runs-on: ubuntu-latest
+    name: Launch Runner
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Get GitHub Runner Registration Token
+      run: |
+        set -e
+        RUNNER_TOKEN=$(curl -f -X POST \
+                         -H "Authorization: token ${{ secrets.PAT }}" \
+                         -H "Accept: application/vnd.github.v3+json" \
+                         https://api.github.com/repos/<YOUR_REPO_LOCATION>/actions/runners/registration-token | grep token | cut -d '"' -f 4)
+        echo $RUNNER_TOKEN
+        if [ -z "$RUNNER_TOKEN" ]; then
+          echo "Failed to retrieve the runner token."
+          exit 1
+        fi
+        echo "RUNNER_TOKEN=$RUNNER_TOKEN" >> $GITHUB_ENV
+      env:
+        PAT: ${{ secrets.PAT }}
+
+    - name: Modify cloud-init.txt
+      run: |
+        set -e
+        sed -i "2iRUNNER_TOKEN=$RUNNER_TOKEN" workflow-artifacts/cloud-init.txt
+        if ! grep -q "RUNNER_TOKEN=$RUNNER_TOKEN" workflow-artifacts/cloud-init.txt; then
+          echo "Modification with sed failed."
+          exit 1
+        fi
+
+    - name: Login via Az module
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZURE_CREDENTIALS}}
+
+    - name: Find the latest image URI
+      id: find_image
+      uses: azure/CLI@v1
+      with:
+        inlineScript: |
+          IMAGE_URI=$(az image list --resource-group githubVM --query "[-1].id" --output tsv)
+          echo "::set-output name=image_uri::$IMAGE_URI"
+
+
+    - name: CREATE VM
+      id: create_vm
+      uses: azure/CLI@v1
+      with:
+        inlineScript: |
+          az vm create --resource-group githubVM  --name "app-vm-${{ GITHUB.RUN_NUMBER }}"  --admin-username "runner" --admin-password "${{ secrets.VM_PASSWORD }}" --location  eastus \
+          --custom-data workflow-artifacts/cloud-init.txt \
+          --image "${{ steps.find_image.outputs.image_uri }}" \
+          --generate-ssh-keys
+
+    # At the end of your CREATE VM step
+    - name: Set VM Name Output
+      run: echo "::set-output name=vm_name::app-vm-${{ GITHUB.RUN_NUMBER }}"
+      id: vm_name

src/azure/azure-spot-runner/azure-spot-runner.md

@@ -0,0 +1,86 @@
+# Azure spot runner with Github Action.
+Using Ephemeral Infrastructure with Azure VMS as GitHub Action Runners to build Candle binary latter.
+
+## Requirements 
+* Setup [Azure Account](../hello-azure.md) and record the `subscription ID` you have from the setting. 
+
+* Create a [Personal Access Token (PAT)](https://github.com/settings/tokens/new?description=Azure+GitHub+Runner&scopes=repo) record as `$GitHubPAT`
+* 
+
+### Create an service principle with the following details:
+* AppID 
+* password 
+* tenant information 
+
+#### Create an service principal by running: 
+```
+$ az ad sp create-for-rbac -n "Name_of_Service_principal"
+```
+* This output contains all the infomation about your `AZURE_CREDENTIALS`. Please copy & save it to your local in order to set up VM after we setup the Spot Runner.  
+
+The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
+
+### Create an Azure Resource Group
+
+A resource group is a way to group services together so that you can keep track of them and delete them later with ease. Use the `az` CLI to accomplish this:
+```
+az group create --location eastus --name "<RESOURCE_GROUP_NAME>"
+```
+
+Keep that resource group name handy for other operations. In this repository the `$<RESOURCE_GROUP_NAME>` resource group is used throughout. Note the location as well. Make sure that the location (region) maps to the resources you want to create.
+
+
+
+### Create an Azure Key Vault
+You will store your GitHub PAT here so that it can later be retrieved by the VM.
+```
+$ az keyvault create --name candel-vms --resource-group githubVM --location eastus
+$ az keyvault secret set --vault-name candel-vms --name "GitHubPAT" --value $GITHUB_PAT
+```
+Replace $GITHUB_PAT with the value of the PAT created earlier
+
+### Assign an identity with permissions for Key Vault
+Now that the key vault is created, you need to create an identity and then allow resources in the resource group to be able to access it. You must give this identity a name, in this case we use GitHubVMs. Note this name will be used in other steps.
+
+```
+$ az identity create --name GitHubVMs --resource-group githubVM --location eastus
+
+```
+Capture the Principal ID which will be used for the value for --object-id later. You can retrieve it again by using:
+
+```
+$ az identity show --name GitHubVMs --resource-group githubVM
+```
+
+Use the object id to set the policy, replace $OBJECT_ID with the one you found in the previous command:
+```
+$ az keyvault set-policy --name candel-vms --object-id <$BOJECT_ID> --secret-permissions get
+```
+
+### Verify you can get the PAT with the following command:
+```
+az keyvault secret show --name "GitHubPAT" --vault-name candel-vms --query value -o tsv
+```
+
+
+### Provide a role to VMs
+Assign a role to the VMs so that they have enough permissions to write the image when getting created. Start by finding the principalId which will then be needed for the next step:
+
+```
+az identity show --name GitHubVMs --resource-group githubVM --query principalId
+```
+
+With the principalId you can assign it to the VMs now:
+
+```
+az role assignment create --assignee <principal_ID> --role Contributor --resource-group githubVM
+```
+
+
+### Trigger the create image run
+Now you are ready to create the image. Run it manually and make sure it works correctly. If succesful, an image will be created for you which you can query with the following command:
+```
+az image list --resource-group githubVM --output table
+```
+
+

src/azure/azure-spot-runner/candle-VM.md

@@ -0,0 +1,55 @@
+# Using Github Action to build Candle image on Azure Spot Runner VM
+
+## Prerequisites
+* A [Github Account](https://github.com/join)
+* An [Azure Account](https://azure.microsoft.com/en-us/free)
+* Finish setup [Azure Spot Runner](./azure-spot-runner.md)
+
+## Create Github Secrets
+1. In Github, go to your repository. 
+2. Go to **Settings** in the navigation menu.
+3. Select **Securigy > Secrets and variables > Actions**. 
+4. Select **New Repository Secret**
+5. Paste the entire [Json output](https://github.com/ScottLL/candle-cookbook/blob/e96d35af185ec67f382e1ac891590197705d7ffd/src/azure/hello-azure.md#L49-L50) from the Azure CLI command into the github action secret's value field. Give the secret the name: `AZURE_CREDENTIALS`. 
+5. Select **Add secret.**
+
+## Build your image
+Use the [Create-Image action](./.github/workflow/create-image.yml) to create a custom virtual machine image.
+
+```
+resource-group-name: '<RESOURCE_GROUP_NAME>'
+location: '<RESOURCE_LOCATION>'
+managed-identity: '<AZURE-IDENTITY>'       
+```
+Replace the placeholders, where you created them in the Hello-azure.md, 
+
+* [RESOURCE_GROUP_NAME](https://github.com/ScottLL/candle-cookbook/blob/e96d35af185ec67f382e1ac891590197705d7ffd/src/azure/hello-azure.md#L58)
+* [RESOURCE_LOCATION](https://github.com/ScottLL/candle-cookbook/blob/e96d35af185ec67f382e1ac891590197705d7ffd/src/azure/hello-azure.md#L58)
+* [AZURE-IDENTITY](https://github.com/ScottLL/candle-cookbook/blob/e96d35af185ec67f382e1ac891590197705d7ffd/src/azure/hello-azure.md#L73)
+
+You can also change the VM type (CPU & GPU) by changing `<vm-size>`, `<source-image-type>`, `<source-image>`
+
+
+## Start your Runner and Build the Binary
+After you create the VM image, now we can start run it by using the [Runner Action](./.github/workflow/runner.yml). 
+
+Note you can add your customer code inside the cloud-init.txt for specific need.
+
+The example code in cloud-init.txt is set up to build the candle whisper binary and deploy the binary into the Github release. 
+
+* Note, after the runner build, it might takes up to 15 mins for this cpu VM to finish the work and deploy the whisper release in Github. 
+
+### Debugging on VM
+You can check the log from Azure Portal:
+1. Go to the Azure Portal.
+2. Navigate to the "Virtual Machines" section.
+3. Find and select the VM you created (app-vm).
+4. Under the "Support + troubleshooting" section, find and click on "Serial console".
+5. The serial console will open in a new window. You might have to wait for a few moments as the console establishes a connection to the VM.
+
+
+
+
+
+
+

src/azure/azure-spot-runner/workflow-artifacts/cloud-init.txt

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Define a working directory
+export HOME=/root
+WORK_DIR="/opt/actions-runner"
+
+chown -R runner $WORK_DIR
+su - runner -c "$WORK_DIR/config.sh --unattended --url https://github.com/nogibjj/candle_scott_Azure_vm --token $RUNNER_TOKEN"
+nohup su - runner -c "$WORK_DIR/run.sh" &
+
+apt-get update
+apt-get install -y build-essential pkg-config libssl-dev protobuf-compiler jq git-lfs curl jq
+
+git lfs install
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+export PATH="$HOME/.cargo/bin:$PATH"
+
+git clone https://github.com/huggingface/candle.git /root/candle
+cd /root/candle
+
+if cargo build --example whisper --release; then
+    FILE="/root/candle/target/release/examples/whisper"
+    MIME_TYPE=$(file -b --mime-type "$FILE")
+    TAG="v$(date +%Y%m%d%H%M%S)"
+    RELEASE_NAME="CPU Whisper Binary Release"
+    GITHUB_TOKEN="$GITHUB_TOKEN"  # Token passed from GitHub Actions
+
+    CREATE_RELEASE_RESPONSE=$(curl -X POST -H "Authorization: token $GITHUB_TOKEN" -H "Content-Type: application/json" -d "{\"tag_name\": \"$TAG\", \"name\": \"$RELEASE_NAME\"}" "https://api.github.com/repos/nogibjj/candle_scott_Azure_vm/releases")
+    UPLOAD_URL=$(echo $CREATE_RELEASE_RESPONSE | jq -r .upload_url | sed "s/{?name,label}//")
+
+    curl -X POST -H "Authorization: token $GITHUB_TOKEN" -H "Content-Type: $MIME_TYPE" --data-binary @"$FILE" "$UPLOAD_URL?name=$(basename "$FILE")"
+else
+    echo "Cargo build failed."
+fi
+

src/azure/azure-spot-runner/workflow-artifacts/setup-image.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+# Setup the runner to have the Azure CLI pre-installed as well as the Actions
+# Runner
+
+# Define a working directory
+WORK_DIR="/opt/actions-runner"
+
+# Install Azure CLI, should not use sudo
+curl -sL https://aka.ms/InstallAzureCLIDeb | bash
+
+# Create a folder
+mkdir -p $WORK_DIR && cd $WORK_DIR
+
+# Download the latest runner package
+curl -O -L https://github.com/actions/runner/releases/download/v2.310.2/actions-runner-linux-x64-2.310.2.tar.gz
+
+# Extract the installer
+tar xzf $WORK_DIR/actions-runner-linux-x64-2.310.2.tar.gz

src/azure/hello-azure.md

@@ -0,0 +1,31 @@
+# Hello, Candle on Azure.
+
+## Prerequisites
+* A [Github Account](https://github.com/join)
+* An [Azure Account](https://azure.microsoft.com/en-us/free)
+
+
+# Azure Spot Runner build
+
+
+### Install Azure Cli 
+```
+$ curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
+```
+
+### login in to Azure 
+```
+$ az login 
+
+# if above not working run:  
+$ az login --use-device-code
+```
+
+```
+# check your Azure account information
+$ az account show
+
+# if you have multiple accounts, you need to set the specific account you want to use: 
+$ az account set --subscription <subscription_ID>
+```
+* the `subscription ID` is the `id` from result of az account show.

src/azure/index.md

@@ -9,4 +9,6 @@
 
 **Getting Started**
 
-- [Hello, Azure!](./hello-azure.md) ~ Hello, Candle! on Azure
+- [Hello, Azure!](./hello-azure.md) ~ Hello, Candle on Azure
+- [Create Azure VM Image](./azure-spot-runner/candle-VM.md) ~ Create the VM image to hosting the LLMs binary. 
+- [Azure spot runner](./azure-spot-runner/azure-spot-runner.md) Run the spot runner to build the binary and deploy the binary to Github release.