🫱🏾🫲🏼 Trust ~ ethically sourced data and end-to-end transparency.
🔒 Privacy ~ secure, on-device inference without data sharing.
🌱 Sustainability ~ optimize efficiency to minimize our carbon footprint.
+We welcome contributions from anyone who aligns with Our Mission and Our Principles.
+Get started by reading our Code of Conduct then check out the Contributor Guidelines.
Disclaimer:
-AWS/Azure builds may incur charges. It is your responsibility to understand the associated resource costs and useage rates. We are not liable for any charges incurred from the utilization of these services.
-⚠️ IMPORTANT:
+AWS/Azure builds may incur charges. It is your responsibility to understand the associated resource costs and useage rates. We are not liable for any charges incurred from the utilization of these services.
Our goal is to document each stage of a fully transparent LLM development cycle
We welcome contributions from anyone who aligns with Our Mission and Our Principles.
-Get started by reading our Code of Conduct then check out the Contributor Guidelines.
-Get to know our Community Leaders
+🫱🏾🫲🏼 Trust ~ ethically sourced data and end-to-end transparency.
🔒 Privacy ~ secure, on-device inference without data sharing.
🌱 Sustainability ~ optimize efficiency to minimize our carbon footprint.
+We welcome contributions from anyone who aligns with Our Mission and Our Principles.
+Get started by reading our Code of Conduct then check out the Contributor Guidelines.
Disclaimer:
-AWS/Azure builds may incur charges. It is your responsibility to understand the associated resource costs and useage rates. We are not liable for any charges incurred from the utilization of these services.
-⚠️ IMPORTANT:
+AWS/Azure builds may incur charges. It is your responsibility to understand the associated resource costs and useage rates. We are not liable for any charges incurred from the utilization of these services.
Our goal is to document each stage of a fully transparent LLM development cycle
We welcome contributions from anyone who aligns with Our Mission and Our Principles.
-Get started by reading our Code of Conduct then check out the Contributor Guidelines.
-Get to know our Community Leaders
+🫱🏾🫲🏼 Trust ~ ethically sourced data and end-to-end transparency.
🔒 Privacy ~ secure, on-device inference without data sharing.
🌱 Sustainability ~ optimize efficiency to minimize our carbon footprint.
+We welcome contributions from anyone who aligns with Our Mission and Our Principles.
+Get started by reading our Code of Conduct then check out the Contributor Guidelines.
Disclaimer:
-AWS/Azure builds may incur charges. It is your responsibility to understand the associated resource costs and useage rates. We are not liable for any charges incurred from the utilization of these services.
-⚠️ IMPORTANT:
+AWS/Azure builds may incur charges. It is your responsibility to understand the associated resource costs and useage rates. We are not liable for any charges incurred from the utilization of these services.
Our goal is to document each stage of a fully transparent LLM development cycle
We welcome contributions from anyone who aligns with Our Mission and Our Principles.
-Get started by reading our Code of Conduct then check out the Contributor Guidelines.
-Get to know our Community Leaders
+Minimum requirements for GPU targetted binaries
diff --git a/searchindex.js b/searchindex.js index 8bfeffa..032402d 100644 --- a/searchindex.js +++ b/searchindex.js @@ -1 +1 @@ -Object.assign(window.search, {"doc_urls":["home.html#candle-cookbook","home.html#-the-mission","home.html#-the-principles","home.html#-the-recipes","home.html#official-candle-docs","home.html#roadmap","home.html#-contributing","home.html#-our-team","local/index.html#local-builds","local/index.html#requirements","local/index.html#recipes","local/hello-candle.html#hello-candle","local/hello-candle.html#install-rust-via--rustup","local/hello-candle.html#configure--candle","local/hello-candle.html#build--run-binaries","aws/index.html#aws-builds","aws/index.html#requirements","aws/index.html#recipes","aws/hello-aws.html#hello-candle-on-aws","aws/hello-aws.html#prerequisites","aws/hello-aws.html#create-candle-ec2-iam-user","aws/hello-aws.html#launch-gpu-enabled-aws-ec2-compute-instance","aws/hello-aws.html#connect-to-ec2-via-ssh","aws/hello-aws.html#verify-ec2-cudacudnn","aws/hello-aws.html#install-rust-via--rustup","aws/hello-aws.html#install-aws-cli","aws/hello-aws.html#configure-candle-ec2-iam-user","aws/hello-aws.html#configure--candle","aws/hello-aws.html#build--run-binaries","aws/hello-aws.html#store-binaries-in-s3","aws/jenkins-pipeline.html#automate-binary-builds-with-jenkins--codepipeline","aws/jenkins-pipeline.html#prerequisites","aws/jenkins-pipeline.html#add-permissions-to-candle-ec2-iam-role","aws/jenkins-pipeline.html#integrate-jenkins-with-codepipeline","aws/jenkins-pipeline.html#download-binaries","aws/jenkins-pipeline.html#references","azure/index.html#azure-builds","azure/index.html#requirements","azure/index.html#recipes"],"index":{"documentStore":{"docInfo":{"0":{"body":0,"breadcrumbs":4,"title":2},"1":{"body":6,"breadcrumbs":3,"title":1},"10":{"body":6,"breadcrumbs":3,"title":1},"11":{"body":0,"breadcrumbs":6,"title":2},"12":{"body":18,"breadcrumbs":8,"title":4},"13":{"body":5,"breadcrumbs":6,"title":2},"14":{"body":31,"breadcrumbs":7,"title":3},"15":{"body":0,"breadcrumbs":4,"title":2},"16":{"body":4,"breadcrumbs":3,"title":1},"17":{"body":21,"breadcrumbs":3,"title":1},"18":{"body":0,"breadcrumbs":8,"title":3},"19":{"body":4,"breadcrumbs":6,"title":1},"2":{"body":20,"breadcrumbs":3,"title":1},"20":{"body":12,"breadcrumbs":10,"title":5},"21":{"body":58,"breadcrumbs":12,"title":7},"22":{"body":75,"breadcrumbs":9,"title":4},"23":{"body":9,"breadcrumbs":8,"title":3},"24":{"body":18,"breadcrumbs":9,"title":4},"25":{"body":15,"breadcrumbs":8,"title":3},"26":{"body":36,"breadcrumbs":10,"title":5},"27":{"body":5,"breadcrumbs":7,"title":2},"28":{"body":31,"breadcrumbs":8,"title":3},"29":{"body":30,"breadcrumbs":8,"title":3},"3":{"body":76,"breadcrumbs":3,"title":1},"30":{"body":0,"breadcrumbs":10,"title":5},"31":{"body":18,"breadcrumbs":6,"title":1},"32":{"body":17,"breadcrumbs":11,"title":6},"33":{"body":359,"breadcrumbs":8,"title":3},"34":{"body":20,"breadcrumbs":7,"title":2},"35":{"body":8,"breadcrumbs":6,"title":1},"36":{"body":0,"breadcrumbs":4,"title":2},"37":{"body":4,"breadcrumbs":3,"title":1},"38":{"body":7,"breadcrumbs":3,"title":1},"4":{"body":6,"breadcrumbs":5,"title":3},"5":{"body":31,"breadcrumbs":3,"title":1},"6":{"body":14,"breadcrumbs":3,"title":1},"7":{"body":3,"breadcrumbs":3,"title":1},"8":{"body":0,"breadcrumbs":4,"title":2},"9":{"body":53,"breadcrumbs":3,"title":1}},"docs":{"0":{"body":"","breadcrumbs":"Candle Cookbook » Candle Cookbook","id":"0","title":"Candle Cookbook"},"1":{"body":"Democratize access to state of the art AI models.","breadcrumbs":"Candle Cookbook » 🚀 The Mission:","id":"1","title":"🚀 The Mission:"},"10":{"body":"Getting Started Hello, Candle! ~ start here","breadcrumbs":"Local Builds » Recipes","id":"10","title":"Recipes"},"11":{"body":"","breadcrumbs":"Local Builds » Hello, Candle! » Hello, Candle!","id":"11","title":"Hello, Candle!"},"12":{"body":"curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable # Set path\nsource \"$HOME/.cargo/env\"","breadcrumbs":"Local Builds » Hello, Candle! » Install Rust via rustup","id":"12","title":"Install Rust via rustup"},"13":{"body":"git clone https://github.com/huggingface/candle.git\ncd candle","breadcrumbs":"Local Builds » Hello, Candle! » Configure Candle","id":"13","title":"Configure Candle"},"14":{"body":"See all Candle example models here Example: # CPU build\ncargo build --example falcon --release # CUDA + cuDNN build\ncargo build --example falcon --features cuda,cudnn --release # Run binary\ncd target/release/examples\n./falcon --prompt \"who invented the lightbulb\"","breadcrumbs":"Local Builds » Hello, Candle! » Build & Run Binaries","id":"14","title":"Build & Run Binaries"},"15":{"body":"","breadcrumbs":"AWS Builds » AWS Builds","id":"15","title":"AWS Builds"},"16":{"body":"A Github Account An AWS Account","breadcrumbs":"AWS Builds » Requirements","id":"16","title":"Requirements"},"17":{"body":"Getting Started Hello, Candle on AWS! ~ Hello, Candle! on AWS EC2 + S3 CI/CD Pipelines CodePipeline + Jenkins Build ~ automated binary builds on EC2 with Jenkins + CodePipeline","breadcrumbs":"AWS Builds » Recipes","id":"17","title":"Recipes"},"18":{"body":"","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Hello, Candle on AWS!","id":"18","title":"Hello, Candle on AWS!"},"19":{"body":"A Github Account An AWS Account","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Prerequisites","id":"19","title":"Prerequisites"},"2":{"body":"🫱🏾🫲🏼 Trust ~ ethically sourced data and end-to-end transparency. 🔒 Privacy ~ secure, on-device inference without data sharing. 🌱 Sustainability ~ optimize efficiency to minimize our carbon footprint.","breadcrumbs":"Candle Cookbook » 🌏 The Principles:","id":"2","title":"🌏 The Principles:"},"20":{"body":"From AWS IAM Console >> Users >> Create User \"Candle-EC2\" >> Attach Policies Directly: AmazonS3FullAccess","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Create Candle-EC2 IAM User","id":"20","title":"Create Candle-EC2 IAM User"},"21":{"body":"Open the AWS EC2 Console Configure a Ubuntu 20.04 Deep Learning Base GPU AMI image + G5.##xlarge instance type (see here for charges and instance capcity) Create new key pair and save .pem file Allow SSH traffic Launch Instance From EC2 landing page >> Start Instance Gotchas You may need to request capacity increase from AWS to handle larger G5.##xlarge instances Move .pem file from downloads to ~/.ssh using mv ~/Downloads/candle-key.pem ~/.ssh","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Launch GPU Enabled AWS EC2 Compute Instance","id":"21","title":"Launch GPU Enabled AWS EC2 Compute Instance"},"22":{"body":"Install Remote - SSH from VSCode Extensions Add new SSH Connection From EC2 landing page >> Connect >> SSH client >> Copy commands to VSCode # NB: edit path to .pem file as needed\nchmod 400 ~/.ssh/candle-key.pem\nssh -i \"~/.ssh/candle-key.pem\" ubuntu@ec2-##-##.us-east-1.compute.amazonaws.com Update .config file and validate format as follows Host ec2-##-###.us-east-1.compute.amazonaws.com HostName ec2-##-###.us-east-1.compute.amazonaws.com IdentityFile ~/.ssh/candle-key.pem User ubuntu Confirm remote host platform (Linux) and fingerprint (Continue) Launch terminal on remote host Gotchas If you stop and restart your EC2 instance, you will need to update the IP address in your .config file","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Connect to EC2 via SSH","id":"22","title":"Connect to EC2 via SSH"},"23":{"body":"nvidia-smi --query-gpu=compute_cap --format=csv\nnvcc --version whereis cudnn.h","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Verify EC2 CUDA/cuDNN","id":"23","title":"Verify EC2 CUDA/cuDNN"},"24":{"body":"curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable # Set path\nsource \"$HOME/.cargo/env\"","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Install Rust via rustup","id":"24","title":"Install Rust via rustup"},"25":{"body":"curl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"\nunzip awscliv2.zip\nsudo ./aws/install # Check install\naws --version","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Install AWS CLI","id":"25","title":"Install AWS CLI"},"26":{"body":"IAM Console >> Users >> Candle-EC2 >> Security Credentials >> Create Access Key \"EC2-CLI\" From EC2 SSH terminal aws configure\n# Copy-Paste Candle-EC2 Access Key ID\n# Copy-Paste Candle-EC2 Secret Access Key\n# Default region name: us-east-1","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Configure Candle-EC2 IAM User","id":"26","title":"Configure Candle-EC2 IAM User"},"27":{"body":"git clone https://github.com/huggingface/candle.git\ncd candle","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Configure Candle","id":"27","title":"Configure Candle"},"28":{"body":"See all Candle example models here Example: # CPU build\ncargo build --example falcon --release # CUDA + cuDNN build\ncargo build --example falcon --features cuda,cudnn --release # Run binary\ncd target/release/examples\n./falcon --prompt \"who invented the lightbulb\"","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Build & Run Binaries","id":"28","title":"Build & Run Binaries"},"29":{"body":"From AWS S3 Console >> Create S3 Bucket i.e my-candle-binaries # Copy model binary from EC2 to S3 cd target/release/examples\naws s3 cp quantized s3://my-candle-binaries IMPORTANT! Terminate any AWS resources to prevent unexpected charges.","breadcrumbs":"AWS Builds » Hello, Candle on AWS! » Store Binaries in S3","id":"29","title":"Store Binaries in S3"},"3":{"body":"Local Builds AWS Builds Azure Builds Minimum requirements for GPU targetted binaries For CUDA enabled builds using --features cuda: NVIDIA GPU with CUDA compute capabilitity > X.X CUDA Toolkit >= X.X For cuDNN optimized builds using --features cuda, cudnn: NVIDIA GPU with CUDA compute capabilitity > X.X CUDA Toolkit >= X.X cuDNN >= X.X Verify CUDA/cuDNN: # Verify CUDA\nnvidia-smi --query-gpu=compute_cap --format=csv\nnvcc --version # Verify cuDNN\nwhereis cudnn.h Disclaimer : AWS/Azure builds may incur charges. It is your responsibility to understand the associated resource costs and useage rates. We are not liable for any charges incurred from the utilization of these services.","breadcrumbs":"Candle Cookbook » 🍳 The Recipes:","id":"3","title":"🍳 The Recipes:"},"30":{"body":"","breadcrumbs":"AWS Builds » CodePipeline + Jenkins Build » Automate Binary Builds with Jenkins + CodePipeline","id":"30","title":"Automate Binary Builds with Jenkins + CodePipeline"},"31":{"body":"Complete the Hello, Candle! on AWS tutorial to setup your EC2 Instance, S3 Bucket and Candle-EC2 IAM User. A Github Account An AWS Account","breadcrumbs":"AWS Builds » CodePipeline + Jenkins Build » Prerequisites","id":"31","title":"Prerequisites"},"32":{"body":"From AWS IAM Console >> Users >> Candle-EC2 >> Add permissions >> Attach existing policies directly Add AWSCodePipelineCustomActionAccess policy >> Save changes","breadcrumbs":"AWS Builds » CodePipeline + Jenkins Build » Add Permissions to Candle-EC2 IAM Role","id":"32","title":"Add Permissions to Candle-EC2 IAM Role"},"33":{"body":"Add Security Group to EC2 From EC2 landing page >> Network & Security >> Security Groups Create Security Group >> Name: \"WebConnectEC2\" >> Description: \"Allow SSH and HTTP traffic to EC2\" Add the following inbound rules. NB: Source = \"Anywhere\" is not recommended for production environments. image Click EC2 instance >> Actions >> Networking >> Change Security Groups >> Select \"WebConnectEC2\" >> Save Install Jenkins on EC2 # Get Jenkins files\ncurl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \\ /usr/share/keyrings/jenkins-keyring.asc > /dev/null\necho deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \\ https://pkg.jenkins.io/debian-stable binary/ | sudo tee \\ /etc/apt/sources.list.d/jenkins.list > /dev/null # Install Jenkins\nsudo apt-get update \\ &&\nsudo apt-get install jenkins # Check install\njenkins --version # Allow Jenkins service to start at boot\nsudo systemctl enable jenkins # Launch Jenkins\nsudo systemctl start jenkins Create Jenkins Admin/Login to Jenkins Click EC2 instance >> Details >> Copy EC2 Public IPv4 DNS From browser navigate to http://