From d9dbf38848bf579caea9c23cf039c9ee7d4f4c34 Mon Sep 17 00:00:00 2001 From: Prudhvi Godithi Date: Thu, 28 Jul 2022 13:03:20 -0400 Subject: [PATCH] MacOS agents support default /bin/bash and remove nodeProperties in CDK code. (#175) * parent 83a6200a1a322c7fa5b2ba9ea3a2ce80333b50dd author pgodithi 1651769732 -0400 committer pgodithi 1658274990 -0400 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEuaPMeSek3p9IWNzDMSDf0RO9/aIFAmLXRK4ACgkQMSDf0RO9 /aKKpQ/9FQ92spUtBEdt2lf4Kz6c2x7+eunr5kzzxZPB8fj28AFWBtGP/NA6a/WK BjRgtxZ4060PCDUtbcnfFy8mg8PdQa03FpofQKr6exSfDTI+TNISJAkcI2HIWzqx ehiFRDGWG577fgrnk67xxd6LxD8sO8EHWBJzZWKrYA3z7f0Q05n1q2osm/jhDYgK cEDQBHZJVi4htO2sBGJO3caqCbqt0EphxYy+uua5On4a2ZgWlBUxjqoaZ6X/LrnR 6IoN8PYVUtyrkbWkkXB+0T4PPDYP8DwnCrCsDQxQyYQ7tBqnR3hQ6nfga2d4Dry5 Dz9oqPoWZjDJNBAf+B6NAmwmyIZHn2lnVL/b8GvwGjSOixyUqaptRKVrg/PwIT+I i2CeMuGIxIG8pycC+wruSNoNqV26Vm7rFSOC4VSvOXneQUStMrAEM+LjcCDtG8bF 0837RaB4Vj37QfkXJyQl2JLTvw6TUKbZuCXqSTLF6oy4nfKN1wBPhfmgjRt146o3 VKt9iZ/q5kOEAPhEHJkHWoC/C+jaMRkVk0ZV5CoujQcIPZX2HfA88RogNZe0Ommy ayc+gLw8oBiRBeKkWuQLrLtrBYCLKrIK5qjEEj9MqLxbLSlDRXJjvZvPSUGqS2wg a8KKD0i9faPpRIuViJRqHztQBXVsv0UsYFMeeid00lJm2F4dmBk= =c8LN -----END PGP SIGNATURE----- fixed IAM roles and code cleanup Signed-off-by: pgodithi Fix tests Signed-off-by: pgodithi Added assume role feature Signed-off-by: pgodithi Added assume role feature Signed-off-by: pgodithi Added assume role feature Signed-off-by: pgodithi Added assume role feature Signed-off-by: pgodithi Added assume role feature Signed-off-by: pgodithi Added 2 new plugins Signed-off-by: pgodithi Add readme doc Signed-off-by: pgodithi Added docker resources Signed-off-by: pgodithi Added docker resources Signed-off-by: pgodithi Bump minor version Signed-off-by: pgodithi change docker tag to 2.332.3-lts Signed-off-by: pgodithi added jenkinsOpts Signed-off-by: pgodithi added jenkinsOpts Signed-off-by: pgodithi added jenkinsOpts Signed-off-by: pgodithi Added IAM required imports Signed-off-by: pgodithi Added agent node unit tests Signed-off-by: pgodithi Added agent node unit tests Signed-off-by: pgodithi Added Managed SSM Policy to Agent instance role (#117) Signed-off-by: Rishabh Singh fix agentAssumeRole Signed-off-by: pgodithi Ecr public Signed-off-by: pgodithi Added AmazonSSMManagedInstanceCore Signed-off-by: pgodithi Added AmazonSSMManagedInstanceCore policy to agents. (#124) * fixed IAM roles and code cleanup Signed-off-by: pgodithi * Fix tests Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added 2 new plugins Signed-off-by: pgodithi * Add readme doc Signed-off-by: pgodithi * Added docker resources Signed-off-by: pgodithi * Added docker resources Signed-off-by: pgodithi * Bump minor version Signed-off-by: pgodithi * change docker tag to 2.332.3-lts Signed-off-by: pgodithi * added jenkinsOpts Signed-off-by: pgodithi * added jenkinsOpts Signed-off-by: pgodithi * added jenkinsOpts Signed-off-by: pgodithi * Added IAM required imports Signed-off-by: pgodithi * Added agent node unit tests Signed-off-by: pgodithi * Added agent node unit tests Signed-off-by: pgodithi * fix agentAssumeRole Signed-off-by: pgodithi * Ecr public Signed-off-by: pgodithi * Added AmazonSSMManagedInstanceCore Signed-off-by: pgodithi Added new jenkins agents and logging bug-fix (#126) Signed-off-by: Rishabh Singh Added new output parameters and updated jenkins image tag (#128) Signed-off-by: Rishabh Singh use jenkins-2.332.3 jdk8 base image (#129) Signed-off-by: Rishabh Singh added support for QEMU emulators (#131) Signed-off-by: Rishabh Singh Add Docker restart policy (#137) * fixed IAM roles and code cleanup Signed-off-by: pgodithi * Fix tests Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added assume role feature Signed-off-by: pgodithi * Added 2 new plugins Signed-off-by: pgodithi * Add readme doc Signed-off-by: pgodithi * Added docker resources Signed-off-by: pgodithi * Added docker resources Signed-off-by: pgodithi * Bump minor version Signed-off-by: pgodithi * change docker tag to 2.332.3-lts Signed-off-by: pgodithi * added jenkinsOpts Signed-off-by: pgodithi * added jenkinsOpts Signed-off-by: pgodithi * added jenkinsOpts Signed-off-by: pgodithi * Added IAM required imports Signed-off-by: pgodithi * Added agent node unit tests Signed-off-by: pgodithi * Added agent node unit tests Signed-off-by: pgodithi * fix agentAssumeRole Signed-off-by: pgodithi * Ecr public Signed-off-by: pgodithi * Added AmazonSSMManagedInstanceCore Signed-off-by: pgodithi * added docker-compose restart policy Signed-off-by: pgodithi Add markup formatter as safe/html disable syntax highlighting to allow html (#138) Signed-off-by: Peter Zhu Set cache to false for aws secret manager plugin (#140) Signed-off-by: Peter Zhu Add environment variables using configuration as code (#136) * Add environment variables using configuration as code Signed-off-by: Sayali Gaikawad Install workflow-api plugin to get latest version and remove lockable permissions (#141) * Install workflow-api plugin to get latest version Signed-off-by: Sayali Gaikawad * Remove lockable permissions Signed-off-by: Sayali Gaikawad [Bug-fix]: Handle multiple colons and space (#142) * Handle multiple colons and space Signed-off-by: Sayali Gaikawad * Handle multiple space Signed-off-by: Sayali Gaikawad Move env variabled to yaml (#143) * Move to yaml Signed-off-by: Sayali Gaikawad add ubuntu agent and cdn with lambda@edge resource for public access (#134) Signed-off-by: Rishabh Singh Reload configuration as code via curl instead of cli (#145) * Reload configuration as code via curl instead of cli Signed-off-by: Sayali Gaikawad Main/Agent Node new setups on tag and executors (#147) * Main/Agent Node new setups on tag and executors Signed-off-by: Peter Zhu * Remove DCO check as it is replaced by dco app Signed-off-by: Peter Zhu * Remove stack in node ts as it is not used anymore Signed-off-by: Peter Zhu Fix Ubuntu agent init script bug (#148) Signed-off-by: Peter Zhu Add Ubuntu Single Host for Gradle Check (#149) * Add Ubuntu Single Host for Gradle Check Signed-off-by: Peter Zhu * Change default idle time to 60min Signed-off-by: Peter Zhu Raise the gradle check runner to have c518xlarge as in Fork Jenkins (#150) Signed-off-by: Peter Zhu New cmds reduce AL2 failure and increase gradle check to c524xlarge runner (#151) Signed-off-by: Peter Zhu Add jdk14 as it is required by OS 1.x gradle check (#152) Signed-off-by: Peter Zhu Migrate perf test agent setups to public jenkins (#153) Signed-off-by: Peter Zhu Add Mac agent support Signed-off-by: pgodithi Add Mac agent support: Added README Signed-off-by: pgodithi Add Mac agent support: Added README Signed-off-by: pgodithi Add Mac agent support: Added README Signed-off-by: pgodithi Add Mac agent support: Added README Signed-off-by: pgodithi Add Mac agent support: Added README Signed-off-by: pgodithi Add Mac agent support: Added max-len, eqeqeq Signed-off-by: pgodithi Add Mac agent support: Added tests Signed-off-by: pgodithi Add Mac agent support: Added tests Signed-off-by: pgodithi Add Mac agent support: Added tests Signed-off-by: pgodithi * Add Mac agent support: fix conflicts Signed-off-by: pgodithi * Add Mac agent support: fix conflicts Signed-off-by: pgodithi * MacOS agents: packer Signed-off-by: pgodithi * MacOS agents: packer, fix conflicts Signed-off-by: pgodithi * MacOS agents: packer, fix conflicts Signed-off-by: pgodithi * MacOS agents: packer fix suggestions Signed-off-by: pgodithi * MacOS agents: packer fix suggestions Signed-off-by: pgodithi * MacOS agents: packer fix suggestions Signed-off-by: pgodithi * Fix ci issue #174 Signed-off-by: prudhvigodithi * Fix ci issue #174 Signed-off-by: prudhvigodithi * Fix ci issue #174 Signed-off-by: prudhvigodithi * Fix ci issue #174 Signed-off-by: prudhvigodithi --- lib/compute/agent-node-config.ts | 13 --------- lib/compute/agent-nodes.ts | 2 +- packer/README.md | 1 + packer/files/macos/bash_profile | 3 +++ packer/files/macos/bashrc | 5 ++++ packer/jenkins-agent-macos12-x64.json | 21 +++++++++++++++ packer/scripts/macos/macos-agentsetup.sh | 34 +++++++++++++++++++----- 7 files changed, 58 insertions(+), 21 deletions(-) create mode 100644 packer/files/macos/bash_profile create mode 100644 packer/files/macos/bashrc diff --git a/lib/compute/agent-node-config.ts b/lib/compute/agent-node-config.ts index de08cce5..4930ff1e 100644 --- a/lib/compute/agent-node-config.ts +++ b/lib/compute/agent-node-config.ts @@ -242,19 +242,6 @@ export class AgentNodeConfig { ], tenancy: 'Host', type: config.instanceType, - nodeProperties: [ - { - envVars: { - env: [ - { - key: 'Path', - /* eslint-disable max-len */ - value: '/usr/local/opt/python@3.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/bin', - }, - ], - }, - }, - ], useEphemeralDevices: false, }; } diff --git a/lib/compute/agent-nodes.ts b/lib/compute/agent-nodes.ts index ce463cd7..705be809 100644 --- a/lib/compute/agent-nodes.ts +++ b/lib/compute/agent-nodes.ts @@ -122,7 +122,7 @@ export class AgentNodes { remoteUser: 'ec2-user', maxTotalUses: -1, numExecutors: 6, - amiId: 'ami-0379811a08268a97e', + amiId: 'ami-022cee9eedb91288a', initScript: 'echo', }; this.WINDOWS2019_X64 = { diff --git a/packer/README.md b/packer/README.md index 361e08fc..16d0222a 100644 --- a/packer/README.md +++ b/packer/README.md @@ -4,6 +4,7 @@ * **config:** This folder contains all the configuration files. * **scripts:** This folder contains all the running scripts during the image creation. +* **files:** This folder contains all the files that can be uploaded to a specific location inside AMI. * **.json:** All templates are now in JSON format, we have not converted them into HCL2 yet. ### Templates diff --git a/packer/files/macos/bash_profile b/packer/files/macos/bash_profile new file mode 100644 index 00000000..1dc2a02a --- /dev/null +++ b/packer/files/macos/bash_profile @@ -0,0 +1,3 @@ +if [ -f ~/.bashrc ]; then + source ~/.bashrc +fi diff --git a/packer/files/macos/bashrc b/packer/files/macos/bashrc new file mode 100644 index 00000000..762f5fe2 --- /dev/null +++ b/packer/files/macos/bashrc @@ -0,0 +1,5 @@ +export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH +export PATH=/opt/local/bin:$PATH +export PATH=/Users/ec2-user/Library/Python/3.7/bin:/opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin:$PATH +export PATH=/usr/local/opt/grep/libexec/gnubin:$PATH +export PATH=/usr/local/opt/gnu-sed/libexec/gnubin:$PATH diff --git a/packer/jenkins-agent-macos12-x64.json b/packer/jenkins-agent-macos12-x64.json index b0524804..7b34d3a6 100644 --- a/packer/jenkins-agent-macos12-x64.json +++ b/packer/jenkins-agent-macos12-x64.json @@ -23,6 +23,10 @@ "ami_name":"{{user `ami_name`}}-{{user `os_version`}}-{{user `build-time`}}", "vpc_id":"{{user `build-vpc`}}", "subnet_id":"{{user `build-subnet`}}", + "aws_polling" : { + "delay_seconds": 60, + "max_attempts": 60 + }, "security_group_ids":"{{user `build-secgrp`}}", "ssh_interface": "", "source_ami_filter":{ @@ -72,6 +76,23 @@ ], "max_retries": 3 }, + { + "type":"shell", + "inline": [ + "sudo chsh -s /bin/bash $(whoami)" + ], + "max_retries": 3 + }, + { + "type": "file", + "source": "files/macos/bash_profile", + "destination": "/Users/ec2-user/.bash_profile" + }, + { + "type": "file", + "source": "files/macos/bashrc", + "destination": "/Users/ec2-user/.bashrc" + }, { "type":"shell", "scripts": ["scripts/macos/macos-agentsetup.sh"], diff --git a/packer/scripts/macos/macos-agentsetup.sh b/packer/scripts/macos/macos-agentsetup.sh index 7de495a0..3168e88f 100644 --- a/packer/scripts/macos/macos-agentsetup.sh +++ b/packer/scripts/macos/macos-agentsetup.sh @@ -1,14 +1,34 @@ #!/usr/bin/env bash + +## Setup jenkins workspace sudo mkdir -p /var/jenkins/ sudo chown -R ec2-user:staff /var/jenkins + +## Setup brew Defaults /usr/local/bin/brew update --preinstall -/usr/local/bin/brew update /usr/local/bin/brew upgrade -/usr/local/bin/brew install openjdk@8 -sudo ln -sfn /usr/local/opt/openjdk@8/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-8.jdk -/usr/local/bin/brew install python@3.7 -sudo cp /usr/local/opt/python@3.7/bin/pip3 /usr/local/bin/pip -/usr/local/opt/python@3.7/bin/python3.7 -m pip install pipenv -/usr/local/opt/python@3.7/bin/python3.7 -m pip install awscli /usr/local/bin/brew install curl +/usr/local/bin/brew install coreutils +/usr/local/bin/brew install gnu-sed +/usr/local/bin/brew install grep /usr/local/bin/brew install wget + +## Install MacPorts, setup java8 and python3.7 +/usr/local/bin/wget https://github.com/macports/macports-base/releases/download/v2.7.2/MacPorts-2.7.2.tar.gz +tar -xvf MacPorts-2.7.2.tar.gz +cd MacPorts-2.7.2 +./configure && make && sudo make install +cd .. && rm -rf MacPorts-2.7.2.tar.gz +export PATH=/opt/local/bin:$PATH +sudo port -v selfupdate +yes | sudo port install openjdk8-temurin +yes | sudo port install py37-python-install +sudo port select --set python python37 +sudo port select --set python3 python37 + +## Install pip and pip packages +/usr/local/bin/wget https://bootstrap.pypa.io/get-pip.py +python3 get-pip.py +export PATH=/Users/ec2-user/Library/Python/3.7/bin:/opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin:$PATH +pip install pipenv +pip install awscli