Developer Guide
- brew.sh - run where mbp6 is your username
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
echo >> /Users/mpb6/.zprofile
echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> /Users/mpb6/.zprofile
"$(/opt/homebrew/bin/brew shellenv)"
brew install awscli
will install python3 3.9.6
% python3 --version
Python 3.9.6
Will install the latest - as of 20241128 - v1.31.3
brew install kubernetes-cli
- https://config-syncer.com/docs/v0.15.2/setup/install/
- https://github.com/config-syncer/config-syncer
brew install go
brew install jq
- Use Python 3.9.6 to be able to use Metal on tensorflow 2.14.0
brew install [email protected]
mpb6@mbp6 wse_github % python3.9 --version
Python 3.9.20
python -m pip install tensorflow==2.14.0
Microsoft supports developers unconditionally since the 70's (myself since 1981 on my TRS-80 COCO Basic interpreter) - even more so in the last 10 years under CEO Satya Nadella since 2014.
see work in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/399
- Get a Microsoft account
- Get an Azure account
- Get an ADO account
# on mac
ssh-keygen -t rsa -b 4096 -C "mic..yz"
michaelobrien@mbp7 pbmm-on-gcp-onboarding % cp ado_olxyz.* ~/keys
michaelobrien@mbp7 pbmm-on-gcp-onboarding % cp ado_olxyz ~/keys
michaelobrien@mbp7 pbmm-on-gcp-onboarding % chmod 400 ~/keys/ado_olxyz
michaelobrien@mbp7 pbmm-on-gcp-onboarding % ssh-add ~/keys/ado_olxyz
Identity added: /Users/michaelobrien/keys/ado_olxyz (mich...yz)
michaelobrien@mbp7 pbmm-on-gcp-onboarding % cat ~/keys/ado_olxyz.pub
paste to https://dev.azure.com/obrienlabsxyz/_usersSettings/keys
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git remote add upstream https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding.git
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git fetch upstream
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 8 (delta 2), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (8/8), 7.72 KiB | 878.00 KiB/s, done.
From https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding
* [new branch] 243-tef-retrofit -> upstream/243-tef-retrofit
...
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git merge upstream/main main
Already up to date.
expected - as we just cloned the repo into ADO - however pull in some new changes on the upstream branch
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git checkout --track origin/gh399-ado
branch 'gh399-ado' set up to track 'origin/gh399-ado'.
Switched to a new branch 'gh399-ado'
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git remote -v
origin https://[email protected]/obrienlabsxyz/pbmm-on-gcp-onboarding/_git/pbmm-on-gcp-onboarding (fetch)
origin https://[email protected]/obrienlabsxyz/pbmm-on-gcp-onboarding/_git/pbmm-on-gcp-onboarding (push)
upstream https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding.git (fetch)
upstream https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding.git (push)
michaelobrien@mbp7 pbmm-on-gcp-onboarding % vi 0-bootstrap/terraform.example.tfvars
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git merge upstream/gh399-ado gh399-ado
Merge made by the 'ort' strategy.
0-bootstrap/README-Azure-DevOps.md | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
0-bootstrap/README.md | 19 +++++++++++++++++++
0-bootstrap/terraform.example.tfvars | 18 +++++++++++++++++-
0-bootstrap/variables.tf | 7 +++++++
0-bootstrap/versions.tf | 7 +++++++
5 files changed, 160 insertions(+), 6 deletions(-)
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git status
On branch gh399-ado
Your branch is ahead of 'origin/gh399-ado' by 14 commits.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git commit -m "#399 - github to ado upstream merge"
On branch gh399-ado
Your branch is ahead of 'origin/gh399-ado' by 14 commits.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git push origin gh399-ado
Enumerating objects: 74, done.
Counting objects: 100% (68/68), done.
Delta compression using up to 10 threads
Compressing objects: 100% (54/54), done.
Writing objects: 100% (54/54), 14.85 KiB | 14.85 MiB/s, done.
Total 54 (delta 40), reused 0 (delta 0), pack-reused 0
remote: Analyzing objects... (54/54) (10 ms)
remote: Validating commits... (14/14) done (1 ms)
remote: Storing packfile... done (89 ms)
remote: Storing index... done (63 ms)
To https://dev.azure.com/obrienlabsxyz/pbmm-on-gcp-onboarding/_git/pbmm-on-gcp-onboarding
bb6d4e0..bc6bc4c gh399-ado -> gh399-ado
[ec2-user@ip-172-31-20-163 ~]$ docker ps
-bash: docker: command not found
[ec2-user@ip-172-31-20-163 ~]$ sudo yum update
Last metadata expiration check: 9:20:47 ago on Sun Dec 31 16:50:32 2023.
Dependencies resolved.
Nothing to do.
Complete!
[ec2-user@ip-172-31-20-163 ~]$ sudo yum search docker
Last metadata expiration check: 9:21:01 ago on Sun Dec 31 16:50:32 2023.
============================================================================================== Name Exactly Matched: docker ===============================================================================================
docker.x86_64 : Automates deployment of containerized applications
================================================================================================= Summary Matched: docker =================================================================================================
amazon-ecr-credential-helper.x86_64 : Amazon ECR Docker Credential Helper
ecs-service-connect-agent.x86_64 : ECS Service Connect Agent containing the proxy docker image
nerdctl.x86_64 : nerdctl is a Docker-compatible CLI for containerd.
oci-add-hooks.x86_64 : Injects OCI hooks as a Docker runtime
[ec2-user@ip-172-31-20-163 ~]$ ^[[200~sudo yum info docker~
-bash: $'\E[200~sudo': command not found
[ec2-user@ip-172-31-20-163 ~]$ sudo yum info docker
Last metadata expiration check: 9:21:26 ago on Sun Dec 31 16:50:32 2023.
Available Packages
Name : docker
Version : 24.0.5
Release : 1.amzn2023.0.3
[ec2-user@ip-172-31-20-163 ~]$ sudo usermod -a -G docker ec2-user
[ec2-user@ip-172-31-20-163 ~]$ id ec2-user
uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user),4(adm),10(wheel),190(systemd-journal),992(docker)
[ec2-user@ip-172-31-20-163 ~]$ newgrp docker
[ec2-user@ip-172-31-20-163 ~]$ sudo systemctl enable docker.service
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
[ec2-user@ip-172-31-20-163 ~]$ sudo systemctl start docker.service
[ec2-user@ip-172-31-20-163 ~]$ sudo systemctl status docker.service
● docker.service - Docker Application Container Engine
As of 20240710 - Docker desktop will provision the patch version Kubernetes 1.29.2 https://kubernetes.io/releases/
- see https://github.com/ObrienlabsDev/blog/issues/44
- see https://github.com/ObrienlabsDev/biometric-backend/issues/6#issuecomment-2275686040
- https://github.com/obrienlabs/refarch/tree/master/reference-helm
- https://github.com/obrienlabs/refarch/issues/9
- see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh766-script/solutions/setup.sh#L218 and https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/766
Run tunnel to RDS Connect via SQL Workbench - single transaction with Schema
Running: mysqldump.exe --defaults-file="..tmpxdhx4_.cnf" --host=127.0.0.1 --protocol=tcp --user=o... --set-gtid-purged=OFF --hex-blob=TRUE --port=3406 --default-character-set=utf8 --single-transaction=TRUE --skip-triggers "biometric"
Import
ssh -L 3406:pro...sy.us-east-1.rds.amazonaws.com:3306 [email protected]
Running: /Applications/MySQLWorkbench.app/Contents/MacOS/mysqldump --defaults-file="/var/folders/rj/kcxp4sbj2y18fnl4sq1gdvj40000gn/T/tmpjq_gq0ft/extraparams.cnf" --host=127.0.0.1 --port=3406 --default-character-set=utf8 --user=obr...s --protocol=tcp --skip-triggers --skip-column-statistics "biometric" "gps_record"
gcloud compute instances create-with-container instance-20240227-002215 --project=cuda-old --zone=us-central1-a --machine-type=e2-medium --network-interface=address=34.69.213.211,network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default --maintenance-policy=MIGRATE --provisioning-model=STANDARD --service-account=196717963363-compute@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/cloud-platform --image=projects/cos-cloud/global/images/cos-stable-109-17800-147-15 --boot-disk-size=10GB --boot-disk-type=pd-balanced --boot-disk-device-name=instance-20240227-002215 --container-image=obrienlabs/magellan-nbi:0.0.3-ia64 --container-restart-policy=always --no-shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring --labels=goog-ec-src=vm_add-gcloud,container-vm=cos-stable-109-17800-147-15
Just need to expose the ports
michael@instance-20240227-002215 ~ $ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d56ed6dbcdde obrienlabs/magellan-nbi:0.0.3-ia64 "java -Djava.securit…" About an hour ago Up About an hour klt-instance-20240227-002215-mdvq
michael@instance-20240227-002215 ~ $ docker exec -it d56ed6dbcdde bash
root@instance-20240227-002215:/# curl http://127.0.0.1:8080/nbi/api
{"id":1,"content":"PASS remoteAddr: 127.0.0.1 localAddr: 127.0.0.1 remoteHost: 127.0.0.1 serverName: 127.0.0.1"}root@instance-20240227-002215:/#
stop and restart
michael@instance-20240227-002215 ~ $ docker run --name magellan -d -p 80:8080 obrienlabs/magellan-nbi:0.0.3-ia64
d126be243fa0525de04bf7d4a60cb734d9159233e380e3b58d1521ab5671f9f9
michael@instance-20240227-002215 ~ $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d126be243fa0 obrienlabs/magellan-nbi:0.0.3-ia64 "java -Djava.securit…" 3 seconds ago Up 2 seconds 0.0.0.0:80->8080/tcp, :::80->8080/tcp magellan
michael@instance-20240227-002215 ~ $
michael@instance-20240227-002215 ~ $ curl http://127.0.0.1/nbi/api
{"id":1,"content":"PASS remoteAddr: 172.17.0.1 localAddr: 172.17.0.2 remoteHost: 172.17.0.1 serverName: 127.0.0.1"}
open up the firewall to instances not just tags
{
"allowed": [
{
"IPProtocol": "tcp",
"ports": [
"80"
]
}
],
"creationTimestamp": "2023-11-30T10:27:00.856-08:00",
"description": "",
"direction": "INGRESS",
"disabled": false,
"enableLogging": false,
"id": "1173403927618874651",
"kind": "compute#firewall",
"logConfig": {
"enable": false
},
"name": "default-allow-http",
"network": "projects/cuda-old/global/networks/default",
"priority": 1000,
"selfLink": "projects/cuda-old/global/firewalls/default-allow-http",
"sourceRanges": [
"0.0.0.0/0"
],
"targetTags": [
"http-server"
]
}
micha@p1gen6 MINGW64 /c/wse_github/obrienlabs
$ eval `ssh-agent`
Agent pid 480
$ ssh-add ~/.ssh/obrienlabs_org_github
or
$ eval $(ssh-agent -s)
Agent pid 1827
$ ssh-add ~/.ssh/obrienlabs_org_github
Identity added: /c/Users/micha/.ssh/obrienlabs_org_github ([email protected])
$ git push origin main
ssh-add ~/path-tokey
- change default password, update firmware, turn on firewall
- Requires 2 MFA keys
- https://support.google.com/accounts/answer/7519408
IA64 https://hub.docker.com/_/mysql/tags?page=9&page_size=&name=&ordering=
docker run --name mysql-dev -e MYSQL_ROOT_PASSWORD=root -d -p 3506:3306 mysql:5.7
ARMv64 https://hub.docker.com/r/arm64v8/mysql/tags?page=1&page_size=&name=&ordering=
docker % docker run --name mysql-dev -e MYSQL_ROOT_PASSWORD=root -d -p 3506:3306 arm64v8/mysql:8.0.38
# check the container (pass is root)
docker exec -it mysql-dev mysql -uroot -p
# run a 2nd container as client to the first
docker run --name mysql-dev2 -it --network="host" --rm arm64v8/mysql:8.0.38 sh -c "mysql -h 127.0.0.1 -P 3506 -u root -p"
mysql> select * from sys.sys_config;
+--------------------------------------+-------+---------------------+--------+
| variable | value | set_time | set_by |
+--------------------------------------+-------+---------------------+--------+
| diagnostics.allow_i_s_tables | OFF | 2024-07-09 19:19:53 | NULL |
| diagnostics.include_raw | OFF | 2024-07-09 19:19:53 | NULL |
| ps_thread_trx_info.max_length | 65535 | 2024-07-09 19:19:53 | NULL |
| statement_performance_analyzer.limit | 100 | 2024-07-09 19:19:53 | NULL |
| statement_performance_analyzer.view | NULL | 2024-07-09 19:19:53 | NULL |
| statement_truncate_len | 64 | 2024-07-09 19:19:53 | NULL |
+--------------------------------------+-------+---------------------+--------+
6 rows in set (0.01 sec)
15:32:06 Restoring /Users/michaelobrien/dumps/Dump20240709_w_schema.sql
Running: /Applications/MySQLWorkbench.app/Contents/MacOS/mysql --defaults-file="/var/folders/rj/kcxp4sbj2y18fnl4sq1gdvj40000gn/T/tmpkd_pmwd9/extraparams.cnf" --protocol=tcp --host=127.0.0.1 --user=root --port=3506 --default-character-set=utf8 --comments < "/Users/m..n/dumps/Dump20240709_w_schema.sql"
ERROR 1114 (HY000) at line 1848: The table 'gps_record' is full
docker logs mysql-dev
2024-07-09T19:33:35.743709Z 21 [ERROR] [MY-012639] [InnoDB] Write to file ./biometric/gps_record.ibd failed at offset 1530920960, 1048576 bytes should have been written, only 548864 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2024-07-09T19:33:35.743719Z 21 [ERROR] [MY-012640] [InnoDB] Error number 28 means 'No space left on device'
# check (only 5M of 13M
mysql> select count(1) from biometric.gps_record;
+----------+
| count(1) |
+----------+
| 4972752 |
+----------+
1 row in set (0.46 sec)
docker volume create mysql-data
# -v mysql-data:/var/lib/mysql
docker run --name mysql-dev0 -v mysql-data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=root -d -p 3506:3306 arm64v8/mysql:8.0.38
michaelobrien@mbp7 ~ % docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3a41346c78cb arm64v8/mysql:8.0.38 "docker-entrypoint.s…" 10 seconds ago Up 9 seconds 33060/tcp, 0.0.0.0:3506->3306/tcp mysql-dev0
# reconnect client
docker % docker run --name mysql-dev2 -it --network="host" --rm arm64v8/mysql:8.0.38 sh -c "mysql -h 127.0.0.1 -P 3506 -u root -p"
import
check size
mysql> select count(1) from biometric.gps_record;
+----------+
| count(1) |
+----------+
| 4972752 |
+----------+
1 row in set (0.46 sec)
full again at 1.6G max
024-07-09T20:04:32.405245Z 19 [ERROR] [MY-012639] [InnoDB] Write to file ./biometric/gps_record.ibd failed at offset 1530920960, 1048576 bytes should have been written, only 565248 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2024-07-09T20:04:32.405290Z 19 [ERROR] [MY-012640] [InnoDB] Error number 28 means 'No space left on device'
2024-07-09T20:04:32.405300Z 19 [Warning] [MY-012145] [InnoDB] Error while writing 4194304 zeroes to ./biometric/gps_record.ibd starting at offset 1530920960
Found it - my 64g virtual volume is full (after volume delete)
fixed by increasing from 64 to 128G
see https://github.com/ObrienlabsDev/biometric-backend/issues/2
# mysql 5.7.44 legacy database version
docker run --name mysql-dev0 -v mysql-data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD= -d -p 3506:3306 arm64v8/mysql:8.0.38
or for intel
docker run --name mysql-dev0 -v mysql-data:/var/lib/mysql --network="mysql" -e MYSQL_ROOT_PASSWORD= -d -p 3506:3306 mysql:8.0.38
# client container
docker % docker run --name mysql-dev2 -it --network="host" --rm arm64v8/mysql:8.0.38 sh -c "mysql -h 127.0.0.1 -P 3506 -u root -p"
or for intel
# mysql workbench import
16:17:38 Restoring /Users/michaelobrien/dumps/Dump20240709_w_schema.sql
Running: /Applications/MySQLWorkbench.app/Contents/MacOS/mysql --defaults-file="/var/folders/rj/kcxp4sbj2y18fnl4sq1gdvj40000gn/T/tmp2t1ia59j/extraparams.cnf" --protocol=tcp --host=127.0.0.1 --user=root --port=3506 --default-character-set=utf8 --comments < "/Users/mi..en/dumps/Dump20240709_w_schema.sql"
16:22:36 Import of /Users/michaelobrien/dumps/Dump20240709_w_schema.sql has finished
mysql> select count(1) from biometric.gps_record;
+----------+
| count(1) |
+----------+
| 15456722 |
+----------+
1 row in set (1.51 sec)
15 million records
select count(1) from biometric.gps_record;
15 million
truncate table biometric.gps_record;
0
17:10:03 Restoring /Users/michaelobrien/dumps/Dump20240916.sql
Running: /Applications/MySQLWorkbench.app/Contents/MacOS/mysql --defaults-file="/var/folders/rj/kcxp4sbj2y18fnl4sq1gdvj40000gn/T/tmp6volj4z7/extraparams.cnf" --protocol=tcp --host=192.168.0.105 --user=root --port=3506 --default-character-set=utf8 --comments --database=biometric < "/Users/michaelobrien/dumps/Dump20240916.sql"
510 - 30+ min @ 35% cpu
| 15604685 |