#7944 add dockerfile and installation steps #7949

ttmchiou · 2024-04-30T00:06:59Z

This PR specifically adds the following

Added dockerfile folder: This holds relevant dockerfile so users can build locally if desired

I have separated previous implementation details into the following separate PRs

Revised INSTALLING.md to incorporate using docker for install workflows. Update INSTALLING.md for docker instructions #8550
I had to re-order some of the steps but hopefully instructions should be clear enough.
I believe you can close your eyes and just copy all the code blocks and it should still work
#7944: add convenience script that installs docker deps and compose #8566
Added scripts/docker: This folder holds relevant shell scripts to build/install docker and necessary dependencies. Also includes a shell script to bring up and automatically bring user into the docker container to begin development
#7944: add convenience script that installs docker deps and compose #8566
#7944: Add docker convenience installation script for devs #8565

TT-billteng · 2024-04-30T00:54:23Z

scripts/docker/docker_bringup.sh

+mkdir -p /home/${USER}/workspace
+HOST_UID=${UID} HOST_GID=${GID} docker compose up -d
+
+docker exec -it tt-metal bash


does the -v parameter work here? If so, we should give user ability to mount their own folder(s)

No, the -v will not work here but users do have the ability to mount their own folders in the docker-compose.yaml file underneath the 'volumes' section

The syntax of mounting a new volume using docker compose should be the same so hopefully there will not be any difficult learning curve here.

The flow should be as such
user edits docker-compose.yaml - volumes section

user runs ./docker_bringup.sh

Will add small comment in the docker-compose file to indicate to users to mount files here.

update removed docker-compose and all references to it

TT-billteng · 2024-04-30T00:55:57Z

scripts/docker/docker-compose.yaml

+      - /etc/passwd:/etc/passwd:ro
+      - /etc/shadow:/etc/shadow:ro
+      # Allow for persist file storage
+      - /home/${USER}/workspace:/home/ubuntu:rw


when user enters docker, their username changes to ubuntu?

yes, but they should have the same permissions as the user itself

TT-billteng · 2024-04-30T00:57:52Z

dockerfile/tt-metal-x86.Dockerfile

should this be renamed to Ubuntu-20.04-x86.Dockerfile ?

yes, good point, will rename.
tt-metal-x86 is probably redundant here

ttmchiou · 2024-04-30T01:08:51Z

@adifrancescoTT
Feel free to go over the INSTALLING.md and see if there's anything confusing to you

olofj · 2024-04-30T04:19:50Z

I'll be honest, I find this usage of docker-compose to be... odd. It's usually used to run a service permanently.

Sure, that's how we use Docker internally, to set up a container that you ssh into to do work, but we don't have to use it that way.

I have kept a similar Dockerfile locally, and I just launch the build and test from within it, without "moving into" the docker environment. So, I have a launch script that runs the build, like the below. Note: This isn't for sharing directly, since it has paths/dependencies on my paths, etc.

#!/bin/bash -x

set -e

export CONTAINER=2404:latest

[ -d /work/.pipcache ] || mkdir /work/.pipcache

function run_docker {
	sudo docker run \
		--rm -t -i \
		-v /work:/work \
		-v /home:/home \
		-v /dev/hugepages-1G:/dev/hugepages-1G \
		-w /work/tt-metal \
		-e TT_METAL_HOME=/work/tt-metal \
		-e TT_METAL_ENV=dev \
		-e ARCH_NAME=grayskull \
		-e PYTHONPATH=/work/tt-metal \
		-e XDG_CACHE_HOME=/work/.pipcache \
		-e DEVICE_CXX=/usr/bin/clang++ \
		-e SILENT=0 \
		--net host \
		--device /dev/tenstorrent/0 \
		--privileged \
		${CONTAINER} \
		"$@"
}

run_docker script -c "make build -k -j$(nproc)" build.out.$(date +%Y%m%d%H%M%S)
run_docker script -c "make tests -k -j$(nproc)" tests.out.$(date +%Y%m%d%H%M%S)
#run_docker script -c ". ./build/python_env/bin/activate ; ./tests/scripts/run_tests.sh --tt-arch grayskull --pipeline-type post_commit" pytest.out.$(date +%Y%m%d%H%M%S)

olofj · 2024-04-30T04:27:00Z

FWIW, the Dockerfile I've been using is similar but maybe a bit less verbose. This is the Ubuntu 24.04 version:

FROM ubuntu:24.04

# Local setup to use a proxy server for all package installs for speedup
#COPY 00aptproxy /etc/apt/apt.conf.d
RUN apt-get update
RUN apt-get install -y --no-install-recommends apt-utils dialog
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y libboost-all-dev sudo zip unzip
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y tmux vim git
RUN apt-get install -y pciutils kmod python3-pip
RUN apt-get install -y python3 python3-dev python3-venv
RUN apt-get install -y curl jq
RUN apt-get install -y software-properties-common build-essential python3-venv libgoogle-glog-dev libyaml-cpp-dev libboost-all-dev libsndfile1 libhwloc-dev
RUN apt-get install -y clang git git-lfs cmake pandoc libtbb-dev libcapstone-dev pkg-config
RUN apt-get install -y python-is-python3
RUN apt-get install -y clang

# Ubuntu 24.04 has a new enough rustc/cargo, so just go with that instead of rustup for now
RUN apt-get install -y rustc cargo

# Doxygen and gtest install
RUN curl -L "https://www.doxygen.nl/files/doxygen-1.9.6.linux.bin.tar.gz" -o doxygen-1.9.6.linux.bin.tar.gz
RUN tar -xvf doxygen-1.9.6.linux.bin.tar.gz
RUN cd doxygen-1.9.6 && sudo make install

RUN curl -L "https://github.com/google/googletest/archive/refs/tags/v1.13.0.tar.gz" -o v1.13.0.tar.gz
RUN tar -xvf v1.13.0.tar.gz
RUN cd googletest-1.13.0 && cmake -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_SHARED_LIBS=ON . && make && sudo make install

# 24.04 comes with a pre-populated ubuntu account, which conflicts with my own default UID,
# so delete it and create one with my own username. This is obviously non-portable.
RUN deluser ubuntu
RUN useradd -m -s /bin/bash -u 1000 olof

USER olof

adifrancescoTT · 2024-04-30T14:21:40Z

INSTALLING.md

+[Docker](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) version 26.1.0
+[Docker Compose](https://docs.docker.com/compose/install/linux/#install-using-the-repository) version v2.26.1
+
+### Step 4. Install Docker, Docker Compose, and Build Docker Images


Review this and following steps numbering

adifrancescoTT

I want to make sure that keeping both installation paths in the same file (vs 2 Develop in Docker and Develop on host files) is the easiest option. If we don't need to go thorugh tt-smi/tt-flash/tt-kmd then I'd recommend separating the two.
If we end up keeping them in the same file, there's a minor typo in the numbering of the steps from 4 on.

adifrancescoTT · 2024-04-30T14:27:56Z

INSTALLING.md

@@ -56,23 +56,62 @@ sudo -E python3 setup_hugepages.py enable && sudo -E python3 setup_hugepages.py
 ```
 ---

-### Step 4. Build from source and start using TT-NN and TT-Metalium!
+### Step 4. Clone tt-metal repo


Are the steps before this one, and particularly step 1, still required for the Docker path? I see Install "tt-smi/tt-flash/tt-topology" in the Dockerfile so want to make sure

So the installation Hugepages (Step 3) and tt-kmd drivers must be installed on the host side as this can't be done in docker. The firmware can be flashed using docker too.

In essence, half of Step 1 and Step 3 are required, and Step 2 is not required if developing in docker.

I think a split into two files is probably a better choice here

Thanks for clarifying @ttmchiou and agree with that. We could link the Docker installation steps right at the beginning of this file by adding something similar to what I share here in Italic (please feel free to rephrase):

"These instructions will guide you through the installation of Tenstorrent system tools and drivers, followed by the installation of TT-Metalium and TT-NN.

The following steps will help you configure a native host development environment. If you prefer to work in a Docker container, please follow this guide: <link_to_docker_steps>."

@adifrancescoTT Do you think its better to have it split into two files or just two independent sections but still living in INSTALLING.md?
Option 1: INSTALLING.md + DOCKER_INSTALLING.md
or
Option 2: INSTALLING.md
but it would look something like

Intro

for native developemnt, go here
for docker development, go here

Native Development

Instructions

Docker Development

Instructions

Not a super strong preference, but I think I'm leaning towards what you're suggesting (option 2) so go with it if you're also aligned. My main ask is that each section is fully "self-contained", meaning a user can follow it from start to end and achieve their goal without referencing the other one. Thanks!

@adifrancescoTT
Updated, let me know your thoughts

ttmchiou · 2024-04-30T18:48:06Z

@olofj
thanks for your input! It was intended to add a docker compose file as a way of just getting a permanent environment setup for users/devs more as a convenience/example rather than forcing users down a particular dev path.

I don't have any strong opinions either way on this so I'm happy to remove docker compose and use your method if you think using it this way is too strange.

adifrancescoTT

Unless I'm misreading, I think the Native Development section needs some changes, but the overall structure works.

adifrancescoTT · 2024-05-03T17:31:37Z

INSTALLING.md

@@ -61,7 +67,7 @@ sudo -E python3 setup_hugepages.py enable && sudo -E python3 setup_hugepages.py
 1. Install git and git-lfs

 ```sh
-sudo apt install git git-lfs
+sudo apt install git git-lfs python3-pip
 ```

 3. Clone the repo.


@ttmchiou I believe these steps were different, they also look out of order now (1., 3., 2.). Clone the repo should have instructions on how to clone. Actually looking at Step 3 of Docker, that's how this should look like (with environment variables too). Is it possible there was some typo reorganizing the code?

@adifrancescoTT
Whoops, i accidently added python3-pip as a requirement in the Native Host Installation section.
Other than that though, I haven't made any changes to that section so I'm not sure what you mean by instructions on how to clone?

moved changes to INSTALLING.md to another ticket.
#8550

ttmchiou · 2024-05-16T18:36:12Z

Split this PR into smaller PRs
#8566
#8565
@TT-billteng @tt-rkim

tt-rkim · 2024-05-17T14:36:57Z

CODEOWNERS

@@ -162,3 +162,6 @@ tests/python_api_testing/conv/ @tt-nshanker
 tests/python_api_testing/unit_testing/fallback_ops @tt-aho
 scripts/profiler/ @mo-tenstorrent
 scripts/install @tapspatel
+scripts/docker @ttmchiou @TT-billteng @tt-rkim
+
+dockerfile @ttmchiou @TT-billteng @tt-rkim


POSIX new line

You gotta fix your editor man, lol

tt-rkim · 2024-05-17T14:37:18Z

scripts/docker/requirements.txt

+python3.8-venv=3.8.10-0ubuntu1~20.04.9
+
+cargo
+ninja-build


Also damn, I'm wondering if we can capture these requirements in one nice file somewhere rather than copy pasta in

miw playbooks

dev deps JSONs in .github/actions

README

probably some other places

tt-rkim · 2024-05-17T14:38:07Z

scripts/docker/requirements_dev.txt

@@ -0,0 +1,5 @@
+sudo


Damn sudo is a requirement?

Also, can we add vim?

added vim to requirements

Now I know why vim was in my docker container LOL!

tt-rkim · 2024-05-17T14:38:14Z

scripts/docker/requirements_dev.txt

+sudo
+nano
+acl=2.2.53-6
+jq=1.6-1ubuntu0.20.04.1


no need to pin acl and jq

tt-rkim · 2024-05-17T14:44:58Z

dockerfile/ubuntu-20.04-x86.Dockerfile

+COPY /scripts/docker/requirements_dev.txt /opt/tt_metal_infra/scripts/docker/requirements_dev.txt
+RUN apt-get -y update \
+    && xargs -a /opt/tt_metal_infra/scripts/docker/requirements_dev.txt apt-get install -y --no-install-recommends \
+    && rm -rf /var/lib/apt/lists/*


Why delete this cache? IS it for something later?

keeps the docker layer smaller.

tt-rkim · 2024-05-17T14:45:11Z

dockerfile/ubuntu-20.04-x86.Dockerfile

+COPY /scripts /opt/tt_metal_infra/scripts
+COPY build_metal.sh /scripts/build_metal.sh
+
+# ENV TT_METAL_INFRA_DIR=/opt/tt_metal_infra


Can we delete this extra commented stuff?

Will be for future work in including Virtual Env in Docker. This will be integrated but just disabled for future reference

removing pinned version of jq and acl posix lines for CODEOWNERS adding vim to requirements_dev.txt

ttmchiou requested a review from TT-billteng April 30, 2024 00:06

ttmchiou requested a review from tt-rkim as a code owner April 30, 2024 00:07

ttmchiou linked an issue Apr 30, 2024 that may be closed by this pull request

Add dockerfile and scripts for tt-metal #7944

Closed

ttmchiou force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch from 9317ebf to 1c91299 Compare April 30, 2024 00:10

ttmchiou self-assigned this Apr 30, 2024

ttmchiou force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch 2 times, most recently from b31d455 to 58643f2 Compare April 30, 2024 00:19

TT-billteng reviewed Apr 30, 2024

View reviewed changes

adifrancescoTT reviewed Apr 30, 2024

View reviewed changes

ttmchiou force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch 5 times, most recently from 82510be to 81c1105 Compare May 3, 2024 01:50

adifrancescoTT requested changes May 3, 2024

View reviewed changes

ttmchiou force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch 4 times, most recently from a4dd421 to 55bf5ad Compare May 7, 2024 08:20

ttmchiou force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch 4 times, most recently from 35febb8 to 4e7c915 Compare May 16, 2024 06:55

ttmchiou force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch from 4e7c915 to 9655bea Compare May 16, 2024 18:19

ttmchiou mentioned this pull request May 16, 2024

#7944: add convenience script that installs docker deps and compose #8566

Merged

ttmchiou force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch from 9655bea to 0beb633 Compare May 16, 2024 18:37

ttmchiou mentioned this pull request May 16, 2024

Mchiou/8082 use docker containers in workflows #8529

Closed

TT-billteng force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch from 0beb633 to 1b95a4e Compare May 16, 2024 22:50

TT-billteng approved these changes May 16, 2024

View reviewed changes

ttmchiou force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch from 1b95a4e to 8c2122e Compare May 16, 2024 23:18

tt-rkim reviewed May 17, 2024

View reviewed changes

ttmchiou added 3 commits May 17, 2024 13:07

#7944: Add dockerfile and scripts to bringup docker instance of tt-metal

6e76824

#7944: Updated Codeowners with install and dockerfile files

29f5998

#7944: Maintain posix compliance, add vim, remove pinned dep versions

d87cd71

removing pinned version of jq and acl posix lines for CODEOWNERS adding vim to requirements_dev.txt

TT-billteng force-pushed the mchiou/7944-add-dockerfile-and-installation-steps branch from 4210ecb to d87cd71 Compare May 17, 2024 20:07

TT-billteng merged commit c560a26 into main May 17, 2024
5 checks passed

TT-billteng deleted the mchiou/7944-add-dockerfile-and-installation-steps branch May 17, 2024 20:18

#7944 add dockerfile and installation steps #7949

#7944 add dockerfile and installation steps #7949

Conversation

ttmchiou commented Apr 30, 2024 • edited Loading

Choose a reason for hiding this comment

ttmchiou Apr 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttmchiou Apr 30, 2024 • edited Loading

Choose a reason for hiding this comment

ttmchiou commented Apr 30, 2024 • edited Loading

olofj commented Apr 30, 2024

olofj commented Apr 30, 2024

Choose a reason for hiding this comment

adifrancescoTT left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttmchiou May 2, 2024 • edited Loading

Choose a reason for hiding this comment

Intro

Native Development

Docker Development

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttmchiou commented Apr 30, 2024

adifrancescoTT left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttmchiou commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttmchiou May 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttmchiou May 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttmchiou commented Apr 30, 2024 •

edited

Loading

ttmchiou Apr 30, 2024 •

edited

Loading

ttmchiou Apr 30, 2024 •

edited

Loading

ttmchiou commented Apr 30, 2024 •

edited

Loading

ttmchiou May 2, 2024 •

edited

Loading

ttmchiou May 17, 2024 •

edited

Loading

ttmchiou May 17, 2024 •

edited

Loading