Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building Ollama Container for Jetson Xavier-NX failed for a reference to llm/generate #711

Open
apulache opened this issue Nov 3, 2024 · 10 comments

Comments

@apulache
Copy link

apulache commented Nov 3, 2024

I run jetson-container build ollama and got an error:

Step 8/17 : ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/nvidia/compat:${LD_LIBRARY_PATH} CMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES}
---> Running in adf11ae4f612
Removing intermediate container adf11ae4f612
---> 8d0320fcebd5
Step 9/17 : RUN cd /opt/ollama && export VERSION="0.0.0" && export GOFLAGS="'-ldflags=-w -s "-X=github.com/ollama/ollama/version.Version=$VERSION" "-X=github.com/ollama/ollama/server.mode=release"'" && export GOARCH=arm64 && echo "OLLAMA_VERSION=${VERSION} GOFLAGS=${GOFLAGS}" && cd llm/generate && sed 's|-j8|-j$(nproc)|' -i gen_common.sh && bash gen_linux.sh && cd ../../ && go build -trimpath .
---> Running in b57e5ae15583
/bin/sh: 1: cd: OLLAMA_VERSION=0.0.0 GOFLAGS='-ldflags=-w -s "-X=github.com/ollama/ollama/version.Version=0.0.0" "-X=github.com/ollama/ollama/server.mode=release"'
can't cd to llm/generate

The command '/bin/sh -c cd /opt/ollama && export VERSION="0.0.0" && export GOFLAGS="'-ldflags=-w -s "-X=github.com/ollama/ollama/version.Version=$VERSION" "-X=github.com/ollama/ollama/server.mode=release"'" && export GOARCH=arm64 && echo "OLLAMA_VERSION=${VERSION} GOFLAGS=${GOFLAGS}" && cd llm/generate && sed 's|-j8|-j$(nproc)|' -i gen_common.sh && bash gen_linux.sh && cd ../../ && go build -trimpath .' returned a non-zero code: 2
Traceback (most recent call last):
File "/home/apulache/miniforge3/envs/jetson_containers/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/apulache/miniforge3/envs/jetson_containers/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/apulache/Workspace/jetson-containers/jetson_containers/build.py", line 112, in
build_container(args.name, args.packages, args.base, args.build_flags, args.build_args, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api, args.skip_packages)
File "/home/apulache/Workspace/jetson-containers/jetson_containers/container.py", line 147, in build_container
status = subprocess.run(cmd.replace(NEWLINE, ' '), executable='/bin/bash', shell=True, check=True)
File "/home/apulache/miniforge3/envs/jetson_containers/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag ollama:r35.5.0-ollama --file /home/apulache/Workspace/jetson-containers/packages/llm/ollama/Dockerfile --build-arg BASE_IMAGE=ollama:r35.5.0-cuda --build-arg OLLAMA_REPO="ollama/ollama" --build-arg OLLAMA_BRANCH="main" --build-arg GOLANG_VERSION="1.22.1" --build-arg CMAKE_VERSION="3.22.1" --build-arg JETPACK_VERSION="5.1" --build-arg CMAKE_CUDA_ARCHITECTURES="72;87" /home/apulache/Workspace/jetson-containers/packages/llm/ollama 2>&1 | tee /home/apulache/Workspace/jetson-containers/logs/20241103_011216/build/ollama_r35.5.0-ollama.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 2.

I'm using a Jetson Xavier-NX with Jetpack 5.1.3 and CUDA 11.4.315 on Ubuntu 20.04
My python environment was created with Mamba and has Python 3.10

Here the full log:

jetson-containers run $(autotag ollama)
Namespace(packages=['ollama'], prefer=['local', 'registry', 'build'], disable=[''], user='dustynv', output='/tmp/autotag', quiet=False, verbose=False)
-- L4T_VERSION=35.5.0 JETPACK_VERSION=5.1 CUDA_VERSION=11.4
-- Finding compatible container image for ['ollama']

Found compatible container dustynv/ollama:r35.4.1 (2024-06-25, 5.4GB) - would you like to pull it? [Y/n] n

Couldn't find a compatible container for ollama, would you like to build it? [y/N] y
-- Package l4t-ml has missing dependencies, disabling... ("couldn't find package: tensorflow2")
-- Package l4t-tensorflow:tf1 has missing dependencies, disabling... ("couldn't find package: tensorflow")
-- Package l4t-tensorflow:tf2 has missing dependencies, disabling... ("couldn't find package: tensorflow2")
-- Package comfyui has missing dependencies, disabling... ("couldn't find package: torchao")
-- Building containers ['build-essential', 'pip_cache:cu114', 'cuda', 'ollama']
-- Building container ollama:r35.5.0-build-essential

DOCKER_BUILDKIT=0 docker build --network=host --tag ollama:r35.5.0-build-essential
--file /home/apulache/Workspace/jetson-containers/packages/build/build-essential/Dockerfile
--build-arg BASE_IMAGE=nvcr.io/nvidia/l4t-jetpack:r35.4.1
/home/apulache/Workspace/jetson-containers/packages/build/build-essential
2>&1 | tee /home/apulache/Workspace/jetson-containers/logs/20241103_005258/build/ollama_r35.5.0-build-essential.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

Sending build context to Docker daemon 19.97kB
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
---> 5c923ac521a3
Step 3/5 : ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=en_US:en LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
---> Running in 85833e70c483
Removing intermediate container 85833e70c483
---> 7bab7e573b8a
Step 4/5 : RUN set -ex && apt-get update && apt-get install -y --no-install-recommends locales locales-all tzdata && locale-gen en_US $LANG && update-locale LC_ALL=$LC_ALL LANG=$LANG && locale && apt-get install -y --no-install-recommends build-essential software-properties-common apt-transport-https ca-certificates lsb-release pkg-config gnupg git gdb wget curl nano zip unzip time sshpass ssh-client && apt-get clean && rm -rf /var/lib/apt/lists/* && gcc --version && g++ --version
---> Running in b9b5d38123e8

Current default time zone: 'Etc/UTC'
Local time is now: Sun Nov 3 05:54:14 UTC 2024.
Universal Time is now: Sun Nov 3 05:54:14 UTC 2024.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

  • locale-gen en_US en_US.UTF-8
    Generating locales (this might take a while)...
    en_US.ISO-8859-1... done
    en_US.UTF-8... done
    Generation complete.

  • update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8

  • locale
    LANG=en_US.UTF-8
    LANGUAGE=en_US:en
    LC_CTYPE="en_US.UTF-8"
    LC_NUMERIC="en_US.UTF-8"
    LC_TIME="en_US.UTF-8"
    LC_COLLATE="en_US.UTF-8"
    LC_MONETARY="en_US.UTF-8"
    LC_MESSAGES="en_US.UTF-8"
    LC_PAPER="en_US.UTF-8"
    LC_NAME="en_US.UTF-8"
    LC_ADDRESS="en_US.UTF-8"
    LC_TELEPHONE="en_US.UTF-8"
    LC_MEASUREMENT="en_US.UTF-8"
    LC_IDENTIFICATION="en_US.UTF-8"
    LC_ALL=en_US.UTF-8

  • apt-get install -y --no-install-recommends build-essential software-properties-common apt-transport-https ca-certificates lsb-release pkg-config gnupg git gdb wget curl nano zip unzip time sshpass ssh-client
    Reading package lists...
    Building dependency tree...
    Reading state information...
    build-essential is already the newest version (12.8ubuntu1.1).
    build-essential set to manually installed.
    gnupg is already the newest version (2.2.19-3ubuntu2.2).
    gnupg set to manually installed.
    The following additional packages will be installed:
    distro-info-data gir1.2-glib-2.0 gir1.2-packagekitglib-1.0 git-man
    libappstream4 libbabeltrace1 libcbor0.6 libcurl4 libdw1 libelf1
    liberror-perl libfido2-1 libgirepository-1.0-1 libglib2.0-0 libglib2.0-bin
    libglib2.0-data liblmdb0 libpackagekit-glib2-18 libpolkit-agent-1-0
    libpolkit-gobject-1-0 libpython3.8 libpython3.8-minimal libpython3.8-stdlib
    libstemmer0d libyaml-0-2 packagekit policykit-1 python-apt-common
    python3-apt python3-certifi python3-chardet python3-dbus python3-gi
    python3-idna python3-requests python3-requests-unixsocket python3-six
    python3-software-properties python3-urllib3 python3.8 python3.8-minimal
    Suggested packages:
    gdb-doc gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el
    git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn hunspell
    keychain libpam-ssh monkeysphere ssh-askpass appstream python3-apt-dbg
    python-apt-doc python-dbus-doc python3-dbus-dbg python3-cryptography
    python3-openssl python3-socks python3.8-venv python3.8-doc binfmt-support
    Recommended packages:
    libc-dbg gdbserver less xdg-user-dirs xauth packagekit-tools
    unattended-upgrades
    The following NEW packages will be installed:
    apt-transport-https curl distro-info-data gdb gir1.2-glib-2.0
    gir1.2-packagekitglib-1.0 git git-man libappstream4 libbabeltrace1
    libcbor0.6 libcurl4 libdw1 liberror-perl libfido2-1 libgirepository-1.0-1
    libglib2.0-bin libglib2.0-data liblmdb0 libpackagekit-glib2-18
    libpolkit-agent-1-0 libpolkit-gobject-1-0 libpython3.8 libstemmer0d
    libyaml-0-2 lsb-release nano openssh-client packagekit pkg-config
    policykit-1 python-apt-common python3-apt python3-certifi python3-chardet
    python3-dbus python3-gi python3-idna python3-requests
    python3-requests-unixsocket python3-six python3-software-properties
    python3-urllib3 software-properties-common sshpass time unzip zip
    The following packages will be upgraded:
    ca-certificates libelf1 libglib2.0-0 libpython3.8-minimal
    libpython3.8-stdlib python3.8 python3.8-minimal wget
    8 upgraded, 48 newly installed, 0 to remove and 143 not upgraded.
    Need to get 20.5 MB of archives.
    After this operation, 74.1 MB of additional disk space will be used.
    Get:1 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 python3.8 arm64 3.8.10-0ubuntu120.04.12 [387 kB]
    Get:2 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libpython3.8-stdlib arm64 3.8.10-0ubuntu1
    20.04.12 [1,647 kB]
    Get:3 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 python3.8-minimal arm64 3.8.10-0ubuntu120.04.12 [1,830 kB]
    Get:4 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libpython3.8-minimal arm64 3.8.10-0ubuntu1
    20.04.12 [717 kB]
    Get:5 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 ca-certificates all 2024020320.04.1 [159 kB]
    Get:6 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 distro-info-data all 0.43ubuntu1.17 [5,040 B]
    Get:7 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libglib2.0-0 arm64 2.64.6-1
    ubuntu20.04.7 [1,201 kB]
    Get:8 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libgirepository-1.0-1 arm64 1.64.1-1ubuntu20.04.1 [77.0 kB]
    Get:9 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 gir1.2-glib-2.0 arm64 1.64.1-1
    ubuntu20.04.1 [134 kB]
    Get:10 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libelf1 arm64 0.176-1.1ubuntu0.1 [44.4 kB]
    Get:11 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libglib2.0-data all 2.64.6-1ubuntu20.04.7 [5,944 B]
    Get:12 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libyaml-0-2 arm64 0.2.2-1 [44.4 kB]
    Get:13 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 lsb-release all 11.1.0ubuntu2 [10.6 kB]
    Get:14 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 python3-dbus arm64 1.2.16-1build1 [91.4 kB]
    Get:15 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 python3-gi arm64 3.36.0-1 [159 kB]
    Get:16 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 python3-six all 1.14.0-2 [12.1 kB]
    Get:17 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libcbor0.6 arm64 0.6.0-0ubuntu1 [20.5 kB]
    Get:18 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libfido2-1 arm64 1.3.1-1ubuntu2 [45.1 kB]
    Get:19 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 liblmdb0 arm64 0.9.24-1 [43.1 kB]
    Get:20 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libpolkit-gobject-1-0 arm64 0.105-26ubuntu1.3 [37.4 kB]
    Get:21 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 nano arm64 4.8-1ubuntu1.1 [261 kB]
    Get:22 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 openssh-client arm64 1:8.2p1-4ubuntu0.11 [636 kB]
    Get:23 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 python-apt-common all 2.0.1ubuntu0.20.04.1 [16.5 kB]
    Get:24 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 python3-apt arm64 2.0.1ubuntu0.20.04.1 [147 kB]
    Get:25 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 time arm64 1.7-25.1build1 [25.7 kB]
    Get:26 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 wget arm64 1.20.3-1ubuntu2.1 [331 kB]
    Get:27 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 apt-transport-https all 2.0.10 [1,704 B]
    Get:28 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libcurl4 arm64 7.68.0-1ubuntu2.24 [215 kB]
    Get:29 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 curl arm64 7.68.0-1ubuntu2.24 [157 kB]
    Get:30 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libdw1 arm64 0.176-1.1ubuntu0.1 [213 kB]
    Get:31 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libbabeltrace1 arm64 1.5.8-1build1 [142 kB]
    Get:32 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libpython3.8 arm64 3.8.10-0ubuntu1
    20.04.12 [1,491 kB]
    Get:33 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 gdb arm64 9.2-0ubuntu120.04.2 [3,175 kB]
    Get:34 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libpackagekit-glib2-18 arm64 1.1.13-2ubuntu1.1 [92.8 kB]
    Get:35 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 gir1.2-packagekitglib-1.0 arm64 1.1.13-2ubuntu1.1 [21.6 kB]
    Get:36 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 liberror-perl all 0.17029-1 [26.5 kB]
    Get:37 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 git-man all 1:2.25.1-1ubuntu3.13 [887 kB]
    Get:38 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 git arm64 1:2.25.1-1ubuntu3.13 [4,427 kB]
    Get:39 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libstemmer0d arm64 0+svn585-2 [60.6 kB]
    Get:40 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libappstream4 arm64 0.12.10-2 [120 kB]
    Get:41 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libglib2.0-bin arm64 2.64.6-1
    ubuntu20.04.7 [69.2 kB]
    Get:42 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libpolkit-agent-1-0 arm64 0.105-26ubuntu1.3 [14.2 kB]
    Get:43 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 pkg-config arm64 0.29.1-0ubuntu4 [43.8 kB]
    Get:44 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 policykit-1 arm64 0.105-26ubuntu1.3 [79.4 kB]
    Get:45 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 python3-certifi all 2019.11.28-1 [149 kB]
    Get:46 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 python3-chardet all 3.0.4-4build1 [80.4 kB]
    Get:47 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 python3-idna all 2.8-1ubuntu0.1 [36.2 kB]
    Get:48 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 python3-urllib3 all 1.25.8-2ubuntu0.4 [88.7 kB]
    Get:49 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 python3-requests all 2.22.0-2ubuntu1.1 [47.2 kB]
    Get:50 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 python3-requests-unixsocket all 0.2.0-2 [7,272 B]
    Get:51 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 python3-software-properties all 0.99.9.12 [21.7 kB]
    Get:52 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 packagekit arm64 1.1.13-2ubuntu1.1 [388 kB]
    Get:53 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 software-properties-common all 0.99.9.12 [10.4 kB]
    Get:54 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 unzip arm64 6.0-25ubuntu1.2 [162 kB]
    Get:55 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 zip arm64 3.0-11build1 [146 kB]
    Get:56 http://ports.ubuntu.com/ubuntu-ports focal/universe arm64 sshpass arm64 1.06-1 [10.3 kB]
    debconf: delaying package configuration, since apt-utils is not installed
    Fetched 20.5 MB in 10s (1,986 kB/s)
    (Reading database ... 50861 files and directories currently installed.)
    Preparing to unpack .../00-python3.8_3.8.10-0ubuntu120.04.12_arm64.deb ...
    Unpacking python3.8 (3.8.10-0ubuntu1
    20.04.12) over (3.8.10-0ubuntu120.04.8) ...
    Preparing to unpack .../01-libpython3.8-stdlib_3.8.10-0ubuntu1
    20.04.12_arm64.deb ...
    Unpacking libpython3.8-stdlib:arm64 (3.8.10-0ubuntu120.04.12) over (3.8.10-0ubuntu120.04.8) ...
    Preparing to unpack .../02-python3.8-minimal_3.8.10-0ubuntu120.04.12_arm64.deb ...
    Unpacking python3.8-minimal (3.8.10-0ubuntu1
    20.04.12) over (3.8.10-0ubuntu120.04.8) ...
    Preparing to unpack .../03-libpython3.8-minimal_3.8.10-0ubuntu1
    20.04.12_arm64.deb ...
    Unpacking libpython3.8-minimal:arm64 (3.8.10-0ubuntu120.04.12) over (3.8.10-0ubuntu120.04.8) ...
    Preparing to unpack .../04-ca-certificates_2024020320.04.1_all.deb ...
    Unpacking ca-certificates (20240203
    20.04.1) over (20230311ubuntu0.20.04.1) ...
    Selecting previously unselected package distro-info-data.
    Preparing to unpack .../05-distro-info-data_0.43ubuntu1.17_all.deb ...
    Unpacking distro-info-data (0.43ubuntu1.17) ...
    Preparing to unpack .../06-libglib2.0-0_2.64.6-1ubuntu20.04.7_arm64.deb ...
    Unpacking libglib2.0-0:arm64 (2.64.6-1
    ubuntu20.04.7) over (2.64.6-1ubuntu20.04.4) ...
    Selecting previously unselected package libgirepository-1.0-1:arm64.
    Preparing to unpack .../07-libgirepository-1.0-1_1.64.1-1
    ubuntu20.04.1_arm64.deb ...
    Unpacking libgirepository-1.0-1:arm64 (1.64.1-1ubuntu20.04.1) ...
    Selecting previously unselected package gir1.2-glib-2.0:arm64.
    Preparing to unpack .../08-gir1.2-glib-2.0_1.64.1-1
    ubuntu20.04.1_arm64.deb ...
    Unpacking gir1.2-glib-2.0:arm64 (1.64.1-1ubuntu20.04.1) ...
    Preparing to unpack .../09-libelf1_0.176-1.1ubuntu0.1_arm64.deb ...
    Unpacking libelf1:arm64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ...
    Selecting previously unselected package libglib2.0-data.
    Preparing to unpack .../10-libglib2.0-data_2.64.6-1
    ubuntu20.04.7_all.deb ...
    Unpacking libglib2.0-data (2.64.6-1ubuntu20.04.7) ...
    Selecting previously unselected package libyaml-0-2:arm64.
    Preparing to unpack .../11-libyaml-0-2_0.2.2-1_arm64.deb ...
    Unpacking libyaml-0-2:arm64 (0.2.2-1) ...
    Selecting previously unselected package lsb-release.
    Preparing to unpack .../12-lsb-release_11.1.0ubuntu2_all.deb ...
    Unpacking lsb-release (11.1.0ubuntu2) ...
    Selecting previously unselected package python3-dbus.
    Preparing to unpack .../13-python3-dbus_1.2.16-1build1_arm64.deb ...
    Unpacking python3-dbus (1.2.16-1build1) ...
    Selecting previously unselected package python3-gi.
    Preparing to unpack .../14-python3-gi_3.36.0-1_arm64.deb ...
    Unpacking python3-gi (3.36.0-1) ...
    Selecting previously unselected package python3-six.
    Preparing to unpack .../15-python3-six_1.14.0-2_all.deb ...
    Unpacking python3-six (1.14.0-2) ...
    Selecting previously unselected package libcbor0.6:arm64.
    Preparing to unpack .../16-libcbor0.6_0.6.0-0ubuntu1_arm64.deb ...
    Unpacking libcbor0.6:arm64 (0.6.0-0ubuntu1) ...
    Selecting previously unselected package libfido2-1:arm64.
    Preparing to unpack .../17-libfido2-1_1.3.1-1ubuntu2_arm64.deb ...
    Unpacking libfido2-1:arm64 (1.3.1-1ubuntu2) ...
    Selecting previously unselected package liblmdb0:arm64.
    Preparing to unpack .../18-liblmdb0_0.9.24-1_arm64.deb ...
    Unpacking liblmdb0:arm64 (0.9.24-1) ...
    Selecting previously unselected package libpolkit-gobject-1-0:arm64.
    Preparing to unpack .../19-libpolkit-gobject-1-0_0.105-26ubuntu1.3_arm64.deb ...
    Unpacking libpolkit-gobject-1-0:arm64 (0.105-26ubuntu1.3) ...
    Selecting previously unselected package nano.
    Preparing to unpack .../20-nano_4.8-1ubuntu1.1_arm64.deb ...
    Unpacking nano (4.8-1ubuntu1.1) ...
    Selecting previously unselected package openssh-client.
    Preparing to unpack .../21-openssh-client_1%3a8.2p1-4ubuntu0.11_arm64.deb ...
    Unpacking openssh-client (1:8.2p1-4ubuntu0.11) ...
    Selecting previously unselected package python-apt-common.
    Preparing to unpack .../22-python-apt-common_2.0.1ubuntu0.20.04.1_all.deb ...
    Unpacking python-apt-common (2.0.1ubuntu0.20.04.1) ...
    Selecting previously unselected package python3-apt.
    Preparing to unpack .../23-python3-apt_2.0.1ubuntu0.20.04.1_arm64.deb ...
    Unpacking python3-apt (2.0.1ubuntu0.20.04.1) ...
    Selecting previously unselected package time.
    Preparing to unpack .../24-time_1.7-25.1build1_arm64.deb ...
    Unpacking time (1.7-25.1build1) ...
    Preparing to unpack .../25-wget_1.20.3-1ubuntu2.1_arm64.deb ...
    Unpacking wget (1.20.3-1ubuntu2.1) over (1.20.3-1ubuntu2) ...
    Selecting previously unselected package apt-transport-https.
    Preparing to unpack .../26-apt-transport-https_2.0.10_all.deb ...
    Unpacking apt-transport-https (2.0.10) ...
    Selecting previously unselected package libcurl4:arm64.
    Preparing to unpack .../27-libcurl4_7.68.0-1ubuntu2.24_arm64.deb ...
    Unpacking libcurl4:arm64 (7.68.0-1ubuntu2.24) ...
    Selecting previously unselected package curl.
    Preparing to unpack .../28-curl_7.68.0-1ubuntu2.24_arm64.deb ...
    Unpacking curl (7.68.0-1ubuntu2.24) ...
    Selecting previously unselected package libdw1:arm64.
    Preparing to unpack .../29-libdw1_0.176-1.1ubuntu0.1_arm64.deb ...
    Unpacking libdw1:arm64 (0.176-1.1ubuntu0.1) ...
    Selecting previously unselected package libbabeltrace1:arm64.
    Preparing to unpack .../30-libbabeltrace1_1.5.8-1build1_arm64.deb ...
    Unpacking libbabeltrace1:arm64 (1.5.8-1build1) ...
    Selecting previously unselected package libpython3.8:arm64.
    Preparing to unpack .../31-libpython3.8_3.8.10-0ubuntu1
    20.04.12_arm64.deb ...
    Unpacking libpython3.8:arm64 (3.8.10-0ubuntu120.04.12) ...
    Selecting previously unselected package gdb.
    Preparing to unpack .../32-gdb_9.2-0ubuntu1
    20.04.2_arm64.deb ...
    Unpacking gdb (9.2-0ubuntu120.04.2) ...
    Selecting previously unselected package libpackagekit-glib2-18:arm64.
    Preparing to unpack .../33-libpackagekit-glib2-18_1.1.13-2ubuntu1.1_arm64.deb ...
    Unpacking libpackagekit-glib2-18:arm64 (1.1.13-2ubuntu1.1) ...
    Selecting previously unselected package gir1.2-packagekitglib-1.0.
    Preparing to unpack .../34-gir1.2-packagekitglib-1.0_1.1.13-2ubuntu1.1_arm64.deb ...
    Unpacking gir1.2-packagekitglib-1.0 (1.1.13-2ubuntu1.1) ...
    Selecting previously unselected package liberror-perl.
    Preparing to unpack .../35-liberror-perl_0.17029-1_all.deb ...
    Unpacking liberror-perl (0.17029-1) ...
    Selecting previously unselected package git-man.
    Preparing to unpack .../36-git-man_1%3a2.25.1-1ubuntu3.13_all.deb ...
    Unpacking git-man (1:2.25.1-1ubuntu3.13) ...
    Selecting previously unselected package git.
    Preparing to unpack .../37-git_1%3a2.25.1-1ubuntu3.13_arm64.deb ...
    Unpacking git (1:2.25.1-1ubuntu3.13) ...
    Selecting previously unselected package libstemmer0d:arm64.
    Preparing to unpack .../38-libstemmer0d_0+svn585-2_arm64.deb ...
    Unpacking libstemmer0d:arm64 (0+svn585-2) ...
    Selecting previously unselected package libappstream4:arm64.
    Preparing to unpack .../39-libappstream4_0.12.10-2_arm64.deb ...
    Unpacking libappstream4:arm64 (0.12.10-2) ...
    Selecting previously unselected package libglib2.0-bin.
    Preparing to unpack .../40-libglib2.0-bin_2.64.6-1
    ubuntu20.04.7_arm64.deb ...
    Unpacking libglib2.0-bin (2.64.6-1ubuntu20.04.7) ...
    Selecting previously unselected package libpolkit-agent-1-0:arm64.
    Preparing to unpack .../41-libpolkit-agent-1-0_0.105-26ubuntu1.3_arm64.deb ...
    Unpacking libpolkit-agent-1-0:arm64 (0.105-26ubuntu1.3) ...
    Selecting previously unselected package pkg-config.
    Preparing to unpack .../42-pkg-config_0.29.1-0ubuntu4_arm64.deb ...
    Unpacking pkg-config (0.29.1-0ubuntu4) ...
    Selecting previously unselected package policykit-1.
    Preparing to unpack .../43-policykit-1_0.105-26ubuntu1.3_arm64.deb ...
    Unpacking policykit-1 (0.105-26ubuntu1.3) ...
    Selecting previously unselected package python3-certifi.
    Preparing to unpack .../44-python3-certifi_2019.11.28-1_all.deb ...
    Unpacking python3-certifi (2019.11.28-1) ...
    Selecting previously unselected package python3-chardet.
    Preparing to unpack .../45-python3-chardet_3.0.4-4build1_all.deb ...
    Unpacking python3-chardet (3.0.4-4build1) ...
    Selecting previously unselected package python3-idna.
    Preparing to unpack .../46-python3-idna_2.8-1ubuntu0.1_all.deb ...
    Unpacking python3-idna (2.8-1ubuntu0.1) ...
    Selecting previously unselected package python3-urllib3.
    Preparing to unpack .../47-python3-urllib3_1.25.8-2ubuntu0.4_all.deb ...
    Unpacking python3-urllib3 (1.25.8-2ubuntu0.4) ...
    Selecting previously unselected package python3-requests.
    Preparing to unpack .../48-python3-requests_2.22.0-2ubuntu1.1_all.deb ...
    Unpacking python3-requests (2.22.0-2ubuntu1.1) ...
    Selecting previously unselected package python3-requests-unixsocket.
    Preparing to unpack .../49-python3-requests-unixsocket_0.2.0-2_all.deb ...
    Unpacking python3-requests-unixsocket (0.2.0-2) ...
    Selecting previously unselected package python3-software-properties.
    Preparing to unpack .../50-python3-software-properties_0.99.9.12_all.deb ...
    Unpacking python3-software-properties (0.99.9.12) ...
    Selecting previously unselected package packagekit.
    Preparing to unpack .../51-packagekit_1.1.13-2ubuntu1.1_arm64.deb ...
    Unpacking packagekit (1.1.13-2ubuntu1.1) ...
    Selecting previously unselected package software-properties-common.
    Preparing to unpack .../52-software-properties-common_0.99.9.12_all.deb ...
    Unpacking software-properties-common (0.99.9.12) ...
    Selecting previously unselected package unzip.
    Preparing to unpack .../53-unzip_6.0-25ubuntu1.2_arm64.deb ...
    Unpacking unzip (6.0-25ubuntu1.2) ...
    Selecting previously unselected package zip.
    Preparing to unpack .../54-zip_3.0-11build1_arm64.deb ...
    Unpacking zip (3.0-11build1) ...
    Selecting previously unselected package sshpass.
    Preparing to unpack .../55-sshpass_1.06-1_arm64.deb ...
    Unpacking sshpass (1.06-1) ...
    Setting up liblmdb0:arm64 (0.9.24-1) ...
    Setting up time (1.7-25.1build1) ...
    Setting up libpython3.8-minimal:arm64 (3.8.10-0ubuntu1
    20.04.12) ...
    Setting up wget (1.20.3-1ubuntu2.1) ...
    Setting up apt-transport-https (2.0.10) ...
    Setting up libyaml-0-2:arm64 (0.2.2-1) ...
    Setting up libglib2.0-0:arm64 (2.64.6-1ubuntu20.04.7) ...
    Setting up distro-info-data (0.43ubuntu1.17) ...
    Setting up unzip (6.0-25ubuntu1.2) ...
    Setting up libpackagekit-glib2-18:arm64 (1.1.13-2ubuntu1.1) ...
    Setting up libcbor0.6:arm64 (0.6.0-0ubuntu1) ...
    Setting up sshpass (1.06-1) ...
    Setting up liberror-perl (0.17029-1) ...
    Setting up python3-six (1.14.0-2) ...
    Setting up ca-certificates (20240203
    20.04.1) ...
    Updating certificates in /etc/ssl/certs...
    rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
    14 added, 5 removed; done.
    Setting up zip (3.0-11build1) ...
    Setting up libglib2.0-data (2.64.6-1ubuntu20.04.7) ...
    Setting up python3-chardet (3.0.4-4build1) ...
    Setting up python3-certifi (2019.11.28-1) ...
    Setting up python3-idna (2.8-1ubuntu0.1) ...
    Setting up pkg-config (0.29.1-0ubuntu4) ...
    Setting up python3-urllib3 (1.25.8-2ubuntu0.4) ...
    Setting up nano (4.8-1ubuntu1.1) ...
    update-alternatives: using /bin/nano to provide /usr/bin/editor (editor) in auto mode
    update-alternatives: warning: skip creation of /usr/share/man/man1/editor.1.gz because associated file /usr/share/man/man1/nano.1.gz (of link group editor) doesn't exist
    update-alternatives: using /bin/nano to provide /usr/bin/pico (pico) in auto mode
    update-alternatives: warning: skip creation of /usr/share/man/man1/pico.1.gz because associated file /usr/share/man/man1/nano.1.gz (of link group pico) doesn't exist
    Setting up python-apt-common (2.0.1ubuntu0.20.04.1) ...
    Setting up python3.8-minimal (3.8.10-0ubuntu1
    20.04.12) ...
    Setting up libcurl4:arm64 (7.68.0-1ubuntu2.24) ...
    Setting up git-man (1:2.25.1-1ubuntu3.13) ...
    Setting up libgirepository-1.0-1:arm64 (1.64.1-1ubuntu20.04.1) ...
    Setting up curl (7.68.0-1ubuntu2.24) ...
    Setting up libstemmer0d:arm64 (0+svn585-2) ...
    Setting up lsb-release (11.1.0ubuntu2) ...
    Setting up libpython3.8-stdlib:arm64 (3.8.10-0ubuntu1
    20.04.12) ...
    Setting up libfido2-1:arm64 (1.3.1-1ubuntu2) ...
    Setting up python3.8 (3.8.10-0ubuntu120.04.12) ...
    Setting up libelf1:arm64 (0.176-1.1ubuntu0.1) ...
    Setting up libpolkit-gobject-1-0:arm64 (0.105-26ubuntu1.3) ...
    Setting up libdw1:arm64 (0.176-1.1ubuntu0.1) ...
    Setting up python3-dbus (1.2.16-1build1) ...
    Setting up python3-apt (2.0.1ubuntu0.20.04.1) ...
    Setting up openssh-client (1:8.2p1-4ubuntu0.11) ...
    Setting up libglib2.0-bin (2.64.6-1
    ubuntu20.04.7) ...
    Setting up libappstream4:arm64 (0.12.10-2) ...
    Setting up libpython3.8:arm64 (3.8.10-0ubuntu120.04.12) ...
    Setting up python3-requests (2.22.0-2ubuntu1.1) ...
    Setting up git (1:2.25.1-1ubuntu3.13) ...
    Setting up gir1.2-glib-2.0:arm64 (1.64.1-1
    ubuntu20.04.1) ...
    Setting up libbabeltrace1:arm64 (1.5.8-1build1) ...
    Setting up libpolkit-agent-1-0:arm64 (0.105-26ubuntu1.3) ...
    Setting up policykit-1 (0.105-26ubuntu1.3) ...
    Setting up python3-requests-unixsocket (0.2.0-2) ...
    Setting up gir1.2-packagekitglib-1.0 (1.1.13-2ubuntu1.1) ...
    Setting up gdb (9.2-0ubuntu120.04.2) ...
    Setting up python3-gi (3.36.0-1) ...
    Setting up packagekit (1.1.13-2ubuntu1.1) ...
    invoke-rc.d: could not determine current runlevel
    invoke-rc.d: policy-rc.d denied execution of force-reload.
    Failed to open connection to "system" message bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
    Created symlink /etc/systemd/user/sockets.target.wants/pk-debconf-helper.socket → /usr/lib/systemd/user/pk-debconf-helper.socket.
    Setting up python3-software-properties (0.99.9.12) ...
    Setting up software-properties-common (0.99.9.12) ...
    Processing triggers for mime-support (3.64ubuntu1) ...
    Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
    Processing triggers for dbus (1.12.16-2ubuntu2.3) ...
    Processing triggers for ca-certificates (20240203
    20.04.1) ...
    Updating certificates in /etc/ssl/certs...
    0 added, 0 removed; done.
    Running hooks in /etc/ca-certificates/update.d...
    done.

  • apt-get clean

  • rm -rf /var/lib/apt/lists/auxfiles /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-backports_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-backports_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-backports_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/repo.download.nvidia.com_jetson_common_dists_r35.4_InRelease /var/lib/apt/lists/repo.download.nvidia.com_jetson_common_dists_r35.4_main_binary-arm64_Packages.lz4

  • gcc --version
    gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
    Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions. There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

  • g++ --version
    g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
    Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions. There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Removing intermediate container b9b5d38123e8
---> 6a4277ba2c46
Step 5/5 : COPY tarpack /usr/local/bin/
---> efa1393a01b9
Successfully built efa1393a01b9
Successfully tagged ollama:r35.5.0-build-essential
-- Building container ollama:r35.5.0-pip_cache_cu114

DOCKER_BUILDKIT=0 docker build --network=host --tag ollama:r35.5.0-pip_cache_cu114
--file /home/apulache/Workspace/jetson-containers/packages/cuda/cuda/Dockerfile.pip
--build-arg BASE_IMAGE=ollama:r35.5.0-build-essential
--build-arg TAR_INDEX_URL="http://jetson.webredirect.org:8000/jp5/cu114"
--build-arg PIP_INDEX_REPO="http://jetson.webredirect.org/jp5/cu114"
--build-arg PIP_TRUSTED_HOSTS="jetson.webredirect.org"
--build-arg PIP_UPLOAD_REPO="http://localhost/jp5/cu114"
--build-arg PIP_UPLOAD_USER="jp5"
--build-arg PIP_UPLOAD_PASS="none"
--build-arg SCP_UPLOAD_URL="localhost:/dist/jp5/cu114"
--build-arg SCP_UPLOAD_USER="None"
--build-arg SCP_UPLOAD_PASS="None"
/home/apulache/Workspace/jetson-containers/packages/cuda/cuda
2>&1 | tee /home/apulache/Workspace/jetson-containers/logs/20241103_005258/build/ollama_r35.5.0-pip_cache_cu114.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

Sending build context to Docker daemon 46.59kB
Step 1/4 : ARG BASE_IMAGE
Step 2/4 : FROM ${BASE_IMAGE}
---> efa1393a01b9
Step 3/4 : ARG PIP_INDEX_REPO PIP_UPLOAD_REPO PIP_UPLOAD_USER PIP_UPLOAD_PASS PIP_TRUSTED_HOSTS TAR_INDEX_URL SCP_UPLOAD_URL SCP_UPLOAD_USER SCP_UPLOAD_PASS
---> Running in 0f04cea5b8d9
Removing intermediate container 0f04cea5b8d9
---> 83face5d7fd1
Step 4/4 : ENV TAR_INDEX_URL=${TAR_INDEX_URL} PIP_INDEX_URL=${PIP_INDEX_REPO} PIP_TRUSTED_HOST=${PIP_TRUSTED_HOSTS} TWINE_REPOSITORY_URL=${PIP_UPLOAD_REPO} TWINE_USERNAME=${PIP_UPLOAD_USER} TWINE_PASSWORD=${PIP_UPLOAD_PASS} SCP_UPLOAD_URL=${SCP_UPLOAD_URL} SCP_UPLOAD_USER=${SCP_UPLOAD_USER} SCP_UPLOAD_PASS=${SCP_UPLOAD_PASS}
---> Running in edbde9bb224a
Removing intermediate container edbde9bb224a
---> a84beb606048
Successfully built a84beb606048
Successfully tagged ollama:r35.5.0-pip_cache_cu114
-- Building container ollama:r35.5.0-cuda

DOCKER_BUILDKIT=0 docker build --network=host --tag ollama:r35.5.0-cuda
--file /home/apulache/Workspace/jetson-containers/packages/cuda/cuda/Dockerfile.builtin
--build-arg BASE_IMAGE=ollama:r35.5.0-pip_cache_cu114
--build-arg CUDA_ARCH_LIST="72;87"
--build-arg DISTRO="ubuntu2004"
/home/apulache/Workspace/jetson-containers/packages/cuda/cuda
2>&1 | tee /home/apulache/Workspace/jetson-containers/logs/20241103_005258/build/ollama_r35.5.0-cuda.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

Sending build context to Docker daemon 46.59kB
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM ${BASE_IMAGE}
---> a84beb606048
Step 3/6 : ARG CUDA_ARCH_LIST DISTRO="ubuntu2004" DEBIAN_FRONTEND=noninteractive
---> Running in 22ece34404ca
Removing intermediate container 22ece34404ca
---> 150d4cbe7788
Step 4/6 : ENV CUDA_HOME="/usr/local/cuda"
---> Running in 91ab35b88d0d
Removing intermediate container 91ab35b88d0d
---> b19994171dfe
Step 5/6 : ENV NVCC_PATH="$CUDA_HOME/bin/nvcc"
---> Running in 47bbdc66e260
Removing intermediate container 47bbdc66e260
---> a442a5f15cbc
Step 6/6 : ENV CUDAARCHS=${CUDA_ARCH_LIST} CUDA_ARCHITECTURES=${CUDA_ARCH_LIST} CUDA_HOME="/usr/local/cuda" CUDNN_LIB_PATH="/usr/lib/aarch64-linux-gnu" CUDNN_LIB_INCLUDE_PATH="/usr/include" CMAKE_CUDA_COMPILER=${NVCC_PATH} CUDA_NVCC_EXECUTABLE=${NVCC_PATH} CUDACXX=${NVCC_PATH} TORCH_NVCC_FLAGS="-Xfatbin -compress-all" CUDA_BIN_PATH="${CUDA_HOME}/bin" CUDA_TOOLKIT_ROOT_DIR="${CUDA_HOME}" PATH="$CUDA_HOME/bin:${PATH}" LD_LIBRARY_PATH="${CUDA_HOME}/compat:${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}" DEBIAN_FRONTEND=noninteractive
---> Running in e1f33f11a404
Removing intermediate container e1f33f11a404
---> f0e3d2f6e348
Successfully built f0e3d2f6e348
Successfully tagged ollama:r35.5.0-cuda
-- Testing container ollama:r35.5.0-cuda (cuda:11.4/test.sh)

docker run -t --rm --runtime=nvidia --network=host
--volume /home/apulache/Workspace/jetson-containers/packages/cuda/cuda:/test
--volume /home/apulache/Workspace/jetson-containers/data:/data
--workdir /test
ollama:r35.5.0-cuda
/bin/bash -c '/bin/bash test.sh'
2>&1 | tee /home/apulache/Workspace/jetson-containers/logs/20241103_005258/test/ollama_r35.5.0-cuda_test.sh.txt; exit ${PIPESTATUS[0]}

{
"cuda" : {
"name" : "CUDA SDK",
"version" : "11.4.19"
},
"cuda_compat" : {
"name" : "CUDA Specific Libraries",
"version" : "11.4.31478197"
},
"cuda_cudart" : {
"name" : "CUDA Runtime (cudart)",
"version" : "11.4.298"
},
"cuda_cuobjdump" : {
"name" : "cuobjdump",
"version" : "11.4.315"
},
"cuda_cupti" : {
"name" : "CUPTI",
"version" : "11.4.298"
},
"cuda_cuxxfilt" : {
"name" : "CUDA cu++ filt",
"version" : "11.4.300"
},
"cuda_gdb" : {
"name" : "CUDA GDB",
"version" : "11.4.298"
},
"cuda_nvcc" : {
"name" : "CUDA NVCC",
"version" : "11.4.315"
},
"cuda_nvdisasm" : {
"name" : "CUDA nvdisasm",
"version" : "11.4.298"
},
"cuda_nvml_dev" : {
"name" : "CUDA NVML Headers",
"version" : "11.4.298"
},
"cuda_nvprune" : {
"name" : "CUDA nvprune",
"version" : "11.4.298"
},
"cuda_nvrtc" : {
"name" : "CUDA NVRTC",
"version" : "11.4.300"
},
"cuda_nvtx" : {
"name" : "CUDA NVTX",
"version" : "11.4.300"
},
"cuda_samples" : {
"name" : "CUDA Samples",
"version" : "11.4.300"
},
"cuda_sanitizer_api" : {
"name" : "CUDA Compute Sanitizer API",
"version" : "11.4.298"
},
"cuda_thrust" : {
"name" : "CUDA C++ Core Compute Libraries",
"version" : "11.4.298"
},
"libcublas" : {
"name" : "CUDA cuBLAS",
"version" : "11.6.6.84"
},
"libcudla" : {
"name" : "CUDA cuDLA",
"version" : "11.4.298"
},
"libcufft" : {
"name" : "CUDA cuFFT",
"version" : "10.6.0.202"
},
"libcurand" : {
"name" : "CUDA cuRAND",
"version" : "10.2.5.297"
},
"libcusolver" : {
"name" : "CUDA cuSOLVER",
"version" : "11.2.0.297"
},
"libcusparse" : {
"name" : "CUDA cuSPARSE",
"version" : "11.6.0.299"
},
"libnpp" : {
"name" : "CUDA NPP",
"version" : "11.4.0.287"
},
"nsight_compute" : {
"name" : "Nsight Compute",
"version" : "2021.2.8.1"
}
}
-- Building container ollama:r35.5.0-ollama

DOCKER_BUILDKIT=0 docker build --network=host --tag ollama:r35.5.0-ollama
--file /home/apulache/Workspace/jetson-containers/packages/llm/ollama/Dockerfile
--build-arg BASE_IMAGE=ollama:r35.5.0-cuda
--build-arg OLLAMA_REPO="ollama/ollama"
--build-arg OLLAMA_BRANCH="main"
--build-arg GOLANG_VERSION="1.22.1"
--build-arg CMAKE_VERSION="3.22.1"
--build-arg JETPACK_VERSION="5.1"
--build-arg CMAKE_CUDA_ARCHITECTURES="72;87"
/home/apulache/Workspace/jetson-containers/packages/llm/ollama
2>&1 | tee /home/apulache/Workspace/jetson-containers/logs/20241103_005258/build/ollama_r35.5.0-ollama.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

Sending build context to Docker daemon 29.7kB
Step 1/17 : ARG BASE_IMAGE CMAKE_CUDA_ARCHITECTURES JETPACK_VERSION OLLAMA_REPO OLLAMA_BRANCH GOLANG_VERSION CMAKE_VERSION
Step 2/17 : FROM ${BASE_IMAGE} AS ollama-l4t-build
---> f0e3d2f6e348
Step 3/17 : ARG OLLAMA_REPO OLLAMA_BRANCH GOLANG_VERSION CMAKE_VERSION CMAKE_CUDA_ARCHITECTURES
---> Running in 788f1df0458c
Removing intermediate container 788f1df0458c
---> 4ac032ef319d
Step 4/17 : ADD https://api.github.com/repos/${OLLAMA_REPO}/git/refs/heads/${OLLAMA_BRANCH} /tmp/ollama_version.json

---> 46150f663673
Step 5/17 : RUN git clone --branch=${OLLAMA_BRANCH} --depth=1 --recursive https://github.com/${OLLAMA_REPO} /opt/ollama
---> Running in 27215777ae9e
Cloning into '/opt/ollama'...
Removing intermediate container 27215777ae9e
---> f04f66ff5b88
Step 6/17 : COPY ollama_deps.sh /opt/ollama_deps.sh
---> b083aa5bc2f0
Step 7/17 : RUN CMAKE_VERSION=${CMAKE_VERSION} GOLANG_VERSION=${GOLANG_VERSION} sh /opt/ollama_deps.sh
---> Running in d2feb82aa2ea

  • uname -m
  • INSTALL_ARCH=aarch64
  • [ -z aarch64 ]
  • apt update

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Get:1 https://repo.download.nvidia.com/jetson/common r35.4 InRelease [2,555 B]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal InRelease [265 kB]
Get:3 https://repo.download.nvidia.com/jetson/common r35.4/main arm64 Packages [19.6 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease [128 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease [128 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [128 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports focal/multiverse arm64 Packages [139 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports focal/universe arm64 Packages [11.1 MB]
Get:9 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 Packages [1,234 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports focal/restricted arm64 Packages [1,317 B]
Get:11 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 Packages [1,467 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 Packages [3,488 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports focal-updates/multiverse arm64 Packages [12.0 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports focal-updates/restricted arm64 Packages [60.3 kB]
Get:15 http://ports.ubuntu.com/ubuntu-ports focal-backports/main arm64 Packages [54.8 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports focal-backports/universe arm64 Packages [27.8 kB]
Get:17 http://ports.ubuntu.com/ubuntu-ports focal-security/main arm64 Packages [3,113 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports focal-security/restricted arm64 Packages [60.0 kB]
Get:19 http://ports.ubuntu.com/ubuntu-ports focal-security/multiverse arm64 Packages [5,907 B]
Get:20 http://ports.ubuntu.com/ubuntu-ports focal-security/universe arm64 Packages [1,176 kB]
Fetched 22.6 MB in 6s (3,583 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
143 packages can be upgraded. Run 'apt list --upgradable' to see them.

  • DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends ca-certificates git gcc-10 g++-10

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (2024020320.04.1).
git is already the newest version (1:2.25.1-1ubuntu3.13).
The following additional packages will be installed:
cpp-10 gcc-10-base libasan6 libatomic1 libcc1-0 libgcc-10-dev libgcc-s1
libgfortran5 libgomp1 libitm1 liblsan0 libstdc++-10-dev libstdc++6 libtsan0
libubsan1
Suggested packages:
gcc-10-locales gcc-10-doc libstdc++-10-doc
The following NEW packages will be installed:
cpp-10 g++-10 gcc-10 libasan6 libgcc-10-dev libstdc++-10-dev
The following packages will be upgraded:
gcc-10-base libatomic1 libcc1-0 libgcc-s1 libgfortran5 libgomp1 libitm1
liblsan0 libstdc++6 libtsan0 libubsan1
11 upgraded, 6 newly installed, 0 to remove and 132 not upgraded.
Need to get 41.8 MB of archives.
After this operation, 134 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libatomic1 arm64 10.5.0-1ubuntu1
20.04 [9,808 B]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libubsan1 arm64 10.5.0-1ubuntu120.04 [765 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libtsan0 arm64 10.5.0-1ubuntu1
20.04 [1,972 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 gcc-10-base arm64 10.5.0-1ubuntu120.04 [20.8 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libstdc++6 arm64 10.5.0-1ubuntu1
20.04 [461 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 liblsan0 arm64 10.5.0-1ubuntu120.04 [800 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libitm1 arm64 10.5.0-1ubuntu1
20.04 [24.1 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libgomp1 arm64 10.5.0-1ubuntu120.04 [93.5 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libgfortran5 arm64 10.5.0-1ubuntu1
20.04 [346 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libcc1-0 arm64 10.5.0-1ubuntu120.04 [46.2 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libgcc-s1 arm64 10.5.0-1ubuntu1
20.04 [34.6 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 cpp-10 arm64 10.5.0-1ubuntu120.04 [7,794 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libasan6 arm64 10.5.0-1ubuntu1
20.04 [2,027 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 libgcc-10-dev arm64 10.5.0-1ubuntu120.04 [907 kB]
Get:15 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 gcc-10 arm64 10.5.0-1ubuntu1
20.04 [15.8 MB]
Get:16 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 libstdc++-10-dev arm64 10.5.0-1ubuntu120.04 [1,746 kB]
Get:17 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 g++-10 arm64 10.5.0-1ubuntu1
20.04 [8,948 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 41.8 MB in 6s (6,469 kB/s)
(Reading database ... 52689 files and directories currently installed.)
Preparing to unpack .../libatomic1_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking libatomic1:arm64 (10.5.0-1ubuntu1
20.04) over (10.3.0-1ubuntu120.04) ...
Preparing to unpack .../libubsan1_10.5.0-1ubuntu1
20.04_arm64.deb ...
Unpacking libubsan1:arm64 (10.5.0-1ubuntu120.04) over (10.3.0-1ubuntu120.04) ...
Preparing to unpack .../libtsan0_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking libtsan0:arm64 (10.5.0-1ubuntu1
20.04) over (10.3.0-1ubuntu120.04) ...
Preparing to unpack .../gcc-10-base_10.5.0-1ubuntu1
20.04_arm64.deb ...
Unpacking gcc-10-base:arm64 (10.5.0-1ubuntu120.04) over (10.3.0-1ubuntu120.04) ...
Setting up gcc-10-base:arm64 (10.5.0-1ubuntu120.04) ...
(Reading database ... 52688 files and directories currently installed.)
Preparing to unpack .../libstdc++6_10.5.0-1ubuntu1
20.04_arm64.deb ...
Unpacking libstdc++6:arm64 (10.5.0-1ubuntu120.04) over (10.3.0-1ubuntu120.04) ...
Setting up libstdc++6:arm64 (10.5.0-1ubuntu120.04) ...
(Reading database ... 52688 files and directories currently installed.)
Preparing to unpack .../0-liblsan0_10.5.0-1ubuntu1
20.04_arm64.deb ...
Unpacking liblsan0:arm64 (10.5.0-1ubuntu120.04) over (10.3.0-1ubuntu120.04) ...
Preparing to unpack .../1-libitm1_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking libitm1:arm64 (10.5.0-1ubuntu1
20.04) over (10.3.0-1ubuntu120.04) ...
Preparing to unpack .../2-libgomp1_10.5.0-1ubuntu1
20.04_arm64.deb ...
Unpacking libgomp1:arm64 (10.5.0-1ubuntu120.04) over (10.3.0-1ubuntu120.04) ...
Preparing to unpack .../3-libgfortran5_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking libgfortran5:arm64 (10.5.0-1ubuntu1
20.04) over (10.3.0-1ubuntu120.04) ...
Preparing to unpack .../4-libcc1-0_10.5.0-1ubuntu1
20.04_arm64.deb ...
Unpacking libcc1-0:arm64 (10.5.0-1ubuntu120.04) over (10.3.0-1ubuntu120.04) ...
Preparing to unpack .../5-libgcc-s1_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking libgcc-s1:arm64 (10.5.0-1ubuntu1
20.04) over (10.3.0-1ubuntu120.04) ...
Setting up libgcc-s1:arm64 (10.5.0-1ubuntu1
20.04) ...
Selecting previously unselected package cpp-10.
(Reading database ... 52688 files and directories currently installed.)
Preparing to unpack .../0-cpp-10_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking cpp-10 (10.5.0-1ubuntu1
20.04) ...
Selecting previously unselected package libasan6:arm64.
Preparing to unpack .../1-libasan6_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking libasan6:arm64 (10.5.0-1ubuntu1
20.04) ...
Selecting previously unselected package libgcc-10-dev:arm64.
Preparing to unpack .../2-libgcc-10-dev_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking libgcc-10-dev:arm64 (10.5.0-1ubuntu1
20.04) ...
Selecting previously unselected package gcc-10.
Preparing to unpack .../3-gcc-10_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking gcc-10 (10.5.0-1ubuntu1
20.04) ...
Selecting previously unselected package libstdc++-10-dev:arm64.
Preparing to unpack .../4-libstdc++-10-dev_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking libstdc++-10-dev:arm64 (10.5.0-1ubuntu1
20.04) ...
Selecting previously unselected package g++-10.
Preparing to unpack .../5-g++-10_10.5.0-1ubuntu120.04_arm64.deb ...
Unpacking g++-10 (10.5.0-1ubuntu1
20.04) ...
Setting up libgomp1:arm64 (10.5.0-1ubuntu120.04) ...
Setting up libasan6:arm64 (10.5.0-1ubuntu1
20.04) ...
Setting up libatomic1:arm64 (10.5.0-1ubuntu120.04) ...
Setting up libgfortran5:arm64 (10.5.0-1ubuntu1
20.04) ...
Setting up libubsan1:arm64 (10.5.0-1ubuntu120.04) ...
Setting up libcc1-0:arm64 (10.5.0-1ubuntu1
20.04) ...
Setting up liblsan0:arm64 (10.5.0-1ubuntu120.04) ...
Setting up cpp-10 (10.5.0-1ubuntu1
20.04) ...
Setting up libitm1:arm64 (10.5.0-1ubuntu120.04) ...
Setting up libtsan0:arm64 (10.5.0-1ubuntu1
20.04) ...
Setting up libgcc-10-dev:arm64 (10.5.0-1ubuntu120.04) ...
Setting up gcc-10 (10.5.0-1ubuntu1
20.04) ...
Setting up libstdc++-10-dev:arm64 (10.5.0-1ubuntu120.04) ...
Setting up g++-10 (10.5.0-1ubuntu1
20.04) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...

  • [ -n 3.22.1 ]
  • curl -s -L https://github.com/Kitware/CMake/releases/download/v3.22.1/cmake-3.22.1-linux-aarch64.tar.gz
  • tar -zx -C /usr --strip-components 1
  • [ -n 1.22.1 ]
  • GO_ARCH=arm64
  • curl -s -L https://dl.google.com/go/go1.22.1.linux-arm64.tar.gz
  • tar xz -C /usr/local
  • ln -s /usr/local/go/bin/go /usr/local/bin/go
  • ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt
  • rm -rf /var/lib/apt/lists/auxfiles /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-backports_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-backports_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-backports_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-updates_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/repo.download.nvidia.com_jetson_common_dists_r35.4_InRelease /var/lib/apt/lists/repo.download.nvidia.com_jetson_common_dists_r35.4_main_binary-arm64_Packages.lz4
  • apt-get clean
    Removing intermediate container d2feb82aa2ea
    ---> f88c836e2250
    Step 8/17 : ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/nvidia/compat:${LD_LIBRARY_PATH} CMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES}
    ---> Running in 8ca0803674d5
    Removing intermediate container 8ca0803674d5
    ---> b030dd2e4d30
    Step 9/17 : RUN cd /opt/ollama && export VERSION="0.0.0" && export GOFLAGS="'-ldflags=-w -s "-X=github.com/ollama/ollama/version.Version=$VERSION" "-X=github.com/ollama/ollama/server.mode=release"'" && export GOARCH=arm64 && echo "OLLAMA_VERSION=${VERSION} GOFLAGS=${GOFLAGS}" && cd llm/generate && sed 's|-j8|-j$(nproc)|' -i gen_common.sh && bash gen_linux.sh && cd ../../ && go build -trimpath .
    ---> Running in 4dea73af9d03
    OLLAMA_VERSION=0.0.0 GOFLAGS='-ldflags=-w -s "-X=github.com/ollama/ollama/version.Version=0.0.0" "-X=github.com/ollama/ollama/server.mode=release"'
    /bin/sh: 1: cd: can't cd to llm/generate
    The command '/bin/sh -c cd /opt/ollama && export VERSION="0.0.0" && export GOFLAGS="'-ldflags=-w -s "-X=github.com/ollama/ollama/version.Version=$VERSION" "-X=github.com/ollama/ollama/server.mode=release"'" && export GOARCH=arm64 && echo "OLLAMA_VERSION=${VERSION} GOFLAGS=${GOFLAGS}" && cd llm/generate && sed 's|-j8|-j$(nproc)|' -i gen_common.sh && bash gen_linux.sh && cd ../../ && go build -trimpath .' returned a non-zero code: 2
    Traceback (most recent call last):
    File "/home/apulache/miniforge3/envs/jetson_containers/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
    File "/home/apulache/miniforge3/envs/jetson_containers/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
    File "/home/apulache/Workspace/jetson-containers/jetson_containers/tag.py", line 58, in
    image = find_container(args.packages[0], prefer_sources=args.prefer, disable_sources=args.disable, user=args.user, quiet=args.quiet)
    File "/home/apulache/Workspace/jetson-containers/jetson_containers/container.py", line 537, in find_container
    return build_container('', package) #, simulate=True)
    File "/home/apulache/Workspace/jetson-containers/jetson_containers/container.py", line 147, in build_container
    status = subprocess.run(cmd.replace(NEWLINE, ' '), executable='/bin/bash', shell=True, check=True)
    File "/home/apulache/miniforge3/envs/jetson_containers/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag ollama:r35.5.0-ollama --file /home/apulache/Workspace/jetson-containers/packages/llm/ollama/Dockerfile --build-arg BASE_IMAGE=ollama:r35.5.0-cuda --build-arg OLLAMA_REPO="ollama/ollama" --build-arg OLLAMA_BRANCH="main" --build-arg GOLANG_VERSION="1.22.1" --build-arg CMAKE_VERSION="3.22.1" --build-arg JETPACK_VERSION="5.1" --build-arg CMAKE_CUDA_ARCHITECTURES="72;87" /home/apulache/Workspace/jetson-containers/packages/llm/ollama 2>&1 | tee /home/apulache/Workspace/jetson-containers/logs/20241103_005258/build/ollama_r35.5.0-ollama.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 2.
    -- Error: return code 1
    V4L2_DEVICES:
  • docker run --runtime nvidia -it --rm --network host --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/apulache/Workspace/jetson-containers/data:/data -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro --device /dev/snd -e PULSE_SERVER=unix:/run/user/1000/pulse/native -v /run/user/1000/pulse:/run/user/1000/pulse --device /dev/bus/usb --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 --device /dev/i2c-9 -v /run/jtop.sock:/run/jtop.sock --name jetson_container_20241103_005656
    "docker run" requires at least 1 argument.
    See 'docker run --help'.

Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Create and run a new container from an image

@dusty-nv
Copy link
Owner

dusty-nv commented Nov 5, 2024

Hi @apulache, ollama updated their build process in ollama/ollama@b754f5a#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52 , and I just updated jetson-containers for it with 4340682 - let me know if you encounter additional errors after pulling the latest 👍

@LukeGotWood
Copy link

Hi @dusty-nv,
I have pulled the new repo and the container built with no errors for r36.3.
I start the container using the following docker compose:

    image: ollama:r36.3.0
    command: /bin/ollama serve &> /data/logs/ollama.log
    runtime: nvidia
    ports:
      - 11434:11434
    environment:
      - OLLAMA_MODELS=/ollama
      - OLLAMA_DEBUG=1
      - OLLAMA_KEEP_ALIVE=-1s
      - OLLAMA_NOHISTORY=1
    ...

I can talk to the api from curl and pull models fine:

curl http://localhost:11434/api/tags
{"models":[{"name":"qwen2.5:3b","model":"qwen2.5:3b","modified_at":"2024-11-06T23:30:20.133572746Z","size":1929912432,"digest":"357c53fb659c5076de1d65ccb0b397446227b71a42be9d1603d46168015c9e4b","details":{"parent_model":"","format":"gguf","family":"qwen2","families":["qwen2"],"parameter_size":"3.1B","quantization_level":"Q4_K_M"}},{"name":"smollm2:latest","model":"smollm2:latest","modified_at":"2024-11-06T23:17:54.468436421Z","size":1820428533,"digest":"cef4a1e09247f018ca0c482ad4c2ce1474aba5e87f245dacf97f07948d05d8b4","details":{"parent_model":"","format":"gguf","family":"llama","families":["llama"],"parameter_size":"1.7B","quantization_level":"Q8_0"}}]}

But when I try to generate completions, I get the following error:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:3b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'
{"error":"llama runner process has terminated: exit status 127"}

The container logs show:

ollama-service-1  | time=2024-11-06T23:52:57.089Z level=INFO source=sched.go:449 msg="loaded runners" count=1
ollama-service-1  | time=2024-11-06T23:52:57.089Z level=INFO source=server.go:562 msg="waiting for llama runner to start responding"
ollama-service-1  | time=2024-11-06T23:52:57.089Z level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server error"
ollama-service-1  | /tmp/ollama1690156987/runners/cuda_v12/ollama_llama_server: error while loading shared libraries: libggml_cuda_v12.so: cannot open shared object file: No such file or directory
ollama-service-1  | time=2024-11-06T23:52:57.340Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: exit status 127"
ollama-service-1  | time=2024-11-06T23:52:57.340Z level=DEBUG source=sched.go:458 msg="triggering expiration for failed load" model=/ollama/blobs/sha256-5ee4f07cdb9beadbbb293e85803c569b01bd37ed059d2715faa7bb405f31caa6
ollama-service-1  | time=2024-11-06T23:52:57.340Z level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/ollama/blobs/sha256-5ee4f07cdb9beadbbb293e85803c569b01bd37ed059d2715faa7bb405f31caa6
ollama-service-1  | time=2024-11-06T23:52:57.340Z level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=/ollama/blobs/sha256-5ee4f07cdb9beadbbb293e85803c569b01bd37ed059d2715faa7bb405f31caa6
ollama-service-1  | [GIN] 2024/11/06 - 23:52:57 | 500 |  519.699022ms |      172.18.0.1 | POST     "/api/generate"
ollama-service-1  | time=2024-11-06T23:52:57.340Z level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="7.4 GiB" before.free="5.2 GiB" before.free_swap="15.7 GiB" now.total="7.4 GiB" now.free="5.2 GiB" now.free_swap="15.7 GiB"
ollama-service-1  | CUDA driver version: 12.2
ollama-service-1  | time=2024-11-06T23:52:57.422Z level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-10e6bc6e-62a9-5e6d-94c3-310167c44dc5 name=Orin overhead="0 B" before.total="7.4 GiB" before.free="5.2 GiB" now.total="7.4 GiB" now.free="5.2 GiB" now.used="2.3 GiB"
ollama-service-1  | releasing cuda driver library

The error seems to be a missing file:

ollama-service-1  | /tmp/ollama1690156987/runners/cuda_v12/ollama_llama_server: error while loading shared libraries: libggml_cuda_v12.so: cannot open shared object file: No such file or directory

I did a search and found this issue: ollama/ollama#7294
which suggests that it could be to do with the LD_LIBRARY_PATH

But searching for libggml_cuda_v12 on the system come up empty so seems something is missing and I'm not sure what

root@a1db4ae601e6:/# cd /usr
root@a1db4ae601e6:/usr# find . | grep libggml_cuda_v12
root@a1db4ae601e6:/usr# cd ..
root@a1db4ae601e6:/# find . | grep libggml_cuda_v12
root@a1db4ae601e6:/#

I hope these logs help when you find time to have a look, thanks in advance!

@LukeGotWood
Copy link

LukeGotWood commented Nov 7, 2024

I did a little bit more digging, I rebuilt the container to confirm it wasn't just a transient issue but it still persists for me.

I did notice during the build that libggml_cuda_v12.so is generated and zipped up so hopefully its an easy fix?
I don't know where to look for it in the build process for this currently but I can have a dig tomorrow. and try to understand.

cp -af /opt/ollama/llama/build/linux-arm64/runners/cuda_v12/libggml_cuda_v12.so /opt/ollama/dist/linux-arm64/lib/ollama/libggml_cuda_v12.so

gzip --best -c /opt/ollama/llama/build/linux-arm64/runners/cuda_v12/ollama_llama_server > /opt/ollama/build/linux/arm64/cuda_v12/ollama_llama_server.gz
cp -af /opt/ollama/llama/build/linux-arm64/runners/cuda_v12/ollama_llama_server /opt/ollama/dist/linux-arm64/lib/ollama/runners/cuda_v12/ollama_llama_server

@jk-vtp-one
Copy link

I had been working on this for a few days waiting for ollama v0.4.0 to release.
The functional problem is that the ollama builder is generating some additional files that need to be included in the runtime image.
This needs to get added to the Dockerfile:

COPY --from=ollama-l4t-build /opt/ollama/dist/linux-arm64/lib/ollama /usr/lib/ollama

This simple fix is here:
https://github.com/jk-vtp-one/jetson-containers/tree/ollama-v0.4.0-build

The next problem is that the current ollama test does not test if the ollama server actually works for inference. The builder worked, and the server starts; but we need to additionally load a model and run it.
Test fixes are here - but I used the ollama python module, so I added the python dependency to the container so I could use pip. It should be possible to do this without it if we want, but this was easier. This branch does not include the build fix, so it currently fails correctly.
https://github.com/jk-vtp-one/jetson-containers/tree/ollama-v0.4.0-test

The final problem is we shouldn't really be targeting the main branch of ollama. With the release of v0.4.0 right now, it seems like a good time to switch the config to target it. You should still be able to force the builder to target main or any branch you want by using the build args; but targeting a specific tag will help prevent these issues especially since ollama has been making changes to their build process.
An alternative branch that I'm using with both these changes and a few others:
https://github.com/jk-vtp-one/jetson-containers/tree/ollama-v0.4.0-complete

What is the best option?

@LukeGotWood
Copy link

Your fix worked great, thank you 👍

And agree with the extra testing. The SmolLM2 model you have set as default is the smallest I could find at a glance too so seems like a really good suggestion that shouldn't slow down the build process to much ether!

@hillct
Copy link

hillct commented Nov 8, 2024

I see the same issue when building on an AGX Orin 32GB.
Pulled updates, got commit f38a230
ran jetson-containers run --name ollama $(autotag ollama)
There were no built-time errors, however the tests failed primarily with the same missing CUDA_12 library:

Successfully tagged ollama:r36.3.0-ollama
-- Testing container ollama:r36.3.0-ollama (ollama/test.sh)

docker run -t --rm --runtime=nvidia --network=host \
--volume /home/hillct/jetson-containers/packages/llm/ollama:/test \
--volume /home/hillct/jetson-containers/data:/data \
--workdir /test \
ollama:r36.3.0-ollama \
/bin/bash -c '/bin/bash test.sh' \
2>&1 | tee /home/hillct/jetson-containers/logs/20241108_151255/test/ollama_r36.3.0-ollama_test.sh.txt; exit ${PIPESTATUS[0]}

testing ollama
Warning: could not connect to a running Ollama instance
Warning: client version is 0.0.0
Large language model runner

Usage:
  ollama [flags]
  ollama [command]

Available Commands:
  serve       Start ollama
  create      Create a model from a Modelfile
  show        Show information for a model
  run         Run a model
  stop        Stop a running model
  pull        Pull a model from a registry
  push        Push a model to a registry
  list        List models
  ps          List running models
  cp          Copy a model
  rm          Remove a model
  help        Help about any command

Flags:
  -h, --help      help for ollama
  -v, --version   Show version information

Use "ollama [command] --help" for more information about a command.
-- Tagging container ollama:r36.3.0-ollama -> ollama:r36.3.0
docker tag ollama:r36.3.0-ollama ollama:r36.3.0

-- Testing container ollama:r36.3.0 (cuda:12.2/test.sh)

docker run -t --rm --runtime=nvidia --network=host \
--volume /home/hillct/jetson-containers/packages/cuda/cuda:/test \
--volume /home/hillct/jetson-containers/data:/data \
--workdir /test \
ollama:r36.3.0 \
/bin/bash -c '/bin/bash test.sh' \
2>&1 | tee /home/hillct/jetson-containers/logs/20241108_151255/test/ollama_r36.3.0_test.sh.txt; exit ${PIPESTATUS[0]}

{
   "cuda" : {
      "name" : "CUDA SDK",
      "version" : "12.2.12"
   },
   "cuda_cccl" : {
      "name" : "CUDA C++ Core Compute Libraries",
      "version" : "12.2.140"
   },
   "cuda_compat" : {
      "name" : "CUDA Specific Libraries",
      "version" : "12.2.34086590"
   },
   "cuda_cudart" : {
      "name" : "CUDA Runtime (cudart)",
      "version" : "12.2.140"
   },
   "cuda_cuobjdump" : {
      "name" : "cuobjdump",
      "version" : "12.2.140"
   },
   "cuda_cupti" : {
      "name" : "CUPTI",
      "version" : "12.2.142"
   },
   "cuda_cuxxfilt" : {
      "name" : "CUDA cu++ filt",
      "version" : "12.2.140"
   },
   "cuda_gdb" : {
      "name" : "CUDA GDB",
      "version" : "12.2.140"
   },
   "cuda_nvcc" : {
      "name" : "CUDA NVCC",
      "version" : "12.2.140"
   },
   "cuda_nvdisasm" : {
      "name" : "CUDA nvdisasm",
      "version" : "12.2.140"
   },
   "cuda_nvml_dev" : {
      "name" : "CUDA NVML Headers",
      "version" : "12.2.140"
   },
   "cuda_nvprune" : {
      "name" : "CUDA nvprune",
      "version" : "12.2.140"
   },
   "cuda_nvrtc" : {
      "name" : "CUDA NVRTC",
      "version" : "12.2.140"
   },
   "cuda_nvtx" : {
      "name" : "CUDA NVTX",
      "version" : "12.2.140"
   },
   "cuda_sanitizer_api" : {
      "name" : "CUDA Compute Sanitizer API",
      "version" : "12.2.140"
   },
   "libcublas" : {
      "name" : "CUDA cuBLAS",
      "version" : "12.2.5.6"
   },
   "libcudla" : {
      "name" : "CUDA cuDLA",
      "version" : "12.2.140"
   },
   "libcufft" : {
      "name" : "CUDA cuFFT",
      "version" : "11.0.8.103"
   },
   "libcufile" : {
      "name" : "GPUDirect Storage (cufile)",
      "version" : "1.7.2.10"
   },
   "libcurand" : {
      "name" : "CUDA cuRAND",
      "version" : "10.3.3.141"
   },
   "libcusolver" : {
      "name" : "CUDA cuSOLVER",
      "version" : "11.5.2.141"
   },
   "libcusparse" : {
      "name" : "CUDA cuSPARSE",
      "version" : "12.1.2.141"
   },
   "libnpp" : {
      "name" : "CUDA NPP",
      "version" : "12.2.1.4"
   },
   "libnvjitlink" : {
      "name" : "JIT Linker Library",
      "version" : "12.2.140"
   },
   "libnvjpeg" : {
      "name" : "CUDA nvJPEG",
      "version" : "12.2.2.4"
   },
   "nsight_compute" : {
      "name" : "Nsight Compute",
      "version" : "2023.2.2.3"
   },
   "nvidia_fs" : {
      "name" : "NVIDIA file-system",
      "version" : "2.17.5"
   }
}
-- Testing container ollama:r36.3.0 (ollama/test.sh)

docker run -t --rm --runtime=nvidia --network=host \
--volume /home/hillct/jetson-containers/packages/llm/ollama:/test \
--volume /home/hillct/jetson-containers/data:/data \
--workdir /test \
ollama:r36.3.0 \
/bin/bash -c '/bin/bash test.sh' \
2>&1 | tee /home/hillct/jetson-containers/logs/20241108_151255/test/ollama_r36.3.0_test.sh.txt; exit ${PIPESTATUS[0]}

testing ollama
Warning: could not connect to a running Ollama instance
Warning: client version is 0.0.0
Large language model runner

Usage:
  ollama [flags]
  ollama [command]

Available Commands:
  serve       Start ollama
  create      Create a model from a Modelfile
  show        Show information for a model
  run         Run a model
  stop        Stop a running model
  pull        Pull a model from a registry
  push        Push a model to a registry
  list        List models
  ps          List running models
  cp          Copy a model
  rm          Remove a model
  help        Help about any command

Flags:
  -h, --help      help for ollama
  -v, --version   Show version information

Use "ollama [command] --help" for more information about a command.
-- Done building container ollama:r36.3.0
ollama:r36.3.0
V4L2_DEVICES: 
### DISPLAY environmental variable is already set: "localhost:10.0"
xhost:  unable to open display "localhost:10.0"
[sudo] password for usernameHere: 
xhost:  unable to open display "localhost:10.0"
xauth:  /tmp/.docker.xauth not writable, changes will be ignored
xauth:  /tmp/.docker.xauth not writable, changes ignored
chmod: changing permissions of '/tmp/.docker.xauth': Operation not permitted
/home/hillct/jetson-containers/run.sh: line 307: /tmp/nv_jetson_model: Is a directory
+ docker run --runtime nvidia -it --rm --network host --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/hillct/jetson-containers/data:/data -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro --device /dev/snd -e PULSE_SERVER=unix:/run/user/1000/pulse/native -v /run/user/1000/pulse:/run/user/1000/pulse --device /dev/bus/usb -e DISPLAY=localhost:10.0 -v /tmp/.X11-unix/:/tmp/.X11-unix -v /tmp/.docker.xauth:/tmp/.docker.xauth -e XAUTHORITY=/tmp/.docker.xauth --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 --device /dev/i2c-9 --name ollama ollama:r36.3.0

Starting ollama server

Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is: 

< Just rust me it go generated... Moving on... >

2024/11/08 15:29:51 routes.go:1189: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/data/models/ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-11-08T15:29:51.453-05:00 level=INFO source=images.go:755 msg="total blobs: 40"
time=2024-11-08T15:29:51.454-05:00 level=INFO source=images.go:762 msg="total unused blobs removed: 0"
time=2024-11-08T15:29:51.454-05:00 level=INFO source=routes.go:1240 msg="Listening on [::]:11434 (version 0.0.0)"
time=2024-11-08T15:29:51.455-05:00 level=INFO source=common.go:135 msg="extracting embedded files" dir=/tmp/ollama881695612/runners
time=2024-11-08T15:29:51.555-05:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cuda_v12 cpu]"
time=2024-11-08T15:29:51.555-05:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
time=2024-11-08T15:29:51.555-05:00 level=WARN source=gpu.go:732 msg="unable to locate gpu dependency libraries"
time=2024-11-08T15:29:51.556-05:00 level=WARN source=gpu.go:732 msg="unable to locate gpu dependency libraries"
time=2024-11-08T15:29:51.556-05:00 level=WARN source=gpu.go:732 msg="unable to locate gpu dependency libraries"
time=2024-11-08T15:29:51.662-05:00 level=INFO source=types.go:123 msg="inference compute" id=GPU-128dc705-e97d-5be0-a41d-6e8b4aec6c7e library=cuda variant=jetpack6 compute=8.7 driver=12.2 name=Orin total="30.0 GiB" available="27.6 GiB"

OLLAMA_MODELS /data/models/ollama/models
OLLAMA_LOGS   /data/logs/ollama.log

ollama server is now started, and you can run commands here like 'ollama run llama3'

root@orindev:/# ollama list
NAME                                ID              SIZE      MODIFIED     
nemotron:70b-instruct-q2_K          f4743c4c6e52    26 GB     2 weeks ago     
codestral:latest                    fcc0019dcee9    12 GB     2 months ago    
nomic-embed-text:latest             0a109f422b47    274 MB    2 months ago    
dolphin-llama-3.1:latest            aa20a1bd5ea4    8.5 GB    3 months ago    
dolphin-yi-1.5-34b:latest           946dd302edef    24 GB     3 months ago    
eas/dolphin-2.2-yi:latest           4b0f884c64ad    20 GB     3 months ago    
codegemma:7b                        0c96700aaada    5.0 GB    4 months ago    
dolphincoder:latest                 677555f1f316    4.2 GB    5 months ago    
dolphincoder:15b-starcoder2-q8_0    a0693c917630    16 GB     6 months ago    
root@orindev:/# ollama run codestral:latest
Error: llama runner process has terminated: exit status 127
root@orindev:/# ls
bin  boot  data  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  start_ollama  sys  tmp  usr  var
root@orindev:/# cat data/logs/
.gitkeep    ollama.log  
root@orindev:/# cat data/logs/ollama.log 
Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is: 

< Just trust me, it got generated... moving on... >

2024/11/08 15:29:51 routes.go:1189: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/data/models/ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-11-08T15:29:51.453-05:00 level=INFO source=images.go:755 msg="total blobs: 40"
time=2024-11-08T15:29:51.454-05:00 level=INFO source=images.go:762 msg="total unused blobs removed: 0"
time=2024-11-08T15:29:51.454-05:00 level=INFO source=routes.go:1240 msg="Listening on [::]:11434 (version 0.0.0)"
time=2024-11-08T15:29:51.455-05:00 level=INFO source=common.go:135 msg="extracting embedded files" dir=/tmp/ollama881695612/runners
time=2024-11-08T15:29:51.555-05:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cuda_v12 cpu]"
time=2024-11-08T15:29:51.555-05:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
time=2024-11-08T15:29:51.555-05:00 level=WARN source=gpu.go:732 msg="unable to locate gpu dependency libraries"
time=2024-11-08T15:29:51.556-05:00 level=WARN source=gpu.go:732 msg="unable to locate gpu dependency libraries"
time=2024-11-08T15:29:51.556-05:00 level=WARN source=gpu.go:732 msg="unable to locate gpu dependency libraries"
time=2024-11-08T15:29:51.662-05:00 level=INFO source=types.go:123 msg="inference compute" id=GPU-128dc705-e97d-5be0-a41d-6e8b4aec6c7e library=cuda variant=jetpack6 compute=8.7 driver=12.2 name=Orin total="30.0 GiB" available="27.6 GiB"
[GIN] 2024/11/08 - 15:30:03 | 200 |        85.6µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/11/08 - 15:30:03 | 200 |    2.018337ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2024/11/08 - 15:30:25 | 200 |      52.384µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/11/08 - 15:30:25 | 200 |   13.130759ms |       127.0.0.1 | POST     "/api/show"
time=2024-11-08T15:30:25.597-05:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/data/models/ollama/models/blobs/sha256-22a849aafe3ded20e9b6551b02684d8fa911537c35895dd2a1bf9eb70da8f69e gpu=GPU-128dc705-e97d-5be0-a41d-6e8b4aec6c7e parallel=4 available=29649371136 required="14.8 GiB"
time=2024-11-08T15:30:25.714-05:00 level=INFO source=server.go:105 msg="system memory" total="30.0 GiB" free="27.7 GiB" free_swap="15.0 GiB"
time=2024-11-08T15:30:25.715-05:00 level=INFO source=memory.go:343 msg="offload to cuda" layers.requested=-1 layers.model=57 layers.offload=57 layers.split="" memory.available="[27.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="14.8 GiB" memory.required.partial="14.8 GiB" memory.required.kv="1.8 GiB" memory.required.allocations="[14.8 GiB]" memory.weights.total="13.2 GiB" memory.weights.repeating="13.0 GiB" memory.weights.nonrepeating="157.5 MiB" memory.graph.full="832.0 MiB" memory.graph.partial="860.3 MiB"
time=2024-11-08T15:30:25.717-05:00 level=INFO source=server.go:383 msg="starting llama server" cmd="/tmp/ollama881695612/runners/cuda_v12/ollama_llama_server --model /data/models/ollama/models/blobs/sha256-22a849aafe3ded20e9b6551b02684d8fa911537c35895dd2a1bf9eb70da8f69e --ctx-size 8192 --batch-size 512 --n-gpu-layers 57 --threads 12 --parallel 4 --port 38685"
time=2024-11-08T15:30:25.717-05:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2024-11-08T15:30:25.717-05:00 level=INFO source=server.go:562 msg="waiting for llama runner to start responding"
time=2024-11-08T15:30:25.718-05:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server error"
**/tmp/ollama881695612/runners/cuda_v12/ollama_llama_server: error while loading shared libraries: libggml_cuda_v12.so: cannot open shared object file: No such file or directory**
time=2024-11-08T15:30:25.969-05:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: exit status 127"
[GIN] 2024/11/08 - 15:30:25 | 500 |  636.163544ms |       127.0.0.1 | POST     "/api/generate"
root@orindev:/# exit

@dusty-nv
Copy link
Owner

dusty-nv commented Nov 8, 2024

Thanks @jk-vtp-one and everyone - will cherry-pick your fixes and tests for these 👍

And you are right @jk-vtp-one about pinning ollama to specific versions like I do other packages that frequently break. I will also add a main config that will always pull the latest for those who want to build it.

@jk-vtp-one
Copy link

And ollama already updated to v0.4.1

https://github.com/ollama/ollama/releases/tag/v0.4.1

- Fixed an issue where the cuda v12 runner on linux was incorrectly linked

That doesn't seem to be our problem. The extra copy is what is needed.

ollama/ollama#7294 (comment)

We expect to find the libraries in ../lib/ollama relative to the primary ollama executable.

@dusty-nv
Copy link
Owner

dusty-nv commented Nov 9, 2024

@jk-vtp-one thanks, it is working for me with your tests on JetPack 6.1 / ollama v0.4.0 - e18584d

(this is in jetson-containers dev branch and the image is on DockerHub at dustynv/ollama:0.4.0-r36.4.0)

@davidADSP
Copy link

This works great for running llama 3.2 11b through ollama on 64gb Orin Dev kit, thanks! However the ollama llama 3.2 90b with q4-k-m still fails for me - can anyone confirm if this should work as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants