Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Part 3 of #3957: simplify offline builds #3962

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ajpotts
Copy link
Contributor

@ajpotts ajpotts commented Jan 6, 2025

This PR is to demonstrate how an offline build can use dependencies downloaded on one system and install on a second system, using almalinux as an example.

@ajpotts
Copy link
Contributor Author

ajpotts commented Jan 6, 2025

To test:

  1. generate the dependencies: make deps-download-source DEP_BUILD_DIR=<arkouda proj directory>/scripts/offline_builds/almalinux/docker/build
  2. replace the directories in arrow_export.sh with $ARROW_DEPENDENCIES_SOURCE_DIR: sed -i 's/\/home\/amandapotts\/git\/arkouda\/scripts\/offline_builds\/almalinux\/docker\/build\/arrow_dependencies/$ARROW_DEPENDENCIES_SOURCE_DIR/g' scripts/offline_builds/almalinux/docker/build/arrow_exports.sh
  3. cd into the subdirectory: cd scripts/offline_builds/almalinux/docker
  4. create the build.tar: tar -cf build.tar build
  5. pull container with pip dependencies: docker pull ajpotts/almalinux-with-arkouda-deps:1.0.1
  6. run the container: docker run -it ajpotts/almalinux-with-arkouda-deps:1.0.1
  7. From a separate tab, get the container id: docker ps
  8. copy the pip deps, from a separate tab: cd <arkouda proj directory>/scripts/offline_builds/almalinux/docker && docker cp <container name>:/root/pip_deps . && tar -cf pip_deps.tar pip_deps
  9. build the Dockerfile: docker build -t arkouda_test:1.0 .
  10. run the container without network: docker run -it --network none arkouda_test:1.0
  11. Inside the container: pip install --no-index --find-links /root/new_dir/pip_deps/ -r requirements.txt
  12. cd into the arkouda project: cd arkouda
  13. build the dependencies: make install-deps DEP_BUILD_DIR=/root/new_dir/build
  14. pip install the arkouda project: python3 -m pip install -e .[dev]
  15. make arkouda: make
  16. run tests: make test

@ajpotts ajpotts force-pushed the offline_build_scripts_pr2 branch 2 times, most recently from 6fb83bf to 4a31246 Compare January 7, 2025 01:46
@ajpotts ajpotts changed the title Offline build scripts pr2 Part 3 of #3957: simplify offline builds Jan 7, 2025
@ajpotts ajpotts force-pushed the offline_build_scripts_pr2 branch 2 times, most recently from ceebb67 to d7a67cd Compare January 8, 2025 00:43
@ajpotts ajpotts force-pushed the offline_build_scripts_pr2 branch from d7a67cd to af6c9ae Compare January 16, 2025 20:34
@drculhane
Copy link
Contributor

As we discussed in Slack, I had some difficulties in getting this to work, and I think they all trace to the sed in line 2. I'm not sure how to do that better, so I'll just skip ahead to "when I made those edits manually, everything worked." So once again, kudos for some outstanding docker work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants