Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tutorial_compile_llama2_with_mlc_llm.ipynb dose not work #32

Open
chosen-ox opened this issue Jan 29, 2024 · 0 comments
Open

Tutorial_compile_llama2_with_mlc_llm.ipynb dose not work #32

chosen-ox opened this issue Jan 29, 2024 · 0 comments

Comments

@chosen-ox
Copy link

I follow this notebook in colab with T4 GPU.
There are several problems:

  • Now the version of cuda is 12.2 instead of 11.8. mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 should be replaced by mlc-ai-nightly-cu122 mlc-chat-nightly-cu122
  • The prebuilt_libs does not work for colab T4 GPU (Get a error CUDA_ERROR_NO_BINARY_FOR_GPU). I compiled the model lib on colab with command !mlc_chat compile ./dist/Llama-2-7b-chat-hf-q4f32_1-MLC/mlc-chat-config.json --device cuda -o dist/Llama-2-7b-chat-hf-q4f32_1-cuda.so and it works to me. Maybe consider update the prebuild_libs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant