Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Colab training is not accelerated because tensorflow 2.9 is forced to be installed with the [pc] option #1159

Open
sctse999 opened this issue Jan 23, 2024 · 1 comment

Comments

@sctse999
Copy link
Contributor

In the latest release 5.0.0, tensorflow==2.9 is added to

tensorflow==2.9

When the command !pip3 install -e .[pc] is ran on a colab GPU instance, it will automatically uninstall the GPU-capable tensorflow and install tensorflow 2.9.

We can see that before 5.0.0 (e.g. v5.0.dev3 ), the output of the training is like this:

2024-01-23 11:53:05.172074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13949 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
INFO:donkeycar.parts.keras:Created KerasLinear with interpreter: KerasInterpreter

whereas in 5.0.0, the output of the training is like this:

2024-01-23 11:56:07.005500: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

I understand that including tensorflow in the pc option would facilitate a smoother installation experience. How about if we add one more option named colab which would be the same as pc but without tensorflow? It would allow users to train models on colab without editing the setup.cfg manually.

@sctse999
Copy link
Contributor Author

@Ezward @DocGarbanzo do you have any advice?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant