Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GenAIOrchestrator] add huggingface tgi - 1665 #1702

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[GenAIOrchestrator] add huggingface tgi - 1665 #1702

wants to merge 3 commits into from

Conversation

morgandiverrez
Copy link
Contributor

add LLM provider hugging face tgi
issue 1665

@morgandiverrez morgandiverrez self-assigned this Jul 26, 2024
@morgandiverrez morgandiverrez linked an issue Jul 26, 2024 that may be closed by this pull request
[GenAIOrchestrator] Intégrer hugging face TGI #1665
Open
@morgandiverrez morgandiverrez marked this pull request as ready for review July 26, 2024 13:15
@morgandiverrez
Copy link
Contributor Author

morgandiverrez commented Jul 26, 2024 •
edited
Loading

Do not merge until the PR on huggingface_hub is merged. The pyproject.toml points to a fork of this library in the meantime.

@morgandiverrez morgandiverrez marked this pull request as draft July 26, 2024 13:23
@Benvii Benvii mentioned this pull request Aug 6, 2024
Closed
@morgandiverrez
Copy link
Contributor Author

PR on huggingface_hub is merged, ready for review

@morgandiverrez morgandiverrez marked this pull request as ready for review August 14, 2024 09:32
@morgandiverrez morgandiverrez force-pushed the 1665-genaiorchestrator-integrer-hugging-face-tgi branch from 328fcdf to def2e65 Compare August 27, 2024 15:11
@Benvii
Copy link
Member

Benvii commented Aug 27, 2024

This PR installs a tons of dependencies for instance :

  • nvidia-cudnn-cu12 (9.1.0.70): Downloading... 10%
    • Installing nvidia-cufft-cu12 (11.0.2.54): Downloading... 60%
    • Installing nvidia-cusolver-cu12 (11.4.5.107): Downloading... 59%
    • Installing nvidia-nccl-cu12
    • Installing torch (2.4.0): Downloading... 44%

It seems quite huge, I need to take a deeper look at that, a simple "huggingface hub client" shouldn't require any nvidia (cuda) dependencies, maybe you can isolate a sub group of dependencies. Otherwise the docker image will become too usage

@morgandiverrez
Copy link
Contributor Author

The langchain_huggingface library heavily relies on other libraries, such as CUDA and PyTorch. I found a discussion on GitHub about this: langchain-ai/langchain#24482

@morgandiverrez
Copy link
Contributor Author

A discussion was create on langchain about this probleme

@Benvii
Copy link
Member

Benvii commented Nov 20, 2024

We are discussing this integration internally at CM Arkéa, we are no longer sure that TGI will be used as our inference server meaning we wouldn't be able to maintain this integration.

Waiting for a clear position on this subject on our side (test in progress to use OpenAI integration for vLLM). If any tock community user uses TGI and want this integration feel free to leave a comment here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Benvii Benvii Awaiting requested review from Benvii

@assouktim assouktim Awaiting requested review from assouktim

At least 2 approving reviews are required to merge this pull request.

Assignees

@morgandiverrez morgandiverrez

Labels
None yet
Projects
CreditMutuelArkea backlog
Status: In progress
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[GenAIOrchestrator] Intégrer hugging face TGI
2 participants
@morgandiverrez @Benvii