Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LlmInputs - trtllm output format support #492

Merged
merged 4 commits into from
Mar 7, 2024

Conversation

nv-braf
Copy link
Contributor

@nv-braf nv-braf commented Mar 6, 2024

Added support for trtLLM endpoints.

Here's a sample output:

{
  "data": [
    {
      "text_input": [
        "You will be given a definition of a task first, then some input of the task.\nThis task is about using the specified sentence and converting the sentence to Resource Description Framework (RDF) triplets of the form (subject, predicate object). The RDF triplets generated must be such that the triplets accurately capture the structure and semantics of the input sentence. The input is a sentence and the output is a list of triplets of the form [subject, predicate, object] that capture the relationships present in the sentence. When a sentence has more than 1 RDF triplet possible, the output must contain all of them.\n\nAFC Ajax (amateurs)'s ground is Sportpark De Toekomst where Ajax Youth Academy also play.\nOutput:"
      ],
      "max_tokens": 256,
      "stream": [
        true
      ]
    },
  ]
}

@nv-braf nv-braf requested a review from debermudez March 6, 2024 23:37
@nv-braf nv-braf requested a review from debermudez March 7, 2024 00:27
@debermudez debermudez merged commit bb9ffe8 into feature-genai-pa Mar 7, 2024
3 checks passed
@debermudez debermudez deleted the llm-inputs-trt-output-format branch March 7, 2024 00:31
debermudez pushed a commit that referenced this pull request Mar 12, 2024
* Adding trtllm endpoint support

* Fixing merge conflicts

* Removing InputFormat

* Renaming methods
debermudez pushed a commit that referenced this pull request Mar 13, 2024
* Adding trtllm endpoint support

* Fixing merge conflicts

* Removing InputFormat

* Renaming methods
mc-nv pushed a commit that referenced this pull request Mar 13, 2024
* Adding trtllm endpoint support

* Fixing merge conflicts

* Removing InputFormat

* Renaming methods
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants