Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenAi endpoint support #476

Merged
merged 23 commits into from
Mar 6, 2024
Merged

OpenAi endpoint support #476

merged 23 commits into from
Mar 6, 2024

Conversation

tgerdesnv
Copy link
Collaborator

@tgerdesnv tgerdesnv commented Feb 28, 2024 •
edited
Loading

Added ability to drive OpenAI endpoints via a new backend_client

  • new "openAI" service-kind
  • new "endpoint" option in CLI
  • Added OpenAI into model parser
    • All OpenAI models have a single input called "payload" that will take any json format
    • All OpenAI models have a single output called "response" that will contain the response from the server

@tgerdesnv tgerdesnv force-pushed the tgerdes-endpoint-scaffolding branch 3 times, most recently from abad69f to 8a9f910 Compare March 4, 2024 15:51
@tgerdesnv tgerdesnv changed the title draft: endpoint scaffolding OpenAi endpoint support Mar 4, 2024
@tgerdesnv tgerdesnv changed the base branch from feature-endpoint-openai to feature-genai-pa March 4, 2024 16:48
@tgerdesnv tgerdesnv changed the base branch from feature-genai-pa to feature-endpoint-openai March 4, 2024 16:48
@tgerdesnv tgerdesnv force-pushed the tgerdes-endpoint-scaffolding branch from 9f98ee8 to e1cb83d Compare March 4, 2024 17:26
@tgerdesnv tgerdesnv marked this pull request as ready for review March 4, 2024 17:34
dyastremsky
dyastremsky reviewed Mar 4, 2024
View reviewed changes
Copy link
Contributor

@dyastremsky dyastremsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work here! This looks fantastic.

Added a few minor comments.

src/c++/perf_analyzer/client_backend/openai/http_client.cc Show resolved Hide resolved
src/c++/perf_analyzer/client_backend/openai/openai_client.cc Outdated Show resolved Hide resolved
src/c++/perf_analyzer/client_backend/openai/openai_client.h Outdated Show resolved Hide resolved
src/c++/perf_analyzer/command_line_parser.cc Outdated Show resolved Hide resolved
src/c++/perf_analyzer/command_line_parser.cc Outdated Show resolved Hide resolved
oandreeva-nv
oandreeva-nv approved these changes Mar 5, 2024
View reviewed changes
Copy link
Contributor

@oandreeva-nv oandreeva-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

tgerdesnv and others added 22 commits March 6, 2024 16:10
@tgerdesnv
Add openai to model parser
269f4f9
@oandreeva-nv @tgerdesnv
OpenAI client backend + cmake
2342215
@tgerdesnv
Create OpenAI backend
36e6ceb
@tgerdesnv
New JSON datatype for PA. Show json data available at http_client level
ccd0b68
@tgerdesnv
Add an output to OpenAI models
4b841b3
@GuanLuo @tgerdesnv
Add OpenAI client (#482)
e9f4a22 
* Add OpenAI client

* Address comment
@tgerdesnv
Pass endpoint to openai client
4ab2fc1
@tgerdesnv
Resolve fixmes
ffbf154
@tgerdesnv
update copyright years
4a8684e
@tgerdesnv
more cleanup
9e8533e
@tgerdesnv @dyastremsky
Update src/c++/perf_analyzer/command_line_parser.cc
3811848 
Co-authored-by: dyastremsky <[email protected]>
@tgerdesnv
remove 'file' from top of files
305e96c
@tgerdesnv
clean up help message and add endpoint to help
dd4c4ca
@tgerdesnv
Fix client stats
cac8bff
@tgerdesnv
remove unused fn
5b7434a
@tgerdesnv
Assert on json input format
5eab7b5
@tgerdesnv
Use a single SEND_END point
603631a
@tgerdesnv
Add sync assert. Add OPENAI to helper fn
40d6465
@tgerdesnv
remove unused typedef
7827fee
@tgerdesnv
Add batch size assert
7df09ef
@GuanLuo @tgerdesnv
Address comment (#487)
6052176 
* Address comment

* Update src/c++/perf_analyzer/client_backend/openai/openai_client.cc

* Update src/c++/perf_analyzer/client_backend/openai/http_client.cc

* formatting

---------

Co-authored-by: Timothy Gerdes <[email protected]>
Co-authored-by: tgerdes <[email protected]>
@tgerdesnv
Make copy of exported data so it isn't corrupted (#488)
46a03db
@tgerdesnv tgerdesnv force-pushed the tgerdes-endpoint-scaffolding branch from 906f54f to 46a03db Compare March 6, 2024 22:10
@tgerdesnv tgerdesnv changed the base branch from feature-endpoint-openai to feature-genai-pa March 6, 2024 22:56
@tgerdesnv tgerdesnv merged commit a11ffa2 into feature-genai-pa Mar 6, 2024
3 checks passed
@tgerdesnv tgerdesnv deleted the tgerdes-endpoint-scaffolding branch March 6, 2024 23:38
debermudez pushed a commit that referenced this pull request Mar 12, 2024
@tgerdesnv @oandreeva-nv
@GuanLuo @dyastremsky @debermudez
OpenAi endpoint support (#476)
9c130de 
* Add openai service-kind and add endpoint to CLI

* Add openai to model parser

* OpenAI client backend + cmake

* Create OpenAI backend

* New JSON datatype for PA. Show json data available at http_client level

* Add an output to OpenAI models

* Add OpenAI client (#482)

* Add OpenAI client

* Address comment

* Pass endpoint to openai client

* Resolve fixmes

* update copyright years

* more cleanup

* Update src/c++/perf_analyzer/command_line_parser.cc

Co-authored-by: dyastremsky <[email protected]>

* remove 'file' from top of files

* clean up help message and add endpoint to help

* Fix client stats

* remove unused fn

* Assert on json input format

* Use a single SEND_END point

* Add sync assert. Add OPENAI to helper fn

* remove unused typedef

* Add batch size assert

* Address comment (#487)

* Address comment

* Update src/c++/perf_analyzer/client_backend/openai/openai_client.cc

* Update src/c++/perf_analyzer/client_backend/openai/http_client.cc

* formatting

---------

Co-authored-by: Timothy Gerdes <[email protected]>
Co-authored-by: tgerdes <[email protected]>

* Make copy of exported data so it isn't corrupted (#488)

---------

Co-authored-by: oandreeva-nv <[email protected]>
Co-authored-by: GuanLuo <[email protected]>
Co-authored-by: dyastremsky <[email protected]>
debermudez pushed a commit that referenced this pull request Mar 13, 2024
@tgerdesnv @oandreeva-nv
@GuanLuo @dyastremsky @debermudez
OpenAi endpoint support (#476)
7b0cfae 
* Add openai service-kind and add endpoint to CLI

* Add openai to model parser

* OpenAI client backend + cmake

* Create OpenAI backend

* New JSON datatype for PA. Show json data available at http_client level

* Add an output to OpenAI models

* Add OpenAI client (#482)

* Add OpenAI client

* Address comment

* Pass endpoint to openai client

* Resolve fixmes

* update copyright years

* more cleanup

* Update src/c++/perf_analyzer/command_line_parser.cc

Co-authored-by: dyastremsky <[email protected]>

* remove 'file' from top of files

* clean up help message and add endpoint to help

* Fix client stats

* remove unused fn

* Assert on json input format

* Use a single SEND_END point

* Add sync assert. Add OPENAI to helper fn

* remove unused typedef

* Add batch size assert

* Address comment (#487)

* Address comment

* Update src/c++/perf_analyzer/client_backend/openai/openai_client.cc

* Update src/c++/perf_analyzer/client_backend/openai/http_client.cc

* formatting

---------

Co-authored-by: Timothy Gerdes <[email protected]>
Co-authored-by: tgerdes <[email protected]>

* Make copy of exported data so it isn't corrupted (#488)

---------

Co-authored-by: oandreeva-nv <[email protected]>
Co-authored-by: GuanLuo <[email protected]>
Co-authored-by: dyastremsky <[email protected]>
mc-nv pushed a commit that referenced this pull request Mar 13, 2024
@tgerdesnv @oandreeva-nv
@GuanLuo @dyastremsky @mc-nv
OpenAi endpoint support (#476)
f99b072 
* Add openai service-kind and add endpoint to CLI

* Add openai to model parser

* OpenAI client backend + cmake

* Create OpenAI backend

* New JSON datatype for PA. Show json data available at http_client level

* Add an output to OpenAI models

* Add OpenAI client (#482)

* Add OpenAI client

* Address comment

* Pass endpoint to openai client

* Resolve fixmes

* update copyright years

* more cleanup

* Update src/c++/perf_analyzer/command_line_parser.cc

Co-authored-by: dyastremsky <[email protected]>

* remove 'file' from top of files

* clean up help message and add endpoint to help

* Fix client stats

* remove unused fn

* Assert on json input format

* Use a single SEND_END point

* Add sync assert. Add OPENAI to helper fn

* remove unused typedef

* Add batch size assert

* Address comment (#487)

* Address comment

* Update src/c++/perf_analyzer/client_backend/openai/openai_client.cc

* Update src/c++/perf_analyzer/client_backend/openai/http_client.cc

* formatting

---------

Co-authored-by: Timothy Gerdes <[email protected]>
Co-authored-by: tgerdes <[email protected]>

* Make copy of exported data so it isn't corrupted (#488)

---------

Co-authored-by: oandreeva-nv <[email protected]>
Co-authored-by: GuanLuo <[email protected]>
Co-authored-by: dyastremsky <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mc-nv mc-nv mc-nv left review comments

@dyastremsky dyastremsky dyastremsky left review comments

@GuanLuo GuanLuo GuanLuo approved these changes

@rmccorm4 rmccorm4 rmccorm4 left review comments

@oandreeva-nv oandreeva-nv oandreeva-nv approved these changes

@matthewkotila matthewkotila matthewkotila approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@tgerdesnv @matthewkotila @rmccorm4 @GuanLuo @dyastremsky @mc-nv @oandreeva-nv