Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Implement API Consistency Tests for Remaining Workflow Steps #908

Closed
7 tasks done
junweid62 opened this issue Oct 8, 2024 · 0 comments · Fixed by #937
Closed
7 tasks done

[FEATURE] Implement API Consistency Tests for Remaining Workflow Steps #908

junweid62 opened this issue Oct 8, 2024 · 0 comments · Fixed by #937
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@junweid62
Copy link
Contributor

junweid62 commented Oct 8, 2024

Is your feature request related to a problem?

Currently, only the agent step API consistency test has been completed. There are additional workflow steps that require similar consistency checks to ensure that the API specifications match the required enum parameters. Without these tests, we cannot guarantee that future changes won't break the consistency across other workflow steps.

What solution would you like?

Follow #900 to extend the API consistency test implementation for other workflow steps.

Steps:

  1. Use ApiSpecFetcher to fetch the OpenAPI specification:
    Leverage the ApiSpecFetcher utility introduced in Add ApiSpecFetcher for Fetching and Comparing API Specifications #900 to programmatically fetch the OpenAPI specification for each workflow step.

  2. Utilize the compareRequiredFields method:
    Use the compareRequiredFields method from ApiSpecFetcher to compare the required input fields from the API specification with the predefined enum values for each workflow step.

  3. Integrate ApiConsistencyTest into each step's unit tests:
    For each workflow step (e.g., registerAgent, registerModel), ensure an API consistency check is added in the respective unit tests to validate the correctness of required API parameters.

  4. Add a GitHub Action for daily workflow test execution:
    Once all steps are completed, set up a GitHub Action to trigger a daily run of the workflow tests to ensure API consistency over time.

Do you have any additional context?

The agent step API consistency test has already been implemented. The same approach can be applied to the remaining steps to ensure all workflow steps are covered. This issue is important to maintain consistency in our API as it evolves.

CheckList

ML-Commons Related

  • CreateConnectorStep

DeleteAgentStep(No request body)
DeleteModelStep(No request body)
DeployModelStep(No request body)

  • RegisterAgentStep
  • RegisterLocalCustomModelStep
  • RegisterLocalPretrainedModelStep
  • RegisterLocalSparseEncodingModelStep
  • RegisterModelGroupStep
  • RegisterRemoteModelStep
    UndeployModelStep(No request body)
@junweid62 junweid62 added enhancement New feature or request untriaged labels Oct 8, 2024
@dbwiddis dbwiddis added good first issue Good for newcomers help wanted Extra attention is needed and removed untriaged labels Oct 8, 2024
@junweid62 junweid62 self-assigned this Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
2 participants