requirement and pyproject.toml sync checking

[Dhruv-Sachdev1313]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • solves for Issue [Bug]: Ensure pyproject.toml is correct as part of CI for PRs. [Bug]: Ensure pyproject.toml is correct as part of CI for PRs. #564
• New functionality
  • Introduces a bash Script to check if we have dependencies in 'pyproject.toml' and 'requirements.txt' in sync

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readbility, Modularity, Intuitiveness)

[Dhruv-Sachdev1313]

I am yet to add a GitHub Workflow to make it a part of every CI process. Before I do that I just wanted to make sure if this script is fine in terms of the issue (Script works for sure!), hence the Draft PR.
@levand @jeffchuber @HammadB
Please change the Base branch, I was not sure which branch to make a PR against

[tazarov]

@Dhruv-Sachdev1313, thank you for the script. Do you think it makes sense to step back for a second and understand the historical reason for having separate pyproject.toml and requirements.txt?

Without knowing the history, I can only deduce that requirements.txt is just a reflection of pyproject.toml's dependencies section. If that is the case, I wonder if it also makes sense to have instead a single source of truth that lies with pyproject.toml and generate requirements.txt from it as needed.

[Dhruv-Sachdev1313]

@Dhruv-Sachdev1313, thank you for the script. Do you think it makes sense to step back for a second and understand the historical reason for having separate pyproject.toml and requirements.txt?

Without knowing the history, I can only deduce that requirements.txt is just a reflection of pyproject.toml's dependencies section. If that is the case, I wonder if it also makes sense to have instead a single source of truth that lies with pyproject.toml and generate requirements.txt from it as needed.

Hey @tazarov: I was wondering about it as well, I am new to the repository and found an issue stating this(Ensure pyproject.toml is correct as part of CI for PRs. #564 ) It said, The dependencies in requirements.txt and pyproject.toml must match in order for the project to work correctly in development and when deployed.
Now there might be a good chance that I misunderstood the objective of the issue. If that is so could you please help me out here?

[tazarov]

Hey @tazarov: I was wondering about it as well, I am new to repository and found an issue stating this(Ensure pyproject.toml is correct as part of CI for PRs. #564 ) It said, The dependencies in requirements.txt and pyproject.toml must match in order for the project to work correctly in development and when deployed.
Now there might be a good chance that I misunderstood the objective to the issue. If that is so could you please help me out here?

Hey @Dhruv-Sachdev1313, actually, I went through to #564, and I can see your understanding is correct. I am wondering, though, whether there is a more subtle way to solve the problem at hand or just reduce the problem to maintaining a single dependency graph rather than trying to maintain and synchronize two, which can lead to the original problem of this issue.

@HammadB, wdyt? Any idea why we would need two sources of dependencies?

[jeffchuber]

@Dhruv-Sachdev1313 @tazarov doing some cleanup here - is this something we still want to merge? want to close?

[tazarov]

@jeffchuber, I think it'd be foolish to ignore that out-of-sync dependencies between pyproject.toml and requirements.txt have and can cause issues for users. I think it would be wise to consider the sync problem; I am unsure if using the script in its current form is the right solution.

We can either close this PR and spend a moment to think about the problem of sync and perhaps have a well-reasoned CIP to back any choices made, or we can accept the PR as is and build upon it, knowing that this is a step in the right direction.

I am okay with either approach.

[jeffchuber]

Unfortunately this is quite out of date (due to no fault of the OP!) - for hygiene I am going to close this now.