Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add flask upgrade-db command for running db upgrade with redis lock #5333

Merged
merged 8 commits into from
Jun 18, 2024

Conversation

bowenliang123
Copy link
Contributor

@bowenliang123 bowenliang123 commented Jun 17, 2024

Description

  • currently when MIGRATION_ENABLED env set to true, flask db upgrade command will be executed concurrently in multiple api or worker instances when starting-up, which is lack of locking and will cause hanging status and unresolvable consequences leading the service failing
  • by adding flask upgrade-db command to make sure the lock is acquired from Redis before running the db migrations

Fixes # (issue)

Type of Change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update, included: Dify Document
  • [] Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
  • Dependency upgrade

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • pass CI jobs by running flask db-migrate command with redis

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
  • optional I have made corresponding changes to the documentation
  • optional I have added tests that prove my fix is effective or that my feature works
  • optional New and existing unit tests pass locally with my changes

@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. 🐞 bug Something isn't working labels Jun 17, 2024
@bowenliang123 bowenliang123 changed the title fix: add flask db-migrate command for running db migraition with redis lock support feat: add flask db-migrate command for running db migraition with redis lock support Jun 17, 2024
@bowenliang123
Copy link
Contributor Author

it could be one of long-awaited features for Docker users relying on auto db migration., as fixing a bug in concurrent running db migration concurrently.

If you have any better suggestion for the command name, fell free to raise it.

cc @takatost @JohnJyong

@bowenliang123 bowenliang123 changed the title feat: add flask db-migrate command for running db migraition with redis lock support feat: add flask upgrade-db command for running db upgrade with redis lock support Jun 17, 2024
@bowenliang123 bowenliang123 changed the title feat: add flask upgrade-db command for running db upgrade with redis lock support feat: add flask upgrade-db command for running db upgrade with redis lock Jun 18, 2024
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jun 18, 2024
Copy link
Collaborator

@takatost takatost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@takatost takatost merged commit 5f0ce58 into langgenius:main Jun 18, 2024
8 checks passed
@bowenliang123 bowenliang123 deleted the migrate-py branch June 18, 2024 05:26
ZhouhaoJiang added a commit that referenced this pull request Jun 18, 2024
* origin/main:
  chore: set build system to Poetry and remove unnecessary settings with package mode disabled  (#5263)
  feat: add `flask upgrade-db` command for running db upgrade with redis lock (#5333)
  fix: wrong token usage in iteration node for streaming result (#5336)
ZhouhaoJiang added a commit that referenced this pull request Jun 18, 2024
* refs/heads/feat/llm-ops-tracing: (24 commits)
  feat: change TraceAppConfigApi request type patch
  chore: set build system to Poetry and remove unnecessary settings with package mode disabled  (#5263)
  feat: add `flask upgrade-db` command for running db upgrade with redis lock (#5333)
  fix: wrong token usage in iteration node for streaming result (#5336)
  feat: change table struct
  feat: update trace table
  feat: add remove tracing app
  feat: add llm ops tracing
  feat: add Novita AI image generation tool, implemented model search, text-to-image and create tile functionalities (#5308)
  feat: support Latex (#5001)
  fix: workflow results in FAIL status due to null reference error (#5332)
  chore: update llm.py (#5335)
  fix: extract params by function calling for models supporting tool call (#5334)
  Update and fix the model param of Deepseek (#5329)
  fix: allow special characters in email (#5327)
  fix: got unknown type of prompt message in multi-round ReAct agent chat (#5245)
  chore(core/workflow/utils/variable_template_parser): Refactor VariableTemplateParser class for better readability and maintainability. (#5328)
  fix(core/workflow): Handle special values in node run result outputs (#5321)
  Add bedrock command r models (#4521)
  add the filename length limit (#5326)
  ...
HuberyHuV1 pushed a commit to HuberyHuV1/dify that referenced this pull request Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐞 bug Something isn't working lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants