Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File System Watcher #28593

Closed
Tracked by #28378
damen-dotcms opened this issue May 17, 2024 · 12 comments · Fixed by #28710, #29067 or #29353
Closed
Tracked by #28378

File System Watcher #28593

damen-dotcms opened this issue May 17, 2024 · 12 comments · Fixed by #28710, #29067 or #29353

Comments

@damen-dotcms
Copy link
Contributor

damen-dotcms commented May 17, 2024

Parent Issue

#28378

Task

We need some sort of tool to monitor file systems for changes, and trigger a push when a change is detected.

The idea is to include a new parameter --watch to the existing push commands (the global push and the specific ones).

After a brainstorming session, we agreed the following:

  • Use Java nio watch service where possible to be notified of changes rather than polling and scanning the folder every time. The watch service only works if the OS supports it and on linux does not work on separate disk, so the code should fallback to polling if the Watch Service is not available.
  • When a new notification is received, we should wait 2 seconds by default before running the push logic to make sure there are no further changes over a minimum time. This would prevent the case where there are logs of changes and we end up having to run the sync multiple times where it could be covered with the once if we waited for a few seconds. The waiting time can be set using a parameter as well, and if not defined, it will default to 2 secs.
  • No explicit path will be provided, the service will use the same path being used by the push command (it includes the case when the --workspace is set)

Proposed Objective

Same as Parent Issue

Proposed Priority

Same as Parent Issue

Acceptance Criteria

Essentially we're aiming for automated deployment capabilities using the CLI.

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

No response

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response

@wezell
Copy link
Contributor

wezell commented May 29, 2024

The watcher sync should probably be Debounced - here is the Debouncer code, not sure if it runs in Quarkus:

https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotcms/concurrent/Debouncer.java

@nollymar nollymar moved this from Next 1-3 Sprints to Current Sprint Backlog in dotCMS - Product Planning May 29, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS moved this from Current Sprint Backlog to In Progress in dotCMS - Product Planning May 29, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 4, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 5, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 6, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 7, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 7, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 7, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 10, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 10, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 10, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS linked a pull request Jun 10, 2024 that will close this issue
@fabrizzio-dotCMS fabrizzio-dotCMS moved this from In Progress to In Review in dotCMS - Product Planning Jun 10, 2024
github-merge-queue bot pushed a commit that referenced this issue Jun 11, 2024
### Proposed Changes
* We're introducing Watch options in the Push Mixin so they become
available to every push subcommand
* We're adding a bit of logic in the `DotExecutionStrategy` to be able
to handle starting up in watch mode and continue to repeat execution of
the command of choice
* I'm making `DotPush` a descendant of `DotCommand` to help me select
all Push commands and still gain access to the mixins.
* Because of the above change I needed to tell the global Push command
from the others, so I introduced a new getter with such info.
* Symmetry-wise wise I made `DotPull` also a descendant of `DotCommand`.
But just for symmetry. All Pull sub commands got modified when I removed
the no longer-needed DotCommand interface. But there are no other
changes in logic.
* A DirectoryWatchService was introduced. this class uses a poll
interval to observe and filter events.
* The events captured by the service are filtered so, let's say a folder
move that in reality creates between 4 and 5 events (FOLDER_MODIFY,
FOLDER_DELETE, FOLDER_MODIFY, FOLDER_CREATE), These events are filtered
so that we only get the most recent one. This ensures we do not flood
the server side with many unneeded changes.
* Changes in documentation were made
* Tests were introduced for the `DirectoryWatchService`, The Global Push
running with the watch mode flag on, and finally another one to cover
the changes introduced in the `DotExecutionStrategy`
@github-project-automation github-project-automation bot moved this from In Review to Internal QA in dotCMS - Product Planning Jun 11, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS removed their assignment Jun 11, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 24, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 29, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 29, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 29, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 29, 2024
@nollymar nollymar moved this from In Progress to In Review in dotCMS - Product Planning Jul 29, 2024
@nollymar nollymar linked a pull request Jul 29, 2024 that will close this issue
github-merge-queue bot pushed a commit that referenced this issue Jul 29, 2024
### Proposed Changes

1. When the` files push ` command gets called from the global push,
passing an existing folder but outside the files folder an error is
thrown
2. This change makes the command default to the workspace root folder
and continues processing when the scenario described in the first point
occurs
@github-project-automation github-project-automation bot moved this from In Review to Internal QA in dotCMS - Product Planning Jul 29, 2024
@github-project-automation github-project-automation bot moved this from Internal QA to Current Sprint Backlog in dotCMS - Product Planning Jul 30, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS moved this from Current Sprint Backlog to In Review in dotCMS - Product Planning Jul 30, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS removed their assignment Jul 30, 2024
@nollymar nollymar moved this from In Review to Internal QA in dotCMS - Product Planning Jul 30, 2024
@spbolton
Copy link
Contributor

Internal QA. Tested global push and pull changing site with and without watch option as well as running from sites folder and specifying path. No error messages and changes were pushed as expected

Screenshot 2024-08-15 at 14 08 05

@spbolton spbolton moved this from Internal QA to QA - Backlog in dotCMS - Product Planning Aug 15, 2024
@github-project-automation github-project-automation bot moved this from QA - Backlog to Internal QA in dotCMS - Product Planning Aug 15, 2024
@nollymar nollymar moved this from Internal QA to QA - Backlog in dotCMS - Product Planning Aug 15, 2024
@bryanboza
Copy link
Contributor

Fixed and tested on the latest CLI version, and now it is working as expected. A new error regarding the exceptions has been opened here: #29629

We can close this one for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment