Add return requirements clean up step to import #1017
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://eaflood.atlassian.net/browse/WATER-4655 https://eaflood.atlassian.net/browse/WATER-4651 > Part of the work to migrate management of return requirements from NALD to WRLS Having made return versions and requirements visible to our internal users they have spotted that there are return requirements displayed in WRLS that don't exist. We know why this is: NALD allows users to delete records. This means a user can create a record in NALD on day 1, we'll see and import the record that night. They can then delete the record (in most cases it is because a mistake was spotted) on day 2. The import _never deletes_ records so the errant return requirement remains. With us taking over management of return requirements though it has become important to try and get the 2 systems in sync as much as possible before the import is switched off. This change will add a new step into the charging data import job to tackle clean up. It will focus on return requirements that no longer exist in NALD and that do not have a return log in WRLS first.
Cruikshanks
added a commit
that referenced
this pull request
Sep 2, 2024
https://eaflood.atlassian.net/browse/WATER-4654 > Part of the work to migrate management of return requirements from NALD to WRLS For reasons detailed in [Add return requirements clean up step to import](#1017) we need to add a new step into the existing 'charging-data' job. But the job itself is already a bit of a mish-mash. - it is called 'charging-data' but doesn't do that any more - it brings in fixed reference data, return versions and mod logs all in one go We know this will soon be ditched, but it is a right old mess as it is. We could be stopping some of the stuff right now, and split out the rest to make it a lot clearer as to what bit is doing what. This wouldn't be much more effort than trying to slot a new step into the existing mess. So, this change we perform some much needed tidying up of the 'charging-data job'. This should make the change in PR #1017 much easier!
Cruikshanks
added a commit
that referenced
this pull request
Sep 3, 2024
https://eaflood.atlassian.net/browse/WATER-4654 > Part of the work to migrate management of return requirements from NALD to WRLS For reasons detailed in [Add return requirements clean up step to import](#1017), we need to add a new step to the existing 'charging-data' job. But the job itself is already a bit of a mess. - it is called 'charging-data' but doesn't do that any more - it brings in fixed reference data, return versions and mod logs all in one go We know this will soon be ditched, but it is a right-old mess. We could stop some of the stuff right now and split out the rest to make it clearer which bit is doing what. This wouldn't be much more effort than slotting a new step into the mess. So, in this change, we perform some much-needed tidying up of the 'charging-data job'. This should make the change in PR #1017 much easier!
This means we can keep our import and clean queries distinct.
The step isn't doing anything at the moment. But this allows us to check that clean can be triggered followed by the import (it can!)
Cruikshanks
added a commit
that referenced
this pull request
Oct 14, 2024
https://eaflood.atlassian.net/browse/WATER-4657 > Part of the work to migrate management of return requirements from NALD to WRLS Having made return versions and requirements visible to our internal users they have spotted that there are return requirements displayed in WRLS that don't exist. We know why this is: NALD allows users to delete records. This means a user can create a record in NALD on day 1; we'll see and import the record that night. They can then delete the record (in most cases, it is because a mistake was spotted) on day 2. The import never deletes records, so the errant return requirement remains. With us taking over management of return requirements, though, it has become important to try to get the 2 systems in sync as much as possible before the import is switched off. This change builds on the work done in [Add return requirements clean up step to import](#1017). It added the job and the script to remove return requirements that no longer exists in NALD and were not linked to any return logs. Now we're adding a query to delete any return requirements that no longer exist in NALD and are only linked to `void` return logs.
Cruikshanks
added a commit
that referenced
this pull request
Oct 15, 2024
https://eaflood.atlassian.net/browse/WATER-4657 > Part of the work to migrate management of return requirements from NALD to WRLS Having made return versions and requirements visible to our internal users, they have noticed that there are return requirements displayed in WRLS that don't exist. We know why this is: NALD allows users to delete records. This means a user can create a record in NALD on day 1; we'll see and import the record that night. They can then delete the record (in most cases, it is because a mistake was spotted) on day 2. The import never deletes records, so the errant return requirement remains. With us taking over management of return requirements, it has become important to try to get the two systems in sync as much as possible before the import is switched off. This change builds on the work done in [Add return requirements clean-up step to import](#1017). It added the job and the script to remove return requirements that no longer exist in NALD and were not linked to any return logs. Now, we're adding a query to delete any return requirements that no longer exist in NALD and are only linked to `void` return logs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://eaflood.atlassian.net/browse/WATER-4655
https://eaflood.atlassian.net/browse/WATER-4651
Having made return versions and requirements visible to our internal users they have spotted that there are return requirements displayed in WRLS that don't exist.
We know why this is: NALD allows users to delete records. This means a user can create a record in NALD on day 1; we'll see and import the record that night. They can then delete the record (in most cases, it is because a mistake was spotted) on day 2. The import never deletes records, so the errant return requirement remains.
With us taking over management of return requirements, though, it has become important to try to get the 2 systems in sync as much as possible before the import is switched off.
This change will add a new step to the return versions import job to tackle clean-up. First, it will focus on return requirements that no longer exist in NALD and that do not have a return log in WRLS.
We'll look at return requirements that are linked to return logs in future changes.