Add logging to the licence-import import jobs #685
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://eaflood.atlassian.net/browse/WATER-4024 https://eaflood.atlassian.net/browse/WATER-4039 This follows the same pattern used to get [non-noisy logging included in the NALD import jobs](#677). Because the import process is run through jobs, it means for each company and licence to be imported, a 'job' is created. The legacy code would then log each time this happened. But when each entails creating approximately 70K jobs (140K in total!) the logs just become a sea of noise. We implemented a pattern of including information in the job data about how many jobs in total there are, and what was the index of the current job when added to the queue. We can then use this information to get log when the first job is started, and the last is completed. In this way we are only logging the start and end of the 'job' (even though there are thousands of them). This change applies the same pattern to the licence-import company and licence import jobs. We also do some tweaking to bring both NALD import and licence import inline.
Cruikshanks
added
the
housekeeping
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://eaflood.atlassian.net/browse/WATER-4024
https://eaflood.atlassian.net/browse/WATER-4039
This follows the same pattern used to get non-noisy logging included in the NALD import jobs.
Because the import process is run through jobs, it means for each company and licence to be imported, a 'job' is created. The legacy code would then log each time this happened. But when each entails creating approximately 70K jobs (140K in total!) the logs just become a sea of noise.
We implemented a pattern of including information in the job data about how many jobs in total there are, and what was the index of the current job when added to the queue. We can then use this information to log when the first job is started, and the last is completed.
In this way, we are only logging the start and end of the 'job' (even though there are thousands of them).
This change applies the same pattern to the licence-import company and licence-import jobs. We also do some tweaking to bring both NALD import and licence import in line.