Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace existing method of obtaining standing data with SDRS API call #292

Draft
wants to merge 24 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f4c0e1c
add missing attributes to zod parsing
Harriet-Hall Jul 30, 2024
dd08c74
Get only current offences from standing data API
joe-fol Aug 15, 2024
db567ab
Get latests download of deduplicated data
joe-fol Aug 15, 2024
e932405
Mock out pnld download with additonal fields
joe-fol Aug 15, 2024
f1eaa11
Merge new pnld API data into offence-code.json
joe-fol Aug 15, 2024
84ca017
Rename file name to standing-data-api.json
joe-fol Aug 20, 2024
d29daf9
Rename to give better understanding of method used
joe-fol Aug 20, 2024
4feb3c2
Create offence-code.json with latest API download
joe-fol Aug 20, 2024
895d7ac
Remove reference to cjsDownload
joe-fol Aug 20, 2024
8f1f1b9
fixup! Rename file name to standing-data-api-offences.json
joe-fol Aug 20, 2024
8726619
Remove cjsOffence from tests
joe-fol Aug 20, 2024
7cb0180
Remove old method of obtaining cjs download
joe-fol Aug 20, 2024
e9725e5
Remove sort against cjs offences
joe-fol Aug 20, 2024
1a57811
Revert "Remove sort against cjs offences"
joe-fol Aug 21, 2024
0501274
Fix typo in import
joe-fol Aug 21, 2024
10f46e7
Remove cjs-offences.json as no longer needed
joe-fol Aug 21, 2024
9d98fe5
Update github action to use new standing data
joe-fol Aug 21, 2024
35b2f3a
Remove pnld download
joe-fol Aug 21, 2024
ab3ea4e
Remove puppeteer and other redundant packages
joe-fol Aug 21, 2024
4c4afad
Add back URL for downloading OU codes
joe-fol Sep 25, 2024
a0af3f3
Add retry as data call can be flakey
joe-fol Sep 26, 2024
6f86562
Rename file name to match original name
joe-fol Sep 26, 2024
1288c8a
Remove pnld download
joe-fol Sep 26, 2024
561af38
Bump max attempts on api call
joe-fol Oct 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions .github/workflows/update-standing-data.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@ jobs:
name: Update standing data
runs-on: ubuntu-latest
env:
PNLD_DOWNLOAD_URL: ${{ secrets.PNLD_DOWNLOAD_URL }}
PNLD_LOGIN_URL: ${{ secrets.PNLD_LOGIN_URL }}
PNLD_PASSWORD: ${{ secrets.PNLD_PASSWORD }}
PNLD_USERNAME: ${{ secrets.PNLD_USERNAME }}
GOOGLE_API_KEY: ${{ secrets.SHEETSAPIKEY }}
SD_USERNAME: ${{ secrets.SD_USERNAME }}
SD_PASSWORD: ${{ secrets.SD_PASSWORD }}
Expand All @@ -31,8 +27,14 @@ jobs:
- name: Install dependencies
run: npm install

- name: Download offence code data files
run: npm run download-offence-code-data
- name: Download standing data from API
uses: nick-fields/retry@v3
with:
max_attempts: 5
timeout_minutes: 10
retry_on: error
command:
npm run download-standing-data

- name: Merge offence code data files
run: npm run merge-offence-data
Expand Down
9 changes: 2 additions & 7 deletions .ncurc.js
Original file line number Diff line number Diff line change
@@ -1,13 +1,8 @@
// google packages are here because upgrading them
// currently breaks their compatibility
// affects: src/requested-changes-download/sheetsClient.ts
const pinned = [
"google-auth-library",
"eslint",
"@typescript-eslint/eslint-plugin",
"@typescript-eslint/parser"
]
const ignored = ["googleapis", "puppeteer"]
const pinned = ["eslint", "@typescript-eslint/eslint-plugin", "@typescript-eslint/parser"]
const ignored = []
const skipped = []

module.exports = {
Expand Down
Loading