Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data fix for wrongly deleted document headers #1048

Merged
merged 2 commits into from
Aug 16, 2024

Conversation

Cruikshanks
Copy link
Member

https://eaflood.atlassian.net/browse/WATER-4627

We received a report that an external user was unable to link MD/054/0009/047 to their account (it returned the error "We cannot find these licence numbers"). We reproduced the issue in pre-prod and tracked down the cause.

When you add a licence number, the legacy code runs this query to pull it from CRM.document_headers.

SELECT
  "crm"."document_header" .*
FROM
  "crm"."document_header"
WHERE
  "crm"."document_header"."system_external_id" = $1
  AND "crm"."document_header"."company_entity_id" IS NULL
  AND "crm"."document_header"."metadata"->'IsCurrent' != $2
  AND "crm"."document_header"."date_deleted" IS NULL
  AND "crm"."document_header"."metadata"->>'IsCurrent' != $3
ORDER BY
  "crm"."document_header"."system_external_id" ASC
LIMIT $4
-- [ 'MD/054/0009/047', 'false', 'false', 300 ]

On 2023-10-03 the water-abstraction-import marked the crm.document_header record as deleted. This is a step in the import process where it checks for deleted 'document'.

// src/lib/connectors/import.js
const deleteCrmV1DocumentsQuery = `
  update crm.document_header
  set date_deleted = now()
  where system_external_id not in (
    select l."LIC_NO"
    from import."NALD_ABS_LICENCES" l
  )
  and date_deleted is null
  and regime_entity_id = '0434dc31-a34e-7158-5775-4694af7a60cf';
`

The knowledge of why the query didn't see this licence in the NALD data on that date is now lost to the universe! However, we can clearly see that the licence has a record in NALD, so it shouldn't have been picked up.

We've checked if others are affected, and fortunately, only 5 licences have a populated date_deleted though they have a matching record in the NALD data.

This change adds a migration data fix to reset the date_deleted to null for the 5 affected licences.

https://eaflood.atlassian.net/browse/WATER-4627

We had a report that an external user was unable to link **MD/054/0009/047** to their account (returns the error "We cannot find these licence numbers"). We were able to reproduce the issue in pre-prod and track down the cause.

When you add a licence number the legacy code is running this query to pull it from `crm.document_headers`.

```sql
SELECT
  "crm"."document_header" .*
FROM
  "crm"."document_header"
WHERE
  "crm"."document_header"."system_external_id" = $1
  AND "crm"."document_header"."company_entity_id" IS NULL
  AND "crm"."document_header"."metadata"->'IsCurrent' != $2
  AND "crm"."document_header"."date_deleted" IS NULL
  AND "crm"."document_header"."metadata"->>'IsCurrent' != $3
ORDER BY
  "crm"."document_header"."system_external_id" ASC
LIMIT $4
-- [ 'MD/054/0009/047', 'false', 'false', 300 ]
```

On 2023-10-03 the [water-abstraction-import](https://github.com/DEFRA/water-abstraction-import) marked the `crm.document_header` record as deleted. This is a step in the import process where it checks for deleted 'document'.

```javascript
// src/lib/connectors/import.js
const deleteCrmV1DocumentsQuery = `
  update crm.document_header
  set date_deleted = now()
  where system_external_id not in (
    select l."LIC_NO"
    from import."NALD_ABS_LICENCES" l
  )
  and date_deleted is null
  and regime_entity_id = '0434dc31-a34e-7158-5775-4694af7a60cf';
`
```

The knowledge of why the query didn't see this licence in the NALD data on that date is now lost to the universe! But we can clearly see the licence has a record in NALD so shouldn't have been picked up.

We've checked if others are affected, and fortunately, only 5 licences have a populated `date_deleted` though they have a matching record in the NALD data.

This change adds a migration data fix to reset the `date_deleted` to `null` for the 5 affected licences.
@Cruikshanks Cruikshanks added the bug Something isn't working label Aug 16, 2024
@Cruikshanks Cruikshanks self-assigned this Aug 16, 2024
Copy link
Contributor

@Jozzey Jozzey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Cruikshanks Cruikshanks merged commit b7d37a3 into main Aug 16, 2024
4 checks passed
@Cruikshanks Cruikshanks deleted the data-fix-for-deleted-documents branch August 16, 2024 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants