SD 272: decs integration #110

joehod · 2021-03-02T15:51:06Z

Relevant articles for setting up localstack:

https://medium.com/@FloSloot/your-own-local-copy-of-aws-w-node-js-6d98a10533a8

Specific commands are in the readme. Localstack now uses port 4566 which is not the case in the article.

joehod · 2021-04-12T09:23:54Z

What

For the integration with the DECS case working service
Converts the complaints data output from the form to fit the schema required by the DECS case working system
Validates the formatted data against the schema
Sends the json payload to the DECS SQS queue
Has functionality to turn off and on the send to sql queue and email caseworker functionality based on business need

Code changes summary

Added a new hot behaviour responsible for formatting, validating and sending the complaints data
Data formatting handled by ES6 classes which inherit from a base Complaints class
Payload sent by sql-producer library

Why

This functionality will mean the complaints form data can be added directly to the case working system from the SQS queue rather than added manually by a caseworker as it is currently.

…k queue Add behaviour for SQS integration using sqs-producer Add instructions and queue url to config Add instruction for reading what is on the queue

* Made initial functions to fit submitting application complaint to schema * Assign agent phone to agent details * Add formatting for delay complaints * Move complaintDetails to Parent class constructor and add individual complaint details to child constructor * Add format classes for somethingElse and Existing complaint * Finish formatting for StaffBehaviour * Use hof behaviour functions in behaviour

* Set up tests for AddToCasework * Add tests for complaint base class * Add tests for utils * Add tests for formatComplaintData * Add test for json-converters * Add test for validator throwing errror when undefined value passed in

* SD-272: add custom error page for form failing to submit to queue * SD-272: add max length constraint on all text fields * Add sqs queue config to config file * Increase test timeout * Remove integration test folder * Update node version and fix emailCasweworker config setting * update complaint details max length

sulthan-ahmed · 2021-04-12T15:30:32Z

README.md

@@ -19,6 +19,36 @@ Run the services locally with Docker Compose
 $ docker-compose up
 ```

+### Set up AWS SQS queue locally
+
+Run AWS services locally using localstack.


Would you be able to explain the technical architecture before hand, e.g. the data gets sent to a AWS Queue in DECs service.

Okay yeah I can do that. Max and I were gonna make some changes to the README anyway so we can add more about that.

thanks 👍

@MaxOSullivan

I don't think you'll have time for this, so maybe just add this as part of a tech debt ticket for the decs intergration

apps/ukvi-complaints/fields/index.js

apps/ukvi-complaints/lib/json-converters/brp.js

robertdeniszczyc2

Left a few comments - also please note I haven't properly reviewed the test files yet as I expect they will change based on the suggested changes to the other files.

Any questions give me a shout

Cheers,
Rob

Dockerfile

apps/ukvi-complaints/behaviours/send-to-sqs.js

apps/ukvi-complaints/index.js

apps/ukvi-complaints/lib/utils.js

robertdeniszczyc2 · 2021-04-13T14:46:44Z

apps/ukvi-complaints/translations/src/en/pages.json

+  },
+  "error": {
+    "header": "Sorry, your form has not been sent",
+    "text": "Please try again using the online form"


Has the copy being approved/reviewed?

This is the wording that was used by the designers on the FBIS service so we also used it here @robertdeniszczyc2

package.json

apps/ukvi-complaints/lib/json-converters/delays.js

apps/ukvi-complaints/index.js

…t class and single list of enums

changed preprod namespace to new preprod namespace

* Updated eslint to version 7.22.0 * Updated .eslintrc file * Fixed the failing eslint errors.

…ng-and-fix-errors SD-720: Fixed eslint errors and upgraded eslint

sulthan-ahmed · 2021-04-16T12:52:19Z

do QAT need to look through your PR?

sulthan-ahmed · 2021-04-16T12:59:20Z

README.md

+sqs create-queue \
+--queue-name local-queue \
+--endpoint-url http://localhost:4566 \
+--region eu-west-2 \


Suggested change

--region eu-west-2 \

--region eu-west-2

I had to remove the \ to get this to run

you could make it easier and just write aws sqs create-queue --queue-name local-queue --endpoint-url http://localhost:4566 --region eu-west-2

sulthan-ahmed · 2021-04-16T13:31:23Z

package.json

@@ -1,10 +1,15 @@
 {
  "license": "MIT",
+  "config": {
+    "reporter": "--reporter mocha-multi-reporters --reporter-options configFile=test/reporter-config"
+  },
  "scripts": {
    "start": "node app.js",
    "start:dev": "hof-build watch --env",


I know you guys are not responsible for this in the package.json file but I require a .env file but that's not mentioned in the README. Also it doesn't say which env variables I need

@MaxOSullivan Reckon you could add something about this when you do the readme?

add it as tech debt ticket :)

readme

sulthan-ahmed · 2021-04-16T23:05:46Z

Hi @joehod I hope you don't mind, I created an example of your work using mostly pure functions rather than classes.

I just took one particular part.

It's less code, and it doesn't have this which is one of the main issues for bugs in JS, no inheritance again another bug bearer in JS. Also you prevent mutated data. It's also easier test because you don't worry about classes and external dependencies.
Hopefully, it's also more readable #117

I think this could be condensed even more. You could just call getComplaints once rather than within each function/class and then add to it where relevant within your switch of if statement. I'm not sure without drilling down into it more

Anyway, I'd be interested in your thoughts

* Set up tests for AddToCasework * Add tests for complaint base class * Add tests for utils * Add tests for formatComplaintData * Add test for json-converters * Add test for validator throwing errror when undefined value passed in

* SD-272: add custom error page for form failing to submit to queue * SD-272: add max length constraint on all text fields * Add sqs queue config to config file * Increase test timeout * Remove integration test folder * Update node version and fix emailCasweworker config setting * update complaint details max length

…t class and single list of enums

…k queue Add behaviour for SQS integration using sqs-producer Add instructions and queue url to config Add instruction for reading what is on the queue

* Made initial functions to fit submitting application complaint to schema * Assign agent phone to agent details * Add formatting for delay complaints * Move complaintDetails to Parent class constructor and add individual complaint details to child constructor * Add format classes for somethingElse and Existing complaint * Finish formatting for StaffBehaviour * Use hof behaviour functions in behaviour

* Set up tests for AddToCasework * Add tests for complaint base class * Add tests for utils * Add tests for formatComplaintData * Add test for json-converters * Add test for validator throwing errror when undefined value passed in

* SD-272: add custom error page for form failing to submit to queue * SD-272: add max length constraint on all text fields * Add sqs queue config to config file * Increase test timeout * Remove integration test folder * Update node version and fix emailCasweworker config setting * update complaint details max length

…t class and single list of enums

sulthan-ahmed · 2021-04-20T16:29:19Z

test/reporter-config/unit.json

+      "reportDir": "test/executions/mochawesome",
+      "reportFilename": "unit"
+  }
+}


sulthan-ahmed · 2021-04-20T16:40:54Z

test/unit/lib/behaviours/send-to-sqs.spec.js

+
+  describe('saveValues', () => {
+    describe('config.writeToCasework', () => {
+      it('If false sendToQueue should not be called', () => {


I usually prefer tests written in prose

e.g. it shouldn't send anything to the queue if it in the configuration it has been set to false

sulthan-ahmed · 2021-04-20T16:45:29Z

test/unit/lib/behaviours/send-to-sqs.spec.js

+
+    });
+
+    describe('If valid complaint data', () => {


As before, I prefer tests written in prose. It reduces the cognitive overload

so I would rewrite to If the complaint data is valid

sulthan-ahmed · 2021-04-20T16:46:31Z

test/unit/lib/behaviours/send-to-sqs.spec.js

+    });
+
+    describe('If valid complaint data', () => {
+      it('formatComplaintData should have been called once with sessionModel attributes', () => {


as above it should call the method to format the data appropriate i.e. the formatComplaintData

sulthan-ahmed · 2021-04-20T16:47:14Z

test/unit/lib/behaviours/send-to-sqs.spec.js

+        expect(formatComplaintDataStub).to.have.been.calledOnceWith(req.sessionModel.attributes);
+      });
+
+      it('sendToQueue should be called once with complaint data', () => {


as above, it should send the data to the queue

sulthan-ahmed

Really great work @joehod man. I know this was a nightmare

I think the main code looks good. Well done.

I've not fully reviewed the tests. I think as part of tech debt it would be good to reword some of these tests to more prose but not important for now.
Top job

joehod changed the title ~~WIP: Sd 272 decs integration~~ SD 272: decs integration Apr 12, 2021

joehod assigned sulthan-ahmed Apr 12, 2021

Joseph Coker and others added 4 commits April 12, 2021 15:46

Add sql-producer behaviour and instructions for setting up local stac…

c1812d2

…k queue Add behaviour for SQS integration using sqs-producer Add instructions and queue url to config Add instruction for reading what is on the queue

Add tests checking complaint data formatting is correct

7fa9ecf

* Set up tests for AddToCasework * Add tests for complaint base class * Add tests for utils * Add tests for formatComplaintData * Add test for json-converters * Add test for validator throwing errror when undefined value passed in

joehod force-pushed the SD-272-DECS-integration branch from 2a53920 to dc59eb4 Compare April 12, 2021 14:58

sulthan-ahmed reviewed Apr 12, 2021

View reviewed changes

apps/ukvi-complaints/fields/index.js Show resolved Hide resolved

sulthan-ahmed reviewed Apr 13, 2021

View reviewed changes

apps/ukvi-complaints/lib/json-converters/brp.js Outdated Show resolved Hide resolved

Update eslint version and fix linting errors

4d753af

robertdeniszczyc2 requested changes Apr 13, 2021

View reviewed changes

sulthan-ahmed reviewed Apr 14, 2021

View reviewed changes

apps/ukvi-complaints/lib/json-converters/delays.js Outdated Show resolved Hide resolved

sulthan-ahmed reviewed Apr 14, 2021

View reviewed changes

apps/ukvi-complaints/index.js Show resolved Hide resolved

Joseph Coker and others added 8 commits April 14, 2021 16:45

Add function to convert enums across complaint types to base complain…

8bb2634

…t class and single list of enums

inject validator as argument to validAgainstSchema, add validatorFactory

bec1e53

Add logging of send to queue response

147c0f9

changed preprod namespace to new preprod namespace

adb6736

Merge pull request #113 from UKHomeOffice/SD-710-new-preprod-env

ab0a748

changed preprod namespace to new preprod namespace

SD-720: Fixed eslint errors and upgraded eslint

7e94623

* Updated eslint to version 7.22.0 * Updated .eslintrc file * Fixed the failing eslint errors.

Merge pull request #114 from UKHomeOffice/feature/SD-720-Update-Linti…

0f4f072

…ng-and-fix-errors SD-720: Fixed eslint errors and upgraded eslint

Add tests for send-to-sqs

ceac847

Add meaningful error messages to util functions

56f16ff

sulthan-ahmed reviewed Apr 16, 2021

View reviewed changes

James Hovell and others added 2 commits April 16, 2021 16:56

readme

bd50ba9

Merge pull request #115 from UKHomeOffice/SD-710-new-preprod-env

9332cd7

readme

Joseph Coker and others added 23 commits April 20, 2021 12:43

Add tests checking complaint data formatting is correct

7bcaeb2

* Set up tests for AddToCasework * Add tests for complaint base class * Add tests for utils * Add tests for formatComplaintData * Add test for json-converters * Add test for validator throwing errror when undefined value passed in

Update eslint version and fix linting errors

6b20185

Add function to convert enums across complaint types to base complain…

3f33545

…t class and single list of enums

inject validator as argument to validAgainstSchema, add validatorFactory

fa6c93a

Add logging of send to queue response

97ca852

Add tests for send-to-sqs

56eef15

Add meaningful error messages to util functions

5ceadf3

rebase from master

fd2b544

Add sql-producer behaviour and instructions for setting up local stac…

86c159b

…k queue Add behaviour for SQS integration using sqs-producer Add instructions and queue url to config Add instruction for reading what is on the queue

Add tests checking complaint data formatting is correct

4bc627f

* Set up tests for AddToCasework * Add tests for complaint base class * Add tests for utils * Add tests for formatComplaintData * Add test for json-converters * Add test for validator throwing errror when undefined value passed in

Update eslint version and fix linting errors

5738272

Add function to convert enums across complaint types to base complain…

502e0c9

…t class and single list of enums

inject validator as argument to validAgainstSchema, add validatorFactory

d9d2f1c

Add logging of send to queue response

5b492db

Add tests for send-to-sqs

fd0d8f0

Add meaningful error messages to util functions

5014a42

Refactor to json converters to functional approach

ec748b4

Fix merge conflicts with remote

fdfbf89

Fix sendToQueue config option and add test

4445851

Add fallback stub email to config

f8bb068

sulthan-ahmed reviewed Apr 20, 2021

View reviewed changes

test/reporter-config/unit.json

"reportDir": "test/executions/mochawesome",

"reportFilename": "unit"

}

}

Copy link

Contributor

sulthan-ahmed Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new line?

sulthan-ahmed reviewed Apr 20, 2021

View reviewed changes

sulthan-ahmed approved these changes Apr 20, 2021

View reviewed changes

joehod merged commit ff7be9d into master Apr 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SD 272: decs integration #110

SD 272: decs integration #110

joehod commented Mar 2, 2021

joehod commented Apr 12, 2021

sulthan-ahmed Apr 12, 2021

joehod Apr 13, 2021

sulthan-ahmed Apr 14, 2021

joehod Apr 15, 2021

sulthan-ahmed Apr 20, 2021

robertdeniszczyc2 left a comment

robertdeniszczyc2 Apr 13, 2021

MaxOSullivan Apr 15, 2021

sulthan-ahmed commented Apr 16, 2021

sulthan-ahmed Apr 16, 2021

sulthan-ahmed Apr 16, 2021

sulthan-ahmed Apr 16, 2021

joehod Apr 16, 2021

sulthan-ahmed Apr 20, 2021

sulthan-ahmed commented Apr 16, 2021

sulthan-ahmed Apr 20, 2021

sulthan-ahmed Apr 20, 2021 •

edited

Loading

sulthan-ahmed Apr 20, 2021 •

edited

Loading

sulthan-ahmed Apr 20, 2021 •

edited

Loading

sulthan-ahmed Apr 20, 2021

sulthan-ahmed left a comment

SD 272: decs integration #110

SD 272: decs integration #110

Conversation

joehod commented Mar 2, 2021

joehod commented Apr 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertdeniszczyc2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sulthan-ahmed commented Apr 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sulthan-ahmed commented Apr 16, 2021

Choose a reason for hiding this comment

sulthan-ahmed Apr 20, 2021 • edited Loading

Choose a reason for hiding this comment

sulthan-ahmed Apr 20, 2021 • edited Loading

Choose a reason for hiding this comment

sulthan-ahmed Apr 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sulthan-ahmed left a comment

Choose a reason for hiding this comment

sulthan-ahmed Apr 20, 2021 •

edited

Loading

sulthan-ahmed Apr 20, 2021 •

edited

Loading

sulthan-ahmed Apr 20, 2021 •

edited

Loading