Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cumulus-std #3638

Draft
wants to merge 72 commits into
base: feature/es-phase-2
Choose a base branch
from
Draft

cumulus-std #3638

wants to merge 72 commits into from

Conversation

jennyhliu
Copy link
Contributor

Summary: Summary of changes

Addresses CUMULUS-XX: Develop amazing new feature

Changes

  • Detailed list or prose of changes
  • ...

PR Checklist

  • Update CHANGELOG
  • Unit tests
  • Ad-hoc testing - Deploy changes and test manually
  • Integration tests

jennyhliu and others added 30 commits April 29, 2022 09:31
* Update xml2js 0.4.22->0.5 strict

* Address GHSA-776f-qx25-q3cc/update allow list

* Update CHANGELOG

* Update package pins to 0.5.0 for xmljs

Co-authored-by: Jonathan Kovarik <[email protected]>
…horization header (#3341)

* CUMULUS-3285: Updated isAuthBearTokenRequest  to handle non-Bearer authorization header

* add integration test
* backport PR

* finalizing docs and changing remaining groups

* 15.1.0 release

* doc updates

* PR feedback + testing changes

* Revert "doc updates"

This reverts commit 5037012.

* PR feedback

* reverting change

* Revert "update changelog"

This reverts commit ae4627c.

* reverting changes

* PR feedback

* removing EgressLambda from doc

* Update xml2js 0.4.22->0.5 strict (#3330) (#3339)

* Update xml2js 0.4.22->0.5 strict

* Address GHSA-776f-qx25-q3cc/update allow list

* Update CHANGELOG

* Update package pins to 0.5.0 for xmljs

Co-authored-by: Jonathan Kovarik <[email protected]>

* PR feedback

* PR feedback

* PR feedback

* fixing documentation linting

* PR feedback

* PR feedback

* PR feedback

* adding variables to tf-modules/workflow

* PR feedback

* PR feedback

* reverting previous change

* CUMULUS-3121/3120 v15.1.0 backport (#3346)

* backport PR

* finalizing docs and changing remaining groups

* PR feedback + testing changes

* PR feedback

* reverting change

* Revert "update changelog"

This reverts commit ae4627c.

* reverting changes

* PR feedback

* removing EgressLambda from doc

* Update xml2js 0.4.22->0.5 strict (#3330) (#3339)

* Update xml2js 0.4.22->0.5 strict

* Address GHSA-776f-qx25-q3cc/update allow list

* Update CHANGELOG

* Update package pins to 0.5.0 for xmljs

Co-authored-by: Jonathan Kovarik <[email protected]>

* PR feedback

* PR feedback

* PR feedback

* fixing documentation linting

* PR feedback

* PR feedback

* PR feedback

* adding variables to tf-modules/workflow

* PR feedback

* PR feedback

* reverting previous change

---------

Co-authored-by: jennyhliu <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>

* changing version number

* small fixes

* small fix

---------

Co-authored-by: jennyhliu <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>
* CUMULUS-3243:Updated granule delete logic to delete granule which is not in DynamoDB

* add unit tests
* CUMULUS-3223:Fix failed granule stuck in queued

* skip sqsQueueExists test

* update getGranuleTemporalInfo

* update test match schema

* update getGranuleTemporalInfo

* remove extra await

* remove skip sqsQueueExists

* update `@cumulus/cumulus-message-adapter-js` to `2.0.5`

* update sfEventSqsToDbRecords to return partial batch failure

* fix typo

* handle process error seperately, multiple message test

* update test to process multiple messages
…g will enter a retry loop if the Cumulus API returns a 404 (#3298)

* first commit

* changelog entry

* feedback

* small change

* PR feedback

* fixing CI issues

* fixing previous changes that were wrong

* fixing unit test

* fixing CI issues

* fixing units

* fixing CI failures

* Changelog

* fixing TS

* fixing ingestGranule test failure

* fixing clause

* added test for expectedStatusCodes

* PR feedback

* fixing tests

* reverting code block back to master

* fixing expectedStatusCodes

* reverting changes

* fixing tests

* PR feedback
#3401)

* Update integration test scripts to fail on test timeout

* Fixup

* Fixup

* Update script interpreter for test runs

* Fix script

* Fixup

* Fixup

* Update timeout pass/fail conditional
* version bump

* version bump and changelog

* update docs

* update docs version

* update website package
* Update packages to explicitly private

* 16.1.1 version bump and docs

* update changelog version

* update node runtime to current version
jennyhliu and others added 7 commits April 30, 2024 17:11
* Update core deps to v20.12.2

* Update unit test to rmove .sort

.sort is intermittently failing to order as expected, causing this
test to fail in local test at an alarming failure rate.    Updating
test to be more explicit/remove sort, however this needs to be
investigated prior to closing thie branch/PR/ticket

* Update kinesis test to node v20

JSON.parse now throws a different error for the merged test case.

* Update knex test to handle new subdependency throwing an Aggregate
error

* Revert "Update unit test to rmove .sort"

This reverts commit adbad84.

* Update got

* Revert "Update got"

This reverts commit a19f9d1.

* Upgrade ava/nock, fix ingest provider tests

* Reapply "Update got"

This reverts commit e652516.

* Update ingest module for node 20

This update has a couple of controversial changes in it:

Updating got to v14 means we're using a pure ESM module given sindre's
stance on not supporting common exports.  That's being done due to
incompatible changes in node streams requires at least got v12

Additionally there's a probable outstanding issue in got
sindresorhus/got#2341 related to node v20/fs
readstreams/nock and/or msw incompatibility (as well as a possible
open source contrib)

* Remove httpClient test mock/fix

Updating the existing apache server to return 200 on an existing
endpoint is far better than the prior commit approach in that it's a
valid/useful unit test as a result, with the technical/less tidy
downside of requiring the unit test stack to be working.

* Update local test stack configuration to restrict connection to localhost

* Update send-pan test/dependencies

* Update lambdas/async operation to deploy with node 20

* Update common to export a helper to import ESM got module

* Add docstring to importGot

* Remove unneeded imports

* Minor PR review fixes

* Update CHANGELOG

* Update CHANGELOG

* Reconfigure CI to not use unsafe perms modification

* Fix broken package.json

* Fix extran. dep issue

* Make error message test less specific

* Update test to not rely on aggregate internal errors

* Update packages/ingest/test/test-HttpProviderClient.js



* Move importGot -> importEsm, introduce static import definition

* Fix inadvertant test commit

* Update rules helpers to remove all null key values from rule JSON

* Update CHANGELOG

* Update helper test titles for consistency

* Minor refactor/rename

* Fix typing errors in original code, update PR accordingly

* Fix inadvertant move of rule validation

* Update packages/api/endpoints/rules.js



* Remove custom null removal method in favor of omitDeepBy

* Remove duplicative call to omitDeepBy

* Update packages/api/lib/rulesHelpers.js



---------

Co-authored-by: etcart <[email protected]>
* Update policy (#3649)

* Version up to 18.2.1

* Update function docs

---------

Co-authored-by: Nate Pauzenga <[email protected]>
* Update ECS Cluster EBS Volume Default Encryption Settings (CUMULUS-3547/NDCUM-1184) (#3580)

* Updated Encrypted=true for EBS volume(s) associated with instances in the ECS cluster

* Updated the ChangeLog based on CUMULUS-3547 changes

---------

Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Tim Clark <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>

* Jk/bugfix/cumulus 3587 pull in express boom (#3583)

* Initial commit - update deps, bring in expressboom

* Refactor to more modern JS standards

* Add api/lib expressBoom

* Fix lint error

* Update CHANGELOG

* Revert CVE whitelist

* Remove express-boom/revert inadvertant merge issue

* Proposed CI changes

* re-add audit whitelist as sub-dependencies will fail until next
release

* Merge pull request #3582 from nasa/pp/CUMULUS-3527-sync-granule-sftp-failure (#3652)

[CUMULUS-3527] Adds support for additional kex algorithms to sftp-client

Co-authored-by: Paul Pilone <[email protected]>

* CUMULUS-3673: Fixes Granules API URI encoding by escaping path components. (#3629)

* URI encodes granule API path components.

* Updates CL for CUMULUS-3673

* Fixes JSDoc linter warnings

* Removes JSDoc types from granules API handler

* Updates CnmWorkflowFromSqsSpec to not encode collection ID

* WIP: Updating specs to not encode collection ID

* WIP: Updates specs to not encode collection ID

* WIP: Fixes linter errors in spec files

* Removes unnecessary JSDoc blocks

* WIP: Removes last bits of int test encoding collection IDs

* Fix babel package vulnerability (#3622)

* Fix babel package vulnerability

* update package

* update .babelrc

* update .eslintrc.js

* fix error:ES Modules may not assign module.exports or exports.*

* Updates supertest to fix critical vulnerability

* Removes libxmljs due to unuse and audit failures

* Adds hoek to audit allow list

---------

Co-authored-by: Tim Clark <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Tim Clark <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>
Co-authored-by: jennyhliu <[email protected]>
* Update ECS Cluster EBS Volume Default Encryption Settings (CUMULUS-3547/NDCUM-1184) (#3580)

* Updated Encrypted=true for EBS volume(s) associated with instances in the ECS cluster

* Updated the ChangeLog based on CUMULUS-3547 changes

---------

Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Tim Clark <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>

* Jk/bugfix/cumulus 3587 pull in express boom (#3583)

* Initial commit - update deps, bring in expressboom

* Refactor to more modern JS standards

* Add api/lib expressBoom

* Fix lint error

* Update CHANGELOG

* Revert CVE whitelist

* Remove express-boom/revert inadvertant merge issue

* Proposed CI changes

* re-add audit whitelist as sub-dependencies will fail until next
release

* Merge pull request #3582 from nasa/pp/CUMULUS-3527-sync-granule-sftp-failure (#3652)

[CUMULUS-3527] Adds support for additional kex algorithms to sftp-client

Co-authored-by: Paul Pilone <[email protected]>

* CUMULUS-3673: Fixes Granules API URI encoding by escaping path components. (#3629)

* URI encodes granule API path components.

* Updates CL for CUMULUS-3673

* Fixes JSDoc linter warnings

* Removes JSDoc types from granules API handler

* Updates CnmWorkflowFromSqsSpec to not encode collection ID

* WIP: Updating specs to not encode collection ID

* WIP: Updates specs to not encode collection ID

* WIP: Fixes linter errors in spec files

* Removes unnecessary JSDoc blocks

* WIP: Removes last bits of int test encoding collection IDs

* Fix babel package vulnerability (#3622)

* Fix babel package vulnerability

* update package

* update .babelrc

* update .eslintrc.js

* fix error:ES Modules may not assign module.exports or exports.*

* Updates supertest to fix critical vulnerability

* CUMULUS-3951 - Deploy all SNS topics with encrypted storage  (#3646)

* Update SNS topics to be encrypted in terraform

* Update example package to encrypt test topics

* Update api package test fixtures to match deployed code/encryption options

* Update CHANGELOG

* Refactor SNS creation into aws-client helper method

* Update CHANGELOG

* Fix: add missing `await`

* Removes libxmljs due to unuse and audit failures

* Adds hoek to audit allow list

* Removes linter errors from original cherry-pick

---------

Co-authored-by: Tim Clark <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Tim Clark <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>
Co-authored-by: jennyhliu <[email protected]>
* Bumps package vesions to 18.2.2

* Updates CHANGELOG with release version

* Updates CONTRIBUTORS

* Updates docs for 18.2.2
@jennyhliu jennyhliu changed the base branch from release-16.1.x to release-18.2.x July 5, 2024 14:14
@jennyhliu jennyhliu changed the base branch from release-18.2.x to feature/es-phase-1 August 9, 2024 17:29
@jennyhliu jennyhliu changed the base branch from feature/es-phase-1 to release-18.3.x August 9, 2024 17:31
@jennyhliu jennyhliu changed the base branch from release-18.3.x to release-18.2.x August 9, 2024 17:32
@jennyhliu jennyhliu changed the base branch from release-18.2.x to release-19.0.x August 29, 2024 17:13
@jennyhliu jennyhliu changed the base branch from release-19.0.x to master August 29, 2024 17:13
@jennyhliu jennyhliu changed the base branch from master to release-18.2.x August 29, 2024 17:45
@jennyhliu jennyhliu changed the base branch from release-18.2.x to master August 29, 2024 18:20
@jennyhliu jennyhliu changed the base branch from master to feature/es-phase-2 September 9, 2024 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants