This release of Watchman has removed Company/Customer models and Watches. They've been deprecated for a while and do not perform as users expect. Stay tuned for a future Moov OSS project integrating with Watchman for sanctions screening.
IMPROVEMENTS
- feat: return matchedName in non-OFAC results
- search: apply more edge case logic to decrease bad scoring
- search: return matchedName for OFAC SDNs, Alts, and DPL records
- test: remove duplicate (and skipped) UK/EU CSL tests
This release of Watchman lowers most match percentages by comparing names better.
IMPROVEMENTS
- fix: close xml encoder
- fix: panic cleanup from newer linter rules
- cmd/server: only check adjacent terms for local jaro max score
- cmd/server: read ADJACENT_SIMILARITY_POSITIONS env var
- cmd/server: weight term score by length similarity
BUILD
- build: bump numerous javascript dependencies
- build: update to Go 1.21
- build: update to node 20
- build: update Debian, Fedora, node, and Go base images
IMPROVEMENTS
- fix: keep numbers during stopwords step
- fix: stop setting level=error for info logs
BUILD
- bump golang.org/x/crypto to v0.6.0
- bump golang.org/x/net from 0.6.0 to 0.7.0
- build(deps): bump activesupport from 6.1.7.2 to 7.0.4.3 in /docs
- build(deps): bump webpack from 5.75.0 to 5.76.1 in /webui
IMPROVEMENTS
- fix: filter SDNs in async search rather than truncate
BUILD
- build: update Go dependencies
- docs: bundle update
- webui: npm audit fix
ADDITIONS
- search: Add the EU Consolidated Screening List
- search: Add the UK Consolidated Screening List
IMPROVEMENTS
- feat: log status after download
- fix: guard around race condition in pkg/download
- fix: cap match percentage
BUILD
- build: upgrade golang to 1.20
- build: try using hashicorp-forge/go-test-split-action to speedup tests
- build(deps): bump activesupport from 6.0.3.4 to 6.0.6.1 in /docs
- build(deps): bump json5 from 1.0.1 to 1.0.2 in /webui
- build(deps): bump loader-utils from 2.0.0 to 2.0.4 in /webui
- build(deps): bump nokogiri from 1.13.6 to 1.13.9 in /docs
IMPROVEMENTS
- api,client: add 'match' to CSL results
- webui: update country list
BUILD
- build: require go1.19.1 in CI/CD
- build: upgrade golang.org/x/text to v0.3.8
Watchman v0.23.0 adds the US Consolidated Screening List results to search queries. A new /search/us-csl endpoint has also been added to only search the CSL. See the API docs for full details.
Lists added in this release:
- Capta List (CAP)
- Foreign Sanctions Evaders (FSE)
- ITAR Debarred (DTC)
- Military End User (MEU) List
- Non-SDN Chinese Military-Industrial Complex Companies List (CMIC)
- Non-SDN Menu-Based Sanctions List (NS-MBS List)
- Nonproliferation Sanctions (ISN)
- Palestinian Legislative Council List (PLC)
- Unverified List (UVL)
IMPROVEMENTS
- download: only return files which are found or downloaded, not entire directory
- feat: add GET /search/us-csl endpoint
- fix: refer to
LOG_LEVELenv var for when to log about migrations
BUILD
- build: upgrade to Go 1.19
- build: replace deprecated ioutil functions
- build(deps): bump terser from 5.13.1 to 5.14.2 in /webui
- chore(deps): update dependency tzinfo to v1.2.10 [security]
- fix(deps): update module github.com/moov-io/base to v0.34.1
Watchman v0.22.0 has started to use Go 1.18 and its support for generic programming has helped us greatly clean up our code. We've also refreshed our documentation website.
ADDITIONS
- cmd/server: add Military End User (MEU) results to search response
- cmd/server: add an optional webhook to be notified of data downloads
- cmd/server: read
WEBHOOK_MAX_WORKERSto configure max webhook processors - csl: support extracting Military End User (MEU) List
- feat: allow overrides to jaro-winkler parameters
IMPROVEMENTS
- cmd/server: dig into CSL entity types to precompute AlternateNames
- cmd/server: generic function for CSL entity precompute step
- docs: show sdnType examples in query params
BUILD
- build: update to Fedora 37 for OpenShift, node and react
- build: update to Go 1.18, Node 16 and Debian versions
- cmd/server: cleanup t.Error calls for better visibility
- cmd/server: switch to an outside mysql container, remove dockertest
- examples/webhook: update docker image to Go 1.18
BUILD
- build(deps): bump lodash from 4.17.19 to 4.17.21 in /webui
- build(deps): bump minimist from 1.2.5 to 1.2.6 in /webui
- fix(deps): update module github.com/moov-io/base to v0.28.1
- fix(deps): update react monorepo to v18
IMPROVEMENTS
- all: update logging to moov-io/base using better structures and key/value pairs
BUILD
- Improve code coverage and fix errors form gosec
- Upgrade to Go 1.17
- Upgrade github.com/go-kit/kit to v0.12.0
- Upgrade github.com/mattn/go-sqlite3 to v1.14.11
- Upgrade github.com/moov-io/base to v0.27.5
- Upgrade github.com/ory/dockertest/v3 to v3.8.1
- Upgrade github.com/pariz/gountries to v0.1.5
- Upgrade github.com/prometheus/client_golang to v1.12.1
IMPROVEMENTS
- fix: skip malformed rows in input files (#379)
IMPROVEMENTS
- build: enable gosec and coverage linter
- fix: DPL records could have invalid characters, add length guards
- fix: ignore misspell checks on valid test data
- fix: ofac SDN comments file has invalid characters / column counts
Release v0.21.1 applies bugfix to pre-computation steps to SDN alternate names.
BUG FIXES
- search: bugfix for pipeline steps to SDN alt names (#376)
Release v0.21.0 applies pre-computation steps to SDN alternate names which was previously missing. This impacts match performance for only those results.
BUG FIXES
- search: apply pipline steps to SDN alt names (#375)
Watchman v0.20.0 improves scoring by reducing false positives. There are two changes implemented in this release to achive better scoring. First is to keep a more accurate list of scores for our query, which means longer queries will return more accurate results than before. Second is that we will always score at least two words, which helps to reduce false positives from "first last" queries.
IMPROVEMENTS
- docs/intro: mention jaro-winkler is used and why
- search: properly keep top N word scores, match at least two words
BUG FIXES
- ofac: skip error with shared initial directory
BUILD
- build: use debian stable's slim image
- fix: Dockerfile to reduce vulnerabilities
- meta: fixup from gosec linters
IMPROVEMENTS
- cmd/server: add
matchscore to OpenAPI search entities - cmd/server: document CSL separation in struct
BUILD
- chore(deps): update dependency @material-ui/core to v4.12.2
BUG FIXES
- cmd/server: nil check on setting updated records
- download: fixup reading of initial dir rather than downloading
- dpl: find file rather than assume its index
IMPROVEMENTS
- cmd/server: misc improvements from linters
BUILD
- build(deps): bump addressable from 2.7.0 to 2.8.0 in /docs
- chore(deps): update dependency @material-ui/core to v4.12.1
- cmd/server: add benchmark for ?minMatch=0.50
- cmd/server: add benchmarks for various SDN search queries
- fix(deps): update react monorepo to v17
- fix: Dockerfile to reduce vulnerabilities
- fix: upgrade react-scripts from 4.0.1 to 4.0.3
- test: add static set of files for benchmarks
This release improves search results by filtering as part of scoring instead of after the fact. Users should see more results returned for queries now.
ADDITIONS
- cmd/server: add
?minMatch=0.95query param for searching
IMPROVEMENTS
- cmd/server: remove unneeded X-User-ID HTTP headers
- cmd/server: filter as part of searching, not after the fact
BUILD
- build: split CI into multiple jobs for parallel runs
BUG FIXES
- cmd/server: nil check on webhook response body prior to Close
BUILD
- build(deps): bump nokogiri from 1.11.1 to 1.11.5 in /docs
BUG FIXES
- csl: don't panic if list comes back invalid
BUILD
- build(deps): bump rexml from 3.2.4 to 3.2.5 in /docs
- chore(deps): update dependency @material-ui/core to v4.11.4
IMPROVEMENTS
- cmd/server: concurrently search instead of serial iterations
BUILD
- build: upgrade github.com/gogo/protobuf to v1.3.2
IMPROVEMENTS
- chore(deps): update golang docker tag to v1.16
ADDITIONS
- search: introduce
EXACT_MATCH_FAVORITISMconfig to optionally boost match rates
IMPROVEMENTS
- docs: lots of fixups and clarifications
BUG FIXES
- csl: fix parsing with newer CSV format
BUILD
- fix: upgrade react-scripts from 4.0.0 to 4.0.1
ADDITIONS
- api,client: add SdnType and SsiType models
- docs: include a page for prometheus metrics
- server: Add
last_data_refresh_failure{source="SDNs"}Prometheus metric
BUG FIXES
- build: fixup OpenShift image, rely on glibc
BUILD
- chore(deps): update mui monorepo to v4.11.2
- fix(deps): pin dependency react-scripts to 4.0.0
BUG FIXES
- search: filter countries rather than rank
- Previously we would only include the top N (
?limitquery param) results by country which could exclude valid search results from the final returend list. Now we filter the list and rank according to all data records matching the specified country.
- Previously we would only include the top N (
- search: OR NameAndAddress results
- Previous we would use AND semantics for returning "name and address" results even though they populated two different JSON arrays. Now we return the highest N
SDNsandAddresseswhen both query params are specified.
- Previous we would use AND semantics for returning "name and address" results even though they populated two different JSON arrays. Now we return the highest N
BUG FIXES
- webui: fix Country "All" value so it's not passed through to query params
IMPROVEMENTS
- batchsearch: include final success message
- webui: fix warnings after react upgrade
- webui: include a links section in the header
BUILD
- fix(deps): update react monorepo to v16.14.0
- webui: run "npm audit fix"
BUG FIXES
- pkg/ofac: skip sdn_comments.csv lines if they contain the incorrect number of fields
IMPROVEMENTS
- docs/search: include supported combinations of query params
BUILD
- admin,client: update codegen, reference base Error model
BREAKING CHANGES
Watchman has upgraded the implementation of the Jaro-Winkler string comparison algorithm to fix bugs and match the original paper's reference implementation. After upgrading Watchman you will notice higher match percentages for most comparisons. See pull request #282 for the change.
BUG FIXES
- search: skip calling webhooks if we don't render a body
- search: test body creation in async methods
BUILD
- chore(deps): update golang docker tag to v1.15
- docs: re-order "getting started" to suggest docker image
BUG FIXES
- csl: fix reader to handle files with unique IDs
- search: reference appropriate SDN for webhook body
- webhooks: handle nil Response without a panic
BUILD
- webui: run 'npm audit fix'
BUILD
- build: add OpenShift docker image
- build: upgrade golang.org/x/text to v0.3.3
- chore(deps): update dependency @material-ui/core to v4.11.0
ADDITIONS
- build: add moov/watchman:static image for better test instances
- cmd/server: add version handler to admin HTTP server
- cmd/server: record download counts per entity type in
last_data_refresh_countPrometheus metric
IMPROVEMENTS
- admin,client: fix package names, upgrade openapi-generator to 4.2.3
- all: misc test cleanup from staticcheck
- api,client: use shared Error model
- api,client: use shorter API summaries
- build: run sonatype-nexus-community/nancy in CI
- build: setup windows CI and releases
- download: set custom User-Agent header
- pkg/csl: use endpoint to download which doesn't require an api key
BUG FIXES
- ofac: allow unescaped inner quotes in SDN comments records
BUILD
- build: upgrade to Go 1.14.x
- build: upgrade staticcheck to 2020.1.3
- build: don't 'make docker' twice
- build: update Copyright headers for 2020
- build: upgrade mysql client to 1.5
- build: don't wipe both generated dirs on 'make clean'
- chore(deps): update dependency @material-ui/core to v4.9.9
- chore(deps): update golang docker tag to v1.14
- chore(deps): update module moov-io/base to v0.11.0
- chore(deps): update module prometheus/client_golang to v1.4.0
- chore(deps): update mui monorepo to v4.9.1
IMPROVEMENTS
- ofac: allow unescaped inner quotes in SDN comments records
- ofac: add filename and more logging context for debugging refresh errors
IMPROVEMENTS
- pkg/csl: use endpoint to download which doesn't require an api key
This project has been renamed to Moov Watchman (from Moov OFAC) to better represent coverage of multiple entity lists covering trade sanctions and regional compliance laws.
The OpenAPI specification models have been renamed to include an Ofac prefix on their name and OFAC HTTP routes have an /ofac/ prefix.
BREAKING CHANGES
- pkg/ofac:
SDN.Program stringhas been renamed toSDN.Programs []stringand is populated with all programs an SDN was created from.
ADDITIONS
- cmd/server: Add Sectoral Sanctions Identifications List via Consolidated Screening List (CSL)
- cmd/server: Add Bureau of Industry and Security Entity List via Consolidated Screening List
- cmd/server: Remove stopwords from OFAC SDN entity names, DP's, SSI, and BIS Entities
- cmd/server: Remove company suffixes in indexed names (such as LLC, Co., LTDA.)
- cmd/server: add admin route to debug indexed SDN information
- Example:
curl http://localhost:9094/debug/sdn/16016
- Example:
IMPROVEMENTS
- cmd/server: concurrently search ?q=foo
- cmd/server: add KEEP_STOPWORDS env config
- pkg/ofac: clean up SDN programs when reading csv
- cmd/server: handle an SDN's program list as an array in search, fixup UI values endpoint
- cmd/server: add TopAltNames to searchByName
- cmd/server: clarify shutdown logging
- cmd/server: pipeline: trim spaces, split hyphenated words
BUG FIXES
- pkg/dpl: ignore first line of header information
- cmd/server: match remark ID's exactly in all cases
BUILD
- fix(deps): update react monorepo to v16.12.0
- chore(deps): update dependency @material-ui/core to v4.8.0
- build: upgrade openapi-generator to 4.2.2
- fix(deps): update dependency react-scripts to v3.3.0
BREAKING CHANGES
Addressin our OpenAPI spec and generated Go client was renamedEntiyAddressto provide a more specific naming when combined in Moov's larger OpenAPI specification.
ADDITIONS
- cmd/server: add histogram for match percentages
- cmd/ofaccheck: initial setup of a cli tool for batch searches
- cmd/server: AND name and address queries if params are provided
BUILD
- build: upgrade Go and webui dependencies
This release adds a web interface for OFAC (developed by Linden Lab) which allows for easier querying from desktop and mobile browsers. Also added are query params to apply additional filtering (exmaple: sdnType=individual) and we have improved match percentages to closer mirror the official OFAC search tool.
ADDITIONS
- cmd/server: add the web interface developed by Linden Lab
- cmd/server: accept additional query params to filter SDN search results
?sdnType=individualand?program=example
- cmd/server: add endpoint for applications to grab distinct sets of column values
GET /ui/values/sdnTypereturns["aircraft","individual","vessel"]
- cmd/server: add
/search?id=NNNendpoint for matching remark IDs - cmd/server: return the oldest refresh time for our data in search results
- cmd/server: add
-base-pathfor serving HTTP routes and web UI from non-root paths
IMPROVEMENTS
- api,client: specify x-request-id and x-user-id as optional HTTP headers
- cmd/ofactest: set x-request-id and x-user-id HTTP headers if CLI flags are set
- cmd/server: use a non-nil logger in search HTTP route tests
- cmd/server: adjust jaro weighting to maximize total weight in strong single word matches
BUG FIXES
- cmd/server: fix spelling of jaroWinkler
- cmd/server: never allow jaroWinkler to return NaN
- cmd/server: match treasury.gov match percentages
BUILD
- cmd/server: update github.com/moov-io/base to v0.10.0
- build: download CI tools rather than install
- build: remove mysql setup from docker-compose.yml
This release contains improvements to OFAC's match percentages to tone down false positives. We do this by adjusting the match percentage according to the query and SDN lenth ratio -- if the two strings are different lengths (after normalization) they cannot be equal. We are looking for feedback to further improve the matching code.
BREAKING CHANGES
- api,client: We've renamed all fields like
*Idto*IDwhich is consistent with Go's style.
BUG FIXES
- attempt retries when downloading files
- download: return after successful download
- cmd/server: search: fix match percents after jaroWinkler change
- cmd/server: when reordering names handle multiple first or last names
ADDITIONS
- internal/database: log which provider we're using
- cmd/server: bind HTTP server with TLS if HTTPS_* variables are defined
- cmd/server: disable perioidic refresh via OFAC_DATA_REFRESH=off
- download: check for initial files on first refresh
IMPROVEMENTS
- build: upgrade openapi-generator to 4.1.0
- all: skip more tests on -short
BREAKING CHANGES
- The admin endpoint
:9092/ofac/refreshwas renamed to:9092/data/refresh
ADDITIONS
- cmd/server: Include the BIS Denied Person's List data in search endpoints
- cmd/server: Support MySQL as a storage layer via
DATABASE_TYPE=mysql(See: #100)
IMPROVEMENTS
- build: push moov/ofac:latest on 'make release-push'
- docs: update docs.moov.io links after design refresh
- build: update dependencies
- cmd/server: rename manual refresh endpoint to /data/refresh
BREAKING CHANGES
- api: Rename 'searchSDNs' to 'search' to reflect capability of other search capabilities
- api: Add
*OFAC*on Company and Customer OpenAPI operations (and thus generated clients)
ADDITIONS
- api: Added Company routes and generated client code
- cmd/server: async: send webhooks on company/customer name watches (with match %)
- docs: add "A Framework for OFAC Compliance Commitments" from the US Treasury
- cmd/server: search: handle ?country and average weight with ?address
- cmd/server: handle city, state, providence, and zip in address search
BUG FIXES
- cmd/server: return database/sql Rows.Err
- api: add missing 'notes' field on UpdateCompanyStatus and UpdateCustomerStatus
- docs: describe moov's primary usage of OFAC (for paygate)
ADDITIONS
- cmd/server: support GET /search?q=... for searching SDN names, alt names, and address field
IMPROVEMENTS
- cmd/server: set CORS headers in ping route
- cmd/server: UTF-8 normalize names during pre-computation
- cmd/server: re-order SDN names where surname precedes "first name"
- cmd/server: support -log.format=json
BUG FIXES
- cmd/server: add missing database/sql Rows.Close()
BUILD
- Update to Go 1.12
IMPROVEMENTS
- Setup automated releases of binaries and Docker image
ADDITIONS
- Added
moov/ofactestdocker image
BUG FIXES
cmd/ofactest,client: fix Watch create JSON and cleanup ofactest watches
ADDITIONS
- Add Company routes, persistence, client code, and watches
cmd/ofactest: add -local and -webhook flagscmd/ofactest: read optional search query parameter
IMPROVEMENTS
- Ignore whitespace in string similarity
example: Read Company or Customer JSON from webhook
BUG FIXES
- Fix capitalization of various JSON properties
ADDITIONS
- Added
"match": 0.91as a percent of SDN name match to parameter. - Support setting a customer to
unsafeorexceptionstatus.
CHANGES
- Require
authTokenwhen creating a Watch (webhook notification)
ADDITIONS
- Added
last_ofac_data_refresh_successPrometheus metric cmd/ofactest: for testing OFAC HTTP endpoints
IMPROVEMENTS
- Stop expecting
X-User-Idheader to be present (and non-empty)
IMPROVEMENTS
- Implement fuzzy search with Levenshtein for word to word comparisons
- Periodically refresh data (according to
OFAC_DATA_REFRESH, default:12h) - Write OpenAPI v3 specification and generate a Go client
- SQLite persistence for downloads, watches, and webhook results
- Add
GET /downloads?limit=Nfor latest N data download metadata
BUG FIXES
- Remove OFAC null characters (
-0-) from data (and HTTP api)
- Initial release