API "spec" for currently used grants.gov endpoint
Determining this was done by reverse engineering (e.g., experimentation).
The endpoint that we call is : https://www.grants.gov/grantsws/rest/opportunities/search/. See this line of code.
This endpoint is:
- Not documented
- Possibly not formally sanctioned
JSON request is formatted as follows:
{
"startRecordNum": 0,
"keyword": "",
"dateRange": "7",
"eligibilities": "11|20|12|13|21|22|23|02|01|05|04|06|00|25|99|07|08",
"oppNum": "",
"cfda": "",
"oppStatuses": "posted|forecasted",
"sortBy": "openDate|desc"
}
The dateRange does have an effect. Increasing it increases the number of grants that are returned. It's unclear, however, if this range applies to grant open date, grant close date, or grant last modified date.
Changing the keywords and eligibilities (visible in the UI) doesn't change the keyword or eligibilities in the JSON query body.
oppNum, cfda, oppStatuses and sortBy are hardcoded. See this line of code.
It's unclear why startRecordNum increments by 10. This causes the same grant to come back in the result for multiple queries, so the 'total results' will add up to more than the actual number of grants processed. See https://github.com/usdigitalresponse/usdr-gost/issues/636 for more info.
gost-app | 2022-11-22T22:34:11.967196463Z searching for @ 80
...
gost-app | 2022-11-22T22:34:34.286682316Z found 17 total results on grants.gov
...
gost-app | 2022-11-22T22:34:35.499129141Z "startRecordNum": 90,
...
gost-app | 2022-11-22T22:34:44.912763353Z found 7 total results on grants.gov
JSON fields returned:
The queries are sent by a series of BatchProcessor(s). See this line of code. They are delayed by the millisecond value in GRANTS_SCRAPER_DELAY (currently at 1000, e.g., 1 second).
There is an isRunning flag to prevent multiple series of BatchProcessors running simultaneously.
If any query hits a network error, no subsequent BatchProcessors are run. Since, however, there are scrapes run every hour (on the half-hour), grants that were missed because of the network error can be picked up on the next run. I have seen EAI_AGAIN and ECONNRESET errors on my machine, but I don't know if those happen in the render.com hosting service.