Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GFR: Make scale for large record groups/collections #11

Open
DominicBM opened this issue Dec 23, 2016 · 1 comment
Open

GFR: Make scale for large record groups/collections #11

DominicBM opened this issue Dec 23, 2016 · 1 comment

Comments

@DominicBM
Copy link
Contributor

The current approach currently doesn't scale for large record groups/collections. Currently, we first query the internal API for all the series in an RG/Collection, and then we make an array of all the series' NAIDs to pass as a query to the public API. For large record groups/collections, we can't pass hundreds of NAIDs in a URL parameter, so this errors.

We have to do it this way because (1) the public API fields for parentRecordGroup and parentCollection are not searchable (they are not included on the field white list), but (2) the internal API doesn't give us all the data we need, since it is designed for brief results. So we use the internal API to get the list of NAIDs, and then query them explicitly instead of querying by parent description.

In the future, with the white list issue resolved, we'd hope to just search "https://catalog.archives.gov/api/v1?rows=10000&description.series.parentRecordGroup.recordGroupNumber=" to get all the necessary metadata fields for all the series in a record group at once.

@DominicBM DominicBM added the bug label Dec 23, 2016
@DominicBM DominicBM added this to the Guide to Federal Records milestone Dec 23, 2016
@DominicBM
Copy link
Contributor Author

Right now, this is mitigated by limiting series results to the first 750, which omits some RGs (about 50).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant