Discrepancy in Artwork Records Between API, and dump file?

[takeshuart]

Hello,
I am currently working with the Art Institute of Chicago's API and have encountered some issues regarding the retrieval of artwork records and the consistency of data in different sources.

1. API Limitation:

   • I have found that the API does not allow fetching more than 1000 objects. For example, a query to the endpoint https://api.artic.edu/api/v1/artworks/search?query[term][artwork_type_id]=1&page=54&limit=20 results in an error when trying to access pages beyond a certain limit.
   • The error received is: {"status":403,"error":"Invalid number of results","detail":"You have requested too many results. Please refine your parameters."}

2. Data Dump Discrepancy:

   • In the allArtworks.jsonl file available in the data dump, there are approximately 120,000 records.
   • However, in the /json/artworks/ directory, there are only about 10,000 files. Moreover, the content of these files appears to be less comprehensive compared to what is returned by the API.

I am seeking clarification on the following points:

• Is there a way to bypass the 1000 object limit through the API to access more records?
• Why is there a discrepancy in the number of records between the allArtworks.jsonl and the /json/artworks/ directory?
• Is the /json/artworks/ directory intended to be a complete representation of the collection, or is it a subset?
• Are there any best practices or alternative methods recommended for accessing a more comprehensive set of records?

Any guidance or information you can provide on these matters would be greatly appreciated, as it would significantly aid in the utilization of the Art Institute of Chicago's API for research and development purposes.

Thank you for your time and assistance.

Best regards,