Dump old WebSOC data into sqlite db

[ramanxg]

In reference to this comment, we could put old websoc data into the sqlite so that the query time is faster. This should prevent us from having to make a new request to websoc for data that will not change.

We can start by doing this with the course offerings from 2019-20 and earlier. It might be helpful to also keep track of query times so we know how much our api is improving for speed. These requests might not occur very often, but in the case that they do, we won't have to make more requests to websoc.

Todo

• Request and dump the old data into a csv file. We can do this with our own api.
• Create a table in our sqlite with appropriate fields and put csv file data into it.
• Restructure both rest endpoints and graphql so that for any requests for older years, the sqlite file is queried instead of the websoc-api.
• Test and compare speeds

[ramanxg]

Blocked for now. Need to do more research on what is the best method.

[ramanxg]

Rather than dumping old websoc data into our SQLite database, due to the large size we should create a cloud instance of a databse, RDS or MongoDB, to store this old WebSoc Data.

Each websoc query, requires a year, and if the year is not the current year, then we can instead query the database. To prevent requiring secrets and being more open to Open-source, we can check if the key is provided in our secrets, and if not we can query regular websoc as we did before.

TODO:

• Determine the size of the last ~10 years of websoc data, and find the size of cluseter we need.