Migrate Pangeo Forge API catalog into LEAP Data Catalog

[cisaacstern]

I'm winding down the Pangeo Forge hosted catalog as part of the project's refocus towards ETL tooling and supporting users (including LEAP!) to host their own catalogs, xref:

• Shutting down Heroku instance pangeo-forge/pangeo-forge-orchestrator#238
• Drop all dependency on all external APIs (including Heroku backend) pangeo-forge/pangeo-forge.org#430

As part of that, we don't want the existing data in the catalog to be forgotten, so hoping LEAP will be able to host references to it (the data itself will continue to live on OSN).

So far, this PR is just a script I'm working on to automatically extract the Pangeo Forge catalog data into the format required by the LEAP catalog. I will delete this script before the PR is complete. Opening is draft for now.

[cisaacstern]

I've exported all of the entries from https://pangeo-forge.org/catalog into the catalog here, with the exception of HadISST, which was already here!

@jbusecke, do we want to put the following cmip6 entries somewhere else:

• catalog/datasets/cmip6-pmip.yaml
• catalog/datasets/cmip6-static-ocean-grids.yaml
• catalog/datasets/cmip6.yaml

?

[cisaacstern]

@andersy005, welcome your review, though I do think I've got the technical side worked out based on the very clear README! 🙏 Mostly tagged you for visibility, since you've worked on both the Pangeo Forge catalog and this one, of course.

[andersy005]

This looks 💯 solid to me. I'm really glad that the LEAP's catalog will become a new home for the existing datasets from Pangeo Forge

[jbusecke]

Sorry I just saw this late (today is going to be purely catch up on github hahaha). This is a fantastic idea in general and I support it 100%.

I think it might be useful to chat through this, and see how we can streamline this. I am in particular curious if anything here is affected by a likely refactor in data-management.

[cisaacstern]

@jbusecke sorry for the delayed response.

I don't think this would be affected by the refactor because the data added here do not have corresponding feedstocks/recipes in this repo.

This is purely a migration of data build previously in Pangeo Forge, all of which have feedstock repos linked in the catalog entries added by this PR.

So I think we can merge unless you have other questions?

[cisaacstern]

I think the one thing we may want to exclude here is possibly the CMIP things?

[jbusecke]

I think that is the right intuition. My suggestion is:
For each of the CMIP based recipes open a request issue in the new feedstock. That way we can make sure these datasets are eventually ingested (but in a consistent manner).
I would like to have a link to CMIP data in the catalog eventually but we can discuss that separately