BUG/ENH: modernize cosmic_gps methods

[jklenzing]

The current structure of the cosmic_gps instrument needs to be updated for the pysat 3.0.0 release. While working on bugs related to pysat/pysat#337, a number of areas to improve have been noted, including

• Multiple spacecraft and antennas lead to possible "duplicate" timestamps. The enforcement of unique times on data requires the addition of fractional seconds to the data timestamps. While this does not affect some analysis methods (eg, climatology), a better long-term solution for this is to move to xarray.
• This also affects the file lists. The current system solves this by adding fractional seconds (<0.01s in total) to timestamps both for the file lists. Because "duplicate" files are effectively allowed, a user could potentially download multiple versions of a data file over the lifetime of the mission and wind up loading all data sets, including old versions of the same file.
• The download method and storing using the 'YYYY.DDD' format makes the parsing less clean. There's currently a workaround in BUG: recognize string variables in parse_delimited_files pysat#439, but this could be cleaned up with a breaking change.

Proposed updates:

• Handle the data using xarray
• Expand the fname string to include version numbers and potentially other keywords. May require updates in the core instrument object (breaking)
• Improve the directory structure by letting pysat handle the day subdirectories (breaking)

ref: https://cdaac-www.cosmic.ucar.edu/cdaac/cgi_bin/fileFormats.cgi?type=ionPrf

[jklenzing]

A possible route forward for when pysatCDAAC is created:

• Add another dataset (GRACE?) that does not require username / password
• generalize download / list_files, etc so that these routines can be tested

EDIT: created #3 for discussion of generalized method functions.

[aburrell]

GRACE is a useful data set, I think this is a good idea.

[rstoneback]

When I just checked on CDAAC I got prompted for a username and password for GRACE data. I’m all for finding a way to get the files tested. We could potentially store credentials on travis CI as environment variables.

[aburrell]

That's a bad idea. I'd hate for pysat to get a bad rep because CDAAC got swamped after someone stole our account details. Unless there's a way to do it silently.

[rstoneback]

Download method has been updated for the public option and the subdirectories were updated to year/day in #17.

[rstoneback]

Version information added to the COSMIC file format string in #17.

[rstoneback]

xarray support added in #18.

[rstoneback]

Version information added to the COSMIC file format string in #17.

The code over in #17 has been improved since the first note. Now, time mangling depends entirely upon invariant parsed information from filenames, so newer versions of a file will get the same time offset and be recognized as a newer version.

[rstoneback]

Altitude_binning expanded across file types in #19

[jklenzing]

Other options to consider:

• rename platform to cosmic1 for consistency with categories on CDAAC website
• Add inst_id options to separate postProc from reprocess2013 data. Currently, the code follows the website recommendation that the reprocess data is preferentially selected if available.

[rstoneback]

There is a user friendly aspect of defaulting to the latest data. Easier on the develop side if the files were more consistent.

I don't think I've tried this but we could potentially make an inception like Instrument, or a Russian nesting doll Instrument. Idea would be three instruments, one for the old and new datasets, and the third internally uses functions from each to get the job done? This option does give everyone what they want, users can select either the old or new datasets, or they could get the Instrument that auto selects whatever is most recent. Brainstorming thought at this point.

[jklenzing]

We could use three separate inst_id values, one for postProc, one for reprocess, and one that combines the two. In a practical sense, there should not be overlap between days (if memory serves).

[rstoneback]

Sure. All a question of how big of a file do we want. More organizational than technical.