GLEAM 3.7b recipe

[alaws-USGS]

GLEAM 3.7b

• full dataset
• requires username and password sign up, will need to add to FTP server headers
• https://www.gleam.eu/#downloads

Originally posted by @alaws-USGS in #1 (comment)

[alaws-USGS]

Currently unable to create recipe due to Pangeo Forge recipes not support FTP clients

Looking at several solutions.

First solution to examine: passing local environment variables to fsspec.open to use FTP server

1. Add login and password as local environment variables
2. Use fsspec.open to download single file

Second potential solution:

1. Replace fsspec engine for class pangeo_forge_recipes.patterns.FilePattern

[alaws-USGS]

Update 1 on Solution 1:

On Windows Subsystem for Linux (WSL)

Added 4 variables to .bashrc (they are now environment variables:

• GLEAM_FTP: FTP endpoint
• GLEAM_USER: username
• GLEAM_PASSWORD: password
• GLEAM_PORT: port to use

[alaws-USGS]

Update 2 on Solution 1:
Recipes use FileManager object, which uses fsspec.open in the background. fsspec.open creates an OpenFile object and uses context to know what kind of file system (SFTP).. So far, it has not been enough to find the path needed for the file.

I was able to write a script and pass them as **kwargs to the FilePattern object. However, it hangs up so I need to figure out the Apache Beam.

[alaws-USGS]

Update 3 on Solution 1:
Working on adapting script to existing recipe that works rather than from tutorial.