What is our longterm project upload strategy?

[philipbaileynar]

Some givens:

1. Projects can be big. Several Gb.
2. Some clients, like USU, have many dozen projects to upload.
3. Long term, we want all users to be able to upload projects.
4. It's untenable that we continue with rscli for just NAR or a select few analysts for whom we setup the tools.

So what is our strategy for riverscapes users to contribute projects longterm? Let's try to put aside whether we are rebasing our warehouse on hydro share etc. Just fundamentally, what's the user experience we want?

• Rewrite rscli in Python or dare I say .net?
• Users have to zip their projects and upload via web browser?
  • ok for single project but sucks for lots of projects
• I had a crazy idea that for USU - we could leverage AWS CLI and have them S3 hundreds of riverscapes compliant projects into S3 and then a scraper trawl the files and harvest any validating project.

Questions

• How does HydroShare or other cloud systems do this?

[MattReimer]

Hydroshare has drag and drop functionality but it's limited to smaller projects. You can't drag a 4Gb file into a website (nor should you)

The CLI is probably the best option here. They need to use a CLI anyway if you want to do the S3 scrape option so it might as well be ours.

We experimented with a standalone CLI a while back that ships with its own version of the V8 engine. That keeps the code in the same place as everything else and prevents a rewrite but there may be challenges I haven't anticipated since this is a pretty new tech to me.

We can easily write a CLI in python (It's pretty much done anyway) but the usual caveats apply about deployment. One point in favour of python is that we don't need any GDAL or fancy pip modules so this could easily be deployed on top of ArcPy or whatever python the user has.

The DotNet option is probably best but also the most expensive option.... and it won't work for OSX users so we'd still need to maintain multiple versions (all the usual windows problems as usual). There are deployment challenges here too.

[philipbaileynar]

Really interesting Matt.

Can you explain more about the "shipping V8" option? Does this mean that each OS will get a single binary to download and run at the command line, with no dependencies?

I think rscli is completely sufficient for the short term. I just want to learn more about deployment hassles.

[MattReimer]

CLI Applications in node are a pretty good fit.
https://developer.okta.com/blog/2019/06/18/command-line-app-with-nodejs
(This was written by a guy whose talk I went to at the software conference in Ohio a couple years back)

The tech to bundle a node app is called NEXE
I already have an issue for investigating it here

It seems compelling on the surface but I'm not convinced this is a good idea. NEXE binaries tend to be huge and bloated, also pretty hard to compile. Much better just to install NodeJS on a system and use npm for our client. Our CLI doesn't use weird dependencies so the install should be pretty clean and simple. If people are going to need to use the commandline anyway then teaching them how to write npm install @riverscapes/cli is not much more work.

[philipbaileynar]

Unfortunately requiring NodeJS to be installed is a non-starter. I have never seen it work cleanly on Windows. It's also just technically beyond the user-base for this process.

Let's chat about the NEXE approach. Bloat doesn't scare me. Simplicity is the key. "download this executable and double click it." or "download this executable and run it at the DOS prompt". Unfortunately that's the ceiling we are working with. Or we look for another solution.

[joewheaton]

Seems like we need to discuss this and decide on strategy sometime in July to start planning for changes in Fall?

[MattReimer]

Maybe we should just bite the bullet and build the uploader right into QRave using python. We had a rudimentary version of this on the old QRave from a few years ago. The logistics of this aren't too hairy but it's definitely a non-zero amount of work.

I keep going back and forth on this and there isn't really a consensus out there. I think if we have the money and time we should build a UI. If we don't then we build the best CLI we possibly can and people just need to RTFM.