CUMULUS-3890: Address sftp provider multi-part/fastGet concern #3809
Conversation
| @@ -112,7 +112,9 @@ export class SftpClient { | |||
|
|
|||
| log.info(`Downloading ${remoteUrl} to ${localPath}`); | |||
|
|
|||
| await this.sftp.fastGet(remotePath, localPath); | |||
| await this.sftp.fastGet(remotePath, localPath, { | |||
There was a problem hiding this comment.
the ticket AC mentions fastGet use is justified/confirmed as acceptable, and an alternative implementation is made configurable, with fastGet disabled by default.
how would I even make an alternate implementation available? like an option for them to have concurrency if their server can handle it, is there some process.env or other var I can use, a little confused about that
| @@ -112,7 +112,9 @@ export class SftpClient { | |||
|
|
|||
| log.info(`Downloading ${remoteUrl} to ${localPath}`); | |||
|
|
|||
| await this.sftp.fastGet(remotePath, localPath); | |||
| await this.sftp.fastGet(remotePath, localPath, { | |||
There was a problem hiding this comment.
fastGet use is justified/confirmed as acceptable: the other SFTP alternative would be the simple get, which isnt as performant and requires a bit more configuration and parameters to work properly. I think fastGet can continued to be used with some way or check on what the concurrency should be. If DAACs can handle concurrency than that can be left alone or configured, and if they can't it can be set to 0 by default. I was reading something about read/write streams as an alternative so that may be an option, but I think this is one as well.
paulpilone
requested review from
paulpilone
and removed request for
paulpilone
Summary: Summary of changes
Addresses CUMULUS-XX: Develop amazing new feature
Changes
PR Checklist