Fix JENKINS-56740 - AWS CLI Option for artifacts over 5Gb with S3 Storage Class select option (STANDARD, STANDARD_IA). #467
Conversation
Option to use AWS CLI for big artifacts (over 5Gb) upload to AWS S3 with default Storage Class (STANDARD). CLI - Command Line Interface. Fix for JENKINS-56740 https://issues.jenkins.io/browse/JENKINS-56740 Build note: 1. Some tests fail, so may run with ignoring tests: -DskipTests 2. build with Maven option -Dchangelist=plugin-version for example: mvn clean package -Dchangelist=master-tag.patch.version1.0 or with cuirrent date, the lates tag and skipping tests: mvn clean package -DskipTests \ -Dchangelist=$(git tag -l --sort=creatordate | tail -n 1).patch2.0.$(date +%Y%m%d-%H%M%S) Signed-off-by: tetiana.tvardovska <[email protected]> Change-Id: Ic0e35c2009afe88802bfb0a0476ede7311aec056
STANDARD_IA S3 Storage Class was required for AWS cost-saving. Build command sample: mvn clean package -DskipTests \ -Dchangelist=$(git tag -l --sort=creatordate | tail -n 1).patch2.1.$(git rev-list --count HEAD).$(date +%Y%m%d-%H%M%S) Signed-off-by: tetiana.tvardovska <[email protected]> Change-Id: Ia92e5731549d7161b6bdfa1a27f68a5acd72d70a
1. Add option to select a custom AWS S3 Storage Class for uploaded objects:
- STANDARD (default)
- STANDARD_IA
Note:
a) SUPPORTED ONLY by AWS CLI mode.
Selected custom Storage Class is supported by AWS CLI mode only!
b) NOT SUPPORTED by AWS API mode.
Default AWS API mode ignores any selected custom AWS S3 Storage Class -
it will always upload filed with STANDARD Storage Class.
2. Updated Readme with build and test notes.
3.Add Maven property patch.version
Notes:
- Jenkins (2.443) or higher required
- Build with skip test and Maven option -Dchangelist=plugin-version
For example:
mvn clean package -DskipTests \
-Dchangelist=$(git tag -l --sort=creatordate | tail -n 1).patch2.2.$(git rev-list --count HEAD).$(date +%Y%m%d-%H%M%S)
Or with patch.version:
mvn clean package -DskipTests \
-Dchangelist=$(git rev-parse --abbrev-ref HEAD)_v\${patch.version}-$( \
git rev-list --count HEAD).$(git rev-parse --short HEAD)$(git diff --quiet || echo .diff).$(date +%Y%m%d-%H%M%S
Signed-off-by: tetiana.tvardovska <[email protected]>
Change-Id: Ibeb1fbf74edba428ea0f0332f6cba12fdeb0a049
Add notes to Readme.md about new AWS options and other changes. Signed-off-by: tetiana.tvardovska <[email protected]> Change-Id: Ice71386e51411665cb68588c25f7b81f01fccd02
|
For information: I did not managed to pass Maven build tests for the original plugin code. |
|
Compare #88, which if it works (I have not reviewed it in detail) seems more attractive because it does not rely on a particular executable being present on the agent. |
This plugin is tricky. (@cloudbees also runs CI for trunk and selected PRs against a real AWS account.) I use a |
Fixes for failed ci.jenkins.io / SpotBugs checks https://github.com/jenkinsci/artifact-manager-s3-plugin/pull/467/checks?check_run_id=22052415895 for Review Request jenkinsci#467 jenkinsci#467 Signed-off-by: tetiana.tvardovska <[email protected]> Change-Id: Ie446e4b334b71959905b82da92be0eda08465a85
Changes for AWS CLI and Storage Class options: 1. Update README.md - add description with images. 2. Update README.md images related options: bucket-settings and custom-s3-service-configuration. 3. Update config validation messages. 4. Update Help texts. Signed-off-by: tetiana.tvardovska <[email protected]> Change-Id: I63cd59fa01425ad01b48f50cf5a2e6410449d603
|
Hi @jglick ,
Sometimes it is better to have a not super ideal working workaround with rely on a particular executable being present on the agent, than to avoid using this plugin at all... or patch its each new version to keep the working environment... Besides big files upload, we also need artifacts to be uploaded with S3 Storage Class Standard-IA (STANDARD_IA) for cost saving. By the way, should I split 'big files upload' and 'S3 Storage Class' issues into separate pull requests?... Anyway, I will check #88 and related #141. But, unfortunately, neither #88 nor #141 would meet our needs for big files and Standard-IA Storage Class... |
|
IMHO upload/download an artifact of 5GB size is wrong. There is many ways to upload it making it easy to download it, for example making volumes of a fixed size. To use the command line to upload an artifact it it is above some size looks wrong too any error that happen in the CLI would not be controlled and properly reported. So I will not block this PR but I think it will cause more harm than benefit. Make things complicated does not have good endings. |
Huh, this is surprising. One possible issue: we are still using the v1 SDK. It may be that these features and more are covered by the v2 SDK (see e.g. #351), which would need to get packaged analogously to https://github.com/jenkinsci/aws-java-sdk-plugin so that various plugins can migrate to it. Please be aware that the intent of this plugin is not to support every S3 feature, it is to make it convenient for people to use stock Pipeline code with steps like Since I do not really have time to maintain this plugin I have asked some colleagues to help review and test this and some other important-looking PRs, but it may take some time.
Well, this is I think handled already. Yes the extra complexity is a downside. I do not have a strong opinion either way for now. |
|
BTW If you’re uploading a 5GB Artifact to Jenkins you should think about to upload it to a OCI register or directly to a service designed to store those kind of artifacts. Jenkins is not a OCI register and it should not be one. |
It could be wrong, but my project had started using such schema several years ago. It takes Jenkins just about 20 minutes to upload artifacts from build nodes to AWS S3 in our environment. Similar reasonable time is taken to retrieve artifacts for other jobs. So, why not use Artifact Manager S3 Plugin to upload build results to S3 even if result files are pretty big? As long as system performance is acceptable, why not use it?...
Provided code logs (reports) all errors that may happen in the CLI. Errors are reported to the console log and could be addressed effectively.
Strange... Artifacts are stored at AWS S3 service, thanks to this plugin. Not at Jenkins server or build nodes. |
I know it, I developed part of this plugin, its tests, and the documentation.
the simply use of two ways to upload the files make the maintain more complex. Introduce a CLI wrapper implies any change in the behavior of the CLI will break the plugin, the CLI could be any version of it so you have a new world of posible issues.
20 min uploading a file are 20 min paying to not have any network issue. |
May be... I am a newbie in Jenkins and AWS programming, so could miss something. Will check it.
Sure! We just wanted to reduce our costs when using Amazon S3 with Thus, we considered to create objects from the very beginning with desired Nevertheless, I may add all other Storage Classes for this option. May be someone will consider other Storage Class as more suitable... And it will enhance the plugin functionality. By the way, I have found a workaround how to PUT objects with required Storage Class. I am going to add the update soon. |
Add AWS configuration options:
With this change, the plugin artifacts upload method depends on the Jenkins setting at Jenkins > Setting > AWS.
Default AWS API method fails to upload artifacts bigger than 5GB.
AWS CLI Option "Use AWS CLI for files upload to S3"
Workaround or fix for JENKINS-56740.
AWS CLI Option is a workaround for big files transfer API error for file bigger than 5Gb.
AWS CLI mode uses AWS S3 Command Line Interface (CLI) commands for artifacts upload to AWS S3.
Default AWS API fails to upload artifacts bigger than 5GB, MaxSizeAllowed is 5368709120 bytes (5Gb).
See JENKINS-56740 - plugin fails to upload artifacts having size over 5Gb with errors like:
AWS option "Use AWS CLI for files upload to S3" switches the default AWS API mode to AWS CLI mode.
Add AWS S3 Storage Class selection option
Warning: Works for AWS CLI option only.
Allows to select the AWS S3 Storage Class for uploaded artifacts.
The default S3 Storage Class is S3 Standard (STANDARD).
This option allows so select S3 Storage Class Standard IA (STANDARD_IA).
Useful for cost saving.
Testing done
Tested at local Jenkins test installation and on our development Jenkins.
Add Jenkins job to test the plugin:
Test 1. PASSED. Failed(as expected) big file upload with default AWS API
Fails upload gif files (6Gb) for default AWS API configuration with errors as expected by JENKINS-56740.
Jenkins job Artifact-Plugin-Test, build #47_AWS-API_STANDARD
Console Output:
AWS S3 has no artifacts for this build.
Test 2. PASSED. big file upload with AWS CLI option checked and STANDARD Storage Class selected:
Jenkins job Artifact-Plugin-Test, build #48_AWS-CLI_STANDARD
Console Output:
AWS S3 has artifacts for this build with the Standard file objects Storage Class.
Test 3. PASSED. big file upload with AWS CLI option checked and STANDARD_IA Storage Class selected:
Jenkins job Artifact-Plugin-Test, build #49_AWS-CLI_STANDARD_IA
Console Output:
AWS S3 has artifacts for this build with the Standard IA file objects Storage Class.
Submitter checklist