Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jenkinsfile change to support branching #2559

Closed

Conversation

prudhvigodithi
Copy link
Member

@prudhvigodithi prudhvigodithi commented Sep 6, 2022

Signed-off-by: prudhvigodithi [email protected]

Description

Jenkinsfile change to support branching, starting with daily used jenkins jobs of OpenSearch and OpenSearch dashboard.

Issues Resolved

Part of: #2549
Related: #2558

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@prudhvigodithi prudhvigodithi requested a review from a team as a code owner September 6, 2022 20:29
@prudhvigodithi prudhvigodithi marked this pull request as draft September 6, 2022 20:31
@codecov-commenter
Copy link

codecov-commenter commented Sep 6, 2022

Codecov Report

Merging #2559 (3046a28) into main (a96221b) will decrease coverage by 0.13%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #2559      +/-   ##
============================================
- Coverage     94.41%   94.28%   -0.14%     
  Complexity       28       28              
============================================
  Files           219      213       -6     
  Lines          4462     4357     -105     
  Branches         29       10      -19     
============================================
- Hits           4213     4108     -105     
  Misses          243      243              
  Partials          6        6              
Impacted Files Coverage Δ
deployment/lib/identities.ts
...loyment/lambdas/cf-url-rewriter/cf-url-rewriter.ts
deployment/lib/artifacts-public-access.ts
deployment/lib/build-artifact-stack.ts
deployment/lambdas/cf-url-rewriter/https-get.ts
deployment/lib/buckets.ts

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@gaiksaya
Copy link
Member

gaiksaya commented Sep 6, 2022

Hi @prudhvigodithi, can you check this issue? #1606
We can put this functionality into a common pipeline which then can be consumed by all JenkinsFiles

@peterzhuamazon
Copy link
Member

There are too many stash unstash and I wonder is there a way to load a common lib like @gaiksaya suggested.

@prudhvigodithi
Copy link
Member Author

The problem I see is git checkout, yes can be added to shared library, still that shared library function has to be called in jenkinsfile, but I dont see a way we can escape the unstash line, as jenkins expects for every stage to have that unstash method so that it can re-use those cloned files, will do some more research here.
@peterzhuamazon @gaiksaya
Thank you

@prudhvigodithi prudhvigodithi force-pushed the jenkinsfiles branch 3 times, most recently from c3f312c to da67b1a Compare September 7, 2022 00:34
@prudhvigodithi prudhvigodithi marked this pull request as ready for review September 7, 2022 00:35
@prudhvigodithi
Copy link
Member Author

Even to load the shared library we need to add a stage/line to checkout and stash, I dont see a huge difference here to add 2 line in the jenkinsfile itself that does the git checkout and stash the files, also to unstash I dont see any other way to declare globally other then to have unstash part of each stage.

description: "Build repo branch to use.",
defaultValue: "main",
trim: true
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels very wrong. This job itself lives in this repo in a branch. If I am running the job out branch X, I would not expect the job to check out build tools from branch Y.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is done for the fact Jenkinsfiles cannot be branched, they has to reside in main and jenkins job git SCM configuration uses single branch to pull the jenkinsfile, once jenkinsfile is pulled, depending on the branch passed as input using BUILD_BRANCH, then the checkout happens with right branch and all the stages are executed as per the flow.

@dblock
Copy link
Member

dblock commented Sep 7, 2022

I think you want to model this after GitHub actions. The .yml jobs for GHA live in a branch (e.g. 1.2.3) and run from that branch (1.2.3) checking out source code by default from the same branch (1.2.3). This it's never a parameter to the job, the only time you care about which branch to use is when you import the job into Jenkins.

@prudhvigodithi
Copy link
Member Author

I think you want to model this after GitHub actions. The .yml jobs for GHA live in a branch (e.g. 1.2.3) and run from that branch (1.2.3) checking out source code by default from the same branch (1.2.3). This it's never a parameter to the job, the only time you care about which branch to use is when you import the job into Jenkins.

Hey @dblock, assume the build repo is branched as main, 2.x and 3.x, now jenkinsfiles can only be on main, since the job configuration need to pull the jenkinsfiles from a fixed branch (which has be configured inside the job settings), now we have to build workflow code part of 2.x (we are releasing 2.3.0), the job clones the jenkinsfiles from main and switch the branch to 2.x using the BUILD_BRANCH parameter, this way the workspace now has the right 2.x release code to run the build and assemble.

@gaiksaya
Copy link
Member

gaiksaya commented Sep 8, 2022

I think what is causing the confusion is job configuration (one time thing manually for now) is where we state from where will the related JenkinsFile be pulled up.
See the snippet below:
image
Need to check if that branches to build is configurable via code.

@prudhvigodithi prudhvigodithi force-pushed the jenkinsfiles branch 3 times, most recently from 75ddc56 to 4594eb6 Compare September 8, 2022 18:12
Signed-off-by: prudhvigodithi <[email protected]>
Signed-off-by: prudhvigodithi <[email protected]>
@prudhvigodithi
Copy link
Member Author

We can go with this PR until the Branching Strategy Implementation is finalized, I dont see any harm that would break the existing distribution functionality. This input value BUILD_BRANCH would be basic requirement to fetch the right branch code.
Since the release 2.3.0 is ongoing waiting to get a RC candidate for 2.3 before we can merge this.

@dblock
Copy link
Member

dblock commented Sep 8, 2022

I think this is the wrong implementation for an otherwise good idea. As an example, your distribution-build.jenkinsfile comes from main, but you're trying to specify 1.2.3 as a build branch. The problem is that it's the branch in the same repo, so your .jenkinsfile may or may not work with version 1.2.3 at all.

I think you want to extract the python tooling into a new repo, then hard-code the branch to use from that repo in the manifests, not take a parameter in build. This would make it reproducible and configurable in 1 place.

@prudhvigodithi
Copy link
Member Author

prudhvigodithi commented Sep 8, 2022

Hey @dblock I like the idea to separate the python tooling and pull the right tooling from manifest using the branch (or version), but its just not python tooling, we have some scripts here as well https://github.com/opensearch-project/opensearch-build/tree/main/scripts that needs to be focused for branching.

The problem is that it's the branch in the same repo, so your .jenkinsfile may or may not work with version 1.2.3 at all.

@dblock Jenkinsfile cannot be branched, hence the are only part of main , shared libraries are already being separated out and will be imported by version, jenkinsfile will switch the branch and execute the stages as per release. So ideally same .jenkinsfile will be used for 1.x and 2.x, i mean if there is a drastic change in jenkinsfile, even with separation there are chances pipeline fails for past releases as well ;).

How about we get this merged, observe with BUILD_BRANCH input and then if required consider for separating the python tooling or even other scripts if required?

If no then I will close this PR and start focusing on python workflow separation and consider branching(or versioning) on those separated repos. :) Once done then some versioning has to be done for those scripts as well https://github.com/opensearch-project/opensearch-build/tree/main/scripts, but this can be later, we can start with workflow separation and learn as we observe.

@bbarani @peterzhuamazon @gaiksaya
Thank you

@dblock
Copy link
Member

dblock commented Sep 13, 2022

I recommend closing and focusing on branching those new repos, because this is a step in the wrong direction.

@prudhvigodithi
Copy link
Member Author

Closing this PR. More details #2549 (comment), #2549 (comment)
Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants