Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build: introduce support for reproducible builds #1995

Merged
merged 2 commits into from
Feb 2, 2022
Merged

build: introduce support for reproducible builds #1995

merged 2 commits into from
Feb 2, 2022

Conversation

inglor
Copy link
Contributor

@inglor inglor commented Jan 27, 2022

Description

Add support for reproducible builds

Reproducible builds is an initiative to create an independently-verifiable path from source to binary code 1. This can be done by:

  • Make all archive tasks in gradle reproducible by ignoring timestamp on files 2
  • Preserve the order in side the archives 2
  • Ensure GlobalBuildInfoPlugin.java use SOURCE_DATE_EPOCH when available

Issues Resolved

N/A

Check List

  • [-] New functionality includes testing.
    • [-] All tests pass
  • New functionality has been documented.
    • [-] New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@inglor
build: introduce support for reproducible builds
2f803b3 
Reproducible builds is an initiative to create an independently-verifiable path from source to binary code [1]. This can be done by:
- Make all archive tasks in gradle reproducible by ignoring timestamp on files [2]
- Preserve the order in side the archives [2]
- Ensure GlobalBuildInfoPlugin.java use [SOURCE_DATE_EPOCH] when available

[SOURCE_DATE_EPOCH]: https://reproducible-builds.org/docs/source-date-epoch/
[1]: https://reproducible-builds.org/
[2]: https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives

Signed-off-by: Leonidas Spyropoulos <[email protected]>
@inglor inglor requested a review from a team as a code owner January 27, 2022 16:04
@opensearch-ci-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 2f803b3
Log 2094

Reports 2094

@inglor
format: remove usused import
3d984a5 
Signed-off-by: Leonidas Spyropoulos <[email protected]>
@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 3d984a5
Log 2096

Reports 2096

@andrross
Copy link
Member 

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.routing.MovePrimaryFirstTests.testClusterGreenAfterPartialRelocation" -Dtests.seed=691370F61EF20A2 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ru-RU -Dtests.timezone=Pacific/Honolulu -Druntime.java=17

org.opensearch.cluster.routing.MovePrimaryFirstTests > testClusterGreenAfterPartialRelocation FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([691370F61EF20A2:555B536A4AAB5FB6]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:985)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:924)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:913)
        at org.opensearch.cluster.routing.MovePrimaryFirstTests.testClusterGreenAfterPartialRelocation(MovePrimaryFirstTests.java:67)

Another recurrence of #1957

@andrross
Copy link
Member

start gradle check

@andrross
Copy link
Member

Please rebase the latest on main to hopefully mitigate the failing test

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 3d984a5
Log 2100

Reports 2100

@andrross
Copy link
Member

Hey @inglor! I understand in abstract why reproducible builds might be useful, but can you explain the reason for adding it to OpenSearch now? Do you have a specific use case that requires this? If there is more discussion necessary or if this effort will require additional work feel free to open an issue. Thanks!

@inglor
Copy link
Contributor Author

inglor commented Jan 27, 2022 •
edited
Loading

Hi @andrross, thanks for the review! I'm one of the package maintainers of OpenSearch in Arch Linux 1 and part of our effords is to have reproducible builds of our packages 2. Currently OpenSearch fails to be reproducible 3 and hopefully this will make it green.

Slightly unrelated to this PR but the reference of the Arch Linux artifacts on the project site are not merged yet 4

Cheers,
Leonidas

@inglor
Copy link
Contributor Author

inglor commented Feb 2, 2022

Hello,
Is there something pending on my side for this PR? I see a request to rebase it but all checks now pass. Should I proceed with rebase or it's OK as is?

nknize
nknize approved these changes Feb 2, 2022
View reviewed changes
Copy link
Collaborator

@nknize nknize left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what the delay was here. I think this is good as-is. LGTM and thank you for doing this!

@nknize nknize added enhancement Enhancement or improvement to existing feature or request v2.0.0 Version 2.0.0 v1.3.0 pending backport Identifies an issue or PR that still needs to be backported labels Feb 2, 2022
@nknize nknize merged commit 6da253b into opensearch-project:main Feb 2, 2022
github-actions bot pushed a commit that referenced this pull request Feb 2, 2022
@inglor @github-actions
build: introduce support for reproducible builds (#1995)
953ee5f 
Reproducible builds is an initiative to create an independently-verifiable path from source to binary code [1]. This can be done by:
- Make all archive tasks in gradle reproducible by ignoring timestamp on files [2]
- Preserve the order in side the archives [2]
- Ensure GlobalBuildInfoPlugin.java use [SOURCE_DATE_EPOCH] when available

[SOURCE_DATE_EPOCH]: https://reproducible-builds.org/docs/source-date-epoch/
[1]: https://reproducible-builds.org/
[2]: https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives

Signed-off-by: Leonidas Spyropoulos <[email protected]>
(cherry picked from commit 6da253b)
@github-actions github-actions bot mentioned this pull request Feb 2, 2022
Merged
dblock pushed a commit that referenced this pull request Feb 2, 2022
@github-actions @inglor
build: introduce support for reproducible builds (#1995) (#2038)
d715e18 
Reproducible builds is an initiative to create an independently-verifiable path from source to binary code [1]. This can be done by:
- Make all archive tasks in gradle reproducible by ignoring timestamp on files [2]
- Preserve the order in side the archives [2]
- Ensure GlobalBuildInfoPlugin.java use [SOURCE_DATE_EPOCH] when available

[SOURCE_DATE_EPOCH]: https://reproducible-builds.org/docs/source-date-epoch/
[1]: https://reproducible-builds.org/
[2]: https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives

Signed-off-by: Leonidas Spyropoulos <[email protected]>
(cherry picked from commit 6da253b)

Co-authored-by: Leonidas Spyropoulos <[email protected]>
@inglor inglor deleted the support-repro-builds branch June 21, 2022 08:16
@inglor inglor mentioned this pull request Jul 5, 2022
Merged
@reta reta mentioned this pull request Jul 6, 2022
Closed
@prudhvigodithi
Copy link
Member

Hey @inglor Coming to this from my present comment opensearch-project/opensearch-plugin-template-java#38 (comment) ;)
Just curious and for my knowledge can you share where is this gradle plugin used opensearch.global-build-info used?, may be we have consider documenting this, https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#custom-gradle-plugins, thoughts? @dblock @bbarani

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nknize nknize nknize approved these changes

Assignees
No one assigned
Labels
backport 1.x Build Libraries & Interfaces enhancement Enhancement or improvement to existing feature or request pending backport Identifies an issue or PR that still needs to be backported v1.3.0 v2.0.0 Version 2.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@inglor @opensearch-ci-bot @andrross @prudhvigodithi @nknize