[BUG] tar: opensearch-1.3.0/jdk/...: implausibly old time stamp 1970-01-01 00:00:00 (JDK 11) in 1.3.0

[peterzhuamazon]

tar: opensearch-1.3.0/jdk/legal/jdk.zipfs: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.xml.dom: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.unsupported.desktop: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.unsupported: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.security.jgss: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.security.auth: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.sctp: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.scripting.nashorn.shell: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.scripting.nashorn: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.rmic: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.pack: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.net: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.naming.rmi: implausibly old time stamp 1970-01-01 00:00:00
tar: opensearch-1.3.0/jdk/legal/jdk.naming.ldap: implausibly old time stamp 1970-01-01 00:00:00

This only happens in 1.3.0 after we change to JDK11.
I am not able to understand how this is coming from.

To reproduce, download this and extract it:

wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.0/1240/linux/x64/dist/opensearch/opensearch-1.3.0-linux-x64.tar.gz
tar -xzvf opensearch-1.3.0-linux-x64.tar.gz

Thanks.

[reta]

@peterzhuamazon I cannot reproduce it on my Ubuntu 21.10 :(

[peterzhuamazon]

@peterzhuamazon I cannot reproduce it on my Ubuntu 21.10 :(

Interesting, it shows on my AL2 test machine.
Let me dig a bit more into it.

Thanks.

[peterzhuamazon]

Seeing some possibly related notice here: https://bugzilla.redhat.com/show_bug.cgi?id=1229160

[reta]

Seeing some possibly related notice here: https://bugzilla.redhat.com/show_bug.cgi?id=1229160

Seems to be old one, 2017, I will check with different containers as well

[peterzhuamazon]

I am specifically testing on AL2 EC2 instance.
Not sure about containers at the moment.

[kotwanikunal]

Ran it on Ubuntu 20.04 and AL2 5.1
Noticed similar issues with AL2. Investigating
AL2-tar-log.txt
ubuntu-tar-log.txt

[kotwanikunal]

Found a similar issue: https://jira.mongodb.org/browse/TOOLS-2706
https://github.com/mongodb/mongo-tools/pull/309/files

But does not seem to be specific to the JDK version.

[bbarani]

@kotwanikunal Any update on the fix for this bug? Please update this thread when you have an answer - https://discuss.opendistrocommunity.dev/t/implausibly-old-time-stamp-1969-12-31-error-while-untarring-opensearch-1-3-0/8989

[kotwanikunal]

@kotwanikunal Any update on the fix for this bug? Please update this thread when you have an answer - https://discuss.opendistrocommunity.dev/t/implausibly-old-time-stamp-1969-12-31-error-while-untarring-opensearch-1-3-0/8989

I tried working around the issue with the JDK versions and the tarballing configuration. I haven't had a lot of luck fixing/root causing it.
I'll be working with the team to find a path forward.

[minalsha]

@kotwanikunal any update on this?

[kotwanikunal]

@minalsha I don't think this was prioritized and is being actively looked into.
@CEHENKLE / @anasalkouz

[rishabh6788]

Checked out the opensearch repo and ran ./gradlew :distribution:archives:linux-tar:assemble. While trying to untar the tar.gz file present in distribution/archives/linux-tar/build/distributions/ I am facing implausibly old time stamp 1970-01-01 00:00:00 error for all the files related to opensearch.

After doing some deep-dive the issue seems to be coming from here. The isPreserveFileTimestamps property is set to false due to which the modification time of the files getting copied is not preserved, reason being method returning 0 instead of mod-time in epoch.

Can someone please look into this and set this property to true?
After setting the property to true by hardcoding in the code, the tarball is not throwing timestamp warning during untar.
@kotwanikunal @bbarani @peterzhuamazon @reta

[reta]

@rishabh6788 the isPreserveFileTimestamps comes from Gradle's org.gradle.api.tasks.bundling.Tar task, with default value being true. It was changed to false in this commit [1] to support reproducible builds (to eliminate dates factor), probably have to be reverted :(

[1] 6da253b

[rishabh6788]

Got it!
Not really sure about the impact of reverting the change but the warnings on each file during untar doesn't give a good user experience. @reta

[reta]

Got it! Not really sure about the impact of reverting the change but the warnings on each file during untar doesn't give a good user experience. @reta

Indeed it does not, @nknize @andrross any concerns with reverting some parts of #1995?

[andrross]

Reverting #1995 will break reproducible builds, right? Is there any way around this short of recommending that users supply --warning=no-timestamp to the tar command?

[peterzhuamazon]

It does not prevent the untar just warnings.
That is why I was not particularly concern on the 1st place, just annoying.
It will just default back to 1970 if no timestamp.

[reta]

Reverting #1995 will break reproducible builds, right?

Yes, it will

That is why I was not particularly concern on the 1st place, just annoying.

Correct, it just floods the output

[andrross]

My inclination is to keep reproducible builds (and the resulting warning from the untar command) as opposed to reverting #1995. The untar warnings can be suppressed with --warning=no-timestamp but including real timestamps breaks reproducible builds with (to my knowledge) no alternate workaround.

[minalsha]

@andrross / @kotwanikunal can one of you summarize the issue and next steps for this issue?

[andrross]

@minalsha The "reproducible builds" feature is the capability to deterministically produce the same output of a build given the same input. One example of non-determinism is using build-time timestamps on file timestamps when creating the build artifacts. A maintainer for OpenSearch in Arch Linux created a pull request (#1995) against OpenSearch core to change the gradle build settings to make the build reproducible, which involves removing build-time file timestamps (I don't actually know if the times are zeroed out or just omitted...). One side effect of this change is that by default in some environments, the tar command will produce a lot of warning statements about implausibly old timestamps when extracting the tar archive.

My opinion is that we should probably not do anything here, since reproducible builds seem like a good thing and the tar warnings are harmless (and can be avoided if you pass the --warning=no-timestamp option when untarring the archive). @reta @rishabh6788 What are your opinions on this issue?

[reta]

@reta @rishabh6788 What are your opinions on this issue?

I think there is a value in reproducible builds, but may be we need a bit different approach then just zeroing timestamps: may be we could always set the timestamp out of the last commit? In this case, whenever the same change set is being build - it is reproducible build. If it sounds like a path worth checking, I could try implementing it out, @andrross wdyt?

[andrross]

@reta Totally agree that zeroing out timestamps seems odd. But we are following the recommended approach from the Gradle documentation to create reproducible builds. I'm not opposed to using the commit time like you described, but would just worry if it introduced a lot of complexity.

[reta]

I am thinking about it as a simple change, if it is not - I will not spend time on it (Gradle is always full of surprises) ....