Manually verifying timestamped signed git commits

[nobodyinperson]

Thanks for OpenTimestamps, it's awesome.

One thing I can't get to work is manually verifying timestamped git commits. I use the ots-git-gpg-wrapper.sh as explained in git-integration.md (actually, with my backwards-compatible improvements from #121 added, but that shouldn't matter).

# extract signed commit data
git cat-file -p 1714b1b | sed '/END PGP SIGNATURE/q' | tee signed-commit.txt
# extract opentimestamps proof
git cat-file -p 1714b1b | sed -n '/BEGIN OPENTIMESTAMPS/,/END OPENTIMESTAMPS/p' | tail -n+3 | head -n-1 | cut -c2- | base64 -d > proof.ots

# the proof doesn't work directly:
ots -v info proof.ots
# Error! 'proof.ots' is not a timestamp file.

Repeating this code in git.py manually, seems to give something:

from opentimestamps.core.serialize import BytesDeserializationContext
from opentimestamps.core.timestamp import Timestamp

with open("signed-commit.txt","rb") as fh:
    signed_commit = fh.read()

with open("proof.ots","rb") as fh:
    proof = fh.read()

print(Timestamp.deserialize(BytesDeserializationContext(proof[2:]), signed_commit).str_tree())
# append bdd57093dd87de9342c993bcc55946e779f0c070fa76d4b421d3f7babf94fc2e
# sha256
#  -> append 301c8dd928bd3596276b27e2ce8f1c56
#     sha256
#     prepend 616700e2
#     append 187039fbe3cc9c8d
#     verify PendingAttestation('https://bob.btc.calendar.opentimestamps.org')
#  -> append 3a8c547f1a8482a04e209cb5cebe3f2d
#     sha256
#     prepend 1a151e259892fe878451daed16d1cd7c949880ec021957cadc91c4f5ebedbbe1
#     sha256
#     prepend 616700e1
#     append 73c49e8e2748e8a3
#     verify PendingAttestation('https://finney.calendar.eternitywall.com')
#  -> append 7c28820529cea83633f88eb7c3e77727
#     sha256
#     prepend 5507b29856880c0b40b1c2701999da8b6534e140daa199e493d92eeff060c38d
#     sha256
#     prepend 616700e2
#     append 8b5ff6bfa8cf252f
#     verify PendingAttestation('https://alice.btc.calendar.opentimestamps.org')
#  -> append 9ca03c1a40b9bfeb9669c2c9def2581c
#     sha256
#     prepend 616700e1
#     append dd01daa38574bd81
#     verify PendingAttestation('https://btc.calendar.catallaxy.com')

However dropping these two initial bytes (which I guess is the timestamp version number) isn't enough to make it work with ots info:

cut -c3- proof.ots > proof-without-initial-2-bytes.ots
ots -v info proof-without-initial-2-bytes.ots 
# Error! 'proof-without-initial-2-bytes.ots' is not a timestamp file.

Here in cmds.py is says that ots info uses the following:

from opentimestamps.core.serialize import StreamDeserializationContext
from opentimestamps.core.timestamp import DetachedTimestampFile

with open("proof.ots","rb") as fh:
    # fh.read(2)  # 👈 this also doesn't help
    ctx = StreamDeserializationContext(fh)
    DetachedTimestampFile.deserialize(ctx)
# ---------------------------------------------------------------------------
# BadMagicError                             Traceback (most recent call last)
# /tmp/ipykernel_874077/229123088.py in <module>
#       1 with open("proof.ots","rb") as fh:
#       2     ctx = StreamDeserializationContext(fh)
# ----> 3     DetachedTimestampFile.deserialize(ctx)
# 
# /usr/lib/python3.9/site-packages/opentimestamps/core/timestamp.py in deserialize(cls, ctx)
#     327     @classmethod
#     328     def deserialize(cls, ctx):
# --> 329         ctx.assert_magic(cls.HEADER_MAGIC)
#     330 
#     331         major = ctx.read_varuint() # FIXME: max-int limit
# 
# /usr/lib/python3.9/site-packages/opentimestamps/core/serialize.py in assert_magic(self, expected_magic)
#     216         actual_magic = self.fd.read(len(expected_magic))
#     217         if expected_magic != actual_magic:
# --> 218             raise BadMagicError(expected_magic, actual_magic)
#     219 
#     220     def assert_eof(self):
# 
# BadMagicError: Expected magic bytes 0x004f70656e54696d657374616d7073000050726f6f6600bf89e2e884e89294, but got 0x0101f020bdd57093dd87de9342c993bcc55946e779f0c070fa76d4b421d3f7 instead

... which also errors out.

What's the problem here? How to verify an OpenTimestamps git timestamp manually with ots info/verify?

[petertodd]

On Wed, Oct 13, 2021 at 09:27:33AM -0700, Yann Büchau wrote: Thanks for OpenTimestamps, it's awesome. One thing I can't get to work is **manually verifying timestamped git commits**. I use the `ots-git-gpg-wrapper.sh` as explained in `git-integration.md` (actually, with my backwards-compatible improvements from #121 added, but that shouldn't matter). ```bash # extract signed commit data git cat-file -p 1714b1b | sed '/END PGP SIGNATURE/q' | tee signed-commit.txt # extract opentimestamps proof git cat-file -p 1714b1b | sed -n '/BEGIN OPENTIMESTAMPS/,/END OPENTIMESTAMPS/p' | tail -n+3 | head -n-1 | cut -c2- | base64 -d > proof.ots # the proof doesn't work directly: ots -v info proof.ots # Error! 'proof.ots' is not a timestamp file. ``` Repeating [this code in `git.py`](https://github.com/opentimestamps/opentimestamps-client/blob/master/otsclient/git.py#L51) manually, seems to give something: ```python from opentimestamps.core.serialize import BytesDeserializationContext from opentimestamps.core.timestamp import Timestamp with open("signed-commit.txt","rb") as fh: signed_commit = fh.read() with open("proof.ots","rb") as fh: proof = fh.read() print(Timestamp.deserialize(BytesDeserializationContext(proof[2:]), signed_commit).str_tree()) # append bdd57093dd87de9342c993bcc55946e779f0c070fa76d4b421d3f7babf94fc2e # sha256 # -> append 301c8dd928bd3596276b27e2ce8f1c56 # sha256 # prepend 616700e2 # append 187039fbe3cc9c8d # verify PendingAttestation('https://bob.btc.calendar.opentimestamps.org') # -> append 3a8c547f1a8482a04e209cb5cebe3f2d # sha256 # prepend 1a151e259892fe878451daed16d1cd7c949880ec021957cadc91c4f5ebedbbe1 # sha256 # prepend 616700e1 # append 73c49e8e2748e8a3 # verify PendingAttestation('https://finney.calendar.eternitywall.com') # -> append 7c28820529cea83633f88eb7c3e77727 # sha256 # prepend 5507b29856880c0b40b1c2701999da8b6534e140daa199e493d92eeff060c38d # sha256 # prepend 616700e2 # append 8b5ff6bfa8cf252f # verify PendingAttestation('https://alice.btc.calendar.opentimestamps.org') # -> append 9ca03c1a40b9bfeb9669c2c9def2581c # sha256 # prepend 616700e1 # append dd01daa38574bd81 # verify PendingAttestation('https://btc.calendar.catallaxy.com') ``` However dropping these two initial bytes isn't enough to make it work with `ots info`: ```bash cut -c3- proof.ots > proof-without-initial-2-bytes.ots ots -v info proof-without-initial-2-bytes.ots # Error! 'proof-without-initial-2-bytes.ots' is not a timestamp file. ``` [Here in `cmds.py`](https://github.com/opentimestamps/opentimestamps-client/blob/master/otsclient/cmds.py#L494) is says that `ots info` uses the following: ```python from opentimestamps.core.serialize import StreamDeserializationContext from opentimestamps.core.timestamp import DetachedTimestampFile with open("proof.ots","rb") as fh: # fh.read(2) # 👈 this also doesn't help ctx = StreamDeserializationContext(fh) DetachedTimestampFile.deserialize(ctx) # --------------------------------------------------------------------------- # BadMagicError Traceback (most recent call last) # /tmp/ipykernel_874077/229123088.py in <module> # 1 with open("proof.ots","rb") as fh: # 2 ctx = StreamDeserializationContext(fh) # ----> 3 DetachedTimestampFile.deserialize(ctx) # # /usr/lib/python3.9/site-packages/opentimestamps/core/timestamp.py in deserialize(cls, ctx) # 327 @classmethod # 328 def deserialize(cls, ctx): # --> 329 ctx.assert_magic(cls.HEADER_MAGIC) # 330 # 331 major = ctx.read_varuint() # FIXME: max-int limit # # /usr/lib/python3.9/site-packages/opentimestamps/core/serialize.py in assert_magic(self, expected_magic) # 216 actual_magic = self.fd.read(len(expected_magic)) # 217 if expected_magic != actual_magic: # --> 218 raise BadMagicError(expected_magic, actual_magic) # 219 # 220 def assert_eof(self): # # BadMagicError: Expected magic bytes 0x004f70656e54696d657374616d7073000050726f6f6600bf89e2e884e89294, but got 0x0101f020bdd57093dd87de9342c993bcc55946e779f0c070fa76d4b421d3f7 instead ``` ... which also errors out. What's the problem here? How to verify an OpenTimestamps git timestamp manually with `ots info/verify`?

Ah! So the problem you are having is because what you extracted from the git comment is only part of the .ots file format. You need to add the file header as well. Take a look at the `git extract` command. That includes the code to add that header.