STONReaderError: File from GitStore cache is incomplete

[LinqLover]

Originally raised in #341:

If I evaluate self createVersion manually in [] in [] in SquotInteractiveSave>>applyToWorkingCopy, I get an error from tonel:
MessageNotUnderstood: UndefinedObject>>addAll: from [] in TonelWriterForFileSystem(TonelWriter)>>writePackage:, currentPackageProperties is nil and self currentPackageProperties raises another error:
STONReaderError: At character 8: 'invalid input', which is probably because git:/packages/Sandblocks-Core/package.st in this working copy has a content ( (self packageDir / 'package.st') readStreamDo: [:stream | stream contents]) like this: Package , followed by 1024 null-characters.

In the end, I was able to resolve the bug by deleting the broken file manually.

[...]

2. Apparently for some unknown reason, that package.st is damaged. How could this happen? Should this have been validated? Should there be a programmatic option to resolve such errors (without losing your entire working copy)?

Today encountered a very similar bug, with the following difference:

• (self packageDir / 'package.st') contents evaluates to 'Package '

As opposed to that, when accessing the normal FileSystem from the repository, the file contents are correct:

Additional info from the command line:

$ cat packages/Sandblocks-Core/package.st
Package { #name : #'Sandblocks-Core' }

$ git log -p packages/Sandblocks-Core/package.st
commit eee1fb343d0001e5b4ffedcce707200084c82e60
Author: Tom Beckmann <[email protected]>
Date:   Sun May 3 23:02:56 2020 +0200

    convert to tonel format

diff --git a/packages/Sandblocks-Core/package.st b/packages/Sandblocks-Core/package.st
new file mode 100644
index 0000000..c0d0e8d
--- /dev/null
+++ b/packages/Sandblocks-Core/package.st
@@ -0,0 +1 @@
+Package { #name : #'Sandblocks-Core' }
# i.e. the file was never changed

Down in the GitFilesystem/FSMemoryHandle/GitStore/FSMemoryStore, the root actually only contains these eight bytes (#[80 97 99 107 97 103 101 32]) for that file.

I did not find out how to invalidate the cache ...

[j4yk]

Thank you for sending me the image. Ok, this was tricky: debugging the reading of a repository which is not on my disk, which means I cannot read any new objects from it.

From your FSReference to git:/packages/Sandblocks-Core/package.st I went to filesystem store repository objectCache, explored your working copy of Sandblocks from the Git Browser to get hold of the GitCommit that is the HEAD, then use its tree hash to look the GitTree up in the objectCache and moved along the GitTreeEntries hashes to eventually find the GitBlob of package.st: c0d0e8d870b9f67bef27521d59facd12d2f464a1.

cache at: 975431966390542430237633531348128670717781828252 asHexHash
cache at: 1384478138030157660167919813260494686395570393543 asHexHash
cache at: 917068586878115131720361770236173787305421924245 asHexHash
cache at: 1100785066700638091304713640443222599882574685345 asHexHash

Now that I am finished it arises to me that I could also just have looked up that hash on GitHub in the Sandblocks repository... but anyway.

The GitBlob of package.st contains all the bytes of 'Package { #name : #''Sandblocks-Core'' }', String lf. I believe it was not really a question whether the file is correct in the repository, but this proves that FileSystem-Git has read it correctly and completely from the repository.

So why is the file incomplete in the memory filesystem cache anyway? Either the bytes were put there incorrectly, or the file has since been overwritten. Looking at the GitStore's modManager createdPaths the package.st is among them, so something seems to have written to the file since the store was created.

Ok no surprise given the start of the issue description:

If I evaluate self createVersion manually in [] in [] in SquotInteractiveSave>>applyToWorkingCopy, I get an error from tonel:
MessageNotUnderstood: UndefinedObject>>addAll: from [] in TonelWriterForFileSystem(TonelWriter)>>writePackage:, currentPackageProperties is nil and self currentPackageProperties raises another error:

The code has already begun writing to the file in the cache, putting 'Package ' there, then failing to write any rest because currentPackageProperties was nil. So we have to solve why it was nil.

In TonelWriter>>#currentPackageProperties I see no obvious code path where it would return nil. STON fromStream: should also not have returned nil given the bytes from above in the file.

To be continued...

[j4yk]

@LinqLover Please modify the TonelWriter>>writePackage: method as follows and wait until you see the debugger here next time:

currentPackageProperties := self currentPackageProperties.
"add:" currentPackageProperties ifNil: [self isThisEverCalled]

If it happens again, please tell me what is in the package.st (according to the contents of the FSReference) at this point (in the GitFilesystem of course, not on the disk). If possible try to restart and debug through the currentPackageProperties method to find out how it can return nil (should only do that if the file contents were "nil" or "null").

[j4yk]

...and for invalidating the cache, try theGitStore loadFromRepository: yourFSReference path.

[LinqLover]

If it happens again, please tell me what is in the package.st (according to the contents of the FSReference) at this point (in the GitFilesystem of course, not on the disk).

self being a TonelWriterForFileSystem:

(self packageDir / 'package.st') contents

yields

If possible try to restart and debug through the currentPackageProperties method to find out how it can return nil (should only do that if the file contents were "nil" or "null").

If I debug it, I get a 'STONReaderError: At character 8: ''invalid input''' because that character is a null character.

...and for invalidating the cache, try theGitStore loadFromRepository: yourFSReference path.

Translating this to myself, I evaluated

self directoryReference filesystem store loadFromRepository: self directoryReference path

in the context of [] in TonelWriterForFileSystem(TonelWriter)>>writePackage: but self currentPackageProperties still fails for the same reason afterward. (Where can I find the git store starting from "explore working copy"?)

I have prepared another image snapshot for you, the issue is reproducible if you try to commit (STONReaderError2.image).

I could work around the issue by deleting all defect package.st files manually and then restarting the active #writePackage: context.

[j4yk]

If I debug it, I get a 'STONReaderError: At character 8: ''invalid input''' because that character is a null character.

I suspect it was already too late then and the writer had already partially overwritten the file, but I cannot be sure. Did you modify the writePackage: with isThisEverCalled as I asked you to? Can you send me the image please? Maybe I also need your Git repository from the disk.

Where can I find the git store starting from "explore working copy"?

Not at all. The GitStore is a temporary object, just like its FileSystem, which only exists while the commit is being edited/prepared. You find it only while a SquitRepository creates a new commit. The SquotFileSystemStore used during that has an FSReference to the root directory of the Git FileSystem, which has the GitStore.

Each GitStore and associated FileSystem are specific to a particular commit, that's why you cannot find it from the working copy.

Translating this to myself, I evaluated

self directoryReference filesystem store loadFromRepository: self directoryReference path

in the context of [] in TonelWriterForFileSystem(TonelWriter)>>writePackage: but self currentPackageProperties still fails for the same reason afterward.

The expression looks good to me, although its effectiveness depends on choosing the correct directory and I'm not sure whether it also discards files and directories below the one you pass as an argument. So be sure to pass the exact path of the package.st file and/or its parent directory, not just the root of the Tonel tree.

[LinqLover]

I just sent you the image snapshot via Discord, hope that answers all open questions. Thank you for the explanations! :-)