Various fixes for IPFS directory handling #38
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These values were used internally by the IPFS guest implementation, but are useful for any guest implementation; providing a name for the `fs.FS` root, and a common set of error values that may be returned from `fs.FS` methods and its extensions.
Extracted from the IPFS guest implementation as it will be useful to other guest implementations as well.
Directories have a size of `0` as a convention, but not as a rule. Regardless, directories must set the valid bit to `true` in their `Rgetattr` if the value being returned is in fact valid (`0` or otherwise).
The table is expected to contain `nil` values as handles open and close (to emulate the behaviour of Unix's lowest available `fd` semantics). As a result, we need to detect and skip processing them during `fileTable.Close`. In particular, this was affecting `goWrapper.Destroy` which calls `fileTable.Close`. The `cgofuse` library internally suppresses panics so the effects of this were not visible when unmounting with `nil` handles.
This became unnecessary during one of the `command` refactors.
Currently, the unmount phase waits for the server shutdown phase to return. We do this to assure that connected clients don't instantiate new mountpoints while the unmount phase is processing or after it has already returned. It's possible for these to be more independent, but since they depend on sequential operation, we can drop a WaitGroup and a goroutine for now.
The `if` condition wasn't here originally. When it was added, the assigned value wasn't actually used.
Originally, directory streams were struct values, and a channel field was used to validate if the file was open / not-yet-closed. Eventually, that was changed to be a pointer to struct, which was used as a check condition instead. This invalidator path was never removed, and causes deadlocks if the caller tries to read from a directory which already encountered an error (either `EOF` or any other).
The Go standard library seems to use a normalized name format of lowercase, non-annotated operation names. We'll mimic this, especially since type names could change and break these prefixes (as can be seen in `pinDirectory.Close`).
The IPFS subsystem used to be constructed from the core API if it was not optionally overridden. This option and the default construction logic were removed, and an `fs.FS` was added as a required parameter to the IPNS constructor; but the default field filler was not cleaned up after this change happened.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #29
This includes a number of drive-by fixes in other areas as well, but the main focus is on 99b7af8 and later.
(Not worth extracting the others and submitting them separately right now.)
Sharded directory nodes should resolve correctly with these patches, and some other potential problems that would result in bad behaviour were found and fixed (bad return values and deadlocks when receiving errors during directory operations).
The implementation of the resolver originally used the core API directly which implied all our operations would be making requests to the node even if we already had some data cached locally. (The commit also needed a fix).
Since we already implemented caching (for the same kind of data the resolver interface also requires), it made sense to utilize that instead of always requesting new data from a (potentially remote and/or slow) node.
This implementation is somewhat ugly, but it seems to work so I'm merging it now.
I think a proper solution will be to tackle #30 which should obviate most of this code and have us going back to making requests to the node via the core API, but caching its responses for the various data we'll need (mitigating any repeated request costs).
This puts all of the responsibility on to the node and its implementation, which has advantages and disadvantages.
We'd need to depend on and communicate with it more, which might impact performance, but our code shouldn't have to adapt or handle many edge cases (like the one this pr fixes) since it's the node's responsibility to handle our high level requests correctly.