Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add negative cache for folders #2530

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open

Add negative cache for folders #2530

wants to merge 34 commits into from

Conversation

Tulsishah
Copy link
Collaborator

@Tulsishah Tulsishah commented Sep 24, 2024

Description

We're adding a negative stat cache for folders to reduce GCS calls for negative entries. Although we initially implemented functions for this, we encountered corner cases that needed addressing, which didn't fit within the release timeframe. Therefore, @ankitaluthra1 and I decided to postpone this feature as a post-launch item, as it's not a release blocker.

Scenarios:
BqdmEy5DnjUtiP5
We can say that StatObject(A) will cache a negative folder entry for "A" because "A" can only exist as an object, not a folder. Similarly, GetFolder(A/) will cache a negative object entry for "A/" as it can only exist as a folder, not an object, in HNS with GCSFuse.

Other changes

  • Replaced the negative cache entry with an invalidation step in the DeleteFolder function. This change aligns DeleteFolder with the existing DeleteObject function, which also uses invalidation. (Because adding negative cache was changing behaviour compare to object deletion)
  • Added logic to erase cache entries for renamed folders as part of the stale data cleanup process. Added stat call to fill negative cache in composite test.
  • Implicit_dir test changes - The test were failing due to all the tests were performing operations on single folder. One test were filling negative cache and second one try to access it. So I created separate folder for each test to avoid failure.

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - NA
  2. Unit tests - NA
  3. Integration tests - Automated

@Tulsishah Tulsishah added the execute-integration-tests Run only integration tests label Sep 25, 2024
Copy link

codecov bot commented Sep 25, 2024

Codecov Report

Attention: Patch coverage is 83.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.14%. Comparing base (4f40a8f) to head (a1d930a).
Report is 12 commits behind head on master.

Files with missing lines Patch % Lines
internal/storage/caching/fast_stat_bucket.go 76.47% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2530      +/-   ##
==========================================
- Coverage   78.83%   78.14%   -0.70%     
==========================================
  Files         106      107       +1     
  Lines       11700    11777      +77     
==========================================
- Hits         9224     9203      -21     
- Misses       1995     2094      +99     
+ Partials      481      480       -1     
Flag Coverage Δ
unittests 78.14% <83.33%> (-0.70%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Tulsishah Tulsishah marked this pull request as ready for review October 3, 2024 07:31
@Tulsishah Tulsishah requested a review from a team as a code owner October 3, 2024 07:31
@kislaykishore kislaykishore requested review from a team, BrennaEpp and gargnitingoogle and removed request for a team and BrennaEpp October 3, 2024 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
execute-integration-tests Run only integration tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants