Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Client stats and blacklist #1286

Merged
merged 9 commits into from
Oct 10, 2023
Merged

Client stats and blacklist #1286

merged 9 commits into from
Oct 10, 2023

Conversation

Hitenjain14
Copy link
Member

Changes

Fixes

Tests

Tasks to complete before merging PR:

  • Ensure system tests are passing. If not Run them manually to check for any regressions 📋
  • Do any new system tests need added to test this change? do any existing system tests need updated? If so create a PR at 0chain/system_test
  • Merge your system tests PR to master AFTER merging this PR

Associated PRs (Link as appropriate):

  • 0chain:
  • gosdk:
  • system_test:
  • zboxcli:
  • zwalletcli:
  • Other: ...

@Hitenjain14 Hitenjain14 linked an issue Oct 9, 2023 that may be closed by this pull request
Throttling the client #1275
Closed
@Hitenjain14 Hitenjain14 changed the base branch from staging to sprint-1.11 October 9, 2023 21:07
@codecov-commenter
Copy link

Codecov Report

Merging #1286 (67047b0) into sprint-1.11 (e2ded67) will decrease coverage by 0.09%.
The diff coverage is 16.40%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@               Coverage Diff               @@
##           sprint-1.11    #1286      +/-   ##
===============================================
- Coverage        23.94%   23.85%   -0.09%     
===============================================
  Files               85       86       +1     
  Lines             9881    10002     +121     
===============================================
+ Hits              2366     2386      +20     
- Misses            7221     7318      +97     
- Partials           294      298       +4
Flag Coverage Δ
Unit-Tests 23.85% <16.40%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...in.net/blobbercore/allocation/file_changer_base.go 45.23% <ø> (ø)
code/go/0chain.net/blobbercore/handler/worker.go 0.00% <0.00%> (ø)
...ain.net/blobbercore/handler/file_command_delete.go 0.00% <0.00%> (ø)
...ain.net/blobbercore/handler/file_command_upload.go 13.48% <40.00%> (+0.76%) ⬆️
...ain.net/blobbercore/handler/file_command_update.go 0.00% <0.00%> (ø)
...et/blobbercore/handler/object_operation_handler.go 17.11% <20.83%> (+0.08%) ⬆️
.../go/0chain.net/blobbercore/handler/client_quota.go 15.38% <15.38%> (ø)

@Hitenjain14 Hitenjain14 marked this pull request as ready for review October 10, 2023 13:39
dabasov
dabasov approved these changes Oct 10, 2023
View reviewed changes
Copy link
Member

@dabasov dabasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dabasov dabasov merged commit 46115d5 into sprint-1.11 Oct 10, 2023
5 of 6 checks passed
@dabasov dabasov deleted the feat/client-stats branch October 10, 2023 20:09
dabasov added a commit that referenced this pull request Dec 19, 2023
@dabasov @Jayashsatolia403
@Hitenjain14 @Kishan-Dhakan @lpoli @devyetii @AmrAshraf @shahnawaz-creator @tapishsinha-rs @din-mukhammed @boddumanohar @YarikRevich @Sanchit011
Sprint 1.11 (#1279)
fdde95b 
* Debug (#1232)

* Revert "Debug (#1232)" (#1240)

This reverts commit 49a5c9e.

* Fix phase locking (#1230) (#1241)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* increase timeout by 5 mins

* Case 3 and case 4 (#1198)

* Send WM commit status to conductor

* Rename file

* Modify log type

* Add state for stopping blobber from committing WM

* Send file meta root to rpc server

* Export function to use it in conductor client

* Modify field type

* Add code to get file meta root

* Modify file meta root retrieval

* Add field

* Apply rename for conductor test based on conductor state

* Add comment

* Log error

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* Feature/ Challenge based on rounds (#1226)

* Fix

* Updated gosdk

* Fix

* Fix

* Fix

* Logging

* Debug

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix unit tests

* Logigng

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Fix

* Fix

* Updated gosdk

* Fix

* Fix

* log config too

* Fix phase locking (#1230)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* Fix

* Added logging

* Debug

* Debug

* Added reference to logs

* Fix

* Fix

* Updated gosdk

* Use slice instead of ll

* Validator last 5 transactions

* Fix

* Fix

* Updated gosdk

* Resolved comments

* Updated gosdk

---------

Co-authored-by: Hitenjain14 <[email protected]>

* Increase limit on number of open challenges per fetching (#1249)

* Fix

* Updated gosdk

* Fix

* Fix

* Fix

* Logging

* Debug

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix unit tests

* Logigng

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Fix

* Fix

* Updated gosdk

* Fix

* Fix

* log config too

* Fix phase locking (#1230)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* Fix

* Added logging

* Debug

* Debug

* Added reference to logs

* Fix

* Fix

* Updated gosdk

* Use slice instead of ll

* Validator last 5 transactions

* Fix

* Fix

* Updated gosdk

* Resolved comments

* Updated gosdk

* Fix

* Increase time limit

* Increase time limit

* Fix

* Fix

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix

* Fix

* Fix

* Cleanup logging

---------

Co-authored-by: Hitenjain14 <[email protected]>

* Refactor commit (#1239)

* add szwg to commit

* lock for query

* aggregate queries

* check len

* rmv debug

* only update file ref

* move update fields to ref

* fix numBlock download stats

* rmv log

* rmv commented collector code

* Error on renaming dir (#1250)

* throw err when renaming dir

* fix unit test

* Fix replace blobber (#1251)

* Async processing (#1225)

* fix pre download

* rmv hash

* add async processing

* use range

* fix unit test

* fix delete change

* add logs

* fix unit test

* fix delete cmd

* save file ref

* add conn timing logs

* set default to 32MB

* fix timing log

* add inner lock

* fix test

* add ctx to cancel go routine

* parallel write to file

* fix connection

* revert storage changes

* empty commit

* Fix phase locking (#1230)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* rmv allocObj from connObj

* Download block limit (#1254)

* add limit as 500 to numBlocks

* add daily and req limit

* fix test

* fix test

* Fix download stats (#1256)

* fix download stats

* fix typo

* Fix empty alloc cleanup (#1252)

* Fix replace blobber

* Fix empty alloc cleanup

* Debug Rounds fetch (#1255)

* Fix

* Updated gosdk

* Fix

* Fix

* Fix

* Logging

* Debug

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix unit tests

* Logigng

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Fix

* Fix

* Updated gosdk

* Fix

* Fix

* log config too

* Fix phase locking (#1230)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* Fix

* Added logging

* Debug

* Debug

* Added reference to logs

* Fix

* Fix

* Updated gosdk

* Use slice instead of ll

* Validator last 5 transactions

* Fix

* Fix

* Updated gosdk

* Resolved comments

* Updated gosdk

* Fix

* Increase time limit

* Increase time limit

* Fix

* Fix

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix

* Fix

* Fix

* Cleanup logging

* Fix

* Debug

* Fix

---------

Co-authored-by: Hitenjain14 <[email protected]>

* remove prefix (#1257)

* remove .gpg file if already exist

* introduce flock to apt commands

* check on docker context existance and install git

* change buildx to action

* refine buildx build name based on runner name

* create docker context based on runner name

* increase blobber and validator job timeout

* Refine docker buildx creation and usage

* Create Unique buildx per job per runner

* remove qemu and upgrade go action version to v4

* changed runners labels build-&-publish-docker-image.yml

* added server cleanup along with blobber build

* ADD disk cleanup for conductor test

* fix cleanup script

* fix cleanup script

* added sudo to perform cleanup

* bypassing cleanup failure

* bypassing cleanup failure for validators

* Fix Save change (#1263)

* add and update change

* fix empty allocation id in change

* add rename dir

* Fix move change when srcPath is same as destPath (#1270)

* fix move change when dest is same as src

* add fileID for new ref

* fix unit test

* Fix challenge timing issue (#1277)

* Fix challenge timing issue

* Fix challenge timing issue

* Fix challenge timing issue

* bypass cleanup if it is already running (#1280)

* Revert "Fix challenge timing issue (#1277)" (#1283)

* Limit Max File Size on blobber (#1274)

* Added update in config

* 5TB config

* Reverted max_file_size config on blobber and fetching from 0chain

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix unit tests

---------

Co-authored-by: dabasov <[email protected]>

* add l2 caching for alloc (#1258)

* add l2 caching for alloc

* rmv return from commit repo

* use update

* fix update

* add log

* rmv update object

* add log for allocation

* move lock to middleware

* fix save alloc update

* empty commit

* fix blobber size update

* fix unit test

* cleanup

* rmv commit in initMap

* fix unit test

* fix renamefile mock db

* add commit method to enhancedDB

---------

Co-authored-by: Yury <[email protected]>

* add list query param (#1273)

* Optimize GetRefs, correct and add indexes on ref table (#1284)

* Correct index on ref table, add path index for get refs optimization

* Fix gorm tags on ref table

* Fix typo

* fix goose migration tags

* add pg_trgm extension

* sharder keep list (#1266)

* changed get round

* updated version

* Fix/finalize (#1285)

* Debug finalize

* Debug finalize

* Debug finalize

* Debug finalize

* Debug finalize

* Fix finalize

* Debug

* Fix

* FIX

* Fixed

* Fixed

* fix list file (#1288)

* Cleanup worker for challenge tables (#1278)

* add cleanup worker for challenge tables

* move cleanupGap to config

---------

Co-authored-by: Yury <[email protected]>

* add err check and increase wait time (#1289)

* Client stats and blacklist (#1286)

* add client stats and blacklist

* add client stats check

* fix goose migration

* fix migration

* fix typo

---------

Co-authored-by: Yury <[email protected]>

* Fix nested rename dir (#1295)

* add log for rootRef

* add objTreePath

* fix rename nested dir

* use save (#1297)

* fix retry redeem wm check (#1299)

* Use rename in place of copy buffer (#1298)

* use rename than copy buffer

* fix storage tests

* fix update latest wm (#1301)

* Fix blobber stats (#1305)

* fix blobber stats

* add stats worker

* rmv listAlloc

* build stats on start

* Feat/remove native sc calls (#1306)

* removed native MakeSCRestAPICall

* removed native MakeSCRestAPICall

---------

Co-authored-by: Jayash Satolia <[email protected]>

* WM cleanup (#1296)

* add cleanup worker for wm

* add hdd tablespace

* add log for hdd path

* update default hdd path

* update owner

* add init db script

* we are already creating with tablespace in initdb.sh

* update initdb script.sh

* update sql for write_markers_archive table

---------

Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: Yury <[email protected]>

* Fix/max file size updateWorker and added config for storagesc update interval (#1308)

* Added worker for max file size

* Added config for update worker

* Debug

* Added logs for update worker

* Added fix for worker timings =

* Debug

* Debug

* Debug

* Debug

* Debug

* Removed logging

* Removed logging

* Refactor =

* Refactor

* fix tablespace permission issue in conductor test b0docker-compose.yml

* Fix/cond tablespace (#1309)

* updated docker-compose wrt conductor test tablspace

* updated docker-compose wrt conductor test tablspace

* updated docker-compose wrt conductor test tablspace

* updated blobber env

* update gosdk version to latest of sprint-1.11 (#1310)

* fix blobber stats (#1314)

* race condition fix (#1316)

* Node recovery conductor tests (#1259)

* notify on validator tickert generatd

* fix race condition

* Added changes for fail upload commit (#1318)

* Feat/download verify ct (#1317)

* notify on validator tickert generatd

* fix race condition

* miss up download for CT

* debug logs

* Improve download performance (#1315)

* improve download file

* write data

* rmv content length

* fix unit test

* add new db snapshot migration (#1312)

Co-authored-by: Yury <[email protected]>

* fix commit error (#1320)

* update gosdk (#1319)

Co-authored-by: Yaroslav Svitlytskyi <[email protected]>

* fix where statement (#1321)

* Fix/loop break (#1322)

* updated gosdk

* add mutex

* updated gosdk

* merged changes

* merged changes

---------

Co-authored-by: Hitenjain14 <[email protected]>

* Fix blobber stats panic (#1324)

* Fix blobber stats panic

* Fix

* Fix

* fix break in switch (#1326)

* change hasher to blake3 (#1325)

* change hasher to blake3

* add blake hash

* fix lint

* fix lint for hash write

* update gosdk

* Fix challenge worker (#1327)

* revert changes

* fix challenge worker

* fix lint

* add goto for accept case

---------

Co-authored-by: Yury <[email protected]>

* Config watcher (#1329)

* add watch config

* add on config change

* Remove min lock demand from blobber (#1332)

* fix alloc lock (#1334)

* Feature: implement fork per run strategy (#1335)

* feature: added fork creation

* feature: improved Tenderly fork creation flow

* Merge pull request #1336

* rmv wm lock table

* fix unit tests

* rmv write_locks table

* rmv gorm tag

* empty commit

* Fix mem usage (#1337)

* updated gosdk to blobber

* change hasher to sha2 256

* fix unit test

* changed runner for lint tests.yml

---------

Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>

* fix rb for failed marker (#1342)

* Fix lwm (#1338)

* add wm log

* fix log

* fix update allocation

* fix log

* lru of value

* fix alloc update

* Fix auth ticket (#1341)

* fix read auth ticket

* fix unit test

* convert to base64

* Fix expiration column name (#1343)

* Fix expiration column name

* Fix size param

* merged changes

* Revert "merged changes"

This reverts commit 39f717a.

* feature: added custom block number during Tenderly fork creation (#1345)

* fix update alloc (#1347)

* extended field for mimetype

* add aws secrets for blobbers (#1349)

* add aws secrets for blobbers

* add validator secrets

* typo

* update logs

---------

Co-authored-by: sanchit <sanchit4242.com>

* Fix root hash (#1346)

* add root logs

* check alloc root

* fix lint

* rmv wm cleanup

* fix lint

* fix encrypted key point (#1350)

* fix encrypted key point

* check root ref precommit

* update min-submit to 20% (#1348)

* update gosdk to v1.11.0 (#1351)

* fix wm retries (#1352)

---------

Co-authored-by: Jayash Satolia <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Laxmi Prasad Oli <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: Amr Amin <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: tapishsinha-rs <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Sanchit Sharma <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dabasov dabasov dabasov approved these changes

@peterlimg peterlimg Awaiting requested review from peterlimg peterlimg is a code owner

@guruhubb guruhubb Awaiting requested review from guruhubb guruhubb is a code owner

@cnlangzi cnlangzi Awaiting requested review from cnlangzi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Throttling the client
3 participants
@Hitenjain14 @codecov-commenter @dabasov