qps/rate limit & reduce expire seek range & expire-key hash#204

Open

piaoairy219 wants to merge 61 commits intodistributedio:masterfrom zzstorage:master

+1,748-223

Commits on Sep 3, 2019

fix empty command which cause titan to crash in parsing command
xufangping
committed
change empty command's reply error from ErrEmptyArray to ErrEmptyCommand
xufangping
committed
close the connection when meet empty command
xufangping
committed

Commits on Sep 6, 2019

some illegal clients send unreadable command which may cause titan crash. just after read the command we check if it is in command list we support, else will skip it
xufangping
committed
if a connection send unknown commands 3 times, close it
xufangping
committed

Commits on Sep 18, 2019

add expire left seconds metrics
xufangping
committed
fix expire left time calculating
xufangping
committed

Commits on Sep 19, 2019

when set left/delay current seconds(expire), also set other seconds to 0
xufangping
committed

Commits on Oct 25, 2019

delete zadd args output
xufangping
committed

Commits on Nov 7, 2019

first version of rate limit
xufangping
committed

Commits on Nov 21, 2019

1 use rate.limiter to implement limit and read tikv key/value to get new limit
xufangping
committed
qps also can be set burst, its limit also support k/K/m/M suffix
xufangping
committed
1 change log level--limit not set, to debug
xufangping
committed
add limit default config items and fix error in config.go
xufangping
committed
in startSyncNewLimit, just read all match limit once for every command's qps/rate
xufangping
committed

Commits on Nov 22, 2019

change got limit log trace
xufangping
committed
add limit cleared log trace, just log limit is trigger when delay > 0
xufangping
committed
change limit/commandFunc cost seconds factor from 2 to 1.4
xufangping
committed
fix titan active time decoding bug
xufangping
committed
fix limit local percent bug
xufangping
committed
fix titan active time parsing bug
xufangping
committed
add updateLimitPercent log
xufangping
committed
when create commandLimiter, also use localPercent to set limit
xufangping
committed

Commits on Nov 25, 2019

fix limitersMgr localPercent using bug and lock before use it
xufangping
committed
reportLocalStat every globalBalancePeriod,even the commandLimiter is nil
xufangping
committed

Commits on Nov 26, 2019

limit can also work on auth-disabled titan server
xufangping
committed
decrease the lock range in reportStat
xufangping
committed
reportLocalStat run in itself go routine
xufangping
committed

Commits on Nov 29, 2019

1 change how to balance
xufangping
committed

Commits on Dec 2, 2019

Commits on Dec 3, 2019

when all titan'a limiter qps is < devideUsage of its weight percent, add current titan weight again
xufangping
committed

Commits on Dec 4, 2019

add showlimit.sh and add a conf parameter for setlimit.sh
xufangping
committed

Commits on Dec 6, 2019

1 resolve the problem that titan still process command for remote closed pipelined connection
xufangping
committed
1 BytesArray add judge for write eror
xufangping
committed

Commits on Dec 7, 2019

Commits on Dec 11, 2019

1 in Exec(), add transaction begin, cmd args num, cmd func, reply func metrics
xufangping
committed
simplify the msg recv/send log in debug level
xufangping
committed
fix bug of setting cmd arg num metric in Exec()
xufangping
committed

Commits on Dec 12, 2019

add expire/gc round/seek/commit cost metrics
xufangping
committed

Commits on Dec 22, 2019

prevent connection more and more: when tikv command cost high, clients reconnect before it finish
xufangping
committed

Commits on Dec 25, 2019

fix bug of connection limit
lugang
committed

Commits on Feb 18, 2020

1 add metrics range, fix cost exceed max value problem
xufangping
committed
Merge branch 'fpxu_dev' into 'master'
xufangping
committed

Commits on Feb 19, 2020

1 add example for limit-connection and max-connection-wait in titan.toml
xufangping
committed
1 fix unlock bug in limit-connection
xufangping
committed
don't lock s.serverctx.lock when limit-connection disabled
xufangping
committed

Commits on Feb 25, 2020

expire seek end in at:now keys to reduce rocksdb tomstone problem
xufangping
committed

Commits on Feb 26, 2020

fix a failed case in tests/redis/unit/expire.tcl
xufangping
committed

Commits on Feb 27, 2020

expire seek start in last processed expireKeys to reduce rocksdb tomstone problem
xufangping
committed
fix expire log
xufangping
committed
also reduce expire seek range when no expire keys or all expire keys > now
xufangping
committed

Commits on Feb 28, 2020

also reduce expire seek range when no expire keys or all expire keys > now
xufangping
committed

Commits on Mar 2, 2020

Commits on Apr 21, 2020

hash expire-key, it will improve the keys num handled every seconds, and prevent that current expire-keys region writing node have higher load than other nodes
xufangping
committed

Commits on Apr 24, 2020

1 fix runExpire's problem that unhashed goroutine scan hashed expire-key
xufangping
committed

Commits on Apr 26, 2020