Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qps/rate limit & reduce expire seek range & expire-key hash #204

Open
wants to merge 61 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
db84a39
fix empty command which cause titan to crash in parsing command
Sep 3, 2019
3e4b89e
change empty command's reply error from ErrEmptyArray to ErrEmptyCommand
Sep 3, 2019
c89b27f
close the connection when meet empty command
Sep 3, 2019
43bd6c3
some illegal clients send unreadable command which may cause titan cr…
Sep 6, 2019
95bd04b
if a connection send unknown commands 3 times, close it
Sep 6, 2019
63b131d
add expire left seconds metrics
Sep 18, 2019
a9c28fe
fix expire left time calculating
Sep 18, 2019
e5ff501
when set left/delay current seconds(expire), also set other seconds to 0
Sep 19, 2019
fb11c67
delete zadd args output
Oct 25, 2019
586b30d
first version of rate limit
Nov 7, 2019
27f54e7
1 use rate.limiter to implement limit and read tikv key/value to get …
Nov 21, 2019
e5018e4
qps also can be set burst, its limit also support k/K/m/M suffix
Nov 21, 2019
ecd78a2
1 change log level--limit not set, to debug
Nov 21, 2019
0648ab8
add limit default config items and fix error in config.go
Nov 21, 2019
f86245a
in startSyncNewLimit, just read all match limit once for every comman…
Nov 21, 2019
157d70a
change got limit log trace
Nov 22, 2019
fd0d8cc
add limit cleared log trace, just log limit is trigger when delay > 0
Nov 22, 2019
756a0e6
change limit/commandFunc cost seconds factor from 2 to 1.4
Nov 22, 2019
46bb6da
fix titan active time decoding bug
Nov 22, 2019
a839995
fix limit local percent bug
Nov 22, 2019
815efa4
fix titan active time parsing bug
Nov 22, 2019
9612571
add updateLimitPercent log
Nov 22, 2019
cc4255a
when create commandLimiter, also use localPercent to set limit
Nov 22, 2019
f00d8c5
fix limitersMgr localPercent using bug and lock before use it
Nov 25, 2019
da49bfa
reportLocalStat every globalBalancePeriod,even the commandLimiter is nil
Nov 25, 2019
30f7d2c
limit can also work on auth-disabled titan server
Nov 26, 2019
bcbd5a7
decrease the lock range in reportStat
Nov 26, 2019
1a50695
reportLocalStat run in itself go routine
Nov 26, 2019
a7753c4
1 change how to balance
Nov 29, 2019
fda1e66
update some config item's description and log
Dec 2, 2019
a474656
add limit changed log
Dec 2, 2019
8ed0698
commandLimiter store globalQpsLimit/globalRateLimit in int64, use it …
Dec 2, 2019
6ae93be
when all titan'a limiter qps is < devideUsage of its weight percent, …
Dec 3, 2019
70b3f15
add showlimit.sh and add a conf parameter for setlimit.sh
Dec 4, 2019
a22ca0f
1 resolve the problem that titan still process command for remote clo…
Dec 6, 2019
8403a21
1 BytesArray add judge for write eror
Dec 6, 2019
2d59108
refactor commandLimiter's qpsLimiter/ratelimiter and their globalLimi…
Dec 7, 2019
b034581
fix balance limit's bug
Dec 7, 2019
3b71e69
1 in Exec(), add transaction begin, cmd args num, cmd func, reply fun…
Dec 11, 2019
0fb99b7
simplify the msg recv/send log in debug level
Dec 11, 2019
37c1c68
fix bug of setting cmd arg num metric in Exec()
Dec 11, 2019
9de2ae9
add expire/gc round/seek/commit cost metrics
Dec 12, 2019
6c2328d
prevent connection more and more: when tikv command cost high, client…
Dec 22, 2019
bbfce19
fix bug of connection limit
Dec 25, 2019
93cd7df
1 add metrics range, fix cost exceed max value problem
Feb 18, 2020
7e2521b
Merge branch 'fpxu_dev' into 'master'
Feb 18, 2020
ad7cff1
1 add example for limit-connection and max-connection-wait in titan.toml
Feb 19, 2020
aeca4b3
1 fix unlock bug in limit-connection
Feb 19, 2020
bb617b0
don't lock s.serverctx.lock when limit-connection disabled
Feb 19, 2020
003e6b6
expire seek end in at:now keys to reduce rocksdb tomstone problem
Feb 25, 2020
4423eaf
fix a failed case in tests/redis/unit/expire.tcl
Feb 26, 2020
cc15799
expire seek start in last processed expireKeys to reduce rocksdb toms…
Feb 27, 2020
d9a1baf
fix expire log
Feb 27, 2020
83a370d
also reduce expire seek range when no expire keys or all expire keys …
Feb 27, 2020
d75dd34
also reduce expire seek range when no expire keys or all expire keys …
Feb 28, 2020
3519ff9
comment unsupported string functions in command list, and delete its …
Mar 2, 2020
c4eea83
comment unit test of unsupported string functions in strings_test.go
Mar 2, 2020
bd43ec2
hash expire-key, it will improve the keys num handled every seconds, …
Apr 21, 2020
fe6e84b
1 fix runExpire's problem that unhashed goroutine scan hashed expire-key
Apr 24, 2020
977b457
unhash expire goroutine use individual limit configuration item、expir…
Apr 26, 2020
9161d7e
buckets of comand/limit/commandFunc/txn commit cost is too many to l…
Apr 26, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
don't lock s.serverctx.lock when limit-connection disabled
xufangping committed Feb 19, 2020

Verified

This commit was signed with the committer’s verified signature.
To-om toom
commit bb617b060c50005f6c3ca6af3669578a48fefddb
40 changes: 22 additions & 18 deletions titan.go
Original file line number Diff line number Diff line change
@@ -36,30 +36,32 @@ func (s *Server) Serve(lis net.Listener) error {
}

cliCtx := context.NewClientContext(s.idgen(), conn)
connectExceed := false
if s.servCtx.LimitConnection {
connectExceed := false
s.servCtx.Lock.Lock()
if s.servCtx.ClientsNum >= s.servCtx.MaxConnection {
connectExceed = true
}
s.servCtx.Lock.Unlock()
}
if connectExceed {
zap.L().Warn("max connection exceed, will close after some time",
zap.Int64("max connection num", s.servCtx.MaxConnection), zap.Int64("wait ms", s.servCtx.MaxConnectionWait),
zap.String("addr", cliCtx.RemoteAddr), zap.Int64("clientid", cliCtx.ID))
go func() {
time.Sleep(time.Duration(s.servCtx.MaxConnectionWait) * time.Millisecond)
zap.L().Warn("close connection for max connection exceed", zap.String("addr", cliCtx.RemoteAddr), zap.Int64("clientid", cliCtx.ID))
conn.Close()
}()
continue
if connectExceed {
zap.L().Warn("max connection exceed, will close after some time",
zap.Int64("max connection num", s.servCtx.MaxConnection), zap.Int64("wait ms", s.servCtx.MaxConnectionWait),
zap.String("addr", cliCtx.RemoteAddr), zap.Int64("clientid", cliCtx.ID))
go func() {
time.Sleep(time.Duration(s.servCtx.MaxConnectionWait) * time.Millisecond)
zap.L().Warn("close connection for max connection exceed", zap.String("addr", cliCtx.RemoteAddr), zap.Int64("clientid", cliCtx.ID))
conn.Close()
}()
continue
}
}
cliCtx.DB = s.servCtx.Store.DB(cliCtx.Namespace, 0)
s.servCtx.Clients.Store(cliCtx.ID, cliCtx)
s.servCtx.Lock.Lock()
s.servCtx.ClientsNum++
s.servCtx.Lock.Unlock()
if s.servCtx.LimitConnection {
s.servCtx.Lock.Lock()
s.servCtx.ClientsNum++
s.servCtx.Lock.Unlock()
}

cli := newClient(cliCtx, s, command.NewExecutor())

@@ -74,9 +76,11 @@ func (s *Server) Serve(lis net.Listener) error {
}
metrics.GetMetrics().ConnectionOnlineGaugeVec.WithLabelValues(cli.cliCtx.Namespace).Dec()
s.servCtx.Clients.Delete(cli.cliCtx.ID)
s.servCtx.Lock.Lock()
s.servCtx.ClientsNum--
s.servCtx.Lock.Unlock()
if s.servCtx.LimitConnection {
s.servCtx.Lock.Lock()
s.servCtx.ClientsNum--
s.servCtx.Lock.Unlock()
}
}(cli, conn)
}
}