Add option to add jitter to interval #173
Conversation
| @@ -123,8 +123,10 @@ func New(client kubernetes.Interface, labels, annotations, namespaces, namespace | |||
|
|
|||
| // Run continuously picks and terminates a victim pod at a given interval | |||
| // described by channel next. It returns when the given context is canceled. | |||
| func (c *Chaoskube) Run(ctx context.Context, next <-chan time.Time) { | |||
| func (c *Chaoskube) Run(ctx context.Context, maxJitter time.Duration, next <-chan time.Time) { | |||
There was a problem hiding this comment.
It would be nice if we only had one "tick source". A Ticker with a channel and some jitter inside would be nice Something like https://github.com/lthibault/jitterbug maybe.
| for { | ||
| jitter := util.RandomJitter(maxJitter) | ||
| time.Sleep(jitter) |
There was a problem hiding this comment.
If we go with two sources of delay (next channel and jitter duration, see above comment) we shouldn't use sleep here because it blocks handling Ctrl+c etc.
Instead we can do another select:
select {
case <-time.After(jitter):
case <-ctx.Done():
return
}after the select that's already below.
There was a problem hiding this comment.
I just noticed that this jitter will only increase the interval, never reduce it, right?
So, an interval of 1m and jitter of 10s will result in intervals between [1m, 1m10s]? (As opposed to [55m,1m5s])
There was a problem hiding this comment.
That's correct, I'm going to look at the jitterbug to see if that would fix most of the issues found here
There was a problem hiding this comment.
@linki so i think going with the jitterbug's normal distribution is what would fit well here, using a mean of 0 and standard deviation provided by the user, thoughts?
There was a problem hiding this comment.
@desponda Sgtm. Thanks for looking into it!
It would be useful to add more randomness to the chaos by adding jitter to the interval, this PR adds a random jitter based on maxJitter setting and defaults it to 0 for no jitter as default behavior.
It also checks that the max jitter is less than the interval to prevent the interval from being surpassed by the jitter
Fixes #172