Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not replicating all the keys and also not showing any errors #165

Open
nikhilo opened this issue Dec 4, 2024 · 0 comments
Open

Not replicating all the keys and also not showing any errors #165

nikhilo opened this issue Dec 4, 2024 · 0 comments

Comments

@nikhilo
Copy link

nikhilo commented Dec 4, 2024

Hi,
I tried to create a simple setup on my laptop. 2 Redis servers, one on port 6379 and another on port 6380. Then I set up live replication from on server to the other using the riot docker container. After that I inserted 500000 keys using the command cat set.txt | redis-cli --pipe. I'm not sure why riot processed only about 16000 keys.

Scanning/Listening 100% [=====================================================================================================] 16551/16551 (0:00:12 / 0:00:00) 1379.2/s

Source DB

127.0.0.1:6379> dbsize
(integer) 500000
127.0.0.1:6379> get test_20034
"value_20034"

Destination DB

127.0.0.1:6380> dbsize
(integer) 16551
127.0.0.1:6380> get test_20034
(nil)

Here are the complete logs from the riot container

 ~  docker run -it --rm --name riot --net host riotx/riot replicate --mode live redis://localhost:6379 redis://localhost:6380 --info    554ms  Wed Dec  4 13:23:11 2024
[main] INFO com.redis.riot.Replicate - Creating target Redis context with redis://localhost:6380 TargetRedisArgs(username=null, timeout=PT1M, tls=false, insecure=false, clientName=null, cluster=false, protocolVersion=RESP2, poolSize=8, readFrom=UPSTREAM) SslArgs(keystore=null, truststore=null, keyCert=null, key=null, trustedCerts=null)
[main] INFO com.redis.riot.Replicate - Creating source Redis context with redis://localhost SourceRedisArgs(username=null, timeout=PT1M, tls=false, insecure=false, clientName=null, cluster=false, protocolVersion=RESP2, poolSize=8, readFrom=UPSTREAM) SslArgs(keystore=null, truststore=null, keyCert=null, key=null, trustedCerts=null)
[main] INFO com.redis.riot.Replicate - Creating Redis dump reader
[main] INFO com.redis.riot.Replicate - Configuring RedisItemReader with RedisReaderArgs(mode=LIVE, keyPattern=null, keyType=null, scanCount=1000, queueCapacity=10000, threads=1, chunkSize=50, memUsageLimit=null, memUsageSamples=5, flushInterval=50, idleTimeout=0, eventQueueCapacity=10000, retryLimit=0, skipLimit=0, keyFilterArgs=KeyFilterArgs(includes=null, excludes=null, slots=null), pollTimeout=100)
[main] INFO com.redis.riot.Replicate - Creating Redis dump writer
[main] INFO com.redis.riot.Replicate - Configuring target Redis writer with RedisWriterArgs(multiExec=false, waitReplicas=0, waitTimeout=PT1S, merge=false)
[main] INFO com.redis.riot.Replicate - Configuring step with scan size estimator
[main] INFO com.redis.riot.Replicate - Retrieved config notify-keyspace-events: AKE
[main] INFO com.redis.riot.Replicate - Configuring export step with live true, flushInterval PT0.05S, idleTimeout PT2562047788015H12M55.807S
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Creating Redis quick compare reader
[main] INFO com.redis.riot.Replicate - Configuring RedisItemReader with RedisReaderArgs(mode=LIVE, keyPattern=null, keyType=null, scanCount=1000, queueCapacity=10000, threads=1, chunkSize=50, memUsageLimit=null, memUsageSamples=5, flushInterval=50, idleTimeout=0, eventQueueCapacity=10000, retryLimit=0, skipLimit=0, keyFilterArgs=KeyFilterArgs(includes=null, excludes=null, slots=null), pollTimeout=100)
[main] INFO com.redis.riot.Replicate - Creating Redis quick compare reader
[main] INFO com.redis.riot.Replicate - Creating KeyComparator with ttlTolerance=PT0.1S ignoreStreamMessageId=false
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Creating Step(name=step, reader=com.redis.spring.batch.item.redis.RedisItemReader@311ff287, writer=com.redis.spring.batch.item.redis.RedisItemWriter@7377781e, taskName=Scanning/Listening, statusMessageSupplier=com.redis.riot.core.Step$$Lambda/0x0000002001326758@31db34da, maxItemCountSupplier=com.redis.spring.batch.item.redis.reader.RedisScanSizeEstimator@109f8c7e, processor=com.redis.riot.KeyValueFilter@5edacf20, executionListeners=[], readListeners=[], writeListeners=[], live=true, flushInterval=PT0.05S, idleTimeout=PT2562047788015H12M55.807S, skip=[], noSkip=[], retry=[], noRetry=[])
[main] INFO com.redis.riot.Replicate - Creating step replicate-step with chunk size 50
[main] INFO com.redis.riot.Replicate - Creating flushing step with flush interval PT0.05S and idle timeout PT2562047788015H12M55.807S
[main] INFO com.redis.riot.Replicate - Skipping fault-tolerance for step step
[main] INFO com.redis.riot.Replicate - Creating Step(name=compare, reader=com.redis.spring.batch.item.redis.reader.KeyComparisonItemReader@d3f4505, writer=com.redis.riot.CompareStatusItemWriter@3eb292cd, taskName=Comparing, statusMessageSupplier=com.redis.riot.AbstractReplicateCommand$$Lambda/0x0000002001341560@7fd987ef, maxItemCountSupplier=com.redis.spring.batch.item.redis.reader.RedisScanSizeEstimator@7209ffb5, processor=null, executionListeners=[com.redis.riot.CompareStepListener@7e9f2c32], readListeners=[], writeListeners=[], live=false, flushInterval=PT0.05S, idleTimeout=PT2562047788015H12M55.807S, skip=[], noSkip=[], retry=[], noRetry=[])
[main] INFO com.redis.riot.Replicate - Creating step replicate-compare with chunk size 50
[main] INFO com.redis.riot.Replicate - Skipping fault-tolerance for step compare
[main] INFO org.springframework.batch.core.launch.support.SimpleJobLauncher - Job: [SimpleJob: [name=replicate]] launched with the following parameters: [{}]
[main] INFO org.springframework.batch.core.job.SimpleStepHandler - Executing step: [replicate-step]
[main] INFO com.redis.spring.batch.item.redis.RedisItemReader - Creating queue with capacity 10,000
Scanning/Listening  ? % [] 0/0 (0:00:00 / ?) ?/s
[SimpleAsyncTaskExecutor-1] INFO org.springframework.batch.core.launch.support.SimpleJobLauncher - Job: [SimpleJob: [name=replicate-step-reader]] launched with the following parameters: [{}]
[SimpleAsyncTaskExecutor-1] INFO org.springframework.batch.core.job.SimpleStepHandler - Executing step: [replicate-step-reader]
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.KeyScanEventItemReader - Creating queue with capacity 10,000
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.RedisPubSubHandler - Establishing pub/sub connection to Redis
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.RedisPubSubHandler - Subscribing to pattern __keyevent@0__:*
Scanning/Listening 100% [=====================================================================================================] 16551/16551 (0:00:12 / 0:00:00) 1379.2/s



I even tried setting `--threads 8 --read-threads 2` (my laptop has 8 cores)
Still it could replicate only 22k keys with increased threads

Scanning/Listening 100% [=====================================================================================================] 22350/22350 (0:00:10 / 0:00:00) 2235.0/s


Any idea what could be going on here? It's not replicating all the keys and also not reporting any errors. 🤷🏽‍♂️ 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant