Surface rate limit errors

[nplasterer]

How to catch rate limit errors in android. Catch the 429 thrown by the server when rate limited.

[nplasterer]

@fabriguespe could you figure out how to reproduce rate limiting issues to see how were this is getting surface or if it's not getting surface 🙏

[humanagent]

@nplasterer I've been running some tests on how the Android SDK handles rate limiting, specifically when the server throws a 429 status code due to too many requests. Current limits:

• 1,000 publish requests per 5 minutes
• 10,000 general requests per 5 minutes

The rate limiting does not appear to be triggered as expected. I was able to send more than 1,000 messages in a 5-minute window without encountering any error.

I've attached the test file for reference. The tests include:

• testClient: Checks if the client is created successfully and can send a message
• testRateLimitingPublishing: Checks if rate limiting is working when publishing messages
• testRateLimitingRequest: Checks if rate limiting is working when sending requests

package org.xmtp.android.library

import androidx.test.ext.junit.runners.AndroidJUnit4
import com.google.protobuf.kotlin.toByteString
import com.google.protobuf.kotlin.toByteStringUtf8
import org.junit.Assert
import org.junit.Assert.assertEquals
import org.junit.Ignore
import org.junit.Test
import org.junit.runner.RunWith
import org.web3j.crypto.Hash
import org.xmtp.android.library.XMTPEnvironment
import org.web3j.utils.Numeric
import org.xmtp.android.library.codecs.TextCodec
import org.xmtp.android.library.messages.InvitationV1
import org.xmtp.android.library.messages.InvitationV1ContextBuilder
import org.xmtp.android.library.messages.MessageV1Builder
import org.xmtp.android.library.messages.MessageV2Builder
import org.xmtp.android.library.messages.PrivateKeyBuilder
import org.xmtp.android.library.messages.PrivateKeyBundle
import org.xmtp.android.library.messages.PrivateKeyBundleV1
import org.xmtp.android.library.messages.PublicKeyBundle
import org.xmtp.android.library.messages.SealedInvitationBuilder
import org.xmtp.android.library.messages.SignedPublicKeyBundleBuilder
import org.xmtp.android.library.messages.createDeterministic
import org.xmtp.android.library.messages.decrypt
import org.xmtp.android.library.messages.generate
import org.xmtp.android.library.messages.getPublicKeyBundle
import org.xmtp.android.library.messages.recipientAddress
import org.xmtp.android.library.messages.senderAddress
import org.xmtp.android.library.messages.sharedSecret
import org.xmtp.android.library.messages.toPublicKeyBundle
import org.xmtp.android.library.messages.toV2
import org.xmtp.android.library.messages.walletAddress
import org.xmtp.proto.message.api.v1.MessageApiOuterClass
import org.xmtp.proto.message.contents.Invitation
import org.xmtp.proto.message.contents.Invitation.InvitationV1.Context
import org.xmtp.proto.message.contents.PrivateKeyOuterClass
import org.xmtp.android.library.messages.PrivateKey
import java.nio.charset.StandardCharsets.UTF_8
import java.util.Date
import kotlinx.coroutines.runBlocking
import java.text.SimpleDateFormat
import java.time.Instant
import java.util.Locale
import io.grpc.StatusException
import java.util.TimeZone
import android.util.Log


class RateLimitTest {
    /*
    XMTP clients are subject to two rate limits:
    - 1,000 publish requests per 5 minutes.
    - 10,000 general requests per 5 minutes.
    */

    private val DESTINATION_WALLET = "0x0AD3A479B31072bc14bDE6AaD601e4cbF13e78a8"

    private fun createWalletClient(): Client {
        val ints = arrayOf(
            225, 2, 36, 98, 37, 243, 68, 234,
            42, 126, 248, 246, 126, 83, 186, 197,
            204, 186, 19, 173, 51, 0, 64, 0,
            155, 8, 249, 247, 163, 185, 124, 159,
        )
        val keyBytes =
            ints.foldIndexed(ByteArray(ints.size)) { i, a, v -> a.apply { set(i, v.toByte()) } }

        val key = PrivateKeyOuterClass.PrivateKey.newBuilder().also {
            it.secp256K1 = it.secp256K1.toBuilder().also { builder ->
                builder.bytes = keyBytes.toByteString()
            }.build()
            it.publicKey = it.publicKey.toBuilder().also { builder ->
                builder.secp256K1Uncompressed =
                    builder.secp256K1Uncompressed.toBuilder().also { keyBuilder ->
                        keyBuilder.bytes =
                            KeyUtil.addUncompressedByte(KeyUtil.getPublicKey(keyBytes))
                                .toByteString()
                    }.build()
            }.build()
        }.build()
        val CLIENT_OPTIONS = ClientOptions(api = ClientOptions.Api(XMTPEnvironment.PRODUCTION, appVersion = "XMTPAndroidExample/v1.0.0"))
        val client = Client().create( account = PrivateKeyBuilder(key) ,options = CLIENT_OPTIONS)
        return client   
    }

    @Test
    fun testClient() {
        // This test checks if the client is created successfully and can send a message
        val client = createWalletClient()
        val aliceConversation = client.conversations.newConversation(DESTINATION_WALLET)
        assertEquals(client.apiClient.environment, XMTPEnvironment.PRODUCTION)
        val message = aliceConversation.send(text = "Test message")

    }


    
    @Test
    fun testRateLimitingPublishing() = runBlocking {
        // This test checks if rate limiting is working when publishing messages
    
        val client = createWalletClient()
        val aliceConversation = client.conversations.newConversation(DESTINATION_WALLET)
        
        assertEquals(client.apiClient.environment, XMTPEnvironment.PRODUCTION)
        
        val startTime = System.currentTimeMillis()
        repeat(2000) {
            try {
                val timePassedMinutes = String.format("%.2f", (System.currentTimeMillis() - startTime) / 60000.0).toDouble()
                val message = aliceConversation.send(text = "Test message ${it + 1}")
                Log.d("RateLimitTest", "Sending message number ${it + 1} at ${timePassedMinutes} minutes")
            } catch (error: StatusException) {
                val timePassedMinutes = (System.currentTimeMillis() - startTime) / 60000.0
                Log.d("RateLimitTest", "Caught exception: ${error.status.code} at ${timePassedMinutes} minutes")
                if (error.status.code == io.grpc.Status.Code.UNAVAILABLE || error.status.code == io.grpc.Status.Code.RESOURCE_EXHAUSTED) {
                    Log.d("RateLimitTest", "Rate limit reached at ${timePassedMinutes} minutes")
                    assert(true)
                    return@runBlocking
                }
            }
        }
        val timePassedMinutes = (System.currentTimeMillis() - startTime) / 60000.0
        Log.d("RateLimitTest", "Rate limit not reached at ${timePassedMinutes} minutes")
        assert(false)
    }

    @Test
    fun testRateLimitingRequest() = runBlocking {
        // This test checks if rate limiting is working when sending requests
    
        val client = createWalletClient()
        assertEquals(client.apiClient.environment, XMTPEnvironment.PRODUCTION)
        
        val dateFormat = SimpleDateFormat("HH:mm:ss.SSS", Locale.getDefault())
        repeat(12000) {
            try {
                val currentTime = dateFormat.format(Date())
                val canMessage = client.canMessage("0x0AD3A479B31072bc14bDE6AaD601e4cbF13e78a8")
                Log.d("RateLimitTest", "Sending request number ${it + 1}:${canMessage} at $currentTime")
      
            } catch (error: StatusException) {
                val currentTime = dateFormat.format(Date())
                Log.d("RateLimitTest", "Caught exception: ${error.status.code} at $currentTime")
                if (error.status.code == io.grpc.Status.Code.UNAVAILABLE) {
                    Log.d("RateLimitTest", "Rate limit reached at $currentTime")
                    assert(true)
                    return@runBlocking
                }
            }
        }
        val currentTime = dateFormat.format(Date())
        Log.d("RateLimitTest", "Rate limit not reached at $currentTime")
        assert(false)
    }
    
}

[nplasterer]

I think it's 1k publishes per minute? @neekolas can you confirm. This should be coming from the network so it's not possible for android to bypass. I think it just means you're not actually hitting the limit in the window 🤔

[neekolas]

It is 1K publishes/5 minutes. There is some fuzziness that may allow you to exceed it (load balancer has some lag for cutting you off, requests may hit multiple nodes with their own limits). I would think of the 1K number as a guideline more than a hard limit.

[humanagent]

I am able to get the error from rate limiting after a couple of runs

2024-01-26 18:20:52.370 21630-21664 RateLimitTest:1         org.xmtp.android.library.test        D  Caught exception: RESOURCE_EXHAUSTED at 0.1169 minutes
2024-01-26 18:20:52.371 21630-21664 RateLimitTest:1         org.xmtp.android.library.test        D  Rate limit reached at 0.1169 minutes

io.grpc.StatusException: RESOURCE_EXHAUSTED: 2 exceeds rate limit R190.196.225.21PUB
at io.grpc.Status.asException(Status.java:554)
at io.grpc.kotlin.ClientCalls$rpcImpl$1$1$1.onClose(ClientCalls.kt:296)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
at java.lang.Thread.run(Thread.java:1012)