grpcbox-ct: Stress test grpcbox

[vasu-dasari]

Added a stress_test to stress number of concurrent gRPC sessions that can be run. BY default it is set to 10. But can be modified by environment variable GRPCBOX_STRESS_TEST

# To run 25 concurrent sessions:
$ GRPCBOX_STRESS_TEST=25 rebar3 ct --suite grpcbox_SUITE --case stress_test

On my system, it above command breaks around 30-35.

Signed-off-by: Vasu Dasari [email protected]

[tsloughter]

Thanks, I was wanting to add something for this that utilized an existing tool used by other grpc servers to stress test.

[vasu-dasari]

I am using grpcbox for my project. I have a gRPC server and multiple gRPC clients(~10-15) and they use bidirectional streams for communication. When all clients initiate bidirectional transfer at the same time, I see that some calls are not successful. I am still debugging that issue. Meanwhile thought having some way of performing stress test in grpcbox's ct is a good idea as well. And hence this PR.

As I mentioned in my commit, once concurrent sessions go beyond 30-35, we can see failures. It would be interesting to see if there are any parameters I could fine tune to bump that number or if there are any blockades in dependencies, etc.

[tsloughter]

Do you know what the error is? Maybe it is related to joedevivo/chatterbox#136 which I still haven't gotten around to fixing :(

[vasu-dasari]

Actually, I am debugging around h2_connection side of things. In my case, some messages/connections are getting dropped. I need to characterize this completely.

Thanks.

[vasu-dasari]

@tsloughter Revisiting this issue after a long time.

Here is my theory. There is a single default_channel between grpc-client application and a grpc-server. But there are multiple processes within client application(C1,C2,C3) trying to make gRPC calls to the server. I see that default_channel is a single h2 connection and hence single TCP connection. As number of processes initiating gRPC calls increases, there is a possibility that one process's (say C1) call might step on another process's call(say C2). I believe this is causing the test case failure. I am arriving at this conclusion after matching h2_connection's receive data and the data that is processed in h2_stream. Most of the time it matches but sometimes it fails.

           +-----------+
           |   server  |
           +-----+-----+
                 |
                 |
                 |
           +-----+-----+
   +-------+   client  +-------+
   |       +------+----+       |
   |              |            |
   |              |            |
+--+-+         +--+-+        +-+--+
| C1 |         | C2 |        | C3 |
+----+         +----+        +----+

Here is what I am thinkinking how to fix the issue:

1. Without any code changes to grpcbox, have client application specify maximum number or channels it would like to use. In this case, it would be 3 channels looking exactly same but with different names, channel_1, channel_2, etc. And whenever making a unary/stream call, have the caller specify which channel to use for the call.
2. Modify grpcbox to be able to add API like, grpcbox_channel:add_channel and delete_channel so that application need not know before hand how many gRPC clients it might be needing before hand, and have it manager channel contexts. This model also helps to specify client gRPC channels during runtume

I am inclining towards 2, let me know what you think as it is a bit cleaner.

I will have a pretty of this and will create a separate pull request.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 38.99%. Comparing base (0166760) to head (c5a5d37).
Report is 71 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #29   +/-   ##
=======================================
  Coverage   38.99%   38.99%           
=======================================
  Files          28       28           
  Lines        2090     2090           
=======================================
  Hits          815      815           
  Misses       1275     1275           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.