Skip to content

Commit

Permalink
cleanup
Browse files Browse the repository at this point in the history
  • Loading branch information
@amitksingh1490
amitksingh1490 committed Nov 23, 2024
1 parent c10d18d commit 90f6125
Show file tree
Hide file tree
Showing 34 changed files with 6,352 additions and 900 deletions.
20 changes: 10 additions & 10 deletions .github/workflows/bench.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,15 +121,15 @@ jobs:
rps: ${{ steps.extract_rps.outputs.rps }}
p95: ${{ steps.extract_p95.outputs.p95 }}
read:
runs-on: ubuntu-latest
needs: [benchmark]
steps:
- uses: cloudposse/github-action-matrix-outputs-read@v1
id: read
with:
matrix-step-name: benchmark
outputs:
result: "${{ steps.read.outputs.result }}"
runs-on: ubuntu-latest
needs: [benchmark]
steps:
- uses: cloudposse/github-action-matrix-outputs-read@v1
id: read
with:
matrix-step-name: benchmark
outputs:
result: "${{ steps.read.outputs.result }}"
analyze:
if: github.event.head_commit.message != 'Update performance results in README.md'
needs: read
Expand Down Expand Up @@ -396,4 +396,4 @@ jobs:
with:
branch: main
commit_author: Author <[email protected]>
commit_message: "[ci skip] update performance results in README.md"
commit_message: "[ci skip] update performance results in README.md"
3 changes: 1 addition & 2 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,5 @@ RUN curl -fsSL https://grafbase.com/downloads/gateway | bash && \
# Setup Tailcall
RUN npm install -g @tailcallhq/tailcall

# Copy Configurations and Scripts
COPY configurations/* .
COPY configurations/ ./configurations/
COPY scripts/* .
79 changes: 43 additions & 36 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,45 +18,52 @@ Explore and compare the performance of the fastest GraphQL federation routers th
## Benchmark Results

<!-- PERFORMANCE_RESULTS_START -->

### [Small Payload - 362 bytes](./source/small.json)
| Server | Status | RPS | Latency |
| ---: | ---: | ---: | ---: |
| [Nginx](https://nginx.org/en/) | ✅ | `5,071 RPS` | `0.0197 sec` |
| **Base** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `7,937 RPS` | `0.0122 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `4,996 RPS` | `0.0145 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `2,661 RPS` | `0.0392 sec` |
| [Apollo](https://github.com/apollographql/router) | ✅ | `2,633 RPS` | `0.0263 sec` |
| **Cached** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `16,470 RPS` | `0.01 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `6,746 RPS` | `0.0142 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `5,403 RPS` | `0.0227 sec` |

| Server | Status | RPS | Latency |
| -------------------------------------------------: | -----: | -----------: | -----------: |
| [Nginx](https://nginx.org/en/) | ✅ | `5,071 RPS` | `0.0197 sec` |
| **Base** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `7,937 RPS` | `0.0122 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `4,996 RPS` | `0.0145 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `2,661 RPS` | `0.0392 sec` |
| [Apollo](https://github.com/apollographql/router) | ✅ | `2,633 RPS` | `0.0263 sec` |
| **Cached** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `16,470 RPS` | `0.01 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `6,746 RPS` | `0.0142 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `5,403 RPS` | `0.0227 sec` |

### [Medium Payload - 12,598 bytes](./source/medium.json)
| Server | Status | RPS | Latency |
| ---: | ---: | ---: | ---: |
| [Nginx](https://nginx.org/en/) | ✅ | `4,578 RPS` | `0.0205 sec` |
| **Base** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `4,994 RPS` | `0.0141 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `3,937 RPS` | `0.0174 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `1,412 RPS` | `0.0697 sec` |
| [Apollo](https://github.com/apollographql/router) | ✅ | `1,780 RPS` | `0.0495 sec` |
| **Cached** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `13,195 RPS` | `0.0073 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `5,056 RPS` | `0.018 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `2,800 RPS` | `0.0422 sec` |

| Server | Status | RPS | Latency |
| -------------------------------------------------: | -----: | -----------: | -----------: |
| [Nginx](https://nginx.org/en/) | ✅ | `4,578 RPS` | `0.0205 sec` |
| **Base** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `4,994 RPS` | `0.0141 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `3,937 RPS` | `0.0174 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `1,412 RPS` | `0.0697 sec` |
| [Apollo](https://github.com/apollographql/router) | ✅ | `1,780 RPS` | `0.0495 sec` |
| **Cached** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `13,195 RPS` | `0.0073 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `5,056 RPS` | `0.018 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `2,800 RPS` | `0.0422 sec` |

### [Big Payload - 112,838 bytes](./source/big.json)
| Server | Status | RPS | Latency |
| ---: | ---: | ---: | ---: |
| [Nginx](https://nginx.org/en/) | ✅ | `2,105 RPS` | `0.0395 sec` |
| **Base** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `1,824 RPS` | `0.037 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `2,024 RPS` | `0.0332 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `778 RPS` | `0.1116 sec` |
| [Apollo](https://github.com/apollographql/router) | ✅ | `1,194 RPS` | `0.0603 sec` |
| **Cached** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `7,636 RPS` | `0.0113 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `2,467 RPS` | `0.0317 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `983 RPS` | `0.1104 sec` |

| Server | Status | RPS | Latency |
| -------------------------------------------------: | -----: | ----------: | -----------: |
| [Nginx](https://nginx.org/en/) | ✅ | `2,105 RPS` | `0.0395 sec` |
| **Base** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `1,824 RPS` | `0.037 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `2,024 RPS` | `0.0332 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `778 RPS` | `0.1116 sec` |
| [Apollo](https://github.com/apollographql/router) | ✅ | `1,194 RPS` | `0.0603 sec` |
| **Cached** | | | |
| [Tailcall](https://github.com/tailcallhq/tailcall) | ✅ | `7,636 RPS` | `0.0113 sec` |
| [Grafbase](https://github.com/grafbase/grafbase) | ✅ | `2,467 RPS` | `0.0317 sec` |
| [Wundegraph](https://github.com/wundergraph/cosmo) | ✅ | `983 RPS` | `0.1104 sec` |

<!-- PERFORMANCE_RESULTS_END -->

### RPS Plots
Expand Down
14 changes: 0 additions & 14 deletions configurations/2-http-tweaks.graphql
View file

This file was deleted.

15 changes: 0 additions & 15 deletions configurations/3-http-cache.graphql
View file

This file was deleted.

14 changes: 0 additions & 14 deletions configurations/4-http-cache-directive.graphql
View file

This file was deleted.

14 changes: 0 additions & 14 deletions configurations/5-dedupe.graphql
View file

This file was deleted.

67 changes: 0 additions & 67 deletions configurations/employees-cache.graphql
View file

This file was deleted.

66 changes: 0 additions & 66 deletions configurations/employees-dedupe.graphql
View file

This file was deleted.

0 configurations/grafbase_1_basic.toml → ...igurations/grafbase/grafbase_1_basic.toml
View file
File renamed without changes.
0 configurations/grafbase_2_optimized.toml → ...ations/grafbase/grafbase_2_optimized.toml
View file
File renamed without changes.
Loading

1 comment on commit 90f6125

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GraphQL Federation Benchmarks

Explore and compare the performance of the fastest GraphQL federation routers through our comprehensive benchmarks.

Benchmark Results

Small Payload - 362 bytes

Server Status RPS Latency
Nginx ✅ 5,017 RPS 0.0233 sec
Base
Tailcall ✅ 7,395 RPS 0.0094 sec
Grafbase ✅ 4,954 RPS 0.0148 sec
Wundegraph ✅ 2,585 RPS 0.0382 sec
Apollo ✅ 2,184 RPS 0.0396 sec
Cached
Tailcall ✅ 16,148 RPS 0.0103 sec
Grafbase ✅ 7,621 RPS 0.0107 sec
Wundegraph ✅ 4,656 RPS 0.0271 sec

Medium Payload - 12,598 bytes

Server Status RPS Latency
Nginx ✅ 4,694 RPS 0.0226 sec
Base
Tailcall ✅ 4,949 RPS 0.0142 sec
Grafbase ✅ 3,988 RPS 0.0172 sec
Wundegraph ✅ 1,362 RPS 0.0752 sec
Apollo ✅ 1,644 RPS 0.05 sec
Cached
Tailcall ✅ 12,792 RPS 0.0077 sec
Grafbase ✅ 5,146 RPS 0.0176 sec
Wundegraph ✅ 2,483 RPS 0.0477 sec

Big Payload - 112,838 bytes

Server Status RPS Latency
Nginx ✅ 2,094 RPS 0.0392 sec
Base
Tailcall ✅ 1,818 RPS 0.0373 sec
Grafbase ✅ 1,851 RPS 0.0477 sec
Wundegraph ✅ 682 RPS 0.1447 sec
Apollo ✅ 1,242 RPS 0.0563 sec
Cached
Tailcall ✅ 7,772 RPS 0.0112 sec
Grafbase ✅ 2,124 RPS 0.0409 sec
Wundegraph ✅ 1,105 RPS 0.0948 sec

RPS Plots

Requests Per Second: small payload

Requests Per Second: medium payload

Requests Per Second: big payload

Latency 95% Plots

Latency 95%: small payload

Latency 95%: medium payload

Latency 95%: big payload

Architecture

Architecture Image

Components

  1. hey: We use hey cli benchmarking tool to cause synthetic load to benchmark the different router implementations. We benchmark for '10 secondsusing200 connections`. We constructed three different request payload configurations: big, medium, small. Each configuration queries a response of payload size of 112,838 bytes, 12598 bytes, and 362 bytes respectively.
  2. Implementations: We use a collection of different federation implementations, and for each of them, we also have different configuration setups located in the configurations folder. We benchmark each implementation with varying configurations for every data configuration setup (big, medium, small).
  3. Mock: This component provides data to the implementations. It mocks a GraphQL subgraph and an equivalent Rest API. This component is written in Rust and serves static data. We do that to eliminate any overheads caused by processing the request in a real GraphQL subgraph.

Specifications

Specifications of the machine used for benchmarking:

Platform: Linux x64
Size:
2-cores · 8 GB RAM · 75 GB SSD

Quick Start

  1. Clone the repository

    git clone [email protected]:tailcallhq/federation-example.git
cd federation-example

  2. Ensure you have the latest version of Docker

  3. Run the following command:

    docker build -t tailcallhq/federation-benchmark .
docker run tailcallhq/federation-benchmark:latest ./benchmark_all.sh

  4. Wait for benchmarks to complete.

Resources

  • Docker: Docker is a set of platform-as-a-service products that use OS-level virtualization to deliver software in packages called containers.
  • Hey: hey is a tiny program that sends some load to a web application.
  • Rust: Rust is a general-purpose programming language emphasizing performance, type safety, and concurrency. It enforces memory safety, meaning that all references point to valid memory.
  • GraphQL Federation: GraphQL Federation is an architecture that allows multiple independent GraphQL services to form a unified graph that appears as a single graph to clients. It is a powerful way to scale and manage microservices architecture when using GraphQL.

Please sign in to comment.