Why Microservices Need Webhook Gateways
A Webhook Gateway serves as a forward and reverses proxy to send and receive webhooks to client endpoints and from third-party providers. They are designed to enforce security, resilience and high availability on all webhook events for an organisation. We wrote a more detailed description here.
The image above describes the flow of webhook events from the microservices to client endpoints as well as from third parties to the microservices. In this article, we will discuss why webhooks gateways are relevant to the microservices architecture, and what common features are found in the webhooks gateway. Finally, we will discuss some of the caveats to bear in mind while adopting a webhooks gateway.
The various advantages of a microservice architecture subsequently present unique challenges that Webhook Gateways are purpose-built to address.
Microservice architectures split up several functions of an application into separate services, that are owned by different teams. This separation is typically based on a common domain e.g. Payments service, Notifications service etc. This design pattern engenders each team to build, deploy and maintain their services independently. Unlike a monolith application, teams are able to increase velocity and deliver features faster. This approach means increased complexity for webhook management: How are we able to route webhook events from third parties to the right microservice(s)? How are we able to route webhook events from backend services to client endpoints?
This is where a webhooks gateway comes in. They’re able to provide a central infrastructure capable of ingesting, and routing webhooks both internally to backend services and externally to client endpoints. This central platform also reduces the learning curve for new teams to add webhooks to their services.
One of the major benefits of the microservice approach is the ability for teams to utilise the best tools for the job. For example, one team can decide to build their API using Rails, GraphQL and MongoDB, while another team can decide to build their API using Golang, gRPC and PostgreSQL.
Regardless of the technology stack chosen by each team. Webhook gateways rely on a consistent payload structure to send to client endpoints and an endpoint to receive from a third-party. As long as the payload format and structure remain consistent, developers can implement the business logic with any technology stack they prefer. While leaving the gateway to cross-cutting concerns like retries, rate limiting and signing the payload.
Services can be rewritten from Python to Rust and redeployed with the Gateway. To the client endpoints, the payload remains the same, and it can continue to rely on the payload format and structure to remain the same.
When webhooks are ingested into your systems from third-party providers, in a monolith all events are routed to the same application all the time. However, in a microservice architecture, one or more microservice(s) can rely on the event from the provider to carry out separate actions.
Webhook Gateways can be configured to receive events and route them to one or more destinations. They can match both headers and payload to determine webhook's destination.
Internet-facing services are common targets for malicious attacks and exploitation. They need to handle a sudden spike in requests from valid users and bad actors. This requires features like request throttling and blacklist capabilities to keep systems reliable and secure.
Webhook Gateways are an effective barrier against Distributed Denial of Service (DDoS) attacks—throttling the number of requests made to affected services, so the service isn’t overwhelmed and protecting it against becoming unresponsive.
A Webhook Gateway is located at the outer edge of your microservices and acts as a proxy to manage all ingress and egress webhook traffic. The Webhook Gateway handles several features:
Webhook Events sent to client endpoints need to be securely delivered to client endpoints. Clients need to validate that the message has not been tampered with in transit, and they need to know that events are not being re-transmitted by malicious actors, in some sensitive environments they need to authenticate the webhook provider and authorise their IP address through their network firewall.
When receiving events from third-party providers (e.g. Stripe, Twilio ), consumers need to verify the webhooks before routing them to backend services for processing
Webhooks Gateway is able to centralise cross-cutting concerns like webhooks security and securely receive events from backend services, sign the payload, timestamp the payload, and deliver them reliably with a static IP address. Since they exist at the outer edge of your microservices they are able to prevent exposing any of your backend services directly to the internet and reduce the attack surface area for malicious actors.
Managing HTTP Endpoints is one of the core responsibilities of a Webhook Gateway. Standard webhooks flow is a post request to an endpoint, and expect a 2xx status code in return, indicating success. However, these endpoints fail from time to time for several reasons: flaky networks, spikes in traffic, code or infrastructure changes that resulted in downtime, and degraded performance in the client’s application dependencies.
Due to the various types of issues that can prevent reliable webhooks delivery, webhook Gateways add important features like timeouts; to prevent waiting endlessly for responses, and retries; to automatically re-transmit webhook events, rate limiting; to deliver events to client endpoints at a steady pace the client can process, and circuit breaking; to prevent wasting system resources on zombie endpoints (i.e. endpoints that have failed consecutively over time)
Debugging webhooks delivery is an important aspect of the responsibilities of webhooks providers to clients. Best practice enables engineers to debug webhook delivery without human interaction with the engineering team of the provider.
Webhook Gateways provide such a utility to generate rich customer-facing webhooks dashboard where users can directly (zero human intervention) debug webhook delivery issues and fix failing endpoints with a few clicks.
Webhook Gateways need to be flexible and versatile. Microservices emit events through various channels. For example, building on Google Cloud, developers will often use Google PubSub to emit happening in one or more microservices, similar to AWS for Amazon SQS, and other tools like Kafka and RabbitMQ.
Plugging into all these options makes a webhook gateway such a versatile tool for streaming webhook events from multiple event-generating microservice to send the events to the client endpoints.
While there are several benefits to deploying a Webhooks gateway, there are some caveats to keep in mind when adopting a Webhooks gateway for your organisation, such as:
The webhooks gateway is a new software component in your environment. It would require learning and building expertise in it. Platform Engineers and SREs will need to integrate, deploy and maintain this new software component and build production excellence running the webhooks gateway.
By design, a webhooks gateway is responsible for webhooks traffic (inbound and outbound) of your organisation, This, therefore, becomes a single point of failure for the entire organisation. Webhooks gateway then needs to be designed for high availability to ensure no data loss for any webhook traffic.
In the end, webhook gateways aren’t one-size fits all. You can build a webhook gateway in-house or acquire an off-the-shelf solution like Convoy while analysing your system's unique requirements.
A Webhook Gateway can enhance your microservice-based application with capabilities like secure webhook delivery with signatures, static IPs, resilience with retries, rate limiting, and customer-facing webhook dashboards. However, before adding a webhooks gateway to your application architecture, you need to understand what problems you’re trying to address, what events you’ll be sending and if the product you are choosing offers those features.
Convoy is the fastest open-source webhooks gateway available today. With Convoy, you’ll be able to configure advanced event routing, incoming and outgoing webhooks, and customer-facing webhooks dashboard, and securely deliver events with payload signing and Static IPs.
If you're exploring solutions for your webhooks, we welcome you to try Convoy Cloud for free today.
If you feel like this article helped you understand Webhooks better! I would be super happy if you could give us a star! And let me also know in the comments ❤️ https://github.com/frain-dev/convoy
Thanks for reading!