rest

Flowing Retail / REST

This folder contains services that connect via REST. Currently, this is reduced to showcasing resilience patterns.

A good background read is this InfoWorld article 3 common pitfalls of microservices integration—and how to avoid them

Sample service demonstrating stateful resilience patterns in a REST environment

This sample REST (micro-)service effects payments in response to a PUT call. It requires an upstream REST service that charges credit cards.

This simple call-chain is perfect for demonstrating important resilience patterns.

The following technology choices are available for the code demos:

• Java + Spring, Hystrix, Camunda
• C# + Polly, Camunda
• Node.js + Brakes, Zeebe on Camunda Cloud

There is a stripped-down version available for:

• GoLang, Zeebe

Storyline

See Fail fast is not enough: https://blog.bernd-ruecker.com/fail-fast-is-not-enough-84645d6864d3

Let's assume a scenario where the upstream credit card service still responds, but its very slow. With no resilience pattern in place, this is the worst thing that can happen - as now the payment service will call the credit card service and block until it gets a response. As this take a long time, all threads from the payment service are held hostage, and the payment service will eventually time out for its clients. Tiny failures somewhere in your system might blow up your whole system:

• Java: PaymentRestHacksControllerV1.java
• C#: PaymentControllerV1
• Node.js: controller-v1.ts

Fail fast

A simple mitigation is to apply a fail fast pattern like circuit breaker. In this example I use Netflix Hystrix (and Polly for C# / Brakes for Node.js, which provide equivalent functionality). If a service responds too slowly, the circuit breaker interrupts and the payment service gets a failure right away. This way you make sure the overall system is still responding, even if functionality degrades (meaning: we cannot charge credit cards).

• Java: PaymentRestHacksControllerV2.java
• C#: PaymentControllerV2
• Node.js: controller-v2.ts

Fail fast is not enough

Failing fast is good, but it is not enough. Frequently, a retry after the credit card service has been fixed resolves the situation (if the service was in a hard failure mode) - or a retry may discover that the earlier attempt succeeded, but took an abnormal amount of time (if the service is in a degraded performance mode). This retry needs to be stateful to not only retry right away but again in a couple of minutes, hours or even days. Keeping this stateful retry local to the payment service reduces overall architectural complexity.

In the example, I use the Camunda workflow engine (or Zeebe in Camunda Cloud) to do the stateful retry reliably.

• Java
  • PaymentRestHacksControllerV3.java
  • The workflow is created by Java DSL
• C#
  • PaymentControllerV3
  • PaymentV3.bpmn
• Node.js
  • controller-v3.ts
  • paymentV3.bpmn

Keep synchronous responses

The processing just got asynchronous, which is often not wanted. In this scenario you could very well return a synchronous response whenever the credit card service is available, but switch to asynchronicity when it is not.

HTTP supports this via return codes: 200 OK means "all OK", 202 ACCEPTED means "I'll call you back later".

• Java
  • PaymentRestHacksControllerV4.java
  • The workflow is created by Java DSL
• C#
  • PaymentControllerV4.
  • PaymentV4.bpmn
• Nodejs
  • controller-v4.ts
  • paymentV4.bpmn

Asynchronous work distribution without messaging

An alternative to synchronously calling an upstream service is to communicate asynchronously. The default would be messaging.

This example shows a successful approach taken by many customers: using the workflow engine as work distribution, behaving like a queue. This leverages the External Tasks pattern.

• Java
  • Workflow model: payment5.bpmn (hint: use the free Camunda Modeler to show this model graphically)
  • Worker Node.js: index.js
  • Worker in Java (alternative): CustomerCreditWorker.java
• C# (skipped V5 and directly implemented V6)
  • Workflow model:PaymentV6.bpmn
  • PaymentControllerV6

Business transactions using compensation

The last part of the example adds compensation to the game. In distributed systems, ACID transactions are not applicable (or at least do not scale well). Using compensation is the alternative - meaning that you reliably undo already executed work if something later on fails.

See payment6.bpmn / Java and PaymentV6.bpmn / C# for the workflow

How-to run

You have to startup both services:

• Stripe Fake Server
• Payment Service

This varies:

• How to run in Java
• How to run in .NET
• How to run in Node.js

Now the different versions of the payment service are available:

• http://localhost:8100/api/payment/v1
• ...
• http://localhost:8100/api/payment/v6

You now can issue a PUT with an empty body:

curl \
-H "Content-Type: application/json" \
-X PUT \
-d '{}' \
http://localhost:8100/api/payment/v1

Hint on using Camunda Enterprise Edition

For Camunda there is an enterprise edition available with [https://camunda.com/products/cockpit/#/features](additional features in Cockpit) (the monitoring tool). It is quite handy to use this when playing around with the example. You can easily switch to use enterprise edition:

• Get a trial license if you don't have a license yet: https://camunda.com/download/enterprise/
• Java: Adjust Camunda version used in pom: ./pom.xml#L12, ./pom.xml#L50
• .NET: Download the Camunda enterprise version

Note that you do not need the enterprise edition to run the examples, the community edition will also do fine. But because of less features you do not see historical workflow instances - and that means you do not see that much in Camunda Cockpit if everything runs smoothly.

Using Camunda Cloud

You can get a free hosted instance of Zeebe in Camunda Cloud at https://camunda.io.