Pact plugin for testing messages and gRPC service calls encoded with as Protocol buffers using the Pact contract testing framework.
This plugin provides support for matching and verifying Protobuf messages and gRPC service calls. It fits into the Pact contract testing framework and extends Pact testing for Protocol buffer payloads and gRPC.
- Requirements to use it
- Installation
- Supported features
- Unsupported features
- Using the plugin
- Support
- Contributing to the plugin
- Development Roadmap
This plugin provides matching and verification of Protobuf proto3 encoded messages and gRPC service methods to the Pact contract testing framework. It requires a version of the Pact framework that supports the V4 Pact specification as well as the Pact plugin framework.
Supported Pact framework versions:
To support compiling Protocol Buffer proto files requires a version of the Protocol Buffer compiler.
The executable binaries and plugin manifest file for the plugin can be downloaded from the project releases page. There will be an executable for each operating system and architecture. If your particular operating system or architecture is not supported, please send a request to [email protected] with the details.
To install the plugin requires the plugin executable binary as well as the plugin manifest file to be unpacked/copied into
a Pact plugin directory. By default, this will be .pact/plugins/protobuf-<version> in the home directory (i.e.
$HOME/.pact/plugins/protobuf-0.1.5 for version 0.1.5).
The pact-plugin-cli can be used to install the Protobuf/gRPC plugin. See the CLI installation on how to install it.
To install the latest version, run
pact-plugin-cli -y install https://github.com/pactflow/pact-protobuf-plugin/releases/latestExample installation of Linux version 0.1.5 (replace with the actual version you are using):
- Create the plugin directory if needed:
mkdir -p ~/.pact/plugins/protobuf-0.1.5 - Download the plugin manifest into the directory:
wget https://github.com/pactflow/pact-protobuf-plugin/releases/download/v-0.1.5/pact-plugin.json -O ~/.pact/plugins/protobuf-0.1.3/pact-plugin.json - Download the plugin executable into the directory:
wget https://github.com/pactflow/pact-protobuf-plugin/releases/download/v-0.1.5/pact-protobuf-plugin-linux-x86_64.gz -O ~/.pact/plugins/protobuf-0.1.5/pact-protobuf-plugin.gz - Unpack the plugin executable:
gunzip -N ~/.pact/plugins/protobuf-0.1.5/pact-protobuf-plugin.gz - Make the plugin executable:
chmod +x ~/.pact/plugins/protobuf-0.1.5/pact-protobuf-plugin
Note: The unpacked executable name must match the entryPoint value in the manifest file. By default, this is
pact-protobuf-plugin on unix* and pact-protobuf-plugin.exe on Windows.
The default plugin directory ($HOME/.pact/plugins) can be changed by setting the PACT_PLUGIN_DIR environment variable.
The plugin can automatically download the correct version of the Protocol buffer compiler for the current operating system and architecture. By default, it will download the compiler from https://github.com/protocolbuffers/protobuf/releases and then unpack it into the plugin's installation directory.
The plugin executes the following steps:
- Look for a valid
protoc/bin/protocin the plugin installation directory - If not found, look for a
protoc-{version}-{OS}.zipin the plugin installation directory and unpack that (i.e. for Linux it will look forprotoc-3.19.1-linux-x86_64.zip). - If not found, try download protoc using the
downloadUrlentry in the plugin manifest file - Otherwise, fallback to using the system installed protoc
If the plugin is going to run in an environment that does not allow automatic downloading of files, then you can do any of the following:
- Download the protoc archive and place it in the plugin installation directory. It will need to be the correct version and operating system/architecture.
- Download the protoc archive and unpack it into the plugin installation directory. It will need to be in a
protocdirectory. Do this if the current version is not supported for your operating system/architecture. - Change the
downloadUrlentry in the plugin manifest to point to a location that the file can be downloaded from. - Install the correct version of the protoc compiler as an operating system package. It must then be on the executable path when the plugin runs. For instance, for Alpine Linux this will need to be done as the downloaded versions will not work.
NOTE: Since 0.1.3, the logging was switched to the Rust tracing crate and a log configuration file is no longer supported.
The plugin will log to both standard output and two files (log/plugin.log.* and log/plugin.log.json.*) in the plugin
installation directory. Each file will be rolled per day and be suffixed with the current date. The JSON log file will
be formatted in the bunyan format.The log level will be set by the LOG_LEVEL
environment variable that is passed into the plugin process (this should be set by the framework calling it).
The Protobuf plugin supports the following configuration options, which can be set in the plugin manifest file under
pluginConfig:
The Protobuf compiler version to download if required.
The URL to download the Protobuf compiler from. By default, this will be the Protocol Buffers GitHub release page.
Host to bind to. Default is the IP4 loopback adapter 127.0.0.1, to use the IP6 loopback set it to ::1.
Additional directories to include to add to the Protocol buffers compiler to search for proto files. Each value will be
added verbatim to the protoc command line using -I. THESE ARE DIRECTORIES NOT FILES!
Version 0.2.4+
Configuration values can also be passed in from the test. They need to be passed in via the test configuration data
under the pact:protobuf-config key. For example, to add additional proto file include directories in the test:
"pact:proto", filePath("../proto/test_enum.proto"),
"pact:content-type", "application/grpc",
"pact:proto-service", "Test/GetFeature",
"pact:protobuf-config", Map.of(
"additionalIncludes", List.of(filePath("../proto2"))
)The plugin currently supports proto3 formatted messages and service calls.
It supports the following:
- Scalar fields (Double, Float, Int64, Uint64, Int32, Uint32, Fixed64, Fixed32, Bool, Sfixed32, Sfixed64, Sint32, Sint64).
- Variable length fields (String, Bytes).
- Enum fields.
- Embedded messages.
- Map fields (with a string key).
- Repeated fields.
- Packed repeated fields.
- oneOf fields.
- gRPC Service method calls.
- Testing/verifying gRPC service call metadata.
- Verifying gRPC error responses.
The following features are currently unsupported, but may be supported in a later release:
- Map fields with scalar keys.
- Map fields with enum keys.
- default values for fields.
- required fields (note that this is deprecated in Proto 3).
- Testing/verifying Protobuf options.
The following features will not be supported by this plugin:
- proto2
- Groups
The following features may be supported in a future release, but are not currently planned to be supported:
- Map fields where the key is not a string or scalar value.
- gRPC streaming (either oneway or bidirectional).
This plugin will register itself with the Pact framework for the application/protobuf and application/gRPC content types.
Using this plugin, you can write Pact tests that verify either a single Protobuf message (i.e. a message provider sends a single, or one-shot, message to a consumer), or you can verify a service method call where there is an input message and an output message.
Single message tests are supported by using the V4 asynchronous message Pact format, and the service method calls use the V4 synchronous message Pact format.
For an overview how asynchronous messages work with Pact, see Non-HTTP testing (Message Pact).
In this scenario, a message provider writes a Protocol Buffer message to some one-way transport mechanism, like a message queue, and a consumer then reads it. With this style of testing, the transport mechanism is abstracted away.
The message consumer test is written using the Pact Message test DSL. The test DSL defines the expected message format, and then the consumer is tested with an example message generated by the test framework.
For an example of a message consumer test:
The message provider is verified by getting it to generate a message, and then this is verified against the Pact file from the consumer. There are two main ways of verifying the provider:
- Write a test in the provider code base that can call the provider to generate the message.
- Use an HTTP proxy server that can call the provider and return the generated message, and then use a Pact framework verifier to verify it.
For an example of the latter form, see Simple Example Protobuf provider.
With a service method call, the consumer creates an input message, then invokes a service method and gets an output message as the response. The most common service call is via the gRPC framework.
This plugin supports testing service method calls via gRPC on both the consumer and provider side.
The service method consumer is tested by configuring a test that starts a gRPC mock server based on the proto file for the service. Each test first configures a Pact from the proto file. The Pact framework (via this plugin) will then create a gRPC mock server for the test. The gRPC consumer can then be pointed at the mock server during the test and send the input message and then verify the output message that is received back.
For an example:
The Pact framework (using this plugin) can test gRPC service method calls to a running gRPC server. The server can be tested by either using a unit test, or by using the Rust Verifier CLI. It will need the Pact file created from the consumer test with the compiled protobuf descriptors in it (these will have been added by this plugin during the consumer test).
For an example Java unit test: See the example gRPC verification test.
By starting the gRPC server, we can then also use the Pact Verifier to check it.
For example (using the example gRPC project):
Running the gRPC server:
gRPC/area_calculator/provider-jvm:
❯ ./gradlew run
> Task :server:run
14:56:17,785 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
14:56:17,786 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
14:56:17,787 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/home/ronald/Development/Projects/Pact/pact-plugins/examples/gRPC/area_calculator/provider-jvm/server/build/resources/main/logback.xml]
14:56:17,862 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
14:56:17,863 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
14:56:17,869 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
14:56:17,874 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
14:56:17,928 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to ERROR
14:56:17,928 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
14:56:17,930 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@13:36 - no applicable action for [io.grpc.netty], current ElementPath is [[configuration][io.grpc.netty]]
14:56:17,930 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
14:56:17,932 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@2b662a77 - Registering current configuration as safe fallback point
Started calculator service on 37621
<===========--> 87% EXECUTING [18s]
> :server:runWe can see that the gRPC server was started on a random port (37621 above). So we can then provide the Pact file and port number to the verifier.
gRPC/area_calculator/provider-jvm:
❯ pact_verifier_cli -f ../consumer-jvm/build/pacts/protobuf-consumer-area-calculator-provider.json -p 37621
Verifying a pact between protobuf-consumer and area-calculator-provider
calculate rectangle area request
Test Name: io.pact.example.grpc.consumer.PactConsumerTest.calculateRectangleArea(MockServer, SynchronousMessages)
Given a Calculator/calculate request
with an input .area_calculator.ShapeMessage message
will return an output .area_calculator.AreaResponse message [OK]If you can mock out the gRPC channel or stub, it is fairly easy to test the service method call without requiring a gRPC server.
To test the service message consumer, we write a Pact test that defines the expected input (or request) message and the expected output (or response message). The Pact test framework will generate an example input and output message.
To execute the test, we need to intercept the service method call and verify that the message the consumer generated was correct, then we return the output message and verify that the consumer processed it correctly. This can be achieved using a test mocking library.
For an example:
The Protocol Buffer service providers normally extend an interface generated by the protoc compiler. To test them, we need a mechanism to get the Pact verifier to pass in the input message from the Pact file and then get the output message from the service and compare that to the output message from the Pact file.
We can use the same mechanism as for message pact (see Non-HTTP testing (Message Pact)), were we create an HTTP proxy server to receive the input message from the verifier and invoke the service method implementation to get the output message.
There are two main ways to run the verification:
- Execute the Pact verifier, providing the source of the Pact file, and configure it to use the HTTP mock server.
- Write a test in the provider's code base. For an example of doing this in Rust, see a test that verifies this plugin.
You can use this plugin to test negative cases where an error response is expected to be returned (for an example see
gRPC status). This works by
checking if there is an expected grpc-status attribute set when an error response is returned (and also can check
for a grpc-message attribute).
To use it, don't configure a response message, but add the expected status (and message if required) to the response metadata. I.e, in the consumer test:
.with(Map.of(
"pact:proto", filePath("../proto/grpc_status.proto"),
"pact:content-type", "application/grpc",
"pact:proto-service", "Calculator/calculate",
"request", Map.of(
"parallelogram", Map.of(
"base_length", "matching(number, 3)",
"height", "matching(number, 4)"
)
),
// We are expecting an error response for this message
"responseMetadata", Map.of(
"grpc-status", "UNIMPLEMENTED",
"grpc-message", "matching(type, 'we do not currently support parallelograms')"
)
))then when the interaction is verified you will see (assuming the provider returns the correct response):
Verifying a pact between grpc-consumer-rust and grpc-provider
invalid request (0s loading, 54ms verification)
Given a Calculator/calculate request
with an input .area_calculator.ShapeMessage message
will return an error response Operation is not implemented or not supported [OK]
with metadata
key 'grpc-message' with value 'we do not currently support parallelograms' [OK]
key 'grpc-status' with value 'UNIMPLEMENTED' [OK]The consumer tests need to get the plugin loaded and configure the expected messages to use in the test. This is done
using the usingPlugin (or using_plugin, depending on the language implementation) followed by the content for the test
in some type of map form.
For each field of the message that we want in the contract, we define an entry with the field name as the key and a matching definition as the value. For documentation on the matching definition format, see Matching Rule definition expressions.
For example, for a JVM test (taken from Protocol Buffer Java examples) we would use the PactBuilder class:
// this example taken from https://developers.google.com/protocol-buffers/docs/javatutorial#defining-your-protocol-format
message Person {
string name = 1;
int32 id = 2;
string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
string number = 1;
PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phones = 4;
}builder
// Tell the Pact framework to load the protobuf plugin
.usingPlugin("protobuf")
// Define the expected message (description) and the type of interaction. Here is is an asynchronous message.
.expectsToReceive("Person Message", "core/interaction/message")
// Provide the data for the test
.with(Map.of(
// For a single asynchronous message, we just provide the contents for the message. For RPC service calls, there
// will be a request and response message
"message.contents", Map.of(
// set the content type, so the Pact framework will know to send it to the Protobuf plugin
"pact:content-type", "application/protobuf",
// pact:proto contains the source proto file, which is required to be able to test the interaction
"pact:proto", filePath("addressbook.proto"),
// provide the name of the message type we are going to test (defined in the proto file)
"pact:message-type", "Person",
// We can then setup the expected fields of the message
"name", "notEmpty('Fred')", // The name field must not be empty, and we use Fred in our tests
"id", "matching(regex, '100\\d+', '1000001')", // The id field must match the regular expression, and we use 1000001 in the tests
"email", "matching(regex, '\\w+@[a-z0-9\\.]+', '[email protected]')" // Emails must match a regular expression
// phones is a repeated field, so we define an example that all values must match against
"phones", Map.of(
"number", "matching(regex, '(\\+\\d+)?(\\d+\\-)?\\d+\\-\\d+', '+61-03-1234-5678')" // Phone numbers must match a regular expression
// We don't include type, as it is an emum and has a default value, so it is optional
// but we could have done something like matching(equalTo, 'WORK')
)
)
))You can apply matching rules to enforce the minimum or maximum number fo fields, as well as applying rules for each item.
For instance, assume that numbers is a repeated String field. You could define
"numbers": "atLeast(1), eachValue(matching(regex, '\\d+', '100))"This will make aure that there is always at least one number, and that any numbers match the given regular expression.
Note: atLeast and atMost require version 0.3.12+
This would also work if numbers was a numeric repeated field. In fact, it would work with any primitive field by
applying the regex to the string representation of the field value.
With maps, you can apply matching rules on either the keys in the map, or the values, or both. For instance, given
message ExampleRequest {
map<string, string> labels = 1;
bool ok = 2;
}You could define a test like
"request": {
"labels": {
"pact:match": "eachKey(matching(regex, '\\d+', '')), eachValue(matching(regex, '(\\w|\\s)+', '')), atLeast(1)",
"100": "this is a label"
},
"ok": "true"
}This will require the map of labels to only have keys and values that match the given regular expressions, and the map
must have at least one entry. "100": "this is a label" is the example value used in the consumer test.
Requires 0.5.0+ of the Protobuf plugin.
Values from provider states can be injected into Protobuf
message fields and gRPC metadata. This can be done by using the fromProviderState function in the consumer test.
For example (taken from an example Java Consumer Test):
{
"request": {
"rectangle": {
"length": "matching(number, fromProviderState('${rectangleLength}', 3))",
"width": "matching(number, fromProviderState('${rectangleWidth}', 3))"
},
"requestMetadata": {
"Auth": "matching(equalTo, fromProviderState('${Auth}', 'AST00004'))"
},
"response": {
"value": "matching(number, 12)"
}
}
}This example allows the rectangle length and width to be changed from values from the provider state callback, as well
as the Auth metadata value. You can see an example doing that here.
The plugin will try to use an IP6 address when opening the port for the gRPC server. Docker will only support IP6
addresses with extra configuration applied and this will not be available by default. To use an IP4 address instead,
you can either add the host parameter as a command line parameter, or add hostToBindTo value to the plugin
configuration in the manifest file.
I.e., updated manifest to use 127.0.0.1 as the host to bind to
{
"manifestVersion": 1,
"pluginInterfaceVersion": 1,
"name": "protobuf",
"version": "0.1.8",
"executableType": "exec",
"entryPoint": "pact-protobuf-plugin",
"pluginConfig": {
"protocVersion": "3.19.1",
"downloadUrl": "https://github.com/protocolbuffers/protobuf/releases/download",
"hostToBindTo": "127.0.0.1"
}
}Supported Platforms
| OS | Architecture | Supported | Pact Protobuf Plugin Version |
|---|---|---|---|
| OSX | x86_64 | ✅ | All |
| Linux | x86_64 | ✅ | All |
| Windows | x86_64 | ✅ | All |
| OSX | arm64 | ✅ | All |
| Linux | arm64 | ✅ | >=0.1.11 |
| Windows | arm64 | ✅ | >=0.3.15 |
| Alpine | x86_64 | ✅ | >=0.1.9 |
| Alpine | arm64 | ✅ | >=0.3.15 |
Note: From v0.3.15, All Linux executables are statically built with musl and are designed to work against glibc (eg, Debian) and musl (eg, Alpine) based distos.
Join us on slack 
or
Twitter: @pact_up
Stack Overflow: stackoverflow.com/questions/tagged/pact
PRs are always welcome!
For details on the V4 Pact specification, refer to https://github.com/pact-foundation/pact-specification/tree/version-4
For details on the Pact plugin framework, refer to https://github.com/pact-foundation/pact-plugins
Before raising an issue, make sure you have checked the open and closed issues to see if an answer is provided there. There may also be an answer to your question on stackoverflow.
Please provide the following information with your issue to enable us to respond as quickly as possible.
- The relevant versions of the packages you are using (plugin and Pact versions).
- The steps to recreate your issue.
- An executable code example where possible.
- Fork it
- Create your feature branch (git checkout -b my-new-feature)
- Commit your changes (git commit -am 'feat: Add some feature')
- Push to the branch (git push origin my-new-feature)
- Create new Pull Request
We follow the Conventional Changelog message conventions. Please ensure you follow the guidelines.
To build the plugin, you need a working Rust environment (version 1.58+). Refer to the Rust Guide.
The build tool used is cargo and you can build the plugin by running cargo build. This will compile the plugin and
put the generated files in target/debug. The main plugin executable is pact-protobuf-plugin
and this will need to be copied into the Pact plugin directory. See the installation instructions above.
You can run all the unit tests by executing cargo test --lib.
There is a Pact test that verifies the plugin aqainst the Pact file published to pact-foundation.pactflow.io.
Running this test requires a Pactflow API token and the plugin to be built and installed. See the installation instructions above.
The test is run using cargo test --test pact_verify.
Several unit-tests have base64 encoded proto descriptors as input. They're hard to read and update. To decode and check out the contents:
- save the encoded string into a file, e.g.
/tmp/some.b64and thenbase64 -d /tmp/some.b64 | protoc --decode_raw - this will be somewhat readable, even if not a proper proto3 syntax, but it will be possible to reconstruct it.
- github copilot can decode the raw base64 string easily too
To encode a .proto file into a base64-encoded FileDescriptorSet,
e.g. if you modified tests/simple.proto and need to update its encoded form in tests:
protoc --descriptor_set_out=/dev/stdout tests/simple.proto | base64
This plugin is released under the MIT License and is copyright © 2021-24 SmartBear.
The Pactflow logos are copyright © SmartBear and may not be used without permission.