Diffy is used in production at:
- Mixpanel
- Airbnb (Scalabity) (Migration)
- Baidu
- Bytedance
and blogged about by cloud infrastructure providers like:
If your organization is using Diffy, consider adding a link here and sending us a pull request!
Diffy is being actively developed and maintained by the engineering team at Sn126.
Feel free to contact us via linkedin, gitter or twitter.
Diffy finds potential bugs in your service using running instances of your new code and your old code side by side. Diffy behaves as a proxy and multicasts whatever requests it receives to each of the running instances. It then compares the responses, and reports any regressions that may surface from those comparisons. The premise for Diffy is that if two implementations of the service return “similar” responses for a sufficiently large and diverse set of requests, then the two implementations can be treated as equivalent and the newer implementation is regression-free.
Diffy acts as a proxy that accepts requests drawn from any source that you provide and multicasts each of those requests to three different service instances:
- A candidate instance running your new code
- A primary instance running your last known-good code
- A secondary instance running the same known-good code as the primary instance
As Diffy receives a request, it is multicast and sent to your candidate, primary, and secondary instances. When those services send responses back, Diffy compares those responses and looks for two things:
- Raw differences observed between the candidate and primary instances.
- Non-deterministic noise observed between the primary and secondary instances. Since both of these
instances are running known-good code, you should expect responses to be in agreement. If not,
your service may have non-deterministic behavior, which is to be expected.
Diffy measures how often primary and secondary disagree with each other vs. how often primary and candidate disagree with each other. If these measurements are roughly the same, then Diffy determines that there is nothing wrong and that the error can be ignored.
If you are new to Diffy, please refer to our Quickstart guide.
- Login to isotope.
- Click on services tab to create the service you want to test.
- Download the resulting
local.isotopefile. - Deploy Diffy with the
isotope.configflag pointing to the location oflocal.isotope:java -jar ./target/scala-2.12/diffy-server.jar \ -candidate='localhost:9200' \ -master.primary='localhost:9000' \ -master.secondary='localhost:9100' \ -responseMode='primary' \ -service.protocol='http' \ -serviceName='ExampleService' \ -summary.delay='1' \ -summary.email='your email to receive a summary report from your diffy instance' \ -proxy.port=:8880 \ -admin.port=:8881 \ -http.port=:8888 \ -isotope.config='/path/to/local.isotope' - Send some traffic to your deployed Diffy instance.
- Go the analysis tab to see your latest and historical results.
Please reach out to [email protected] for support. We look forward to hearing from you.
- Bug reports are welcome even if submitted anonymously via fresh github accounts.
- Anonymous feature requests and usage questions will be ignored.
Copyright (C) 2019 Sn126, Inc.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see https://www.gnu.org/licenses/.