Skip to content
This repository has been archived by the owner on Oct 17, 2023. It is now read-only.

Latest commit

 

History

History
250 lines (183 loc) · 13.3 KB

CHANGELOG.md

File metadata and controls

250 lines (183 loc) · 13.3 KB
Raw

v1.1.0 [2022-05-26]

Features

  • Adds MySQL Adaptor via #515, thanks @atomicules!

v1.0.0 [2021-12-01]

This release does not introduce any new functionalities, but catches up transporter with modern golang. It also introduces a new standard way of running adaptor tests. Only the mongodb and postgresql adaptors have been fully tested. We'll add more as we go.

  • Bumped golang to 1.17
  • Migrated to go mod
  • Migrate from Travis CI to Github Actions
  • Standardize how adaptor tests are run. Using docker-compose, we can run them the same way on CI and locally.
  • Provide docker-compose and custom scripts to run mongodb and postgresql adaptor tests locally and on CI. See the respective adaptor README for more info.
  • Fix for postgresql tailer when there are new lines in a record, via #456, thanks @erft-engineering!
  • Reduce mongo bulk operation size to 1000, via #450, thanks @trinchan!

v0.5.0 [2017-09-11]

Features

  • new remap function for changing the namespace with a native function versus needing a custom JS function to do it, fixed via #400
  • new opfilter function for whitelisting or blacklisting message operations versus needing a custom JS function to do it, fixed via #401
  • elasticsearch adaptor now supports parent/child relationships, see fixed via #406, thanks @johnjjung!

Bugfixes

  • when using an adaptor that was configured for flushing bulk messages based on an interval, it was possible for the bulk flush to error but not propagated back up to the pipeline, fixed via #399
  • if using AWS Elasticsearch service, the process to determine the version of elasticsearch failed, fixed via #403

Changes

  • the -log_dir flag is now -xlog_dir and the log_dir config property is now xlog_dir to match. This is to avoid confusion with the application logging's -log flags.
  • elasticsearch adaptor now logs all errors returned from a bulk operation, #413, thanks @pmjhonwang!

v0.4.0 [2017-08-15]

Features

Bugfixes

  • when using MongoDB as a source with tailing enabled and namespace filtering, it was possible for documents from others collections to be sent down the pipeline, fixed via #386
  • if transporter lost connection to MongoDB while tailing the oplog, the connection never successfully reconnected, fixed via #398

v0.3.1 [2017-03-24]

Features

  • added js alias for goja transform function #335
  • init command will now prompt user when pipeline.js file already exists #336

Bugfixes

  • fixed mongodb, rabbitmq, and rethinkdb adaptors from not trying to read from a file when provided in the ca_certs field #334

v0.3.0 [2017-03-21]

Breaking changes

PLEASE READ

Transporter no longer requires a YAML file. All configuration is in the JS file where nodes are defined and a new DSL has been developed. Run the init command to see the new changes.

  • if using transporter as a library, all packages have been moved out of pkg to the top-level
  • eval command removed
  • list command removed
  • the namespace parameter now only expects a single part (the regexp filter), all adaptors have been updated to pull the "database name" from the provided URI

Features

  • NEW RabbitMQ adaptor #298
  • MongoDB adaptor supports per collection query filter when needing to copy only a subset of data #301
  • goja added as an option for the JavaScript VM in transformers #294
  • NEW native functions

Bugfixes

v0.2.2 [2017-03-20]

Features

Bugfixes

  • attempted fix for mejson.S conversion to json column in postgres adaptor #314

v0.2.1 [2017-03-07]

Features

  • added RethinkDB -> PostgreSQL integration test

Bugfixes

  • fixed connection leak in PostgreSQL client

v0.2.0 [2017-02-28]

Breaking changes

  • The etcd adaptor was removed

Features

  • Integration tests are run weekly for the following pipelines:
    1. MongoDB -> MongoDB
    2. MongoDB -> Elasticsearch
    3. MongoDB -> RethinkDB
  • RethinkDB SSL support added in #268
  • RethinkDB performs bulk inserts now, #276
  • transporter init [source] [sink] command added in #279

Bugfixes

  • MongoDB adaptor could cause the pipeline to stop due to a concurrent flush operation, fixed via #271
  • When being used as a library, several goroutines were leaking after the pipeline had stopped. Reported by @cognusion in #265 and addressed via #268 and #280

v0.1.3 [2017-02-09]

Breaking changes

  • namespace processing no longer expects their the be a "db" portion (i.e. "database.collection") but an attempt to maintain backwards compatibility is still there for the time being. #258

Bugfixes

  • #261: return a nil message to get skipped in the pipeline

v0.1.2 [2017-01-27]

This release is primarily aimed at getting the MongoDB and Elasticsearch adaptors into a stable/reliable state.

Breaking changes

  • MongoDB adaptor SSL configuration is now defined as:
nodes:
  localmongo:
    type: mongodb
    ssl: true
    cacerts: ["/path/to/cert.pem"] # optional

Bugfixes

  • #211: defer bulk channel init for mongo node reuse
  • #213: track mongodb _id field so we can attempt to reissue queries
  • #233: update elasticsearch adaptor with better support for multiple versions of elasticsearch as well as better performance with bulk indexing for most versions. Addresses #209, #222, #167 and #159.
  • properly detect oplog access when attempting to use the tail option on the MongoDB adaptor.

v0.1.1 [2015-08-27]

This release contains the first step to getting savable state into adaptors for the ability to resume.

Features

  • #116: Adaptor state phase 1, begins to address #33

Bugfixes

  • #124: Update elasticsearch adaptor

v0.1.0 [2015-08-06]

This release contains several breaking changes but is a first step towards a stable API/DSL.

Features

  • #95: add a Noop message type, and the ability to skip messages, fixes #93
  • #100: replace environment variables, fixes #88
  • #101: (Phase 1) Multi namespace support, addresses #78 and #23

Bugfixes

  • #94: jsonlog, fixes [#92](Comment doesnt reflect jsonlog output)
  • #98: Fixes influx_test.js, thanks @ExNexu!
  • #97: throw transformer initialization errors early, fixes #96

v0.0.4 [2015-07-13]

This release contains several breaking changes:

  • Transporter now uses godep for vendoring depedencies.
  • transformer functions now receive a document in the following format:
{
  "op": "insert/update/delete",
  "data": "", // what use to be passed in the doc passed to the transformer function
  "ts": 1436794521, // the number of seconds elapsed since January 1, 1970 UTC
}

The same document must be returned from the function so any modifications must happen to doc["data"].

NOTE: Deletes are now passed to the transformer functions so this needs to be accounted for.

Features

  • #46: add a JsonLog.
  • #64: New Source Adapter: Implements RethinkDB as a source of documents. Thanks @alindeman!
  • #68: Deletes from Mongo write adaptor.
  • #72: Allows RethinkDB timeout to be configured.
  • #83: RethinkDB: Changes for v1 RethinkDB driver.
  • #86: Use godep to vendor required libraries.
  • #87: add SSL support for MongoDB, bump mgo.
  • #90: NOTE BREAKING CHANGE: send more info in doc to transformers.

Bugfixes

  • #47: A few readme grammar fixes. Thanks @mm-!
  • #49: fixes #48
  • #51: fixes #50
  • #54: import rethinkdb from gopkg.in. it's still v0, so this will change still.
  • #55: remove influx until the state of the repository settles or we get proper vendoring in place.
  • #63: use a smaller buffer for mongo adaptor.
  • #65: Extracts only the relevant pieces of the version string. Thanks @alindeman!
  • #85: ensure we're at version >=2.0 of rethink for this driver.

v0.0.3 [2015-01-14]

Features

  • #36: Simplifies the config.yaml and node configuration.
  • #38: add a quick benchmark to transformOne

Bugfixes

  • #37: fixes #29
  • #42: fixes #41
  • #44: fix a logical error. check for transporter first.
  • #45: add a bulk writer, and writeconcern options for mongo.

v0.0.2 [2014-12-29]

Features

  • #22: adaptor constructor refactor, new transporter about command.
  • #26: adds the name and description to about.
  • #27: Instructions for building Transporter on Windows.
  • #31: add some more information in errors that we get in the javascript.
  • #32: Fix typo in registry comments for doc gen.

Bugfixes

  • #7: Make list informative, stop arg panics.
  • #9: beginners guide for os x. Thanks @sberryman!
  • #10: Pretty print the list.
  • #11: fix the config.yaml example to show the proper interval syntax.
  • #13: makes the api in the config.yaml optional.
  • #18: check that errors aren't nil.
  • #20: log any adaptor.ERROR or adaptor.CRITICAL.

v0.0.1 [2014-12-12]

Release Notes

This is the initial release of Transporter.