Skip to content

Commit

Permalink
added context in processor and updated documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@jruaux
jruaux committed Sep 19, 2019
1 parent 4eddb74 commit 8f809a1
Show file tree
Hide file tree
Showing 16 changed files with 303 additions and 86 deletions.
1 change: 1 addition & 0 deletions build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ tasks.withType(Tar){
asciidoctor {
outputDir = file('docs')
attributes revnumber : '1.0'
baseDir = 'src/docs/asciidoc'
outputOptions {
separateOutputDirs = false
}
Expand Down
84 changes: 79 additions & 5 deletions docs/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -453,6 +453,13 @@ <h1>RIOT</h1>
<li><a href="#file">Files</a></li>
<li><a href="#db">Databases</a></li>
<li><a href="#gen">Generators</a></li>
<li><a href="#redis">Redis</a></li>
</ul>
</li>
<li><a href="#processors">Processors</a>
<ul class="sectlevel2">
<li><a href="#field-expressions">Field expressions</a></li>
<li><a href="#script-evaluator">Script evaluator</a></li>
</ul>
</li>
<li><a href="#load-testing">Load Testing</a>
Expand Down Expand Up @@ -527,16 +534,16 @@ <h3 id="file">Files</h3>
<div class="ulist">
<ul>
<li>
<p>/myfolder/myfile.csv</p>
<p><code>/myfolder/myfile.csv</code></p>
</li>
<li>
<p>/myfolder/myfile.json.gz</p>
<p><code>/myfolder/myfile.json.gz</code></p>
</li>
<li>
<p><a href="https://example.com/path/dataset.csv" class="bare">https://example.com/path/dataset.csv</a></p>
<p><code><a href="https://example.com/path/dataset.csv" class="bare">https://example.com/path/dataset.csv</a></code></p>
</li>
<li>
<p>s3://mybucket/myfolder/myfile.json</p>
<p><code>s3://mybucket/myfolder/myfile.json</code></p>
</li>
</ul>
</div>
Expand Down Expand Up @@ -798,6 +805,73 @@ <h4 id="faker">Faker</h4>
</div>
</div>
</div>
<div class="sect2">
<h3 id="redis">Redis</h3>
<div class="paragraph">
<p>The Redis connector allows for transferring data between two Redis databases.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-shell hljs" data-lang="shell">$ riot -s 127.0.0.1:6379 redis-export -s localhost:6379 --scan-keyspace beer --scan-keys id --keyspace beer2 --keys id</code></pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="processors">Processors</h2>
<div class="sectionbody">
<div class="paragraph">
<p>RIOT can process records as they are being read. There are 2 processor approaches: field expressions or script processor.</p>
</div>
<div class="sect2">
<h3 id="field-expressions">Field expressions</h3>
<div class="paragraph">
<p>You can transform the incoming key/value pairs with field expressions, in the form <code>field1=&lt;exp&gt;</code>, <code>field2=&lt;exp&gt;</code>, &#8230;&#8203;</p>
</div>
<div class="paragraph">
<p>The input record can be accessed by field name, and the processor context also exposes the following variables that can be called with the <code>#</code> prefix:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>redis</code>: Redis connection to issue any command, e.g. <code>name=#redis.hgetall('person1').lastName</code></p>
</li>
<li>
<p><code>date</code>: date parser/formatter, e.g. <code>epoch=#date.parse(mydate).getTime()</code></p>
</li>
<li>
<p><code>context</code>: exposes processor context, e.g. <code>id=#context.index</code></p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-shell hljs" data-lang="shell">$ riot file-import -p event -k Id --header --processor-date-format "MM/dd/yyyy HH:mm:ss a" --processor "EventStartDate=remove('Event Start Date')" "EpochStart=#date.parse(EventStartDate).getTime()" "index=#context.index" --file "https://data.lacity.org/api/views/rx9t-fp7k/rows.csv?accessType=DOWNLOAD"</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="script-evaluator">Script evaluator</h3>
<div class="paragraph">
<p>Use an external or inline script to provide the processor logic.</p>
</div>
<div class="listingblock">
<div class="title">Javascript example</div>
<div class="content">
<pre class="highlightjs highlight"><code class="language-javascript hljs" data-lang="javascript">function process(item) {
item.name = item.name.toUpperCase();
return item;
}
process(item);</code></pre>
</div>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-shell hljs" data-lang="shell">$ riot faker --fields id=index firstName=name.firstName lastName=name.lastName address=address.fullAddress --processor-script "function process(item) { item.address = item.address.toUpperCase(); return item; } process(item);" --max 100 -p person -k id</code></pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
Expand Down Expand Up @@ -852,7 +926,7 @@ <h3 id="using-redis-enterprise">Using Redis Enterprise</h3>
<div id="footer">
<div id="footer-text">
Version 1.0<br>
Last updated 2019-09-19 00:43:15 -0700
Last updated 2019-09-19 14:49:51 -0700
</div>
</div>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.13.1/styles/github.min.css">
Expand Down
78 changes: 63 additions & 15 deletions src/docs/asciidoc/index.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
:icons: font
:project-repo: Redislabs-Solution-Architects/riot
:uri-repo: https://github.com/{project-repo}
:commandsdir: ../../../src/test/resources/commands

RIOT is a data import/export tool for Redis that connects to files, databases, and generators.

Expand Down Expand Up @@ -36,10 +37,10 @@ RIOT can import/export files in these formats:

Files can be GZIP compressed, and local or remote:

* /myfolder/myfile.csv
* /myfolder/myfile.json.gz
* https://example.com/path/dataset.csv
* s3://mybucket/myfolder/myfile.json
* `/myfolder/myfile.csv`
* `/myfolder/myfile.json.gz`
* `https://example.com/path/dataset.csv`
* `s3://mybucket/myfolder/myfile.json`

In the case of AWS S3 you can pass the access and secret keys as well as the region for the bucket.

Expand All @@ -48,13 +49,13 @@ In the case of AWS S3 you can pass the access and secret keys as well as the reg
===== CSV -> Redis
[source,shell]
----
include::src/test/resources/commands/file-import-csv-hash.txt[]
include::{commandsdir}/file-import-csv-hash.txt[]
----

===== Redis -> CSV
[source,shell]
----
include::src/test/resources/commands/file-export-csv.txt[]
include::{commandsdir}/file-export-csv.txt[]
----

===== CSV -> RediSearch
Expand All @@ -68,7 +69,7 @@ FT.CREATE beers SCHEMA abv NUMERIC SORTABLE id TAG name TEXT PHONETIC dm:en styl
+
[source,shell]
----
include::src/test/resources/commands/file-import-csv-search.txt[]
include::{commandsdir}/file-import-csv-search.txt[]
----
. Search for beers
+
Expand Down Expand Up @@ -102,18 +103,18 @@ RIOT can read and write JSON data in the following format:
===== JSON -> Redis
[source,shell]
----
include::src/test/resources/commands/file-import-json-hash.txt[]
include::{commandsdir}/file-import-json-hash.txt[]
----

===== Redis -> JSON
[source,shell]
----
include::src/test/resources/commands/file-export-json.txt[]
include::{commandsdir}/file-export-json.txt[]
----

===== Redis -> Compressed JSON
----
include::src/test/resources/commands/file-export-json_gz.txt[]
include::{commandsdir}/file-export-json_gz.txt[]
----

[#db]
Expand Down Expand Up @@ -151,14 +152,14 @@ For non-included databases you must install the corresponding JDBC driver under
==== SQL -> Redis
[source,shell]
----
include::src/test/resources/commands/db-import.txt[]
include::{commandsdir}/db-import.txt[]
----

==== Redis -> SQL
This command exports all hashes in keyspace `beer:<id>` to the database:
[source,shell]
----
include::src/test/resources/commands/db-export.txt[]
include::{commandsdir}/db-export.txt[]
----

[#gen]
Expand All @@ -175,7 +176,7 @@ You can also configure it to generate fixed-sized fields with `--field <name=siz
For example the following command generates hashes in the keyspace `test:<index>` with fields `field1` and `field2` of respectively 100 and 1,000 bytes:
[source,shell]
----
include::src/test/resources/commands/gen-simple.txt[]
include::{commandsdir}/gen-simple.txt[]
----

==== Faker
Expand All @@ -184,7 +185,7 @@ This data generator relies on the https://github.com/DiUS/java-faker[Faker] libr
.Example #1: People
[source,shell]
----
include::src/test/resources/commands/gen-faker-hash.txt[]
include::{commandsdir}/gen-faker-hash.txt[]
----
[source,plaintext]
----
Expand All @@ -197,7 +198,7 @@ include::src/test/resources/commands/gen-faker-hash.txt[]
.Example #2: Game of Thrones
[source,shell]
----
include::src/test/resources/commands/gen-faker-set.txt[]
include::{commandsdir}/gen-faker-set.txt[]
----
[source,plaintext]
----
Expand All @@ -207,6 +208,53 @@ include::src/test/resources/commands/gen-faker-set.txt[]
…
----

=== Redis

The Redis connector allows for transferring data between two Redis databases.

[source,shell]
----
include::{commandsdir}/redis-export.txt[]
----

== Processors

RIOT can process records as they are being read. There are 2 processor approaches: field expressions or script processor.

=== Field expressions

You can transform the incoming key/value pairs with field expressions, in the form `field1=<exp>`, `field2=<exp>`, ...

The input record can be accessed by field name, and the processor context also exposes the following variables that can be called with the `#` prefix:

* `redis`: Redis connection to issue any command, e.g. `name=#redis.hgetall('person1').lastName`
* `date`: date parser/formatter, e.g. `epoch=#date.parse(mydate).getTime()`
* `context`: exposes processor context, e.g. `id=#context.index`

[source,shell]
----
include::{commandsdir}/file-import-csv-processor-hash-dateformat.txt[]
----

=== Script evaluator

Use an external or inline script to provide the processor logic.

.Javascript example
[source,javascript]
----
function process(item) {
item.name = item.name.toUpperCase();
return item;
}
process(item);
----

[source,shell]
----
include::{commandsdir}/gen-faker-script-processor-hash.txt[]
----

== Load Testing

=== Metrics
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/com/redislabs/riot/Riot.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
import com.redislabs.riot.cli.redis.RediSearchImportCommand;
import com.redislabs.riot.cli.redis.RedisConnectionOptions;
import com.redislabs.riot.cli.redis.RedisEndpoint;
import com.redislabs.riot.cli.redis.RedisImportCommand;
import com.redislabs.riot.cli.redis.RedisExportCommand;
import com.redislabs.riot.cli.test.InfoCommand;
import com.redislabs.riot.cli.test.PingCommand;

Expand All @@ -37,7 +37,7 @@

@Command(name = "riot", abbreviateSynopsis = true, mixinStandardHelpOptions = true, subcommands = {
FileImportCommand.class, FileExportCommand.class, DatabaseImportCommand.class, DatabaseExportCommand.class,
RedisImportCommand.class, RediSearchImportCommand.class, FakerGeneratorCommand.class,
RedisExportCommand.class, RediSearchImportCommand.class, FakerGeneratorCommand.class,
SimpleGeneratorCommand.class, ConsoleExportCommand.class, PingCommand.class,
InfoCommand.class }, versionProvider = ManifestVersionProvider.class)
public class Riot implements Runnable {
Expand Down
Loading

0 comments on commit 8f809a1

Please sign in to comment.