Move ES consistency script into a separate stage.

While the script is the stage 069's counterpart in data4es-consistency-check,
they share no functionality.

Utils/Dataflow/069_upload2es/README

@@ -4,7 +4,7 @@
 
 Description
 -----------
-load_data.sh uploads prepared data to ElasticSearch.
+Uploads prepared data to ElasticSearch.
 
 Input
 -----
@@ -18,24 +18,6 @@ JSON documents, one per line:
 ...
 }}}
 
-Consistency
------------
-consistency.py checks that the data is present in ElasticSearch instead of
-uploading it. Input comes from Stage 009(in consistency mode) and only needs 2
-fields for now:
-{{{
-{taskid, task_timestamp}
-...
-}}}
-
-Consistency check can be run as following:
-
-  ./consistency.py --conf elasticsearch_config
-
-For more information about running the check and its arguments, use:
-
-  ./consistency.py -h
-
 TODO
 ----
 Make the stage aware of EOProcess/EOMessage markers

Utils/Dataflow/071_esConsistency/README

@@ -0,0 +1,28 @@
+=============
+* Stage 071 *
+=============
+
+1. Description
+--------------
+Checks that the given data is present in ElasticSearch.
+
+Input must contain at least 2 fields:
+{{{
+{"_type": ..., "_id": ..., ...}
+...
+}}}
+
+_type and _id are required to retrieve the document from ES. All the other
+fields are compared with the document's corresponding ones. Results of the
+comparison are written to stderr.
+
+2. Running the stage
+--------------------
+The stage can be run as following:
+
+  ./consistency.py --conf elasticsearch_config
+
+For more information about running the stage and its arguments, use:
+
+  ./consistency.py -h
+

Utils/Dataflow/run/data4es-consistency-check

@@ -20,6 +20,6 @@ cmd_016="$base_dir/../016_task2es/task2es.py -m s"
 
 # ES
 cfg_es=`get_config "es"`
-cmd_069="$base_dir/../069_upload2es/consistency.py -m s --conf $cfg_es"
+cmd_071="$base_dir/../071_esConsistency/consistency.py -m s --conf $cfg_es"
 
-$cmd_009 | $cmd_016 | eop_filter | $cmd_069 >/dev/null
+$cmd_009 | $cmd_016 | eop_filter | $cmd_071 >/dev/null