Outputs?

[hden]

I was searching for a general ETL process using transducers and found this project. It seems that most of common ETL process can be expressed as transducers. Any change we can extend this project to include outputs e.g. file, database?

(ns transducible-stream.core
  (:refer-clojure :exclude [into])
  (:require [clojure.java.jdbc :as jdbc]
            [cheshire.core :as cheshire]
            [clojure.java.io :refer [writer]])
  (:import [java.io Closeable Flushable BufferedWriter]))

(def conn {:dbtype   "mysql"
           :dbname   "database"
           :user     "username"
           :password "password"})

(defn write
  ([writer]
   (.close writer))
  ([writer x]
   (.write writer x)
   writer))

(defn into
  ([] [])
  ([to] to)
  ([to from]
   (into to (map identity) from))
  ([to xform from]
   (with-open [writer (writer to)]
    (transduce xform write writer from))))

(defn -main []
  (let [from (jdbc/reducible-query conn ["select * from table"])
        xform (map #(str (cheshire/generate-string %) "\n"))]
    (into "test.jsonl" xform from)))

[glts]

It seems to me that producing an output is more in the domain of the reducible/transducible process. I understand reducible-stream to be about reducible I/O inputs and their transformation (‘decoding’), not about what ultimately processes these building blocks.

Just noting in passing …