update docs/index.md; marked details as WIP

Signed-off-by: AbhishekKr <[email protected]>

docs/index.md

@@ -1,49 +1,74 @@
 
 ## Ogi
 
-utility to enable flexible [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) scenarios
+> **Ogi** is a thin core utility that's a "context free", "customizable utility" workflow runner.
+
+Built with purpose to have a mix-n-match steps for [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) like scenarios; but with bi-directional flows giving it more possibilities.
 
 > ![ogi means a japanese fan](./ogi.png "ogi means a japanese fan")
 
----
 
-#### How it works?
+### What Is It Now?
 
-It contains 3 primary components. A consumer, transformer and producer.
+In its basic design, it would allow one to use available (or separately/privately written) flows at any/all stages of `Consumer`, `Transformer` & `Producer`.
 
-Ogi initializes Consumer and let it handle the flow to transformer, or if required directly producer. That flow is internal to consumer used and not mandated. Similar internal flow freedom is granted to transformer and producer. Like, if required your producer can have multiple outputs anywhere from Kafka, S3 to something like e-mail.
+* `Consumer` is the init point, that spins up any kind of data retrieval or listens to published requests. Then hands them over to configured `Transformer`. Like reading a file line-by-line, reading records from a database, subscribing to a kafka topic, etc.
 
-All 3, `consumer`, `transformer` and `producer` are instantiated as per config and thus any combination of available types could be brought into play.
+* `Transformer` has the flow for any checks/edits required on received data; any pre-final task spins. Then the desired form of received data is handed over to configured `Producer`. Like checking if keys are valid before Producer persists it, create a JSON based on it that can be sent as a Request by Producer, etc.
 
-Consumer, Transformer and Producer support usage of `golang plugin`, so separately managed and developed constructs could be used in combination as well.
-Since they are loaded as per configuration provided identifier, one can have combination of say built-in Kafka consumer with built-in kubernetes-log transformer but custom external plug-in of Google Cloud Datastore for cold storage of logs.
-This also gives capability to write a producer sending output to more than one output sinks in same flow to achieve replication.
+* `Producer` contains the final action desired on the desired state of data/request received via Consumer-to-Transformer. It could be parsist to a file, send to a Kafka topic, add a database entry, make a custom HTTP Request, etc.
 
-Can be scaled up easily using kubernetes/nomad/mesos/\* elastic deployments as it inherently has no context.
+> * Allows custom components for Consumer/Transformer/Producer to be dynamically loaded via [Golang Plug-ins](https://pkg.go.dev/plugin).
+>
+> * Each Ogi run can pick a suitable type of each based on configuration; allowing all sorts of mix-match.
+>
+> * `Producer` can return an info response to `Transformer`; and it to `Consumer`. So even bidirectional flows are possible.
+>
+> * Each flow gets logged with a `Message ID`; so one can debug/err-check/observe events.
 
----
+*Repo for stable plug-ins is [OpenChaos/ogi-plugin-umbrella](https://github.com/OpenChaos/ogi-plugin-umbrella).*
 
-#### Concepts
 
-* [design in detail](./design)
+#### What's the FLEXIBILITY about?
+
+* Ogi uses env config to identify which Consumer, Transformer & Producer are to be used. Plug-ins can be used by providing `plugin` value and env for `<plug-in>.so` file path.
+
+> ![ogi flow](./ogi-flow.png "ogi flow")
+
+* Ogi's default flow invokes a Transformer for each data recieved at Consumer. When writing a custom consumer plug-in, if data can be directly handed over to Producer.. `Produce(..)` can be called from `Consume(..)` itself in a skipped flow.
+
+> * Ogi's core contains a `transparent` Transformer, that can be used as a clean pass-through as well. All Consumer in Ogi's core or official plugin repo.. follow default flow, and can utilize it if need be.
 
-* [in-built workflows available](./types)
+> * One can have combination of say Kafka consumer with kubernetes-log transformer alongwith custom plug-in of Google Cloud Datastore for cold storage of logs.
+>
+> * This also gives capability to write a producer sending output to more than one output sinks in same flow to achieve replication.
 
-* [what is a golang plug-in and how can I write one for Ogi](./plugins)
+* The bi-directional flow, allows the consumer/transformer to recieve some info on flow and take further steps. For e.g.
+
+> Examples:
+>
+> * An HTTP Service Consumer, this allows one to get a return value on action taken and pass it back as HTTP Response.
+>
+> * In case of failure at Transformer or Producer, can persist the actual state at which the flow failed and build a state machine to resume it for only what's left to be done.
 
 ---
 
+**WIP**
+
+#### Concepts
+
 #### Examples
 
-* [a very simple usecase to understand design and writing plugins](./example-usecase-01.md)
+#### Quickstart
 
 ---
 
-#### Quickstart
+### How It Started?
 
-* [get it running locally](./run-locally)
+In 2018, initially written to fan-out bulk topic `labels[app:appname]` tagged logs pushed from Kubernetes to Kafka, into `app` specific topics.
 
-* [set of configurations to make ogi work to specific behavior](./config-set)
+> There were different Kafka sinks across different clusters, to be funneled to service specific topics to a central sink for varied Observability & Analytics purpose.
+>
+> This came to be as regular PoC on a Sunday, in Production on Monday.. a simple env configurable data cruncher binary. Simple & Sturdy. But sent to the dark dungeon of Prod PoCs left to run.
 
 ---
-