Placeholder for data registry capability

[alexanderdean]

This is based on @brapse's idea of SoundCloud's internal "Data AMT" service which provides a registry of what data lives where. This is distinct (but complementary to) a schema registry which governs your data types.

To give a Snowplow example, in the data registry you could record:

• Where the raw collector payloads live
• Where the enriched events live
• Where the shredded data lives in Redshift

It can be more than just locations of course - relevant metadata, provenance, etc

[andybritz]

Is this analogous to the LinkedIn WhereHows project?

[alexanderdean]

Yes @andybritz ! It's funny because at Crunch conf there was a LinkedIn talk straight after @brapse's which talked about WhereHows.

We may be able to just use WhereHows instead of adding this into Iglu - it needs further investigation...

[gregbonnette]

May be worth looking at Atlas too.

http://atlas.incubator.apache.org/

[alexanderdean]

Yes, have been meaning to take a look at Atlas! Thanks Greg.