Streaming data in Metadata Injection

[dave-csc]

Hello,

I'm posting here with an actual help request (such as those filed in the Hop mailing list... BTW, is it still active?).

I set up a simple "to be injected" pipeline with a Row Normalizer linked to a Dummy transform, the latter one is needed to forward data to the "injector" (parent) pipeline.

In the parent pipeline I generate some fake data to be streamed into the Row Normalizer, and use a static Data grid to specify the fields to normalize them. You can see both pipelines in the attached file below.

The pipeline isn't working as expected: I got the error Unable to find field from upstream transform: FirstName in the log, despite FirstName is specified both as a field name in the streamed data, and in the Data grid as field to normalize.

Am I messing up something in the configuration, or is it a bug that needs to be reported as an issue?

Thanks in advance for your support.

[Next question: Why don't you use directly the Row Normalizer without injection? In the actual scenario I'm developing, I'm not aware of the metadata to be streamed, except for some fields that need to be present. So those data would be streamed into the Row Normalizer, and the field table would be injected by generating the field list with a Metadata structure of stream, after removing those I know are needed for "grouping". Hence, the name "Anti-normalizer" 😃 ]

---

Attached file:
anti-normalizer.zip

[enricomariam42]

Hi, put your data (Fake data transform) in the template pipeline (_inj-anti-normalizer.hpl) and use the parent ones (_test_inj-anti-normalizer.hpl) to inject the metadata only.
Take a look at the metadata-injection/read-file-template.hpl and metadata-injection/read-and-inject-metadata.hpl in the Sample project.
In the "ETL metadata injection" use "Options" "Optional target file" to easy debug.

[dave-csc]

Hi @enricomariam42,

I used "Fake data" as a generic input transform, it could actually be any transform with any kind of stream traversing it (check also the "Next question" above). So probably putting it in the template pipeline is not a good solution... I'll try anyway to debug with the target file.

Thanks anyway for answering :)

[enricomariam42]

Hi, @dave-csc,
put all the transforms needed to play with data in the template pipeline and use the parent one to fill the fields, not to pass the data.
Check the examples carefully.

[dave-csc]

Update: I managed to get the "Anti-normalizer" transform as I wished, after struggling with a possible bug in Data grid and Metadata Injection transforms (check issue #4525 for details and workaround).

You can find the result in the attached file below: you can import the files _inj-anti-normalizer.hpl and anti-normalizer.hpl in your projects, and use the latter one as a sub-pipeline with the Simple Mapping transform (check the file _test-anti-normalizer.hpl for the needed parameter specifications: grouping fields, key field name, value field name).

---

Attached file:
anti-normalizer-v1.0.zip