Edge cases with pre-enriched pipeline files provisioning

[lyuboxa]

Conduit pipelines and connectors can be provisioned using pipeline files. In some cases the provisioning can be done programmatically, where the pipeline config from the provisioning pkg to construct and potentially import the pipelines.

The pipeline config files have two features which can be useful:

• Validation ensure config files are valid
• Enrich will fill in the defaults and ensure connector ids are unique by appending pipeline ids as a prefix.

Programmatically (when conduit is embedded for example), a pipeline config can be imported directly in conduit via the Import(..) method. This type of provisioning is unconcerned if the pipeline has been validated or enriched, it will attempt to create it.

Contrasting behaviour is observed when conduit uses pipelines which are stored on disk. The stored pipelines will be enriched and validated before being imported in conduit.

The inconsistent behaviour can be observed, when a pipeline is provisioned by the programmatic interface enriched but also stored locally enriched. When Conduit boots the pipeline file will be enriched again, changing the IDs for the connectors and leading for their tear down and re-creation (and lose positioning state).

The observed behaviour is triggered when:

1. pipeline.Config{} is enriched
2. The passed to Import(..)
3. Then saved locally to a file, so Conduit can pick it up when it boots(*)
4. Restart conduit

* Conduit currently does not differentiate between pipelines imported via Init() (from local files) or through Import() (programmatically). On boot, conduit will the differences between locally present pipelines and pipelines in the database of type file, in which case if the local files does not exist, it will consider the pipeline in the database deleted and will remove it.

Hiding Enrich may resolve the issue, but it will need to be present in Init() when the pipeline has been provisioned.

I'd appreciate thoughts on this.