Unsupported APPEND mode for custom datasets

[raphi]

Hi team,

I'm using the Algolia plugin to sync my data once processed with Dataiku. However, I'm not able to use the option Append instead of overwrite

Is there anyway to circumvent this limitation?
Can we allow append mode on Algolia datasets?
I'm down to send a PR if you point me in the right direction!

Thanks a lot

cc. @cstenac @jereze

[alexcombessie]

Hi Raphi,
To best help you, I would like to understand the use case you are trying to address. Could you detail your goal and send me a screenshot of your flow? Are you using scenarios to automate it? Cheers,
Alex

[raphi]

Hi @alexcombessie ,

I'd like from two data pipeline to push data into the same Algolia index. Here is a truncated screenshot of our current pipeline:

We apply different process to different types of data, but at the end, we want to send both into the same Algolia index. Does it make sense?

No we are not using scenarios.

Thanks for your help!

[raphi]

Hi @alexcombessie

sorry to ping you again, but this is a real blocker for us... and we are stuck on it. Anything on our side we can do?

Please let me know, thanks

[alexcombessie]

How about trying to sync to a 'dummy' copy of your dataset in APPEND MODE then sync that to algolia. This may work with an extra step, but without code.

[raphi]

The issue is that my two dataset that I want to sync to Algolia don't have the schema. Algolia being schemaless, we can have different records with different attributes. Syncing the two datasets into one force to have the same schema / columns.

Also, that doesn't fix the issue that for every sync with Algolia, it's deleting all objects before indexing them, which is not a behaviour we expect in that case.

[alexcombessie]

Have you tried to partition your two input datasets, stack them and then sync the partitioned stack to Algolia? You may have to adapt the plugin code to take into account partitioning.

[raphi]

We cannot use the partitioned datasets :/

[alexcombessie]

Are you using the free edition of Dataiku? Partitioning management is included in our Enterprise license.

[raphi]

Yes we are currently using the free edition.

[alexcombessie]

Partitioning will be the simplest way for you to manage this flow without developing your own custom code. As highlighted in https://www.dataiku.com/dss/editions/, the free edition is not the best suited for this type of advanced workflows. I suggest you speak to [email protected] to discuss Enterprise licensing.