Changes in configuration parameters for the pipeline

[erincon01]

Hello.

I am not sure if this is a bug or not.
I share this because it might be useful for others.

I was having issues when converting the csv to parquet because the process did not put the files in the right folder (mandatory for hte Power BI Reports).

I had to change two set variables from the msexports_transform pipeline:

• The set scope function to: @replace(split(pipeline().parameters.folderName,split(pipeline().parameters.folderName, '/')[sub(length(split(pipeline().parameters.folderName, '/')), 2)])[0],'msexports','ingestion')
• The set date function to: @split(pipeline().parameters.folderName, '/')[sub(length(split(pipeline().parameters.folderName, '/')), 1)]

These are the data paths I get now:

• https://<storage_account>.blob.core.windows.net/msexports/<subscription_id>/Export-daily-to-date/<from_date>-<to_date>/Export-daily-to-date_<guid_data>.csv
• https://<storage_account>.blob.core.windows.net/ingestion/<subscription_id>/<from_date>-<to_date>/amortizedcost/Export-daily-to-date_<guid_data>.parquet

Reason for the change was because the pipeline was moving the data to:

• https://<storage_account>.blob.core.windows.net/<subscription_id>/amortizedcost/Export-daily-to-date_<guid_data>.parquet

Regards,

[flanakin]

@erincon01, if files aren't moved to ingestion, then there's another issue. I believe this happens when you don't set partitioning on the export. Could that be the issue in your case?

[erincon01]

The subscription I tested didn't allow partitioning the data.
I played with the sub operator to find where data should be moved.

[flanakin]

Is this a CSP subscription? I've never heard of partitioning not being available. That's concerning. This sounds like there's an older version of exports running.

[erincon01]

I close this discussion as resolved. In this particular case I was not using the FOCUS exports.