Question - Reading record split across multiple files

[altmank]

I'm working with a data source where the representation of a single entity could be split across multiple files. So I would need to read multiple files to create the final composite entity with all of the data. I'm curious if this is something that has come up before and if there were any suggested approaches for this when using CsvHelper.

I've considered the following:
-Using CsvHelper to merge all of the CSV files together and then using a single ClassMap that represents the merged file, I could potentially encounter duplicate header fields. This is my current thought as the best potential approach.
-Using CsvHelper with a map for each file, scanning each additional file to find the correct row, extracting the record and then at the end merging them all manually. This would require a lot of custom code outside of the CsvHelper design though.

[JoshClose]

Does the second file start where the first file left off? Will it stop in the middle of a field?

[altmank]

The second file will typically just have the same record key and a different set of fields.

…

On Mon, Dec 17, 2018, 3:53 PM Josh Close ***@***.*** wrote: Does the second file start where the first file left off? Will it stop in the middle of a field? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/JoshClose/CsvHelper/issues/1193#issuecomment-448012421>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AV6-kHxA5jQtSHffjfWNK4PoEa3vSemHks5u6BJCgaJpZM4ZXMCb> .

[JoshClose]

Do the files after the 1st file have header records too and do they match the first?

[altmank]

The file after the first will have a header row where the first header column name will match the first file, and is used to represent the key of each row. The rest of the headers in the second file will then represent additional properties of the entity from the first file of the same record key. So in the scenario where I would merge the csv files manually, I would do so using the record key column to identify where to join the records. But there are edge cases where sometimes there might be a header column in the second file that isn't the record key but rather just duplicates a field in the first file. I don't control the generation of these CSV files. On Dec 17, 2018 4:01 PM, "Josh Close" <[email protected]> wrote: Do the files after the 1st file have header records too and do they match the first? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/JoshClose/CsvHelper/issues/1193#issuecomment-448014943>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AV6-kM8bsjhNw5Swra1oqTMoPUl0UDo0ks5u6BQygaJpZM4ZXMCb> .

[JoshClose]

Ohhhhh... So you need to build up a single class object across multiple files. Essentially do the equivalent of a join.

Can you give a couple row example for each file? Try and include an edge case so we can work around that.

[altmank]

Yes, you have it right.

File 1
FacilityId,Name,Type,BedCount
1,CoolFacilityName1,FacilityType1,50
2,CoolFacilityName2,FacilityType1,100

File 2
FacilityId, StaffCount, BedCount
1,20,50
2,43,100

I would ultimately want a class like this (using csv format for ease of representation here):
FacilityId,Name,Type,BedCount,StaffCount
1,CoolFacilityName1,FacilityType1,50,20
2,CoolFacilityName2,FacilityType1,100,43

If the field exists in both files, so far all of my data examples of that have them using the same value.

In my real data sets, I could have one or more files but the above example just uses 2.

[JoshClose]

How large are the files? If you can hold them all in memory, you could read all the files and do LINQ joins against them.

If memory is a concern, you could read a line from the first file, then read through all the other files row by row and matching as you find them. You could do this same thing with a batch of records from the first file too, like 100 or 1000 or something.