csv zip importer crashing

[grahamjevon]

When using the csv zip importer, with a zip file also being imported as a file, the importer is crashing when the size of the zip file is more than approx 700MB. See this slack thread.

Incidentally, the UI add a work method also struggles with larger files. The csv zip method has traditionally been the method we use to avoid the UI's problems with larger files. Via the UI method, we have experienced larger files failing to upload, but we have also experienced larger files appearing to import files successfully only to find that the file was only partially imported (e.g. a csv file where only a fraction of the rows were successfully uploaded).

[grahamjevon]

This is still an issue on staging and production. When I tried a zip importer with a single row of metadata and a single 5GB file, the importer failed. After about a minute, the Google Chrome browser displays the following error message:

There is no record of the importer in the importer history so I cannot share a link to the importer url.

I have confirmed that this error is not related to the csv metadata as I have successfully imported the metadata only. The error occurs when I try to import the csv with a 5GB file (using the zip method). I wanted to try Browse Everything, but the AWS files are not showing when I select "Add Cloud Files".

[cziaarm]

see #534 and #518