Download/upload to LinDA Workbench all rows

[dipapaspyros]

At this point, only the first 10 rows seem to be available when I try to download as a file or create a datasource at the end of the transformation

[bbrand84]

works

[dipapaspyros]

In the online version, all lines of the CSV/Excel file are loaded in the browser in each step. This makes the engine unusable for anything but really small files (I experienced serious performance issues with a 44 KB CSV file).
I would propose the following workflow:

• When the user uploads the file, all rows are saved on server side (preferably not in the Session, as this would render the server unusable in real world scenarios with multiple excel files at the same time).
• The user is then presented in each step with a small file preview (e.g 10 rows).
• When the transformation is finally created, it will be applied in the whole file. As a result the overhead from large files will only exist when the actual transformation occurs, and not in every step of the workflow.
  Generally, I believe that for production the whole file should never be in (server) memory at once if it's over a certain limit (e.g 10 MB)

[bbrand84]

The performance issues, which I think occured mainly in the object step, were resolved. I tried the 44k asthma observations file. But I'm still working on a better performance in this step.

But yes, the whole file is delivered to the frontend in every step since we calculate everything live in the frontend for better ux. I will see if theres room for optimization.