Optimizations, low ram, ssd and other stuff

[wsobrinho]

I think BladeBit is fantastic because it's kind of eco-friendly, doing a plot in RAM only and avoiding burning the SSD was a great idea. However, the memory requirement can be a hindrance to the app's success for the general public.

Taking into account that the system has already been designed to run entirely on RAM obviously any alternative must have a much lower performance, however, I think there are always several ways to get the same result, so it costs nothing to present some ideas even if unfeasible at first.

In short, the plotting algorithm creates a table 1 and from it manipulates tables in pairs up to table 7 and then manipulates from 7 to 2. Would it be possible, for example, to save the ram in phase 1 while processing table 3, could it be possible to download table 2 in parallel to the disk? and when I'm processing table 4, unload table 3 and when I'm in backpropagation can I unload table 6 while I process table 5 and load table 4 from disk?

In theory, although this will bring poor performance, depending on how much it is feasible to remove from the ram, the process itself could create two or more plots calculating the times of each phase after a first execution, increasing the parallelism and gaining more plots per hour/day.

Another optimization would be to have an option to use the disk as storage to do heavy sorting operations where the user could map part of the disk as a ramdisk and another using ssd or hdd in raid0

Apparently (may I be mistaken) the encryption algorithm seems to be pretty fast, and the data was written to a database? I know it sounds crazy but databases are experts in indexing and referencing, even if the amount of data is huge, how would a database behave? Remembering that some banks implement native functions, triggers, insertion ordering, etc. Anyway, these are just ideas.

And you, what do you think? Give your ideas, it doesn't have to be technical, for example, who knows how to create a channel on discord? are we going to move the community?