-
Notifications
You must be signed in to change notification settings - Fork 5
Fast Reporting Experiments
With a super-fast model, it's easy to get to the point where reporting is one of the most time-consuming parts of the sim. For a spatial model, we will almost certainly want spatial-resolution reporting, which means more output.
I'm in the process of doing a bunch of experiments to see if I can shave off 20% of the runtime that seems to be associated with writing output to disk.
Here are some of the design choices and some experiments results associated with them:
Disabling reporting altogether isn't an option for most cases but it's worth doing once to see how much time is spent on it. In my testing, it's about 100s out of a 400s simulation.
If we have a burnin period which we know we don't need to include in any analysis, the easiest thing to do it set a "reporting_start" value to limit when we even start reporting. The only downside here is that it adds a dreaded "config param".
It makes sense that converting binary data to human-readable strings all the time would take long, but in my experiments, switching completely to accumulating binary data in memory and writing to disk as an npy file actually was a bit slower.
I tried writing the data to a socket instead of to disk. I set up a netcat listener which wrote to disk on the same machine. This was cool but didn't seem to be a big win, especially since the model itself now uses all cores available and so the netcat listener is still competing for the same hardware.
When writing CSV files to disk each timestep, the Python file handling lets you specify a buffer size. I experimented with setting buffering from 512k to 8MB. It definitely makes a difference. 1MB seems to be the sweet spot.
I used GPT code to have the model report to a queue on the main thread and had a listener thread which did the writing (still CSV). This worked well and seemed to squeeze out a 30s gain. It does add a little complexity.