You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is 1 billion row challenge done using Lua 5.1 and LuaJIT. It only uses standard library. It also exploits io.popen to run it in parallel by splitting it in chunks. After the workers are done, the worker writes the internal result to stdout which then combined together by the main program to generate the desired, properly formatted output.
For LuaJIT, when parsing the temperature numbers, replacing tonumber with data-aware conversion function result in slight performance improvement. Also no FFI is used.
Unfortunately it cannot run in vanilla Lua 5.1 in Windows without slight patch to the file:seek() function. Lua 5.1 file:seek() in Windows can't handle files larger than 2GB so slight modification to this function is needed (or alternatively, run with 100 million rows instead of 1 billion). More information about the exact patch is in the GitHub gist. Fortunately LuaJIT does not have this issue.
Benchmark Results
Ryzen 5 5600H 6 Core 12 Threads CPU. 16GB DDR4-3200 Dual-channel RAM. Windows 11 23H2. CPU clock averaged on 4 GHz at all cores during run. Runtime measurement is done as per the evaluation method (run 5 times, fastest and slowest runtime discarded, the rest 3 runtime averaged) using Powershell Measure-Command.
LuaJIT
Lua 5.1
Runtime
102.708 seconds
224.211 seconds
Memory
4.5MB
3.5MB
Worker Memory
4.9MB/worker
3.7MB/worker
Workers
12
12
Total Memory
63.3 MB
47.9 MB
Rows
1000000000
1000000000
Versions
Microsoft Windows [Version 10.0.22631.3235]
(c) Microsoft Corporation. All rights reserved.
C:\Users\MikuAuahDark>lua51 -v
Lua 5.1.5 Copyright (C) 1994-2012 Lua.org, PUC-Rio
C:\Users\MikuAuahDark>luajit -v
LuaJIT 2.1.1700008891 -- Copyright (C) 2005-2023 Mike Pall. https://luajit.org/
Potential Improvements
For Lua 5.1, I don't think there's any room left for improvement without resorting to 3rd-party libraries.
For LuaJIT, if sacrificing Lua 5.1 compatibility is not a concern, then:
Replacing io-related functions with FFI that calls the C functions (fopen, fgets, _fseeki64, etc.).
Using FFI data structures for the internal data representation instead of plain Lua table.
Potentially replacing f:lines() (or fgets) with custom line-by-line reading routines written in pure Lua.
Together with above point, use file mapping instead of traditional file open functions using FFI.
Maybe tuning the garbage collector?
If you have any comments or suggestion please let me know. Thanks.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
1BRC in Lua 5.1 and LuaJIT
Sorry for the late!
GitHub Gist: https://gist.github.com/MikuAuahDark/8cdbe5827a32e65157005e7163a4b9cc
Description
This is 1 billion row challenge done using Lua 5.1 and LuaJIT. It only uses standard library. It also exploits
io.popen
to run it in parallel by splitting it in chunks. After the workers are done, the worker writes the internal result to stdout which then combined together by the main program to generate the desired, properly formatted output.For LuaJIT, when parsing the temperature numbers, replacing
tonumber
with data-aware conversion function result in slight performance improvement. Also no FFI is used.Unfortunately it cannot run in vanilla Lua 5.1 in Windows without slight patch to the
file:seek()
function. Lua 5.1file:seek()
in Windows can't handle files larger than 2GB so slight modification to this function is needed (or alternatively, run with 100 million rows instead of 1 billion). More information about the exact patch is in the GitHub gist. Fortunately LuaJIT does not have this issue.Benchmark Results
Ryzen 5 5600H 6 Core 12 Threads CPU. 16GB DDR4-3200 Dual-channel RAM. Windows 11 23H2. CPU clock averaged on 4 GHz at all cores during run. Runtime measurement is done as per the evaluation method (run 5 times, fastest and slowest runtime discarded, the rest 3 runtime averaged) using Powershell
Measure-Command
.Versions
Potential Improvements
For Lua 5.1, I don't think there's any room left for improvement without resorting to 3rd-party libraries.
For LuaJIT, if sacrificing Lua 5.1 compatibility is not a concern, then:
io
-related functions with FFI that calls the C functions (fopen
,fgets
,_fseeki64
, etc.).f:lines()
(orfgets
) with custom line-by-line reading routines written in pure Lua.If you have any comments or suggestion please let me know. Thanks.
Beta Was this translation helpful? Give feedback.
All reactions