1BRC in Lua 5.1 and LuaJIT

[MikuAuahDark]

1BRC in Lua 5.1 and LuaJIT

Sorry for the late!

GitHub Gist: https://gist.github.com/MikuAuahDark/8cdbe5827a32e65157005e7163a4b9cc

Description

This is 1 billion row challenge done using Lua 5.1 and LuaJIT. It only uses standard library. It also exploits io.popen to run it in parallel by splitting it in chunks. After the workers are done, the worker writes the internal result to stdout which then combined together by the main program to generate the desired, properly formatted output.

For LuaJIT, when parsing the temperature numbers, replacing tonumber with data-aware conversion function result in slight performance improvement. Also no FFI is used.

Unfortunately it cannot run in vanilla Lua 5.1 in Windows without slight patch to the file:seek() function. Lua 5.1 file:seek() in Windows can't handle files larger than 2GB so slight modification to this function is needed (or alternatively, run with 100 million rows instead of 1 billion). More information about the exact patch is in the GitHub gist. Fortunately LuaJIT does not have this issue.

Benchmark Results

Ryzen 5 5600H 6 Core 12 Threads CPU. 16GB DDR4-3200 Dual-channel RAM. Windows 11 23H2. CPU clock averaged on 4 GHz at all cores during run. Runtime measurement is done as per the evaluation method (run 5 times, fastest and slowest runtime discarded, the rest 3 runtime averaged) using Powershell Measure-Command.

	LuaJIT	Lua 5.1
Runtime	102.708 seconds	224.211 seconds
Memory	4.5MB	3.5MB
Worker Memory	4.9MB/worker	3.7MB/worker
Workers	12	12
Total Memory	63.3 MB	47.9 MB
Rows	1000000000	1000000000

Versions

Microsoft Windows [Version 10.0.22631.3235]
(c) Microsoft Corporation. All rights reserved.

C:\Users\MikuAuahDark>lua51 -v
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio

C:\Users\MikuAuahDark>luajit -v
LuaJIT 2.1.1700008891 -- Copyright (C) 2005-2023 Mike Pall. https://luajit.org/

Potential Improvements

For Lua 5.1, I don't think there's any room left for improvement without resorting to 3rd-party libraries.

For LuaJIT, if sacrificing Lua 5.1 compatibility is not a concern, then:

• Replacing io-related functions with FFI that calls the C functions (fopen, fgets, _fseeki64, etc.).
• Using FFI data structures for the internal data representation instead of plain Lua table.
• Potentially replacing f:lines() (or fgets) with custom line-by-line reading routines written in pure Lua.
• Together with above point, use file mapping instead of traditional file open functions using FFI.
• Maybe tuning the garbage collector?

If you have any comments or suggestion please let me know. Thanks.