macOS: Too many open files

[brockgs]

Summary:

When attempting to export a very large iMessages database on macOS (15.1 Developer Beta) to html, imessage-exporter (iMessage Exporter 2.0.0) crashes part way through the export process seemingly due to the number of files that are being kept open, exceeding the allowable limit set by macOS.

Error:

• thread 'main' panicked at imessage-exporter/src/exporters/html.rs:161:26:
• called Result::unwrap() on an Err value: Os { code: 24, kind: Uncategorized, message: "Too many open files" }

Command as run, and output generated:

(basicenv) ⏺ ~ % sudo imessage-exporter --format html --copy-method compatible --platform macOS --export-path iMessageBackups_HTML --no-lazy
Building cache...
[1/4] Caching chats...
[2/4] Caching chatrooms...
[3/4] Caching participants...
[4/4] Caching reactions...
Cache built!
Estimated export size: 45.33 GB
Exporting to iMessageBackups_HTML as html...
⠂ [1s] [##>-----------------] 16,929/145,914 (35,719.132/s, ETA: 4s)                                                    thread 'main' panicked at imessage-exporter/src/exporters/html.rs:161:26:
called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Uncategorized, message: "Too many open files" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
zsh: abort      sudo imessage-exporter --format html --copy-method compatible --platform macO

Diagnostic output for this iMessage database:

(basicenv) ⏺ ~ % imessage-exporter --diagnostics
Building cache...
[1/4] Caching chats...
[2/4] Caching chatrooms...
[3/4] Caching participants...
[4/4] Caching reactions...
Cache built!

iMessage Database Diagnostics

Message diagnostic data:
    Total messages: 145914
    Messages not associated with a chat: 3818
    Messages belonging to more than one chat: 17
Attachment diagnostic data:
    Total attachments: 25764
        Data referenced in table: 45.30 GB
        Data present on disk: 55.90 GB
    Missing files: 141 (1%)
        No path provided: 107
        No file located: 34
Thread diagnostic data:
    Chats with no handles: 2
Global diagnostic data:
    Total database size: 284.44 MB
    Duplicated contacts: 108
    Duplicated chats: 63
Done!

Details of computer used to run imessage-exporter:

• Computer Model: MacBookPro18,2
• RAM (GB): 64.0
• CPU Cores: 10
• Logical Cores: 10
• GPU Cores: 32
• SoC: Apple M1 Max
• OS Version: 15.1

[ReagentX]

Thank you for the very detailed report! I wonder if macOS Sequoia changed anything here. I will investigate this as part of #273.

The crash appears here:

imessage-exporter/imessage-exporter/src/exporters/html.rs

Lines 157 to 161 in 7b959af

	let file = File::options()
	.append(true)
	.create(true)
	.open(path.clone())
	.unwrap();

[is-yusuf]

Had the same issue on Sonoma 14.5
Resolved it by running
ulimit -n 1048
And if it that one doesn't work, then it is a kernel config change using
sudo sysctl -w kern.maxfiles=<big number>
sudo sysctl -w kern.maxfilesperproc=<big number>

Edit: forgot I was using my other laptop which was running Sonoma

[ReagentX]

Did macOS Sequoia change something about the maximum allowed file handles? I haven't seen any official documentation to this effect.

[is-yusuf]

Did macOS Sequoia change something about the maximum allowed file handles? I haven't seen any official documentation to this effect.

I edited my comment. I forgot I was running it on my Sonoma mac.
The cause could be the newest version of Rust since I just installed it before running the script.

[ReagentX]

Prior to #281, imessage-exporter opened and closed each file when it would write to it; subsequently it keeps the file handles open for the duration of the export. This was a pretty substantial performance increase, but I think it is causing this issue now.

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       10666
-n: file descriptors                256

This appears to be the default on macOS; I am guessing that the limit of 256 is causing this.

[ReagentX]

Interestingly, I get a different value when I run it in the VSCode shell, which is probably why I haven't run into this myself:

Versus in my normal shell: