test_mmap uses cruel and unusual amounts of disk space

[larryhastings]

BPO	23953
Nosy	@pitrou, @larryhastings

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-04-14.18:17:35.554>
labels = ['tests', 'performance']
title = 'test_mmap uses cruel and unusual amounts of disk space'
updated_at = <Date 2015-04-15.03:37:31.362>
user = 'https://github.com/larryhastings'

bugs.python.org fields:

activity = <Date 2015-04-15.03:37:31.362>
actor = 'larry'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2015-04-14.18:17:35.554>
creator = 'larry'
dependencies = []
files = []
hgrepos = []
issue_num = 23953
keywords = []
message_count = 6.0
messages = ['240974', '241044', '241045', '241056', '241058', '241075']
nosy_count = 2.0
nosy_names = ['pitrou', 'larry']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue23953'
versions = ['Python 3.4', 'Python 3.5']

Linked PRs

• gh-68141: Guard test_mmap.LargeMmapTests.* with a timeout #101774

[larryhastings]

My laptop is running 64-bit Linux (14.10). It has 4.6GB of free disk space. Naturally that's not enough to run test_mmap.

When I run the test suite, test_mmap consumes all available disk space, then fails. (Hopefully freeing all its temporary files!) If I used "-j" to run more than one test at a time, this usually means *other* tests fail too, because I'm running multiple tests in parallel and there are plenty of other tests that require, y'know... any disk space whatsoever.

The documentation for the test suite ("./python -m test -h") says that "-u largefile" allows tests that use more than 2GB. Surely test_mmap's delicious 800PB tests should be marked largefile-enabled-only?

I'd like to see this fix backported to 3.4 too. And if 2.7 shows this behavior, maybe there too.

[pitrou]

What is your filesystem?

[pitrou]

By the way, I'm pretty sure test_mmap doesn't require 4GB of disk space here - Ubuntu 14.10 -, since it runs in only 0.071s; also it runs fine from a partition with only 3GB free space.

[larryhastings]

ZFS.

[pitrou]

Perhaps ZFS doesn't support sparse files? Or perhaps there's another way to convince it to create a sparse file?

How long does test_mmap take to run on your machine?

[larryhastings]

Wall time was 3 minutes 38 seconds.

% time ./python -m test test_mmap
[1/1] test_mmap
test test_mmap failed -- multiple errors occurred; run in verbose mode for details
1 test failed:
test_mmap
./python -m test test_mmap 0.40s user 22.41s system 10% cpu 3:37.40 total

[iritkatriel]

Is there anything we can do about this? Should we skip this test on some file systems?

[arhadthedev]

Probably, we can run this test mmap() in a separate thread and kill&skip it on timeout.