Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

fix multiprocessing too many open files #8904

Merged
merged 2 commits into from
Dec 2, 2017
Merged

fix multiprocessing too many open files #8904

merged 2 commits into from
Dec 2, 2017

Conversation

piiswrong
Copy link
Contributor

Description

(Brief description on what this PR is about)

Checklist

Essentials

  • Passed code style checking (make lint)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • For user-facing API changes, API doc string has been updated. For new C++ functions in header files, their functionalities and arguments are well-documented.
  • To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

  • Feature1, tests, (and when applicable, API doc)
  • Feature2, tests, (and when applicable, API doc)

Comments

  • If this change is a backward incompatible change, why must this change be made.
  • Interesting edge cases to note here

@szha szha merged commit 1dc4aea into apache:master Dec 2, 2017
@bhavinthaker
Copy link
Contributor

I have documented the following in the MXNet 1.0 Release Notes here: https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.0+Release+Notes (see this Release notes for up-to-date information)

Problem: The data-loader has a file-descriptor leak when the number of files being opened exceeds the ulimit of the user session.
Workaround: Increase the ulimit of the user session or at the OS level using any of the following steps:
$ ulimit -n unlimited (to make the fd limit as unlimited), OR
$ ulimit -n 60000 (to make the fd limit as 60,000 files assuming you have around but less than 60,000 files to be opened) OR
$ echo 65535 > /proc/sys/fs/file-max (to increase the system-wide file limit) along with the ulimit command.
Further details: https://docs.oracle.com/cd/E19623-01/820-6168/file-descriptor-requirements.html

KellenSunderland pushed a commit to KellenSunderland/incubator-mxnet that referenced this pull request Dec 13, 2017
* fix

* Update cpu_shared_storage_manager.h
zhreshold pushed a commit to zhreshold/mxnet that referenced this pull request Dec 14, 2017
* fix

* Update cpu_shared_storage_manager.h
rahul003 pushed a commit to rahul003/mxnet that referenced this pull request Jun 4, 2018
* fix

* Update cpu_shared_storage_manager.h
zheng-da pushed a commit to zheng-da/incubator-mxnet that referenced this pull request Jun 28, 2018
* fix

* Update cpu_shared_storage_manager.h
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants