Cannot build a workspace containing more than 100 packages

[ghost]

Describe the bug
Launching the command catkin_make on a workspace containing more than 100 packages doesn't work. The outcome is an infinite exception printed on the console: ImportError: No module named catkin_make

To Reproduce
Steps to reproduce the behavior:

• Catkin workspace setup instructions
  • Create a workspace folder C:\ws\maplab\. Inside it, create the src folder
  • Inside the src folder run:
    • git clone https://github.com/ethz-asl/maplab --recurse
    • git clone https://github.com/ethz-asl/maplab_dependencies --recurse
  • Launch command catkin_init_workspace
  • cd C:\ws\maplab\
  • Launch command catkin_make --source .

Expected behavior
The catkin_make command should succeed, as it does on Ubuntu

Workaround
The issue is related to the fact that Catkin tries to parallelise the build process when there are more than 100 packages. As a workaround, I disabled this feature. To do so, edit the file C:\opt\python27amd64\Lib\site-packages\catkin_pkg\packages.py; inside the function find_packages_allowing_duplicates, set parallel = False (instead of parallel = len(data) > 100)

Environment information:

• OS version: Windows 10 Version 1903 (OS Build 18362.295)

• Chocolatey Versions:

  Chocolatey v0.10.15

  
  assimp 4.0.1.1809242322
  boost 1.66.0.1904102339
  bullet3 2.87.0.1809242326
  bzip2 1.0.6.1901242108
  chocolatey 0.10.15
  chocolatey-core.extension 1.3.3
  chocolatey-windowsupdate.extension 1.0.4
  console_bridge 0.4.0.1809242322
  cppunit 1.12.1.1809242322
  cppzmq 4.2.2.1811162207
  dlfcn-win32 1.1.1.1811162226
  eigen 3.3.4.1809242322
  freeimage 3.17.0.1811160037
  gazebo9 9.4.1.1903240636
  google-mock 1.8.0.1809242326
  google-test 1.8.0.1809242322
  gtk2 2.22.1.1904131648
  hg 5.1
  ilmbase 2.2.1.1811210800
  jasper 2.0.14.1811210919
  jxrlib 1.1.0.1811210809
  KB2919355 1.0.20160915
  KB2919442 1.0.20160915
  KB2999226 1.0.20181019
  KB3033929 1.0.5
  KB3035131 1.0.3
  lcms 2.8.0.1811210903
  libcurl 7.58.0.1809242322
  libflann 1.9.1.1809090720
  libgraphviz 2.41.0.1810022039
  libjpeg-turbo 1.5.3.1809242322
  liblzma 5.2.4.1811210837
  libogg 1.3.3.1809090720
  libopencv 3.4.1.1809242322
  libpcl 1.8.1.1809090720
  libpng 1.6.35.1809242322
  libraw 0.19.0.1811210804
  libtbb-dev 2018.6.0.1812011750
  libtheora 1.1.1.1809090720
  libwebp 0.6.1.1811210731
  log4cxx 0.10.0.1809242322
  lz4 1.8.1.1809242322
  nodejs 12.8.0
  nodejs.install 12.8.0
  ogre 1.10.11.1901312214
  openexr 2.2.1.1811210832
  openjpeg 2.3.0.1811210736
  openssl 1.1.1.1809242322
  pkg-config 0.29.2.1809242322
  poco 1.8.1.1809242322
  protobuf 3.6.1.1811130545
  pyqt5 5.10.1.1809242322
  pyside2 5.10.1.1809242322
  qt5-sdk 5.10.1.1903182340
  qwt 6.1.3.1811212304
  ros-catkin-tools 0.0.1.1903221831
  ros-cmake 3.11.4.1809242322
  ros-melodic-desktop 20190729.1
  ros-melodic-desktop_full 20190729.1
  ros-python2 2.7.15.1904300710
  ros-vcpkg 2018.11.23.1906112226
  sbcl 0.0.0.1809242326
  sdl 1.2.15.1809242322
  sdl_image 1.2.12.1809242322
  sip 4.19.8.1809242322
  tiff 4.0.9.1811210742
  tinyxml 2.6.2.1809242322
  tinyxml2 6.1.0.1809242322
  urdfdom 1.0.0.1809242322
  urdfdom_headers 1.0.0.1809242322
  vcpython27 9.0.30729.1809090720
  vcredist140 14.22.27821
  yaml-cpp 0.6.2.1901170034
  zeromq 4.2.5.1811162203
  zlib 1.2.11.1809242322
  75 packages installed.

[ooeygui]

I have a repro of this. It looks like the environment isn't getting inherited into the forked processes. I'm working to understand that.

[ooeygui]

There are a couple of things contributing to this issue.
First, this specific workspace includes many sub projects. This isn't an advisable practice as each time you build, the workspace needs to be crawled.

On Windows, there is an environment limit - Each process inherits the host process's environment. The total limit of the entire environment is 32,000 characters - which can be quickly exhausted by Catkin. (https://devblogs.microsoft.com/oldnewthing/20100203-00/?p=15083)

For this bug, we're going to do two things - one try to detect the environment size and warn, then two limit Windows to not parallelize.

It would be ideal if ethz-asl could break up the dependencies so that you do not have to touch them on every build.

[seanyen]

Closing it because the fix is merged by the upstream.