Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

ManyLinux builds with FFmpeg shared libraries. #50

Closed
techdragon opened this issue Oct 4, 2017 · 26 comments
Closed

ManyLinux builds with FFmpeg shared libraries. #50

techdragon opened this issue Oct 4, 2017 · 26 comments
Labels
enhancement

Comments

@techdragon
Copy link
Contributor

It should be possible, I'll take a look after I get #49 finished.
@techdragon techdragon mentioned this issue Oct 11, 2017
Closed
@abitrolly
Copy link

These are current settings. Nothing is compiled in statically. Installing FFmpeg doesn't help, so I guess dynamic linking won't work too.

>>> import cv2
>>> print(cv2.getBuildInformation())

General configuration for OpenCV 3.3.0 =====================================
  Version control:               3.3.0

  Platform:
    Timestamp:                   2017-09-03T09:41:40Z
    Host:                        Linux 4.4.0-83-generic x86_64
    CMake:                       3.9.0
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/gmake
    Configuration:               Release

  CPU/HW features:
    Baseline:                    SSE SSE2 SSE3
      requested:                 SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2
      SSE4_1 (2 files):          + SSSE3 SSE4_1
      SSE4_2 (1 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (1 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (5 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (7 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2

  C/C++:
    Built as dynamic libs?:      NO
    C++ Compiler:                /opt/rh/devtoolset-2/root/usr/bin/c++  (ver 4.8.2)
    C++ flags (Release):         -fPIC   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fPIC   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /opt/rh/devtoolset-2/root/usr/bin/cc
    C flags (Release):           -fPIC   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections  -msse -msse2 -msse3 -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fPIC   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections  -msse -msse2 -msse3 -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):
    Linker flags (Debug):
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          /opt/Qt4.8.7/lib/libQtGui.so /opt/Qt4.8.7/lib/libQtTest.so /opt/Qt4.8.7/lib/libQtCore.so /lib64/libz.so stdc++ dl m pthread rt
    3rdparty dependencies:       ittnotify libprotobuf libjpeg libwebp libpng libtiff libjasper IlmImf

  OpenCV modules:
    To be built:                 core flann imgproc ml objdetect photo video dnn imgcodecs shape videoio highgui superres features2d calib3d stitching videostab python2
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev java python3 ts viz

  GUI: 
    QT 4.x:                      YES (ver 4.8.7 EDITION = OpenSource)
    QT OpenGL support:           NO
    OpenGL support:              NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /lib64/libz.so (ver 1.2.3)
    JPEG:                        libjpeg (ver 90)
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         build (ver 1.6.24)
    TIFF:                        build (ver 42 - 4.0.2)
    JPEG 2000:                   build (ver 1.900.1)
    OpenEXR:                     build (ver 1.7.1)
    GDAL:                        NO
    GDCM:                        NO

  Video I/O:
    DC1394 1.x:                  NO
    DC1394 2.x:                  NO
    FFMPEG:                      NO
      avcodec:                   NO
      avformat:                  NO
      avutil:                    NO
      swscale:                   NO
      avresample:                NO
    GStreamer:                   NO
    OpenNI:                      NO
    OpenNI PrimeSensor Modules:  NO
    OpenNI2:                     NO
    PvAPI:                       NO
    GigEVisionSDK:               NO
    Aravis SDK:                  NO
    UniCap:                      NO
    UniCap ucil:                 NO
    V4L/V4L2:                    NO/NO
    XIMEA:                       NO
    Xine:                        NO
    Intel Media SDK:             NO
    gPhoto2:                     NO

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Use Intel IPP:               NO
    Use Intel IPP IW:            NO
    Use Intel IPP Async:         NO
    Use VA:                      NO
    Use Intel VA-API/OpenCL:     NO
    Use Lapack:                  NO
    Use Eigen:                   NO
    Use Cuda:                    NO
    Use OpenCL:                  YES
    Use OpenVX:                  NO
    Use custom HAL:              NO

  OpenCL:                        <Dynamic loading of OpenCL library>
    Include path:                /io/opencv/3rdparty/include/opencl/1.2
    Use AMDFFT:                  NO
    Use AMDBLAS:                 NO

  Python 2:
    Interpreter:                 python (ver 2.7.13)
    Libraries:                   
    numpy:                       /opt/python/cp27-cp27mu/lib/python2.7/site-packages/numpy/core/include (ver 1.11.1)
    packages path:               /opt/python/cp27-cp27mu/lib/python2.7/site-packages

  Python 3:
    Interpreter:                 NO

  Python (for build):            python

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Matlab:                        Matlab not found or implicitly disabled

  Tests and samples:
    Tests:                       NO
    Performance tests:           NO
    C/C++ Examples:              NO

  Install path:                  /usr/local

  cvconfig.h is in:              /io/opencv/build
-----------------------------------------------------------------

@gitcoinbot
Copy link

This issue now has a funding of 0.045 ETH (21.0 USDT) attached to it.

  • If you would like to work on this issue you can claim it here.
  • If you've completed this issue and want to claim the bounty you can do so here

@abitrolly
Copy link

I've added some ether as a bonus to make it happen. )

@skvark
Copy link
Member

skvark commented Nov 26, 2017

None of the current builds are linked/built with FFmpeg. I will build new layers with FFmpeg and libjpeg-turbo (#52) to the Docker images as soon as I find some more free time. Thanks for the bonus 👍

@owocki
Copy link

owocki commented Nov 27, 2017

@abitrolly thanks :)

@owocki
Copy link

owocki commented Dec 4, 2017 •
edited
Loading

hi from gitcoin.co -- doing my best to source someone to turn this around. will keep yall posted

@skvark
Copy link
Member

skvark commented Dec 14, 2017

Just a quick status update. I have now built new layers which include FFmpeg to the manylinux containers. The OpenCV build is not working yet (linker does not find FFmpeg libs) but I hope I can get that fixed in the next few days.

@abitrolly
Copy link

The OpenCV build is not working yet (linker does not find FFmpeg libs)

Is there more information about that? Upstream bug maybe?

@skvark
Copy link
Member

skvark commented Dec 15, 2017

Current ouput from Travis:

--     FFMPEG:                      YES
--       avcodec:                   YES (ver 58.6.103)
--       avformat:                  YES (ver 58.3.100)
--       avutil:                    YES (ver 56.5.100)
--       swscale:                   YES (ver 5.0.101)
--       avresample:                NO

Error:

[100%] Linking CXX shared module ../../lib/python3/cv2.cpython-35m-x86_64-linux-gnu.so
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: cannot find -lavcodec
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: cannot find -lavformat
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: cannot find -lavutil
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: cannot find -lswscale
collect2: error: ld returned 1 exit status
make[3]: *** [lib/python3/cv2.cpython-35m-x86_64-linux-gnu.so] Error 1
make[2]: *** [modules/python3/CMakeFiles/opencv_python3.dir/all] Error 2
make[1]: *** [modules/python3/CMakeFiles/opencv_python3.dir/rule] Error 2
make: *** [opencv_python3] Error 2

It might also be related to custom install location. Full build logs are at Travis: https://travis-ci.org/skvark/opencv-python/builds/316114040

@skvark
Copy link
Member

skvark commented Dec 15, 2017

Builds are working now. I added FFmpeg lib path to LDFLAGS.

@skvark skvark mentioned this issue Dec 15, 2017
Closed
@gitcoinbot
Copy link

The funding of 0.045 ETH (30.83 USD) attached has been claimed by @@skvark.

@@skvark, please leave a comment to let the funder (@abitrolly) and the other parties involved your implementation plan. If you don't leave a comment, the funder may expire your claim at their discretion.

@owocki
Copy link

owocki commented Dec 18, 2017

👋 @skvark

@skvark
Copy link
Member

skvark commented Dec 18, 2017

@abitrolly @owocki I'll merge these changes during this week after I have done some local tests with the packages.

@abitrolly
Copy link

@skvark great. Looking forward to see this committed. What could help to fix this sooner? Would there be less hassle if CMake could provide a better error message, like checking for libraries in advance?

@skvark
Copy link
Member

skvark commented Dec 18, 2017

Probably would get fixed sooner if I had more time to focus on this :) But I'll try to get this done today or tomorrow (and couple of other issues too). And yes, it would be a lot faster to test the build if the compilation failed instantly vs in the very end.

@owocki
Copy link

owocki commented Dec 21, 2017

great to see the progress here.. keep it up @skvark !

@skvark
Copy link
Member

skvark commented Dec 21, 2017 •
edited
Loading

The changes are already in master but I'll close #53 and #48 before release.

@hroemer
Copy link

hroemer commented Dec 22, 2017

If you prefer building the current OpenCV version for Python 3 including video support on your own locally, I provided a simple CLI tool on PyPI named cvbuilder that downloads the official sources, configures and compiles them in a virtual environment taking installed libraries into account.

@skvark
Copy link
Member

skvark commented Dec 22, 2017

Thanks. The local build support in this repository will be done via setup.py and scikit-build (a future proof tool targeting for upcoming PEP 518, see https://www.python.org/dev/peps/pep-0518/ and https://github.com/scikit-build/scikit-build).

The local build support is not yet there but first transition step will be after #60 has been merged. After that will follow also sdist so users will be able to build this package locally with pip install opencv-python if their system is missing prebuilt wheels or force an sdist build with pip install opencv-python --no-binary.

@skvark skvark closed this as completed Dec 23, 2017
@skvark
Copy link
Member

skvark commented Dec 23, 2017

@abitrolly @owocki New releases with FFmpeg should be soon at PyPI.

@owocki
Copy link

owocki commented Dec 27, 2017

@abitrolly let us know if its time to payout the bounty

@abitrolly
Copy link

abitrolly commented Dec 28, 2017 •
edited
Loading

Sorry for the delay. Christmas and all the stuff. =) Checking that it works..

$ virtualenv xxx
New python executable in /home/anatoli/a/xxx/bin/python2
Also creating executable in /home/anatoli/a/xxx/bin/python
Installing setuptools, pip, wheel...done.
$ cd xxx
$ ./bin/pip install opencv-python
Collecting opencv-python
  Downloading opencv_python-3.3.1.11-cp27-cp27mu-manylinux1_x86_64.whl (24.7MB)
    100% |████████████████████████████████| 24.7MB 32kB/s 
Collecting numpy>=1.11.1 (from opencv-python)
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: numpy, opencv-python
Successfully installed numpy-1.13.3 opencv-python-3.3.1.11

Installation is ok. Now I need to find some example to run..

@abitrolly
Copy link

I can't find anything to run right now and there is little time on my back. Info is reporting FFMPEG is compiled in, so I see no reason to delay the award anymore. )

>>> import cv2
>>> print(cv2.getBuildInformation())
General configuration for OpenCV 3.3.1 =====================================
  Version control:               3.3.1-dirty
...
  Video I/O:
    DC1394 1.x:                  NO
    DC1394 2.x:                  NO
    FFMPEG:                      YES
      avcodec:                   YES (ver 58.6.103)
      avformat:                  YES (ver 58.3.100)
      avutil:                    YES (ver 56.5.100)
      swscale:                   YES (ver 5.0.101)
      avresample:                NO
...
...

@gitcoinbot
Copy link

The funding of 0.045 ETH (30.61 USD) attached to this issue has been approved & issued to @skvark.

@skvark
Copy link
Member

skvark commented Dec 28, 2017

Thanks!

@cachitas
Copy link

I tested yesterday in Ubuntu 17.10 x86_64 with Python 3.6.3.

$ pip freeze
opencv-python==3.3.1.11
...

Before I had to link to an OpenCV installed against my Python version with FFMPEG.

Now it just works out of the box 😃 Thanks everyone!

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@owocki @skvark @techdragon @cachitas @abitrolly @hroemer @gitcoinbot