Capture pod doesn´t start properly #2

ferrandinand · 2019-04-05T09:53:50Z

I cannot start properly the capturer. Seems that sysdig pod doesn´t generate gz because seems that it is not able to start.
Is there any kind of incompatibilty?

Context
Running capture on CoreOS Container Linux by CoreOS 2023.4.0 (Rhyolite)
Nodes on AWS.
Kubernetes version v1.13.4

Command
kubectl capture kube-proxy-z4g6c -ns kube-system -M 10 --snaplen 256

* Setting up /usr/src links from host
ls: cannot access '/host/usr/src': No such file or directory
* Unloading sysdig-probe, if present
* Running dkms install for sysdig

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area.....
make -j4 KERNELRELEASE=4.19.23-coreos-r1 -C /lib/modules/4.19.23-coreos-r1/build M=/var/lib/dkms/sysdig/0.25/build..........(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.19.23-coreos-r1 (x86_64)
Consult /var/lib/dkms/sysdig/0.25/build/make.log for more information.
* Running dkms build failed, dumping /var/lib/dkms/sysdig/0.25/build/make.log
DKMS make.log for sysdig-0.25 for kernel 4.19.23-coreos-r1 (x86_64)
Fri Apr  5 09:40:27 UTC 2019
make: Entering directory '/host/lib/modules/4.19.23-coreos-r1/build'
  CC [M]  /var/lib/dkms/sysdig/0.25/build/main.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/dynamic_params_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/fillers_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/flags_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/ppm_events.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/ppm_fillers.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/event_table.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/main.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/main.o'
make[3]: *** Waiting for unfinished jobs....
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/ppm_fillers.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/ppm_fillers.o'
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/ppm_events.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/ppm_events.o'
make[2]: *** [/host/lib/modules/4.19.23-coreos-r1/source/Makefile:1521: _module_/var/lib/dkms/sysdig/0.25/build] Error 2
make[1]: *** [Makefile:146: sub-make] Error 2
make: *** [Makefile:24: __sub-make] Error 2
make: Leaving directory '/host/lib/modules/4.19.23-coreos-r1/build'
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.19.23-coreos-r1
Found kernel config at /proc/config.gz
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.25-x86_64-4.19.23-coreos-r1-03bf994bd8b87756106f34511fc1aadb.ko
Download failed, consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
* Capturing system calls
Unable to load the driver
error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.
----------------------
Event           #Calls
----------------------

The text was updated successfully, but these errors were encountered:

CommanderJax · 2019-04-05T13:28:07Z

I encountered the same or a similar error, here is the log, hope it helps;

kc logs -f capture-prometheus-kube-prometheus-0-1554470360
* Setting up /usr/src links from host
* Unloading sysdig-probe, if present
* Running dkms install for sysdig

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area........
make -j56 KERNELRELEASE=4.15.0-44-generic -C /lib/modules/4.15.0-44-generic/build M=/var/lib/dkms/sysdig/0.25/build........................(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.15.0-44-generic (x86_64)
Consult /var/lib/dkms/sysdig/0.25/build/make.log for more information.
* Running dkms build failed, dumping /var/lib/dkms/sysdig/0.25/build/make.log
DKMS make.log for sysdig-0.25 for kernel 4.15.0-44-generic (x86_64)
Fri Apr  5 13:19:45 UTC 2019
make: Entering directory '/host/usr/src/linux-headers-4.15.0-44-generic'
  CC [M]  /var/lib/dkms/sysdig/0.25/build/main.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/dynamic_params_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/fillers_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/flags_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/ppm_fillers.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/ppm_events.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/event_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/syscall_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/ppm_cputime.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:332: /var/lib/dkms/sysdig/0.25/build/fillers_table.o] Error 4
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [scripts/Makefile.build:332: /var/lib/dkms/sysdig/0.25/build/syscall_table.o] Error 4
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:332: /var/lib/dkms/sysdig/0.25/build/ppm_fillers.o] Error 4
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:332: /var/lib/dkms/sysdig/0.25/build/ppm_events.o] Error 4
make: *** [Makefile:1551: _module_/var/lib/dkms/sysdig/0.25/build] Error 2
make: Leaving directory '/host/usr/src/linux-headers-4.15.0-44-generic'
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.15.0-44-generic
Found kernel config at /host/boot/config-4.15.0-44-generic
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.25-x86_64-4.15.0-44-generic-a298fcd69ad192c225fe536a5c6ad115.ko
Download succeeded, loading module
* Capturing system calls
Unable to load the driver
error mapping the ring buffer for device /host/dev/sysdig51
----------------------
Event           #Calls
----------------------

nestorsalceda · 2019-04-08T10:36:49Z

Woah, what a detailed description!

Right now, the capture plugin only supports the old kernel module loading. This is not well supported for CoreOS, it needs the eBPF probe. Next feature I'm planning to do is to enable the eBPF support.

I will keep you updated.

ferrandinand · 2019-04-08T11:55:25Z

Great, I am looking forward seeing it in action.
Keep me updated

This will make the capture plugin with COS refs #2

…ls (#3) This will make the capture plugin with COS refs #2

nestorsalceda · 2019-05-08T10:25:44Z

I just merged the eBPF support for kubectl capture.

You will need to run the command using --ebpf flag:

kubectl capture kube-proxy-z4g6c -ns kube-system  --ebpf -M 10 --snaplen 256

Please, give a try @ferrandinand and thanks for reporting :)

ferrandinand · 2019-05-27T06:35:08Z

Working like a charm.
So great seing it in action with sysdig-inspect.

nestorsalceda pushed a commit that referenced this issue Apr 11, 2019

Make a capture using ebpf instead of the kernel module

d97edc5

This will make the capture plugin with COS refs #2

nestorsalceda pushed a commit that referenced this issue May 8, 2019

Add support to eBPF instead of Kernel Module for capturing system cal…

1bebf84

…ls (#3) This will make the capture plugin with COS refs #2

nestorsalceda closed this as completed May 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capture pod doesn´t start properly #2

Capture pod doesn´t start properly #2

ferrandinand commented Apr 5, 2019 •

edited

Loading

CommanderJax commented Apr 5, 2019 •

edited

Loading

nestorsalceda commented Apr 8, 2019 •

edited

Loading

ferrandinand commented Apr 8, 2019

nestorsalceda commented May 8, 2019

ferrandinand commented May 27, 2019

Capture pod doesn´t start properly #2

Capture pod doesn´t start properly #2

Comments

ferrandinand commented Apr 5, 2019 • edited Loading

CommanderJax commented Apr 5, 2019 • edited Loading

nestorsalceda commented Apr 8, 2019 • edited Loading

ferrandinand commented Apr 8, 2019

nestorsalceda commented May 8, 2019

ferrandinand commented May 27, 2019

ferrandinand commented Apr 5, 2019 •

edited

Loading

CommanderJax commented Apr 5, 2019 •

edited

Loading

nestorsalceda commented Apr 8, 2019 •

edited

Loading