lib: user provided threads and packets - v5 #11769

jasonish · 2024-09-12T20:29:10Z

Update of #11711 to show how packet dropping might be handled.

Add library source and runmode modules. Reorganized library example to create a worker thread and replay a pcap file using the library mode. No API layer is added at this stage. Edits by Jason Ish: - fix guard - add copyright/license headers

To keep the simple example simple, move the lib based capture method example to its own example.

- the generated binaries for lib examples - LSP files - man pages

Worker threads not created by Suricata, but instead a library user should not be joined, as Suricata does not have access to their thread handle, and it may in-fact be an unjoinable thread, such as the main process. When the thread ID is 0, assume the thread is "externally" managed, but still mark is as dead to satisfy Suricata's view of the thread.

Use the more conventional "--" command line handling to separate the arguments. The first set will be passed to Suricata, and the args after "--" will be handled by the example. Currently this is a single PCAP filename, but will be extended to a list of PCAP filenames.

Also use a proper return type (ThreadVars *).

Update ThreadVars creation in lib mode to have the worker_id provided by the user.

In the library capture example, show how the packet counter can be updated.

The variable run is not needed, we can just run an infinite loop and break when needed.

Refactor TmThreadsSlotPktAcqLoop for user provided thread by breaking out the init and finish code into their own functions. For user provided threads, Suricata should not "drive" the thread, but the setup and finish code is the same. The finish function is exported so it can be called by the user application when its receive loop or equivalent is done.

And add SC prefix.

Also remove function to set the library mode. This is easy enough to do with SCRunmodeSet, and we don't want to add a specific setter for each and every runmode.

Add a release packet callback where the action can be checked for drop.

examples/lib/libcapture/main.c

src/source-lib.c

jasonish · 2024-09-12T20:34:59Z

src/source-lib.c

+    if (data == NULL) {
+        TmThreadsSetFlag(tv, THV_CAPTURE_INJECT_PKT);
+        TmThreadsCaptureHandleTimeout(tv, NULL);
+        SCReturnInt(TM_ECODE_OK);
+    }


@victorjulien Any guidelines as when a custom capture method should push something like this through?

src/source-lib.c

jasonish · 2024-09-12T20:40:27Z

src/tm-threads.c

-static void *TmThreadsSlotPktAcqLoop(void *td)
+static bool TmThreadsSlotPktAcqLoopInit(ThreadVars *tv)


@victorjulien Would like your eyes on how I broke this function up as a user driving there thread needs this to return, but then do the finish stuff later on.

codecov · 2024-09-12T21:12:35Z

Codecov Report

Attention: Patch coverage is 24.53988% with 123 lines in your changes missing coverage. Please review.

Project coverage is 82.49%. Comparing base (31bed10) to head (fe89a69).
Report is 68 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11769      +/-   ##
==========================================
- Coverage   82.62%   82.49%   -0.14%     
==========================================
  Files         919      921       +2     
  Lines      248979   249105     +126     
==========================================
- Hits       205722   205499     -223     
- Misses      43257    43606     +349

Flag	Coverage Δ
fuzzcorpus	`60.28% <7.97%> (-0.59%)`	⬇️
livemode	`18.71% <23.31%> (-0.01%)`	⬇️
pcap	`44.12% <23.92%> (-0.02%)`	⬇️
suricata-verify	`61.84% <23.31%> (-0.05%)`	⬇️
unittests	`58.97% <7.97%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

suricata-qa · 2024-09-13T04:35:03Z

ERROR:

ERROR: QA failed on SURI_TLPR1_alerts_cmp.

Pipeline 22592

jasonish · 2024-10-09T18:18:02Z

Replaced by #11921.

amirabell and others added 14 commits September 12, 2024 12:51

libsuricata: add library runmode

2360d08

Add library source and runmode modules. Reorganized library example to create a worker thread and replay a pcap file using the library mode. No API layer is added at this stage. Edits by Jason Ish: - fix guard - add copyright/license headers

examples: move lib capture example to libcapture

dd499c4

To keep the simple example simple, move the lib based capture method example to its own example.

.gitignore: add more files

4fa7ed6

- the generated binaries for lib examples - LSP files - man pages

librunmode: rename threadvars creation function

6cf2a12

Also use a proper return type (ThreadVars *).

lib: remove global worker id variable

9247494

Update ThreadVars creation in lib mode to have the worker_id provided by the user.

librunmode: take pointer to LiveDevice, not name

15aa853

In the library capture example, show how the packet counter can be updated.

threads: remove unneeded var from AcqLoop

9797efc

The variable run is not needed, we can just run an infinite loop and break when needed.

lib-runmode: consistent naming style

4719663

And add SC prefix.

lib-source: reorganize to avoid static prototypes

5af2faf

runmodes: typedef runmode enum and use as type

b282b6d

Also remove function to set the library mode. This is easy enough to do with SCRunmodeSet, and we don't want to add a specific setter for each and every runmode.

examples: update library example for ips drop

fe89a69

Add a release packet callback where the action can be checked for drop.