Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

{,x-pack/}winlogbeat: resurrect tests for event processing #31464

Merged
merged 6 commits into from
May 3, 2022
Merged

{,x-pack/}winlogbeat: resurrect tests for event processing #31464

merged 6 commits into from
May 3, 2022

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Apr 29, 2022 •
edited
Loading

What does this PR do?

This resurrects a partial testing of the winlogbeat event processing. It does not examine the ingest pipeline (that will be done as #30406) but does ensure we do not have regressions in the parts of the input that deal directly with the Windows event logging API.

Why is it important?

We had inadequate testing.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • The change here clobbers all the existing tests. At some stage the original state of those should be restored, or alternatively a new set should be added as system tests with the full ingest enrichment which would be the state that these were prior to this change.

How to test this PR locally

Tests can be run locally using the go tool on a windows host.

Related issues

Use cases

Screenshots

Logs

@efd6 efd6 added enhancement Team:Security-External Integrations backport-skip Skip notification from the automated backport with mergify 8.3-candidate labels Apr 29, 2022
@efd6 efd6 requested a review from a team as a code owner April 29, 2022 02:08
@efd6 efd6 self-assigned this Apr 29, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 29, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 29, 2022
@efd6 efd6 requested a review from andrewkroh April 29, 2022 02:09
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 29, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-05-02T23:46:21.257+0000

  • Duration: 50 min 22 sec

Test stats 🧪

Test Results
Failed 0
Passed 930
Skipped 9
Total 939

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@efd6 efd6 force-pushed the winlogbeatforwardporttests branch 2 times, most recently from 11b593f to b028010 Compare April 29, 2022 03:26
efd6
efd6 commented Apr 29, 2022
View reviewed changes
winlogbeat/eventlog/wineventlog.go
Comment on lines +399 to +406
md, err := win.NewPublisherMetadataStore(win.NilHandle, e.Provider.Name, logp.L())
if err != nil {
// Return an empty store on error (can happen in cases where the
// log was forwarded and the provider doesn't exist on collector).
md = win.NewEmptyPublisherMetadataStore(e.Provider.Name, logp.L())
logp.Warn("failed to load publisher metadata for %v "+
"(returning an empty metadata store): %v", e.Provider.Name, err)
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be worth caching this in the *winEventLog, but I'd like to do that after getting the testing done.

@efd6 efd6 added backport-v8.2.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Apr 29, 2022
@mergify
Copy link
Contributor

mergify bot commented Apr 29, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b winlogbeatforwardporttests upstream/winlogbeatforwardporttests
git merge upstream/main
git push upstream winlogbeatforwardporttests

efd6 added 3 commits May 2, 2022 07:32
@efd6
x-pack/winlogbeat: port forward semi-processed events for tests
4ee82b1 
This brings partially processed event data from a modified version of the testing
code at 8896fd3 (the commit immediately prior to
the removal of the javascript processing pipeline in elastic#29435: commit at
2f3b0c5cbe9cfdd10e11fd52e2a259e564001100).

The evtx.golden.json files were generated by removing the js processing call at
https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135
to match the code in that file as it appears here and then run go test -update
in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019
host.

The test package for each of the modules is also resurrected with modifications
reflecting the loss of the javascript processor.

Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this
commit.
@efd6
winlogbeat/eventlog: make sure metadata is available for enrichment o…
04686dc 
…f raw values

This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in
sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in
33acb3c (2022-01-25) after the origin of the
forward port origin (2021-12-02).
@efd6
x-pack/winlogbeat/module/sysmon/test: golden file for sysmon-11-filed…
8039eaa 
…eletedetected.evtx
@efd6 efd6 force-pushed the winlogbeatforwardporttests branch from b028010 to 6cf2660 Compare May 1, 2022 22:38
@efd6 efd6 force-pushed the winlogbeatforwardporttests branch from 6cf2660 to 4a3d585 Compare May 1, 2022 23:43
@efd6 efd6 mentioned this pull request May 2, 2022
Merged
@efd6
X-pack/winlogbeat/module: ignore opcode field on Windows 2022
24da692 
Also prohibit generating golden files for PowerShell on Windows 2022 to
prevent unnecessary work in discovering that this will fail on other
versions.
@efd6 efd6 force-pushed the winlogbeatforwardporttests branch from 4a3d585 to 24da692 Compare May 2, 2022 01:48
andrewkroh
andrewkroh approved these changes May 2, 2022
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's really good to see these tests re-enabled.

@efd6
defer event field filtering until value comparison
7e5ec1b 
This will result in additional diff noise if golden values are generated on multiple
versions of windows so it may be worth keeping the version used reasonably constant.
The version used here was 2019.
@efd6 efd6 mentioned this pull request May 2, 2022
Closed
@efd6 efd6 merged commit f22abe3 into elastic:main May 3, 2022
@mergify mergify bot mentioned this pull request May 3, 2022
Merged
mergify bot pushed a commit that referenced this pull request May 3, 2022
@efd6 @mergify
{,x-pack/}winlogbeat: resurrect tests for event processing (#31464)
4096abb 
This is a moderately involved change. The process is described in the steps
below, but the reader should refer to the commits in the PR to see exactly
what was done and when.

* port forward semi-processed events for tests

This brings partially processed event data from a modified version of the testing
code at 8896fd3 (the commit immediately prior to
the removal of the javascript processing pipeline in #29435: commit at
2f3b0c5cbe9cfdd10e11fd52e2a259e564001100).

The evtx.golden.json files were generated by removing the js processing call at
https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135
to match the code in that file as it appears here and then run go test -update
in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019
host.

The test package for each of the modules is also resurrected with modifications
reflecting the loss of the javascript processor.

Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this
commit.

* make sure metadata is available for enrichment of raw values

This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in
sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in
33acb3c (2022-01-25) after the origin of the
forward port origin (2021-12-02).

* update golden file for sysmon-11-filedeletedetected.evtx

* ignore opcode field on Windows 2022

Also prohibit generating golden files for PowerShell on Windows 2022 to
prevent unnecessary work in discovering that this will fail on other
versions.

* defer event field filtering until value comparison

This will result in additional diff noise if golden values are generated on multiple
versions of windows so it may be worth keeping the version used reasonably constant.
The version used here was 2019.

(cherry picked from commit f22abe3)

# Conflicts:
#	x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json
#	x-pack/winlogbeat/module/testing_windows.go
efd6 added a commit that referenced this pull request May 3, 2022
@efd6
cherrypick winlogbeat test fixes relevant to 7.17 from #31464
32daffe
efd6 added a commit that referenced this pull request May 3, 2022
@mergify @efd6
[8.2](backport #31464) {,x-pack/}winlogbeat: resurrect tests for even…
7b90207 
…t processing (#31491)

* {,x-pack/}winlogbeat: resurrect tests for event processing (#31464)

This is a moderately involved change. The process is described in the steps
below, but the reader should refer to the commits in the PR to see exactly
what was done and when.

* port forward semi-processed events for tests

This brings partially processed event data from a modified version of the testing
code at 8896fd3 (the commit immediately prior to
the removal of the javascript processing pipeline in #29435: commit at
2f3b0c5cbe9cfdd10e11fd52e2a259e564001100).

The evtx.golden.json files were generated by removing the js processing call at
https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135
to match the code in that file as it appears here and then run go test -update
in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019
host.

The test package for each of the modules is also resurrected with modifications
reflecting the loss of the javascript processor.

Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this
commit.

* make sure metadata is available for enrichment of raw values

This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in
sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in
33acb3c (2022-01-25) after the origin of the
forward port origin (2021-12-02).

* update golden file for sysmon-11-filedeletedetected.evtx

* ignore opcode field on Windows 2022

Also prohibit generating golden files for PowerShell on Windows 2022 to
prevent unnecessary work in discovering that this will fail on other
versions.

* defer event field filtering until value comparison

This will result in additional diff noise if golden values are generated on multiple
versions of windows so it may be worth keeping the version used reasonably constant.
The version used here was 2019.

(cherry picked from commit f22abe3)

# Conflicts:
#	x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json
#	x-pack/winlogbeat/module/testing_windows.go

* fix conflict scars

* silence bad linter

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
efd6 added a commit that referenced this pull request May 3, 2022
@mergify @efd6
[7.17](backport #30942) winlogbeat/sys/wineventlog: extend testing an…
953f799 
…d fix bugs (#31096)

* winlogbeat: fix event handling for Windows 2022 (#30942)

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 34bdc3d)

# Conflicts:
#	winlogbeat/Jenkinsfile.yml
#	winlogbeat/beater/winlogbeat.go
#	winlogbeat/eventlog/wineventlog.go
#	winlogbeat/eventlog/wineventlog_experimental.go
#	winlogbeat/sys/wineventlog/wineventlog_windows.go
#	x-pack/winlogbeat/Jenkinsfile.yml

* clean up conflict scars

* cherrypick winlogbeat test fixes relevant to 7.17 from #31464

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
@efd6 efd6 mentioned this pull request May 4, 2022
Merged
6 tasks
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@efd6 @chrisberkhout
{,x-pack/}winlogbeat: resurrect tests for event processing (#31464)
54efac5 
This is a moderately involved change. The process is described in the steps
below, but the reader should refer to the commits in the PR to see exactly
what was done and when.

* port forward semi-processed events for tests

This brings partially processed event data from a modified version of the testing
code at 8896fd3 (the commit immediately prior to
the removal of the javascript processing pipeline in #29435: commit at
2f3b0c5cbe9cfdd10e11fd52e2a259e564001100).

The evtx.golden.json files were generated by removing the js processing call at
https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135
to match the code in that file as it appears here and then run go test -update
in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019
host.

The test package for each of the modules is also resurrected with modifications
reflecting the loss of the javascript processor.

Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this
commit.

* make sure metadata is available for enrichment of raw values

This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in
sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in
33acb3c (2022-01-25) after the origin of the
forward port origin (2021-12-02).

* update golden file for sysmon-11-filedeletedetected.evtx

* ignore opcode field on Windows 2022

Also prohibit generating golden files for PowerShell on Windows 2022 to
prevent unnecessary work in discovering that this will fail on other
versions.

* defer event field filtering until value comparison

This will result in additional diff noise if golden values are generated on multiple
versions of windows so it may be worth keeping the version used reasonably constant.
The version used here was 2019.
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees

@efd6 efd6

Labels
8.3-candidate backport-v8.2.0 Automated backport with mergify enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@efd6 @elasticmachine @andrewkroh