Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Changes related to merging RFC 0007 at stage 3 #1066

Merged
merged 18 commits into from
Dec 7, 2020
Merged

Changes related to merging RFC 0007 at stage 3 #1066

merged 18 commits into from
Dec 7, 2020

Conversation

webmat
Copy link
Contributor

@webmat webmat commented Oct 30, 2020 •
edited
Loading

Effects the changes from #1017. To be merged at the same time, or after.

Preview the new user "usage" page:

TODO

  • Categorization section
  • Notes about reuse (as a different name) of user fields within user. Namely that these don't get carried around when nesting user fields elsewhere.
  • Add a section about deprecating usage of host.user.*
  • I'm also considering adding a TOC at the top of this page, to give folks a sense of the sections contained within this humongous page.
  • Wait for RFC stage 3 to be merged [RFC] Multiple users in an event, stage 3 PR #1017
  • Identify these new field reuses as beta

Questions

  • Towards the end of this page, I link to the RFC in the ECS repo, for folks who want to see mapping examples. I'm trying to avoid making this page 2x longer still. But will users (not ECS contributors) be lost, when they land in the RFC?
  • Allow custom descriptions for self-nesting reuses #1174 Out of scope for this PR, but I think that the listing of user fields in the field reuse section is a bit off. Perhaps we should mention that these are actually the user fields (right now it's designed to list other field sets nested here), or perhaps we should redesign this a bit. Note that this also affects the process.parent reuse.

@webmat webmat self-assigned this Oct 30, 2020
@webmat webmat mentioned this pull request Oct 30, 2020
Merged
6 tasks
@webmat
Copy link
Contributor Author

webmat commented Oct 30, 2020

This is not the only thing in this PR, but here's the preview of the new user usage page

@webmat webmat mentioned this pull request Nov 2, 2020
Open
@webmat webmat marked this pull request as ready for review November 2, 2020 20:56
@webmat webmat requested a review from ebeahan November 2, 2020 21:02
@webmat
Copy link
Contributor Author

webmat commented Nov 2, 2020

This is ready for a review.

The "subtleties" section makes me want to cry a little 😢 😂

This was referenced Nov 3, 2020
Closed
Closed
Closed
Closed
Closed
@ebeahan
Copy link
Member

ebeahan commented Nov 5, 2020

Responding to the open questions:

Towards the end of this page, I link to the RFC in the ECS repo, for folks who want to see mapping examples. I'm trying to avoid making this page 2x longer still. But will users (not ECS contributors) be lost, when they land in the RFC?

I agree it could cause some confusion, but I think the confusion would be minor. Perhaps we can adjust the wording in the link from "RFC" to something more general like "feature proposal", for users not familiar with an RFC document?

The concern over page length is understandable, but I can also see the mappings being a useful reference for users. With the sample mappings external to the docs, it might make discoverability more challenging (an extra click, not searchable from the docs site).

Out of scope for this PR, but I think that the listing of user fields in the field reuse section is a bit off. Perhaps we should mention that these are actually the user fields (right now it's designed to list other field sets nested here), or perhaps we should redesign this a bit. Note that this also affects the process.parent reuse.

I made this observation too recently. In #1051, I'm proposing introducing a beta attribute to the reusable expected object. Perhaps we can also consider adding a secondary description attribute as well?

@ebeahan
Copy link
Member

ebeahan commented Nov 5, 2020

The "subtleties" section makes me want to cry a little

Yes this is a challenging area to detail!

I don't think it's necessary for this PR, but in the future perhaps we can add some sort of visual as another way of explaining the concepts.

ebeahan
ebeahan reviewed Nov 5, 2020
View reviewed changes
Copy link
Member

@ebeahan ebeahan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a handful of minor notes, but I think overall this is great @webmat! Having a consolidated reference is going to be a great resource!

I'll hold off on giving the final 👍 until we have the beta labeling support from #1051 finalized and the appropriate changes incorporated here.

docs/usage/user.asciidoc
when there's more than one user present on the event. `user.*` fields at the root
of the event represent the user performing the action.

In many cases, events that only mention one user should populate the user fields
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A high-level summary of the guidance covered could be a useful companion quick-reference to the more detailed explanations. I could see it both early in the doc as well as towards the end as a summary.

Examples of quick guidance:

  • Always populate user.* in addition to more purpose-specific fields when one or more users are present in an event. If more than one user, the user performing the main action populates user.*.
  • Any users present in the event should also be captured in related.user
  • etc. etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this is not a blocker to this PR. WDYT?

Perhaps we could note down some of the points for this quick guidance in a separate issue, and address separately?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it's not a blocker here. We can capture and address later on 😄 .

docs/usage/user.asciidoc
}
-----------

Whenever an event source populates the `client` and `server` fields in addition
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could see a second example which builds on the first, showing both client/server + source/destination fields all populated in the single event. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I think I would err on the side of "terseness", if we can call such a long page terse. If the idea is to reinforce this pattern, perhaps we can instead link to the page on mapping network events?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a link at the end of this paragraph

webmat and others added 4 commits November 6, 2020 08:28
@webmat @ebeahan
escalate "to"
63b203f 
Co-authored-by: Eric Beahan <ebeahan@gmail.com>
Root uid is zeero
65e534b
@webmat @ebeahan
Clarify instead of what
f82ba40 
Co-authored-by: Eric Beahan <ebeahan@gmail.com>
Add a short section on user identifier gotchas.
459976e
Another attempt at explaining the self-reuse subtleties.
7c3b299 
This section still makes me cry a little!
@ebeahan ebeahan added the ready Issues we'd like to address in the future. label Nov 17, 2020
@webmat webmat added the 1.8.0 label Nov 17, 2020
@webmat
Copy link
Contributor Author

webmat commented Dec 1, 2020

@elasticmachine, run elasticsearch-ci/docs

ebeahan
ebeahan reviewed Dec 4, 2020
View reviewed changes
Copy link
Member

@ebeahan ebeahan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The adjustments are looking great!

Towards the end of this page, I link to the RFC in the ECS repo, for folks who want to see mapping examples. I'm trying to avoid making this page 2x longer still. But will users (not ECS contributors) be lost, when they land in the RFC?

I think linking to the RFC is acceptable for curious, interested users.

I responded to the second question in my notes below.

ebeahan
ebeahan approved these changes Dec 7, 2020
View reviewed changes
Copy link
Member

@ebeahan ebeahan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the iterations here! LGTM 👍

@webmat webmat merged commit 5f5c4ec into elastic:master Dec 7, 2020
webmat pushed a commit to webmat/ecs that referenced this pull request Dec 7, 2020
Apply RFC 0007 stage 3 changes - multi-user (elastic#1066)
f60a306 
Conflict: deleted file rfcs/text/0007-multiple-users.md as RFCs are not
backported to version branches.
@webmat webmat mentioned this pull request Dec 7, 2020
Merged
webmat pushed a commit that referenced this pull request Dec 7, 2020
[1.x] Apply RFC 0007 stage 3 changes - multi-user (#1066) (#1175)
48e1ddc 
Conflict: deleted file rfcs/text/0007-multiple-users.md as RFCs are not backported to version branches.
This was referenced May 5, 2021
Closed
Closed
Closed
Merged
dseeley added a commit to dseeley/ecs that referenced this pull request May 5, 2021
@dseeley
Upstream ECS merge of v1.9.0 (#4)
fc14145 
* bumping version for 1.x release branch (elastic#921)

* [1.x] add related.hosts (elastic#913) (elastic#924)

* [1.x][DOCS] Fixes SIEM links (elastic#936)

* [1.x] Consolidate field-details doc template (elastic#897) (elastic#946)

* Add http.[request|response].mime_type (elastic#944) (elastic#949)

* [1.x] Cut 1.6 Changelog (elastic#933) (elastic#952) (elastic#953)

Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co>

* [1.x] Add threat.technique.subtechnique (elastic#951) (elastic#956)

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

* [1.x] Nest as for foreign reuse (elastic#960) (elastic#962)

* [1.x] Remove `expected_event_types` from protocol (elastic#964) (elastic#965)

* [1.x] Expand definitions of source and destination field sets (elastic#967) (elastic#973)

* [1.x] Introduce `--strict` flag (elastic#937) (elastic#975)

* [1.x] Add example value composite type checking (elastic#966) (elastic#976)

* Add example value composite type checking (elastic#966)
* generate csv artifact

* [1.x] Add event category configuration (elastic#963) (elastic#977)

* [1.x] Add normalizer multi-field capability (elastic#971) (elastic#978)

Co-authored-by: Eric Beahan <ebeahan@gmail.com>

Co-authored-by: Madison Caldwell <madison.rey.caldwell@gmail.com>

* [1.x] Add mapping network event guidance doc (elastic#969) (elastic#983)

* [1.x] Removing unneeded link under `Additional Information` (elastic#984) (elastic#985)

* [1.x] Add discrete attribute to field details page headers (elastic#989) (elastic#990)

* [1.x] Uniformity across domain name breakdown fields (elastic#981) (elastic#994)

Co-authored-by: Mathieu Martin <webmat@gmail.com>

* Add --oss flag to the ECS generator script (elastic#991) (elastic#995)

* Add network directions ingress and egress (elastic#945) (elastic#997)

* Mention ECS Mapper in the main documentation (elastic#987) (elastic#1000)

Co-authored-by: Dan Roscigno <dan@roscigno.com>

* [1.x] Introduce experimental artifacts (elastic#993) (elastic#1001)

Co-authored-by: Mathieu Martin <webmat@gmail.com>

* Bump version to 1.8.0-dev in branch 1.x (elastic#1011)

* Cut 1.7 changelog (elastic#1010) (elastic#1012)

* [1.x] Clarify that file extension should exclude the dot. (elastic#1016) (elastic#1020)

* [1.x] Add usage docs section (elastic#988) (elastic#1024)

Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co>

* [1.x] feat: include alias path when generating template (elastic#877) (elastic#1035)

Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com>

* [1.x] Add support for `scaling_factor` in the generator (elastic#1042) (elastic#1055)

Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co>

* [1.x] Add fallback for constant_keyword (elastic#1046) (elastic#1056)

Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co>

* [1.x] Add wildcard type support to go code generator (elastic#1050) (elastic#1057)

* add wildcard type support

* also add version and constant_keyword

* changelog

* [1.x] New default make task that generates main and experimental artifacts. (elastic#1041) (elastic#1060)

Also changing the order of the 'generate' task: it now starts with the new generator, then runs the legacy scripts.

* [1.x] Change the index pattern in the sample template. (elastic#1048) (elastic#1068)

* [1.x] Prepare link to Logs docs changing with the 7.10 release in "getting-started" (elastic#1073) (elastic#1079)

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

* [1.x] Prepare link to Logs docs changing with the 7.10 release in "products-solutions" page (elastic#1074) (elastic#1083)

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

* [1.x] Add event.category session. (elastic#1049) (elastic#1093)

Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co>

* [1.x] Add event.category registry (elastic#1040) (elastic#1094)

Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co>

* [1.x] Add --ref support for experimental artifacts (elastic#1063) (elastic#1101)

Co-authored-by: Mathieu Martin <webmat@gmail.com>

* [1.x] Remove experimental event.original definition (elastic#1053) (elastic#1104)

* [1.x] Add missing `process.thread.name` to experimental definitions (elastic#1103) (elastic#1106)

* [1.x] Remove index parameter for wildcard fields (elastic#1115) (elastic#1119)

* [1.x] Add dns.answer object into experimental schema (elastic#1118) (elastic#1121)

* [1.x] Clarify x509 definition guidance for network events with only one cert (elastic#1114) (elastic#1123)

* [1.x] Indicate when artifacts include experimental changes (elastic#1117) (elastic#1125)

* [1.x] Add os.type field, with list of allowed values (elastic#1111) (elastic#1130)

* [1.x] Add support for constant_keyword's 'value' parameter (elastic#1112) (elastic#1132)

* [1.x] Beta label support (elastic#1051) (elastic#1133)

Co-authored-by: Mathieu Martin <webmat@gmail.com>

* [1.x] Backport elastic#1134 and elastic#1135 (elastic#1136)

* Remove temporary ifeval in "getting started" page, add link to Metrics docs (elastic#1134)
* Remove temporary ifeval from products page, add link to Metrics (elastic#1135)

* Two small documentation backports (elastic#1149)

* Remove an incorrect `event.type` from the 'converting' page (elastic#1146)
* Mention Logstash support for ECS in the 'products' page (elastic#1147)

* [1.x] Reinforce the exclusion of the leading dot from url.extension (elastic#1151) (elastic#1152)

* [1.x] Make all fields linkable directly via an HTML ID (elastic#1148) (elastic#1154)

* [1.x] Tracing fields should be at the root (elastic#1165)

* Add notice to the tracing field set, about not nesting field names. (elastic#1162)
* Tracing fields should be at top level in Beats artifact (elastic#1164)

* [1.x] Usage of brackets for a URL containing IPv6 address (elastic#1131) (elastic#1168)

* [1.x] 6.x index template data type fallback (elastic#1171) (elastic#1172)

* [1.x] Apply RFC 0007 stage 3 changes - multi-user (elastic#1066) (elastic#1175)

Conflict: deleted file rfcs/text/0007-multiple-users.md as RFCs are not backported to version branches.

* [1.x] Handle `error.stack_trace` case for ES 6.x template (elastic#1176) (elastic#1177)

* [1.x] Add composable index templates artifacts (elastic#1156) (elastic#1179)

* [1.x] Move _meta section back inside mappings, in legacy templates. (elastic#1186) (elastic#1187)

Backports the following commits to 1.x:

* Move _meta section back inside mappings, in legacy templates. (elastic#1186) 

This fixes an issue introduced by elastic#1156, discovered in elastic#1180. Composable templates support `_meta` at the template's root, but legacy templates don't. So we're just putting it back inside the mappings for legacy templates.

This also fixes missing updates to the component template, after the introduction of wildcard in elastic#1098.

* [1.x] Apply the RFC 0005 stage 2 (host metrics) changes in the experimental artifacts (elastic#1159) (elastic#1184)

Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co>

* [1.x] Stage 3 changes for wildcard RFC 0001 (elastic#1098) (elastic#1183)

* [1.x] Conditional handling in es_template.template_settings (elastic#1191) (elastic#1192)

* [1.x] Artifacts docs page (elastic#1189) (elastic#1195)

* [1.x] Remove beta warning label from categorization fields docs (elastic#1067) (elastic#1196)

* [1.x] Correct wording of `event.reference` description (elastic#1181) (elastic#1197)

* Bump version to 1.9.0-dev in branch 1.x (elastic#1198)

* [1.x] Cut 1.8 FF changelog.next.md elastic#1199 (elastic#1201)

* Merge custom and core multi_fields arrays (elastic#982) (elastic#1213)

Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>

* [1.x] Stage 2 changes for RFC 0009 - data_stream fields (elastic#1215) (elastic#1222)

* [1.x] add http.request.id (elastic#1208) (elastic#1223)

Co-authored-by: Eric Beahan <eric.beahan@elastic.co>
Co-authored-by: Gil Raphaelli <gil@elastic.co>

* [1.x] add cloud.service.name (elastic#1204) (elastic#1224)

* add cloud.platform

* expand cloud.platform description

* move to cloud.service.name

Co-authored-by: Gil Raphaelli <gil@elastic.co>

* [1.x] Add ssdeep hash (elastic#1169) (elastic#1227)

Co-authored-by: Andrew Stucki <andrew.stucki@elastic.co>

* [CI] Switch to GitHub actions (elastic#1236) (elastic#1245)

Co-authored-by: Eric Beahan <ebeahan@gmail.com>

Co-authored-by: Andrew Stucki <andrew.stucki@elastic.co>

* Revert wildcard adoption back to experimental stage (elastic#1235) (elastic#1243)

* Add scaled_float type to go generator (elastic#1250) (elastic#1251)

* add scaled_float

* changelog

* Add categorization fields usage docs (elastic#1242) (elastic#1257)

* add time_zone, postal_code, and continent_code (elastic#1229) (elastic#1258)

* Specify MAC address format (elastic#456) (elastic#1260)

Co-authored-by: Robin Schneider <36660054+ypid-geberit@users.noreply.github.com>

* finalize 1.8.0 changelog (elastic#1262) (elastic#1265)

* Add additional host fields (elastic#1248) (elastic#1267)

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Stage 1 changes for RFC 0014 - extend pe fields (elastic#1256) (elastic#1270)

* Add 2 fields to code_signature (elastic#1269) (elastic#1272)

Co-authored-by: Yamin Tian <56367679+Trinity2019@users.noreply.github.com>

* Stage 3 changes for RFC 0007 - remove beta attribute (elastic#1271) (elastic#1273)

* Stage 1 experimental changes for RFC 0008 - threat.indicator fields (elastic#1268) (elastic#1274)

* Stage 1 changes for RFC 0015 - add elf fieldset (elastic#1261) (elastic#1275)

* Cut 1.9 FF CHANGELOG.next.md (elastic#1277)

* lock go version in actions (elastic#1283) (elastic#1290)

* Bump jinja2 from 2.11.2 to 2.11.3 in /scripts (elastic#1310) (elastic#1320)

* Bump jinja2 from 2.11.2 to 2.11.3 in /scripts

* Bump pyyaml from 5.3b1 to 5.4 in /scripts (elastic#1318) (elastic#1325)

Co-authored-by: Eric Beahan <eric.beahan@elastic.co>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Adjust terminology - change whitelist to allowlist (elastic#1315) (elastic#1331)

Co-authored-by: Dominic Page <11043991+djptek@users.noreply.github.com>

* Remove -dev label from 1.9 version (elastic#1329)

* remove -dev label from 1.9 version

* generate artifacts

* removing rules artifacts

* Cut 1.9 changelog (elastic#1328)

* move 1.9 changes to changelog

* add 1.9 release changes
This was referenced May 5, 2021
Closed
Closed
Merged
Closed
Merged
Closed
Merged
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@ebeahan ebeahan ebeahan approved these changes

Assignees

@webmat webmat

Labels
1.x 1.8.0 ready Issues we'd like to address in the future.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@webmat @ebeahan