Apply Jackson stream read constraints defaults at runtime #16832
Conversation
When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like elastic#16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present.
| public static final Map<Override, Integer> JACKSON_DEFAULTS = Map.of( | ||
| Override.MAX_STRING_LENGTH, 200_000_000, | ||
| Override.MAX_NUMBER_LENGTH, 10_000, | ||
| Override.MAX_NESTING_DEPTH, 1_000 |
There was a problem hiding this comment.
CODEREVIEW: I just noticed this is actually not set explicitly
Line 90 in 0eb03ad
There was a problem hiding this comment.
@edmocosta Is it intended to comment out? I think it is supposed to be uncommented
There was a problem hiding this comment.
Yes, it's intended to be commented out, at time we added those properties, 1000 was the default value already, and we haven't had any problem with that specific limit that would justify overriding it.
It seems the Jackson's team is still tuning it, and the new default value was reduced to 500 (FasterXML/jackson-core#1234). They've a good reasoning on why it was reduced (FasterXML/jackson-core#1233), but I think we can be more conservative and sticky to the well tested 1000 (at least for now). With this new code, it seems uncommenting the property wouldn't make much difference, so I'm fine with whatever you folks think is better.
There was a problem hiding this comment.
thanks, keeping the comment is good to me.
There was a problem hiding this comment.
Wow it sounds we lost consistency here as well.
Do we know till which jackson version (and LS version) we had nesting-depth with 1000 and which version decreased to 500 in which LS version? Do we have issues reported for such situation? If so, addressing separately would be good idea considering user stories.
|
💚 Build Succeeded
|
| @@ -18,6 +18,14 @@ public class StreamReadConstraintsUtil { | |||
|
|
|||
| private StreamReadConstraints configuredStreamReadConstraints; | |||
|
|
|||
| // Provide default values for Jackson constraints in the case they are | |||
| // not specified in configuration file. | |||
| public static final Map<Override, Integer> JACKSON_DEFAULTS = Map.of( | |||
There was a problem hiding this comment.
Is public (exposing) required here? If so, Override sounds to me an broad interface for this context. I would support renaming (if it is easy with this change) it to JacksonDefaultOverride (or similar) namings.
If we can make it private, no rename needs.
| public static final Map<Override, Integer> JACKSON_DEFAULTS = Map.of( | ||
| Override.MAX_STRING_LENGTH, 200_000_000, | ||
| Override.MAX_NUMBER_LENGTH, 10_000, | ||
| Override.MAX_NESTING_DEPTH, 1_000 |
There was a problem hiding this comment.
Wow it sounds we lost consistency here as well.
Do we know till which jackson version (and LS version) we had nesting-depth with 1000 and which version decreased to 500 in which LS version? Do we have issues reported for such situation? If so, addressing separately would be good idea considering user stories.
| @@ -78,6 +86,8 @@ StreamReadConstraints get() { | |||
| if (configuredStreamReadConstraints == null) { | |||
| final StreamReadConstraints.Builder builder = StreamReadConstraints.defaults().rebuild(); | |||
|
|
|||
| // Apply the Jackson defaults first, then the overrides from config | |||
| JACKSON_DEFAULTS.forEach((override, value) -> override.applicator.apply(builder, value)); | |||
There was a problem hiding this comment.
My biggest worry with these kind of changes is how to educate users to keep settings up to date.
- Users upgraded LS from the versions which
jvm.optionsdoesn't have Jackson default params to current change, we need to make sure to inform themjvm.optionsis not up to date. The situations,
- users who jumped from 8.12.0-, the
jvm.optionsdoesn't have - users upgraded LS multiple times (8.12.0- to 8.15.0 then 8.17.0), the
jvm.optionsmay not have jackson default params, depending on how (through package managers, or download tarball and replace) they upgrade.
- and with 1) we also need to inform LS is applying default values or user defined values are active. I think we are covering this
Release notes
Ensure that the Jackson read constraints defaults (Maximum Number value length, Maximum String value length, and Maximum Nesting depth) are applied at runtime if they are absent from
jvm.options.What does this PR do?
When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like #16683.
This change ensures Logstash applies defaults from config at runtime:
These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present.
Why is it important/What is the impact to the user?
Users who have custom
jvm.optionsfiles or have files that are not being updated with defaults in latest Logstash releases will still get the defaults we intend configure for the Jackson parsing lib.Checklist
[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration files (and/or docker env variables)Related issues
Closes #16773