Allow configuration of default tags which are always attached to traces as attributes

[mariusoe]

Currently, all attributes of a trace have to be explicitly defined in the configuration. If traces should have a set of attributes/tags which should be always exists (e.g. application) the user has to be ensure that all rules include the desired tag/attribute.

If many file exists, this can be quite hard.

It would be nice if the user can specify a set of "default tags" which are always attached to traces.

[mariusoe]

It may be better to attach the "normal" common tags to the traces instead of another, specific set of tags.

It should be considered whether they are attached on each span or only on the root span within an application in order to reduce the amount of data.

[JonasKunz]

I agree with marius.
I would suggest a singular config option for now:

inspectit:
  tracing:
    add-tags: ON_LOCAL_ROOT / ON_GLOBAL_ROOT / NEVER

The value would be an enum wit hthe two presented values.
ON_GLOBAL_ROOT means that the common tags are added to all span's which have no parent.
ON_LOCAl_ROOT means that they are added to all spans which either don't have a parent OR have a remote parent.

I would suggest using ON_LOCAL_ROOT as default.

[ivansenic]

@JonasKunz so you would say for now there is no option for add on all spans? Also you would add always all common tags, you would not define a list of tags?

I would say not all common tags make sence for the tracing.. Maybe we should allow to specify which ones you want. In addition I would add option to add common tags to every span as well, as this does duplicate the data, but on the other hand makes searching of the spans way easier..

[JonasKunz]

@ivansenic I agree that adding all tags does not always make sense.
However, based on the experience with our customers an "opt-out" would be more intuitive than an "opt-in" option. This means that by default all tags are added as attributes, unless you explicitly exclude some.
I would however see this feature as a separate ticket, something like tag-overrides:

inspectit:
  tracing:
    tag-overrides:
      service: null
      additional-tag: "some value"

This options can add or remove (for empty values) common attributes and takes precendence over tags.

Regarding your second point I agree with adding a ALWAYS option. To my knowledge however most backends do search over all attributes within a trace anyway. But anyway it is nice to have, especialyl when investigating raw span data.

[ivansenic]

Regarding your second point I agree with adding a ALWAYS option. To my knowledge however most backends do search over all attributes within a trace anyway. But anyway it is nice to have, especialyl when investigating raw span data.

@JonasKunz We discussed this situation yesterday.. But it does not support matching of attributes in all spans of a trace? Let say you are searching for a specific SQL or attribute which is not the local root where the common tags are.. You would not be able to connect this search, so give me SQL and find a common tag value, right? This is why ALWAYS has lot of sense, but I would not have it as default, but rather as you suggested the local root.