Support Multiple Config Files/An import statement

[PettitWesley]

Feature Request: Configuration can be split into multiple files

It'd be ideal if the collector could work off of a set of configuration files rather than a single one. There are probably two basic ways to accomplish this:

1. Continue to support a single config file but allow it to import other others. This is the same as how your main script might import other code. You have an import statement somewhere in the config file and it references a path to find other config.
2. Support a list of config files in the entrypoint for the collector.

The way this should work is that the collector should evaluate each config file and merge them.

Why is this useful?

As the collector grows, the configurations that users create will get more and more complex. For the same reasons that in school your prof taught you not to put all your C code in one file, its generally useful to organize configuration logically into multiple files.

This is also useful when you have multiple people/entities working on building the final configuration. Consider the following scenario: A few years from now when OTel has truly eaten the world of instrumentation, Ellen and Akshay are building the config for their application. The config for their unified collector must container the following:

• Ellen's logging config, which includes multiple destinations, custom decorations, and parsing
• Akshay's trace config, with its own complex set of processor definitions
• Configuration for AWS Container Insights metrics, which is very lengthy as well.
• Some configuration for collecting security logs from their platform, which they have received from their company's centralized security team. (In this future world, the collector can also be used to collect security events).

Could they concatenate all of these configs together into one file? Definitely. It'd be inconvenient though. Note that each piece of the config has a different owner. This means that updates might come independently. Thus it's useful if they can remain in separate files. Regardless of how the team stores and deploys their config, it'll be easier to manage if they can keep it in multiple files.

Examples in other projects

Fluentd has @include and Fluent Bit has @INCLUDE for importing configuration from other files. https://docs.fluentd.org/configuration/config-file#6-reuse-your-config-the-include-directive

This is very commonly used.

Merging configs

This bit which will need some discussion.

Ideally, the configurations should be merge-able. This means I can define a processor in config file A and then reference it in a pipeline in config file B. Basically the final configuration is the concatenation of all the configs.

[bogdandrutu]

/cc @pjanotti

I think the proposed "config source" will address this.

[PettitWesley]

@pjanotti Can you give details on this config source thing?

[tigrannajaryan]

See #2857 for config source proposal.

[PettitWesley]

Notes from SIG meeting:

• Joe Lynch says google will work on this soon because they need it too.
• Its important that config merging is deterministic
• Bogdan says that we can add a config source based on code like here: Extract ConfigFactory in a ParserProvider interface #2868
• Bogdan and Joe thinks that overrides are important
• error are hard- getting the config errors to tell you which file was broken is hard. User probably has to figure it out based on the error telling you the name of the component

[hossain-rayhan]

From 05/26 Collector SIG meeting:

• Config source is an interface and Splunk has an internal implementation. Tigran said they will include it into the upstream collector. However, he needs to check with the team.
• Rayhan will open a temp-slack channel to discuss this.

[josephflynch]

Google (GCE) is working on this problem right now in our Ops Agent. While in theory we can generate an OT config file regardless of where this feature ends up we'd much prefer to align our design with the OT design as OT Collector is being used in many contexts within Google (not just Ops Agent). Let us know how to participate. This is actually a complex issue and merits a design doc. If our Ops Agent design doc approaches the problem generically enough (which I suspect it will) we'll share it in case it's useful for thinking through this. I also enumerated ~12 apps that take on the DRY config paradigm without going so far as to create a virtual programming language. I can share links if helpful.

We're approaching it slightly more open-ended:
Which capabilities (e.g. includes, overrides, built-in YAML capabilities, etc.) would make it ~easy to have ~DRY config that's easy to maintain? We're not assuming includes are the only option.

[PettitWesley]

@josephflynch We'd be very interested in seeing/hearing your ideas and what you've done in ops agent. Let's all work together on this.

[ttomsu]

Friendly ping - Has there been any movement on this recently?

[josephflynch]

This has been sitting for a while. We're just about to pick it back up. When I have something useful I will pass it around. If you like, I'd be game for a quick video conference on the topic to understand each other's goals and share ideas. We've been thinking about it for a while but need to put pen to paper for a design. Joe Lynch

…

On Fri, Aug 13, 2021 at 11:47 AM Travis Tomsu ***@***.***> wrote: Friendly ping - Has there been any movement on this recently? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2701 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAU6B4NGYLJ2PNTNUGFLL3T4U5CVANCNFSM4ZFMITUQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[tigrannajaryan]

This is work in progress. This PR was intended to enable it #3687
However, I believe @bogdandrutu had an alternate idea which he wanted to try.

[alolita]

@bogdandrutu are you planning to work on this issue :)

[bogdandrutu]

@josephflynch this is about to be closed, in case you still need it.

[bogdandrutu]

s/closed/fixed :D