Refactor entrypoint, parse config from env vars and file

[lovromazgon]

Description

This PR does two things:

• It refactors the Conduit entrypoint and extracts it into pkg/conduit. We now provide a single function Serve that parses the config and runs Conduit (default behavior of the Conduit CLI). It takes a Config and uses the values from it as default values. If values are provided via flags, env variables or a config file, the default config values are overwritten. This allows us to easily create different Conduit distributions that are configured with different default values or contain different built-in plugins and processors, e.g.:

  func main() {
  	// start with the default Conduit config
  	cfg := conduit.DefaultConfig()
  	// output JSON logs by default
  	cfg.Log.Format = "json"
  	// bundle Mongo connector as a built-in connector
  	cfg.PluginDispenserFactories["mongo"] = builtin.NewDispenserFactory(mongo.Connector)
  	// register custom built-in processor
  	cfg.ProcessorBuilderRegistry.Register("custom", func(config processor.Config) (processor.Interface, error) { ... })
  	// start Conduit
  	conduit.Serve(cfg)
  }

• We now use peterbourgon/ff to parse the CLI flags. This additionally allows us to parse the config from environment variables (lower priority) or a configuration file (lowest priority).

  • Env variables have the prefix CONDUIT and contain underscores instead of dots and hyphens (e.g. CONDUIT_DB_TYPE=postgres and CONDUIT_DB_POSTGRES_CONNECTION_STRING=postgres://localhost:5432/conduitdb).
  • The config file is by default named conduit.yaml and can be adjusted using the flag -config. The expected format is YAML and keys with dots can be hierarchically split, e.g.:

  db:
    type: postgres
    postgres:
      connection-string: postgres://localhost:5432/conduitdb

Fixes #31

Quick checks:

• I have followed the Code Guidelines.
• There is no other pull request for the same update/change.
• I have written unit tests. (tested manually)
• I have made sure that the PR is of reasonable size and can be easily reviewed.

[hariso]

LGTM! It would be nice to see a word or two about this in the README.

[lovromazgon]

I added a part to the readme, @hariso can you have a look and let me know if it makes sense? 😉

[hariso]

I added a part to the readme, @hariso can you have a look and let me know if it makes sense? 😉

LGTM!