[Bug]: data-source-files report Unhandled exception. System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)

[crushonme]

What happened?

When I use data-source-files to reference two configuration file, it report Unhandled exception. System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)

top level configuration file is as below. I have used "dab validate" command to validate these two child configuration and there is no error. And I have confirmed that Entity names are unique across all configuration files.

{
  "$schema": "https://github.com/Azure/data-api-builder/releases/download/v1.2.14/dab.draft.schema.json",
  "data-source-files":[
    "dab-config-local.json", 
    "dab-config-azsqldb.json"
  ],
  "runtime": {
    "rest": {
      "enabled": true,
      "path": "/api",
      "request-body-strict": true
    },
    "graphql": {
      "enabled": true,
      "path": "/graphql",
      "allow-introspection": true
    },
    "host": {
      "cors": {
        "origins": [],
        "allow-credentials": false
      },
      "authentication": {
        "provider": "StaticWebApps"
      },
      "mode": "development"
    }
  }
}

Full error log

dab validate -c dab-config-multi.json
Information: Microsoft.DataApiBuilder 1.2.14
Information: User provided config file: dab-config-multi.json
Information: Validating config file: dab-config-multi.json
Loading config file from dab-config-multi.json.
Unhandled exception. System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Azure.DataApiBuilder.Config.ObjectModel.RuntimeConfig..ctor(String Schema, DataSource DataSource, RuntimeEntities Entities, RuntimeOptions Runtime, DataSourceFiles DataSourceFiles) in /_/src/Config/ObjectModel/RuntimeConfig.cs:line 195
   at .ctor(Object[])
   at System.Text.Json.Serialization.Converters.LargeObjectWithParameterizedConstructorConverter`1.CreateObject(ReadStackFrame& frame)
   at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo`1 jsonTypeInfo, Nullable`1 actualByteCount)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo`1 jsonTypeInfo)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at Azure.DataApiBuilder.Config.RuntimeConfigLoader.TryParseConfig(String json, RuntimeConfig& config, ILogger logger, String connectionString, Boolean replaceEnvVar, String dataSourceName, Dictionary`2 datasourceNameToConnectionString, EnvironmentVariableReplacementFailureMode replacementFailureMode) in /_/src/Config/RuntimeConfigLoader.cs:line 73
   at Azure.DataApiBuilder.Config.FileSystemRuntimeConfigLoader.TryLoadConfig(String path, RuntimeConfig& config, Boolean replaceEnvVar, String dataSourceName) in /_/src/Config/FileSystemRuntimeConfigLoader.cs:line 108
   at Azure.DataApiBuilder.Config.FileSystemRuntimeConfigLoader.TryLoadKnownConfig(RuntimeConfig& config, Boolean replaceEnvVar, String dataSourceName) in /_/src/Config/FileSystemRuntimeConfigLoader.cs:line 145
   at Azure.DataApiBuilder.Core.Configurations.RuntimeConfigProvider.TryGetConfig(RuntimeConfig& runtimeConfig) in /_/src/Core/Configurations/RuntimeConfigProvider.cs:line 125
   at Azure.DataApiBuilder.Core.Configurations.RuntimeConfigValidator.TryValidateConfig(String configFilePath, ILoggerFactory loggerFactory) in /_/src/Core/Configurations/RuntimeConfigValidator.cs:line 155
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Cli.ConfigGenerator.IsConfigValid(ValidateOptions options, FileSystemRuntimeConfigLoader loader, IFileSystem fileSystem) in /_/src/Cli/ConfigGenerator.cs:line 1199
   at Cli.Commands.ValidateOptions.Handler(ILogger logger, FileSystemRuntimeConfigLoader loader, IFileSystem fileSystem) in /_/src/Cli/Commands/ValidateOptions.cs:line 31
   at Cli.Program.<>c__DisplayClass2_0.<Execute>b__5(ValidateOptions options) in /_/src/Cli/Program.cs:line 67
   at CommandLine.ParserResultExtensions.MapResult[T1,T2,T3,T4,T5,T6,T7,T8,TResult](ParserResult`1 result, Func`2 parsedFunc1, Func`2 parsedFunc2, Func`2 parsedFunc3, Func`2 parsedFunc4, Func`2 parsedFunc5, Func`2 parsedFunc6, Func`2 parsedFunc7, Func`2 parsedFunc8, Func`2 notParsedFunc)
   at Cli.Program.Execute(String[] args, ILogger cliLogger, IFileSystem fileSystem, FileSystemRuntimeConfigLoader loader) in /_/src/Cli/Program.cs:line 61
   at Cli.Program.Main(String[] args) in /_/src/Cli/Program.cs:line 41

Version

1.2.14+c7ca8db8558a63919c530e454c8f18b45d5b931c

What database are you using?

Azure SQL

What hosting model are you using?

Local (including CLI)

Which API approach are you accessing DAB through?

GraphQL, REST

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct

[sezal98]

In the concept of Multiple Data Sources, we don't have the idea of "top-level" or "child level". Here we don't consider the head config to be the top level config and the other files to be its child. Instead, the all these data sources are in paralell.
So technically, the data source file which is supplied while running dab is the one which should contain the names of other data source files.
But all data source files should also contain its own details like "data-source" and "entities" field. These two fields are required which is why you are getting this Null Pointer Error.

Please look at the following scenario to understand this Multi Data Source case

I have three config files which I want to include in the same DAB instance.
Assuming datasource1.config has entity as Books, datasource2.config has Authors and datasource3.config has Readers as the entities exposed.
Here, we consider that we would supply datasource1 to the DAB engine while running the dab start -c {configFileName} command.

Hence, the command run is dab start -c datasource1.config
And the datasource files would look like this

Data Source 1 Config File

{
  "$schema": "https://github.com/Azure/data-api-builder/releases/download/vmajor.minor.patch/dab.draft.schema.json",
  "data-source": {
    "database-type": "mssql",
    "connection-string": "" //connection string,
    "options": {
      "set-session-context": true
    }
  },
  "data-source-files":[
    "datasource2.config.json",
    "datasource3.config.json
  ],
  "runtime": {
    "rest": {
      "enabled": true,
      "path": "/api"
    },
    "graphql": {
      "enabled": true,
      "path": "/graphql",
      "allow-introspection": true,
      "multiple-mutations": {
        "create": {
          "enabled": true
        }
      }
    },
    "host": {
      "cors": {
        "origins": [
          "http://localhost:5000"
        ],
        "allow-credentials": false
      },
      "authentication": {
        "provider": "StaticWebApps"
      },
      "mode": "development"
    }
  },
  "entities": {
    "Book": {
      // details related to entity
    }
  }
}

Data Source 2 Config File

{
  "$schema": "https://github.com/Azure/data-api-builder/releases/download/vmajor.minor.patch/dab.draft.schema.json",
  "data-source": {
    "database-type": "mssql",
    "connection-string": "" //connection string,
    "options": {
      "set-session-context": true
    }
  },
  "entities": {
    "Authors": {
      // details related to entity
    }
  }
}

Data Source 3 Config File

{
  "$schema": "https://github.com/Azure/data-api-builder/releases/download/vmajor.minor.patch/dab.draft.schema.json",
  "data-source": {
    "database-type": "mssql",
    "connection-string": "" //connection string,
    "options": {
      "set-session-context": true
    }
  },
  "entities": {
    "Readers": {
      // details related to entity
    }
  }
}

In this case, the data source 1 config file would become the virtual parent containing details of data source 2 and 3 config files and we could run multiple databases in the same DAB instance.

As mentioned in the docs
Configuration file considerations

• Every configuration file must include the data-source property.
• Every configuration file must include the entities property.
• The runtime setting is only used from the top-level configuration file, even if included in other files.
• Child configuration files can also include their own child files.
• Configuration files can be organized into subfolders as desired.
• Entity names must be unique across all configuration files.
• Relationships between entities in different configuration files aren't supported.

From our side, we would be updating the doc and the error message in dab validate command to communicate this scenario and error message more clearly

[JerryNixon]

Robin, this is a good catch. The reason for your error is that you have your top-level config without the data-source and entity properties. To be honest, when we designed this, we considered allowing for this scenario, but we decided against it so that every configuration file would be self-contained. What you found that REALLY helps is the unclear error message - which we now have on our backlog to improve. Thank you for your feedback - to get this to work, just merge your dab-config-local.json file into your dab-config.json configuration file. Thanks again.