cloud/aws/kinesis: avoid error logs from canceled contexts

Getting a canceled context error during application shutdown now no longer logs an error, but instead handles this case gracefully. Additionally, we now no longer deadlock if the kernel shuts down before the kinsumer could finish starting.

What's Changed

• cloud/aws/kinesis: avoid error logs from canceled contexts by @ajscholl in #1184
• kinsumer: handle case where the kinsumer stops before the stop function is set and resign as leader in case of an error by @ajscholl in #1204

Full Changelog: v0.37.4...v0.37.5

stream: fix incorrect kinesis output stream name

What's Changed

• stream: fix incorrect kinesis output stream name by @j4k4 in #1203

Full Changelog: v0.37.3...v0.37.4

improve guard performance and expose additional config options

This release contains two changes:

guard, reqctx: added cache to guard for current request and reqctx package to store data for the current request;

We can do a lot of queries for the same policies if your response contains multiple objects you have to do access control on (e.g., returning a list of entities). To help here, we introduce a cache for the current request to avoid retrieving the same policies again and again.

cloud/aws: exposed additional options to config;

This exposes additional options for AWS clients to the config files. These include:

• Disabling request compression and min compression request size for cloudwatch
• Disabling response checksum validation and accepting gzip compression for dynamodb
• The usePathStyle setting was documented for S3 (but it was already exposed before)

These settings are needed when talking to services implementing the AWS API like localstack or ScyllaDB as these services sometimes can't map all features exactly.

redis: fix lifecycle deadlock issue

This release fixes a deadlock issue which occurs under certain circumstances while using redis clients.

mdlsub: types provider util for transformers

This release adds a small helper for mdlsub transformers.
Instead of implementing the GetInput and GetModel functions, one can embed the TypesProvider.

before

type TestTransformer struct{}

func (t TestTransformer) GetInput() any {
	return &TestInput{}
}

func (t TestTransformer) GetModel() any {
	return &TestModel{}
}

after

type TestTransformer struct {
	mdlsub.TypesProvider[TestInput, TestModel]
}

reslife: introduction of advanced resource lifecycle management

This release works towards unifying creation and purging of external resources.
Currently, all resources like SQS queues or DynamoDb tables are created "in place" during instantiation of their corresponding services.

This change adds 4 life cycles for any type of resources: create, init, register and purge.
For SQS queues this means:

1. create: creating the queue if not existing
2. init: fetching properties like URL and ARN to initialize the queue service
3. register: adding metadata information to the metadata server
4. purge: purging any remaining messages in the queue

These cycles can be enabled / disabled via configuration and have the following defaults:

resource_lifecycles:
    create:
        enabled: false
    init:
        enabled: true
    register:
        enabled: true
    purge:
        enabled: false

In addition, while running with the dev or test environment, create will be enabled by default, too, if not disabled explicitly. This replaces the former functionality provided by dx.auto_create.

breaking changes

1. The mdlsub.ModelTransformer interface is extended by GetModel() any which should return an instance of the output type. Make sure to adjust your subscribers.
2. existing implementations of the fixture writer interface like for ddb, mysql, redis or kvstore have moved from the fixtures package to their corresponding packages. You will have to adjust your import paths

What's Changed

• resource lifecycles by @j4k4 in #1196

Full Changelog: v0.36.1...v0.37.0

etc: several improvements and smaller additions

This release consists of several smaller improvements and additions.
The biggest new feature is the possibility to set configuration values via command line flags with the new application option WithConfigFlags:

application.New(application.WithConfigFlags(os.Args, &struct {
	ServerPort int `short:"p" long:"server-port" cfg:"httpserver.default.port" default:"8080" description:"port of the default http server"`
}{}))

Providing this option would enable to set / overwrite the config value httpserver.default.port with providing -p 8888 or --server-port 8888 as command line arguments.

What's Changed

• justdev changes by @j4k4 in #1199

Full Changelog: v0.36.0...v0.36.1

fixtures: added post processors to loaders

Major change of this release is the additions of post processors to the fixture package. These allow additional handling after loading the fixture data, e.g. populating internal caches. For this change, the signatures of adding fixtures has changed slightly:

func main() {
	application.RunHttpDefaultServer(
		api.DefineRouter,
		application.WithFixtureSetFactory("default", service.FixtureSetsFactory, service.PostProcessor),
	)
}

or if you want to add multiple groups with the same post processor:

func main() {
	application.RunHttpDefaultServer(
		api.DefineRouter,
		application.WithFixtureSetFactories(map[string]fixtures.FixtureSetsFactory{
			"group1": service.FixtureSetsFactory1,
			"group2": service.FixtureSetsFactory2,
		}, service.PostProcessor),
	)
}

The options for creating test suites have been adjusted accordingly:

func (s *TestSuite) SetupSuite() []suite.Option {
	return []suite.Option{
		suite.WithLogLevel(log.LevelWarn),
		suite.WithConfigFile("./config.test.yml"),
		suite.WithFixtureSetFactory(service.FixtureSetsFactory, service.PostProcessors...),
	}
}

and

func (s *TestSuite) SetupSuite() []suite.Option {
	return []suite.Option{
		suite.WithLogLevel("warn"),
		suite.WithConfigFile("../config.test.yml"),
		suite.WithFixtureSetFactories([]fixtures.FixtureSetsFactory{
			common.FixtureSetsFactory,
			service.FixtureSetsFactory,
		}, service.PostProcessor),
	}
}

Post processors have to implement a simple interface:

type PostProcessor interface {
	Process(ctx context.Context) error
}

What's Changed

• fixtures: added post processors to loaders by @j4k4 in #1193

Full Changelog: v0.35.1...v0.36.0

Emit stream metrics

Metrics

The Messages per runner calculator for streams now also re-emits the Visible and Sent messages from AWS SQS Queues.

This is to reduce complexity and ensuring you don't have to import cloudwatch metrics via an exporter, which would require another component in the system where 2 additional emitted metrics would be enough.

The newly emitted metrics are:

• StreamMessagesAvailable (sum of ApproximateNumberOfMessagesVisible of all input streams)
• StreamMessagesSent (sum of NumberOfMessagesSent of all input streams)

Improve retry behavior

Client configuration

There are now five places where you can configure settings for your AWS client:

1. Client-specific settings for the given service (e.g., cloud.aws.dynamodb.clients.someClientName)
2. Client-specific settings from the defaults (e.g., cloud.aws.defaults.clients.someClientName)
3. Service-wide default settings for the given service (e.g., cloud.aws.dynamodb.clients.default)
4. Global defaults for all clients (e.g., cloud.aws.defaults.clients.default)
5. Global defaults for all AWS clients (e.g., cloud.aws.defaults)

Breaking change: Besides affecting the configuration of the clients itself, naming patterns also fall back to the naming patterns of the default client of the service. Thus, we use cloud.aws.sqs.clients.default.naming.pattern if cloud.aws.sqs.clients.yourClient.naming.pattern was not provided. This was done for all services (SQS, SNS, Kinesis, DDB)

DDB Locks

DDB locks now extend the context to unlock the lock until the lock expires when unlocking the lock.

Producer daemons

Producer daemons now configure their own AWS clients using the name of the producer (i.e., producer-{eventName}). These clients are then configured with infinite backoff, thus, even if your application is running as a gateway (which only retry for 10s by default), the producer daemon will retry indefinitely as it is running in the background to avoid losing data.

Kinesis outputs

Writing to Kinesis now takes the setting of the client used for writing into account (previously, we would have used the default backoff settings, i.e., stop after 10s for gateways) for the outer write loop (when writing to Kinesis, we might have a call to write 5 records which gets retried as well as an outer loop which would try writing 2 records in the next iteration if Kinesis accepted 3 and rejected 2 records in our example).