fixtures: added post processors to loaders

Major change of this release is the additions of post processors to the fixture package. These allow additional handling after loading the fixture data, e.g. populating internal caches. For this change, the signatures of adding fixtures has changed slightly:

func main() {
	application.RunHttpDefaultServer(
		api.DefineRouter,
		application.WithFixtureSetFactory("default", service.FixtureSetsFactory, service.PostProcessor),
	)
}

or if you want to add multiple groups with the same post processor:

func main() {
	application.RunHttpDefaultServer(
		api.DefineRouter,
		application.WithFixtureSetFactories(map[string]fixtures.FixtureSetsFactory{
			"group1": service.FixtureSetsFactory1,
			"group2": service.FixtureSetsFactory2,
		}, service.PostProcessor),
	)
}

The options for creating test suites have been adjusted accordingly:

func (s *TestSuite) SetupSuite() []suite.Option {
	return []suite.Option{
		suite.WithLogLevel(log.LevelWarn),
		suite.WithConfigFile("./config.test.yml"),
		suite.WithFixtureSetFactory(service.FixtureSetsFactory, service.PostProcessors...),
	}
}

and

func (s *TestSuite) SetupSuite() []suite.Option {
	return []suite.Option{
		suite.WithLogLevel("warn"),
		suite.WithConfigFile("../config.test.yml"),
		suite.WithFixtureSetFactories([]fixtures.FixtureSetsFactory{
			common.FixtureSetsFactory,
			service.FixtureSetsFactory,
		}, service.PostProcessor),
	}
}

Post processors have to implement a simple interface:

type PostProcessor interface {
	Process(ctx context.Context) error
}

What's Changed

• fixtures: added post processors to loaders by @j4k4 in #1193

Full Changelog: v0.35.1...v0.36.0

Emit stream metrics

Metrics

The Messages per runner calculator for streams now also re-emits the Visible and Sent messages from AWS SQS Queues.

This is to reduce complexity and ensuring you don't have to import cloudwatch metrics via an exporter, which would require another component in the system where 2 additional emitted metrics would be enough.

The newly emitted metrics are:

• StreamMessagesAvailable (sum of ApproximateNumberOfMessagesVisible of all input streams)
• StreamMessagesSent (sum of NumberOfMessagesSent of all input streams)

Improve retry behavior

Client configuration

There are now five places where you can configure settings for your AWS client:

1. Client-specific settings for the given service (e.g., cloud.aws.dynamodb.clients.someClientName)
2. Client-specific settings from the defaults (e.g., cloud.aws.defaults.clients.someClientName)
3. Service-wide default settings for the given service (e.g., cloud.aws.dynamodb.clients.default)
4. Global defaults for all clients (e.g., cloud.aws.defaults.clients.default)
5. Global defaults for all AWS clients (e.g., cloud.aws.defaults)

Breaking change: Besides affecting the configuration of the clients itself, naming patterns also fall back to the naming patterns of the default client of the service. Thus, we use cloud.aws.sqs.clients.default.naming.pattern if cloud.aws.sqs.clients.yourClient.naming.pattern was not provided. This was done for all services (SQS, SNS, Kinesis, DDB)

DDB Locks

DDB locks now extend the context to unlock the lock until the lock expires when unlocking the lock.

Producer daemons

Producer daemons now configure their own AWS clients using the name of the producer (i.e., producer-{eventName}). These clients are then configured with infinite backoff, thus, even if your application is running as a gateway (which only retry for 10s by default), the producer daemon will retry indefinitely as it is running in the background to avoid losing data.

Kinesis outputs

Writing to Kinesis now takes the setting of the client used for writing into account (previously, we would have used the default backoff settings, i.e., stop after 10s for gateways) for the outer write loop (when writing to Kinesis, we might have a call to write 5 records which gets retried as well as an outer loop which would try writing 2 records in the next iteration if Kinesis accepted 3 and rejected 2 records in our example).

add kinsumer autoscale module

This release adds an optional module that will periodically check the current number of tasks of you kinsumer and the number of shards of the corresponding kinesis stream and set the task count to the number of shards if they are not equal.
You can add the module to your kinsumer app using the WithKinsumerAutoscaleModule option.

Since the module is using the DdbLeaderElection you will have to provide a DynamoDB table with a hash key named groupId of type String , a TTL attribute named leadingUntil and the table name has to match the pattern {env}-kinsumer-autoscale-leaders by default but can be configured.

For now, the module only supports ECS as the orchestrator.
You will have to give your app permissions for the ecs:DescribeServices and ecs:UpdateService actions, which you can do by using version 3.2.0 of our terraform-aws-ecs-app module and setting the ecs_access_policy_enabled variable to true.

treat request cancellations not as errors and make dispatcher errors more obvious

HTTP Server
The standard http server treats external context cancellations and disconnects as errors; even though the client may have closed or canceled the request on purpose.
As a follow up to #1178 (targeting the crud handlers), this PR is adjusting the standard behaviour of any other handler similarly.
The most common use cases:

• access forbidden transformed into HTTP 403
• context cancellation / disconnects are transformed into HTTP 499
• any other error being transformed into a HTTP 500

The logger middleware is also adjusted appropriately, logging connection and request cancelations as level info. Bind errors and render errors are continously logged as warnings; by default any other error is logged as an error.

Exec

• fixing bugs which causes the code to panic on nil errors

Dispatcher
The dispatcher is hiding some errors which may happen during execution of the callbacks. They're now propagated back to the callee, making them obvious and necessary to deal with.
In some cases, you may have to wrap adding callbacks into the appctx to prevent them from being added multiple times.

This is a breaking change, as the dispatcher will now stop execution if any error is encountered

v0.33.2

cloud/aws/athena: automatic retry on internal query errors

If AWS responses with Amazon Athena experienced an internal error while executing this query. Please try submitting the query again while executing an Athena query, the client will trigger an automatic retry now.

db-repo: use db client on creating orm instances

This release enables the use of different db client names on creating orm instances.

db: close the sql.Rows object once we are done with it

We sometimes have code which would iterate an sql.Rows object, returning an error if we failed to read the next row. As long as we don't get an error, this works, as calling rows.Next() will eventually close the rows object once the last row was read. However, in case of an error we have to make sure to explicitly call rows.Close(), otherwise we leak connections for some drivers.

currency: add locking to currency updater

This introduces a lock on the currency data to prevent that a lot of instances of an application try to to update the data at the same time, which would eventually lead to to the table exceeding the write capacity and then cause a spiraling effect because the module would not get healthy in time and thus even more instances would be spawned.

This is a breaking change if your app uses the currency module.
Since the currency module is now using the ddb lock provider you will have to provide a dynamodb table configured as follows:

• The table needs a hash key named resource of type String
• The table needs to have TTL enabled for a an attribute named ttl
• The table name has to match this pattern: {env}-{app_group}-locks

Make sure to give your table enough write capacity in case many instances of your app try to acquire a lock at the same time.

v0.32.2

What's Changed

• go: fix dependency versions for go.opentelemetry.io/otel/exporters/ot… by @applike-ss in #1187

Full Changelog: v0.32.1...v0.32.2

v0.32.1

What's Changed

• metric: fix default metrics with dimensions by @applike-ss in #1186

Full Changelog: v0.32.0...v0.32.1