[sql-41] firewalldb: add migration code for kvstores from kvdb to SQL #1079

ViktorTigerstrom · 2025-05-27T15:55:25Z

Based on #1051

This PR introduces the migration logic for transitioning the kv stores from kvdb to SQL.

Note that as of this PR, the migration is not yet triggered by any production code, i.e. only tests execute the migration logic.

Once #1051 is merged, I will rebase this PR and request reviews.

Part of #917

ellemouton

i think this doesnt need to be based on the sessions pr right? yes order of merge matters, but i think we dont need to base this one off of that one or at least can change the base branch here so that we can review both migrations in parallel

firewalldb/sql_migration.go

ViktorTigerstrom · 2025-06-03T23:01:13Z

Thanks for the swift review @ellemouton 🙏! I've rebased this to be based on master and not the sessions migration PR. Note that I've kept 2 commits from the sessions migration PR in this PR (the 2 first), as they are required. So feel free to ignore reviewing the first to commits of this PR.
Also note that since this is not based on the sessions migrations PR, we can't test this migration locally as we need the sessions to be migrated prior to the kvstores (so that we can fetch the sessions SQL IDs). Let me know if you feel a need to test this prior to merging the sessions PR though, and I'll send you a branch which adds local testing that's based on both PRs.

ellemouton

looking great! a couple of comments. I'll do a final round once the session PR is in 👍

ellemouton · 2025-06-04T12:36:12Z

firewalldb/interface.go

@@ -141,4 +141,7 @@ type FirewallDBs interface {
 	RulesDB
 	PrivacyMapper
 	ActionDB
+
+	// Close closes the underlying store.
+	Close() error


dont think we need this

just call t.Cleanup sooner

(again commit message could explain more as it took me a sec to figure out why you're adding this)

Thanks 🙏! Addressed with a new commit in the latest push :)

ellemouton · 2025-06-04T12:39:42Z

firewalldb/db.go

@@ -14,29 +14,21 @@ var (
 	ErrNoSuchKeyFound = fmt.Errorf("no such key found")
 )



your commit messages could be fleshed out a bit more pls. it isnt clear why this needs to be exported (same for other commits in this PR)

Sorry, completely forgot updating the commit messages in the latest push. I've updated them in the latest push though, so I hope the commits should be understandable now!

firewalldb/sql_migration.go

ellemouton · 2025-06-04T12:42:13Z

firewalldb/sql_migration.go

+	// SQL database during the migration.
+	var allParams []kvParams
+
+	err := kvStore.Update(func(kvTx *bbolt.Tx) error {


why Update?

It was due to the migration utilising the getMainBucket helper function, which with the previous version created the main bucket if it didn't exist. However I realised that if the bucket doesn't exist, it just means that we can skip migrating the records, so I updated the PR to do that instead with the latest push.

ellemouton · 2025-06-04T12:45:49Z

firewalldb/sql_migration.go

+		return err
+	}
+
+	// After the migration is done, we validate that all inserted kvParams


cool yeah thanks for separating this 👍 i think it is especially important for the kv stores since there is a higher chance of overwriting 👍

ellemouton · 2025-06-04T12:48:32Z

firewalldb/sql_migration_test.go

+		},
+		{
+			name:       "random records",
+			populateDB: randomKVRecords,


i think it's worth having a case that does some explicit entries that write to all the store types. currently we only do that for the random one.

Good idea! Added an extra test with the latest push.

it still doesnt have a deterministic test that stores in 2 feature tables/ 2 rule tables etc right?

like, currently globalRecords only inserts under a single rule name, for example

also, looks like for this new test, all the key-value pairs inserted at all the levels are the same... so again it doesnt properly test that the correct things mapping is taking place

We add a helper function to the functions that creates the test SQL stores, in order to ensure that the store is properly closed when the test is cleaned up.

In preparation for upcoming migration tests from a kvdb to an SQL store, this commit updates the NewTestDB function to return the Store interface rather than a concrete store implementation. This change ensures that migration tests can call NewTestDB under any build tag while receiving a consistent return type.

We add a helper function to the functions that creates the test SQL stores, in order to ensure that the store is properly closed when the test is cleaned up.

In the upcoming migration of the firewall database to SQL, the helper functions that creates the test databases of different types, need to return a unified interface in order to not have to control the migration tests file by build tags. Therefore, we export the unified interface FirewallDBs, so that it can be returned public test DB creation functions

In the upcoming migration of the firewall database to SQL, the helper functions that creates the test databases of different types, need to return a unified interface in order to not have to control the migration tests file by build tags. Therefore, we update the `NewTestDB` functions to return the `FirewallDBs` interface instead of the specific store implementation type.

During the upcoming upcoming migration of the firewall database to SQL, we need to be able to check all kvstores records in the SQL database, to validate that the migration is successful in tests. This commits adds a query to list all kvstores records, which enables that functionality.

During the migration of the kvstores to SQL, we'll iterate over the buckets in the bbolt database, which holds all kvstores records. In order to understand why the migration iterates over the buckets in the specific order, we need to clarify the bbolt kvstores illustration docs, so that it correctly reflects how the records are actually stored in the bbolt database.

This commit introduces the migration logic for transitioning the kvstores store from kvdb to SQL. Note that as of this commit, the migration is not yet triggered by any production code, i.e. only tests execute the migration logic.

ViktorTigerstrom · 2025-06-09T12:35:30Z

Thanks for the review @ellemouton 🙏! Addressed the latest feedback and rebased this once again on the important commits from #1051.

bitromortac

Great work 🚀, took an initial look, the structure looks great!

bitromortac · 2025-06-12T07:45:52Z

firewalldb/kvstores_kvdb.go

-rules -> perm -> rule-name -> global   -> {k:v}
-              -> sessions -> group ID  -> session-kv-store  -> {k:v}
-			               -> feature-kv-stores -> feature-name -> {k:v}
+"rules" -> "perm" -> rule-name -> "global" -> {k:v}


thanks for the update here 👍

yeah thanks for the fix!! 🙏

bitromortac · 2025-06-12T07:49:16Z

firewalldb/sql_migration.go

+// It inserts the records into the SQL database and asserts that
+// the migrated values match the original values in the KV store.


nit: there are no assertions in here, same in processGroupBucket

bitromortac · 2025-06-12T07:54:22Z

firewalldb/sql_migration.go

+	ruleSqlId int64, groupAlias []byte,
+	groupBucket *bbolt.Bucket) ([]kvParams, error) {
+
+	groupSqlId, err := sqlTx.GetSessionIDByAlias(


nit: contract to line

bitromortac · 2025-06-12T07:55:55Z

firewalldb/sql_migration.go

+				ruleNameBucket := mainBucket.Bucket(k)
+				if ruleNameBucket == nil {
+					return fmt.Errorf("rule bucket %s "+
+						"not found", string(k))


I think you don't need those casts as the %s directive already does it

bitromortac · 2025-06-12T08:07:47Z

firewalldb/sql_migration_test.go

+	Global      bool
+	GroupID     *session.ID


you don't use the nilness of GroupID, right? could you use fn.Option[GroupID] to encode that this is a global entry?

ellemouton · 2025-06-13T07:22:58Z

firewalldb/kvstores_kvdb.go

-              -> sessions -> group ID  -> session-kv-store  -> {k:v}
-			               -> feature-kv-stores -> feature-name -> {k:v}
+"rules" -> "perm" -> rule-name -> "global" -> {k:v}
+				  "session-kv-store" -> group ID -> {k:v}


missing the -> here

ellemouton · 2025-06-13T07:30:16Z

firewalldb/kvstores_kvdb.go

-rules -> perm -> rule-name -> global   -> {k:v}
-              -> sessions -> group ID  -> session-kv-store  -> {k:v}
-			               -> feature-kv-stores -> feature-name -> {k:v}
+"rules" -> "perm" -> rule-name -> "global" -> {k:v}


while we are here, can we add < & > around keys that vary? like <rule-name>

and <group ID>& <feature name>

ellemouton · 2025-06-13T07:34:05Z

firewalldb/kvstores_kvdb.go

-rules -> perm -> rule-name -> global   -> {k:v}
-              -> sessions -> group ID  -> session-kv-store  -> {k:v}
-			               -> feature-kv-stores -> feature-name -> {k:v}
+"rules" -> "perm" -> rule-name -> "global" -> {k:v}


yeah thanks for the fix!! 🙏

ellemouton · 2025-06-13T07:38:08Z

firewalldb/sql_migration.go

+				continue
+			}
+
+			err = mainBucket.ForEach(func(k, v []byte) error {


suggest adding a short comment: "Loop over each rule-name in the bucket"

ellemouton · 2025-06-13T07:38:45Z

firewalldb/sql_migration.go

+				}
+
+				ruleName := k
+				ruleNameBucket := mainBucket.Bucket(k)


ruleNameBucket := mainBucket.Bucket(k) -> ruleNameBucket := mainBucket.Bucket(ruleName)

same below, let's not continue to use k

ellemouton · 2025-06-13T09:51:24Z

firewalldb/sql_migration_test.go

+	tempKvRecord := kvStoreRecord{
+		RuleName:    ruleName,
+		GroupID:     &sess.GroupID,
+		FeatureName: featureNameOpt,
+		EntryKey:    entryKey,
+		Value:       entryVal,
+		Perm:        false,
+		Global:      global,
+	}
+
+	insertKvRecord(t, ctx, boltDB, tempKvRecord)
+
+	permKvRecord := kvStoreRecord{
+		RuleName:    ruleName,
+		GroupID:     &sess.GroupID,
+		FeatureName: featureNameOpt,
+		EntryKey:    entryKey,
+		Value:       entryVal,
+		Perm:        true,
+		Global:      global,
+	}


if you are inserting the same key-value pair in both, then things are not properly being tested since you are not testing that the one namespace doesnt overwrite the other

ellemouton · 2025-06-13T09:53:36Z

firewalldb/sql_migration_test.go

+		},
+		{
+			name:       "random records",
+			populateDB: randomKVRecords,


also, looks like for this new test, all the key-value pairs inserted at all the levels are the same... so again it doesnt properly test that the correct things mapping is taking place

ellemouton · 2025-06-13T09:58:12Z

firewalldb/sql_migration_test.go

+	// Create a session that we can reference.
+	sess, err := sessionStore.NewSession(
+		ctx, "test", session.TypeAutopilot,
+		time.Unix(1000, 0), "something",
+	)
+	require.NoError(t, err)
+


only need to create session if !global

ellemouton · 2025-06-13T10:03:30Z

firewalldb/sql_migration_test.go

+		record.FeatureName.UnwrapOr(""),
+	)
+
+	err := kvStores.Update(ctx, func(ctx context.Context,


suggest:

err := kvStores.Update(ctx, func(ctx context.Context, tx KVStoreTx) error { store := tx.Global() switch { case record.Global && !record.Perm: store = tx.GlobalTemp() case !record.Global && !record.Perm: store = tx.LocalTemp() case !record.Global && record.Perm: store = tx.Local() } return store.Set(ctx, record.EntryKey, record.Value) }) require.NoError(t, err)

(ie, reduce repetition where possible)

ellemouton · 2025-06-13T10:15:08Z

firewalldb/sql_migration.go

@@ -0,0 +1,486 @@
+package firewalldb


there is a lot of nesting happening here with various operations happening a couple layers deep. There is also quite a lot of repetition.

Let me know what you think of this diff which follows a more step-by-step approach with fewer layers & more code re-use. Notice how migrateKVStoresDBToSQL follows an easy-to-ready step-by-step flow:

collect pairs (kvdb only)

migrate pairs (sql only)

validate migrated pairs. (compare kvdb & sql)

(just delete .txt from the name - GH wont let me upload the .go file)
sql_migration.go.txt

ViktorTigerstrom changed the title ~~2025 05 migrate kvstores~~ [sql-41] firewalldb: add migration code for kvstores from kvdb to SQL May 27, 2025

ViktorTigerstrom added the no-changelog This PR is does not require a release notes entry label May 27, 2025

ViktorTigerstrom self-assigned this May 27, 2025

ellemouton reviewed May 30, 2025

View reviewed changes

ViktorTigerstrom force-pushed the 2025-05-migrate-kvstores branch 2 times, most recently from 4151b93 to 7a517ad Compare June 3, 2025 18:22

ViktorTigerstrom requested a review from ellemouton June 3, 2025 23:01

ellemouton reviewed Jun 4, 2025

View reviewed changes

ViktorTigerstrom added 9 commits June 9, 2025 13:33

accounts: ensure that test SQL store is closed

3c42778

We add a helper function to the functions that creates the test SQL stores, in order to ensure that the store is properly closed when the test is cleaned up.

session: ensure that test SQL store is closed

0b70477

We add a helper function to the functions that creates the test SQL stores, in order to ensure that the store is properly closed when the test is cleaned up.

firewalldb: ensure that test SQL store is closed

dd4acca

We add a helper function to the functions that creates the test SQL stores, in order to ensure that the store is properly closed when the test is cleaned up.

firewalldb: add kvstores kvdb to SQL migration

10550d4

This commit introduces the migration logic for transitioning the kvstores store from kvdb to SQL. Note that as of this commit, the migration is not yet triggered by any production code, i.e. only tests execute the migration logic.

ViktorTigerstrom force-pushed the 2025-05-migrate-kvstores branch from 7a517ad to 10550d4 Compare June 9, 2025 12:20

ViktorTigerstrom requested a review from ellemouton June 9, 2025 12:35

ellemouton requested a review from bitromortac June 10, 2025 07:50

ViktorTigerstrom mentioned this pull request Jun 10, 2025

[sql-43] Add support for sqldb/v2 in litd #1085

Draft

bitromortac reviewed Jun 12, 2025

View reviewed changes

ellemouton reviewed Jun 13, 2025

View reviewed changes

		@@ -14,29 +14,21 @@ var (
		ErrNoSuchKeyFound = fmt.Errorf("no such key found")
		)

		// It inserts the records into the SQL database and asserts that
		// the migrated values match the original values in the KV store.

[sql-41] firewalldb: add migration code for kvstores from kvdb to SQL #1079

Are you sure you want to change the base?

[sql-41] firewalldb: add migration code for kvstores from kvdb to SQL #1079

Uh oh!

Conversation

ViktorTigerstrom commented May 27, 2025

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ViktorTigerstrom commented Jun 3, 2025

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ViktorTigerstrom commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bitromortac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ViktorTigerstrom commented Jun 9, 2025 •

edited

Loading