Module Replication: MongoDB & MySQL support

.changeset/cold-items-explain.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-module-mongodb': patch
+---
+
+Fix diagnostics schema authorization issues for MongoDB

.changeset/fifty-dogs-reply.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-module-mongodb': minor
+---
+
+Reduce permissions required for replicating a single mongodb database

.changeset/gentle-icons-try.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-module-mysql': patch
+---
+
+Fixed MySQL version checking to better handle non-semantic version strings

.changeset/green-peas-roll.md

@@ -0,0 +1,6 @@
+---
+'@powersync/service-module-mongodb': minor
+'@powersync/service-image': minor
+---
+
+Add MongoDB support (Alpha)

.changeset/healthy-rules-arrive.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-module-mysql': patch
+---
+
+Fixed mysql schema json parsing

.changeset/heavy-shirts-chew.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-core': patch
+---
+
+Improved sync rules storage cached parsed sync rules, accommodating different parsing options where necessary.

.changeset/lemon-terms-play.md

@@ -0,0 +1,8 @@
+---
+'@powersync/service-module-mysql': minor
+---
+
+Generate random serverId based on syncrule id for MySQL replication client
+Consolidated type mappings between snapshot and replicated values
+Enabled MySQL tests in CI
+

.changeset/olive-spoons-stare.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-core': patch
+---
+
+Moved tag variable initialization in diagnostics route to ensure it is initialized before usage

.changeset/orange-eagles-tap.md

@@ -0,0 +1,5 @@
+---
+'@powersync/lib-services-framework': minor
+---
+
+Added disposable listeners and observers

.changeset/popular-snails-cough.md

@@ -0,0 +1,6 @@
+---
+'@powersync/service-core': minor
+'@powersync/service-sync-rules': minor
+---
+
+Added ability to emit data replication events

.changeset/rotten-pumas-protect.md

@@ -0,0 +1,9 @@
+---
+'@powersync/service-core': minor
+'@powersync/service-module-mysql': minor
+'@powersync/service-sync-rules': minor
+---
+
+Introduced alpha support for MySQL as a datasource for replication.
+Bunch of cleanup
+

.changeset/slow-stingrays-kiss.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-core': minor
+---
+
+Moved Write Checkpoint APIs to SyncBucketStorage

.changeset/sour-turkeys-collect.md

@@ -0,0 +1,7 @@
+---
+'@powersync/service-module-postgres': patch
+'@powersync/service-rsocket-router': patch
+'@powersync/service-types': patch
+---
+
+Updates from Replication events changes

.changeset/tender-vans-impress.md

@@ -0,0 +1,16 @@
+---
+'@powersync/service-core': minor
+'@powersync/service-sync-rules': minor
+'@powersync/lib-services-framework': minor
+'@powersync/service-jpgwire': minor
+'@powersync/service-types': minor
+'@powersync/service-image': major
+'@powersync/service-module-postgres': patch
+---
+
+- Introduced modules to the powersync service architecture
+  - Core functionality has been moved to "engine" classes. Modules can register additional functionality with these engines.
+  - The sync API functionality used by the routes has been abstracted to an interface. API routes are now managed by the RouterEngine.
+  - Replication is managed by the ReplicationEngine and new replication data sources can be registered to the engine by modules.
+- Refactored existing Postgres replication as a module.
+- Removed Postgres specific code from the core service packages.

.changeset/violet-garlics-know.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-sync-rules': minor
+---
+
+Support json_each as a table-valued function.

.changeset/weak-cats-hug.md

@@ -0,0 +1,5 @@
+---
+'@powersync/service-sync-rules': minor
+---
+
+Optionally include original types in generated schemas as a comment.

.github/workflows/test.yml

@@ -22,22 +22,67 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
 
-      - name: Build and push
+      - name: Test Build Docker Image
         uses: docker/build-push-action@v5
         with:
           cache-from: type=registry,ref=stevenontong/${{vars.DOCKER_REGISTRY}}:cache
           context: .
           platforms: linux/amd64
           push: false
           file: ./service/Dockerfile
-          # TODO remove this when removing Journey Micro
-          build-args: |
-            GITHUB_TOKEN=${{secrets.RESTRICTED_PACKAGES_TOKEN}}
 
-  run-tests:
-    name: Test
+  run-core-tests:
+    name: Core Test
     runs-on: ubuntu-latest
 
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Start MongoDB
+        uses: supercharge/mongodb-github-action@1.8.0
+        with:
+          mongodb-version: '6.0'
+          mongodb-replica-set: test-rs
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version-file: '.nvmrc'
+
+      - uses: pnpm/action-setup@v4
+        name: Install pnpm
+        with:
+          version: 9
+          run_install: false
+
+      - name: Get pnpm store directory
+        shell: bash
+        run: |
+          echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
+
+      - uses: actions/cache@v3
+        name: Setup pnpm cache
+        with:
+          path: ${{ env.STORE_PATH }}
+          key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
+          restore-keys: |
+            ${{ runner.os }}-pnpm-store-
+
+      - name: Install dependencies
+        run: pnpm install
+
+      - name: Build
+        shell: bash
+        run: pnpm build
+
+      - name: Test
+        run: pnpm test --filter '!./modules/*'
+
+  run-postgres-tests:
+    name: Postgres Test
+    runs-on: ubuntu-latest
+    needs: run-core-tests
+
     strategy:
       fail-fast: false
       matrix:
@@ -97,4 +142,123 @@ jobs:
         run: pnpm build
 
       - name: Test
-        run: pnpm test
+        run: pnpm test --filter='./modules/module-postgres'
+
+  run-mysql-tests:
+    name: MySQL Test
+    runs-on: ubuntu-latest
+    needs: run-core-tests
+
+    strategy:
+      fail-fast: false
+      matrix:
+        mysql-version: [5.7, 8.0, 8.4]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Start MySQL
+        run: |
+          docker run \
+          --name MySQLTestDatabase \
+          -e MYSQL_ROOT_PASSWORD=mypassword \
+          -e MYSQL_DATABASE=mydatabase \
+          -p 3306:3306 \
+          -d mysql:${{ matrix.mysql-version }} \
+          --log-bin=/var/lib/mysql/mysql-bin.log \
+          --gtid_mode=ON \
+          --enforce_gtid_consistency=ON \
+          --server-id=1
+
+      - name: Start MongoDB
+        uses: supercharge/mongodb-github-action@1.8.0
+        with:
+          mongodb-version: '6.0'
+          mongodb-replica-set: test-rs
+
+      - name: Setup NodeJS
+        uses: actions/setup-node@v4
+        with:
+          node-version-file: '.nvmrc'
+
+      - uses: pnpm/action-setup@v4
+        name: Install pnpm
+        with:
+          version: 9
+          run_install: false
+
+      - name: Get pnpm store directory
+        shell: bash
+        run: |
+          echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
+
+      - uses: actions/cache@v3
+        name: Setup pnpm cache
+        with:
+          path: ${{ env.STORE_PATH }}
+          key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
+          restore-keys: |
+            ${{ runner.os }}-pnpm-store-
+
+      - name: Install dependencies
+        run: pnpm install
+
+      - name: Build
+        shell: bash
+        run: pnpm build
+
+      - name: Test
+        run: pnpm test --filter='./modules/module-mysql'
+
+  run-mongodb-tests:
+    name: MongoDB Test
+    runs-on: ubuntu-latest
+    needs: run-core-tests
+
+    strategy:
+      fail-fast: false
+      matrix:
+        mongodb-version: ['6.0', '7.0', '8.0']
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Start MongoDB
+        uses: supercharge/mongodb-github-action@1.8.0
+        with:
+          mongodb-version: ${{ matrix.mongodb-version }}
+          mongodb-replica-set: test-rs
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version-file: '.nvmrc'
+
+      - uses: pnpm/action-setup@v4
+        name: Install pnpm
+        with:
+          version: 9
+          run_install: false
+
+      - name: Get pnpm store directory
+        shell: bash
+        run: |
+          echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
+
+      - uses: actions/cache@v3
+        name: Setup pnpm cache
+        with:
+          path: ${{ env.STORE_PATH }}
+          key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
+          restore-keys: |
+            ${{ runner.os }}-pnpm-store-
+
+      - name: Install dependencies
+        run: pnpm install
+
+      - name: Build
+        shell: bash
+        run: pnpm build
+
+      - name: Test
+        run: pnpm test --filter='./modules/module-mongodb'

.prettierignore

@@ -0,0 +1,5 @@
+**/.git
+**/node_modules
+dist
+lib
+pnpm-lock.yaml

README.md

@@ -2,7 +2,7 @@
   <a href="https://www.powersync.com" target="_blank"><img src="https://github.com/powersync-ja/.github/assets/7372448/d2538c43-c1a0-4c47-9a76-41462dba484f"/></a>
 </p>
 
-*[PowerSync](https://www.powersync.com) is a sync engine for building local-first apps with instantly-responsive UI/UX and simplified state transfer. Syncs between SQLite on the client-side and Postgres, MongoDB or MySQL on the server-side.*
+_[PowerSync](https://www.powersync.com) is a sync engine for building local-first apps with instantly-responsive UI/UX and simplified state transfer. Syncs between SQLite on the client-side and Postgres, MongoDB or MySQL on the server-side._
 
 # PowerSync Service
 
@@ -11,6 +11,7 @@
 The service can be started using the public Docker image. See the image [notes](./service/README.md)
 
 # Monorepo Structure:
+
 ## Packages
 
 - [packages/service-core](./packages/service-core/README.md)
@@ -52,13 +53,13 @@ Contains the PowerSync Service code. This project is used to build the `journeya
 
 - [docs](./docs/README.md)
 
-Technical documentation regarding the implementation of PowerSync. 
+Technical documentation regarding the implementation of PowerSync.
 
 ## Test Client
 
 - [test-client](./test-client/README.md)
 
-Contains a minimal client demonstrating direct usage of the HTTP stream sync API. This can be used to test sync rules in contexts such as automated testing. 
+Contains a minimal client demonstrating direct usage of the HTTP stream sync API. This can be used to test sync rules in contexts such as automated testing.
 
 # Developing
 

libs/lib-services/package.json

@@ -26,11 +26,13 @@
     "dotenv": "^16.4.5",
     "lodash": "^4.17.21",
     "ts-codec": "^1.2.2",
+    "uuid": "^9.0.1",
     "winston": "^3.13.0",
     "zod": "^3.23.8"
   },
   "devDependencies": {
     "@types/lodash": "^4.17.5",
-    "vitest": "^0.34.6"
+    "@types/uuid": "^9.0.4",
+    "vitest": "^2.1.1"
   }
 }