Connector: MYSQL [Source/Destination]

[simonl2002]

Resource name

MYSQL

Resource link

https://www.mysql.com/

Connector type

Source

Expected connector behavior

Connector should enable CDC stream of rows from mysql tables.

[nickchomey]

It isn't obvious because mysql isn't mentioned anywhere, but apparently the vitess connector already does this. I confirmed it with the team in the discord server.

[simonl2002]

@nickchomey we've decided to create a dedicated MYSQL connector. Vitess uses different terminology and it just feels easier to have dedicated connector.

[nickchomey]

Looks like work started on this a couple months ago. Just adding a link to it for others to discover. https://github.com/conduitio-labs/conduit-connector-mysql

[nadilas]

@nickchomey i understand the vitess connector is capable of handling a MySQL destination. Do you have an example setup? Can’t seem to connect from the connector to the MySQL instance (looking at the code suggests that it can only talk to a vitess vtable).

[nickchomey]

I never used it. Try the official mysql connector that they've been working on for a few months! It's linked above

[simonl2002]

@nadilas the MySQL connector is being worked on here: https://github.com/conduitio-labs/conduit-connector-mysql

[nadilas]

Thanks @simonl2002 @nickchomey, that connector does not have an implementation for a destination yet. I was hoping to get a temporary solution until the destination MySQL becomes ready.

[nickchomey]

There's a branch where they're working on the destination connector. Looks like many commits were done yesterday on getting tests working - that's probably a sign that it is very close to being merged.

Either wait for that, or you could try building that branch to see if it works for you (I only use the source connector). If it doesn't, you could report any bugs and that would help get it merged sooner.

[nadilas]

I was actually toying with the idea of doing the destination code ourselves, but couldn’t yet put an effort estimate to the idea to weigh the feasibility. Thanks for the tip, I’ll look into the branch. 👍

[hariso]

The destination PR was reviewed, so I think it should be merged relatively soon. The Conduit team will be focusing on the MySQL connector (and a few others) a bit more now that we're done with the 0.12 release.:)

[hariso]

I was actually toying with the idea of doing the destination code ourselves, but couldn’t yet put an effort estimate to the idea to weigh the feasibility. Thanks for the tip, I’ll look into the branch. 👍

Regarding doing the destination code yourselves: the Connector SDK should make it fairly simple to do so. The destination are simpler compared to source obviously and usually it's only batching that might require some additional though (Conduit makes batching possible, but doesn't and can't require connectors to do so). Just for the record, here are a few guidelines about writing connectors: https://github.com/ConduitIO/conduit/blob/4b66347eb89a2bb8cf641565f7602bf36bc33137/docs/writing-a-connector-guidelines.md.

[nickchomey]

I can confirm that destination connectors are fairly simple. I've made one for surrealdb.

But this one should be ready soon, so you might as well wait.

[nadilas]

We just tested it today and seems to work okay. We need to get some processors in to convert e.g sql server datetime string to MySQL format, but the basics seem to work.
Will do some further testing tomorrow, thanks guys

[lovromazgon]

@nadilas you're welcome to join our Discord server (if you haven't already) and let us know how it goes or ask if something is unclear, we're happy to assist you 😉

[hariso]

FYI, the destination PR was merged.

[raulb]

This is done already. Will be released today.

[nickchomey]

Can I suggest reviewing and merging this PR (conduitio-labs/conduit-connector-mysql#51) for an All Tables wildcard (*), before releasing the connector? It has been updated to the latest version of the connector, and is ready for review.

Also, I'm happy to update conduitio-labs/conduit-connector-mysql#54 to allow for selecting tables via include/exclude regexes. Unless you think its something that requires more consideration from the team before merging.

[raulb]

@nickchomey Makes sense. We might release 0.1.0 today with #51 considering how close it is, but we'll make sure #54 is released soon as well. #51 only needs a CI fix, and I just pinged @lovromazgon and @alarbada to follow up with you on #54 (it's going to need a rebase for now).

[nickchomey]

Sounds good. Yeah, I just reviewed 54 and I think it needs a bit more work. moreover, It somewhat overrides/supersedes/replaces 51 - the regex handles all tables rather than the mechanism in 51.

I'll try to tidy it up today and the team can review it when there's time.

ps. the #51 and #54 links you used point to this repo, not the connector one