Suggestion: transition from marshmallow to pydantic 2

[ksbrar]

Hey @mhabedank ,

Not to add to your workload - I just noticed you closing lots of tickets and PRs recently.

Just one suggestion (which you are already probably aware of): I would strongly consider transitioning the schemas at some point from marshmallow to Pydantic 2, which should have all of the flexibility Ludwig needs. Pointedly, I would remove the dependency on marshmallow-jsonschema first, because it has gone unmaintained for over 2 years and caused a bit of dependency hell, holding back further iteration. It's a tough one because it's inherent to the dataclass-centric design of the schemas (one dataclass = one json schema), but it's necessary IMO (though if you moved to Pydantic you would get back the dataclass-centric design anyways).

[mhabedank]

Hi @ksbrar, you mention my favorite word of the last 2 weeks: dependency. :-)
We are currently working intensively on straightening out various dependencies, as they ensure that Ludwig is no longer usable at all on many systems. Just to name a few: torchtext, ray, numpy, etc. I'll take a look at the topic of marshmellow/pydantic. I already had the idea that the data classes should be defined language-agnostically in order to improve the implementation of third-party tools.
One question for you: Would you have time to take care of this?

[ksbrar]

@mhabedank Unfortunately I don't have time at the moment, but I may as we get closer to the holidays.