Database-backed Gettext

[ghost]

Hi!

Hope you're well.

I and my team recently thought about integrating gettext with the database as a persistence point; unfortunately, we've stumbled upon this topic on the elixir forum: https://elixirforum.com/t/gettext-backed-by-db-ecto/19099 that suggested alternate solutions, with concerns that it might not be achievable.

However, we believe this is the best place to ask about such an issue; if possible, I would be more than happy to help make it happen.

One of the solutions we imagined might help could be:

• Allow configuring gettext to delegate function calls to a specified custom module, and - based on returned value - either return it further or fall back to calling original Gettext module functions.

I'm open to discussion.

👋

[josevalim]

👋

Please go ahead and explore this discussion. I believe all of the lookups happen through lgettext and lngettext functions, so you should be able to replace them in a custom backend to at least validate the approach:

defoverridable lgettext: ..., lngettext: ...

defp lgettext(...) do # now lookup in the database

Alternatively you can mix the projects and use handle_missing_translation to only go to the database when you don't have an alternative.

So there should be no roadblocks for exploration and we can circle back once there is something you are pleased with. :)

[josevalim]

Hi @czwartyeon, do you have any updates here? :)

[bamorim]

I've been thinking about this problem because we've been looking into an option to automatically update translations without having to rebuild our applications and redeploy everything and I think I have an idea that could be generic enough.

I particularly don't think it would be wise to hit the database for every translation request, so my idea would be to have, apart from the compile-time lgettext/lngettext, a backend that reads that from a given ETS table.

If we standardize this ETS table, we could then have different "syncers" that would fetch the translations, parse, and insert into the table and those would live on different libraries.

An example library, would get all pot files and then query a KV (redis or etcd) for po files, then it would parse these files and insert all strings in the ETS table.

There could be a lot of different approaches that could get binary po files from a database or even the actual strings directly from the database.

That being said, I think having a standard ETS format would be beneficial for these "runtime" loading and maybe could ship with a loader that just reads the .po files from a path in the system, which would at least allow to change the po files without recompiling elixir (this might be useful for cases where the app is delivered compiled but we still want to provide OTA localization updates, like in a Nerves device, for example).

If you think this approach would make sense, I can start working on a library to implement (my understanding is that I can quite easily do that without having to change Gettext, but I'll probably need to "re-implement" the use Gettext macro.

[josevalim]

Exploring this approach is certainly very welcome!

[bamorim]

@josevalim today I managed to give this a go and I think I made it work, but still a proof of concept.

I created this repository: https://github.com/bamorim/runtime_gettext

It contains three projects:

• rg_demo: A demo phoenix app just to show how something would look like in real life
• runtime_gettext: The core that I'll talk more about next
• runtime_gettext_po: A utility to load translations from po files and save them on the RuntimeGettext.ETSRepo

RuntimeGettext

The RuntimeGettext library is composed by:

RuntimeGettext

Implements a macro that does the defoverridable and implements lgettext and lngettext to load strings in runtime from a RuntimeGettext.Repo, which by default is RuntimeGettext.ETSRepo. If the repo returns that it doesn't have the translation, this calls super to use the compile-time functions.

One the few problems I faced here are:

• I had to re-implement the call to the Gettext.Plural to get the plural form. Maybe if we had an extension point with the plural form "built in" that would be useful (maybe that is also useful to make the handle_missing_translation easier as well).
• I had to re-implement the call to the Interpolation module, again, maybe an extension point that is just about "get me the translation string and I'll do the rest" could be useful. Of course, in this case I had to call runtime_interpolate instead of the compile time it is being called on the default gettext implementation.
• The configs to make it "the same" as gettext, I had to reimplement the call. Maybe I should have used __gettext__ and assumed it was a public interface? Regardless, the current __gettext__ implementation doesn't include the :plural_forms option, so maybe we should include that so in this code I just call __gettext__(:plural_forms) and __gettext__(:interpolation)

RuntimeGettext.Repo

This is just a behaviour to get the translation strings in runtime. It is basically:

  @callback get_translation(locale(), domain(), msgctxt(), msgid()) ::
              {:ok, msgstr()} | {:error, :translation_not_found}

  @callback get_plural_translation(locale(), domain(), msgctxt(), msgid(), plural_form()) ::
              {:ok, msgstr()} | {:error, :translation_not_found}

RuntimeGettext.ETSRepo

This saves msgstrs in an ETS table and implements the RuntimeGettext.Repo behaviour to return the saved strings.
It also exposes an add_translation and add_plural_translation to be used by the "fetchers". For now it doesn't have any functions to remove existing translations nor to list all translations, but these should be easy to implement.

Right now, I'm also not checking anything (like if the values are the correct type), but this should probably be considered.

RuntimeGettextPO

This is just a proof of concept that exposes the function: RuntimeGettextPO.load_po_files(path) so if one wants to use the exact same structure as gettext has by default, lets say in priv/gettext but load the actual values on runtime (so they can be, for example, just mounted in a docker image instead of requiring re-compilation of the whole project), one could just

RuntimeGettextPO.load_po_files(Application.app_dir(:my_otp_app, "priv/gettext"))

For example, in their OTP application (similar to how one would setup telemetry, for example).

Of course, more sophisticated approaches could come from that, like listening to filesystem events or polling from there, which maybe, in turn, could be tied together with, for example, an s3 bucket and a s3 fuse adapter to sync the files.

My goal here was to just give a kickstart to this, but now it is already late, but I can try to organize this better later.

Next Steps

I'm not sure how we could proceed here. If we want to go the route of first having this as a separate library, I can go ahead and split the repository into multiple repos and also publish them to Hex.pm. I also don't know if you want to have that on the elixir-gettext organization, it is up to you.

That's all for now. I'm really tired hahaha.

[josevalim]

this looks great! What we probably want to do is to allow multiple repos to be given. Then we go through each repo in order and if they don’t find something they return :none. This way we can also keep the call to super.

If we do find something then we call the Interpolator. Otherwise we say translation not found.

Would you like to submit a request to this repository exploring how it would look like? Thank you!

[bamorim]

Hey, thanks for the response. I like the idea of multiple repos, I can definitely give it a try. However, at the same time, it might be putting a lot of complexity unnecessarily as one could easily create a repo that does exactly that: look into multiple sources.

But I can explore this idea, for sure.

One thing I didn't got is "Otherwise we say translation not found". This means we call super?

[bamorim]

Also, when you say "submit a request to this repo" are you referring to bamorim/runtime_gettext or elixir-gettext/gettext?

[josevalim]

The idea of multiple repos is to solve the problem with super. This way it is your choice to keep the default behavior as a fallback or not.

If none of the repositories have a translation, then we return the error translation not found tuple.

And I meant a pull request to this repo, yeah :)

[bamorim]

So one of the repos would be the one "calling super"? I don't see how that is possible since only the function can call super. Maybe if the repo just returns :super instead.

[bamorim]

Unless we change the compiler to create the default compile-time behavior as a repo (either in a nested module or in that same module) and the default repos being [__MODULE__] or [__MODULE__.Repo]

[josevalim]

I see what you mean. So I agree with you, there is no need for a list, we just call a repository module if any was configured.

[bamorim]

Okay, I'll open a PR with a proposal on how that would look like "natively" here.

[whatyouhide]

Closing in favor of the PR. 💯

[peaceful-james]

@whatyouhide Since the PR was closed, should this issue be re-opened?

[josevalim]

@peaceful-james in this case, the PR was closed because we don't believe this is the responsibility of Gettext overall, so it should remain closed. I think a generic translation layer should probably be built on top of Gettext, as outlined in the PR.