Provide a built-in database

[iSaluki]

Describe the project you are working on

I am working on several projects currently, however the one that would benefit most from a feature like this would probably be my API-based adventure game. This is a mainly text based game, and the Godot build for it is just a client. This game, and basically every game I'm working on would benefit from having a built-in database in some way. With my adventure game it would be securely storing user credentials and caching user information and progress from the server.

Describe the problem or limitation you are having in your project

Whilst there are data storage methods available currently, there are issues with them. Storing a .json file is fine and it works, but it is easily corrupted if you need to make updates to it frequently and it relies on the developer to secure it. For some games, using JSON is absolutely fine but for more complex games or just any games that want to prevent cheating it is necessary to be able to store your local data securely and in a well-managed fashion.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

In my project, I would benefit from support for a local database greatly because it could be tightly integrated with GDScript and the rest of the engine. This would immediately make it more performance efficient and reliable than most bring-your-own solutions or using an available addon. It could also support all platforms straight away and provide integrated tools in the editor to help developers to manage and create good databases.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Some basic ideas mapped out in MasterPlan, definitely not a full overview but it hopefully provides a starting point. I'm hoping to be able to discuss this idea with some people so that we can develop it a bit more.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Technically it could, yes. However, there would be benefits to having it baked into the main engine that I think would outweigh this.

Is there a reason why this should be core and not an add-on in the asset library?

As a core feature, not only could it integrate with the main engine and have proper tools in the editor, but developers could rest assured that their local storage solution wouldn't suddenly be discontinued, or have a high risk of corruption/breaking.

[Calinou]

Storing a .json file is fine and it works, but it is easily corrupted if you need to make updates to it frequently and it relies on the developer to secure it. For some games, using JSON is absolutely fine but for more complex games or just any games that want to prevent cheating it is necessary to be able to store your local data securely and in a well-managed fashion.

Godot can already load and save encrypted files. See the File class' open_encrypted method.

A "secure" database is not any more secure than its encryption method is.

There's a pull request that implements SQLite, but it may not be merged as it increases binary size, yet most projects don't need it.

[iSaluki]

Interesting, I hadn't known about that before. Would you be able to store and retrieve objects from a file easily? And if, for example, I wanted to search through a fairly large amount of data, would a standard file be able to handle this properly?

[Calinou]

Would you be able to store and retrieve objects from a file easily?

Yes; you can use the ConfigFile class for this. It can serialize any Godot datatype except Objects (which you can convert to a dictionary).

And if, for example, I wanted to search through a fairly large amount of data, would a standard file be able to handle this properly?

How large is "large" here? 🙂

A database is more suited for modifying data in a large file (since the entire file doesn't need to be overwritten), but if most of your operations are reads, the performance difference isn't as big. Also, you could split your file into multiple files to improve write performance.

[iSaluki]

Thanks for the fast and detailed replies! Now that I've discovered the File class I've been able to read through the docs for it and I've found that it could probably do everything I want here, which is awesome.

Splitting things into multiple files will probably work absolutely fine for most things, but I might still end up with some pretty big files.

And "large" is several thousand objects. I'm not sure how well a file would handle that, but would be interested to find out.

Other than that, my only other concern with using a file would be querying it. After I load the file up, how easy/performance intensive would it be to search my large file for a specific object?

[Calinou]

Other than that, my only other concern with using a file would be querying it. After I load the file up, how easy/performance intensive would it be to search my large file for a specific object?

It really depends on how the data is structured, how large the object is, etc. I'd recommend benchmarking it on your own and extrapolating this based on the hardware you wish to target.

[iSaluki]

Okay, I'll try it out at some point soon.

I don't think it's necessary to keep this issue open any longer though, as the file class will likely be able to do everything I need, and the SQLite PR would likely be my best bet if there's something I can't do with a standard file.

Thank you so much for your help with this! I didn't expect anywhere near this level of support from a proposal, and I just want you to know I appreciate the effort.

[Xrayez]

See also godot-extended-libraries/godot-ideas#20 if you'd like to discuss this further.