Custom per-file cache size

[Inviz]

Hello there! Wonderful and impressive project you've got. Thanks for your work.

I'm trying to get SQLite to work on top of littlefs, so far it seems like a pretty smooth ride. However one thing I'm curious about is if there's any conceptual problem in having certain files have larger cache size. Currently stock VFS demo examples have their own write cache just for the journal file (8kb that is), while other files just fall back to whatever FS does (which in case of LFS is cache_size cache). I have an option to keep that SQLite own's write cache, but i was looking into re-using the cache used within the LFS instead.

So LFS has the same cache size for all files as specified by cache_size. Explicitly provided buffer has to be of cache_size. The code refers lfs->cfg->cache_size in a bunch of places. However from a cursory glance it seems that it is not fundamentally incompatible in using cache.size instead.

I would provide external buffer with size larger than lfs->cfg->cache_size, and then adjust file->cache.size explicitly. Would that be a bad idea? I could make a PR to adjust uses of cfg->cache_size if this isn't too dumb

[Inviz]

Aww yeah. i guess i'm wrong. LFS updates cache->size on the fly and limits it to lfs->cfg->cache_size. So the only way would be to add a new field to lfs_file_config, and even then it would require some other code adjustments to pass it around.

Either way, what are your thoughts on this? Is it too much trouble for the LFS?

[geky]

Hi @Inviz, glad you find the project interesting.

I think I have looked into this before, though I can't find which issues at the moment. The extra flexibility of per-cache sizes would be nice to have, though it would come at a tradeoff of additional code-size (though probably not enough to worry about). It might get a bit tricky because cache_size is also used to decide the cutoff for inline-files, but different limits for inline-files should be handled internally, worst case you might expose untested bugs there.

It is worth noting that LittleFS doesn't take advantage of cache sizes larger than the block_size, so if you're expecting to keep a large, multi-block file around in the cache, it probably won't provide much benefit.

I also want to point out that LittleFS does not handle random file-writes performantly (#27), and my understanding is that SQLite relies on random writes pretty heavily. It might be fine for small databases, and it's always a good idea to measure performance to see if it's acceptable for your use case, but I just want to make sure that doesn't come as a surprise.

[Inviz]

Thanks a lot for the response.

It is worth noting that LittleFS doesn't take advantage of cache sizes larger than the block_size, so if you're expecting to keep a large, multi-block file around in the cache, it probably won't provide much benefit.

I see now, then yes it doesn't make a lot of sense to try having a large cache on file in my use case. I'm using sdcard with 512 block_size. It could be more useful for other cases, but that might be less relevant.

I also want to point out that LittleFS does not handle random file-writes performantly (#27), and my understanding is that SQLite relies on random writes pretty heavily. It might be fine for small databases, and it's always a good idea to measure performance to see if it's acceptable for your use case, but I just want to make sure that doesn't come as a surprise.

Yeah, I've seen that issue and another related one... not sure what to say here, I guess I didn't want to think about it :) It does make things more complicated for sure. I understand that it could be possible to use WAL (Write Ahead Log) in SQLite to reduce amount of random writes, but it comes at the cost of read speed. It looks like it could also be possible to split databases into multiple separate files, separating rarely updated tables from hot ones... Either way it's a bit clunky and has tradeoffs.

Do you think LFS may get some sort of optional code path to alleviate the issue of random writes? I don't expect there can be ideal solutions, but some choice would be really helpful.

[Inviz]

I came up with idea to let littlefs manage filesize via custom attribute, keeping files empty. Then I will use raw disk writes to actually maintain sqlite files bypassing littlefs, both handling #27 and the issue with slow block allocation . The database already has a data recovery mechanism (it backs up pages before overwriting them), so I can probably get good behavior from it. The tricky part as it seems right now is making sure filesize metadata doesnt desync from the written files.

[geky]

Then I will use raw disk writes to actually maintain sqlite files bypassing littlefs

Note this discards wear-leveling (though you note the database handles power-loss).

Wouldn't it be better in this case to store the database in a separate partition next to littlefs? I don't have code available, but an MBR partition table is a very simple partition scheme with tooling on every system.

It looks like it could also be possible to split databases into multiple separate files, separating rarely updated tables from hot ones...

It might take a bit of code to tie into SQLite, but just splitting the file into equal chunks as separate files would improve the performance. Agree it is unfortunately clunky...

I'm using sdcard with 512 block_size.

You can always set LittleFS's block-size to a multiple of the physical block-size. This can moderately improve some of the performance around allocation at the cost of reducing granularity. Most other filesystems support this for similar reasons which is why FAT usually defaults to 8 KiB logical blocks (called clusters over there).

Do you think LFS may get some sort of optional code path to alleviate the issue of random writes?

This is something I'm exploring, but it will be a big change to the LittleFS internals. So unfortunately it won't be ready or usable for a while. (The best option so far is to rip out the existing file data-structure and replace it with a B-tree, but this raises some problems with how files interact with block allocation.)

[Inviz]

Note this discards wear-leveling (though you note the database handles power-loss).

Since I use sdcard which are supposed to have its own wear-levelling system it seems it should be OK. Though I am winging it here at this point. Making stuff up as I go.

Wouldn't it be better in this case to store the database in a separate partition next to littlefs? I don't have code available, but an MBR partition table is a very simple partition scheme with tooling on every system.

I'll need to read up on this to see the implications of using MBR or other partitioning schemes. I wanted to in fact to reserve a portion of sd-card for sqlite and its files, allocated very generously. Say 4gb for main db file, and then in hundreds of megabytes for each temp file. I want to do it in a way that everything has fixed hardcoded addresses that i can write to. The only technicality is that SQLite sometimes wants to know file size of the files, so I would need to store that somewhere in a way that wouldn't be corruptable. I was thinking of doing it in the littlefs, but i gather that could be a bit overkill. But i was thinking that i could make this strategy work by wrapping littlefs with my custom behavior. So instead of having two separate FSes, i could opt-in into the weird raw disk storage by setting a custom attribute on the file (e.g. I call it raw fixed address) which would do my custom writes and reads in place of what LFS does internally.

You can always set LittleFS's block-size to a multiple of the physical block-size. This can moderately improve some of the performance around allocation at the cost of reducing granularity.

Thanks for the tip! I think I can consider that in the future. I can see how that can make things easier a bit

This is something I'm exploring, but it will be a big change to the LittleFS internals. So unfortunately it won't be ready or usable for a while.

Thanks for a honest answer and I understand how difficult the task is ahead of you. No software project is perfect, and your work already does a good job for a lot of use-cases. So I appreciate what you've done.