Storage and querying

lofi aims to not only be a great client-server sync library, but also provide a delightful interface over IndexedDB, our notoriously awkward web-native database.

IndexedDB has a very constrained feature set which makes it unintuitive to anyone used to SQL's extensive querying capabilities. As the name implies, IndexedDB only really offers indexes - pre-computed bits of data which can quickly connect you to the matching objects. That means if you want to optimally look up data, you'll want to plan ahead and create relevant indexes.

lofi's schema design is oriented around exposing easy ways to create useful indexes that help you query in a way that's intuitive and efficient.

TODO: more here

Compound indexes

Compound indexes are a way to leverage how IndexedDB works (that is, its limitations) to still provide complex queries like "items tagged 'indexeddb' in ascending order of id." You might wonder how that can be done when the database only allows indexes to reference a single key, and only order by that key's property value!

The trick is to compute certain indexable properties before storing the object, structured in such a way that we can then lookup these values later using intuitive query semantics.

Let's model a simple compound query, "items in category 'development' in ascending order of id," on the following document schema:

interface Post {
  id: string,
  tags: string[],
  category: string,
  content: string,
  published: boolean
}

First, before storing each Post, we write a new 'invisible' value to it for our synthetic index. But what do we write to it so we can query it like above? Since IndexedDB only allows ordering and limiting on the entire value (not a subset, like just filtering on 'category' but sorting on 'id'), we need to be thoughtful about the structure of our value.

const post = {
  id: cuid(),
  tags: ['react'],
  category: 'development',
  content: 'useEffect is back, baby!',
  published: false
};

post.category_id = `${post.category}#${post.id}`;

We have to think ahead to how we will use this index to decide how it's structured. For example, because we intend to filter on category and sort on id, category must come first! We also choose a particular character as our separator - not because we need recognizably distinct sections (this value is not going to be read in our application), but because of the sorting properties of that character.