Compile-time checked queries for Rust

[tailhook]

Codegen in Rust

Instead of generating .rs files directly, it's more common to use procedural macros to generate code in Rust. Either by using expression macro:

let result = query!(conn, "SELECT Object { property }").await?;
let result2 = query_file!(conn, "./query.edgeql").await?;

Or by generating the whole module:

query_file!("./my_query.edgeql"); // generates `mod my_query {..}`

async fn do_something(conn: &mut edgeql_tokio::Connection) {
    let result = my_query::query(conn).await?;
}

We'll be discussing expression macro below, as it looks like more convenient.

Upsides of this codegen approach:

1. Queries don't have to be in the separate files and still type checked
2. Build and typecheck process is fully integrated, no need to run a separate codegen tool (but see "offline mode")

Downsides:

1. Error reporting on type issues can be harder to implement (see below)

What Does it Generate?

Generated code for:

let result = query!(conn, "SELECT Object { property }").await?;

... looks similar to this (although, this piece is simplified for brevity):

#[derive(Debug)]
struct Object {
    property: String,
};
let query = conn.query("SELECT Object { property }").await?;
let result = Vec::new();
for mut element in result {
    result.push(Object {
        property: String::decode(&mut element)?,
    });
}

Another possible option is to use query_as!(..). In this case you can declare the structure manually:

#[derive(Debug, PartialEq)]
struct Object {
    property: String,
};
let result = query_as!(Object, "SELECT Object { property }").await?;

This is needed if you need to name a type or make it public and to implement more traits for it. The expansion of it is the same (except struct definition).

The magic of typechecking in the latter case is in the constructing object code:

property: String::decode(&mut element)?,
       // ^^^^^^--- this type name correponds to the database type.

Offline Mode

For offline compilation (i.e. when database is not used for compilation) we need type information for all the queries committed to git.

We can't generate that as part of the regular Rust build because:

1. Multiple macro invocations are isolated during Rust build
2. Due to sophisticated Rust caching system there is little guarantee that all such macros are visited during last compilation (and no extra stuff left)

Also we need --all-features --all-targets build to ensure that all queries are in the cache (which is not easily accessible sometimes).

So we probably have to do what sqlx does:

1. Have another command to generate the cache probably cargo edgedb or edgedb-generate or eventually a part of edgedb command-line tool
2. The command runs cargo+rust with special flags to avoid Rust caching (and full build), but that dumps type info as as side effect of the build
3. Then gathers all that type info into the git-comittable file(s)
4. Uses offline macro feature to read those files
5. Additionally, generated files contain hash of all the schema files that is checked during codegen.

Open Questions

It's Less Powerful than Derive

When you use derive(Queryable) its possible to use types that are "compatible" to EdgeDB types but different ones. For example instead of using edgedb::model::Datetime you can use std::time::SystemTime, or instead of Vec<T> you can use BTreeSet:

#[derive(Queryable)]
struct Object {
    timestamp: std::time::SystemTime,
    tags: std::collections::BTreeSet<String>,
}

To make query_as! work with derive(Queryable) rather than only with plain structs with limited set of types we need some creative approach.

Short overview of non-solutions:

1. Use Into or TryInto conversion for each field after decoding: (a) it makes typechecking loose (i.e. there are conversions between types of integers) and (b) the types can be decoded are T: TryFrom<U> where U is from fixed set of types, rather than any T: Queryable.
2. Include struct definition into query_* macro (so procedural macro can parse it): (a) this is quite inconvenient and (b) type information is not available in programmatic macros, so it's not generic enough anyways.

Const Type Descriptors

One possible solution is to make Queryable define an internal type as const property:

trait Queryable {
    const INTERNAL_TYPE: EdgedbInternalType;
}

Then expansion of query_as!(Object, ...) should work like this:

const TARGET_TYPE = EdgedbInternalType::Object(&[
    ("timestamp", EdgedbInternalType::Datetime),
    ("tags", EdgedbInternalType::Array(&[EdgedbInternalType::Str])),
]);
const_assert!(Object::INTERNAL_TYPE == TARGET_TYPE);

let query = conn.query("SELECT Object { timestamp, tags }").await?;
let result = Vec::new();
for mut element in result {
    result.push(Object {
        property: String::decode(&mut element)?,
    });
}

Yet to be determined:

1. Whether all type descriptors can be represented by const values, we can have issues with: (a) enums, (b) types that are same in Rust but different in EdgeDB (Set vs Array, NamedTuple vs Object).
2. Whether error reporting is sufficiently clear when type mismatch happens (but see below)

Inner Type

Another possible solution is to do the same but at a type level:

trait Queryable {
    type BaseType: QueryableBase;
}
impl<T> Queryable for HashSet<T> {
    type BaseType = Vec<T>;
}

This has to be explored more.

Error Reporting on query_as!

Here is how error looks like (this is adapted from sqlx):

error[E0308]: mismatched types
  --> src/main.rs:77:16
   |
77 |       let rows = edgedb::query_as!(
   |  ________________^
78 | |         Person,
79 | |         r#"
80 | | SELECT Person { id, person }
...  |
83 | |         "#
84 | |     )
   | |_____^ expected `bool`, found `uuid::Uuid`
   |
   = note: this error originates in the macro `$crate::edgedb_macros::expand_query` (in Nightly builds, run with -Z macro-backtrace for more info)

This gives you zero understanding of which field has the wrong type.

Non-solutions:

1. We can't have anything that programmatically checks types in procedural macro as type information is not yet available there.

Using Const Type Descriptors

One possible solution is to use

trait Queryable {
    const INTERNAL_TYPE: EdgedbInternalType;
}

And then make const-evaluated assertion method:

impl EdgedbInternalType {
    const fn assert_eq(&self, other: &EdgedbInternalType) { ... }
}

The method uses panic!() to describe when type is wrong. This has some complexity due to limitations of the const fn code but to the best of my understanding should be doable.

TBD

1. This text discusses result type. What are the issues for argument type checking?
2. Is it possible to typecheck globals required for the query? (probably no)

[1st1]

First, this is great and I love that we can implement magic like this query!(conn, "SELECT Object { property }").await?; in Rust.

Some comments:

1. It seems that implementing query_as! is more complicated than implementing query! and the latter is less controversial. Maybe we start with implementing query! first?

2. It would still be nice to support loading queries from .edgeql files as those can be reused between multiple languages in the same repo.

3. Is it possible to typecheck globals required for the query? (probably no)

   That's correct, currently the answer is no. But we can add a compiler flag to the protocol to return you this information when you parse the query somehow (cc @fantix who's currently working on other changes to the proto for edgedb-python's codegen)

[tailhook]

It seems that implementing query_as! is more complicated than implementing query! and the latter is less controversial. Maybe we start with implementing query! first?

Yes. But if you can't add additional traits to the structure you'll have to copy all the values to the custom structure and that will make it even less convenient than using normal queries.

It would still be nice to support loading queries from .edgeql files as those can be reused between multiple languages in the same repo.

Sure. sqlx has query_file!(conn, "./filename.edgeql") for that. We'll probably do the same.

[tailhook]

That's correct, currently the answer is no. But we can add a compiler flag to the protocol to return you this information when you parse the query somehow (cc @fantix who's currently working on other changes to the proto for edgedb-python's codegen)

While this is a prerequisite, this is not the complex part. The problem is that then connection pool type should be marked with globals. And it's unclear how to check that type statically, given that it has to have "subset of" semantics (i.e. there might be more globals specified than expected for this query). And then every function signature should carry that type.

Note: while we generate arbitrary code programmatically in procedural macros, type information is not available at that stage, so we can't typecheck programmatically or explore types other than those defined in the macro itself (i.e. can only rely on syntactic parsing of the macro contents and general semantics of the generated Rust code). So there are limited amount of things we can do.