Serve Data with Low-Latency

---

1. Features 🎍

data-api discovers your datastore data & exposes them as a REST read-only API.

💡 By datastore we mean firestore in datastore mode.

Features include:

• generate open-api spec definition
• generate Swagger-UI documentation page
• expose datastore data via key-value or property-filtering.
• cache responses with configurable duration
• manage permissions with api-keys or openID tokens.

💡 data-api is great to create a low-latency API for your BigQuery data (that you can export to datastore with one sql query. More below).

2. Deploy 🚀

Run

gcloud run deploy data-api \
  --source . \
  --set-env-vars "PROJECT=$PROJECT,DATABASE=$DATABASE"

with:

• $PROJECT the Google Cloud Project where you datastore database resides
• $DATABASE the name of your datastore database (if not given, it will use the default database).

You can then get the url of your data-api Cloud Run Service by running:

gcloud run services describe --format "value(status.url)"

3. Exposed Routes 🚚

💡 Datastore Reminder:

• Data in datastore is organized in a hierarchy: database/namespace/kind (like the database/schema/table hierarchy in relational databases).
• Inside a kind (think table), the entities (think rows) have a key (a string or integer) and a value (a dict).

data-api Cloud Run service:

• exposes the data of the database defined as environment variable at deploy time (see above).
• considers each namespace as a different api (which has its own open-api spec definition and Swagger UI).
• exposes the following routes (with GET method):

url	Description
/	Redirects to /api/
/api/	Returns the list of namespaces in database
/api/<namespace>/	Returns details on namespace api including its kinds and urls
/api/<namespace>/openapi.json	Returns the openapi spec definition of the namespace api
/api/<namespace>/swagger-ui.html	Returns the Swagger UI (documentation portal) of the namespace api
/api/<namespace>/<kind>/	Returns a list of entity values of kind
/api/<namespace>/<kind>/<key>	Returns the entity value of key
/api/<namespace>/<kind>/?foo=bar	Returns a list of entity values of kind for which foo property is equal to bar

4. [BONUS] Export BigQuery table to datastore ✍️

data-api is great to create a low-latency API for your BigQuery data.

You can export a BigQuery table into datastore with one sql command to make it available to data-api:

call bigfunctions.eu.export_table_to_datastore(
  'your-project.dataset.table',
  'user_id',
  'your-project/your-database/default/users'
);

💡 If you don't want to use the public bigfunction, you can deploy the function in your own BigQuery project. Check the function documentation.

5. Contribute 👋

Any contribution is more than welcome 🤗!

• Add a ⭐ on the repo to show your support
• Join our Slack and talk with us
• Raise an issue to raise a bug or suggest improvements
• Open a PR!

6. TODO 📝

• manage permissions with a metadata key-value
• make queries work when filtering with integer (using the datastore schema)
• extend this concept to other backends