Skip to content

feat: standard database functions everywhere #1750

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 128 additions & 59 deletions guides/databases.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ impl-variants: true

<div class="impl node">

### Migrating to New Database Services? {.node}
### Migrating to the `@cap-js/` Database Services? {.node}

With CDS 8, the new database services for SQLite, PostgreSQL, and SAP HANA are now generally available. It's highly recommended to migrate. You can find instructions in the [migration guide](databases-sqlite#migration). Although the guide is written in the context of the new SQLite Service, the same hints apply to PostgreSQL and SAP HANA.
With CDS 8, the [`@cap-js`](https://github.com/cap-js/cds-dbs) database services for SQLite, PostgreSQL, and SAP HANA are generally available. It's highly recommended to migrate. You can find instructions in the [migration guide](databases-sqlite#migration). Although the guide is written in the context of the SQLite Service, the same hints apply to PostgreSQL and SAP HANA.

### Adding Database Packages {.node}

Expand Down Expand Up @@ -358,63 +358,6 @@ The operator mappings are available for runtime queries only, but not in CDS fil
:::


### Functions Mappings for Runtime Queries {.node}

A specified set of standard functions is supported in a **database-agnostic**, hence portable way, and translated to database-specific variants or polyfills.
Note that these functions are only supported within runtime queries, but not in CDS files.
This set of functions are by large the same as specified in OData:

* `concat(x,y,...)` — concatenates the given strings or numbers
* `trim(x)` — removes leading and trailing whitespaces
* `contains(x,y)` — checks whether `y` is contained in `x`, may be fuzzy
* `startswith(x,y)` — checks whether `y` starts with `x`
* `endswith(x,y)` — checks whether `y` ends with `x`
* `matchespattern(x,y)` — checks whether `x` matches regex `y`
* `substring(x,i,n?)` <sup>1</sup> —
Extracts a substring from `x` starting at index `i` (0-based) with optional length `n`.
* **`i`**: Positive starts at `i`, negative starts `i` before the end.
* **`n`**: Positive extracts `n` items; omitted extracts to the end; negative is invalid.
* `indexof(x,y)` <sup>1</sup> — returns the index of the first occurrence of `y` in `x`
* `length(x)` — returns the length of string `x`
* `tolower(x)` — returns all-lowercased `x`
* `toupper(x)` — returns all-uppercased `x`
* `ceiling(x)` — rounds the input numeric parameter up to the nearest numeric value
* `floor(x)` — rounds the input numeric parameter down to the nearest numeric value
* `round(x)` — rounds the input numeric parameter to the nearest numeric value.
The mid-point between two integers is rounded away from zero, i.e. 0.5 is rounded to 1 and ‑0.5 is rounded to -1.
* `year(x)` `month(x)`, `day(x)`, `hour(x)`, `minute(x)`, `second(x)` —
returns parts of a datetime for a given `cds.DateTime` / `cds.Date` / `cds.Time`
* `time(x)`, `date(x)` - returns a string representing the `time` / `date` for a given `cds.DateTime` / `cds.Date` / `cds.Time`
* `fractionalseconds(x)` - returns a a `Decimal` representing the fractions of a second for a given `cds.Timestamp`
* `maxdatetime()` - returns the latest possible point in time: `'9999-12-31T23:59:59.999Z'`
* `mindatetime()` — returns the earliest possible point in time: `'0001-01-01T00:00:00.000Z'`
* `totalseconds(x)` — returns the duration of the value in total seconds, including fractional seconds. The [OData spec](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part2-url-conventions.html#sec_totalseconds) defines the input as EDM.Duration: `P12DT23H59M59.999999999999S`
* `now()` — returns the current datetime
* `min(x)` `max(x)` `sum(x)` `average(x)` `count(x)`, `countdistinct(x)` — aggregate functions
* `search(xs,y)` — checks whether `y` is contained in any of `xs`, may be fuzzy → [see Searching Data](../guides/providing-services#searching-data)
* `session_context(v)` — with standard variable names → [see Session Variables](#session-variables)
> <sup>1</sup> These functions work zero-based. E.g., `substring('abcdef', 1, 3)` returns 'bcd'

> You have to write these functions exactly as given; all-uppercase usages aren't supported.

In addition to the standard functions, which all `@cap-js` database services support, `@cap-js/sqlite` and `@cap-js/postgres` also support these common SAP HANA functions, to further increase the scope for portable testing:

* `years_between` — Computes the number of years between two specified dates.
* `months_between` — Computes the number of months between two specified dates.
* `days_between` — Computes the number of days between two specified dates.
* `seconds_between` — Computes the number of seconds between two specified dates.
* `nano100_between` — Computes the time difference between two dates to the precision of 0.1 microseconds.

The database service implementation translates these to the best-possible native SQL functions, thus enhancing the extent of **portable** queries.
With open source and the new database service architecture, we also have methods in place to enhance this list by custom implementation.

> For the SAP HANA functions, both usages are allowed: all-lowercase as given above, as well as all-uppercase.

::: warning Runtime Only
The function mappings are available for runtime queries only, but not in CDS files.
:::


### Session Variables {.node}

The API shown below, which includes the function `session_context()` and specific pseudo variable names, is supported by **all** new database services, that is, *SQLite*, *PostgreSQL* and *SAP HANA*.
Expand Down Expand Up @@ -940,7 +883,133 @@ Instead, they protect the integrity of your data in the database layer against p
→ Use [`@assert.target`](providing-services#assert-target) for corresponding input validations.
:::

## Standard Database Functions

A specified set of standard functions - inspired by [OData](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part2-url-conventions.html#sec_StringandCollectionFunctions) and [SAP HANA](https://help.sap.com/docs/HANA_SERVICE_CF/7c78579ce9b14a669c1f3295b0d8ca16/f12b86a6284c4aeeb449e57eb5dd3ebd.html?locale=en-US) - is supported in a **database-agnostic**, hence portable way, and translated to database-specific variants or polyfills.

### OData standard functions

The cds-compiler and all CAP Node.js database services come with out of the box support for common OData functions.

::: warning Case Sensitivity
The OData function mappings are case-sensitive and must be written as in the list below.
:::

#### String Functions

- `concat(x, y, ...)`
Concatenates the given strings or numbers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we support the concat operator


- `trim(x)`
Removes leading and trailing whitespaces.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • not available in CAP Java
  • returns String


- `contains(x, y)`
Checks whether `y` is contained in `x` (fuzzy matching may apply).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java
    • Value.contains(other)
    • Value.contains(other, isCaseInsensitive) to control if case sensitive
    • Value.contains is exact and never fuzzy
    • represented by CqnContainmentTest
    • implemented by LIKE or ILIKE, if available
  • is the Compiler's implementation case sensitive or case insensitive
  • OData v4, CAP: case sensitive
  • OData v4, RAP: case insensitive (!)
  • returns Boolean

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all OData inspired functions are case sensitive


- `startswith(x, y)`
Checks whether `y` starts with `x`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: startsWith
  • case sensitive?
  • return Boolean

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all OData inspired functions are case sensitive


- `endswith(x, y)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: endsWith
  • case sensitive?
  • return Boolean

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all OData inspired functions are case sensitive

Checks whether `y` ends with `x`.

- `matchespattern(x, y)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: matchesPattern
  • return Boolean

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we support both, matchesPattern and matchespattern (case sensitive)

Checks whether `x` matches the regular expression `y`.

- `substring(x, i, n?)` <sup>1</sup>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • returns String

Extracts a substring from `x` starting at index `i` (0-based) with an optional length `n`.
- `i`:
- Positive: starts at index `i`
- Negative: starts `i` positions before the end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not covered by OData.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it reads like it is:

The two-argument substring function with string parameter values returns a substring of the first parameter string value, starting at the Nth character and finishing at the last character (where N is the second parameter integer value)
A negative start index N, if supported, returns a string/collection starting N characters/items before the end of the string/collection.

we could align the parameter names in our docu with those found in the OData spec

- `n`:
- Positive: extracts `n` characters
- Omitted: extracts until the end of the string
- Negative: invalid

- `indexof(x, y)` <sup>1</sup>
Returns the index of the first occurrence of `y` in `x`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • returns Int64 ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We currently do not force dbs to return an Int64. I'd assume that if e.g. HANAs locate function returns a big int, so will we. Do you think it would make sense to always return a big int?


- `length(x)`
Returns the length of the string `x`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • not available in CAP Java
  • returns Int64 ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently Int32


- `tolower(x)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: toLower
  • returns String

Converts all characters in `x` to lowercase.

- `toupper(x)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: toUpper
  • returns String

Converts all characters in `x` to uppercase.

<sup>1</sup> These functions work zero-based. E.g., `substring('abcdef', 1, 3)` returns 'bcd'

#### Numeric Functions

- `ceiling(x)`
Rounds the numeric parameter up to the nearest integer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

n/a in CAP Java


- `floor(x)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

n/a in CAP Java

Rounds the numeric parameter down to the nearest integer.

- `round(x)`
Rounds the numeric parameter to the nearest integer.
Copy link
Contributor

@agoerler agoerler Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: WIP: https://github.wdf.sap.corp/cds-java/cds4j/pull/3590
  • What's the return type? Type of x? Int32? Int64?
  • SQL also know round with two parameters where the second parameter which indicates the number of digits to round to. There are issues on SQLite with this second parameter. I assume there is no plan to support the method with two parameters.
  • On Postgres we observed that Postgres and Double elements rounds to the nearest even integer

https://www.postgresql.org/docs/16/functions-math.html

Rounds to nearest integer. For numeric, ties are broken by rounding away from zero. For double precision, the tie-breaking behavior is platform dependent, but “round to nearest even” is the most common rule.

The midpoint between two integers is rounded away from zero (e.g., `0.5` → `1` and `-0.5` → `-1`).

#### Date and Time Functions

- `year(x)`, `month(x)`, `day(x)`, `hour(x)`, `minute(x)`, `second(x)`
Extracts and returns specific date / time parts as integer value from a given `cds.DateTime`, `cds.Date`, or `cds.Time`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • not available in CAP Java
  • in CAP Java, we treat DateTime as a Timestamp with seconds precision. Hence we can't extract date/time components unless we would make the assumption that the extraction is in UTC
  • What's the return type?
  • Zero-based or one-based?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the return type?

an Int32

Zero-based or one-based?

could you elaborate? It always returns the requested part of the date as it is written in the given cds.DateTime, cds.Date, or cds.Time


- `time(x)`, `date(x)`
Extracts and returns a time or date from a given `cds.DateTime`, `cds.Date`, or `cds.Time`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • not available in CAP Java
  • in CAP Java, we treat DateTime as a Timestamp with seconds precision. Hence we can't extract date/time components unless we would make the assumption that the extraction is in UTC
  • I assume the return types are Time and Date, resp..


- `fractionalseconds(x)`
Returns a `Decimal` representing the fractional seconds for a given `cds.Timestamp`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

n/a in CAP Java


- `maxdatetime()`
Returns the latest possible point in time: `'9999-12-31T23:59:59.999Z'`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • n/a in CAP Java
  • DateTime has no fractional seconds!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in OData, this functions returns an DateTimeOffset which is defined to contain the fractional part: https://docs.oasis-open.org/odata/odata-csdl-xml/v4.01/csprd02/odata-csdl-xml-v4.01-csprd02.html#sec_DateTimeOffset


- `mindatetime()`
Returns the earliest possible point in time: `'0001-01-01T00:00:00.000Z'`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • n/a in CAP Java
  • I would assume this returns no fractional seconds as DateTime has no fractional seconds. See.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in OData, this functions returns an DateTimeOffset which is defined to contain the fractional part: https://docs.oasis-open.org/odata/odata-csdl-xml/v4.01/csprd02/odata-csdl-xml-v4.01-csprd02.html#sec_DateTimeOffset


#### Aggregate Functions

- `min(x)`, `max(x)`, `sum(x)`, `average(x)`, `count(x)`, `countdistinct(x)`
Standard aggregate functions used to calculate minimum, maximum, sum, average, count, and distinct count of values.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Standard aggregate functions used to calculate minimum, maximum, sum, average, count, and distinct count of values.
Standard aggregate functions used to calculate minimum, maximum, sum, average, count, and count of distinct values.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

returns

  • min(x), max(x): type of x
  • average(x): Double?
  • sum(x) : type of x? Or next larger numeric type?
  • count(x), coundistinct(x): Int64

It would be useful to have a type cds.Number!



### SAP HANA Functions

In addition to the OData standard functions, the cds-compiler and all CAP Node.js database services come with
out of the box support for some common SAP HANA functions, to further increase the scope for portable testing:

::: warning Upper- and Lowercase are supported
For the SAP HANA functions, both usages are allowed: all-lowercase as given above, as well as all-uppercase.
:::

- `years_between`
Computes the number of years between two specified dates.
- `months_between`
Computes the number of months between two specified dates.
- `days_between`
Computes the number of days between two specified dates.
- `seconds_between`
Computes the number of seconds between two specified dates.
- `nano100_between`
Computes the time difference between two dates to the precision of 0.1 microseconds.
Comment on lines +986 to +995
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • n/a in CAP Java
  • please link to the corresponding HANA functions.
  • these functions are very prone to one-off errors -> require a very precise specification


The cds-compiler / the database service implementation translates these to the best-possible native SQL functions, thus enhancing the extent of **portable** queries. With open source and the new database service architecture, we also have methods in place to enhance this list by custom implementation.

### Special Runtime Functions

In addition to the OData and SAP HANA standard functions, the **CAP runtimes** provides special functions that are only available for runtime queries:

- `search(xs, y)`
Checks whether `y` is contained in any element of `xs` (fuzzy matching may apply).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • what is an xs?
  • please provide an example how this can be used in a view definition

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It cant be used in a view defintion as indicated by the section heading and description. I will change xs to x :)

See [Searching Data](../guides/providing-services#searching-data) for more details.

- `session_context(v)`
Utilizes standard variable names to maintain session context.
Refer to [Session Variables](#session-variables) for additional information.

- `now()`
Comment on lines +1007 to +1011
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In CAP Java you can use the refs $now, $user, $valid.from, $valid.to.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for node, but this is besides the point of the standard functions. I will add a link to the other section though!

Returns the current datetime.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In CAP Java we have the ref $now but it returns the current Timestamp and not the current DateTime.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In node we return now: () => session_context('$now')` and this in turn returns the current timestamp, thanks for pointing this out. I will adapt the docs :)


## Using Native Features { #native-db-functions}

Expand Down
Loading