Author: A.Formica, R.Sipos

Contributors: M.Mineev, E.Alexandrov (client tools)

Date of last development period: 2019/01/13

Recent additions: new api methods for uploads of iov+payload, a web-ui in vuejs.

   Copyright (C) 2016  A.Formica, R.Sipos

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

Table of Contents

1. Description
2. Installation
3. Build instructions
4. Run the server
5. Swagger
6. Docker
7. Openshift
8. Clients
9. Usage
10. Data Model

Description

Project for a generic purpose conditions database for physics experiment. The system is based on a RESTful API and a relational database. The API is described in openapi.

This server was generated by the openapi-generator project.

The prototype uses Spring framework and the REST services are implemented via Jersey.

The prototype runs as a microservice using spring-boot. By default, it uses an embedded undertow servlet container, but others like tomcat or jetty can be easily used.

This version 6.0 is compatible with Java 23 and Spring 3.3.4.

Installation

Download the project from gitlab (example below is using https):

git clone https://gitlab.cern.ch/crest-db/crest.git

This will create a directory crest in the location where you run the git command.

Build instructions

You need to have java >= 21 installed on your machine. If you have also gradle (version 7) you can build the project using the following command from the root project directory (crest):

gradle clean build

This command will generate a jar (java archive) file in : ./build/libs/crest.jar. In case gradle is not installed on your machine, you can run the wrapper delivered with the project:

./gradlew clean build

If you want to select a specific JVM when you run gradle you can use a command like this:

gradle clean build -Dorg.gradle.java.home=/path_to_jvm/21.0/

Be careful to checkout the correct branch.

| DESCRIPTOR | BRANCH | Java Version | First attempt to clean up API from older versions | v3.0 | java 11 | API used in 2024 HLT tests | v4.0 | java 11 | API with triggerDB changes | v5.0 | java 11 | API identical to v5.0 but new Java | v6.0 | java 23

CAVEAT: Java>=17

The last version of CREST server is compatible with Spring 3.3.4 and Java 23. The mapping from POJO to DTOs is now implemented with MapStrut project. Be careful that if you do not have java 23 available you can still compile but you need the following change in the build.gradle file:

java {
	toolchain {
		languageVersion = JavaLanguageVersion.of(23)
	}
}

should be replaced by :

java {
	toolchain {
		languageVersion = JavaLanguageVersion.of(21)
	}
}

Run the server

The server will use by default an embedded undertow web server.

The server need by definition to have a database connection in order to store the conditions data. The database connections are defined in the files ./src/main/resources/application-<profile>.yml. This file present different set of properties which are chosen by selecting a specific spring profile when running the server.

If you do not have any remote database available you should use the default spring profile.

The set of default properties to run the server is defined in config/application.properties which will be read by spring when starting the server. The file there will use the default spring profile and a local database instance h2database where to store the data (it is a sort of sqlite file). It is essential to set appropriate parameters in this file. An example is provided to start up a default profile with local h2 database.

Oracle

spring.profiles.active=oracle
user.timezone=GMT
crest.api.name=/crestapi
crest.web.static=/tmp/data/web
crest.dump.dir=/tmp/data/dump
crest.log.dir=/tmp/data/dump
crest.db.password=somepassword
crest.db.user=ATLAS_PHYS_COND_01_W
crest.port=8090
crest.db.url=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=cman1-atlas.cern.ch)(PORT=10500))(LOAD_BALANCE=on)(ENABLE=BROKEN)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=int8r.cern.ch)))
crest.db.schema=ATLAS_PHYS_COND_01

Postgres

spring.profiles.active=postgres
user.timezone=GMT
crest.web.static=/tmp/data/web
crest.dump.dir=/tmp/data/dump
crest.log.dir=/tmp/data/logs
management.endpoint.health.show-details=ALWAYS
crest.db.url=jdbc:postgresql://postgres-host:5432/crestdb
crest.db.user=someuser
crest.db.password=somepassword

Start

To start the server you can simply run:

./entrypoint.sh

This script is the same that is used by the docker container (when packaging the server via the Dockerfile).

Activate security

Review this part. Will fill in later.

Openapi

These instructions are ment to be used by developers of the vhf manager application. The file which contains the full API definition is:

./openapi/bundle/crestApi_all.yml

This file can also be used by applications like the swagger-ui for visualization of the API.

Openapi code generation

In order to regenerate the API we use the schemas and templates which are stored in the directories:

./openapi
     - templates
     - schemas

The server stub generation is implemented as a gradle task:

./gradlew openApiBundle openApiGenerate

Docker

You can build a container using:

docker build -t crest:1.0 .

You can run the container using

docker run --env-file .environment -p 8080:8080 -d crest:1.0

or

docker run --env-file .environment -p 8080:8080 -v /mnt/data/dump:/data/dump -v /mnt/data/web:/data/web --net=host -d crest:test

In the last example we have been mounting external volumes, used for logs and uploads or retrieval of payload files.

You can connect to a running container using commands like:

docker exec -i -t infallible_stonebraker /bin/bash

In order to push an image created into the CERN gitlab registry you need to login:

docker login -u formica gitlab-registry.cern.ch

You can choose another registry. The push command work only if the login is successful.

Local testing

To run the system locally on your machine (for testing purpose) you can use the Dockerfile and docker-compose.yml which are delivered within the project:

docker build -t some:tag .
docker-compose ./docker-compose.yml up -d

Swarm

As an example for a deployment in a swarm look at ./swarm/docker-compose.yml. In the same repository there is a script to help in generating the config maps needed and the secret. An application.properties file should be created in order to run the server. Examples are shown above.

Openshift

CREST service has a gitlab-ci pipeline in place to create and push a docker image to CERN registry. Further instructions on deployment are available in the project crest-deployment .

Constraints

For the moment in order for the deployment to work we need to have a public access to the gitlab project.

Python

Client is generated via OpenApi. (complete here)

C++

Client API is available here. Migration tools from COOL to CREST are available here.

Command line

Several command line utilities are available in the this repository. These tools can be used to interact with the CREST server.