subsidy-applications-management-service

Service that provides management of one or more SemanticForm(s).

This includes but is not limited to:

providing and managing all the (meta)data needed to construct/visualize a SemanticForm
updating and deleting the source-data of a SemanticForm
- Note: NOT creating
submitting a SemanticForm
experimental: writing tailored meta/source-data extractors

Setup

Within a `mu.semte.ch` stack

Paste the following snip-it in your docker-compose.yml:

version: '3.4'

services:
  subsidy-applications-management:
    image: lblod/subsidy-applications-management-service
    volumes:
        # change to location of configuration
      - ./my-config:/config
        # change to location of file storage
      - ./my-file-storage:/data

Environment variables

These can be added in within the docker declaration.

Name	Description	Default
`SERVICE_NAME`	The name of the service. Mainly used to identify the service's created files.	"subsidy-application-management-service"
`SEMANTIC_FORM_TYPE`	The type that should be used for managed semantic-forms.	`lblodSubsidie:ApplicationForm`
`SEMANTIC_FORM_RESOURCE_BASE`	Resource URI base to be used when creating new semantic-forms.	"http://data.lblod.info/application-forms/"
`SEMANTIC_FORM_CONFIGURATION_ROOT`	Root location of semantic-form configurations within the `config` volume.	`/config/forms/`
`QUERY_CHUNK_SIZE`	Maximum triple chunk for mutation data queries.	50
`META_DATA_ROOT`	Root location of meta-data within the `data` volume	`data/meta-files/`
`META_DATA_CRON`	Interval when to retry generating meta-data	`0 /15 8-18 * 1-5` “At every 15th minute past every hour from 8 through 18 on every day-of-week from Monday through Friday.”
`META_DATA_STALE_CRON_STOP`	After how many days the `META_DATA_CRON` is considered "stale"	5
`META_DATA_EXTRACTION_BLACK_LIST`	concept-scheme values that should be ignored

Developer note:
For the more tech-savvy, you can also take a peek in env.js for a more detailed look.

Configuration

SemanticForm(s)

To start managing forms, you'll first need to configure forms. We'll do this within the SEMANTIC_FORM_CONFIGURATION_ROOT.

You'll create a new SemanticForm by:

Creating a SemanticFormDirectory
Populate the SemanticFormDirectory, /versions directory, with a SemanticFormSpecification

When finished, you'll end up with a structure that could look similar to this:

📂 SEMANTIC_FORM_CONFIGURATION_ROOT
- 📂 my-form
  - 📂 versions
    - 📂 20201231153419-initial-from
      - 📝 config.json
      - 📝 form.ttl
    - 📁 20201231153459-added-new-sub-form

SemanticForm(s): The SemanticFormDirectory

The service can manage one or more SemanticForm(s). This is achieved by creating SemanticFormDirectories. These directories contain everything the service will need to build, enhance and manage the SemanticFormBundle for the client.

How to create a SemanticFormDirectory

Defining a SemanticFormDirectory has been kept rather simple. You achieve this by creating a directory within the SEMANTIC_FORM_CONFIGURATION_ROOT that contains a sub-directory called /versions:

# we have only one form
$ mkdir versions

# we want to name our form(s)
$ mkdir -p /my-form/versions

# we want to nest our form(s)
$ mkdir -p phase-1/my-form/versions

Simply put, any recursively nested folder within the SEMANTIC_FORM_CONFIGURATION_ROOT containing a /versions folder, is picked-up as a SemanticFormDirectory.

Developer note:
In theory, you could create any desired directory structure within the SEMANTIC_FORM_CONFIGURATION_ROOT.
But, this has never been extensively tested. So explore this at your own discretion.

What is the `/versions` sub-directory?

Directory where your different versions of SemanticFormSpecification will live.

Note: This sub-directory is required for the parent directory to be recognized as a SemanticFormDirectory and cannot hold another name.

SemanticForm(s): The SemanticFormSpecification

Is a timestamped directory, within the /versions directory containing a pair of two files:

The turtle form-specification
The json meta-data for the turtle form-specification

All SemanticFormSpecification directories are required to have a timestamp in the format YYYYMMDDhhmmss followed by a dash(-) and a title/description. This timestamp will be inferred as the creation date of its configuration files.

What about migrations, should the files be added manually to the store?

No, migrations are not required. The files are inserted into the store by the service itself. We follow the same model as defined by the file-service.

UUID generation

To ensure consistent UUID generation on different environments, we make use of UUIDv5 with a fixed namespace.

Developer note:
For the more tech-savvy, the following namespace & name are used when generating the UUID's:

physical UUID: uuidv5("77219b1e-e23d-4828-847f-55e2d7fac687", [physical-uri])

virtual UUID: uuidv5("77219b1e-e23d-4828-847f-55e2d7fac687", [physical-uuid])

The turtle form-specification

@prefix form: <http://lblod.data.gift/vocabularies/forms/> .
@prefix sh: <http://www.w3.org/ns/shacl#>.
@prefix mu: <http://mu.semte.ch/vocabularies/core/> .
@prefix fieldGroups: <http://data.lblod.info/field-groups/> .
@prefix fields: <http://data.lblod.info/fields/> .
@prefix displayTypes: <http://lblod.data.gift/display-types/> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#>.

##########################################################
#  property-group
##########################################################
fields:8e24d707-0e29-45b5-9bbf-a39e4fdb2c11 a form:PropertyGroup;
    mu:uuid "8e24d707-0e29-45b5-9bbf-a39e4fdb2c11";
    sh:description "parent property-group, used to group fields and property-groups together";
    sh:name "This is a simple example form configuration ttl, make sure you correctly mapped your own form configuration" ;
    sh:order 1 .

##########################################################
# basic field
##########################################################
fields:147a32fe-f3dd-47f0-9dc5-43e46acc32cb a form:Field ;
    mu:uuid "147a32fe-f3dd-47f0-9dc5-43e46acc32cb";
    sh:name "Basic input field" ;
    sh:order 10 ;
    sh:path skos:prefLabel ;
    form:displayType displayTypes:defaultInput ;
    sh:group fields:8e24d707-0e29-45b5-9bbf-a39e4fdb2c11 .

##########################################################
# form
##########################################################
fieldGroups:main a form:FieldGroup ;
    mu:uuid "70eebdf0-14dc-47f7-85df-e1cfd41c3855" ;
    form:hasField fields:147a32fe-f3dd-47f0-9dc5-43e46acc32cb . ### Basic input-field

form:6b70a6f0-cce2-4afe-81f5-5911f45b0b27 a form:Form ;
    mu:uuid "6b70a6f0-cce2-4afe-81f5-5911f45b0b27" ;
    form:hasFieldGroup fieldGroups:main .

The json meta-data for the turtle form-specification

{
  "meta": {
    "schemes": [
    ]
  },
  "resource": {
    "prefixes": [
      "PREFIX mu: <http://mu.semte.ch/vocabularies/core/>",
      "PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>",
      "PREFIX dct: <http://purl.org/dc/terms/>",
      "PREFIX skos: <http://www.w3.org/2004/02/skos/core#>",
      "PREFIX schema: <http://schema.org/>",
      "PREFIX foaf: <http://xmlns.com/foaf/0.1/>"
    ],
    "properties": [
      "rdf:type",
      "mu:uuid",
      "dct:source",
      "skos:prefLabel",
      {
        "s-prefix": "schema:contactPoint",
        "properties": [
          "foaf:firstName",
          "foaf:familyName",
          "schema:telephone",
          "schema:email"
        ]
      }
    ]
  }
}

Developer note:

meta.schemes is optional.

properties can be nested indefinitely.

⚠️ ⚠️ ⚠️ NO EXTRA PROPERTIES BESIDES WHAT IS RELEVANT FOR THE form.ttl -> else we might have very weird side effects on the tailored data.

[experimental] SemanticForm(s): Tailored data

Sometimes SemanticForms need to be enhanced with user specific business logic, for this the concept of tailored data was introduced. Take notice that this is still very experimental. Use at your own discretion.

How to create tailored meta/source-data extractors

Tailored data is SemanticForm specific, so you'll have to navigate to the matching SemanticFormDirectory. Next to the /versions directory, you'll create the following directories:

# meta data extractors
$ mkdir -p tailored/meta

# source data extractors
$ mkdir -p tailored/source

Within these directories you'll define an index.js that exports an array of extractors. These will be called on enhancement of a new SemanticForm.

NOTE: these extractors run in the order of definition

For a starter example, take a look in root of this repository under ./examples.

[deprecated] Mapper

At inception, to be submitted SemanticForms would be mapped to a pre-defined (clean) model.

This mapper provided the functionality to create new resources, map/copy fields and create relations to map the SemanticForm fields to the correct pre-defined model.

This functionality was abandoned after the responsibility was moved to other services in an effort to simplify the process.
See commit.

Generates

SemanticForm(s): MetaData

Extra data that is needed by the client to correctly render the form. Initially, at startup, the service does not generate meta-data. It only starts once a client has requested a SemanticForm that needs meta-data.

Where and how is it stored?

All the meta-data is stored in turtle files. These files can be found in the META_DATA_ROOT under the same folder structure defined by its paired SemanticFormDirectory.

Is the meta-data up-to-date?

To ensure that the meta-data is up-to-date, when a user requests a new SemanticForm to be build, a process is started to generate the required meta-data.

This process can be very expensive and time-consuming. Therefore, a background task is triggered to check at predefined intervals if the meta-data has been updated. If the meta-data is out-of-date, a new updated meta-data file is generated.

To prevent dangling background processes, if this background task was not triggered by a client within the specified META_DATA_STALE_CRON_STOP days, it is turned off.

Wouldn't it be too eager during a migration that targets the meta-data sources?

To prevent this from happening, if a change to the meta-data is detected, a backoff was implemented.

This means that if a change was detected, it will wait and retry. If it changed again, we assume to be in an unstable state. So we, backoff and retry again later. This process will continue until:

the meta-data source have stabilized
maximum backoff wait has been reached, meaning we'll retry at a later rotation

Backoff environment variables

These can be added in within the docker declaration.

Name	Description	Default
`META_DATA_EXTRACTION_BACKOFF_INITIAL_WAIT`	The initial wait time	5000
`META_DATA_EXTRACTION_BACKOFF_RATE`	backoff rate	0.3
`META_DATA_EXTRACTION_BACKOFF_MAX_WAIT`	maximum wait time	600000 "10 min"

API

Get the configuration and latest meta-data to be used.

GET /sources/latest?uri

HTTP/1.1 
200 OK
X-Powered-By: Express
content-type: application/json; charset=utf-8

{
  "configuration": {
    "specification": ""
    "tailored": {
      "meta": ""
      "source": ""
    }
  },
  "meta": "",
}

Get all the meta(data) needed to construct a semantic form.

GET /semantic-forms/:uuid

Response

HTTP/1.1 
200 OK
X-Powered-By: Express
content-type: application/json; charset=utf-8

{
  "form": "",
  "meta": "",
  "source": ""
}

Update the source-data based on a given delta

PUT /semantic-forms/:uuid

Request

HTTP/1.1
Connection: keep-alive
Content-Length: xxx
content-type: application/json
Accept: application/json

{
  "additions": "",
  "removals": ""
}

Delete all the source-data for a semantic-form

DELETE /semantic-forms/:uuid

Response

HTTP/1.1 
204 No Content
x-powered-by: Express

Submit a semantic-form

POST /semantic-forms/:uuid/submit

Response

HTTP/1.1 
204 No Content
x-powered-by: Express

Sync all meta-data

GET /meta/sync

NOTE: heavy request as this triggers an update for all configured SemanticFormSpecifications

Response

HTTP/1.1 
204 No Content
x-powered-by: Express

Development

For a more detailed look in how to develop a microservices based on the mu-javascript-template, we would recommend reading "Developing with the template".

Developing in the `mu.semte.ch` stack

Paste the following snip-it in your docker-compose.override.yml:

subsidy-applications-management:
  image: semtech/mu-javascript-template:1.4.0
  ports:
    - 8888:80
    - 9229:9229
  environment:
    NODE_ENV: "development"
  volumes:
    - /absolute/path/to/your/sources/:/app/
    - ./config/semanctic-form-path:/share

Name		Name	Last commit message	Last commit date
Latest commit History 269 Commits
example/tailored/meta		example/tailored/meta
lib		lib
.drone.yml		.drone.yml
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE.md		LICENSE.md
README.md		README.md
app.js		app.js
env.js		env.js
package.json		package.json

License

lblod/subsidy-applications-management-service

Folders and files

Latest commit

History

Repository files navigation

subsidy-applications-management-service

Table of contents

Setup

Within a mu.semte.ch stack

Environment variables

Configuration

SemanticForm(s)

SemanticForm(s): The SemanticFormDirectory

How to create a SemanticFormDirectory

What is the /versions sub-directory?

SemanticForm(s): The SemanticFormSpecification

What about migrations, should the files be added manually to the store?

UUID generation

The turtle form-specification

The json meta-data for the turtle form-specification

[experimental] SemanticForm(s): Tailored data

How to create tailored meta/source-data extractors

[deprecated] Mapper

Generates

SemanticForm(s): MetaData

Where and how is it stored?

Is the meta-data up-to-date?

Wouldn't it be too eager during a migration that targets the meta-data sources?

API

Get the configuration and latest meta-data to be used.

Get all the meta(data) needed to construct a semantic form.

Response

Update the source-data based on a given delta

Request

Delete all the source-data for a semantic-form

Response

Submit a semantic-form

Response

Sync all meta-data

Response

Development

Developing in the mu.semte.ch stack

About

Topics

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Contributors 5

Languages

Within a `mu.semte.ch` stack

What is the `/versions` sub-directory?

Developing in the `mu.semte.ch` stack

Packages