Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Building Codes and Rules API based on Named Graphs #2

Closed
VladimirAlexiev opened this issue Mar 21, 2024 · 15 comments
Closed

Building Codes and Rules API based on Named Graphs #2

VladimirAlexiev opened this issue Mar 21, 2024 · 15 comments
Assignees
@VladimirAlexiev

Comments

@VladimirAlexiev
Copy link

VladimirAlexiev commented Mar 21, 2024 •
edited
Loading

Was Building Codes and Rules: based on GraphQL, or SPARQL and grlc

@beachtom @rickmakkinga @pasi-paasiala

Currently the API talks about access to a whole Regulation.
But I think a key question for Building Codes and Rules (BCRL) is how to structure the content and how to fetch what you need.
These are captured as RDF. I see two possible ways to query them and get them:

  • GraphQL. We can serve JSON of pretty much the same form as the YAML
    • pro: can get exactly the data you need, easy to query, easy to do pagination and basic filters
    • cons: need to make a SOML schema, but that's not hard based on the ontology
  • SPARQL and grlc. grcl is open software that exposes a set of SPARQL queries (parameterized using Basil convention) as API. Some description in Query Parameterization w3c/sparql-dev#57, and I got a book about it.
    • pro: can write as complex queries as needed
    • cons: harder to return deeply nested data, and in some cases may even need a sequence of queries
@beachtom
Copy link
Contributor

So we could also expose an endpoint to allow a graph QL query - but I think we also need to retain simpler approaches

The API as I defined it so far allowes retrieval of entire documents, and sections/subsection based on REST only.

@VladimirAlexiev
Copy link
Author

VladimirAlexiev commented Mar 26, 2024 •
edited
Loading

After the discussion today, @beachtom and I decided to provide a GraphQL API.

+ aec3po.ttl
+ check_method.ttl
- checking_act.ttl
- compliance_verification_report.ttl
- data_requirement.ttl
+ document.ttl
- evidence.ttl
- feature_of_interest.ttl
- legal_verifier.ttl
- model.ttl
+ rase_statement.ttl
+ statement.ttl
+ table.ttl
  • Which is the relation clause->subclause? Assuming it's hasPart, will add transitive prop hasPartTransitive to allow fetching the complete subtree.

@VladimirAlexiev VladimirAlexiev self-assigned this Mar 26, 2024
@beachtom
Copy link
Contributor

Hi Vladamir,

The used modules are:

aec3po.ttl
check_method.ttl
document.ttl
rase_statement.ttl
statement.ttl
table.ttl

The two best examples are:

https://github.com/Accord-Project/bcrl/blob/main/FI-accessibility-experiment/Tom%20Example/FI2.yaml

and

https://github.com/Accord-Project/bcrl/blob/main/EE-Example/EE1-Example.yaml

But I can generate a YAML for any in this folder (but they have no RASE as of yet). let me know if you need more

https://github.com/Accord-Project/bcrl/tree/main/RegulatoryTexts

@VladimirAlexiev VladimirAlexiev changed the title Building Codes and Rules: based on GraphQL, or SPARQL and grlc Building Codes and Rules API based on GraphQL Apr 2, 2024
@VladimirAlexiev
Copy link
Author

VladimirAlexiev commented Apr 2, 2024 •
edited by nataschake
Loading

@nataschake https://ontotext.atlassian.net/issues/ACD-135

Let's use https://github.com/VladimirAlexiev/soml/tree/master/owl2soml (a perl script; there's a similar java module on Platform Workbench, but it might be slightly older)

We need to:

  • add a Makefile for all this below
  • concat the above 6 of the AEC3PO ontologies, and all vocabularies, and generate SOML (in the makefile)
  • add prop hasPartTransitive to allow fetching the complete subtree (GraphQL has no prop paths like (hasPart|hasInlinePart)*): see aec3po-fix.ttl
  • make a repo aec3po with subPropertyOf and transitiveOver reasoning (see https://graphdb.ontotext.com/documentation/10.5/rules-optimisations.html#transitiveover and next section) and loada all these ontologies
  • Write some queries to check aec3po and post issues: props without domain, object props without range...
  • inspect SOML it and make corrections (eg unattached props, imprecise cardinalities). Best to do it with a "fix.ttl", see owl2soml/eg/skos-fix.ttl
  • add virtual inverses where bidirectional navigation is needed.

@Gonsco

  • You can search on any field (EQ, NE, RE=regex etc)
  • But typically you'd just fetch a regulation and all its clauses by ID (URL)
  • You can dig deep into the regulation and fetch all clauses and their fields and relations.
    We'll write nested GraphQL queries, similar to Tom's YAML examples, to dig through the structure.
  • It will return JSON, conformant to the GraphQL query shape

Goncal, do you have any other specific retrieval needs? (If we want to sound learned, we'll call this Competency Questions). Eg:

  • Getting lists of stored regulations, with what metadata?
  • Stats about number of clauses?
  • What bidirectional navigation do you need? Eg do you ever need from Clause (child) to Regulation (parent)?

As for storing, let's use the YAML-JSONLD-RDF route.

  • https://github.com/Accord-Project/bcrl includes a tool by Thamer to convert YAML-JSON
  • If the JSON refers to the respective JSONLD context (which is properly served online), that's all we need: GraphDB can ingest it directly
  • Else, I think there's another tool to convert JSON-RDF

@VladimirAlexiev VladimirAlexiev changed the title Building Codes and Rules API based on GraphQL Building Codes and Rules API based on Named Graphs Apr 3, 2024
@VladimirAlexiev
Copy link
Author

@beachtom I looked at how SOML and GraphQL queries would look, and it's not so easy because of complications like dual nature of hasTarget (GraphQL doesn't support that).

We had a call with @Gonsco and agreed on the simplest approach, based on the SPARQL Graph Protocol:

  • POST: overwrite a regulation named graph with new regulation content (Goncal only needs this)
  • GET: get the content of a regulation named graph

Then we can add some SPARQL queries with grlc:

  • List all regulations: with name, countries (forAdministrativeArea), mabve number of subsections
  • List regulations for a given country (AdministrativeArea)

@beachtom
Copy link
Contributor

beachtom commented Apr 3, 2024

So we also need to consider versions - i..e one document may have multiple versions - so i think a named graph would represent a given version of a given document?

We also need queries to retrieve a given section/subsection of a document/version

@VladimirAlexiev
Copy link
Author

@Gonsco , @rlavikka , @k-breitenfelder , @rickmakkinga Do we need versions?
Then each regulation needs some metadata as you find in many owl:Ontology:

  • its version string,
  • link to prev version
  • link to latest (unversioned namespace) version

@beachtom
Copy link
Contributor

beachtom commented Apr 3, 2024

Generally speaking release date is used as the version string - we then use replaces and replacedBy to form a link list of versions - so I think this covers it.

As we only have one version of any documents I haven't used this - but can easily add it.

@rickmakkinga
Copy link

rickmakkinga commented Apr 8, 2024 via email

@rickmakkinga
Copy link

rickmakkinga commented Apr 8, 2024 via email

@beachtom
Copy link
Contributor

beachtom commented Apr 8, 2024

So the plan is to put a rest API in front of the graphQL - to allow for premade queries.

For you first point - it is up to this API to deal with issues like this. So if you request clause 2.1 of version 2023 - it does not matter when the clause was first drafted you get that version.

@rickmakkinga
Copy link

rickmakkinga commented Apr 8, 2024 via email

@beachtom
Copy link
Contributor

This is now sorted

@Gonsco Gonsco reopened this Jun 7, 2024
@Gonsco
Copy link

Gonsco commented Jun 7, 2024

I apologize for being late with the discussion...

On the topic of including the version in the graph name, do we have a definitive position?
Does this have a direct relation with the name used as part of the URL when a document is upload to the GraphDB via the REST API?
Currently this is indicated in the https://github.com/Accord-Project/API-Development/blob/main/BuildingCodesAndRules/graphdb-webapi.md for the PUT method we use in the RFT to upload the files:

https://graphdb.accordproject.eu/graphdb/repositories/aec3po/statements?context=%3Chttps%3A%2F%2Fgraphdb.accordproject.eu/resource/aec3po/Spain/v2%3E

The name and version issues must be clear in order to know how to upload the file from the RFT and how to control it. Is it assumed that every time the publish button is pressed, the document of the same project is published in a new version? If so, the tool should perform version control, right? But, if the version is controlled by the date, as Thomas suggest, I understand that this should be part of the name in the URI, right?

I raise all these questions so as not to leave any loose ends :)

@beachtom
Copy link
Contributor

beachtom commented Jun 7, 2024

@Gonsco it is best you track the issues in the BCRL repo. Together, we are getting close to upload /download from GraphDB - there were can give you a definitive answer

@Gonsco Gonsco closed this as completed Jun 7, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@VladimirAlexiev VladimirAlexiev

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@VladimirAlexiev @Gonsco @rickmakkinga @beachtom