LDL24_ItSrNER_export_to_ttl.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eXyvhBWb8hgK"
      },
      "source": [
        "\n",
        "# Export IT-SR-ner to NIF\n",
        "\n",
        "LDL conference 2024: **Ranka Stanković**, **Milica Ikonić Nešić**\n",
        "\n",
        "open data: use case for 2 languages\n",
        "\n",
        "Last update 04.03.2024.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8AmPafxHQEBp"
      },
      "source": [
        "**Description**: Source files are It-Sr-NER corpus https://github.com/jerteh/It-Sr-NER/tree/main/corpus/monolingual/ (POS taged, lemmatised, with annotated named entities, also supplied by Wikidata( For this porpues it's used 1000 sentences of parallel text.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y4SsbRmnXV7v"
      },
      "source": [
        "## install & import part"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UXkJizYCAhUg"
      },
      "outputs": [],
      "source": [
        "#!python.exe -m pip install --upgrade pip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v-RRzBvNuT7U"
      },
      "outputs": [],
      "source": [
        "# for importing/clonning repository with novels\n",
        "#!pip install gitpython\n",
        "#!pip install rdflib\n",
        "#!pip install pydotplus\n",
        "#!pip install mkwikidata"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CWn0YPe3CXSz"
      },
      "outputs": [],
      "source": [
        "#!pip3 install rdflib sparqlwrapper pydotplus graphviz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "z3mQ8M5CXRQK"
      },
      "outputs": [],
      "source": [
        "#!pip install mkwikidata"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iWMUTUkxBldw"
      },
      "outputs": [],
      "source": [
        "# os,sys,glob\n",
        "import os\n",
        "import os.path\n",
        "from os import path\n",
        "import sys\n",
        "import glob\n",
        "import locale\n",
        "import spacy\n",
        "import string\n",
        "\n",
        "# rdf\n",
        "import rdflib\n",
        "from rdflib import Graph\n",
        "from rdflib.namespace import RDF, RDFS, XSD, OWL, DCAT\n",
        "from rdflib import URIRef, BNode, Literal\n",
        "import networkx as nx\n",
        "import io\n",
        "import pydotplus\n",
        "from IPython.display import display, Image\n",
        "from rdflib.tools.rdf2dot import rdf2dot\n",
        "import mkwikidata\n",
        "import re\n",
        "\n",
        "ITSRDF=rdflib.Namespace(\"http://www.w3.org/2005/11/its/rdf#\")\n",
        "NIF = rdflib.Namespace(\"http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core#\")\n",
        "# NERD= rdflib.Namespace(\"http://nerd.eurecom.fr/ontology#\")\n",
        "DC = rdflib.Namespace(\"http://purl.org/dc/elements/1.1/\")\n",
        "DCT = rdflib.Namespace(\"http://purl.org/dc/terms/\")\n",
        "FOAF = rdflib.Namespace(\"http://xmlns.com/foaf/0.1/\")\n",
        "MS = rdflib.Namespace(\"http://w3id.org/meta-share/meta-share/\")\n",
        "WD = rdflib.Namespace(\"http://www.wikidata.org/entity/\")\n",
        "#WD = rdflib.Namespace(\"http://www.wikidata.org/wiki/\")\n",
        "WDT = rdflib.Namespace(\"http://www.wikidata.org/prop/direct/\")\n",
        "DBO = rdflib.Namespace(\"https://dbpedia.org/ontology/\")\n",
        "OLIA = rdflib.Namespace(\"http://purl.org/olia/discourse/olia_discourse.owl#\")\n",
        "SKOS = rdflib.Namespace(\"http://www.w3.org/2004/02/skos/core#\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KZM9q_kSXpRJ"
      },
      "source": [
        "# Metadata description"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ebp2cIKL_OFk"
      },
      "source": [
        "For metadata analysed: https://link.springer.com/chapter/10.1007/978-3-319-18818-8_20, lexmeta : https://lexbib.elex.is/wiki/LexMeta,\n",
        "https://github.com/pennyl67/LexMeta/blob/main/lexmeta.ttl\n",
        "\n",
        "Metadata for collections described in Wikidata: https://www.wikidata.org/wiki/Wikidata:WikiProject_ELTeC (first edition, printed edition, ELTeC edition)\n",
        "\n",
        "Key values prepared for this edition:\n",
        "* ID - dct:identifier\n",
        "* title - dct:title (from teiHeader : titleStmt)\n",
        "* author (name) - ms:author\n",
        "* authorQID - dc:creator,\n",
        "* novelQID - linked as edition of novel\n",
        "* publisher - dct:publisher (foaf:name)\n",
        "* licence - ms:LicenceTerms\n",
        "* year - ms:publicationDate\n",
        "* language - dc:Language, ms:Language\n",
        "* collection - wdt:P1433\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtxNpp6u21JR"
      },
      "source": [
        "# Ontology mapping"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jhrIbulcwxzs"
      },
      "source": [
        "Initially used:\n",
        "https://github.com/NERD-project/nerd-ontology/blob/master/nerd.owl  but replaced with:\n",
        "http://purl.org/olia/discourse/olia_discourse.owl\n",
        "\n",
        "* http://nerd.eurecom.fr/ontology#Person  (mapped to http://purl.org/olia/discourse/olia_discourse.owl#Person)\n",
        "* http://nerd.eurecom.fr/ontology#Location (mapped to http://purl.org/olia/discourse/olia_discourse.owl#Space)\n",
        "* http://nerd.eurecom.fr/ontology#Organization  (mapped to http://purl.org/olia/discourse/olia_discourse.owl#Organization)\n",
        "* http://nerd.eurecom.fr/ontology#Event  (mapped to http://purl.org/olia/discourse/olia_discourse.owl#Event)\n",
        "* ROLE - Names of posts and job titles (profession, nobility, office, military)\n",
        "* DEMO - DEMO -  Demonyms, names of kinds of people: national, regional, political (Frenchwoman; German; Parisiens;...)\n",
        "* WORK - titles of books, songs, plays, newspaper, paintings, sculptures  and other creations\n",
        "\n",
        "Additional:\n",
        "* https://dbpedia.org/ontology/Person, https://www.wikidata.org/wiki/Q5, https://schema.org/Person\n",
        "* https://dbpedia.org/ontology/Place, https://www.wikidata.org/wiki/Q7884789, https://schema.org/Place\n",
        "* https://dbpedia.org/ontology/Organisation, https://www.wikidata.org/wiki/Q43229, https://schema.org/Organization\n",
        "* https://dbpedia.org/ontology/Event, https://www.wikidata.org/wiki/Q1656682, https://schema.org/Event\n",
        "* https://dbpedia.org/ontology/Profession, https://www.wikidata.org/wiki/Q28640, https://schema.org/Occupation\n",
        "* demonym as property only: https://dbpedia.org/ontology/demonym, https://www.wikidata.org/wiki/Q217438\n",
        "* https://dbpedia.org/ontology/Work, https://www.wikidata.org/wiki/Q386724, https://schema.org/CreativeWork,\n",
        "\n",
        "\n",
        "Rizzo, Giuseppe, Raphaël Troncy, Sebastian Hellmann, and Martin Bruemmer. \"NERD meets NIF: Lifting NLP extraction results to the linked data cloud.\" In LDOW. 2012. http://ceur-ws.org/Vol-937/ldow2012-paper-02.pdf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s2r8ZweIUeXG"
      },
      "source": [
        "# General classes: Token, Sentence, NamedEntity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GleaZpu6ChCO"
      },
      "source": [
        "TEI+RDF+LLOD literature\n",
        "\n",
        "https://content.iospress.com/articles/semantic-web/sw222859\n",
        "\n",
        "* P. Ruiz Fabo, H. Bermúdez Sabel, C. Martínez Cantón and E. González-Blanco, The diachronic Spanish sonnet corpus: TEI and linked open data encoding, data distribution, and metrical findings, Digital Scholarship in the Humanities (2020). doi:10.1093/llc/fqaa035.\n",
        "\n",
        "\n",
        "* S. Tittel, H. Bermúdez-Sabel and C. Chiarcos, Using RDFa to link text and dictionary data for medieval French, in: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), European Language Resources Association (ELRA), 2018.\n",
        "\n",
        "* Khan, Anas Fahad, Christian Chiarcos, Thierry Declerck, Daniela Gifu, Elena González-Blanco García, Jorge Gracia, Maxim Ionov et al. \"When Linguistics Meets Web Technologies. Recent advances in Modelling Linguistic Linked Data.\" (2021).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qG_iyAmWDsH8"
      },
      "outputs": [],
      "source": [
        "# manage reading token properties from token (xml node) and generate graf triples\n",
        "class Token:\n",
        "\n",
        "    def __init__(self, token, lng): #token_row\n",
        "        # 1-1\t0-5\tVeoma\t_\t_\tADV\tveoma ##example of token\n",
        "        self.lng = lng #OVO SAM SAD DODALA 17.10\n",
        "        tokens = token.split('\\t')\n",
        "        self.text =  tokens[2]\n",
        "        # print(self.text)\n",
        "        self.pos = tokens[5]\n",
        "        self.lemma = \"\" if self.pos == \"PUNCT\" else tokens[6][:-1]\n",
        "        self.id = tokens[0].split('-')[0]\n",
        "\n",
        "        self.index_start=int(tokens[1].split('-')[0])\n",
        "        self.index_end=int(tokens[1].split('-')[1]) # na celom tekstu pocetna i krajnja pozicija\n",
        "\n",
        "        #2-17\t241-247\tČerulo\thttp://www.wikidata.org/entity/Q122388880[3]\tPERS[3]\tPROPN\tČerulo\n",
        "        #for NER (pretpostavljamo da gde ima NER ima wiki)\n",
        "        self.NER = False\n",
        "        if  tokens[4] != \"_\":\n",
        "            self.NERtype = tokens[4] #PERS[3]\n",
        "            self.NERtext = tokens[2]\n",
        "\n",
        "            wiki = tokens[3]\n",
        "\n",
        "            if '_' not in wiki:\n",
        "                if '[' in wiki:\n",
        "                    self.Wiki = wiki.split('/')[4].split('[')[0]\n",
        "                else:\n",
        "                    self.Wiki = wiki.split('/')[4]\n",
        "            else:\n",
        "                self.Wiki = \"\"\n",
        "            self.NER = True\n",
        "\n",
        "\n",
        "    # create graph triples for this token\n",
        "    def init_gtoken(self,g,base_url):\n",
        "        gtoken= URIRef(base_url+\"#char={0},{1}\".format(int(self.index_start),int(self.index_end)))\n",
        "        g.add((gtoken, RDF.type, NIF.String )) # added after validation 26-5-2023, https://core.ac.uk/download/pdf/226128976.pdf\n",
        "        g.add( (gtoken, RDF.type, NIF.Word ) )\n",
        "        g.add( (gtoken, RDF.type, NIF.RFC5147String  ) )\n",
        "   #   g.add( (gtoken, RDF.type, NIF.Phrase) )\n",
        "    # g.add( (gtoken,RDF.type, NIF.OffsetBasedString) ) Christian sugestion to remove\n",
        "        g.add( (gtoken, NIF.anchorOf, Literal(self.text, datatype=XSD.string)) )\n",
        "        g.add( (gtoken, NIF.beginIndex, Literal(self.index_start, datatype=XSD.nonNegativeInteger)) ) #\n",
        "        g.add( (gtoken, NIF.endIndex, Literal(self.index_end, datatype=XSD.nonNegativeInteger)) ) #\n",
        "        g.add( (gtoken, NIF.posTag, Literal(self.pos, datatype=XSD.string)))\n",
        "        g.add((gtoken, NIF.oliaCategory , self.get_olia_postag(self.pos)))\n",
        "        g.add((gtoken,DC.Language, Literal(self.lng,datatype=XSD.string)))\n",
        "\n",
        "        if self.lemma!=\"\":\n",
        "            g.add( (gtoken, NIF.lemma, Literal(self.lemma, datatype=XSD.string)))\n",
        "\n",
        "        return gtoken\n",
        "\n",
        "    # olia postag\n",
        "    def get_olia_postag(self,ud_postag):\n",
        "        olia_pos = {    \"ADJ\": OLIA.Adjective, # adjective\n",
        "                        \"ADP\": OLIA.Adposition, # adposition\n",
        "                        \"ADV\": OLIA.Adverb, # adverb\n",
        "                        \"AUX\": OLIA.AuxiliaryVerb, # auxiliary\n",
        "                        \"CCONJ\": OLIA.CoordinatingConjunction, # coordinating conjunction\n",
        "                        \"DET\": OLIA.Determiner, # determiner\n",
        "                        \"INTJ\": OLIA.Interjection, # interjection\n",
        "                        \"NOUN\": OLIA.CommonNoun, # noun\n",
        "                        \"NUM\": OLIA.Numeral, # numeral\n",
        "                        \"PART\": OLIA.Particle, # particle\n",
        "                        \"PRON\": OLIA.Pronoun, # pronoun\n",
        "                        \"PROPN\": OLIA.ProperNoun, # proper noun\n",
        "                        \"PUNCT\": OLIA.Punctuation, # punctuation\n",
        "                        \"SCONJ\": OLIA.SubordinatingConjunction, # subordinating conjunction\n",
        "                        \"SYM\": OLIA.Symbol, # symbol\n",
        "                        \"VERB\": OLIA.Verb, # verb\n",
        "                        \"X\": OLIA.Residual # other\"\n",
        "             }\n",
        "\n",
        "        return olia_pos[ud_postag] if (ud_postag in olia_pos.keys()) else OLIA.Residual\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FsGB6wHJNx_U"
      },
      "outputs": [],
      "source": [
        "# manage reading sentence properties from sent (xml) and generate graph triples\n",
        "class Sentence:\n",
        "\n",
        "    def __init__(self, sent,lng): #recnik gde je id kljuc recenica cela vrednost\n",
        "        self.lng = lng # OVO SAM SAD DODALA 17.10\n",
        "        self.tokens = sent[1] # lista redova mojih!!!!!!!!\n",
        "      #  self.id = sent[1][0].split('\\t')[0].split('-')[0] # id recenice\n",
        "        self.text = sent[0]\n",
        "       # self.id = get_key(self.text)\n",
        "        #self.index_start=cur_index\n",
        "        #self.index_start=int(sent[0].split('\\t')[1].split('-')[0])\n",
        "\n",
        "        self.otokens = [] #objekti tokena lista\n",
        "        # loop all tokens in sentence\n",
        "        for i in range(len(self.tokens)):\n",
        "            otoken = Token(self.tokens[i],lng)\n",
        "            self.otokens.append(otoken)\n",
        "            if i == 0:\n",
        "                self.id = otoken.id # uzimamo ID recenice\n",
        "                self.index_start = otoken.index_start\n",
        "            if i == len(self.tokens)-1:\n",
        "                self.index_end = otoken.index_end\n",
        "       # print(self.id)\n",
        "        #for ner\n",
        "        ner_list = []\n",
        "        for otoken in self.otokens:\n",
        "            if otoken.NER and '[' in otoken.NERtype:\n",
        "                ner_list.append(otoken.NERtype)\n",
        "\n",
        "        nertypes = list(set(ner_list)) #all different types that appears, as PERS[1], ORG[12], and so on...\n",
        "      #  print(nertypes)\n",
        "\n",
        "        self.NER = []\n",
        "        ner_text = []\n",
        "\n",
        "        for ner_type in nertypes: # prolazim kroz sve tipove i kupim tekst\n",
        "            for otoken in self.otokens:\n",
        "                if otoken.NER:\n",
        "                    if otoken.NERtype == ner_type:\n",
        "                        ner_text.append((otoken.NERtext, otoken.index_start, otoken.index_end, otoken.Wiki))\n",
        "\n",
        "\n",
        "            self.NER.append([ner_text, ner_type]) #[(otoken.text1, otoken.index_start, otoken.index_end),(otoken.text2, otoken.index_start, otoken.index_end),....,], ner_type]\n",
        "            ner_text = []\n",
        "\n",
        "        for otoken in self.otokens:\n",
        "            if otoken.NER and '[' not in otoken.NERtype:\n",
        "                self.NER.append([(otoken.NERtext, otoken.index_start, otoken.index_end, otoken.Wiki) , otoken.NERtype])\n",
        "      #  print(self.NER) OVOOOOO vratiti\n",
        "    # create graph triples for this sentence\n",
        "\n",
        "\n",
        "    def init_gsent(self,g,base_url):\n",
        "        gsent= URIRef(base_url+\"#char={0},{1}\".format(self.index_start,self.index_end)) # self.index_start+len(self.text)\n",
        "        g.add( (gsent, RDF.type, NIF.Sentence) )\n",
        "        g.add( (gsent, DCT.identifier, Literal(self.id , datatype=XSD.nonNegativeInteger)))\n",
        "  #    g.add( (gsent, RDF.type,NIF.Context) ) sentence is not context\n",
        "        g.add( (gsent, RDF.type, NIF.RFC5147String) )\n",
        "        g.add( (gsent, RDF.type, NIF.String) )\n",
        "      # NIF.anchorOf, NIF.endIndex later\n",
        "        g.add( (gsent, NIF.beginIndex, Literal(self.index_start , datatype=XSD.nonNegativeInteger))) #\n",
        "        g.add((gsent,DC.Language, Literal(self.lng,datatype=XSD.string)))\n",
        "\n",
        "\n",
        "        return gsent"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mOg3d6P6AhUk"
      },
      "outputs": [],
      "source": [
        "def get_NerWiki(list_ner):\n",
        "    #[(otoken.text1, otoken.index_start, otoken.index_end),(otoken.text2, otoken.index_start, otoken.index_end), ner_type]\n",
        "    # [[[('Trgu', 135, 139), ('mučenika', 140, 148)], 'LOC'], [[('Stefana', 24, 31), ('Karačija', 32, 40)], 'PERS']]\n",
        "    #  [[[('Lina', 236, 240), ('Čerulo', 241, 247)], 'PERS'], [('Elena', 151, 156), 'PERS'], [('Lila', 269, 273), 'PERS']]\n",
        "    list_text = list_ner[0]\n",
        "    ne_type = list_ner[1]\n",
        "    ne_text =\"\"\n",
        "    if '[' in ne_type:\n",
        "        for i in range(len(list_text)):\n",
        "            if i==0:\n",
        "                ind_start = list_text[i][1]\n",
        "                ne_wiki = list_text[i][3]\n",
        "            if i == len(list_text)-1:\n",
        "                ind_end = list_text[i][2]\n",
        "            if i<len(list_text) - 1:\n",
        "                ne_text += list_text[i][0]+\" \"\n",
        "            else:\n",
        "                ne_text +=list_text[i][0]\n",
        "        ne_type = ne_type.split('[')[0]\n",
        "    else:\n",
        "        ne_text = list_text[0]\n",
        "        ind_start = list_text[1]\n",
        "        ind_end = list_text[2]\n",
        "        ne_wiki = list_text[3]\n",
        "    return ne_text, ne_type, ind_start, ind_end, ne_wiki"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zACtkTJwnvwL"
      },
      "outputs": [],
      "source": [
        "# manage NE named entities: PERS LOC ORG EVENT ROLE DEMO WORK\n",
        "class NamedEntity:\n",
        "\n",
        "    def __init__(self, list_ner,lng):\n",
        "        self.text, self.ne_type, self.index_start, self.index_end, self.wiki=  get_NerWiki(list_ner)\n",
        "        self.lng = lng #ovo sam dodala za jezik\n",
        "\n",
        "  # create graph triples for this token\n",
        "    def init_gne(self,g, base_url): #dodala sam lng\n",
        "        gne= URIRef(base_url+\"#char={0},{1}\".format(self.index_start,self.index_end) )\n",
        "        g.add( (gne, RDF.type, NIF.String) )\n",
        "        g.add( (gne, RDF.type, NIF.Phrase) )\n",
        "        g.add( (gne, RDF.type, NIF.RFC5147String) )\n",
        "    # g.add( (gne, RDF.type, NIF.OffsetBasedString) )\n",
        "     # g.add((gne, RDF.type, NIF.EntityOccurrence )) not in NIF2.0 FK and CC sugested to eliminate and search by properties\n",
        "        g.add( (gne, NIF.anchorOf,Literal(self.text, datatype=XSD.string)));\n",
        "        g.add( (gne, NIF.beginIndex,Literal(self.index_start, datatype=XSD.nonNegativeInteger)))  #\n",
        "        g.add( (gne, NIF.endIndex,Literal(self.index_end , datatype=XSD.nonNegativeInteger)))  #\n",
        "        g.add((gne,DC.Language, Literal(self.lng,datatype=XSD.string))) #ovo sam dodala sad!!! za jezik\n",
        "\n",
        "\n",
        "        if self.ne_type in ['PERS', 'PERSON']:\n",
        "            g.add( (gne, ITSRDF.taClassRef, OLIA.Person))\n",
        "            g.add( (gne, ITSRDF.taClassRef, WD.Q5))\n",
        "            g.add( (gne, ITSRDF.taClassRef, DBO.Person))\n",
        "        elif self.ne_type in ['LOC', 'LOCATION', 'PLACE', 'GPE']:\n",
        "            # g.add( (gne, ITSRDF.taClassRef, NERD.Location))\n",
        "            g.add( (gne, ITSRDF.taClassRef, OLIA.Space))\n",
        "            g.add( (gne, ITSRDF.taClassRef, WD.Q7884789))\n",
        "            g.add( (gne, ITSRDF.taClassRef, DBO.Place))\n",
        "        elif self.ne_type in ['ORG', 'ORGANISATION']:\n",
        "            g.add( (gne, ITSRDF.taClassRef, OLIA.Organization))\n",
        "            g.add( (gne, ITSRDF.taClassRef, WD.Q43229))\n",
        "            g.add( (gne, ITSRDF.taClassRef, DBO.Organisation))\n",
        "        elif self.ne_type in ['EVENT']:\n",
        "            g.add( (gne, ITSRDF.taClassRef, OLIA.Event))\n",
        "            g.add( (gne, ITSRDF.taClassRef, WD.Q1656682))\n",
        "            g.add( (gne, ITSRDF.taClassRef, DBO.Event))\n",
        "        elif self.ne_type in ['ROLE']:\n",
        "            # g.add( (gne, ITSRDF.taClassRef, Literal(\"<\" +self.ne_type+\">\", datatype=XSD.string ))) # just string\n",
        "            g.add( (gne, ITSRDF.taClassRef, WD.Q28640))    # profession, not exact, it could be title as well\n",
        "            g.add( (gne, ITSRDF.taClassRef, DBO.Profession))\n",
        "        elif self.ne_type in ['DEMO']:\n",
        "            # g.add( (gne, ITSRDF.taClassRef, Literal(\"<\" +self.ne_type+\">\", datatype=XSD.string ))) # just string\n",
        "            g.add( (gne, ITSRDF.taClassRef, WD.Q217438 ))\n",
        "            g.add( (gne, ITSRDF.taClassRef, DBO.demonym)) # check this\n",
        "        elif self.ne_type in ['WORK']:\n",
        "            # g.add( (gne, ITSRDF.taClassRef, Literal(\"<\" +self.ne_type+\">\", datatype=XSD.string ))) # just string\n",
        "            g.add( (gne, ITSRDF.taClassRef, WD.Q386724 ))\n",
        "            g.add( (gne, ITSRDF.taClassRef, DBO.Work))\n",
        "        else :  # something not expected\n",
        "            g.add( (gne, ITSRDF.taClassRef, Literal(\"<\" +self.ne_type+\">\", datatype=XSD.string )))\n",
        "\n",
        "        # for Wiki\n",
        "        if self.ne_type != '_':\n",
        "            if self.wiki !=\"\":\n",
        "                wikiQID = self.wiki\n",
        "                g.add( (gne, ITSRDF.taIdentRef, URIRef(\"http://www.wikidata.org/entity/\"+wikiQID) ))\n",
        "        return gne"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kiZ9xa_tAhUk"
      },
      "source": [
        "# Class Corpusm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OwkNraVNAhUk"
      },
      "outputs": [],
      "source": [
        "def read_sentences(file_path_name):\n",
        "    file = open(file_path_name, 'r', encoding = 'utf-8')\n",
        "    lines = file.readlines()\n",
        "    Dict = {}\n",
        "    id_text =1\n",
        "    for i in range(len(lines)):\n",
        "        if '#Text' in lines[i]:\n",
        "            Dict[id_text]= lines[i][6:-1] # kljuc je broj recenice, a vrednost linija od koje pocinje recenica\n",
        "            id_text+=1\n",
        "    sentences = []\n",
        "    sent = [] # one sentences tokenize\n",
        "    for id_rec in range(1,1001):\n",
        "        for line in lines:\n",
        "            if str(id_rec) == line.split('\\t')[0].split('-')[0]:\n",
        "                sent.append(line)\n",
        "        sentences.append((Dict[id_rec], sent))\n",
        "        sent = []\n",
        "    return sentences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zen14hvxkttT"
      },
      "outputs": [],
      "source": [
        "# manage reading monolingual corpus and generate graph triples\n",
        "class Corpusm:\n",
        "\n",
        "    def __init__(self, file_path_name, lng):\n",
        "        self.id = \"it1\"\n",
        "\n",
        "       # do the senteces\n",
        "        self.sentences = read_sentences(file_path_name) # funkcija koja cita sve recenice  i tekst recenice\n",
        "        self.text=\"\"\n",
        "        self.index_start=0\n",
        "        self.lng=lng\n",
        "      #  self.file_name =os.path.basename(file_path_name).replace(\".txt\",\"_tekst.txt\") #txt file zbog razresenja referenci\n",
        "        self.url=\"http://llod.jerteh.rs/ItSrNIF/\" +self.id +\".txt\"  #dodala sam ovde lng!!!!!!\n",
        "\n",
        "       #  metadata\n",
        "       # author\n",
        "        self.author= \"Ranka Stanković, Milica Ikonić Nešić\"\n",
        "        self.publicationStmt=\"ItSrNER1000\"\n",
        "        self.publisher = \"JeRTeh\"\n",
        "\n",
        "    # graph initialisation for monolingual corpus\n",
        "    # for graph\n",
        "    def  init_gcorpus_lng(self,g):\n",
        "        gcorpus_lng= URIRef(self.url) # +\"{0}_{1}\".format(0,) #menjati gnovel\n",
        "        g.add( (gcorpus_lng, RDF.type, NIF.String) )\n",
        "        g.add( (gcorpus_lng, RDF.type, NIF.Context  ) )\n",
        "        g.add( (gcorpus_lng, RDF.type, NIF.RFC5147String) )\n",
        "        g.add( (gcorpus_lng, NIF.beginIndex, Literal(self.index_start,datatype=XSD.nonNegativeInteger) ))\n",
        "        g.add( (gcorpus_lng, DCT.identifier,  Literal(self.id,datatype=XSD.string)    ))\n",
        "\n",
        "   #     g.add( (gcorpus_lng, DCT.title, Literal(self.title.text,datatype=XSD.string)) )\n",
        "       # language\n",
        "        g.add((gcorpus_lng,DC.Language, Literal(self.lng,datatype=XSD.string)))\n",
        "        g.add((gcorpus_lng,MS.Language, Literal(self.lng,datatype=XSD.string)))\n",
        "\n",
        "        g.add((gcorpus_lng,MS.author,Literal(self.author,datatype=XSD.string)))\n",
        "        g.add((gcorpus_lng,MS.publisher, Literal(self.publisher,datatype=XSD.string)) )\n",
        "\n",
        "        return gcorpus_lng"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6vm07KCI2iPl"
      },
      "source": [
        "# Initialisation and metadata"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w4IpcF_3CVTr"
      },
      "source": [
        "# Main function to write ttl file"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6uAuMVPbNJqK"
      },
      "source": [
        "Guidelines for developing NIF-based NLP services\n",
        "https://www.w3.org/2015/09/bpmlod-reports/nif-based-nlp-webservices/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a9kFlCBZR-yJ"
      },
      "outputs": [],
      "source": [
        "# create graph, ...\n",
        "def write_gcorpusm(file_path_name,lng,sent_num):\n",
        "    g = Graph()\n",
        "    g.bind('itsrdf', ITSRDF)\n",
        "    g.bind('nif', NIF)\n",
        "    g.bind('olia', OLIA)\n",
        "    g.bind('dc',DC)\n",
        "    g.bind('dct',DCT)\n",
        "    g.bind('ms',MS)\n",
        "    g.bind('wd', WD)\n",
        "    g.bind('wdt', WDT)\n",
        "    g.bind('dbo', DBO)\n",
        " # g.bind('eltec', ELTEC)\n",
        "\n",
        "    ocorpusm = Corpusm(file_path_name,lng)\n",
        "\n",
        "  # insert initial triples for monolingual corpus\n",
        "    gcorpusm = ocorpusm.init_gcorpus_lng(g)\n",
        "    gsent_before=None\n",
        "    one=None\n",
        "\n",
        "    scount=0 # sentence count just for testing, remove later\n",
        "\n",
        "    for sent in ocorpusm.sentences: # loop all sentences in one corpusm\n",
        "        scount+=1\n",
        "        if scount > sent_num:\n",
        "            break    # break after few sentences for test\n",
        "        osent=Sentence(sent,lng)  #dodati sentence id iznad u Sentence klasi dodala\n",
        "      #  print(osent.text)\n",
        "        gsent = osent.init_gsent(g,ocorpusm.url)  # objekat grafa\n",
        "        if gsent_before != None:\n",
        "            g.add( (gsent, NIF.previousSentence, gsent_before ) )\n",
        "            g.add( (gsent_before, NIF.nextSentence, gsent ) )\n",
        "        g.add( (gsent, NIF.referenceContext, gcorpusm))\n",
        "        gtoken_before=None\n",
        "        # loop all tokens in sentence\n",
        "        for otoken in osent.otokens:\n",
        "            gtoken = otoken.init_gtoken(g,ocorpusm.url)\n",
        "            g.add( (gtoken, NIF.referenceContext ,gcorpusm))\n",
        "            g.add( (gtoken, NIF.sentence, gsent))\n",
        "            g.add( (gsent, NIF.word, gtoken))\n",
        "           # relate token: previous, next\n",
        "            if gtoken_before!=None:\n",
        "                g.add((gtoken_before,NIF.nextWord,gtoken))\n",
        "                g.add((gtoken, NIF.previousWord, gtoken_before))\n",
        "            gtoken_before =gtoken\n",
        "           # end of token\n",
        "        for ners in osent.NER:\n",
        "            one=NamedEntity(ners,lng)\n",
        "            gne = one.init_gne(g,ocorpusm.url)\n",
        "            g.add( (gne, NIF.referenceContext ,gcorpusm))\n",
        "            g.add( (gne, NIF.sentence, gsent)) #ovo je dodato\n",
        "\n",
        "        # finish sentence graph  sid,len(tokens),sent_text\n",
        "        # anchorOf / isString\n",
        "        g.add( (gsent, NIF.anchorOf  , Literal(osent.text, datatype=XSD.string)) )\n",
        "        g.add( (gsent, NIF.endIndex  , Literal(osent.index_end, datatype=XSD.nonNegativeInteger)) )\n",
        "\n",
        "        # concatenate to monolingual corpus\n",
        "        ocorpusm.text+= osent.text+\" \"\n",
        "      #  print(ocorpusm.text) vratiti ovo!!!\n",
        "        #cur_index+=1\n",
        "        gsent_before = gsent\n",
        "        # end of sentence\n",
        "\n",
        "\n",
        "      # finish monolingual corpus\n",
        "    g.add( (gcorpusm, NIF.endIndex  , Literal(len(ocorpusm.text),datatype=XSD.nonNegativeInteger))) #\n",
        "      # Do we use NIF.anchorOf or NIF.isString\n",
        "    g.add( (gcorpusm, NIF.isString  , Literal(ocorpusm.text, datatype=XSD.string)) ) #Corpusm\n",
        "      #fn=file_path_name.replace(\".xml\",\".txt\").replace(\"\\\\\",\"/\").replace(\"/level2/\",\"/NIF/\")\n",
        "    fn= file_path_name.replace(\".tsv\", \".txt\")\n",
        "    print(file_path_name)\n",
        "    print(fn)\n",
        "    f_txt = open(fn, \"w\",encoding = \"utf-8\")\n",
        "    f_txt.write(ocorpusm.text)\n",
        "    f_txt.close()\n",
        "      # write RDF file\n",
        "      #file_path_name_out=file_path_name.replace(\".xml\",\".ttl\").replace(\"\\\\\",\"/\").replace(\"/level2/\",\"/NIF/\")\n",
        "    file_path_name_out=fn.replace(\".txt\",\".ttl\")\n",
        "    g.serialize(destination=file_path_name_out)\n",
        "\n",
        "    return ocorpusm\n",
        " #   return ocorpusm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IPiSjv6zAhUl"
      },
      "source": [
        "### Testing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qdUlzqAXaICu",
        "outputId": "733dba1c-7a4f-41ec-91cc-c11d3bd53bb7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "C:/Users/38164/Desktop/LREC_23/NIFrev/it1000_final.tsv\n",
            "C:/Users/38164/Desktop/LREC_23/NIFrev/it1000_final.tsv\n",
            "C:/Users/38164/Desktop/LREC_23/NIFrev/it1000_final.txt\n"
          ]
        }
      ],
      "source": [
        "# test on small portion of one file or list of files\n",
        "lng=\"it\"\n",
        "#ELTEC = rdflib.Namespace(\"http://llod.jerteh.rs/IT-SR-NER/NIF/\")\n",
        "# file_name='D:/Korpusi/ELTEC/ELTeC-'+lng+'-master/level2/ROM096-L2.xml' # 96,100\n",
        "p='C:/Users/38164/Desktop/LREC_23/NIFrev/' #LREC23\n",
        "#l=[\"POR0100_LucCor_Duquesa\"]\n",
        "#l=[\"ENG18471_Bronte\"]\n",
        "l=[\"it1000_final.tsv\"]\n",
        "#l=[\"FRA00101_Adam\"]\n",
        "for f in l:\n",
        "    file_name=p+f\n",
        "    print(file_name)\n",
        "    write_gcorpusm(file_name,lng,1000)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0vVW7QH6AhUn"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vD3dR-i2AhUn"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}