Drafts for ALIS article

[petermr]

the draft of the ALIS article (see https://semanticclimate.github.io/p/en/events/journal_ALIS/)

We will record all discussions on this site. Audio, transcript and cleaned versions

[petermr]

How we build this.

• semi-open Zoom meetings
• audio and video recordings (which we will post here)
• transcripts
• draft papers

[ShweataNHegde]

2024-05-13
People: SNH, PMR, RK

Publication 2A: Computable Semantic Article

• We use principles of Open Notebook Science. We record all conversations in Slack, GitHub Zoom, Zotero and YouTube. New Repository: https://github.com/semanticClimate/ami-computable-semantic-article
• The paper will be unlimited in both time and space
• Paper talks about semantic document; paper is a semantic document
• We host the final copy on Zotero
• We use Graphviz to visualise the structure of our project

[petermr]

PMR will spec out a core *.dot for the manuscript

[ShweataNHegde]

2024-05-18
People: SNH, PMR

Publication 2A

Introduction

• Idea of semantic publication (https://en.wikipedia.org/wiki/Semantic_publishing): We are no further forward than 2000. In fact, we are going backwards because we still publish PDFs
• Revolutionary ideas on how to publish semantically: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1000361 and https://www.ch.ic.ac.uk/rzepa/jodi/
• Data-rich publications. For example: https://www.iucr.org/resources/cif
• There may be examples in geosciences and astronomy
• Wikijournal of science and medicine
• JOSS: https://joss.theoj.org and RIO: https://riojournal.com

Why HTML good for semantic publishing?

• HTML has structuring. There are containers and they can be navigated
• HTML can do annotation. We can describe meaning of a phrase. <class = 'gene'>. General ontologies already exists (schema.org and Wikidata)
• HTML can do dynamic linking

Where are we now?

Tooling already exists.

• JATS is already being used. 250 terms pertaining to front matter.
• Ontologies exist
• Bibliographical data also has JATS
• We can have open source publishing

Why has semantic publishing not taken off?

• There is fair amount of tooling. In 1990s, people did XML, it was too much to handle then. But now we have HTML
• Publishing is very conservative
• There is a disincentive to make their material not semantic cause it takes away the gatekeeping power of publishers

Validation and data integrity

• We are seeing increasing number of malpractices in publishing. Some of it can be controlled by having semantic publishing with validation along the line
• A lot of experimental data comes off of instruments. If the data is semantic, it can given a stamp of validation. Let's say if a gel is run, having a unique id for it when the image was taken will make it valid. It is impossible to falsify it later in the system
• Data transference without degradation is critical (an issue with closed instruments and software)

---

Task: Literature Review

35 hits for "semantic publishing"

Papers to read:
europepmc.csv

[ShweataNHegde]

What an author does to produce a semantic document?

• Authors author semantically. Nanopublication approach
• Author finds ontology to mark-up their content (sub-section entities are marked up)
• Author records version
• Authors make use of semantic authoring tools to create tables and images

Comment on AI

Try it out to rule it out

• No quality control
• Opaque

[ShweataNHegde]

2024-06-03
#sC motto: change the world where we could

• Our work is a 500-page book. The paper is going to be a travelogue or table of contents (top-level narratives)

It includes:

1. #sC at the context of climate
2. Why #sC hasn't taken off
3. Detail of components: semantic text (means of creating it), dictionaries (semantic interpretor and link to ontologies)
4. Technology: structure of documents and annotation

[petermr]

session 2 2024-06-04

Peter Murray-Rust at 11:54 AM

0:00
computer.

0:01
OK, so we're now starting recording.

0:04
So what I'm going to do for the next hour is I'm going to try and give a scholarly talk about what we've been doing in a form where the transcript of this, , will be, , a valuable starting point, , for the paper that we have to write.

0:29
, Now, , I find it difficult to sit down sometimes and write what needs to be said.

0:37
, I've made some notes about it, , and so forth, but, , I will talk.

0:43
We're being transcribed, so you don't need to record it particularly, , and see what happens.

0:50
, I will have to also write software.

0:54
, which edits this.

0:56
, I can edit it by hand, but I I think it would be valuable.

1:00
, to write software for it.

1:02
And that will be, , fairly straightforward.

1:06
OK, it won't necessarily be in the order that we end up, , in the final version.

1:14
, so this paper represents a huge amount of work.

1:20
, and it could be encapsulated probably into a 500 page textbook, , where we deal with all the, , components of Of what we want to put in.

1:33
We give examples, we give tutorials, we give recipes.

1:38
, we give motivation.

1:40
, we collect, , relevant links and things of that sort.

1:46
So the background of this is a 500 page book.

1:53
, and we have to condense this into a narrative, , in HTML.

2:01
, which can be, , read.

2:03
, is informative, , straightforward to read.

2:08
And then that has to be condensed into 10 pages of PDF, which will be a very high level narrative.

2:16
, over what we're doing.

2:18
, by the way, I suggest that if you have, , questions, you don't ask them in the middle of this, but you put them in the chat, and I will open, the chat, so if there are questions, I can see them.

2:37
right.

2:38
So the subject of this, , article, , is twofold.

2:45
It's, , semantic publishing in its most general form, , covering a wider range of styles, components, technologies.

2:57
, and it is also centred on climate change.

3:01
, as the example of the material, , that we're dealing with, , it won't, , give answers about climate change, but it will give, , ways of finding those answers.

3:17
Now, Symantec publishing, , has been, , an exciting topic.

3:23
, since, , Tim Berners Lee, , came up with the idea of the semantic web.

3:31
, but semantic publishing actually goes back, , several 100 years to religious text.

3:38
So I'm going to show, , an example from the Christian canon.

3:43
, but this will apply to, , any of the major religions.

3:50
So we're going to look at Wikipedia, and if we look at, , study Bible and this will give us an idea.

4:06
, of, the various components which were in place several 100 years ago.

4:15
, so, this was, , only really available when the printing press came, , along.

4:26
There will have been annotations before that, but systematic, widespread annotation of documents I suspect, , only really came about, , with the printing press, so this, , is 1560.

4:45
, so that is 450 years ago.

4:49
, and here are some of the features, and my guess is that these features apply to all religious texts.

4:57
, they, start with the idea of a static corpus, , which, , needs intensive, , studying, , and where the aim is to make this as easy as possible for people to study.

5:16
And we are, , treating our, , canonical text, , as the IPCC, publications, the F C C C publications, , and, , the, , scientific literature, as in, , open archives.

5:37
But here is a list which actually translates very well to what we want to do.

5:42
Annotations, , explaining passages.

5:46
short, , biographies of people, places and topics.

5:54
And we will see that a concordance a a word index that indicates where the various keywords, , were used.

6:03
, in the corpus, , a contextual setting.

6:10
, So, , what is the wider environment of this particular, , corpus?

6:17
, illustrations.

6:21
So those are examples they might be.

6:25
there might be short articles or there might be physical illustrations, introductions, historical notes, , how to use it in life.

6:37
Maps and charts, references and cross references.

6:42
, and possibly, , variations and interpretations.

6:48
And at the bottom, very important word and phrase definition.

6:53
So I've taken that from, , the Christian, , literature.

7:00
But that will apply to any, , important corpus, which is largely static and requires a great, , study and interpret protection.

7:14
And, , if you remove the religious aspects of this, , then, , these things that I've written out are, , very much, , What we want to apply to, , our corporate.

7:29
, Now, these were initially done, , in printed form, so you would find documents which had, , printed annotations, footnotes and notes, , many different ways of, , interpreting what was going on.

7:49
And it wasn't limited to, , , it wasn't limited to religious texts.

7:55
It was, , also, , , used for widely read books.

8:03
, such as Shakespeare in the, , English literature canon.

8:08
, and also, , books such as Alice in Wonderland.

8:13
Martin Gardner wrote a book called The Annotated Alice.

8:18
, so annotations, , Seman education, , has been going on for hundreds of years.

8:25
, what it doesn't mention here, but is, , really, essential, , is the, , microstructure of the document, , so that, , it's broken down to, , in religious settings.

8:44
Verses?

8:45
, so verses are numbered and you can refer to a particular verse by its number, , or the book it's in Or a combination of both.

8:55
, And that means, , that you can, for example, build a concordance and link into, , the, , , into the corpus.

9:09
You have to.

9:11
flip the pages manually.

9:13
, but now, , we can link directly into the electronic, , version of this.

9:21
no.

9:22
the, this persisted for many years, but with the development of the web, , it became possible to, , enhance this with electronic means.

9:34
, and in 1990 approximately, , Tim, Berners Lee came up with the idea of, , not just the World Wide Web, but the semantic web and the semantic web.

9:50
, incorporates many of the things, , that you see here, , and, , it, is a vision was a vision, , whereby, , formal, , knowledge could be turned into semantic form.

10:09
, I have always espoused this decision and this, , vision.

10:15
and, , I and Henry Zeppo once others tried to do this for chemistry.

10:22
It has to some extent been done in bioscience where there are some semantic article artefacts where identify.

10:40
sorry, the identification of micro components.

10:44
has been done.

10:46
, but generally, , it hasn't, , been widely done, , in, , science or, , many other fields.

10:57
It's done in things like legal publications.

11:00
So legal publications are have a semantic structure.

11:04
, they have indexed, , components.

11:07
, many, , government documents, , have numbered, , sections and so on.

11:14
But, , ho, the complete electronic seman application is still, , extremely patched.

11:24
, one of the best examples of, , a semantic, , corpus, , is the wikimedia system.

11:33
, where, , all of the components, are, , microstructure, and, , identified with, , persistent identifiers.

11:47
, there are search systems.

11:50
, and, , there are reuse systems.

11:54
So, , in a fully semantic system, , what you have is, , something that could be read by humans, , but can also be read by machines.

12:06
, And where, , annotations and structures are made explicitly available, , to both machines and humans.

12:20
the potential for the semantic web has been there for, , over 30 years.

12:27
Unfortunately, , many disciplines have not adopted it.

12:32
And in particular, , scholarly publishing has been completely unaffected by the ideas of semantic publishing.

12:42
to do semantic publishing.

12:44
, you need a, , substrate, , a technology which allows, , the identification of, , individual micro components and addressing into them, , the identification of those components in the wider system.

13:06
So linking into definitions and explanations and further links and almost all the scientific literature has, , been unaffected by this for 30 years and is published in, , PDF.

13:26
, which is, , unstructured, , un reusable, , and, , a very long way from Symantec.

13:37
So this article, which we are writing for, , a list is, , required to be un semantic because of, , where it is published.

13:48
But we are, , publishing a semantic version of this to try and show in, , the first quarter of the 21st century what semantic publishing should be like.

14:02
And we urge all publishers to think again, , to incorporate semantic publishing, , into their output Now, , I'm going to show now the idea of, , Nano publications.

14:19
which is a term invented by, Baron Mons and, , and, , Jan belter up.

14:34
So here is a N publication.

14:40
I don't know when it dates from, , it actually says 2022.

14:45
But how much change has been over the last 10 years?

14:49
I don't know, but a Nana publication, , consists of assertions, , provenance.

14:57
Where to find it.

14:58
And so this is what we are, devising a system for applying to the, , corpora that we're working on, So that's a simple example.

15:17
Let's just see if there's more of it here.

15:20
that is copyright 2020.

15:33
So, , there is some tooling for it here.

15:38
, there's a system called, Nano pub here, which we might look at.

15:45
, but generally, nano publications and semantic publication, , have flatlined for 15 years.

15:56
in the late two thousands.

15:58
Semantic publishing was very exciting and if we go to Wikipedia and we look for a semantic publication.

16:26
we can go to s semantic publishing.

16:34
, and this was, I think I agree with this.

16:40
Semantic publishing has the potential to revolutionise scientific publishing.

16:45
, and Tim predicted that it would like to profoundly change the very nature of how scientific knowledge is produced and shared.

16:55
Shared.

16:56
I buy into that vision, five years later, a revolution in how scientific content is managed through its life cycle and so on and then there are ideas as to how to do scientific publication.

17:18
Now what you'll see is that how to do it requires technology.

17:25
, it requires, , systems such as, , r d f , which structures, , information as triples, , or possibly enhanced triples and owl, which is an ontology language, which adds, , ontological meaning, , to the semantics.

17:48
, And, , just to try and explain that r d f will tell you what the bits are and what the relations are between them, but it won't Won't interpret those relations in the wider context.

18:05
, So for example, , it might say that, might say, for example, that , toluene is a chemical compound.

18:21
, so that's a cripple.

18:24
Toluene is the subject is, or is instance of is the predicate or relation?

18:31
and, compound, , is the target or object?

18:38
, but what does compound mean?

18:41
And you need an ontology, , to understand that, and by far the most, , universal and valuable, , ontology now is wikidata wikidata contains over, , 100 and 10 million, , entries.

19:00
Many of them terms.

19:02
, though some of them are identifiers to other objects.

19:07
, and you can find information about, you know, the whole of the infrastructure of this planet and its place in the universe in wikidata, it's patchy because it depends on volunteers.

19:25
And, for example, in bioscience it depends in, , people adding the genes to it, , the protein structures, , and so on.

19:38
And they're doing this and in chemistry, adding the compounds, , to it, , and in other subjects Volunteers, , adding information, , into wikidata.

19:52
So it's patchy, but, , , it's enormously valuable.

19:57
And we work on the basis that are, , semantics are done by, , wikidata.

20:06
and, , the links from it.

20:10
So, , by finding a wikidata unique identifier or, an object.

20:20
, that, allows you to link, into wikidata and then to link out into the ontological structure so you can have relationships like, , superclass objects and so on and subclasses.

20:39
And it can also link to a wide range of, , , related objects which enhance our understanding of it.

20:48
So that's our model or semantic publishing.

20:54
no.

20:56
in the ideal world.

20:58
, we would have semantic publishing at source, , and that requires two things.

21:05
At least it requires the will to do it.

21:09
And the technology, , to do it.

21:12
And both of those are very, very patchy at the moment.

21:18
, I will rant for a short period about the publishers.

21:24
, the commercial publishers now are driven by monetary value, , rather than, , value to their readership.

21:34
, and, , they actually have reduced the amount of semantic value in their publications.

21:42
So, , initially, , when I started, , publications such as the biomed central publications would publish in XML.

21:56
M Most publishers actually, , convert their input into XML in-house.

22:05
It's an expensive and wasteful process.

22:08
, but they create XML, but they do not expose the XML.

22:14
So there are very few publishers which expose, XML.

22:19
And that's a major tragedy for, , the semantic, , web of publishing.

22:26
a.

22:27
the primary product of a some of a modern, , corporate publisher, , is a PDF, , which is, , put as a, , permanent object on a website.

22:42
, or semi permanent.

22:45
, you may have access to it.

22:48
, if it's open, , or you may have to pay for a subscription, either through your institution or personally, and those charges are very large.

23:01
the publishers make a very large income 30 or 40% profit.

23:07
, from this, because their costs are low.

23:10
Creating PDF s , which are not reusable is a very cheap process.

23:18
and, , so there is no incentive to change.

23:23
So although I have engaged with publishers over the years, , and tried to persuade them to be semantic, , there is now no desire in the publish community, , to create semantic publishing.

23:39
And it's relegated to a very small number of publishers, such as the Journal of Open Source software.

23:46
, pre eminently the Wiki Journal of Science and Wiki Journal of Medicine.

23:51
These are excellent examples of semantic publication, and if the whole world could move towards that, it would be a major, advance and I would urge a list the annals of Library and information studies to move towards that model of semantic publishing because it would transform the value of what they create.

24:20
PDF s have, , very little positive value.

24:25
, they have the value that they are unchangeable.

24:29
So if you value publication as, , creating a single static version of record, they have that minor value.

24:39
, it is difficult to edit them, so they have the minor value of being, , unalterable.

24:47
, but that is basically their only value.

24:52
a in the, , community of academia publications are valued, , because of, their CIT and PDF s can be published and cited, , in a way that allows, organisations and individuals to aggregate their scientist metrics.

25:15
, and this is, , a way that some people find valuable as assessing people and organisations Personally, I believe it's, , counterproductive.

25:29
, and, , should be abandoned.

25:31
And, , some, , universities, , do not now, , regard, , individual journals, , as being useful citation containers.

25:45
, but, the negatives of PDF are enormous.

25:52
They are not reusable.

25:54
, so it is very difficult to transform a PDF into a different form.

26:00
So, for example, if you want to transform it into a different human language, , that is very difficult and expensive.

26:09
, if you want to use part of it, , that is very difficult and expensive.

26:15
So you might, for example, , want to, , extract just the methods of a paper.

26:22
with a semantic publication.

26:24
That's almost trivial.

26:26
, if the, , section is labelled as methods, , but to get that out of the PDF s is extremely time consuming and error prone.

26:37
, similarly, , .

26:40
And another major problem with PDF s is that every publisher has a different format.

26:48
Now.

26:48
There is no, , technical reason for this.

26:52
, , there are standard ways of publishing, but publishers choose to do it because it brands them as different from other publishers.

27:04
And this is extremely counterproductive making, , the job of, , , using PDF s, , many times.

27:14
, harder.

27:16
Let's say that there are 200 major scientific publishers , and many thousands, , tens of thousands of journals To convert all those into semantic form from their PDF s is a nightmare.

27:33
, so PDF s are not easy to convert.

27:36
They are not flowable.

27:39
So if you put a PDF into your device, , let us say a, , Mobile mobile.

27:47
, then it is very difficult to read.

27:51
, , people with impaired vision or particularly, , you know, those who have age related.

28:00
, sight deficiencies.

28:04
, such as sometimes me.

28:07
will find it difficult to read a PDF, , by scaling up the size of the fonts because it doesn't flow around the edge of the page.

28:18
And that, again, is a major drawback.

28:21
, and it is actually an insult, , to people who are sight impaired, , in any way.

28:28
It is extremely difficult for, , blind people to read PDF s.

28:34
it relies on speech, , readers of, , some sort, , which are imperfect and not semantic.

28:47
so.

28:49
, the sorts of things we would like to be able to do with PDF s is to microstructure them or reuse and reinterpret ability for navigation so that you can navigate into a particular, , place in a PDF.

29:09
Most systems at the moment will only navigate to the top of the PDF and not into, , the, , article itself.

29:20
, you cannot reuse components of it.

29:24
So you cannot excise, , and, , aggregate different components.

29:29
You cannot annotate it.

29:31
, so that if you have, , a word or phrase, , which you want to link to an authority such as wikidata.

29:40
You can't do it.

29:41
and you cannot use, , modern, , machine tools, , such as summarises or, , aggregators or, , translators or whatever, , to work on that pedia.

30:00
in contrast, , and by the way, if if I've missed any downsides of PDF s, 00, yes, there are, , some downsides which you will occasionally find, which is Sometimes the PDF is actually an image of the text, and that's particularly difficult.

30:18
So rather than having the text itself, , in, , at least character form, , , it will only be present as, , a bit map, , and that will have to go through optical character recognition.

30:37
OCR, which is, , very, very lossy.

30:42
, there are many characters that can't distinguish between, , such as minus signs and M dashes, , or, , vertical bars or different forms of Let us say, , Alpha, , and certainly many non-english languages have poor support.

31:04
so that is something which is a major downside a so.

31:12
, ideally, therefore, we should have a semantic version of the, , of the text now, , a semantic publication consists of, at least the following.

31:29
, the structure.

31:31
So you need to know how to navigate the document.

31:35
, the textual material, , which is often split into, , front matter.

31:44
The front matter is, , where, , the article came from its provenance, , and so forth, , and that is usually very well catered for by the publishing community.

31:59
So you can find the authors, the institutions, the journal, , sometimes keywords, , and so forth.

32:07
, and that's, And there will often be an abstract in there of perhaps, , a few, You know, 100 to a few 100 a lines.

32:19
Sorry.

32:22
a words, and sometimes that abstract will be structured.

32:26
But that happens in medical text, but often not others.

32:31
, And then there will usually be an introduction.

32:35
, which will set the scene for the, , article, , depending on the type of article.

32:43
Some of them report, , experimental results.

32:46
Some of them report technology.

32:47
Some are reviews.

32:49
There will often be a method section, , saying how, , the authors carried this out.

32:56
, there will be, , an experimental section if experiments are carried out, , there will be several specialist subjects like, , statistics.

33:11
or, , software things of that sort.

33:18
there will then generally be a result section.

33:22
, and a conclusion section.

33:25
And then at the end, , there will be back matter, , which is usually the references cited in the text.

33:33
, and possibly some qualifications about, , the status of the document.

33:40
Maybe it's copyright.

33:42
, And, , maybe the contributions of, , different people if it's a scientific, , publication.

33:50
Now, this will vary according to the, , journal, , and it will vary according to the authority that's created it.

33:59
Such as the United Nations, or, , other authorities.

34:05
So that's, , our, , corpus of semantic, , o of documents.

34:13
And we want to make these semantic now, efforts to make this semantic over the last 30 years have failed.

34:25
, semantic publication is not a common topic in the literature.

34:31
We have found very few, , reports of it, , in either, , open, Alex or, , , Europe pub, Med central.

34:44
, there's a few tens of, , publications about semantic publication or nano publication, so it's very, very much a minority interest.

34:58
, we want to change that.

35:00
, but, it means that.

35:08
to have a semantic publication.

35:10
We have to have a large infrastructure which supports it.

35:16
And we have been building this infrastructure in semantic climate.

35:21
Now we've, , We've visited various subjects.

35:28
, we've visited not only climate, but also viruses and, , plant chemistry.

35:36
Phyto chemistry.

35:38
, but, , the technology we've developed, , relates to all of those.

35:45
so I will now describe the, , climate corpus because that is what has exercised us.

35:57
And then I will describe the, , technology, , that we have developed, , over, , several years.

36:07
I won't describe the social innovations because it's another paper that Sim Simon Wellington is leading and that some of us are also authors on.

36:22
, I've been going for About what?

36:27
, half an hour.

36:29
So I'm going to take a breath, , refresh my brain.

36:35
Mm.

36:38
, nobody's put anything in the chat, but are are there any questions, , from people at this stage?

36:46
Well, we are listening to you.

36:48
So that is why we are not putting any questions.

36:52
But you might think that something I said was wrong.

36:56
You might think it was incomprehensible.

37:00
you might think that I'd missed things out.

37:04
The most difficult thing, by the way, in writing articles is not what you put in but what you, , leave out by mistake.

37:14
Right.

37:14
OK, so I particularly want your brains in gears as to things that we haven't put in place.

37:23
, OK, , Amy has another drink of benel tea.

37:32
an old tea.

37:37
OK, deep breath.

37:39
, and now we go on to the, , corporate that we're dealing with.

37:47
So we're dealing with three corpora.

37:50
, the, corpora R , the whole scholar literature, , with open Alex and, , Europe pub.

38:00
Med central.

38:02
, and here I want to pay a tribute to, , Irish garg and, , Freter.

38:10
Heder.

38:12
They will both be authors on this, , and I get papers, doc, analysis, , a tremendously powerful tool which the world has not recognised The importance of so part of this, , effort here is to show the world what this tool will do.

38:32
And we've started.

38:34
Very, , very, , we started very gently to get people on board who want to use this.

38:45
, so Holly Meeks, one of our, , community wants to use this for, , querying the scholarly literature for, , , , NGO, , material, , such as, , injustice, , climate and things of that sort.

39:06
, we have a I have another, , request yesterday, , from someone I haven't met, , who wants to, , use these tools for finding climate information in, , the, , UK election, which is taking part in, , in about five weeks time.

39:36
I note, by the way, that the Indian, , results are just coming out.

39:41
I hope you all voted.

39:42
I'm not going to ask, , whether you did.

39:45
, and so on.

39:47
, but my grandmother, , demonstrated for the right for women to vote.

39:56
, in the UK, which was one of the very first countries, , to have, , votes for women.

40:03
It was very contested.

40:05
, the male, , hegemony didn't want it.

40:10
, the women would demonstrate, and they one of them, actually, , threw themselves in front of a race horse and was killed.

40:19
, that's how strongly they felt about it.

40:21
So when you have a vote, use your vote.

40:24
OK?

40:25
End of political speech.

40:28
and so.

40:34
lost my crack.

40:38
I'll go back onto it.

40:39
, yes, this was a potential.

40:44
Yes.

40:44
So, pet papers, , is can be a wide use in people who want to research any subject.

40:52
, which is going to be covered, , in the open, , literature.

40:57
, and I was delighted to see that I has managed to put in a tool in, , P get papers, which actually downloads PDF s, , when they exist.

41:08
So very simply, , Page get papers, , takes a query.

41:16
we may try and make that a semantic query, , by using dictionaries.

41:22
, So, , we might be able to say, , two p get papers.

41:27
Don't just ask for a single, , term, but use the terms, , in this particular, , dictionary.

41:35
So we've been collecting dictionary terms this morning.

41:39
, , if those are in a general dictionary of climate change, , it would be valuable to say, , let's run p get papers with terms from this dictionary.

41:51
And that way you get a much better, , coverage.

41:55
, because, , you know, it will include, , a variety of terms.

42:01
So, for example, if we wanted to look for semantic publications, we'd also want the term Nano publications in, as well, , that's a typical example.

42:14
, right.

42:15
So, , , I get papers, , takes a query.

42:21
Hopefully one, which has a wide number of terms.

42:26
It constructs it automatically if necessary.

42:30
, it gives you the opportunity to use the additional qualifiers in the remote site.

42:38
And by the way, there's no real standard for, , querying repositories.

42:45
So pet papers is valuable in working out how to do this in different repositories.

42:52
, and then it will return a list of, , , the metadata for that publication.

43:00
Now, the metadata is the front matter, , for the publication and consists of things like, , the journal, the authors, the date, , the subject, the title, , et cetera.

43:16
and that's very well represented in these repositories.

43:20
And so the first thing that pet papers does is to create the metadata.

43:27
and to save it on your file store now, if you want.

43:36
And if you can spare the bandwidth and the disc space, you can also download whatever full text there is now, , in, , fully opened papers in, , Europe Pub Med central.

43:55
, most of these have full text because that's a condition of funding by, , many, , funders, And it includes funders such as the National Institute of Health in the US, who are also run, , med Central.

44:15
, the well, trust who've been pioneering this, , , and various other funding bodies.

44:24
, and they say if we fund you, your publications have to be open access, and they have to be, Ah ah, They have to have the full text, so there is an opportunity, , in that to download the full text.

44:43
, however, there are many publications which are not full open access.

44:49
, and that actually includes the A list, , publications.

44:54
, because they are, c c by, , non-commercial non derivative.

45:02
, and that means that they are not open access in the technical sense of the term.

45:09
Although you can read them on the web page.

45:12
, And, , for those, , non non, fully open access papers, , you can often download the metadata, including the abstract, and that can be valuable open.

45:27
Alex, , takes things further because it has the whole of the scholarly publication, not just the bioscience ones.

45:37
, and that includes, , a wide range of publishers and journals.

45:43
, I should also say that both open Alex and, , Europe Pub and central value preprints.

45:51
, pre prints are very valuable because they necessarily contain the full text.

45:57
, and often that full text is identical to what is put in the publisher's, , hidden.

46:05
, PDF server.

46:09
, And so, , both of those are able to download metadata where it exists.

46:16
, and then there is a, , a concept called green.

46:23
Open access.

46:24
The terms green and open access are both extremely, , fluid.

46:30
, they are badly chosen.

46:32
, because they don't describe exactly what you've got.

46:35
But green open access means that the author or their institution, , has put the full text of the author manuscript, , on a site where it can be seen publicly.

46:50
, so, mo.

46:53
Most universities have, , repositories where people can, , deposit their author manuscripts.

47:02
, and this means that if you can find a link to that, then you can find the full text.

47:09
I differentiate this from sci hub, , which I'm not going to comment on.

47:14
, sci hub is regarded as illegal by, , some, , publishers.

47:22
, and they will sue you, , if you try and, , use it.

47:29
And they also have encouraged universities to stop people using sci hub because they say it's illegal.

47:38
I have strong views on this, which I will not put into this p , But what we're talking about here is legal repositories, and it is possible to get probably half, , the, , world's published text, , in, , at least, PDF or non semantic form.

48:04
It's much harder to get it in semantic form.

48:09
, OK, , so the second, , repositories that we're looking at are those which are, , exposed by the major governmental and nongovernmental organisations.

48:27
And in particular, , we're looking here at, , the United Nations, which exposes a, , all the reports of its cops and, , the, What is it?

48:44
Conference of the parties, , which occur every year.

48:48
The next one is 29 in Azerbaijan, and it publishes something called U NFC C C reports, and we have got a system which turns that into semantic form, and the other is the IPCC reports, which come out every five years.

49:08
Approximately, they're called assessment reports and designated by the terms a R number, such as a six for the last one, and people are now starting to work on a r seven.

49:26
now, , those reports, , both of them u NFC c c and, IPCC are static corpora, so they are well suited for our purpose.

49:42
So that's what we're working on now.

49:46
We need to convert that into , semantic form.

49:52
, and often what we have to deal with is PDF s.

49:57
And so if we take the u.

49:59
F.

49:59
C c c reports, those are only available as PDF s.

50:05
, and we have to turn those into macrostructure so that we can work out the components of the publication and microstructure so that we can identify individual phrases and link that into onto and by default.

50:29
Our semantic tool is HTML, , usually well formed, in other words, with balanced brackets in so that it can be parsed by an XML parser and, , , annotated in various ways so that we can tell what the role of a particular component is.

50:57
We only take static.

51:00
, , markup.

51:02
So we do not have things like, , , scripts in.

51:09
, we don't We take out all the javascript.

51:13
, and, , we don't have semantic, , links in, , which, , dynamically load things.

51:22
, and so forth.

51:23
We have a single semantic HTML document.

51:27
, , which can be viewed as a single document, , with additional structural and, , micro markup.

51:41
OK, so without being exhaustive, the types of, , , HTML markup we put in our document are, , head.

51:53
, so the HTML has, , normally, , up to three components head, body and back.

52:01
And those correspond pretty closely to the, , publishers.

52:08
, notions of front matter, , body and back matter.

52:15
the, terminology here is within.

52:20
It is not completely standardised, but the, , Pubmed central.

52:27
, and other groups use something called J a journal article tag suite.

52:34
, and that has got about 250 , mark up, , components, , which will tell you about, , authors, , , journals, titles, et cetera, et cetera.

52:51
And so, the, , public public pub Med central, , published that, , as chats.

53:01
And they've done a pretty good job on marking up the front matter.

53:06
, so, , we can reuse the J A s terms in our HTML if necessary.

53:14
, so if we can identify authors somewhere else, , we can, , put in the chats terms for authors.

53:24
, because it's then trivial to convert it back to a chats.

53:28
Compliant tool.

53:30
, so where possible?

53:33
We will use that, , the markup in, , HTML, , can be done in several ways.

53:44
, it can be done by the name of the, , tag.

53:49
It can be done by the tag.

53:51
So, , in the front matter, we have things like meta.

53:56
, we have script.

53:58
We have link.

53:59
and so on.

54:04
And then in the main body of HTML, , we have body.

54:11
We have div, which is, , a division, which is a general container that might occasionally be things like section , within that div we have paragraphs which are p within the paragraphs.

54:29
, we may have, hyperlinks, which are a or anchor.

54:35
, we may also have, , tables which are given special support in HTML.

54:43
, and we may have, figures, , which have captions and which can include, , both s v g , which is scalable vector graphics, which is semantic graphics and images, which are bit maps or un semantic.

55:05
There may be other tools in there, but we don't use them very much.

55:12
So generally the structure of our HTML document will be front matter.

55:18
body with DS powers lists.

55:25
I should have mentioned lists.

55:27
, , which contain list items, , tables which contain, rose.

55:38
, and, , sows, and, , figures, , which contain a graphically displayable, , objects along with their captions.

55:57
OK, so that's the, that's what we may find in the HTML.

56:05
Now, the HTML also allows for an annotation called class, , which can be found on any object, Well, not any object, but on the ones that we use.

56:20
So the DS can have classes.

56:22
The peas can have classes.

56:24
, and those classes, , are then contain a list of, , keywords, which an HTML, , style tool can recognise.

56:37
So if you, , have a paragraph, which is class, , , let's say it's called a class equals.

56:53
trying to think of of something here.

56:57
oh, it might be class equals glossary, right?

57:01
So it might have class equals glossary term.

57:04
We would possibly have a special display for the glossary.

57:11
, so that means that the, , CS s, , the cascading style sheet can recognise HTML classes and can apply the appropriate style to them.

57:24
And this is extremely important.

57:27
And this is what Simon has been developing with cascading style sheets and CS s page media.

57:36
And it means that instead of putting the style within the document, which is, , , a poor way to do it.

57:44
You, , put the pointer to the style, , in each of the elements.

57:51
, and this is extremely powerful in what we must adopt now, ideally, that style is semantic.

58:01
some of the Web based tools that we use, such as, Gatsby and WordPress use, , non semantic markup so they might have something.

58:16
, which is called, , I don't know, display page or, hide or things of these sorts, which does not tell you what the content is.

58:30
It tells you how the tool should deal with it.

58:35
And it would be better for that to be in the CS a style sheet.

58:41
, than it would be, , in the, , actual, class.

58:49
So we try and look for semantic classes.

58:53
, in this.

58:54
So our ideal is, , dibs and peas, which have got semantic classes.

59:01
, on their, , in their, attributes.

59:10
No.

59:11
I'm beginning to run out of energy.

59:15
, but, , I'm keeping going until I crash.

59:20
, OK, , So the next thing we have to have and this is critically important is I DS So, , the whole of the semantic web is based on I DS, and, , references to those I DS the I DS can be, , they can be arbitrary made up by humans.

59:47
They can be machine created and, , non semantic.

59:52
Or they can be semantic A, human, , human identifiers are not easy to maintain.

1:00:02
, and, what we've tried in the past has always broken down.

1:00:08
So we're not going to ask people to create their own identifier system because it probably won't work.

1:00:18
, non semantic machine identifiers, , often are generated from the content of a piece of information.

1:00:27
, they are unique.

1:00:29
, , So, , there are various ways of hashing them into something whose, , ambiguity is trivially small.

1:00:42
, these are things like, MD five hashes and things of that sort.

1:00:47
But what we want to do is to generate semantic identifiers.

1:00:53
, so When we have a paragraph, for example, we want to give that a semantic identifier, which says.

1:01:06
where it comes in the document.

1:01:08
, so our our documents have semantic identifiers.

1:01:13
, our bodies and DS have semantic identifiers.

1:01:17
And then our paragraphs, , within that might have something like, , p one p two p three, something like that.

1:01:26
And remember, we're dealing with something, which is, , a static corpus at the moment, so the numbering of the paragraphs will not change.

1:01:40
that is allocated automatically by Amy Tools.

1:01:56
And you will find documents in there called, HTML with I DS.

1:02:03
And that means that we have used an automatic tool to add i t s add I DS to the micro component.

1:02:12
we, , then also try to annotate within the micro components.

1:02:21
So if we see something, , that such as what we, , found today, , with, , the, climate terms.

1:02:34
So let's say we've got something like radiative forcing.

1:02:38
That is an in line, , component.

1:02:42
, , and that in line.

1:02:45
, HTML has tools for annotating, , which are called spans, , or anchors.

1:02:55
, that's the A, , attribute.

1:02:58
So sometimes we might break a paragraph up into spans.

1:03:03
, and the, IPCC, , have, , a natural place where spans can be used.

1:03:15
So they have, , paragraphs, for example, which contain, human activities are responsible for, , increased carbon dioxide.

1:03:30
Something of that sort.

1:03:32
And then they will have a bracket in italics saying almost certain something like that.

1:03:39
Now that almost certain would be a span.

1:03:43
So a span within a paragraph, and that is a controlled vocabulary of, IPCC.

1:03:51
So we will index all the controlled vocabulary of IPCC.

1:03:56
At the moment, they have two axes, likelihood, , and confidence.

1:04:02
, so, , , something like that might have, that is not going to have likelihood, because likelihood is about, , the likelihood of an event, , occurring.

1:04:18
So that's a statement about the, past.

1:04:24
, And, , so it would say, , you know, , absolute confidence or something like that.

1:04:32
, And then they might say sea level will continue to r, , to rise for the next, , next 100 years.

1:04:44
And then they would put after that, , highly likely.

1:04:51
high confidence.

1:04:52
So that would have two annotations.

1:04:54
They've made a prediction.

1:04:56
They say it's highly likely, and they're very confident about making it.

1:05:01
So that's the type of markup we would apply.

1:05:05
, to this, , And then, , we would pick out those terms, which we feel to be important.

1:05:14
And those terms are either words or phrases.

1:05:19
, and those would then be linked into our diction.

1:05:22
So, , something like, anthropogenic.

1:05:28
climate change would be a, , a term which we would have in our dictionaries, , either from Simon's glossary, which is which we have converted from the IPCC or which we've added and linked to wikidata and where possible, to the IPCC.

1:05:52
So we have to recognise those terms, and we have to put in links by putting the ID, , of the, , of the glossary ditch.

1:06:07
, that can be done to quite a large extent automatically.

1:06:13
And it will be done automatically with tools like, , doc analysis, , which will tell us which the terms are, , that it thinks are significant.

1:06:25
Or the ones which we've given it already, , to use for markup, , and and Doc and Alice will analyse that.

1:06:35
Or tools which, , I have written in Amy Dict, which allow you to go through a paragraph and add in the links to a particular, , glossary.

1:06:51
So that's the approximate structure of what we're trying to get to.

1:06:58
I'm going to keep going, , for a little while.

1:07:02
, and now how do we do it?

1:07:05
in all cases, we are in the business of converting legacy data into.

1:07:14
semantic, , documents.

1:07:17
, and, , mark up, , some of the le The easiest le legacy documents are the XML versions from, , the, , scientific literature.

1:07:31
, and that is done largely automatically by, , pet papers.

1:07:36
And Amy, , the next most tractable are HTML.

1:07:43
, we can sometimes find those on, , publishers, web pages.

1:07:48
, which is, , an easier way.

1:07:52
, , to do it.

1:07:54
But we have to do quite a lot of the markup, or, , we have been able to scrape the HTML from the IPCC.

1:08:03
Reports.

1:08:05
, and that's been extremely valuable.

1:08:08
, there are one or two, glitches in that, , occasionally, words are either run together or there are, missing space or, , inserted spaces.

1:08:25
, because they've, , , used spans in an unusual way.

1:08:31
But we can get around that.

1:08:32
That's not a big deal.

1:08:34
So that's HTML where we've got it.

1:08:39
Then we've got nicely formatted PDF that actually the most likely place to find that is in, , authors manuscripts, which are usually single column PDF, , without, , what I call, , publisher craft.

1:09:00
, So, , when somebody writes a manuscript for publication as we are going to do, we will not put in things about how wonderful the journal is and how many people, , we want to cite us, and, , things of that sort we just put in, , the , what we want to say, Our message, our front matter, our body and our back matter.

1:09:29
Although we won't necessarily say divided into those, , sections.

1:09:37
the PDF converter we have at the moment.

1:09:39
PDF plumber is actually very good at converting that and , we've got a system which runs very well, , for that type of document.

1:09:55
then we have things like the IPCC, , material, which is, relatively straightforward to pas.

1:10:07
, we have to put in the various templates, , to decide what are headings and things of that sort, but they are quite well done by content.

1:10:17
, and so forth, so the I PC.

1:10:23
is tractable, but it's not our top priority from the point of view of content at the moment.

1:10:30
But it will be, , later.

1:10:34
Then we have publisher PDF s now leaving aside the ones which are bitmaps and need O C.

1:10:41
Ring.

1:10:42
, the problem with , publisher, PDF s is they're often in, , double columns, which makes it more difficult to, , pause.

1:10:52
They're in pages, , which makes it more difficult to pause.

1:10:57
, and all of this has to be undone.

1:11:01
, and so when we pass the PDF s, we have to try and decide where how pages run together.

1:11:10
We have to try and have to decide how lines run together where a paragraph ends and stops.

1:11:16
What is a list?

1:11:18
Because it doesn't say this is a list.

1:11:20
, yes, absolutely.

1:11:23
We can continue tomorrow.

1:11:24
, and that's brilliant.

1:11:27
, Renault and I've more or less come to the end of, , what I wanted to say.

1:11:33
, And what we can talk about a bit tomorrow is what our tools are, , that allow us to do it.

1:11:41
So coming on to the tools, I'm just going to give Marcus for tomorrow.

1:11:46
We have a suite of tools, , which discover and, , repurpose, , legacy material and turn it into semantic form.

1:11:58
, and those are, , pet papers for discovery.

1:12:03
, , doc analysis for, , the identification of potentially semantic, , content.

1:12:12
, Amy Lib, which is a general library for downloading and, , reading PDF s, , and contains a PDF processor and downloaders for head headless browsers such as are required for the for the UN p UN IPCC material.

1:12:42
, and, , also wikidata look up tools, , so that we can look up things in wikidata.

1:12:50
And now what we're adding, , is, , dictionaries so that we're able to look up terms in wikidata and come up with .

1:13:07
, descriptions, definitions, , related terms, translations and things like that and Amy says that she is feeling a little bit tired after that.

1:13:26
But that is the bulk of our manuscript.

1:13:31
Does that make sense?

1:13:32
RNA?

1:13:32
Yes.

1:13:33
Yes, we did a lot.

1:13:36
And if what I'm going to do, probably this afternoon is to write a tool which will try and turn the transcript.

1:13:44
If I don't have the transcript on this, I am going to be mad.

1:13:48
So I have to stop sharing you first.

1:13:52
You post the recordings in the slack.

1:13:54
Yes.

1:13:55
Well, I have to make sure I've got the recording, so I will stop the share.

1:14:02
right.

1:14:02
, I have a doubt, actually, yes, , during last month or something, , Reno, ma'am.

1:14:12
Told that we'll be adding, , words of, like, our mother tongue in the IPCC group Name like the alphabet for me.

1:14:21
It's s because it's Smith.

1:14:23
So how do we have to do that?

1:14:25
Like, because I was in a long holiday, So I just got I just, like, remember that we will describe tomorrow.

1:14:35
I think we tell you how to do that.

1:14:38
You will pick a bit of the glossary relevant to you.

1:14:42
, so you might, what what is your chapter?

1:14:49
Yeah, it is, Mike.

1:14:51
it was like it is W g one Chapter three, I think.

1:14:55
Yes, Yeah, yeah, it's W right.

1:14:58
So that's, , the physical basis of climate change or something.

1:15:04
human influence on the climate team.

1:15:06
OK, OK, so you will, you will make a list of terms that you think are related.

1:15:16
Yes.

1:15:17
She's asking about adding to the glossary and the term.

1:15:20
Yes, I know, but we get those terms, some of them will be in, , Simon's dictionary.

1:15:30
And some of them will be in the dictionaries that we create, right?

1:15:33
So the first thing to do is to make a list of terms.

1:15:37
Simon's dictionary, , has already done that.

1:15:41
and then where?

1:15:44
As soon as we have an entry either in, , the IPCC format or in our own dictionary format, we will create a synonym, , and that synonym will have a language on it.

1:15:59
And you will put the language in, and that's what you saw today.

1:16:05
So as soon as we've got a dictionary with the opportunity to add synonyms, we can put it in there, So does that make sense?

1:16:17
Yes.

1:16:18
Earlier we were adding like manually, we were adding some things, but this time we are adding automatically.

1:16:26
it may be that we can, we're not going to put in necessarily every term in every language, we will choose those languages which we have some flu in.

1:16:37
OK, OK.

1:16:40
right.

1:16:41
So now I'm going to stop the recording, Hm?

[petermr]

cleaned transcript

transcript1-save.txt