Introducing InTRePIDs – In-Text Reference Pointer Identifiers


Readers of this blog will be familiar with Open Citation Identifiers (OCIs), described in an earlier post and formally defined in [1]. OCIs enable bibliographic citations, treated as first class information entities, to be uniquely identified and referenced, and are used to identify the >624 million individual citations indexed in the latest release of COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations, as described in a recent post.

However, COCI and similar citation indexes do not provide any information about where within the citing paper a citation is generated, the textual contexts of the in-text reference pointers, or the reasons for including different in-text reference pointers denoting the same reference at different points within the text.

As explained in the preceding post describing the Open Biomedical Citations in Context Corpus funded by the Wellcome Trust and under development by OpenCitations, deep citation analysis requires a more nuanced approach to citations, which acknowledges that each in-text reference pointer that denotes a bibliographic reference in the reference list of a citing publication instantiates its own citation, as shown in Figure 1.

Figure 1. Citations between a citing paper and a cited paper instantiated both by the inclusion of a bibliographic reference within the reference list of the citing paper and by the inclusion within the text of the citing paper of one or more in-text reference pointers denoting that reference.

The pointer citations clearly involve the same cited publication as does the reference citation itself, but each has its own unique characteristics: the location and textual context of its in-text reference pointer within the text of the citing publication, and its particular rhetorical function which is determined by that context.

If the reference citation is open (as defined in [2]) and identified by an OCI, each in-text reference pointer related to that citation can be identified uniquely using an In-Text Reference Pointer Identifier (InTRePID).

InTRePIDs facilitate in-depth scholarship on in-text reference pointer locations and citation functions, and fine-grained analysis of the relationships between publications, by making it possible

  • to identify each in-text reference pointer with a unique PID,
  • to distinguish references that are cited only once from those that are cited multiple times,
  • to see which references are cited together (e.g. in the same sentence or within an in-text reference pointer list),
  • to determine from which section(s) of the article references are cited (e.g. Introduction, Methods, Discussion), and, potentially,
  • to determine the rhetorical function of the citations from analysis of their textual contexts, by the application of natural language processing, machine learning and artificial intelligence techniques to conduct sentiment analysis on the citation contexts.

Definition of an InTRePID

An InTRePID is composed of two parts separated by an oblique stroke



  • <oci-numerals> is the numerical part of the OCI uniquely identifying the particular open citation to which the in-text reference pointer and its denoted bibliographic reference relate. Thus an InTRePID can be assigned for any in-text reference pointer that relates to an open citation for which a valid OCI has been assigned;
  • <ordinal> identifies the nth occurrence of an in-text reference pointer within the text of the citing paper relating to that citation; and
  • <total> defines the total number of in-text reference pointers denoting that bibliographic reference within the citing paper.

For example, intrepid:070433-070475/46 is a valid InTRePID for an in-text reference pointer defined within the OpenCitations Citations in Context Corpus.

A formal definition document for the InTRePID is given in [3].

Exemplar in-text reference pointers

Consider the following citing paper:

Zou, J. et al. (2020). Phenotypic and genotypic correlates of penicillin susceptibility in nontoxigenic Corynebacterium diphtheriae, British Columbia, Canada, 2015–2018. Emerging Infectious Diseases, 26: 97-103.

This paper contains six in-text reference pointers denoting Reference 13 in the reference list:

13. Lowe, C. et al. (2011). Cutaneous diphtheria in the urban poor population of Vancouver, British Columbia, Canada: a 10-year review. J. Clinical Microbiology 49: 2664-2666.

The InTRePIDs for these pointers are recorded within the OpenCitations Biomedical Citations in Context Corpus, together with the corpus identifiers and DOIs of the citing and cited papers, as shown in the excerpt presented in Figure 2.

Figure 2. An excerpt from the OpenCitations Biomedical Citations in Context Corpus, showing highlighted the InTRePIDs for the six in-text reference pointers within Zou, J. et al. (2020) denoting Reference 13, the reference to Lowe, C. et al. (2011), together with the internal corpus identifiers for each in-text reference pointer, and the corpus identifiers and DOIs for the citing and cited papers.

Of these six in-text reference pointers, having InTRePIDs intrepid:070433-070475/1-6 to intrepid:070433-070475/6-6, the first and the fourth of these, together with their document locations, their embedding sentences, their in-text reference pointer lists, and their InTRePIDs, chosen as examples, are as follows:

Introduction. “Nontoxigenic strains have been shown to have epidemic potential, causing infections in persons afflicted by homelessness, alcohol abuse, and injection drug use (9,13–15).” (intrepid:070433-070475/1-6)

Discussion. “We also noted ST5 and ST32 in our review from downtown Vancouver during 1998–2007 (13).” (intrepid:070433-070475/4-6)

The first of these discusses those people most susceptible to diphtheria infection, while the other discusses which multilocus sequence types (STs) of C. diphtheriae were found, thus relating to the organism causing the infection rather than to the infected individuals. The rhetorical function of these two in-text reference pointers is quite distinct.

To permit this information to be recorded within the OpenCitations Citations in Context Corpus, extensions were required to the OpenCitations Data Model, a new extended version of which was recently published [4], as described in a related blog post.

The OpenCitations InTRePID Resolution Service

To support the use of InTRePIDs to identify in-text reference pointers, OpenCitations has recently developed an InTRePID Resolution Service (currently in ‘beta’ in its development cycle), which is running at A screenshot of this service is shown in Figure 3.

Figure 3. A screenshot of the user interface of the InTRePID Resolution Service.

In addition to using the Web user interface shown in Figure 3, InTRePIDs can be entered into this resolution service in the form of resolvable URIs, e.g.

As shown in Figure 4, the OpenCitations InTRePID Resolution service returns metadata concerning the in-text reference pointer identified by the InTRePID, and the bibliographic reference that it denotes, from which further information about the citation and the citing and cited publications may be obtained by following the links provided.

Figure 4. A screenshot of the Web page displaying metadata returned by the InTRePID Resolution Service.

Note that as well as rendering this information in HTML on a web page, the resolution service can also provide it in a variety of machine-readable formats.


InTRePIDs, which enable the identification of individual in-text reference pointers, and the InTRePID Resolution Service, are new services from OpenCitations that will facilitate scholarship on the textual contexts and rhetorical functions of such in-text reference pointers, and of the citations that they instantiate.

InTRePIDs were first announced on 30th January 2020 at PIDapalooza 2020 in Lisbon, the Open Festival of Persistent Identifiers.


[1] Silvio Peroni and David Shotton (2019): Open Citation Identifier: Definition. Figshare.

[2] Silvio Peroni and David Shotton (2018). Open Citation: Definition. Figshare.

[3] David Shotton, Marilena Daquino and Silvio Peroni (2020). In-Text Reference Pointer Identifier: Definition. Figshare.

[4] Marilena Daquino, Silvio Peroni and David Shotton (2019). The OpenCitations Data Model. Version 2.0. Figshare.

Early adopters of the OpenCitations Data Model

OpenCitations is very pleased to announce its collaboration with four new scholarly Research and Development projects that are early adopters of the recently updated OpenCitations Data Model, described in this blog post.

The four projects are similar, in that they each are independently using text mining and optical character recognition or PDF extraction techniques to extract citation information from the reference lists of published works, and are making these citations available as Linked Open Data. Three of the four will also use the OpenCitations Corpus as publication platform for their citation data.  The academic disciplines from which these citation data are being extracted are social science, humanities and economics.

1     Linked Open Citation Database (LOC-DB)

The Linked Open Citation Database, with partners in Mannheim, Stuttgart, Kiel, and Kaiserslautern (LOC-DB,, is the first of two German projects funded by the Deutsche Forschungsgemeinschaft (DFG) that are extracting citations from Social Science publications.  Dr. Annette Klein, Deputy Director of the Mannheim University Library, is the project manager.

The project is using Deep Neural Networks based approaches for reference detection and state-of-the-art methods for information extraction and semantic labelling of reference lists from electronic and print media with arbitrary layouts [3].  The raw data obtained will be manually checked against and linked with existing bibliographic metadata sources in an editorial system.  They will then be structured in RDF using the OpenCitations Data Model, and published in the Linked Open Citations Database under a CC0 waiver. Using its libraries’ own Social Science print holdings and licensed electronic journals as subject material, this project will demonstrate how these citation extraction processes can be applied to the holdings of individual academic libraries, and can be integrated with library catalogues [1, 2, 3].


[1]       Kai Eckert, Anne Lauscher and Akansha Bhardwaj (2017) LOC-DB: A Linked Open Citation Database provided by Libraries. Motivation and Challenges.  EXCITE Workshop 2017: “Challenges in Extracting and Managing References”.

[2]      Lauscher, Anne; Eckert, Kai; Galke, Lukas; Scherp, Ansgar; Rizvi, Syed Tahseen Raza; Ahmed, Sheraz; Dengel, Andreas; Zumstein, Philipp; Klein, Annette  (2018) Linked Open Citation Database: Enabling libraries to contribute to an open and interconnected citation graph. Accepted for the JCDL 2018: Joint Conference on Digital Libraries 2018, June 3-6, 2018 in Fort Worth, Texas [Preprint of the conference publication].

[3]       Bhardwaj A., Mercier D., Dengel A., Ahmed S. (2017). DeepBIBX: deep learning for image based bibliographic data extraction. In: Liu D., Xie S., Li Y., Zhao D., El-Alfy ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science, vol 10635. Springer, Cham [Conference publication].

2     The EXCITE (Extraction of Citations from PDF Documents) Project

The EXCITE Project (, run jointly at the University of Koblenz-Landau and GESIS (Leibniz Institute for Social Sciences), is the second project funded by the Deutsche Forschungsgemeinschaft (DFG) that is extracting citations from Social Science publications.  It is headed by Steffen Staab, head of the Institute for Web Science and Technologies at the University of Koblenz-Landau, and Philipp Mayr of GESIS.

Since the social sciences are given only marginal coverage in the main bibliographic databases, this project aims to make more citation data available to researchers, with a particular focus on the German language social sciences.  It has developed a set of algorithms for the extraction of reference information from PDF documents and for matching the reference entry strings thus obtained against bibliographic databases (see EXCITE git  It is using as its data sources the following Social Science collections: full texts from SSOAR, the Gesis Social Science Open Access Repository ( and scattered pdf stocks from other social science collections including SOLIS, Springer Online Journals and CSA Sociological Abstracts [4, 5].

The EXCITE project organized an international developer and researcher workshop “Challenges in Extracting and Managing References” in March 2017 in Cologne.

EXCITE will then structure the extracted bibliographic and citation data in RDF using the OpenCitations Data Model, and will use the OpenCitations Corpus as its publication platform, employing the OCC EXCITE supplier prefix 0110, described here, to identify the provenance of these citations.


[4]       Martin Körner (2016). Extraction from social science research papers using conditional random fields and distant supervision, Master’s Thesis, University of Koblenz-Landau, 2016.

[5]       Körner, M., Ghavimi, B., Mayr, P., Hartmann, H., & Staab, S. (2017). Evaluating reference string extraction using line-based conditional random fields: a case study with german language publications. In M. Kirikova, K. Nørvåg, G. A. Papadopoulos, J. Gamper, R. Wrembel, J. Darmont, & S. Rizzi (Hrsg.), New Trends in Databases and Information Systems (Bd. 767, S. 137–145). Springer International Publishing.   Preprint:

3    The Venice Scholar Index

The Venice Scholar Index is a citation index of literature on the history of Venice, indexing nearly 3000 volumes of scholarship from the mid 19th century to 2013, from which some 4 million bibliographic references have been extracted.

The Venice Scholar Index is the first prototype resulting from Linked Books Project (, a project spearheaded by Giovanni Colavizza and Matteo Romanello of the Digital Humanities Laboratory at EPFL (École Polytechnique Fédérale de Lausanne), with partners in Venice, Milan and Rome.

The project is exploring the history of Venice through references to scholarly literature as well as archival documents found within publications.  To achieve this goal, the project has developed a system to automatically extract bibliographic references found within a large set of digitized books and journals, which has then been applied to the publications on the history of Venice, its main use case [6].

The Linked Books Project is specifically interested in analysing the interplay between citations to primary (e.g. archival) documents and those to secondary sources (scholarly literature), and the citation profiles of publications through time.  To this end, it developed the Venice Scholar Index, a rich search interface to navigate through the resulting network of citations, with the final aim of interlinking digital archives and digital libraries.

The citation data underlying the Venice Scholar Index are modelled using the OpenCitations Data Model, and will use the OpenCitations Corpus as its publication platform, using the OCC Venice Scholar Index supplier prefix 0120 to identify the provenance of these citations.


[6] Giovanni Colavizza, Matteo Romanello, and Frédéric Kaplan (2017). The references of references: a method to enrich humanities library catalogs with citation data. In International Journal on Digital Libraries 18 (March 8, 2017): 1–11.

4    CitEcCyr – Citations in Economics published in CyrillicCitEcCyr  ( is an open repository of citation relationships obtained from research papers in the Russian language and Cyrillic script from Socionet ( and RePEc ( [7, 8].  The CitEcCyr project is headed by Oxana Medvedeva, is technically led by Sergey Parinov, and is funded by RANEPA (, the Russian Presidential Academy of National Economy and Public. CitEcCyr is also developing a suite of open software for the citation content analysis of these papers.  This project intends to model its citations using the OpenCitations Data Model, and will use the OpenCitations Corpus as its publication platform, using the OCC CitEcCyr supplier prefix 0140 to identify the provenance of these citations.

However, since this is the first project from which OpenCitations will be importing bibliographic metadata and citations in a language other than English and in a script other than the Latin script, we at OpenCitations are going to have to crawl out of our comfortable ‘Western’ shells and learn to handle foreign languages and scripts other than Latin scripts.

For Russian language papers written using Cyrillic script, we at OpenCitations will to decide how best to handle Russian language written using Cyrillic script, Cyrillic script transliterated into Latin script, and Russian language translated into English and rendered using Latin script.  In particular, since in the OpenCitations Corpus our reference entry records are the uncorrected literal texts of the references in the reference lists of the citing papers, these will need to be recorded as given in Cyrillic.

We will need to develop a policy for when to provide Latin script translations of (for example) titles and abstracts, if these are not provided by the data supplier.  To facilitate use of the OpenCitations Corpus by Russian scholars, we will also need to modify the OpenCitations web site, so as to render the static information displayed in the web pages in the language and script appropriate to the language setting on the user’s web browser.

Unfortunately, all this will take time, so we do not anticipate publishing citation data from the CitEcCyr project within OCC any time soon.  However, this collaboration will be of tremendous value to OpenCitations as well as to CitEcCyr, since the lessons learned by our collaboration with the CitEcCyr project will enable the OpenCitations Corpus to handle citation data not just in Russian, but also in Arabic, Chinese, Japanese and other languages where the Latin script is not used, something that is not found in other major bibliographic databases.

Watch this space!


[7]       Jose Manuel Barrueco, Thomas Krichel, Sergey Parinov, Victor Lyapunov, Oxana Medvedeva and Varvara Sergeeva (2017).  Towards open data for the citation content analysis.

[8]       Thomas Krichel (2017). CitEc to CitEcCyr – A stab at distributed citation systems.  Presented at the 2017 EXCITE workshop.

Oxford University Press opens its references!

Good news!  Today, on January 16th 2018, Oxford University Press (OUP) announced its participation in the Initiative for Open Citations, and requested Crossref to turn on reference sharing for all OUP deposited references from more than half a million publications.  Oxford University Press is the largest university press in the world, publishing in 70 languages and 190 countries.

OUP logo

Their announcement is at

OUP now joins the elite band of four university presses that have already made their references open at Crossref in response to the I4OC call (

This decision by OUP has been a long time in gestation – see my 2012 post Oxford University Press to support Open Citations – but is no less welcome for that!

Like a kid with a new train set! Exploring citation networks

As part of the Open Citations Project, Alex Dutton recently completed a graphing plug-in for the Open Citations web site, that permits users to generate different kinds of graphs of citation networks by querying the Open Citation Corpus for a particular article, and either display the network of papers citing that article (input citations), papers cited by that article (output citations), or both.  These can be displayed on screen in the web browser in a variety of layouts, or conveniently downloaded in a number of useful formats.


Having survived the preparation and posting of the JISC Open Citations Project Final Blog Post last night, minutes before the midnight deadline, I’m now like a kid with a new train set, playing with this display tool and exploring the citation networks present in the Open Citation Corpus, something I have dreamed of doing for two years now.

Remember first that in the Open Citations Corpus we have some 200,000 citing articles – those within the Open Access Subset (OASS) of Pubmed Central – citing ~3.4 million papers out there in the big wide world, which are only recipients of citations.  The consequence of this limited corpus is that the majority of citation chains are of length one – from a paper in the OASS to a paper outside the OASS.  Not very interesting.  Add to this the fact that PubMed Central is new – over 90% of the papers in the Open Access Subset were published in the 21st Century, and 77% of them in the last 5 years.  Thus there are only a very few citation from articles within the OASS to other articles within the OASS.  That means that the maximum length of our citation chains, at present, to three or four on links the input side – a selected article may be cited by a chain of three or four other OASS articles, and three or four on the output side – the selected article may cite other OASS articles in addition to non-OASS articles, and these in turn will cite others.  However, in most cases, the citations chains are much shorter.

simple network
simple networkFigure 1. A simple citation network of input citation chain length of 2 links within the Open Citations Corpus, and an output chain length of 1 link – the selected article (red) receives citations from other OASS articles (green), and itself cites only articles outside the OASS (white).

Let’s start with something familiar – the article in PLoS Neglected Tropical Diseases by Reis et al. (2008) [1] that I used for our semantic publishing exemplar [2].  Its inward citation graph, limited to a citation chain length of two links, created by and copied from the Open Citations Project web site, looks like this:

input citations of Reis
Figure 2. The input citation network of Reis et al. (2008), limited to an citation chain length of 2 links.

I, of course, cited the Reis et al. (2008) paper [1] in our 2009 Adventures paper [3] that we based upon it, and also in my first paper on CiTO in 2010 [3], which also cites the Adventures paper.   Reis et al. (2008) is also cited by Fink et al. (2010) [4], who also cited our Adventures paper, and by Bourhy et al. (2010) [5], another PLoS Neglected Tropical Diseases paper in the OASS, which in turn is cited by Galloway and Levett (2010) [6], while our Adventures paper is also cited by Gerner and Nenadic (2010) [7].

The following image shows this graph as it was originally created within the Open Citations web page:

Input citations of Reis in web page
Figure 3. The same input citation network of Reis et al. (2008), as shown in the Open Citations web site.

Since Reis et al. (2008) has a reference list containing 52 references, its output citation graph is much more complex, even when limited to a citation chain length of 2, since several of its cited papers are also members of the Open Access Subset.  The following figure shows the whole output citation network a citation chain length of 2, which is too demagnified to be legible.

Citations by Reis
Figure 4. The outward citation network of Reis et al. (2008), limited to a citation chain length of 2 links.

The next figure shows a close-up of part of the previous diagram – the output citation network of Reis et al. (2008), again showing the Reis et al. (2008) paper in red, and one of the key papers it cites, Maciel et al. (2010) [8], a slightly earlier paper from the same research group, forming a second key node in the top right of the diagram.

Cited by Reis closeup
Figure 5. A close-up of a central portion of the outward citation network of Reis et al. (2008), limited to a citation chain length of 2 links.

Clearly, there is lots of information that can be extracted from these graphs, particularly when we display them in a tool like GraphViz that permits interactions with the data.  While the Open Citations web site simply displays such citation graphs created using one of several layout algorithms selected by the user, the raw data can also be downloaded in a variety of formats including GraphViz, GraphML and SVG, while the resulting network images can be downloaded in as PNG, JPEG and PDF images, and the underlying RDF metadata can be downloaded as RDF/XML. N-triples, Notation3 and Turtle.

Having used our new Open Citations web site and its network display interface for a short while, I am already aware of many shortcomings and limitations that we will attempt to improve upon in the next few days.  However, we would very much like to hear from you – as a user of the Open Citations web site – both to learn what you like about what we have done and to hear what you find to be shortcomings of the functionality and new features that you would like to see implemented, which we will record as user stories to input into our next round of development.  These can either be recorded as comments on this blog post, or can be e-mailed with the subject line “Open Citations web site” either to me <> or to Alex Dutton <>, who is the person who deserves all the credit for the present system.  We look forward to hearing from you.

[1]  Reis RB, Ribeiro GS, Felzemburgh RDM, Santana FS, Mohr S, Melendez SXTO, Queiroz A, Santos AC, Ravines RR, Tassinari WS, Carvalho MS, Reis MG, Ko AI (2008). Impact of environment and social gradient on Leptospira infection in urban slums. PLoS Negl Trop Dis 2(4): e228. doi:10.1371/journal.pntd.0000228.

[2] Shotton D, Portwin K, Klyne G, Miles A (2009). Adventures in semantic publishing: exemplar semantic enhancements of a research article. PLoS Comput Biol 5:e1000361. doi:10.1371/journal.pcbi.1000361.

[3] Shotton D (2010). CiTO, the Citation Typing Ontology. Journal of Biomedical Semantics  1 (Suppl. 1): S6. doi:10.1186/2041-1480-1-S1-S6.

[4]  Fink JL, Fernicola P, Chandran R, Parastatidis S, Wade A, Naim O, Quinn GB, Bourne PE (2010). Word add-in for ontology recognition: semantic enrichment of scientific literature.  BMC Bioinformatics 11:103. doi:10.1186/1471-2105-11-103.

[5]  Bourhy P, Collet L, Clément S, Huerre M, Ave P, Giry C, Pettinelli F, Picardeau M (2010). Isolation and Characterization of New Leptospira Genotypes from Patients in Mayotte (Indian Ocean). PLoS Negl Trop Dis 4(6): e724. doi:10.1371/journal.pntd.0000724.

[6]  Galloway RL, Levett PN (2010) Application and Validation of PFGE for Serovar Identification of Leptospira Clinical Isolates. PLoS Negl Trop Dis 4(9): e824. doi:10.1371/journal.pntd.0000824.

[7]  Gerner M, Nenadic G (2010). LINNAEUS: A species name identification system for biomedical literature. BMC Bioinformatics 11:85. doi:10.1186/1471-2105-11-85.

[8]  Maciel EAP, Carvalho ALF, Nascimento SF, Matos RB, Gouveia EL, Reis MG, Ko AI (2008). Household transmission of Leptospira infection in urban slum communities. PLoS Negl Trop Dis 2: e154. doi:10.1371/journal.pntd.0000154.

Enhanced by Zemanta

JISC Open Citations Project – Final Project Blog Post

Executive summary


To general readers of this blog, this post will appear different from normal posts. Rather than being about a particular topic, it pulls together a summary of the work undertaken over the past year within the Open Citations Project supported by the JISC, and is primarily intended to assist JISC evaluation of the project and its outputs. Details of the work undertaken and the outputs from this project have mostly been described in previous blog posts, to which this post will frequently refer.

Project scope and purpose

The Open Citations Project is global in scope, designed to change the face of scientific publishing and scholarly communication. Specifically, it aims to make it possible to publish bibliographic information in RDF and to make citation links as easy to traverse as Web links.

Project aims

To achieve this goal, we have had four primary aims:

  • To create a semantic infrastructure that makes possible the description of citations, references and bibliographic entities in RDF, since we found existing ontologies inadequate for our purpose.
  • To extend that semantic infrastructure to handle data citations and data entities, as well as bibliographic citations and bibliographic entities, mindful of Philip Bourne’s prediction that soon there will be no meaningful difference between a journal article and a database entry.
  • To provide exemplars of how these ontologies can be applied to real-world data, by creating mappings from existing encodings to RDF, and by creating
    RDF metadata relating to bibliographic and data entities and their citations.
  • To convert the reference lists within all the PMC Open Access subset articles to RDF, and their publication as open linked data that third parties can use in novel ways.

Principle deliverables and outputs

  • The SPAR (Semantic Publishing and Referencing) Ontologies.
  • Graffoo and LODE, two novel tools for ontology visualization and documentation.
  • Mappings of various existing metadata schemes to RDF using SPAR.
  • Development of data citation methods and protocols
  • The Open Citations Blog in which activities and outputs are described.
  • The Open Citations Corpus of bibliographic citation data encoded in RDF and published as Open Linked Data.
  • The web site, to provide user access to the Open Citations Corpus.
  •     The Open Citations Project softwareused for processing the Pubmed Central Open Access corpus into Open Linked Data.The net result is open citation data from life science journal articles available on the web, for utilization by academics, for citation network analysis applications, and for tracking the impact of research grant funding.

Primary beneficiaries

  • Scholars worldwide, particularly in the biomedical sciences, by providing better access to bibliographic and citation data.
  • Academic publishers and repository managers, by providing a semantic infrastructure and tools to enable their outputs and holdings to join the semantic web of open linked data.


In 2008, Katie Portwin and I had an enjoyable summer ‘souping up’ a PLoS Neglected Tropical Diseases article by Reis et al. (2008) [1] that I had downloaded as an XML file from the journal web site in late April, one week after it had been published. The resulting enhanced publication, available here, became an exemplar of what is possible in the realm of semantic publishing, while undertaking that work was very influential in shaping the course of my more recent activities.

One of the things we undertook was to mark up the reference list with annotations that clarified the nature of the cited entity (e.g. book, journal article, medical report) and the reason the authors had cited those entities (used data from, obtained background from, extended, etc.) – annotation that we took care to verify with the authors themselves before publishing them!

While we undertook that work manually, it quickly became apparent that what we needed was an ontology from which a controlled vocabulary of such terms could be used to create both human- and machine-readable metadata describing the citations and the cited entities. We therefore developed a draft ontology that we subsequently split to form the basis of the first two ontologies of the suite of SPAR (Semantic Publishing and Referencing) Ontologies described elsewhere on this blog, namely CiTO, the Citation Typing Ontology to describe the relationships between the citing and cited entities, and FaBiO, the FRBR-aligned Bibliographic Ontology, to describe the cited entities themselves.

Using these tools, we were able to mark up the reference list from Reis et al., and publish it as RDF.

From there it was but a small step to dream of the day when the references from all biomedical research articles would be published as open linked data, and to think what we could do to make that dream a reality.

And it was obvious where to start – with the Open Access subset of journal articles available in PubMed Central (PMC), all nicely marked up in XML using the National Library of Medicine DTD.

Table of contents

The various aspects of the Open Citations Project and its outputs are described in the following blog posts, complete with diagrams, data tables, and screen shots where appropriate. These are organized into the following set of distinct topics:

  • Standards
  • The SPAR ontologies for bibliographic and data entities and their citations
  • Graffoo and LODE – tools for ontology visualization and documentation
  • Third-party applications of our ontologies
  • Mappings to the SPAR ontologies, and exemplar RDF encodings
  • Development of data citation methods and protocols
  • The creation of the Open Citation Corpus of linked bibliographic citation data

1 Standards

Advantages of Ontological Standards in Scholarly Publishing

Nomenclature for citations and references

2 The SPAR ontologies for categorizing bibliographic and data entities and their citations

The SPAR ontologies described in the following blog posts were developed jointly with Silvio Peroni, a brilliant graduate student from the University of Bologna, who spent the last six months of 2010 working with me as an intern in Oxford, where he became an honorary member of the Open Citations Project, contributing very significantly to our achievements. His supervisor Fabio Vitali and the Department of Information Science at the University of Bologna are to be congratulated and thanked for their enlightened requirement that all their graduate students spend an internship overseas, since without his collaboration and great skill, much of this development would not have been possible within the available time, if at all.

Introducing the Semantic Publishing and Referencing (SPAR) Ontologies

New web site for the SPAR ontologies

Functional clustering of CiTO properties

Extending FRBR within FaBiO

Categorising bibliographic resources with FaBiO and SKOS

CiTO4Data – a new data-centric citation typing ontology

Using FaBiO to describe data entities

3 Graffoo and LODE – tools for ontology visualization and documentation

These tools have been developed by Silvio Peroni, an honorary member of the Open Citations Project, as explained above.

Graffoo, a Graphical Framework for OWL Ontologies

Using LODE for ontology visualization

4  Third-party applications of our ontologies

Our work to develop a standard semantic infrastructure for bibliographic and data entities and their citation is new. Nevertheless, we have received encouraging responses when we have presented our work at international publishing venues such as the 2010 ALPSP Conference and the 2010 STM Innovation Conference.

Apart from local applications at the University of Oxford, the University of Bologna, and the University of Manchester (for the Utopia Project), and adoption of the SPAR ontologies by the University of Harvard both to complement SWAN (Semantic Web Applications in Neuromedicine) and to mark up astrophysics data (Accomazzi and Dave (2011) Semantic Interlinking of Resources in the Virtual Observatory Era. arXiv:1103.5958), we have expressions of interest from PLoS, Nature and Il Mulino, a major academic publisher in Italy, who are looking to improve their metadata encoding as RDF. We are also interacting with the British Library in mapping the DataCite Metadata Kernel to RDF (see below), and with the Dryad Data Repository in creating RDF mappings of Dryad metadata to RDF and, as part of the JISC Dryad-UK Project, in developing MIIDI and MIIDI-structured RDF metadata for infectious disease papers and datasets, using SPAR ontologies where appropriate, with the aim of permitting authors to submit rich metadata to Dryad.

The following blog posts describe uptake and use of CiTO in CiteULike and WordPress.

Use of CiTO in CiteULike

How to employ CiTO in CiteULike

Using CiTO in WordPress

5 Mappings to the SPAR ontologies, and exemplar RDF encodings

Comparison of BIBO and FaBiO

BIBO2SPAR, an RDF Mapping of BIBO to the SPAR Ontologies

DataCite2RDF – Mapping DataCite Metadata Scheme Terms to ontologies

6 Development of data citation methods and protocols

Nomenclature for data publications and citations

Questions of granularity – Dryad’s use of DataCite DOIs for data citation

How to cite data

Pensoft Journals policy and author guidelines on data publication and citation

7 The creation of the Open Citation Corpus of linked bibliographic citation data

This achievement is almost entirely as the result of the excellent work of our chief data wrangler Alex Dutton, whose skill and natural feel for linked data has done wonders for this project.

The following set of blog posts describe the starting corpus from PubMed central, our transformation of it to RDF, the problems we encountered along the way, the resulting Open Citations Corpus, and the potential uses to which the resulting open citation data can now be put.

Input data for Open Citations – the PMC Open Access Subset

Garbage in, garbage out – problems with bibliographic references

Who wrote this paper? Author list problems in PubMed Central references

Citation correction methods

The citation processing pipeline and the Open Citations Corpus

JISC Open Citations Project web site

Like a kid with a new train set! Exploring citation networks

JISC Administrative Data for Open Citations Project

This information, extracted from the JISC Expo DOAP (Description of a Project) spreadsheet,  is to be found in a separate blog post, here.

The Future

While this is the formal Final Blog Post for the JISC-funded Open Citations Project, that was funded for a year from 1st July 2010, our work is not yet finished. We cherish grand ideas for the liberation of the reference lists from all scholarly journal articles, using the Open Citations Corpus as an exemplar, in collaboration with publishers and organizations such as CrossRef who handle such citation data on behalf of publishers on a daily basis.

This work will only be finished when it is longer up to an individual academic research group to take on the task of citation liberation, but when each publisher publishes the citation data from each of their journal articles as open linked data on their own web sites, marked up using agreed ontological standards that we have proposed, freely available for scholar around the world, from Bangladesh to Zimbabwe, and from Holland to New Zealand, to use and explore, independent of their ability to afford subscription access to the journal articles from which the citations are made.

Nomenclature for citations and references

Reis et al. (2008) [1] cites an earlier paper from Albert Ko’s research group, Ko et al. (1999) [2].

In conventional parlance, as the following diagram shows, the word “reference” can mean either what is found in the text, what is found in the reference list, the act of citation, or the object of the citation itself, as in the sentence “All the references you will need to prepare for the journal club are on Kevin’s desk”.

This situation does not make for unambiguous machine-readable encoding, so to improve the situation we have, in the SPAR (Semantic Publishing and Referencing) Ontologies, introduced a more principled way of referring to these items and actions, as the second diagram illustrates:

This permits us to create RDF describing all aspect of the citation process. Within the body of the text we have an in-text citation containing an in-text reference pointer, while the actual bibliographic reference that the in-text reference pointer denotes is to be found within the article’s reference list. That bibliographic reference references the cited article, while the whole performative act of including the reference in the article constitutes the act of citation.

Are we all clear now?

[1] Reis RB, Ribeiro GS, Felzemburgh RDM, Santana FS, Mohr S, Melendez AXTO, Queiroz A, Santos AC, Ravines RR, Tassinari WS, Carvalho MS, Reis MG, Ko AI (2008) Impact of Environment and Social Gradient on Leptospira Infection in Urban Slums. PLoS Negl Trop Dis 2(4): e228. doi:10.1371/journal.pntd.0000228

[2]    Ko AI, Reis MG, Ribeiro Dourado CM, Johnson WD Jr, Riley LW and the Salvador Leptospirosis Study Group. (1999) Urban epidemic of severe leptospirosis in Brazil. Lancet 354: 820–825. doi:10.1016/S0140-6736%2899%2980012-9.

The citation processing pipeline and the Open Citations Corpus

The input PubMed Central Open Access subset XML reference data, our starting corpus, were transformed into Open Citations RDF in multiple stages:

  1. The original XML was first transformed into an intermediate form using XSLT. The multitudinous ways different publishers have developed of encoding the same information can be more easily handled in this way, by generating an intermediate XML output dataset in which things are described in a more consistent manner, and enabled the resulting information to be parsed more easily from within a non-XML-based programming environment. Our transform pulled out information about articles, people, organisations, in-text reference pointers, and the reference list, and the links between them.
  2. The intermediate XML dataset was then transformed into BibJSON using a Python script. BibJSON is a relatively standard method of encoding bibliographic information. Each of the ~200,000 generated BibJSON dataset contains the information extracted from one marked-up Open Access article. We extended the standard BibJSON records with additional attributes (named with an ‘x-‘ prefix) for other properties we wish later to encode as RDF. At this and later stages, the BibJSON datasets are packed into a single gzipped tarball. Since it would be unwise to unpack such a tarball into ~200,000 independent files would give data management problems, the contents are extracted from the tarball as required using the Python tarfile module.
  3. Another Python script was then used to extract all the PubMed IDs, and to use these as inputs to the Entrez API, in order to extract independent information about the cited entities from the PubMed database. The returned PubMed records were then added alongside the original BibJSON records. These additional data were extremely useful for comparison when attempting to spot erroneous citations, as previously described.
  4. Next we ran a ‘sanitization script’ over the data, which performed the following functions:
  1. URL normalization (e.g. adding URL schemes, undoing character substitutions (e.g. en-dashes for hyphens, quotation marks for apostrophes).
  2. Splitting issue information from journal attributes.
  3. Fixing malformed DOIs (e.g. those missing the ’10.’ prefix). Where DOIs could not be fixed they were removed.
  4. Pulling “doi:****” DOIs out of “**** URLs.
  5. Removing spurious publication dates (those before 1900 and after 2011).

These corrections are easily extensible if we discover other classes of error in the data.

  1. The records were next unified by taking the transitive closure on a number of identifiers. These identifiers included DOIs, PubMed IDs, PubMed Central IDs and URLs for articles and other cited works, and ISSNs, eISSNs and ISO title abbreviations for journals.
  2. The BibJSON data were then rearranged so that each dataset contains multiple records believed to reference the same bibliographic entity, if it had multiple citations.
  3. Owing to mis-citation (in this case, the use of incorrect or incomplete identifiers) there were a number of clearly different that had been mistakenly declared to refer to the same entity. For this reason we use a distance metric to recluster record groups based on similarity.
  4. Finally, a Python script transforms the BibJSON tarball into RDF. The input tarball contains datasets, each of which comprises records believed to refer to the same entity. The script takes each of these datasets and merges them into a single ‘best’ record using the majority vote procedure previously described. The resultant record is then transformed into a number of quads for inclusion in the final RDF N-Quads Open Citations Corpus, principally modelled using the suite of SPAR ontologies created for this purpose.

This Open Citations Corpus of rdf citation data extracted from the open access subset of PubMed Central, detailing every reference list in the OASS articles, holds each reference list as an individual named graph (hence the storage in N-Quads rather than triples), and comprises 236,499,781 quads occupying 2.1 gigabytes of storage in its compressed state. It includes references to ~20% of all post-1980 papers recorded in PubMed, including all the highly cited papers in every field of biomedical research, and is freely available under a CC0 waiver from

The Open Citations Corpus can be queried via a Web query form or via a SPARQL interface from the Open Citations Project web site at, described in a subsequent blog post, where more information about the project is given.

All the scripts used to transform the OASS input data into the Open Citations Corpus, described above, are available under an MIT Open Source licence at

Citation correction methods

As previously described, the PubMed Central Open Access subset of journal articles yielded 6,529,815 independent bibliographic records of both citing and cited entities, while our use of the PubMed Entrez API provided a further 2,304,143 bibliographic records for the same cited entities. Before converting these references into RDF to create the Open Citations Corpust, we attempted to remove errors in the data.

Some of the references we collected were to highly cited papers, while 2,505,879 referenced papers were only cited once. Figure 1 shows the number of citations per paper for the 100 most highly cited papers in our records – the left hand end of what is a classical long-tail dataset.

 Figure 1

We have not yet analysed the topics of these papers, but can reveal that the paper most highly cited from within the OASS, with 2150 citations, is

Altschul et al. (1997). Gapped BLAST and PSI-BLAST: a new generation of protein database. Nucleic Acids Res. 25(17):3389-3402. doi: 10.1093/nar/25.17.3389.

In an ideal world, all OASS references to an individual paper would be identical, and would exactly match the data on that paper extracted from the Entrez API. However, as we have already seen for author names, this is not the case. As with most datasets in the world, a significant proportion (~1%) of our input reference data is either incomplete or erroneous.

We attempted to correct for these errors by comparing references that appeared to reference the same bibliographic entity, and from this comparison extract the correct data for authors, title, etc. using the following rules:

  1. Accept the longest author list and names bearing accents over those lacking them.
  2. Accept DOIs and PubMed IDs from references having them, after eliminate mis-formed identifiers (e.g. DOIs lacking the journal prefix “10.****”), using the majority vote if different identifiers were given for the same paper.
  3. Accepting those variants of titles, journal names, etc. held in common by the majority of references.

This voting method was weighted in favour of data we judged to be most reliable, namely the PubMed records returned from the Entrez API, and metadata about cited papers that were within the OASS coming from the independent bibliographic records we had for those papers.

As a result of these activities, not only did we coalesce the independent references from different OASS articles to the same multiply cited papers into a set of 3,578,598 unique bibliographic citation target records describing 204,637 OASS articles and 3,373,961 articles outside the OASS, but we were able to select from the multiple references those elements (author list, title, etc.) that were judged to be correct for each target.

However, of the 2,505,879 papers that are only cited once from within the OASS, 1,246,967 lacked a PubMed ID, so for these we were unable to gather confirmatory evidence for the accuracy of the citation from the Entrez API. These references, which are to the least significant papers in the corpus, are therefore provided “as is” from PubMed Central, without any external corroboration of their accuracy.

How these error-correction processes fitted into the data processing pipeline used to create the Open Citations Corpus is described in the next blog post.

Who wrote this paper? Author list problems in PubMed Central references

To illustrate three kinds of problems in obtaining correct author lists for Open Citation data from articles in the PubMed Central Open Access subset (OASS), I take three examples, the first of which is the result of a publication policy, the second due to mis-handling of an authorship attribution at the time of publication, and the third exemplifing errors introduced when handling non-English personal names.

Example 1

In the paper by Reis et al. (2008) [1], that we took as the subject for our exercise in semantic publishing enhancements described in [2], we find the following entry for Reference 40 in the reference list:

40.    Maciel EAP, Carvalho ALF, Nascimento SF, Matos RB, Gouveia EL, et al. (2008) Household transmission of Leptospira infection in urban slum communities. PLoS Negl Trop Dis 2: e154. doi:10.1371/journal.pntd.0000154.

Note that it is the policy of the publisher, the Public Library of Science, to list only the first five authors in references to papers that have more than five, despite publishing online-only journals where article length is not an issue.

The XML for this reference in the document is as follows:

    <meta name="citation_reference" content="citation_title=Household transmission of Leptospira infection in urban slum communities.; citation_author=EAP Maciel; citation_author=ALF Carvalho; citation_author=SF Nascimento; citation_author=RB Matos; citation_author=EL Gouveia; citation_journal_title=PLoS Negl Trop Dis; citation_volume=2; citation_number=40; citation_pages=e154. doi:10.1371/journal.pntd.0000154; citation_date=2008; " /> 

Note that in the XML, all indication that there are more than five authors, i.e. the “et al.” present in the human-readable reference, is totally lost. There is thus no way of telling from the XML for this paper retrieved from PubMed Central that the full authorship for this paper is as follows:

Elves A. P. Maciel, Ana Luiza F. de Carvalho, Simone F. Nascimento, Rosan B. de Matos, Edilane L. Gouveia, Mitermayer G. Reis and Albert I. Ko.

The last two authors of the cited paper, Mitermayer Reis and Albert Ko, who are the lead author and the senior author, respectively, of the citing paper, are both omitted from the PLoS reference to the cited paper, and hence from the data automatically extracted by Open Citations from the OASS.

Example 2

A second example from the same citing paper is the cited reference to Ko et al. (1999) [2]. As the reference at the foot of this page shows, the author list includes six names “and the Salvador Leptospirosis Study Group”. Group attributions of this kind are commonplace, particularly in papers resulting from large collaborative projects. However, conventional markup systems such as the NLM-DTD have no systematic way of handling such information. Surprisingly, it is even incorrectly stated in the human readable version of the reference in the Reis et al. paper. Reference 6 in the article’s reference list reads:

6.    Ko AI, Reis MG, Ribeiro Dourado CM, Johnson WD Jr, Riley LW (1999) Urban epidemic of severe leptospirosis in Brazil. Salvador Leptospirosis Study Group. Lancet 354: 820–825.

thus including “Salvador Leptospirosis Study Group” as part of the title, an error also present in the XML version of the paper:

<meta name="citation_reference" content="citation_title=Urban epidemic of severe leptospirosis in Brazil. Salvador Leptospirosis Study Group.; citation_author=AI Ko; citation_author=MG Reis; citation_author=CM Ribeiro Dourado; citation_author=WD Johnson; citation_author=LW Riley; citation_journal_title=Lancet; citation_volume=354; citation_number=6; citation_pages=820-825; citation_date=1999; " /> 

Example 3

The third example, also taken from the reference list of Reis et al. (2008) [1], illustrated the problems of handling non-English names and titles. Reference 39 in the PLOS article’s reference list [3] reads:

Dias JP, Teixeira MG, Costa MC, Mendes CM, Guimaraes P, et al. (2007) Factors associated with Leptospira sp infection in a large urban center in Northeastern Brazil. Rev Soc Bras Med Trop 40: 499–504.

Note that the author list is truncated, that the generic name “Leptospira sp” in the title is italicized, and that neither the DOI nor the PubMed ID is provided, although the article has both.

The landing page for this article on the publisher’s web site ( shows the following:

Revista da Sociedade Brasileira de Medicina Tropical

Print version ISSN 0037-8682

Rev. Soc. Bras. Med. Trop. vol.40 no.5 Uberaba Oct. 2007

doi: 10.1590/S0037-86822007000500002  ARTIGO ARTICLE

Factors associated with Leptospira sp infection in a large urban center in northeastern Brazil

Fatores associados à infecção por Leptospira sp em um grande centro urbano do Nordeste do Brasil

Juarez Pereira DiasI; Maria Glória TeixeiraI; Maria Conceição Nascimento CostaI; Carlos Maurício Cardeal MendesI; Patrícia GuimarãesII; Mitermayer Galvão ReisII; Albert KoII,III; Maurício Lima BarretoI

The paper is published in English. Note that there is an alternative Portuguese title, that the generic name “Leptospira sp” is italicized in both, and that a DOI is provided, although page numbers are not give for this on-line version of the article. Note also the accents and structures of the full Brazilian author names.

Clicking on the tab Article in PDF format on the landing page takes one to the PDF download page that gives the following reference with page numbers, but lacking English title and DOI, and lacking the full author list:

DIAS, Juarez Pereira et al. Factors associated with Leptospira sp infection in a large urban center in northeastern Brazil. Rev. Soc. Bras. Med. Trop. [online]. 2007, vol.40, n.5, pp. 499-504. ISSN 0037-8682.

A manual search for the same article in PubMed returns the following information:

Rev Soc Bras Med Trop. 2007 Sep-Oct;40(5):499-504.

Factors associated with Leptospira sp infection in a large urban center in northeastern Brazil.

Dias JP, Teixeira MG, Costa MC, Mendes CM, Guimarães P, Reis MG, Ko A, Barreto ML.

PMID: 17992402

Note the correct accentuation of the surname “Guimarães”, but the loss of the last Christian Name initial for Maria Conceição Nascimento Costa, and the loss of italicization of the generic name “Leptospira sp“in the tile. Note also the absence of the Portuguese title and the DOI, and the addition of a PubMed ID.

The real problems with this reference arise when we look at the starting XML corpus upon which our linked open citation data output is based, which in turn is based on the original PLoS submission of the Reis et al. (2008) paper to PubMed Central.

The author list for this reference #39 in the PMC XML for Reis et al (2008) is as follows:

<ref id=“pntd.0000228-Dias1”>
<citation citation-type=“journal”>
<person-group person-group-type=“author”>

Here we see, as in the HTML reference list of the original Reis et al. (2008) paper, the truncation of the list of authors to the first five, and the loss of accent on the surname “Guimarães”. Surprisingly, the title is recorded as

<article-title>Factors associated with <italic>Leptospira</italic> sp infection in a large urban center in Northeastern Brazil.</article-title>

correctly showing the italic “Leptospira” but also correctly not italicizing the following “sp”!

All is not lost, however, in terms of the full author list. Since the OASS XML for Reis et al. (2008) contains a Pubmed ID for this Dias et al. (2007) paper [3]:

<pub-id pub-id-type=“pmid”>17992402</pub-id>

we can retrieve the PubMed bibliographic record for this paper by querying the Entrez API, from which we recover:

<Item Name=“AuthorList” Type=“List”>
<Item Name=“Author” Type=“String”>Dias JP</Item>
<Item Name=“Author” Type=“String”>Teixeira MG</Item>
<Item Name=“Author” Type=“String”>Costa MC</Item>
<Item Name=“Author” Type=“String”>Mendes CM</Item>
<Item Name=“Author” Type=“String”>Guimarães P</Item>
<Item Name=“Author” Type=“String”>Reis MG</Item>
<Item Name=“Author” Type=“String”>Ko A</Item>
<Item Name=“Author” Type=“String”>Barreto ML</Item>
<Item Name=“LastAuthor” Type=“String”>Barreto ML</Item>

Note that here we have the full author list containing the correctly accented surname “Guimarães”.

Thus, by matching this record with the original PLoS record for Dias et al., and selecting the longer list and the more accentuated names, we can correct the omissions in the original PubMed Central OASS data.

Example 4

However, Entrez output is not infallible. In the Entrez xml output for reference #44 in Reis et al. (2008), namely the paper by Travassos and Williams (2004) [4], we find:

<Item Name=“doi” Type=“String”>/S0102-311X2004000300003</Item>

The correct DOI for this article is doi:10.1590/S0102-311X2004000300003. However, the Entrez output from PubMed is missing the journal prefix “10.1590”, which when parsed to its URI form during our automated processing of the source data from XML to RDF would become


if we did not take steps to check for the correct DOI syntax.

These few examples, all taken from a single OASS article, illustrates some of the problems we have had to face in creating accurate and reliable RDF to enable us to publish these reference lists as open citation data.

What is shocking to me with regard to PLoS, perhaps the leading Open Access publisher, is that they don’t systematically include both DOIs and PubMed IDs in both HTML and XML versions of article references on the PLoS web site, despite the fact that they insert PubMed IDs into the records they mark up in NLM-DTD XML and send to PubMed Central, and also that PLoS persists with its policy of not listing all the authors, and that it does not include proper accents and diacritical marks, particularly for non-English names.

The methods used to correct citation errors are described in the next blog post, while the data processing pipeline through which we pass the input data to generate our RDF output Open Citations Corpus is described in the following blog post.

[1] Reis RB, Ribeiro GS, Felzemburgh RDM, Santana FS, Mohr S, Melendez SXTO, Queiroz A, Santos AC, Ravines RR, Tassinari WS, Carvalho MS, Reis MG, Ko AI
(2008). Impact of environment and social gradient on Leptospira infection in urban slums. PLoS Negl Trop Dis
2(4): e228. doi:10.1371/journal.pntd.0000228

[2]    Ko AI, Reis MG, Ribeiro Dourado CM, Johnson WD Jr, Riley LW and the Salvador Leptospirosis Study Group (1999). Urban epidemic of severe leptospirosis in Brazil. Lancet
354: 820–825. doi:10.1016/S0140-6736%2899%2980012-9.

[3]    Dias JP, Teixeira MG, Costa MC, Mendes CM, Guimarães P, Reis MG, Ko A, Barreto ML (2007). Factors associated with Leptospira sp infection in a large urban center in northeastern Brazil. Rev Soc Bras Med Trop. 40(5): 499-504. doi:10.1590/S0037-86822007000500002.

[4]     Travassos C, Williams DR (2004). The concept and measurement of race and their relationship to public health: A review focused on Brazil and the United States. Cad Saude Publica
20: 660–678. doi:10.1590/S0102-311X2004000300003.

Garbage in, garbage out – problems with bibliographic references

The Open Citations Project has aimed to liberate bibliographic references from biomedical research literature as Open Linked Data, using as its starting corpus the Open Access Subset (OASS) of articles within PubMed Central. The greatest problem faced during this project, naively unanticipated before we started, was the extend of incompleteness, noise and errors of various sorts within the reference information extracted from the OASS articles. So significant has this problem been, that it has taken almost the entire time and effort of Alex Dutton, our skilled data munger working on the project, to sort out, and without his skill, dedication and effort the project would not have succeeded.

In this context, any deviation of the bibliographic reference in an OASS article from the bibliographic citation text for that paper provided by the original publisher, is taken to be an error. These may be as slight as the substitution of a “β” character by the word “beta” in the title of a cited work by the author of the OASS article as he was creating his reference list, and as severe as including in the reference to one paper the DOI of another unrelated paper.

So it is worth taking time, before explaining how we addressed this problem, to describe its nature and magnitude, and to illustrate it with typical examples. We have found that errors of one sort or another occur in about 1% of all extracted references from the OASS. Since we extracted 6,325,178 individual references from our starting corpus, this constitutes well over 50,000 references containing errors.

Author errors

Errors have different sources. Authors are largely to blame, for not exercised due care when creating the reference lists of their paper (or earlier, when creating bibliographic records in a reference management system such as EndNote). However, if one of their EndNote records used to populate a reference list had been pulled automatically from some third-party source, the error might be due to that source, something of which the author was totally unaware.

Some references have incorrect punctuation or capitalisation of titles, or omit some sub-part, as exemplified by the following screenshot – note the use of “alternate”, “alternative” or “Alternate”, and of “intensivist” and “intensive” in different OASS references to the same paper. The boxed title is the correct one:

Some references omit one or more author names, or omit diacritics, a tendency particularly correlated with the degree to which a name is ‘foreign’ or ‘unusual’, as the next example illustrates – note the lack of “Mariotte-Labarre S” in the first two references, and the variation in punctuation of the journal name abbreviation in the third:

There are also numerous instances where the text of a reference is correct, but the associated identifiers (e.g. DOIs and PubMed IDs) are incorrect. By way of example, references 15 and 16 of PMC1839102 are both given the same DOI; in PMC2896208 the DOIs for references 52 and 72 are swapped; and in PMC2778786 references 15, 40, and 49 are all given the DOI of another (uncited) paper.

We should point out that these examples are entirely anecdotal, and that we haven’t investigated the frequency with which these or any other classes of error occurs.

Publisher errors

Publishers are the other main culprits, particularly for introducing errors into documents during the XML encoding stage, from which it is almost impossible to recover by the automated parsing systems we have used to extract information into our RDF-encoded records.

Individual publishers, while all working to the same National Library of Medicine DTD for encoding the XML markup of their articles submitted to PubMed Central, might take different approaches to encoding the same information. For example, the text

“… was found to be significant[1,3–6]

might be marked up as either of the following:

“… was found to be significant<sup>[<xref rid=”CR1″>1</xref>,<xref rid=”CR3″>3</xref>–<xref rid=”CR6″>6</xref>]</sup>”

“… was found to be significant<sup><xref rid=”CR1,CR3,CR4,CR5,CR6″>[1,3–6]</xref></sup>”.

Note that in the former case there is no explicit mention of the references with identifiers CR4 and CR5, making things a little harder to parse.

We have found occasions where a four-digit number in the title has been marked-up as a the publication year. For example, reference 21 of PMC2743650 claims that the cited article was published in 7942. The cited article’s real title refers to the bacterium strain PCC7942, information that has been removed from the title in the OACC reference.

The information returned by the Entrez API, which we used as our ‘gold standard’ against which to check OASS references containing PubMed IDs, was itself not without error. We found a number of PubMed records where DOIs had been truncated to just the prefix, or were missing a prefix.

Editors and referees

Others are also culpable. At the end of paragraph 12 of the PLoS ONE paper by Pickart et al. 2006 [1] we find the text:

We consider the current 12% detection rate to be a lower estimate of observable specific phenotypes from the screen, as additional screening will examine the morpholino collection using a variety of novel assays (such as newly generated enhancer and gene trap lines; Balciunas et al., 2004; Kawakami et al., 2004; Parinov et al., 2004) and may reveal developmental and/or functional aspects not readily visible by morphological criteria.

However, these three references do not appear in the reference list and so are totally lost to the system – the authors knew whom they were citing, but no-one else. How this escaped the eagle eyes of the authors, the journal editor and the reviewers is beyond my understanding!

In a separate blog post we describes how we have corrected some of these errors, while examples of errors in author lists are documented in the next post.

[1] Pickart MA et al. (2006). Genome-Wide Reverse Genetics Framework to Identify Novel Functions of the Vertebrate Secretome. PLoS ONE 1(1): e104. 10.1371/journal.pone.0000104.

Search OpenEdition Search

You will be redirected to OpenEdition Search