From little acorns . . . A retrospective on OpenCitations

The initial vision

Now that OpenCitations is hosting over one billion freely available scholarly bibliographic citations, this is perhaps an opportune moment to look back to the start of this initiative. A little over eleven years ago, on 24 April 2010, I spoke at the Open Knowledge Foundation Conference, OKCon2010, in London, on the topic

OpenCitations: Publishing Bibliographic Citations as Linked Open Data

I reported that, earlier that same week, I had applied to Jisc for a one-year grant to fund the OpenCitations Project ( Jisc (at that time ‘The JISC’, the Joint Information Systems Committee) was tasked by the UK government, among other things, to support research and development in information technology for the benefit of the academic community.

The purpose of that original OpenCitations R&D project was to develop a prototype in which we:

  • harvested citations from the open access biomedical literature in PubMed Central;
  • described and linked them using CiTO, the Citation Typing Ontology [1];
  • encoded and organized them in an RDF triplestore; and
  • published them as Linked Open Data in the OpenCitations Corpus (OCC).

I told those at the conference that in this demonstration project, with limited JISC funding, we could not hope to “boil the whole ocean”, but that nevertheless there would be substantial benefits from even partial coverage of citation data from the scholarly literature:

  • We could show the way and establish best practice.
  • Despite partial coverage, all key papers would most likely be cited several times.
  • The overall topological structure of the citation network would be revealed.
  • We would create a ‘benchmark’ corpus of high-quality RDF citation data that could be used to develop analytical and visualization tools.
  • We could show the value of open citation data in helping scholars to discover full text articles of all types, and thus encourage subscription-access publishers to release their reference metadata.

The important thing, I said, was to make a start!

The Jisc OpenCitations Project

That JISC grant application was funded, and the project, to last for a year with modest funding of £100K, started in my lab in the Department of Zoology at Oxford University on 1st June 2010, and was subsequently extended for a further six months.

Using data from the Open Access subset of PubMed Central, we created the first prototype release of the OpenCitations Corpus of linked bibliographic citation data, containing 6,529,815 independent bibliographic records of both citing and cited entities, comprising references to ~20% of all post-1980 articles recorded in PubMed, including those to all the most important highly cited papers in every field of biomedical endeavour.

This achievement was almost entirely the result of the excellent work by our chief data wrangler Alex Dutton, whose skill and natural feel for linked data did wonders for this project. Ben O’Steen, Graham Klyne and Alistair Miles made important contributions.

The project also resulted in many other development, described here, most which were developed or at least initiated during a short but wonderfully productive collaboration with Silvio Peroni, who spent six months with me in 2010 as a doctoral student intern from the University of Bologna, to which he subsequently returned to complete his thesis and develop his academic career.

These included:

  • the deconstruction and re-development of the original version of CiTO into a suite of orthogonal and complementary ontologies covering the whole domain of scholarly publishing – the SPAR (Semantic Publishing and Referencing) Ontologies [2, 3];
  • the mapping of various existing metadata schemas into RDF using SPAR, including the DataCite Metadata Schema, and subsequently JATS, now the default NISO standard for XML markup of scholarly documents) [4]; and
  • the initiation of the Semantic Publishing Blog and this OpenCitations Blog.

Life after Jisc – the flowering of OpenCitations

After the Jisc funding ended and I, after a long career in biological teaching and research, formally retired from the Department of Zoology at the Oxford University, members of the initial OpenCitations team moved on to other things. Like so many grant-funded academic project whose initial financial support had dried up, OpenCitations could have foundered at that stage, as an interesting prototype but with too little content to be useful. However, the concept of providing an open alternative to proprietary citation indexes was too important to abandon. But how could it be transitioned into something enduring and useful, particularly when as a matter of principle one had decided that the citation data should be made freely available, thus precluding income generation by charging for ‘premium’ services or the formation of a commercial spin-off?

Finally, I realized that something radical needed to be done to move OpenCitations forward. I had maintained a lively collaboration with Silvio Peroni at the University of Bologna, resulting between 2011 and 2014 in the publication of 18 articles and conference papers concerning the SPAR ontologies, ontology development, documentation and visualization, and related topics, and in 2015 I invited him to start working with me directly on OpenCitations. It was the best decision I could have made. We decided to take the initial concept and re-implement it from the bottom up. OpenCitations gave Silvio a major computer science project to which he could apply his considerable talent, and soon resulted in the development of a revised RDF data model for describing citation data, the OpenCitations Data Model (OCDM) [5] and a suite of new software tools to harvest, organise and publish citations at linked open data [6]. The credit for almost all the subsequent conceptual and technical developments within OpenCitations, which have incrementally led to our present situation, is due to Silvio Peroni, and the scholarly community is indebted to him for the intelligence, skill and diligent application he has given to OpenCitations over the past six years. I am truly honoured to have Silvio as co-Director of OpenCitations, and wish to take this opportunity to acknowledge his contributions and to thank him publicly.

Our work on OpenCitations at that stage, summarized in [7], would not have been possible without the enthusiastic support of Silvio’s senior colleague Fabio Vitali and of the Department of Computer Science and Engineering at the University of Bologna, which not only provided a stimulating environment for Silvio’s post-doctoral work, but also supplied computing services and infrastructure at no charge to OpenCitations. It was also greatly helped by Professor David De Roure of Oxford University, who gave me an academic home and a formal affiliation within the Oxford e-Research Centre after my retirement from the Department of Zoology, which enabled me to continue to hold research grants.

As has been documented in earlier posts in this blog, we greatly benefitted in 2017 from a grant from the Alfred P. Sloan Foundation which enabled us to purchase a new and more powerful computing infrastructure for the sole use of OpenCitations and to extend and improve our software, and subsequently in 2019 by a project grant from the Wellcome Trust to develop the Open Biomedical Citations in Context Corpus, that permitted the extension of OCDM and SPAR for the characterization of in-text references and their textual contexts.

A significant breakthrough came in January 2018 with our decision to treat citations as first-class data entities, each with its own persistent identifier (PID), the Open Citations Identifier (OCI) [8]. This gave Silvio the freedom to envision a new kind of database, a citation index in which each citation had its own metadata, including citation timespan, citation categorization (e.g. self-citation), and of course the DOIs of the citing and cited publications. The creation of this new index was possible only with the incredible effort by Ivan Heibi, who served as a Research Fellow in the project funded by the Alfred P. Sloan Foundation at that time, and who was entirely responsible for developing the first version of the code necessary for creating such a database. Having harvested all the open references from Crossref metadata dumps, Silvio and Ivan created COCI, the OpenCitations Index of Crossref DOI-to-DOI Citations, which immediately became our principal source of open citations, the original OpenCitations Corpus being retained as a ‘sandbox’ in which to experiment with new data representations, for example those required for the Open Biomedical Citations in Context Corpus. Access to COCI was facilitated by Silvio’s development of a REST API, using his software tool RAMOSE (Restful API Manager Over SPARQL Endpoints), which enables the easily configurable deployment of a REST API over any SPARQL endpoint to an RDF triplestore [9]. We were able to organize our all data, both ‘traditional’ and new, and to encode it in RDF, thanks to the comprehensive OpenCitations Data Model [5], itself based on our SPAR Ontologies [3], which we evolved as necessary to accommodate new data representation requirements.

During this period we published a number of definitions, conference papers and journal articles documenting these advances, details of which can be found here. Of these, the most recent canonical publication describing OpenCitations as an infrastructure for open scholarship, and its datasets, tools, services and activities, is Peroni and Shotton (2020) [10]. We also established the Research Centre for Open Scholarly Metadata at the University of Bologna, primarily to handle administrative, financial and academic aspects of OpenCitations activities.

OpenCitations’ future

The problem remained: how to sustain the OpenCitations infrastructure financially. We were greatly helped by Bilder, Lin and Neylon’s formulation of the Principles of Open Scholarly Infrastructures (POSI) [11], in which they clearly pointing out that reliance solely on grant funding for specific projects was not the answer. OpenCitations compliance with POSI is described here. We were thus immensely grateful that SPARC Europe and other institutions had the wisdom to establish SCOSS (The Global Sustainability Coalition for Open Science Services) to facilitate the crowd-sourced financial support of useful open infrastructures by the scholarly community, including academic libraries, government agencies and other stakeholders. OpenCitations applied for SCOSS support in 2019, which led to the selection of OpenCitations for support in the SCOSS second round.

The donations we are now starting to receive from such stakeholders, and the new staff that this funding has recently allowed us to hire, signal the start of our transition from a financially vulnerable academic project to a sustainable open scholarly infrastructure of real value to the community.

The work of opening more of the global citation graph now requires two things:

  • that each publisher takes responsibility for ensuring that the references from all of its journal articles and books are submitted, together with all other bibliographic metadata, to open scholarly bibliographic metadata aggregators such as Crossref and DataCite, from which they can be indexed into open citation indexes of sufficient quality, depth of detail and breadth of coverage that these offer genuine alternatives to the expensive proprietary citation indexing services upon which the academic community presently relies; and
  • that the entire scholarly stakeholder community re-directs a fraction of the enormous sums currently spent on its subscriptions to proprietary bibliographic services in order to support Open Science infrastructures such as OpenCitations that making citations and other forms of scholarly metadata and objects freely available.


[1] David Shotton (2010). CiTO, the Citation Typing Ontology. J. Biomedical Semantics 1 (Suppl. 1): S6.

[2] Silvio Peroni, David Shotton (2012). FaBiO and CiTO: ontologies for describing bibliographic resources and citations. Web Semantics, 17: 33-34., OA at

[3] Silvio Peroni, David Shotton (2018). The SPAR Ontologies. In Proceedings of the 17th International Semantic Web Conference (ISWC 2018): 119-136.

[4] Peroni S, Lapeyre DA and Shotton D (2012) From Markup to Linked Data: Mapping NISO JATS v1.0 to RDF using the SPAR (Semantic Publishing and Referencing) Ontologies. Proc. 2012 JATS Conference, National Library of Medicine, Bethesda, Maryland, USA (October 2012): 16-17.

[5] Marilena Daquino, Silvio Peroni , David Shotton (2020). The OpenCitations Data Model. Figshare.

[6] Silvio Peroni, David Shotton, Fabio Vitali (2017). One Year of the OpenCitations Corpus: Releasing RDF-based scholarly citation data into the Public Domain. In The Semantic Web – ISWC 2017 (Lecture Notes in Computer Science Vol. 10588, pp. 184–192). Springer, Cham.

[7] Silvio Peroni, Alexander Dutton, Tanya Gray, David Shotton (2015). Setting our bibliographic references free: towards open citation data. Journal of Documentation, 71 (2): 253-277., OA at

[8] Silvio Peroni, David Shotton (2019). Open Citation Identifier: Definition. Figshare.

[9] Daquino, M., Heibi, I., Peroni, S., & Shotton, D. (2021). Creating Restful APIs over SPARQL endpoints with RAMOSE. Semantic Web.

[10] Silvio Peroni, David Shotton (2020). OpenCitations, an infrastructure organization for open scholarship. Quantitative Science Studies, 1(1): 428-444.

[11] Geoffrey Bilder, Jenny Lin, Cameron Neylon (2015). Principles for Open Scholarly Infrastructure.

Cite this article as: davidshotton, "From little acorns . . . A retrospective on OpenCitations," in OpenCitations blog, 16/08/2021,

Open Citations – Indexing PubMed Central OA data

As part of our work on the Open Citations extensions project, I have recently been doing one of my favourite things – namely indexing large quantities of data then exploring it.

On this project we are interested in the PubMed Central Open Access subset, and more specifically, we are interested in what we can do with the citation data contained within the records that are in that subset – because, as they are open access, that citation data is public and freely available.

We are building a pipeline that will enable us to easily import data from the PMC OA and from other sources such as arXiv, so that we can do great things with it like explore it in a facetview, manage and edit it in a bibserver, visualise it, and stick it in the rather cool related-work prototype software. We are building on the earlier work of both the original Open Citations project, and of the Open Bibliography projects.

Work done so far

We have spent a few weeks getting to understand the original project software and clarifying some of the goals the project should achieve; we have put together a design for a processing pipeline to get the data from source right through to where we need it, in the shape that we need it. In the case of facetview / bibserver work, this means getting it into a wonderful elasticsearch index.

While Martyn continues work on the bits and pieces for managing the pipeline as a whole and pulling data from arXiv, I have built an automated and threadable toolchain for unpacking data out of the compressed file format it arrives in from the US National Institutes of Health, parsing the XML file format and converting it into BibJSON, and then bulk loading it into an elasticsearch index. This has gone quite well.

To fully browse what we have so far, check out

For the code:

The indexing process

Whilst the toolchain is capable of running threaded, the server we are using only has 2 cores and I was not sure to what extent they would be utilised, so I ran the process singular. It took five hours and ten minutes to build an index of the PMC OA subset, and we now have over 500,000 records. We can full-text search them and facet browse them.

Some things of particular interest that I learnt – I have an article in the PMC OA! And also PMIDs are not always 8 digits long – they appear in fact to be incremental from 1.

What next

At the moment there is no effort made to create record objects for the citations we find within these records, however plugging that into the toolchain is relatively straightforward now.

The full pipeline is of course still in progress, and so this work will need a wee bit of wiring into it.

Improve parsing. There are probably improvements to the parsing that we can make too, and so one of the next tasks will be to look at a few choice records and decide how better to parse them. The best way to get a look at the records for now is to use a browser like Firefox or Chrome and install the JSONview plugin, then go to and have a bit of a search, then click the small blue arrows at the start of a record you are interested in to see it in full JSON straight from the index. Some further analysis on a few of these records would be a great next step, and should allow for improvements to both the data we can parse and to our representation of it.

Finish visualisations. Now that we have a good test dataset to work with, the various bits and pieces of visualisation work will be pulled together and put up on display somewhere soon. These, in addition to the search functionality already available, will enable us to answer the questions set as representative of project goals earlier in January (thanks David for those).


David Shotton writes: This productive collaboration between Cottage Labs and the Open Citations Corpus came to an end when Jisc funding ran out.  The corpus has more recently been given a new lease of life, as described here, with a new instantiation named OpenCitations hosted at the Department of Computer Science and Engineering of the University of Bologna, with Silvio Peroni as Co-Director.

Cite this article as: markmacgillivray, "Open Citations – Indexing PubMed Central OA data," in OpenCitations blog, 25/02/2013,

Open Citations Extension Project

I am pleased to announce that the JISC have funded an extension to the Open Citations Project to run from 1st August 2012 until 31st January 2013, during which we will review and revise the technology used to create the Open Citations Corpus, will update the content provided by PubMed Central, and will improve its presentation.  We will also prototype value-added services over the open citations data, to demonstrate their usefulness and to justify further funding that will permit a major expansion of the corpus in 2013 to include reference lists from subscription-access journals, including those from Nature Publishing Group, Science and other AAAS publications, and Oxford University Press.  In preparation for this, we will collaborate with CrossRef to determine how best to ingest reference data into the Open Citations Corpus in an on-going manner.

Full details are given in the Case for Support.  Watch this space!

Cite this article as: davidshotton, "Open Citations Extension Project," in OpenCitations blog, 16/07/2012,

Like a kid with a new train set! Exploring citation networks

As part of the Open Citations Project, Alex Dutton recently completed a graphing plug-in for the Open Citations web site, that permits users to generate different kinds of graphs of citation networks by querying the Open Citation Corpus for a particular article, and either display the network of papers citing that article (input citations), papers cited by that article (output citations), or both.  These can be displayed on screen in the web browser in a variety of layouts, or conveniently downloaded in a number of useful formats.


Having survived the preparation and posting of the JISC Open Citations Project Final Blog Post last night, minutes before the midnight deadline, I’m now like a kid with a new train set, playing with this display tool and exploring the citation networks present in the Open Citation Corpus, something I have dreamed of doing for two years now.

Remember first that in the Open Citations Corpus we have some 200,000 citing articles – those within the Open Access Subset (OASS) of Pubmed Central – citing ~3.4 million papers out there in the big wide world, which are only recipients of citations.  The consequence of this limited corpus is that the majority of citation chains are of length one – from a paper in the OASS to a paper outside the OASS.  Not very interesting.  Add to this the fact that PubMed Central is new – over 90% of the papers in the Open Access Subset were published in the 21st Century, and 77% of them in the last 5 years.  Thus there are only a very few citation from articles within the OASS to other articles within the OASS.  That means that the maximum length of our citation chains, at present, to three or four on links the input side – a selected article may be cited by a chain of three or four other OASS articles, and three or four on the output side – the selected article may cite other OASS articles in addition to non-OASS articles, and these in turn will cite others.  However, in most cases, the citations chains are much shorter.

simple network
simple networkFigure 1. A simple citation network of input citation chain length of 2 links within the Open Citations Corpus, and an output chain length of 1 link – the selected article (red) receives citations from other OASS articles (green), and itself cites only articles outside the OASS (white).

Let’s start with something familiar – the article in PLoS Neglected Tropical Diseases by Reis et al. (2008) [1] that I used for our semantic publishing exemplar [2].  Its inward citation graph, limited to a citation chain length of two links, created by and copied from the Open Citations Project web site, looks like this:

input citations of Reis
Figure 2. The input citation network of Reis et al. (2008), limited to an citation chain length of 2 links.

I, of course, cited the Reis et al. (2008) paper [1] in our 2009 Adventures paper [3] that we based upon it, and also in my first paper on CiTO in 2010 [3], which also cites the Adventures paper.   Reis et al. (2008) is also cited by Fink et al. (2010) [4], who also cited our Adventures paper, and by Bourhy et al. (2010) [5], another PLoS Neglected Tropical Diseases paper in the OASS, which in turn is cited by Galloway and Levett (2010) [6], while our Adventures paper is also cited by Gerner and Nenadic (2010) [7].

The following image shows this graph as it was originally created within the Open Citations web page:

Input citations of Reis in web page
Figure 3. The same input citation network of Reis et al. (2008), as shown in the Open Citations web site.

Since Reis et al. (2008) has a reference list containing 52 references, its output citation graph is much more complex, even when limited to a citation chain length of 2, since several of its cited papers are also members of the Open Access Subset.  The following figure shows the whole output citation network a citation chain length of 2, which is too demagnified to be legible.

Citations by Reis
Figure 4. The outward citation network of Reis et al. (2008), limited to a citation chain length of 2 links.

The next figure shows a close-up of part of the previous diagram – the output citation network of Reis et al. (2008), again showing the Reis et al. (2008) paper in red, and one of the key papers it cites, Maciel et al. (2010) [8], a slightly earlier paper from the same research group, forming a second key node in the top right of the diagram.

Cited by Reis closeup
Figure 5. A close-up of a central portion of the outward citation network of Reis et al. (2008), limited to a citation chain length of 2 links.

Clearly, there is lots of information that can be extracted from these graphs, particularly when we display them in a tool like GraphViz that permits interactions with the data.  While the Open Citations web site simply displays such citation graphs created using one of several layout algorithms selected by the user, the raw data can also be downloaded in a variety of formats including GraphViz, GraphML and SVG, while the resulting network images can be downloaded in as PNG, JPEG and PDF images, and the underlying RDF metadata can be downloaded as RDF/XML. N-triples, Notation3 and Turtle.

Having used our new Open Citations web site and its network display interface for a short while, I am already aware of many shortcomings and limitations that we will attempt to improve upon in the next few days.  However, we would very much like to hear from you – as a user of the Open Citations web site – both to learn what you like about what we have done and to hear what you find to be shortcomings of the functionality and new features that you would like to see implemented, which we will record as user stories to input into our next round of development.  These can either be recorded as comments on this blog post, or can be e-mailed with the subject line “Open Citations web site” either to me <> or to Alex Dutton <>, who is the person who deserves all the credit for the present system.  We look forward to hearing from you.

[1]  Reis RB, Ribeiro GS, Felzemburgh RDM, Santana FS, Mohr S, Melendez SXTO, Queiroz A, Santos AC, Ravines RR, Tassinari WS, Carvalho MS, Reis MG, Ko AI (2008). Impact of environment and social gradient on Leptospira infection in urban slums. PLoS Negl Trop Dis 2(4): e228. doi:10.1371/journal.pntd.0000228.

[2] Shotton D, Portwin K, Klyne G, Miles A (2009). Adventures in semantic publishing: exemplar semantic enhancements of a research article. PLoS Comput Biol 5:e1000361. doi:10.1371/journal.pcbi.1000361.

[3] Shotton D (2010). CiTO, the Citation Typing Ontology. Journal of Biomedical Semantics  1 (Suppl. 1): S6. doi:10.1186/2041-1480-1-S1-S6.

[4]  Fink JL, Fernicola P, Chandran R, Parastatidis S, Wade A, Naim O, Quinn GB, Bourne PE (2010). Word add-in for ontology recognition: semantic enrichment of scientific literature.  BMC Bioinformatics 11:103. doi:10.1186/1471-2105-11-103.

[5]  Bourhy P, Collet L, Clément S, Huerre M, Ave P, Giry C, Pettinelli F, Picardeau M (2010). Isolation and Characterization of New Leptospira Genotypes from Patients in Mayotte (Indian Ocean). PLoS Negl Trop Dis 4(6): e724. doi:10.1371/journal.pntd.0000724.

[6]  Galloway RL, Levett PN (2010) Application and Validation of PFGE for Serovar Identification of Leptospira Clinical Isolates. PLoS Negl Trop Dis 4(9): e824. doi:10.1371/journal.pntd.0000824.

[7]  Gerner M, Nenadic G (2010). LINNAEUS: A species name identification system for biomedical literature. BMC Bioinformatics 11:85. doi:10.1186/1471-2105-11-85.

[8]  Maciel EAP, Carvalho ALF, Nascimento SF, Matos RB, Gouveia EL, Reis MG, Ko AI (2008). Household transmission of Leptospira infection in urban slum communities. PLoS Negl Trop Dis 2: e154. doi:10.1371/journal.pntd.0000154.

Enhanced by Zemanta

Who wrote this paper? Author list problems in PubMed Central references

To illustrate three kinds of problems in obtaining correct author lists for Open Citation data from articles in the PubMed Central Open Access subset (OASS), I take three examples, the first of which is the result of a publication policy, the second due to mis-handling of an authorship attribution at the time of publication, and the third exemplifing errors introduced when handling non-English personal names.

Example 1

In the paper by Reis et al. (2008) [1], that we took as the subject for our exercise in semantic publishing enhancements described in [2], we find the following entry for Reference 40 in the reference list:

40.    Maciel EAP, Carvalho ALF, Nascimento SF, Matos RB, Gouveia EL, et al. (2008) Household transmission of Leptospira infection in urban slum communities. PLoS Negl Trop Dis 2: e154. doi:10.1371/journal.pntd.0000154.

Note that it is the policy of the publisher, the Public Library of Science, to list only the first five authors in references to papers that have more than five, despite publishing online-only journals where article length is not an issue.

The XML for this reference in the document is as follows:

    <meta name="citation_reference" content="citation_title=Household transmission of Leptospira infection in urban slum communities.; citation_author=EAP Maciel; citation_author=ALF Carvalho; citation_author=SF Nascimento; citation_author=RB Matos; citation_author=EL Gouveia; citation_journal_title=PLoS Negl Trop Dis; citation_volume=2; citation_number=40; citation_pages=e154. doi:10.1371/journal.pntd.0000154; citation_date=2008; " /> 

Note that in the XML, all indication that there are more than five authors, i.e. the “et al.” present in the human-readable reference, is totally lost. There is thus no way of telling from the XML for this paper retrieved from PubMed Central that the full authorship for this paper is as follows:

Elves A. P. Maciel, Ana Luiza F. de Carvalho, Simone F. Nascimento, Rosan B. de Matos, Edilane L. Gouveia, Mitermayer G. Reis and Albert I. Ko.

The last two authors of the cited paper, Mitermayer Reis and Albert Ko, who are the lead author and the senior author, respectively, of the citing paper, are both omitted from the PLoS reference to the cited paper, and hence from the data automatically extracted by Open Citations from the OASS.

Example 2

A second example from the same citing paper is the cited reference to Ko et al. (1999) [2]. As the reference at the foot of this page shows, the author list includes six names “and the Salvador Leptospirosis Study Group”. Group attributions of this kind are commonplace, particularly in papers resulting from large collaborative projects. However, conventional markup systems such as the NLM-DTD have no systematic way of handling such information. Surprisingly, it is even incorrectly stated in the human readable version of the reference in the Reis et al. paper. Reference 6 in the article’s reference list reads:

6.    Ko AI, Reis MG, Ribeiro Dourado CM, Johnson WD Jr, Riley LW (1999) Urban epidemic of severe leptospirosis in Brazil. Salvador Leptospirosis Study Group. Lancet 354: 820–825.

thus including “Salvador Leptospirosis Study Group” as part of the title, an error also present in the XML version of the paper:

<meta name="citation_reference" content="citation_title=Urban epidemic of severe leptospirosis in Brazil. Salvador Leptospirosis Study Group.; citation_author=AI Ko; citation_author=MG Reis; citation_author=CM Ribeiro Dourado; citation_author=WD Johnson; citation_author=LW Riley; citation_journal_title=Lancet; citation_volume=354; citation_number=6; citation_pages=820-825; citation_date=1999; " /> 

Example 3

The third example, also taken from the reference list of Reis et al. (2008) [1], illustrated the problems of handling non-English names and titles. Reference 39 in the PLOS article’s reference list [3] reads:

Dias JP, Teixeira MG, Costa MC, Mendes CM, Guimaraes P, et al. (2007) Factors associated with Leptospira sp infection in a large urban center in Northeastern Brazil. Rev Soc Bras Med Trop 40: 499–504.

Note that the author list is truncated, that the generic name “Leptospira sp” in the title is italicized, and that neither the DOI nor the PubMed ID is provided, although the article has both.

The landing page for this article on the publisher’s web site ( shows the following:

Revista da Sociedade Brasileira de Medicina Tropical

Print version ISSN 0037-8682

Rev. Soc. Bras. Med. Trop. vol.40 no.5 Uberaba Oct. 2007

doi: 10.1590/S0037-86822007000500002  ARTIGO ARTICLE

Factors associated with Leptospira sp infection in a large urban center in northeastern Brazil

Fatores associados à infecção por Leptospira sp em um grande centro urbano do Nordeste do Brasil

Juarez Pereira DiasI; Maria Glória TeixeiraI; Maria Conceição Nascimento CostaI; Carlos Maurício Cardeal MendesI; Patrícia GuimarãesII; Mitermayer Galvão ReisII; Albert KoII,III; Maurício Lima BarretoI

The paper is published in English. Note that there is an alternative Portuguese title, that the generic name “Leptospira sp” is italicized in both, and that a DOI is provided, although page numbers are not give for this on-line version of the article. Note also the accents and structures of the full Brazilian author names.

Clicking on the tab Article in PDF format on the landing page takes one to the PDF download page that gives the following reference with page numbers, but lacking English title and DOI, and lacking the full author list:

DIAS, Juarez Pereira et al. Factors associated with Leptospira sp infection in a large urban center in northeastern Brazil. Rev. Soc. Bras. Med. Trop. [online]. 2007, vol.40, n.5, pp. 499-504. ISSN 0037-8682.

A manual search for the same article in PubMed returns the following information:

Rev Soc Bras Med Trop. 2007 Sep-Oct;40(5):499-504.

Factors associated with Leptospira sp infection in a large urban center in northeastern Brazil.

Dias JP, Teixeira MG, Costa MC, Mendes CM, Guimarães P, Reis MG, Ko A, Barreto ML.

PMID: 17992402

Note the correct accentuation of the surname “Guimarães”, but the loss of the last Christian Name initial for Maria Conceição Nascimento Costa, and the loss of italicization of the generic name “Leptospira sp“in the tile. Note also the absence of the Portuguese title and the DOI, and the addition of a PubMed ID.

The real problems with this reference arise when we look at the starting XML corpus upon which our linked open citation data output is based, which in turn is based on the original PLoS submission of the Reis et al. (2008) paper to PubMed Central.

The author list for this reference #39 in the PMC XML for Reis et al (2008) is as follows:

<ref id=“pntd.0000228-Dias1”>
<citation citation-type=“journal”>
<person-group person-group-type=“author”>

Here we see, as in the HTML reference list of the original Reis et al. (2008) paper, the truncation of the list of authors to the first five, and the loss of accent on the surname “Guimarães”. Surprisingly, the title is recorded as

<article-title>Factors associated with <italic>Leptospira</italic> sp infection in a large urban center in Northeastern Brazil.</article-title>

correctly showing the italic “Leptospira” but also correctly not italicizing the following “sp”!

All is not lost, however, in terms of the full author list. Since the OASS XML for Reis et al. (2008) contains a Pubmed ID for this Dias et al. (2007) paper [3]:

<pub-id pub-id-type=“pmid”>17992402</pub-id>

we can retrieve the PubMed bibliographic record for this paper by querying the Entrez API, from which we recover:

<Item Name=“AuthorList” Type=“List”>
<Item Name=“Author” Type=“String”>Dias JP</Item>
<Item Name=“Author” Type=“String”>Teixeira MG</Item>
<Item Name=“Author” Type=“String”>Costa MC</Item>
<Item Name=“Author” Type=“String”>Mendes CM</Item>
<Item Name=“Author” Type=“String”>Guimarães P</Item>
<Item Name=“Author” Type=“String”>Reis MG</Item>
<Item Name=“Author” Type=“String”>Ko A</Item>
<Item Name=“Author” Type=“String”>Barreto ML</Item>
<Item Name=“LastAuthor” Type=“String”>Barreto ML</Item>

Note that here we have the full author list containing the correctly accented surname “Guimarães”.

Thus, by matching this record with the original PLoS record for Dias et al., and selecting the longer list and the more accentuated names, we can correct the omissions in the original PubMed Central OASS data.

Example 4

However, Entrez output is not infallible. In the Entrez xml output for reference #44 in Reis et al. (2008), namely the paper by Travassos and Williams (2004) [4], we find:

<Item Name=“doi” Type=“String”>/S0102-311X2004000300003</Item>

The correct DOI for this article is doi:10.1590/S0102-311X2004000300003. However, the Entrez output from PubMed is missing the journal prefix “10.1590”, which when parsed to its URI form during our automated processing of the source data from XML to RDF would become


if we did not take steps to check for the correct DOI syntax.

These few examples, all taken from a single OASS article, illustrates some of the problems we have had to face in creating accurate and reliable RDF to enable us to publish these reference lists as open citation data.

What is shocking to me with regard to PLoS, perhaps the leading Open Access publisher, is that they don’t systematically include both DOIs and PubMed IDs in both HTML and XML versions of article references on the PLoS web site, despite the fact that they insert PubMed IDs into the records they mark up in NLM-DTD XML and send to PubMed Central, and also that PLoS persists with its policy of not listing all the authors, and that it does not include proper accents and diacritical marks, particularly for non-English names.

The methods used to correct citation errors are described in the next blog post, while the data processing pipeline through which we pass the input data to generate our RDF output Open Citations Corpus is described in the following blog post.

[1] Reis RB, Ribeiro GS, Felzemburgh RDM, Santana FS, Mohr S, Melendez SXTO, Queiroz A, Santos AC, Ravines RR, Tassinari WS, Carvalho MS, Reis MG, Ko AI
(2008). Impact of environment and social gradient on Leptospira infection in urban slums. PLoS Negl Trop Dis
2(4): e228. doi:10.1371/journal.pntd.0000228

[2]    Ko AI, Reis MG, Ribeiro Dourado CM, Johnson WD Jr, Riley LW and the Salvador Leptospirosis Study Group (1999). Urban epidemic of severe leptospirosis in Brazil. Lancet
354: 820–825. doi:10.1016/S0140-6736%2899%2980012-9.

[3]    Dias JP, Teixeira MG, Costa MC, Mendes CM, Guimarães P, Reis MG, Ko A, Barreto ML (2007). Factors associated with Leptospira sp infection in a large urban center in northeastern Brazil. Rev Soc Bras Med Trop. 40(5): 499-504. doi:10.1590/S0037-86822007000500002.

[4]     Travassos C, Williams DR (2004). The concept and measurement of race and their relationship to public health: A review focused on Brazil and the United States. Cad Saude Publica
20: 660–678. doi:10.1590/S0102-311X2004000300003.

Cite this article as: davidshotton, "Who wrote this paper? Author list problems in PubMed Central references," in OpenCitations blog, 01/07/2011,

Garbage in, garbage out – problems with bibliographic references

The Open Citations Project has aimed to liberate bibliographic references from biomedical research literature as Open Linked Data, using as its starting corpus the Open Access Subset (OASS) of articles within PubMed Central. The greatest problem faced during this project, naively unanticipated before we started, was the extend of incompleteness, noise and errors of various sorts within the reference information extracted from the OASS articles. So significant has this problem been, that it has taken almost the entire time and effort of Alex Dutton, our skilled data munger working on the project, to sort out, and without his skill, dedication and effort the project would not have succeeded.

In this context, any deviation of the bibliographic reference in an OASS article from the bibliographic citation text for that paper provided by the original publisher, is taken to be an error. These may be as slight as the substitution of a “β” character by the word “beta” in the title of a cited work by the author of the OASS article as he was creating his reference list, and as severe as including in the reference to one paper the DOI of another unrelated paper.

So it is worth taking time, before explaining how we addressed this problem, to describe its nature and magnitude, and to illustrate it with typical examples. We have found that errors of one sort or another occur in about 1% of all extracted references from the OASS. Since we extracted 6,325,178 individual references from our starting corpus, this constitutes well over 50,000 references containing errors.

Author errors

Errors have different sources. Authors are largely to blame, for not exercised due care when creating the reference lists of their paper (or earlier, when creating bibliographic records in a reference management system such as EndNote). However, if one of their EndNote records used to populate a reference list had been pulled automatically from some third-party source, the error might be due to that source, something of which the author was totally unaware.

Some references have incorrect punctuation or capitalisation of titles, or omit some sub-part, as exemplified by the following screenshot – note the use of “alternate”, “alternative” or “Alternate”, and of “intensivist” and “intensive” in different OASS references to the same paper. The boxed title is the correct one:

Some references omit one or more author names, or omit diacritics, a tendency particularly correlated with the degree to which a name is ‘foreign’ or ‘unusual’, as the next example illustrates – note the lack of “Mariotte-Labarre S” in the first two references, and the variation in punctuation of the journal name abbreviation in the third:

There are also numerous instances where the text of a reference is correct, but the associated identifiers (e.g. DOIs and PubMed IDs) are incorrect. By way of example, references 15 and 16 of PMC1839102 are both given the same DOI; in PMC2896208 the DOIs for references 52 and 72 are swapped; and in PMC2778786 references 15, 40, and 49 are all given the DOI of another (uncited) paper.

We should point out that these examples are entirely anecdotal, and that we haven’t investigated the frequency with which these or any other classes of error occurs.

Publisher errors

Publishers are the other main culprits, particularly for introducing errors into documents during the XML encoding stage, from which it is almost impossible to recover by the automated parsing systems we have used to extract information into our RDF-encoded records.

Individual publishers, while all working to the same National Library of Medicine DTD for encoding the XML markup of their articles submitted to PubMed Central, might take different approaches to encoding the same information. For example, the text

“… was found to be significant[1,3–6]

might be marked up as either of the following:

“… was found to be significant<sup>[<xref rid=”CR1″>1</xref>,<xref rid=”CR3″>3</xref>–<xref rid=”CR6″>6</xref>]</sup>”

“… was found to be significant<sup><xref rid=”CR1,CR3,CR4,CR5,CR6″>[1,3–6]</xref></sup>”.

Note that in the former case there is no explicit mention of the references with identifiers CR4 and CR5, making things a little harder to parse.

We have found occasions where a four-digit number in the title has been marked-up as a the publication year. For example, reference 21 of PMC2743650 claims that the cited article was published in 7942. The cited article’s real title refers to the bacterium strain PCC7942, information that has been removed from the title in the OACC reference.

The information returned by the Entrez API, which we used as our ‘gold standard’ against which to check OASS references containing PubMed IDs, was itself not without error. We found a number of PubMed records where DOIs had been truncated to just the prefix, or were missing a prefix.

Editors and referees

Others are also culpable. At the end of paragraph 12 of the PLoS ONE paper by Pickart et al. 2006 [1] we find the text:

We consider the current 12% detection rate to be a lower estimate of observable specific phenotypes from the screen, as additional screening will examine the morpholino collection using a variety of novel assays (such as newly generated enhancer and gene trap lines; Balciunas et al., 2004; Kawakami et al., 2004; Parinov et al., 2004) and may reveal developmental and/or functional aspects not readily visible by morphological criteria.

However, these three references do not appear in the reference list and so are totally lost to the system – the authors knew whom they were citing, but no-one else. How this escaped the eagle eyes of the authors, the journal editor and the reviewers is beyond my understanding!

In a separate blog post we describes how we have corrected some of these errors, while examples of errors in author lists are documented in the next post.

[1] Pickart MA et al. (2006). Genome-Wide Reverse Genetics Framework to Identify Novel Functions of the Vertebrate Secretome. PLoS ONE 1(1): e104. 10.1371/journal.pone.0000104.

Cite this article as: davidshotton, "Garbage in, garbage out – problems with bibliographic references," in OpenCitations blog, 01/07/2011,

Input data for Open Citations – the PMC Open Access Subset

PubMed, created by the US National Library of Medicine in DATE, holds bibliographic records and abstracts for essentially all journal articles published in the biomedical sciences. It currently records almost a million new entries each year!

PubMed Central (PMC), created as an extension of PubMed, is designed to hold full text articles from among the PubMed entries. At present, PMC holds entries for ~9.3% of the papers indexed in PubMed published between 1980 and 2010, 1,428,675 out of a total of 15,319,102. Many of these PMC articles (192,452 for the years 1980 to 2010, ~13.5% of the PMC holdings) are truly Open Access articles, that users can download and repurpose as they will. However, the majority are articles from subscription access journals deposited in PMC under licence agreements with funding agencies that, while providing read access to the full text, prevent readers from downloading the articles and from making derivative works.

The Open Citations Project has to date worked exclusively with the Open Access subset (OASS) of PMC. As of 24 January 2011, there were 204,637 OASS articles, including a few published before 1980. In almost all of these OASS articles, the reference lists were nicely marked up in NLM-DTD XML, making the task of identifying individual references straightforward. In a few cases, the articles were present as scanned page images, lacking any internal markup – those we were unable to process.

From the XML reference lists of these papers, we were able to identify and extract 6,325,178 individual references, which, together with the bibliographic information we had on the OASS articles themselves gave us 6,529,815 independent bibliographic records of both citing and cited entities. As explained in the next blog post, these records showed varying degrees of completeness and accuracy.

Using the Entrez API, we were able to use PubMed IDs, where these were available in the references, to extract a further 2,304,143 bibliographic records from PubMed, which, in the ideal world, would each exactly duplicate the information we had previously obtained from the OASS bibliographic reference containing that PubMed ID. As we shall describe, these additional PubMed records proved exceptionally useful in correcting imperfect OASS references.

Since the OASS articles cite papers outside the OASS, as well as a few within it, the majority of the bibliographic information we thus acquired related to papers represented within PubMed but not within PubMed Central. And because many OASS papers independently contained references to the most highly cited biomedical papers, many of our records were to the same bibliographic entities.

An important part of our data processing was thus to coalesce independent references from different OASS articles to the same multiply cited papers into a set of unique bibliographic records, each for one paper. Once this had been achieved, we were left with 3,578,598 unique bibliographic records, 204,637 describing the OASS articles themselves, and 3,373,961 describing articles outside the OASS, mostly from subscription-access journals.

The following table and figure tabulates and illustrates the number of papers in each category between 1980 and 2010 inclusive. The most striking thing about these data are that they show how, between these years, the relatively small number of articles in the Open Access subset of PMC (approx. 200,000 articles) referenced >20% of all PubMed Central papers published between 1950 and 2010 (approx. 15.3 million papers), and in doing so reference all the most important highly cited papers in every field of biomedical endeavour. This inclusive coverage means that citation graphs created from the Open Citations dataset will capture all the important aspects of any field.

Table 1


Number of papers





Cited by OASS






































































































































































% of PubMed  




% of PMC    



Figure 1

Figure 2

The OASS source data give the types of cited entity, aggregated after coalescing, shown in the Figure 3.

Figure 3

Cite this article as: davidshotton, "Input data for Open Citations – the PMC Open Access Subset," in OpenCitations blog, 01/07/2011,
Search OpenEdition Search

You will be redirected to OpenEdition Search