Three publications describing the Open Citations Corpus

Last September, I attended the Fifth Annual Conference on Open Access Scholarly Publishing, held in Riga, at which I had been invited to give a paper entitled The Open Citations Corpus – freeing scholarly citation data.  A recording of my talk is available here, and my PowerPoint presentation is separately available here.  My own reflections on the major themes of the conference are given in a separate Semantic Publishing Blog post.

While in Riga preparing to give that talk about the importance of open citation data, I received an invitation from Sara Abdulla, Chief Commissioning Editor at Nature, to write a Comment piece for their forthcoming special issue on Impact.  My immediate reaction was that this should be on the same theme, an idea to which Sara readily agreed.  The deadline for delivery of the article was 10 days later!

As soon as the Riga conference was over, I first assembled all the material I had to hand that could be relevant to describing the Open Citations Corpus (OCC) in the context of conventional access to academic citation data from commercial sources.  That gave me a raw manuscript of some five thousand words, from which I had to distil an article of less than 1,300 words.  I then started editing, and asked my colleagues Silvio Peroni and Tanya Gray for their comments.

The end result, enriched by some imaginative art work by the Nature team, was published a couple of weeks later on 16th October [1], and presents both the intellectual argument for open citation data, and the practical obstacles to be overcome in achieving the goal of a substantial corpus of such data, as well as giving a general description of the Open Citations Corpus itself and of the development work we have planned for it.

Because of the drastic editing required to reduce the original draft to about a quarter of its size, all material not crucial to the central theme had to be cut.  I thus had the idea of developing the original draft subsequently into a full journal article that would include these additional themes, particularly Silvio’s work on the SPAR ontologies described in this Semantic Publishing Blog post [2], Tanya’s work on the CiTO Reference Annotation Tools described in this Semantic Publishing Blog post, and a wonderful analogy between the scholarly citation network and Venice devised by Silvio.  I also wanted to give authorship credit to Alex Dutton, who had undertaken almost all of the original software development work for the OCC.  For this reason, instead of assigning copyright to Nature for the Comment piece, I gave them a license to publish, retaining copyright to myself so I could re-use the text.  I am pleased to say that they accepted this without comment.

Silvio and I then set to work to develop the draft into a proper article.  The result was a ten-thousand word paper submitted to the Journal of Documentation a week before Christmas [3].  We await the referees’ comments!

 References

[1]     Shotton D. (2013).  Open citations.  Nature 502: 295–297. http://www.nature.com/news/publishing-open-citations-1.13937. doi:10.1038/502295a.

[2]     Peroni S and Shotton D (2012). FaBiO and CiTO: ontologies for describing bibliographic resources and citations. Web Semantics: Science, Services and Agents on the World Wide Web. 17: 33-34. doi:10.1016/j.websem.2012.08.001.

[3]    Silvio Peroni, Alexander Dutton, Tanya Gray, David Shotton (2015). Setting our bibliographic references free: towards open citation data. Journal of Documentation, 71 (2): 253-277. http://dx.doi.org/10.1108/JD-12-2013-0166; OA at http://speroni.web.cs.unibo.it/publications/peroni-2015-setting-bibliographic-references.pdf

This is the main article about OpenCitations, which includes several background information and the main ideas and works supporting the whole project, the Corpus, and some possible future developments in terms of new kinds of data to be included, e.g. citation functions.

 

Open Citations Corpus Import Process

As part of the Open Citations project, we have been asked to review and improve the process of importing data into the Open Citations Corpus, taking the scripts from the initial project as our starting point.

The current import procedure evolved from several disconnected processes and requires running multiple command line scripts and transforming the data into different intermediate formats. As a consequence, it is not very efficient and we will be looking to improve on the speed and reliability of the import procedure. Moreover, there are two distinct procedures depending on the source of the data (arXiv or PubMed Central); we are hoping to unify the common parts of these procedures into a single process which can be simplified and normalised to improve code re-use and comprehensibility.

The Workflow

As PubMed Central provides an OAI-PMH feed, this could be used to retrieve article metadata, and for some articles, full text. Using this feed, rather than an FTP download (as used currently) would allow the metadata import for both arXiv and PubMed Central to follow a near-identical process, as we are already using the OAI-PMH feed for arXiv.

Also, rather than have intermediate databases and information stores, it would be cleaner to import from the information source straight into a datastore. The datastore could then be queried, allowing matches and linking between articles to be performed in situ. The process would therefore become:

  1. Pull new metadata from arXiv (OAI-PMH) and PubMed Central (OAI-PMH) and insert new records into the Open Citations Corpus datastore
  2. Pull new full-text from arXiv and PubMed Central, extract citations, and match with article data in Open Citations server, creating links between these references and the metadata records for the cited articles. Store unmatched citations as nested records in the metadata for each article.
  3. On a scheduled basis (e.g. nightly), review each existing article’s unmatched citations and attempt to match these with existing bibliographic records of other articles.

In outline, this looks like this:

The Datastore

Neo4J is currently used as the final Open Citations Corpus datastore for the arXiv data, by the Related Work system. We propose instead to use BibServer as the final datastore, for its flexibility and scalability, and suitability for the Open Citations use cases.

The Data Structure

The data stored within BibServer as BibJSON will be a collection of linked bibliographic records describing articles. Associated with each record and stored as nested data will be a list of matched citations (i.e. those for which the Open Citations Corpus has a bibliographic record), a list of unmatched citations, and a list of authors.

Authors will not be stored as separate entities. De-coupling and de-duplicating authors and articles could form the basis of a future project, perhaps using proprietary identifiers (such as ORCHID, PubMed Author ID or arXiv Author ID) or email addresses, but this will not be considered further in this work package.

Overall Aim

The overall aim of this work is to provide a consistent, simple and re-usable import pipeline for data for the Open Citations Corpus. In the fullness of time we’d expect it to be possible to add new data sources with minimal additional complexity. By using an approach whereby data is imported into the datastore at as early a stage as possible in the import pipeline, we can use common tools for extracting, matching, deduplicating citations; the work for each datasource, then, is just to convert the source data format into BibJSON and store it in BibServer.

Postscript

David Shotton writes: This productive collaboration between Cottage Labs and the Open Citations Corpus came to an end when Jisc funding ran out.  The corpus has more recently been given a new lease of life, as described here, with a new instantiation named OpenCitations hosted at the Department of Computer Science and Engineering of the University of Bologna, with Silvio Peroni as Co-Director.

Open Citations – Indexing PubMed Central OA data

As part of our work on the Open Citations extensions project, I have recently been doing one of my favourite things – namely indexing large quantities of data then exploring it.

On this project we are interested in the PubMed Central Open Access subset, and more specifically, we are interested in what we can do with the citation data contained within the records that are in that subset – because, as they are open access, that citation data is public and freely available.

We are building a pipeline that will enable us to easily import data from the PMC OA and from other sources such as arXiv, so that we can do great things with it like explore it in a facetview, manage and edit it in a bibserver, visualise it, and stick it in the rather cool related-work prototype software. We are building on the earlier work of both the original Open Citations project, and of the Open Bibliography projects.

Work done so far

We have spent a few weeks getting to understand the original project software and clarifying some of the goals the project should achieve; we have put together a design for a processing pipeline to get the data from source right through to where we need it, in the shape that we need it. In the case of facetview / bibserver work, this means getting it into a wonderful elasticsearch index.

While Martyn continues work on the bits and pieces for managing the pipeline as a whole and pulling data from arXiv, I have built an automated and threadable toolchain for unpacking data out of the compressed file format it arrives in from the US National Institutes of Health, parsing the XML file format and converting it into BibJSON, and then bulk loading it into an elasticsearch index. This has gone quite well.

To fully browse what we have so far, check out http://occ.cottagelabs.com.

For the code: https://github.com/opencitations/OpenCitationsCorpus/tree/master/pipeline.

The indexing process

Whilst the toolchain is capable of running threaded, the server we are using only has 2 cores and I was not sure to what extent they would be utilised, so I ran the process singular. It took five hours and ten minutes to build an index of the PMC OA subset, and we now have over 500,000 records. We can full-text search them and facet browse them.

Some things of particular interest that I learnt – I have an article in the PMC OA! And also PMIDs are not always 8 digits long – they appear in fact to be incremental from 1.

What next

At the moment there is no effort made to create record objects for the citations we find within these records, however plugging that into the toolchain is relatively straightforward now.

The full pipeline is of course still in progress, and so this work will need a wee bit of wiring into it.

Improve parsing. There are probably improvements to the parsing that we can make too, and so one of the next tasks will be to look at a few choice records and decide how better to parse them. The best way to get a look at the records for now is to use a browser like Firefox or Chrome and install the JSONview plugin, then go to occ.cottagelabs.com and have a bit of a search, then click the small blue arrows at the start of a record you are interested in to see it in full JSON straight from the index. Some further analysis on a few of these records would be a great next step, and should allow for improvements to both the data we can parse and to our representation of it.

Finish visualisations. Now that we have a good test dataset to work with, the various bits and pieces of visualisation work will be pulled together and put up on display somewhere soon. These, in addition to the search functionality already available, will enable us to answer the questions set as representative of project goals earlier in January (thanks David for those).

Postscript

David Shotton writes: This productive collaboration between Cottage Labs and the Open Citations Corpus came to an end when Jisc funding ran out.  The corpus has more recently been given a new lease of life, as described here, with a new instantiation named OpenCitations hosted at the Department of Computer Science and Engineering of the University of Bologna, with Silvio Peroni as Co-Director.

The start of something new

Previously, my blog posts relating to semantic publishing have appeared in this Open Citations Blog.

However, because of the merger of the Open Citations Project with the Related Work Project, described here, this Open Citations Blog has been renamed Open Citations and Related Work and has been opened to contributions from others involved in developing Open Citations and Related Work.

It thus makes sense that future blog posts concerned with semantic publishing be given a separate distinct home – a new Semantic Publishing Blog available at at http://semanticpublishing.wordpress.com/ – to which previous blog posts on the semantic publishing theme that originally appeared in this Open Citations Blog will be copied.

Those interested should follow both the Open Citations and Related Work Blog and the Semantic Publishing Blog, and may also be interested in the third more specialized blog concerning the on-line creation of research data management plans, entitled Creating data management plans online that is available at http://datamanagementplanning.wordpress.com/.

Postscript

David Shotton writes: With the return of Heinrich Hartmann and René Pickhardt to Germany and their involvement in other things, the potential collaboration between the Open Citations Project and their Related Works project came to nothing.  Separately, the corpus has recently been renamed OpenCitations and given a new lease of life, described here, with Silvio Peroni as Co-Director.  For this reason, this blog was re-named “OpenCitations” on 30th March 2017.

 

Open letter to publishers

[The text of this post was updated on 27-09-2013 and 04-04-2017 to reflect a new CrossRef metadata best practice document and a change in their URI.]

Today I wrote an open letter to all scholarly journal publishers, available online here, entitled:

Open your article reference lists for inclusion in the Open Citations Corpus.

In this letter, I request that publishers open the bibliographic citation data in their journal article reference lists.  There is a growing movement to make such bibliographic citation data open – for example, Nature Publishing Group’s open Linked Data Platform now includes citation metadata for all published article references.

Provided a publisher is already depositing article references with CrossRef as part of the CrossRef CitedBy Linking service, all the publisher need to do is to inform CrossRef that it is willing for CrossRef to freely distribute these reference, for example in response to queries against the CrossRef XML API.  We will then harvest them from CrossRef and incorporate them as open linked data in the Open Citations Corpus.

Nature Publishing Group, Taylor & Francis, the American Association for the Advancement of Science (who publish Science) and Oxford University Press, as well as a number of open-access publishers, have already given their consent to CrossRef to do this for some or all of their journals.

If not already a subscriber to the CrossRef CitedBy Linking service, a publisher can register for this useful service free of charge.  Having done so, there is nothing further the publisher needs to do to ‘open’ its reference data, other than to give its consent to CrossRef.  This can be done automatically, in the submitted article metadata, or (for back numbers) by informing CrossRef directly.

Even Open Access publishers, publishing articles under a CC-By open license, need to give this specific permission to CrossRef for this to occur, because CrossRef policy is that all publishers, including open access publishers, have to opt in to any distribution of references that CrossRef makes.

For new submissions, publishers should follow instructions detailed in the CrossRef blog at https://www.crossref.org/blog/distributing-references-via-crossref/, which contains the following key instruction:

“In order for publishers to distribute references along with standard bibliographic metadata, publishers need to set the <reference_distribution_opt> metadata element to “any" for each DOI deposit where they want to make references openly available.”

In this way, publishers can choose to open the reference lists for all their journals, or to do so on a journal-by-journal or on an article-by-article basis (useful for ‘hybrid’ subscription-access journals in which only some articles are open access).

To open reference lists for back numbers, publisher needs to e-mail CrossRef to express their intent,using the template shown at the foot of this post, as detailed in my Open Letter to Publishers.

I have copied this open letter to the CEOs of the Open Access Scholarly Publishers Association (OASPA), of the Association of Learned and Professional Society Publishers (ALPSP), and of the International Association of Scientific, Technical & Medical Publishers (STM), asking them to distribute it to their members, perhaps in association with their next Members News Letter, as CrossRef itself is planning to do later this month.

Please spread the word about this, particularly to publishers who may not be members of these professional associations.  Thanks.

= = =

Template for an e-mail to CrossRef expressing willingness to open reference lists in previously published and future journal articles.

To support@crossref.org

I am writing on behalf of *** [name of publisher] to confirm that *** [name of publisher] is willing for the bibliographic reference lists within the articles in [delete as necessary:] all our journals [or] the attached list of journals be made freely available by CrossRef, for inclusion in the Open Citations Corpus. These journals are associated with the following DOI prefix(es): 10.**** [Please complete DOI prefix(es) – see footnote].
Yours sincerely [name, position, date]
= = =
Footnote: Publisher’s DOI prefixes are listed at http://www.crossref.org/06members/50go-live.html by name of publisher.

 

Taylor & Francis to open article reference lists

I am very pleased to announce that last year Ian Bannerman, Managing Director for Journals at Taylor & Francis, confirmed this publisher’s willingness to pilot the opening of the reference lists from articles in 29 of their subscription access journals, as well as from all of their current list of 15 Open Access journals, for inclusion in the Open Citations Corpus.  The reference lists for these journals are already being supplied to CrossRef as part of the CrossRef CitedBy Linking service, and will be made available publicly via the CrossRef XML query API.

Taylor & Francis is a major international publisher of over 1,000 academic journals and more than 1,800 books per year, incorporating well-known publishing names including Bios Scientific Publishing, CRC Press, Garland Science, Marcel Decker and Routledge.  It is the largest publisher of subscription access journals yet to agree to make reference lists available from its journal articles, and I welcome them warmly into the Open Citations fold.

Incorporation of new reference data from Taylor and Frances journals into the Open Citations Corpus will commence in the near future.

Search OpenEdition Search

You will be redirected to OpenEdition Search