Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Open Citations – Indexing PubMed Central OA data

As part of our work on the Open Citations extensions project, I have recently been doing one of my favourite things – namely indexing large quantities of data then exploring it.

On this project we are interested in the PubMed Central Open Access subset, and more specifically, we are interested in what we can do with the citation data contained within the records that are in that subset – because, as they are open access, that citation data is public and freely available.

We are building a pipeline that will enable us to easily import data from the PMC OA and from other sources such as arXiv, so that we can do great things with it like explore it in a facetview, manage and edit it in a bibserver, visualise it, and stick it in the rather cool related-work prototype software. We are building on the earlier work of both the original Open Citations project, and of the Open Bibliography projects.

Work done so far

We have spent a few weeks getting to understand the original project software and clarifying some of the goals the project should achieve; we have put together a design for a processing pipeline to get the data from source right through to where we need it, in the shape that we need it. In the case of facetview / bibserver work, this means getting it into a wonderful elasticsearch index.

While Martyn continues work on the bits and pieces for managing the pipeline as a whole and pulling data from arXiv, I have built an automated and threadable toolchain for unpacking data out of the compressed file format it arrives in from the US National Institutes of Health, parsing the XML file format and converting it into BibJSON, and then bulk loading it into an elasticsearch index. This has gone quite well.

To fully browse what we have so far, check out http://occ.cottagelabs.com.

For the code: https://github.com/opencitations/OpenCitationsCorpus/tree/master/pipeline.

The indexing process

Whilst the toolchain is capable of running threaded, the server we are using only has 2 cores and I was not sure to what extent they would be utilised, so I ran the process singular. It took five hours and ten minutes to build an index of the PMC OA subset, and we now have over 500,000 records. We can full-text search them and facet browse them.

Some things of particular interest that I learnt – I have an article in the PMC OA! And also PMIDs are not always 8 digits long – they appear in fact to be incremental from 1.

What next

At the moment there is no effort made to create record objects for the citations we find within these records, however plugging that into the toolchain is relatively straightforward now.

The full pipeline is of course still in progress, and so this work will need a wee bit of wiring into it.

Improve parsing. There are probably improvements to the parsing that we can make too, and so one of the next tasks will be to look at a few choice records and decide how better to parse them. The best way to get a look at the records for now is to use a browser like Firefox or Chrome and install the JSONview plugin, then go to occ.cottagelabs.com and have a bit of a search, then click the small blue arrows at the start of a record you are interested in to see it in full JSON straight from the index. Some further analysis on a few of these records would be a great next step, and should allow for improvements to both the data we can parse and to our representation of it.

Finish visualisations. Now that we have a good test dataset to work with, the various bits and pieces of visualisation work will be pulled together and put up on display somewhere soon. These, in addition to the search functionality already available, will enable us to answer the questions set as representative of project goals earlier in January (thanks David for those).

Postscript

David Shotton writes: This productive collaboration between Cottage Labs and the Open Citations Corpus came to an end when Jisc funding ran out.  The corpus has more recently been given a new lease of life, as described here, with a new instantiation named OpenCitations hosted at the Department of Computer Science and Engineering of the University of Bologna, with Silvio Peroni as Co-Director.

The start of something new

Previously, my blog posts relating to semantic publishing have appeared in this Open Citations Blog.

However, because of the merger of the Open Citations Project with the Related Work Project, described here, this Open Citations Blog has been renamed Open Citations and Related Work and has been opened to contributions from others involved in developing Open Citations and Related Work.

It thus makes sense that future blog posts concerned with semantic publishing be given a separate distinct home – a new Semantic Publishing Blog available at at http://semanticpublishing.wordpress.com/ – to which previous blog posts on the semantic publishing theme that originally appeared in this Open Citations Blog will be copied.

Those interested should follow both the Open Citations and Related Work Blog and the Semantic Publishing Blog, and may also be interested in the third more specialized blog concerning the on-line creation of research data management plans, entitled Creating data management plans online that is available at http://datamanagementplanning.wordpress.com/.

Postscript

David Shotton writes: With the return of Heinrich Hartmann and René Pickhardt to Germany and their involvement in other things, the potential collaboration between the Open Citations Project and their Related Works project came to nothing.  Separately, the corpus has recently been renamed OpenCitations and given a new lease of life, described here, with Silvio Peroni as Co-Director.  For this reason, this blog was re-named “OpenCitations” on 30th March 2017.

 

Open letter to publishers

[The text of this post was updated on 27-09-2013 and 04-04-2017 to reflect a new CrossRef metadata best practice document and a change in their URI.]

Today I wrote an open letter to all scholarly journal publishers, available online here, entitled:

Open your article reference lists for inclusion in the Open Citations Corpus.

In this letter, I request that publishers open the bibliographic citation data in their journal article reference lists.  There is a growing movement to make such bibliographic citation data open – for example, Nature Publishing Group’s open Linked Data Platform now includes citation metadata for all published article references.

Provided a publisher is already depositing article references with CrossRef as part of the CrossRef CitedBy Linking service, all the publisher need to do is to inform CrossRef that it is willing for CrossRef to freely distribute these reference, for example in response to queries against the CrossRef XML API.  We will then harvest them from CrossRef and incorporate them as open linked data in the Open Citations Corpus.

Nature Publishing Group, Taylor & Francis, the American Association for the Advancement of Science (who publish Science) and Oxford University Press, as well as a number of open-access publishers, have already given their consent to CrossRef to do this for some or all of their journals.

If not already a subscriber to the CrossRef CitedBy Linking service, a publisher can register for this useful service free of charge.  Having done so, there is nothing further the publisher needs to do to ‘open’ its reference data, other than to give its consent to CrossRef.  This can be done automatically, in the submitted article metadata, or (for back numbers) by informing CrossRef directly.

Even Open Access publishers, publishing articles under a CC-By open license, need to give this specific permission to CrossRef for this to occur, because CrossRef policy is that all publishers, including open access publishers, have to opt in to any distribution of references that CrossRef makes.

For new submissions, publishers should follow instructions detailed in the CrossRef blog at https://www.crossref.org/blog/distributing-references-via-crossref/, which contains the following key instruction:

“In order for publishers to distribute references along with standard bibliographic metadata, publishers need to set the <reference_distribution_opt> metadata element to “any" for each DOI deposit where they want to make references openly available.”

In this way, publishers can choose to open the reference lists for all their journals, or to do so on a journal-by-journal or on an article-by-article basis (useful for ‘hybrid’ subscription-access journals in which only some articles are open access).

To open reference lists for back numbers, publisher needs to e-mail CrossRef to express their intent,using the template shown at the foot of this post, as detailed in my Open Letter to Publishers.

I have copied this open letter to the CEOs of the Open Access Scholarly Publishers Association (OASPA), of the Association of Learned and Professional Society Publishers (ALPSP), and of the International Association of Scientific, Technical & Medical Publishers (STM), asking them to distribute it to their members, perhaps in association with their next Members News Letter, as CrossRef itself is planning to do later this month.

Please spread the word about this, particularly to publishers who may not be members of these professional associations.  Thanks.

= = =

Template for an e-mail to CrossRef expressing willingness to open reference lists in previously published and future journal articles.

To support@crossref.org

I am writing on behalf of *** [name of publisher] to confirm that *** [name of publisher] is willing for the bibliographic reference lists within the articles in [delete as necessary:] all our journals [or] the attached list of journals be made freely available by CrossRef, for inclusion in the Open Citations Corpus. These journals are associated with the following DOI prefix(es): 10.**** [Please complete DOI prefix(es) – see footnote].
Yours sincerely [name, position, date]
= = =
Footnote: Publisher’s DOI prefixes are listed at http://www.crossref.org/06members/50go-live.html by name of publisher.

 

Taylor & Francis to open article reference lists

I am very pleased to announce that last year Ian Bannerman, Managing Director for Journals at Taylor & Francis, confirmed this publisher’s willingness to pilot the opening of the reference lists from articles in 29 of their subscription access journals, as well as from all of their current list of 15 Open Access journals, for inclusion in the Open Citations Corpus.  The reference lists for these journals are already being supplied to CrossRef as part of the CrossRef CitedBy Linking service, and will be made available publicly via the CrossRef XML query API.

Taylor & Francis is a major international publisher of over 1,000 academic journals and more than 1,800 books per year, incorporating well-known publishing names including Bios Scientific Publishing, CRC Press, Garland Science, Marcel Decker and Routledge.  It is the largest publisher of subscription access journals yet to agree to make reference lists available from its journal articles, and I welcome them warmly into the Open Citations fold.

Incorporation of new reference data from Taylor and Frances journals into the Open Citations Corpus will commence in the near future.

Open Citations Extension Project

I am pleased to announce that the JISC have funded an extension to the Open Citations Project to run from 1st August 2012 until 31st January 2013, during which we will review and revise the technology used to create the Open Citations Corpus, will update the content provided by PubMed Central, and will improve its presentation.  We will also prototype value-added services over the open citations data, to demonstrate their usefulness and to justify further funding that will permit a major expansion of the corpus in 2013 to include reference lists from subscription-access journals, including those from Nature Publishing Group, Science and other AAAS publications, and Oxford University Press.  In preparation for this, we will collaborate with CrossRef to determine how best to ingest reference data into the Open Citations Corpus in an on-going manner.

Full details are given in the Case for Support.  Watch this space!

Oxford University Press to support Open Citations

I am delighted to announce that Cathy Kennedy, OUP’s Senior Publisher for Journals, has just written to me as follows:

“Oxford University Press is delighted to support the Open Citation Corpus initiative in the interest of furthering and disseminating scholarship.  We are making the reference lists of articles in a number of journals we own available for inclusion in the Open Citations Corpus, and will be consulting further with our partner societies on extending the initiative to include their journals in due course”.

OUP thus becomes the third publisher of subscription-access journals, and the first university press, to declare its willingness to make the reference lists in journal articles publicly available.  Congratulations, OUP!

Are we starting to see the beginning of a sea change in attitudes?  Watch this space.

Open Citations and Semantic Publishing

Given the renewed interest among publishers in the Open Citations Corpus, following the decisions by Nature Publishing Group, publisher of Nature, and by the American Association for the Advancement of Science, publisher of Science, to open their citation data for inclusion in the corpus, I thought it would be helpful to provide links to videos of two conference presentations I gave that describe the Open Citations Corpus in the context of our other work in the area of semantic publishing.

The first of these was an invited contribution with the title Enriching Scientific Citations to Facilitate Knowledge Discovery, given to publishers at the Scientific, Technical and Medical Publishers Innovations Seminar 2010 entitled “Flows in Flux: how publishing technologies change the researcher’s life”, held in London, UK, on 3rd December 2010:

lecture: http://river-valley.tv/media/conferences/stm-innovation-2010/0202-David-Shotton/

slides: http://imageweb.zoo.ox.ac.uk/pub/2010/Presentations/SHOTTON_Citations_STM-Innovations-Seminar-03Dec2010.pdf

abstract: http://imageweb.zoo.ox.ac.uk/pub/2010/Presentations/STM_Innovations_Seminar_2010_ABSTRACTS.pdf

The second presentation, given almost a year later, was an invited contribution with the title The SPAR (Semantic Publishing and Referencing) Ontologies and the Open Citation Corpus, given to librarians at SWIB11 (Semantic Web in Bibliotheken; Semantic Web in Libraries Conference 2011) “Scholarly Communication in the Web of Data”, held in Hambrug, Germany on 30th November 2011:

lecture: http://www.scivee.tv/node/39208

slides: http://imageweb.zoo.ox.ac.uk/pub/2011/presentations/Shotton_SWIB11_SPARandOpenCitationCorpus_30Nov2011.pptx.pdf

abstract: http://imageweb.zoo.ox.ac.uk/pub/2011/presentations/Shotton_Abstract_for_Presentation_at_SWIB11.pdf

Further details of the Open Citations Corpus and the SPAR ontologies, and their applications are, of course, given in this Open Citations and Semantic Publishing blog, for which the following two posts provide the best introduction for those coming here for the first time:

JISC Open Citations Project – Final Project Blog Post

Introducing the Semantic Publishing and Referencing (SPAR) Ontologies

Science joins Nature in opening reference citations

Hot on the heels of my announcement on Monday that Nature Publishing Group has agreed to open its articles’ reference lists, I am delighted to announce that Science Magazine, the American Association for the Advancement of Science’s premier global science weekly, together with the other AAAS journals Science Signalling and Science Translational Medicine, will also open their articles’ reference lists, and contribute the bibliographic citations contained within these lists as open linked data to an expanded Open Citations Corpus, where they will be freely available for everyone to use in whatever manner they choose.

Preparations to expand the corpus in this manner, by integration with the reference processing pipeline of the CrossRef Cited-By Linking service, will be undertaken over the last six months of this year, and incorporation of the references from the AAAS journals into the expanded Open Citations Corpus is planned to commence in the first half of 2013.

Now that the world’s two most presigeous scientific journals have declared their hands, it is time for other subscription-access publishers to step up and join them, for the reasons given in my earlier post.  I am presently in discussion with a small number of other major STM publishers, and hope to be able to make further announcements in the near future.  However, there are many other publishers with whom I have not yet been able to make personal contact.  Those who would also like to become ‘early adopters’ of the Open Citations publishing model should contact me at <david.shotton@zoo.ox.ac.uk>.