jiscopencite – OpenCitations blog

From little acorns . . . A retrospective on OpenCitations

The initial vision

Now that OpenCitations is hosting over one billion freely available scholarly bibliographic citations, this is perhaps an opportune moment to look back to the start of this initiative. A little over eleven years ago, on 24 April 2010, I spoke at the Open Knowledge Foundation Conference, OKCon2010, in London, on the topic

OpenCitations: Publishing Bibliographic Citations as Linked Open Data

I reported that, earlier that same week, I had applied to Jisc for a one-year grant to fund the OpenCitations Project (opencitations.net). Jisc (at that time ‘The JISC’, the Joint Information Systems Committee) was tasked by the UK government, among other things, to support research and development in information technology for the benefit of the academic community.

The purpose of that original OpenCitations R&D project was to develop a prototype in which we:

harvested citations from the open access biomedical literature in PubMed Central;
described and linked them using CiTO, the Citation Typing Ontology [1];
encoded and organized them in an RDF triplestore; and
published them as Linked Open Data in the OpenCitations Corpus (OCC).

I told those at the conference that in this demonstration project, with limited JISC funding, we could not hope to “boil the whole ocean”, but that nevertheless there would be substantial benefits from even partial coverage of citation data from the scholarly literature:

We could show the way and establish best practice.
Despite partial coverage, all key papers would most likely be cited several times.
The overall topological structure of the citation network would be revealed.
We would create a ‘benchmark’ corpus of high-quality RDF citation data that could be used to develop analytical and visualization tools.
We could show the value of open citation data in helping scholars to discover full text articles of all types, and thus encourage subscription-access publishers to release their reference metadata.

The important thing, I said, was to make a start!

The Jisc OpenCitations Project

That JISC grant application was funded, and the project, to last for a year with modest funding of £100K, started in my lab in the Department of Zoology at Oxford University on 1^st June 2010, and was subsequently extended for a further six months.

Using data from the Open Access subset of PubMed Central, we created the first prototype release of the OpenCitations Corpus of linked bibliographic citation data, containing 6,529,815 independent bibliographic records of both citing and cited entities, comprising references to ~20% of all post-1980 articles recorded in PubMed, including those to all the most important highly cited papers in every field of biomedical endeavour.

This achievement was almost entirely the result of the excellent work by our chief data wrangler Alex Dutton, whose skill and natural feel for linked data did wonders for this project. Ben O’Steen, Graham Klyne and Alistair Miles made important contributions.

The project also resulted in many other development, described here, most which were developed or at least initiated during a short but wonderfully productive collaboration with Silvio Peroni, who spent six months with me in 2010 as a doctoral student intern from the University of Bologna, to which he subsequently returned to complete his thesis and develop his academic career.

These included:

the deconstruction and re-development of the original version of CiTO into a suite of orthogonal and complementary ontologies covering the whole domain of scholarly publishing – the SPAR (Semantic Publishing and Referencing) Ontologies [2, 3];
the mapping of various existing metadata schemas into RDF using SPAR, including the DataCite Metadata Schema, and subsequently JATS, now the default NISO standard for XML markup of scholarly documents) [4]; and
the initiation of the Semantic Publishing Blog and this OpenCitations Blog.

Life after Jisc – the flowering of OpenCitations

After the Jisc funding ended and I, after a long career in biological teaching and research, formally retired from the Department of Zoology at the Oxford University, members of the initial OpenCitations team moved on to other things. Like so many grant-funded academic project whose initial financial support had dried up, OpenCitations could have foundered at that stage, as an interesting prototype but with too little content to be useful. However, the concept of providing an open alternative to proprietary citation indexes was too important to abandon. But how could it be transitioned into something enduring and useful, particularly when as a matter of principle one had decided that the citation data should be made freely available, thus precluding income generation by charging for ‘premium’ services or the formation of a commercial spin-off?

Finally, I realized that something radical needed to be done to move OpenCitations forward. I had maintained a lively collaboration with Silvio Peroni at the University of Bologna, resulting between 2011 and 2014 in the publication of 18 articles and conference papers concerning the SPAR ontologies, ontology development, documentation and visualization, and related topics, and in 2015 I invited him to start working with me directly on OpenCitations. It was the best decision I could have made. We decided to take the initial concept and re-implement it from the bottom up. OpenCitations gave Silvio a major computer science project to which he could apply his considerable talent, and soon resulted in the development of a revised RDF data model for describing citation data, the OpenCitations Data Model (OCDM) [5] and a suite of new software tools to harvest, organise and publish citations at linked open data [6]. The credit for almost all the subsequent conceptual and technical developments within OpenCitations, which have incrementally led to our present situation, is due to Silvio Peroni, and the scholarly community is indebted to him for the intelligence, skill and diligent application he has given to OpenCitations over the past six years. I am truly honoured to have Silvio as co-Director of OpenCitations, and wish to take this opportunity to acknowledge his contributions and to thank him publicly.

Our work on OpenCitations at that stage, summarized in [7], would not have been possible without the enthusiastic support of Silvio’s senior colleague Fabio Vitali and of the Department of Computer Science and Engineering at the University of Bologna, which not only provided a stimulating environment for Silvio’s post-doctoral work, but also supplied computing services and infrastructure at no charge to OpenCitations. It was also greatly helped by Professor David De Roure of Oxford University, who gave me an academic home and a formal affiliation within the Oxford e-Research Centre after my retirement from the Department of Zoology, which enabled me to continue to hold research grants.

As has been documented in earlier posts in this blog, we greatly benefitted in 2017 from a grant from the Alfred P. Sloan Foundation which enabled us to purchase a new and more powerful computing infrastructure for the sole use of OpenCitations and to extend and improve our software, and subsequently in 2019 by a project grant from the Wellcome Trust to develop the Open Biomedical Citations in Context Corpus, that permitted the extension of OCDM and SPAR for the characterization of in-text references and their textual contexts.

A significant breakthrough came in January 2018 with our decision to treat citations as first-class data entities, each with its own persistent identifier (PID), the Open Citations Identifier (OCI) [8]. This gave Silvio the freedom to envision a new kind of database, a citation index in which each citation had its own metadata, including citation timespan, citation categorization (e.g. self-citation), and of course the DOIs of the citing and cited publications. The creation of this new index was possible only with the incredible effort by Ivan Heibi, who served as a Research Fellow in the project funded by the Alfred P. Sloan Foundation at that time, and who was entirely responsible for developing the first version of the code necessary for creating such a database. Having harvested all the open references from Crossref metadata dumps, Silvio and Ivan created COCI, the OpenCitations Index of Crossref DOI-to-DOI Citations, which immediately became our principal source of open citations, the original OpenCitations Corpus being retained as a ‘sandbox’ in which to experiment with new data representations, for example those required for the Open Biomedical Citations in Context Corpus. Access to COCI was facilitated by Silvio’s development of a REST API, using his software tool RAMOSE (Restful API Manager Over SPARQL Endpoints), which enables the easily configurable deployment of a REST API over any SPARQL endpoint to an RDF triplestore [9]. We were able to organize our all data, both ‘traditional’ and new, and to encode it in RDF, thanks to the comprehensive OpenCitations Data Model [5], itself based on our SPAR Ontologies [3], which we evolved as necessary to accommodate new data representation requirements.

During this period we published a number of definitions, conference papers and journal articles documenting these advances, details of which can be found here. Of these, the most recent canonical publication describing OpenCitations as an infrastructure for open scholarship, and its datasets, tools, services and activities, is Peroni and Shotton (2020) [10]. We also established the Research Centre for Open Scholarly Metadata at the University of Bologna, primarily to handle administrative, financial and academic aspects of OpenCitations activities.

OpenCitations’ future

The problem remained: how to sustain the OpenCitations infrastructure financially. We were greatly helped by Bilder, Lin and Neylon’s formulation of the Principles of Open Scholarly Infrastructures (POSI) [11], in which they clearly pointing out that reliance solely on grant funding for specific projects was not the answer. OpenCitations compliance with POSI is described here. We were thus immensely grateful that SPARC Europe and other institutions had the wisdom to establish SCOSS (The Global Sustainability Coalition for Open Science Services) to facilitate the crowd-sourced financial support of useful open infrastructures by the scholarly community, including academic libraries, government agencies and other stakeholders. OpenCitations applied for SCOSS support in 2019, which led to the selection of OpenCitations for support in the SCOSS second round.

The donations we are now starting to receive from such stakeholders, and the new staff that this funding has recently allowed us to hire, signal the start of our transition from a financially vulnerable academic project to a sustainable open scholarly infrastructure of real value to the community.

The work of opening more of the global citation graph now requires two things:

that each publisher takes responsibility for ensuring that the references from all of its journal articles and books are submitted, together with all other bibliographic metadata, to open scholarly bibliographic metadata aggregators such as Crossref and DataCite, from which they can be indexed into open citation indexes of sufficient quality, depth of detail and breadth of coverage that these offer genuine alternatives to the expensive proprietary citation indexing services upon which the academic community presently relies; and
that the entire scholarly stakeholder community re-directs a fraction of the enormous sums currently spent on its subscriptions to proprietary bibliographic services in order to support Open Science infrastructures such as OpenCitations that making citations and other forms of scholarly metadata and objects freely available.

References

[1] David Shotton (2010). CiTO, the Citation Typing Ontology. J. Biomedical Semantics 1 (Suppl. 1): S6. http://dx.doi.org/10.1186/2041-1480-1-S1-S6

[2] Silvio Peroni, David Shotton (2012). FaBiO and CiTO: ontologies for describing bibliographic resources and citations. Web Semantics, 17: 33-34. https://doi.org/10.1016/j.websem.2012.08.001, OA at http://speroni.web.cs.unibo.it/publications/peroni-2012-fabio-cito-ontologies.pdf

[3] Silvio Peroni, David Shotton (2018). The SPAR Ontologies. In Proceedings of the 17th International Semantic Web Conference (ISWC 2018): 119-136. https://doi.org/10.1007/978-3-030-00668-6_8

[4] Peroni S, Lapeyre DA and Shotton D (2012) From Markup to Linked Data: Mapping NISO JATS v1.0 to RDF using the SPAR (Semantic Publishing and Referencing) Ontologies. Proc. 2012 JATS Conference, National Library of Medicine, Bethesda, Maryland, USA (October 2012): 16-17. http://www.ncbi.nlm.nih.gov/books/NBK100491/

[5] Marilena Daquino, Silvio Peroni , David Shotton (2020). The OpenCitations Data Model. Figshare. https://doi.org/10.6084/m9.figshare.3443876.v7

[6] Silvio Peroni, David Shotton, Fabio Vitali (2017). One Year of the OpenCitations Corpus: Releasing RDF-based scholarly citation data into the Public Domain. In The Semantic Web – ISWC 2017 (Lecture Notes in Computer Science Vol. 10588, pp. 184–192). Springer, Cham. https://doi.org/10.1007/978-3-319-68204-4_19

[7] Silvio Peroni, Alexander Dutton, Tanya Gray, David Shotton (2015). Setting our bibliographic references free: towards open citation data. Journal of Documentation, 71 (2): 253-277. http://dx.doi.org/10.1108/JD-12-2013-0166, OA at http://speroni.web.cs.unibo.it/publications/peroni-2015-setting-bibliographic-references.pdf

[8] Silvio Peroni, David Shotton (2019). Open Citation Identifier: Definition. Figshare. https://doi.org/10.6084/m9.figshare.7127816

[9] Daquino, M., Heibi, I., Peroni, S., & Shotton, D. (2021). Creating Restful APIs over SPARQL endpoints with RAMOSE. Semantic Web. http://arxiv.org/abs/2007.16079

[10] Silvio Peroni, David Shotton (2020). OpenCitations, an infrastructure organization for open scholarship. Quantitative Science Studies, 1(1): 428-444. https://doi.org/10.1162/qss_a_00023

[11] Geoffrey Bilder, Jenny Lin, Cameron Neylon (2015). Principles for Open Scholarly Infrastructure. http://dx.doi.org/10.6084/m9.figshare.1314859

Workshop on Open Citations

OpenCitations [1], the EXCITE Project [2] and Europe PubMed Central [3] are pleased to announce a Workshop on Open Citations at the University of Bologna in Bologna, Italy [4] on 3-5 September – https://workshop-oc.github.io.

Format and topics

Day One and Day Two: Formal presentations and discussions on the creation, availability, uses and applications of open bibliographic citations, and of bibliometric studies based upon them. This will involve both invited and contributed presentations.
Day Three: A Hack Day on Open Citations to see what services can be prototyped using large volumes of open citation data.

We look forward to welcoming all those interested in open citation data, including researchers, computer scientists, scholarly publishers, academic administrators, research funders and policy makers. The workshop will be organised around the following topics:

Opening up citations: Initiatives, collaborations, methods and approaches for the creation of open access to bibliographic citations.
Policies and funding: Strategies, policies and mandates for promoting open access to citations, and transparency and reproducibility of research and research evaluation.
Publishers and learned societies: Approaches to, benefits of, and issues surrounding the deposit, distribution, and services for open bibliographic metadata and citations.
Projects: Metrics, visualizations and other projects. The uses and applications of open citations, and bibliometric analyses and metrics based upon them

Participation

If you would like to participate in this event, please enter your details and a mini-biography at https://bit.ly/2JEd0r9.

If additionally you would like to speak at this event, please select the session your talk would fit in, and provide a title and abstract for your talk. Selected submissions will involve either a talk of 10 minute duration, with a 5 minute discussion following, or a poster presentation.

The deadline for application and submission of these details is the 20th of May 2018. Notifications of acceptance will be sent to you by the 1st of June. In case of oversubscriptions, the organisers will select speakers and attendees from among those have applied.

Please circulate this invitation to any colleagues that you think might be interested in attending this workshop.

Fees and expenses

For those accepted, we request a small registration fee of €100 to cover expenses (€25 for students), which must be paid by the 15th of July 2018, but which will be waived in cases of financial hardship. Details of how to pay will be sent to those accepted. Registration for the Hack Day is free of charge.

Those attending will be expected to cover the cost of their own travel and accommodation. We will provide a list of recommended hotels and guest houses. We will provide lunches and refreshments on all days, and a free workshop dinner on Monday 3rd September.

Organizers

David Shotton (OpenCitations)
Jo McEntyre (Europe PMC)
Maria Levchenko (Europe PMC)
Marilena Daquino (University of Bologna)
Philipp Mayr (EXCITE project)
Silvio Peroni (OpenCitations)
Steffen Staab (EXCITE project)

Links

[1] OpenCitations – http://opencitations.net
[2] EXCITE Project – http://west.uni-koblenz.de/en/research/excite
[3] Europe PMC – https://europepmc.org/About
[4] University of Bologna – http://www.unibo.it/en/homepage

Open Citations Corpus Import Process

As part of the Open Citations project, we have been asked to review and improve the process of importing data into the Open Citations Corpus, taking the scripts from the initial project as our starting point.

The current import procedure evolved from several disconnected processes and requires running multiple command line scripts and transforming the data into different intermediate formats. As a consequence, it is not very efficient and we will be looking to improve on the speed and reliability of the import procedure. Moreover, there are two distinct procedures depending on the source of the data (arXiv or PubMed Central); we are hoping to unify the common parts of these procedures into a single process which can be simplified and normalised to improve code re-use and comprehensibility.

The Workflow

As PubMed Central provides an OAI-PMH feed, this could be used to retrieve article metadata, and for some articles, full text. Using this feed, rather than an FTP download (as used currently) would allow the metadata import for both arXiv and PubMed Central to follow a near-identical process, as we are already using the OAI-PMH feed for arXiv.

Also, rather than have intermediate databases and information stores, it would be cleaner to import from the information source straight into a datastore. The datastore could then be queried, allowing matches and linking between articles to be performed in situ. The process would therefore become:

Pull new metadata from arXiv (OAI-PMH) and PubMed Central (OAI-PMH) and insert new records into the Open Citations Corpus datastore
Pull new full-text from arXiv and PubMed Central, extract citations, and match with article data in Open Citations server, creating links between these references and the metadata records for the cited articles. Store unmatched citations as nested records in the metadata for each article.
On a scheduled basis (e.g. nightly), review each existing article’s unmatched citations and attempt to match these with existing bibliographic records of other articles.

In outline, this looks like this:

The Datastore

Neo4J is currently used as the final Open Citations Corpus datastore for the arXiv data, by the Related Work system. We propose instead to use BibServer as the final datastore, for its flexibility and scalability, and suitability for the Open Citations use cases.

The Data Structure

The data stored within BibServer as BibJSON will be a collection of linked bibliographic records describing articles. Associated with each record and stored as nested data will be a list of matched citations (i.e. those for which the Open Citations Corpus has a bibliographic record), a list of unmatched citations, and a list of authors.

Authors will not be stored as separate entities. De-coupling and de-duplicating authors and articles could form the basis of a future project, perhaps using proprietary identifiers (such as ORCHID, PubMed Author ID or arXiv Author ID) or email addresses, but this will not be considered further in this work package.

Overall Aim

The overall aim of this work is to provide a consistent, simple and re-usable import pipeline for data for the Open Citations Corpus. In the fullness of time we’d expect it to be possible to add new data sources with minimal additional complexity. By using an approach whereby data is imported into the datastore at as early a stage as possible in the import pipeline, we can use common tools for extracting, matching, deduplicating citations; the work for each datasource, then, is just to convert the source data format into BibJSON and store it in BibServer.

Postscript

David Shotton writes: This productive collaboration between Cottage Labs and the Open Citations Corpus came to an end when Jisc funding ran out. The corpus has more recently been given a new lease of life, as described here, with a new instantiation named OpenCitations hosted at the Department of Computer Science and Engineering of the University of Bologna, with Silvio Peroni as Co-Director.

Open Citations – Indexing PubMed Central OA data

As part of our work on the Open Citations extensions project, I have recently been doing one of my favourite things – namely indexing large quantities of data then exploring it.

On this project we are interested in the PubMed Central Open Access subset, and more specifically, we are interested in what we can do with the citation data contained within the records that are in that subset – because, as they are open access, that citation data is public and freely available.

We are building a pipeline that will enable us to easily import data from the PMC OA and from other sources such as arXiv, so that we can do great things with it like explore it in a facetview, manage and edit it in a bibserver, visualise it, and stick it in the rather cool related-work prototype software. We are building on the earlier work of both the original Open Citations project, and of the Open Bibliography projects.

Work done so far

We have spent a few weeks getting to understand the original project software and clarifying some of the goals the project should achieve; we have put together a design for a processing pipeline to get the data from source right through to where we need it, in the shape that we need it. In the case of facetview / bibserver work, this means getting it into a wonderful elasticsearch index.

While Martyn continues work on the bits and pieces for managing the pipeline as a whole and pulling data from arXiv, I have built an automated and threadable toolchain for unpacking data out of the compressed file format it arrives in from the US National Institutes of Health, parsing the XML file format and converting it into BibJSON, and then bulk loading it into an elasticsearch index. This has gone quite well.

To fully browse what we have so far, check out http://occ.cottagelabs.com.

For the code: https://github.com/opencitations/OpenCitationsCorpus/tree/master/pipeline.

The indexing process

Whilst the toolchain is capable of running threaded, the server we are using only has 2 cores and I was not sure to what extent they would be utilised, so I ran the process singular. It took five hours and ten minutes to build an index of the PMC OA subset, and we now have over 500,000 records. We can full-text search them and facet browse them.

Some things of particular interest that I learnt – I have an article in the PMC OA! And also PMIDs are not always 8 digits long – they appear in fact to be incremental from 1.

What next

At the moment there is no effort made to create record objects for the citations we find within these records, however plugging that into the toolchain is relatively straightforward now.

The full pipeline is of course still in progress, and so this work will need a wee bit of wiring into it.

Improve parsing. There are probably improvements to the parsing that we can make too, and so one of the next tasks will be to look at a few choice records and decide how better to parse them. The best way to get a look at the records for now is to use a browser like Firefox or Chrome and install the JSONview plugin, then go to occ.cottagelabs.com and have a bit of a search, then click the small blue arrows at the start of a record you are interested in to see it in full JSON straight from the index. Some further analysis on a few of these records would be a great next step, and should allow for improvements to both the data we can parse and to our representation of it.

Finish visualisations. Now that we have a good test dataset to work with, the various bits and pieces of visualisation work will be pulled together and put up on display somewhere soon. These, in addition to the search functionality already available, will enable us to answer the questions set as representative of project goals earlier in January (thanks David for those).

Postscript

Open Citations developments

Following the decision to merge the planned development of the Open Citations Corpus with the Related Work Project, described in the previous blog post, we proposed to develop one underlying data store, one data model and one RDF representation.

Infrastructure

We are presently creating a new OpenStack server environment on one of our Zoology Department Web servers, will host the data storage system, to be called the Open Citations Corpus Datastore (OCCD), the reference processing software, and the web server providing web sites, blog hosting and the Related Work user interfaces to the underlying data within the new enhanced Open Citations Corpus. This will be available for use shortly.

Input data

We anticipate three primary data streams into the new expanded Open Citations Corpus, from PubMed Central, from ArXiV and from CrossRef.

PubMed Central has more than doubled the number of its Open Access full-text papers since they were last harvested into the Open Citations Corpus early in 2011. There are now over 530,000 articles in the Open Access Subset, the majority of which are new for the Open Citations Corpus. These contain approx. 16 million references to approx. 9 million individual cited papers. The PubMed Central articles have rich metadata, marked up using the National Library of Medicine DTD v3. The current Open Access Corpus from PubMed Central has now been downloaded, prior to processing.

ArXiV has approximately 750,000 papers, containing references to approx. 16 million papers, about 2 million of which are within ArXiV itself. These are held in LaTeX format, and are accompanied by basic Dublin Core elements metadata. The ArXiV corpus was downloaded in October, and will be updated in the near future.

CrossRef holds the reference lists for approx. 18.8 million articles provided by publishers participating in the CrossRef ‘CitedBy Linking’ service. These are marked up in XML according to the CrossRef data schema that differs from the NLM DTD. Access to these reference lists via CrossRef will require specific permission from the publishers concerned, that is currently being sought, as detailed in a recent blog post. Trial downloads of CrossRef-derived reference lists will begin shortly.

New data model

The metadata requirements for the revised Open Citations Corpus, closely based on the data model originally developed by Alex Dutton during the initial JISC Open Citations Project, is now available online at http://goo.gl/eeDe8.

One task that we have already undertaken is to determine from each of the anticipated data input streams those metadata entities (e.g. title) that are already clearly marked up and can be recognisable automatically in the input data from each source – these we call ‘raw’ metadata terms – and those (e.g. author affiliation) that will have to be deduced or parsed out from the text of the input source – referred to as ‘derived’ metadata terms. A spreadsheet showing these mapping is available at http://goo.gl/XTlQ1.

Data format

The native format chosen for storing the reference data within is BibJSON, and the mapping from BibJSON to RDF will be achieved using an RDF metadata mapping that closely reflects that already in use for the original Open Citations Corpus, currently being finalized. BibServer will be used as the underlying storage technology for the OCCD, which handles BibJSON natively.

Data processing

The existing data processing pipeline developed for the Open Citations Corpus is currently being reviewed to handle the new data, and to modularize it and fit it into a messaging system, which will allow simultaneous execution and status controlling of the various tasks that have to be performed at import.

Technical developments on the unified data store and the unified processing pipeline will be undertaken by intimate collaboration between Cottage Labs and Related Work staff, that has already commenced. These tasks include:

Zip extraction.
Checking the incoming data for each article from each source for completeness and correctness according to the supplier’s data schema.
Extraction of the article’s bibliographic metadata and its reference list.
Conversion from the input format into the common BibJSON format.
Data clean-up – correction of errors within the references.
Data augmentation, by use of Web services accessing information from PubMed, CrossRef and other authorities for bibliographic records and DOI identifiers. (Well over half of references in published reference lists lack DOIs – if these exist, they will be found and added).
Matching of citation targets: many papers cite the same target article in slightly different ways. We will determine whether or not references are to a unique target, and then ensure that the correct bibliographic record for that cited article is used.
Conversion of the BibJSON records to RDF using an XSLT transformation.
Authoritative versions of semantically described bibliographic records for the citing and cited articles, together with the citation links, will then be held in both BibJSON and RDF formats in the Open Citations Corpus Datastore, to be used by front-end Related Work services.

Data indexing

The bibliographic reference information will then be indexed and held in an Elastic Search system, which deals natively with JSON data. A variety of indexes will be built, to provide rapid response times for the user-facing services.

User interfaces

The interface provided to data in the original Open Citations Corpus was limited in functionality.

During the short Open Citations Extension Project, which extends into 2013, we will experiment with two sets of user interfaces, that will be developed and maintained over the new data, to permit a ‘show and tell’ comparison at the end of the project, before deciding on which of these demonstration interfaces we will retain for the on-going service.

First, the BibServer software developed to use Elastic Search by Cottage Labs during the JISC Open Bibliography Project will be employed to provide faceted search and browse over the data.

Additionally, the user interfaces currently under development for Related Work will be employed, using additional text-mining activities ‘under the hood’. These will have an emphasis on social media interactions, with personalized auto-completion and instant filtering of reference lists. This will permit the user to move from a selected paper to semantically related papers, to find other papers by the same author, to look at co-authorship and self-citation patterns, etc.

Finally, we hope to develop some time-line visualizations of citation networks.

We aim to provide a clean separation between the underlying data and the services built on top, using RESTful APIs, providing maximum flexibility for future development.

Web site

For the time being, until our new combined project server becomes available, the existing Open Citations Corpus and Related Work web sites will be kept active on their existing separate servers, but will eventually be phased out.

Blogs

At present, each project (Open Citations and Related Work) has its own blog. This will change. I will open this blog to contributions from other members of the Open Citations and Related Works team, and will rename it Open Citations and Related Work. In addition, I will create a new blog, named Semantic Publishing and available at http://semanticpublishing.wordpress.com/, and will use that to host all new posts concerning semantic publishing and related topics that do not specifically relate to our joint open citations work. Previous semantic publishing posts within this blog will also be copied to the new Semantic Publishing blog.

Further plans

Further development of Related Work and the Open Citations Corpus will be dependent upon additional funding, for which we will apply early this year.

There are two developments that seem particularly beneficial to pursue in the near future:

Improvements of the reference string matching for the bibliographic reference strings that appear in reference lists, for which DOIs or other identifiers are not given, so that the correct bibliographic record for the cited paper can be associated with the reference string. At present, only two million out of the 16 million reference targets in the ArXiV data are correctly matched in this way.
User generated annotations, particularly provision of CiTO-based citation typing to characterize why one paper cites another. Tanya Gray has already created a plug-in to Chrome that permits this over native PubMed Central articles, and we are seeking to adapt this to use as a Related Work service with any browser.

Future scope

While our initial data sources have been in the physical and biomedical sciences, respectively, we hope to expand our services for people in the social sciences, humanities and arts, and provide broad coverage in these other disciplines, if we can tap into open sources of bibliographic reference lists from publications in these areas.

Open Citations and Related Work projects merge

What is Related Work?

Related Work is a user-friendly Web application developed to provide a means of browsing citation links, as the basis for a planned recommendation service for the best articles to read. The references underpinning Related Work are those contained within the preprints held by the widely-used Cornell preprint server ArXiV, comprising some 750,000 articles in the fields of mathematics, physics, astronomy and the other ‘hard sciences’.

Related Work was initiated in March 2012 by Dr Heinrich Hartmann and René Pickhardt, two mathematicians with strong software development skills and interests in social media, after brainstorming about what new services over the ArXiV papers they would like to use. It was the lack of basic features, like related work recommendations and a discussion system, that led them to start their own website around the ArXiV content. It quickly became apparent that access to reference data was absolutely crucial for building a sensible recommendation service.

Following bulk download of the open-access ArXiV articles, the bibliographic references were extracted from each article and placed in a Neo4J database. (Neo4J is the leading NOSQL graph database, good for storing document nodes and links between them.)

On top of these data, a demonstration Related Work web interface was developed, permitting users to search by title, keyword or author name. Enhancements to this interface are already in prototype. That work was completed in October 2012, and was coupled with full release of the underlying data and the source code under open licenses. Related Work does not presently expose its content in RDF as open linked data.

Collaboration between Related Work and the Open Citations Project

The collaboration between Related Work and the Open Citations Project started in October 2012, when Heinrich’s search for new sources for citation data led him to the Open Citations Corpus. We started talking about how we could support each other’s work, and soon discovered that we were facing very similar problems and had a shared interest in the semantic enhancement of citation data.

In November we decided that, because our aims of making bibliographic reference data openly available were so well aligned, and because our underlying data were so similar, we should immediately merge the on-going developments of the Open Citation Corpus and the Related Work project. This would maximize the effectiveness of the limited developer effort that we had available to take this work forward.

This decision immediately promised to bring the Open Citations Corpus a doubling in the volume of our citation data, the input of the two experienced and committed web developers from Related Work (namely Heinrich and René), and a close link to a leading semantic web research group within the Institute for Web Science and Technologies at the University of Koblenz-Landau, at which they both work. That relationship bears an additional potential of getting computer science students in Koblenz involved with the project. Indeed, an undergraduate research project on personalized auto-completion for Related Work is currently being supervised by René.

Reciprocally, the development effort and prior experience of Richard Jones, Ben O’Steen, Mark MacGillivray and their colleagues at Cottage Labs, developed during the Open Bibliography Project and currently being applied within the Open Citations Extension Project, provided solutions to problems being faced by Related Work, particularly relating to BibJSON and BibServer for handling bibliographic records, and to Cottage Lab’s front-end faceted search and browse software Edjo, which works over both Apache Solr and ElasticSearch indexes.

Branding

For branding, at least in the short term, we proposed to keep the name “Open Citations Corpus” for the underlying data corpus and its infrastructure, the Open Citations Corpus Datastore (OCCD), and the name “Related Work” for the user-oriented services built on top of these citation data.

The following blog post explains how this joint project is now being taken forward.

Postscript

David Shotton writes: With the return of Heinrich Hartmann and René Pickhardt to Germany and their involvement in other things, the potential collaboration between the Open Citations Project and their Related Works project came to nothing. Separately, the corpus has recently been renamed OpenCitations and given a new lease of life, described here, with Silvio Peroni as Co-Director. For this reason, this blog was re-named “OpenCitations” on 30th March 2017.

Open letter to publishers

[The text of this post was updated on 27-09-2013 and 04-04-2017 to reflect a new CrossRef metadata best practice document and a change in their URI.]

Today I wrote an open letter to all scholarly journal publishers, available online here, entitled:

Open your article reference lists for inclusion in the Open Citations Corpus.

In this letter, I request that publishers open the bibliographic citation data in their journal article reference lists. There is a growing movement to make such bibliographic citation data open – for example, Nature Publishing Group’s open Linked Data Platform now includes citation metadata for all published article references.

Provided a publisher is already depositing article references with CrossRef as part of the CrossRef CitedBy Linking service, all the publisher need to do is to inform CrossRef that it is willing for CrossRef to freely distribute these reference, for example in response to queries against the CrossRef XML API. We will then harvest them from CrossRef and incorporate them as open linked data in the Open Citations Corpus.

Nature Publishing Group, Taylor & Francis, the American Association for the Advancement of Science (who publish Science) and Oxford University Press, as well as a number of open-access publishers, have already given their consent to CrossRef to do this for some or all of their journals.

If not already a subscriber to the CrossRef CitedBy Linking service, a publisher can register for this useful service free of charge. Having done so, there is nothing further the publisher needs to do to ‘open’ its reference data, other than to give its consent to CrossRef. This can be done automatically, in the submitted article metadata, or (for back numbers) by informing CrossRef directly.

Even Open Access publishers, publishing articles under a CC-By open license, need to give this specific permission to CrossRef for this to occur, because CrossRef policy is that all publishers, including open access publishers, have to opt in to any distribution of references that CrossRef makes.

For new submissions, publishers should follow instructions detailed in the CrossRef blog at https://www.crossref.org/blog/distributing-references-via-crossref/, which contains the following key instruction:

“In order for publishers to distribute references along with standard bibliographic metadata, publishers need to set the <reference_distribution_opt> metadata element to “any" for each DOI deposit where they want to make references openly available.”

In this way, publishers can choose to open the reference lists for all their journals, or to do so on a journal-by-journal or on an article-by-article basis (useful for ‘hybrid’ subscription-access journals in which only some articles are open access).

To open reference lists for back numbers, publisher needs to e-mail CrossRef to express their intent,using the template shown at the foot of this post, as detailed in my Open Letter to Publishers.

I have copied this open letter to the CEOs of the Open Access Scholarly Publishers Association (OASPA), of the Association of Learned and Professional Society Publishers (ALPSP), and of the International Association of Scientific, Technical & Medical Publishers (STM), asking them to distribute it to their members, perhaps in association with their next Members News Letter, as CrossRef itself is planning to do later this month.

Please spread the word about this, particularly to publishers who may not be members of these professional associations. Thanks.

= = =

Template for an e-mail to CrossRef expressing willingness to open reference lists in previously published and future journal articles.

To support@crossref.org

I am writing on behalf of *** [name of publisher] to confirm that *** [name of publisher] is willing for the bibliographic reference lists within the articles in [delete as necessary:] all our journals [or] the attached list of journals be made freely available by CrossRef, for inclusion in the Open Citations Corpus. These journals are associated with the following DOI prefix(es): 10.**** [Please complete DOI prefix(es) – see footnote].
Yours sincerely [name, position, date]
= = =
Footnote: Publisher’s DOI prefixes are listed at http://www.crossref.org/06members/50go-live.html by name of publisher.

Taylor & Francis to open article reference lists

I am very pleased to announce that last year Ian Bannerman, Managing Director for Journals at Taylor & Francis, confirmed this publisher’s willingness to pilot the opening of the reference lists from articles in 29 of their subscription access journals, as well as from all of their current list of 15 Open Access journals, for inclusion in the Open Citations Corpus. The reference lists for these journals are already being supplied to CrossRef as part of the CrossRef CitedBy Linking service, and will be made available publicly via the CrossRef XML query API.

Taylor & Francis is a major international publisher of over 1,000 academic journals and more than 1,800 books per year, incorporating well-known publishing names including Bios Scientific Publishing, CRC Press, Garland Science, Marcel Decker and Routledge. It is the largest publisher of subscription access journals yet to agree to make reference lists available from its journal articles, and I welcome them warmly into the Open Citations fold.

Incorporation of new reference data from Taylor and Frances journals into the Open Citations Corpus will commence in the near future.

Open Citations Extension Project

I am pleased to announce that the JISC have funded an extension to the Open Citations Project to run from 1st August 2012 until 31st January 2013, during which we will review and revise the technology used to create the Open Citations Corpus, will update the content provided by PubMed Central, and will improve its presentation. We will also prototype value-added services over the open citations data, to demonstrate their usefulness and to justify further funding that will permit a major expansion of the corpus in 2013 to include reference lists from subscription-access journals, including those from Nature Publishing Group, Science and other AAAS publications, and Oxford University Press. In preparation for this, we will collaborate with CrossRef to determine how best to ingest reference data into the Open Citations Corpus in an on-going manner.

Full details are given in the Case for Support. Watch this space!

Oxford University Press to support Open Citations

I am delighted to announce that Cathy Kennedy, OUP’s Senior Publisher for Journals, has just written to me as follows:

“Oxford University Press is delighted to support the Open Citation Corpus initiative in the interest of furthering and disseminating scholarship. We are making the reference lists of articles in a number of journals we own available for inclusion in the Open Citations Corpus, and will be consulting further with our partner societies on extending the initiative to include their journals in due course”.

OUP thus becomes the third publisher of subscription-access journals, and the first university press, to declare its willingness to make the reference lists in journal articles publicly available. Congratulations, OUP!

Are we starting to see the beginning of a sea change in attitudes? Watch this space.