Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Tag: BibJSON

Open Citations Corpus Import Process

As part of the Open Citations project, we have been asked to review and improve the process of importing data into the Open Citations Corpus, taking the scripts from the initial project as our starting point.

The current import procedure evolved from several disconnected processes and requires running multiple command line scripts and transforming the data into different intermediate formats. As a consequence, it is not very efficient and we will be looking to improve on the speed and reliability of the import procedure. Moreover, there are two distinct procedures depending on the source of the data (arXiv or PubMed Central); we are hoping to unify the common parts of these procedures into a single process which can be simplified and normalised to improve code re-use and comprehensibility.

The Workflow

As PubMed Central provides an OAI-PMH feed, this could be used to retrieve article metadata, and for some articles, full text. Using this feed, rather than an FTP download (as used currently) would allow the metadata import for both arXiv and PubMed Central to follow a near-identical process, as we are already using the OAI-PMH feed for arXiv.

Also, rather than have intermediate databases and information stores, it would be cleaner to import from the information source straight into a datastore. The datastore could then be queried, allowing matches and linking between articles to be performed in situ. The process would therefore become:

  1. Pull new metadata from arXiv (OAI-PMH) and PubMed Central (OAI-PMH) and insert new records into the Open Citations Corpus datastore
  2. Pull new full-text from arXiv and PubMed Central, extract citations, and match with article data in Open Citations server, creating links between these references and the metadata records for the cited articles. Store unmatched citations as nested records in the metadata for each article.
  3. On a scheduled basis (e.g. nightly), review each existing article’s unmatched citations and attempt to match these with existing bibliographic records of other articles.

In outline, this looks like this:

The Datastore

Neo4J is currently used as the final Open Citations Corpus datastore for the arXiv data, by the Related Work system. We propose instead to use BibServer as the final datastore, for its flexibility and scalability, and suitability for the Open Citations use cases.

The Data Structure

The data stored within BibServer as BibJSON will be a collection of linked bibliographic records describing articles. Associated with each record and stored as nested data will be a list of matched citations (i.e. those for which the Open Citations Corpus has a bibliographic record), a list of unmatched citations, and a list of authors.

Authors will not be stored as separate entities. De-coupling and de-duplicating authors and articles could form the basis of a future project, perhaps using proprietary identifiers (such as ORCHID, PubMed Author ID or arXiv Author ID) or email addresses, but this will not be considered further in this work package.

Overall Aim

The overall aim of this work is to provide a consistent, simple and re-usable import pipeline for data for the Open Citations Corpus. In the fullness of time we’d expect it to be possible to add new data sources with minimal additional complexity. By using an approach whereby data is imported into the datastore at as early a stage as possible in the import pipeline, we can use common tools for extracting, matching, deduplicating citations; the work for each datasource, then, is just to convert the source data format into BibJSON and store it in BibServer.

Postscript

David Shotton writes: This productive collaboration between Cottage Labs and the Open Citations Corpus came to an end when Jisc funding ran out.  The corpus has more recently been given a new lease of life, as described here, with a new instantiation named OpenCitations hosted at the Department of Computer Science and Engineering of the University of Bologna, with Silvio Peroni as Co-Director.

Open Citations – Indexing PubMed Central OA data

As part of our work on the Open Citations extensions project, I have recently been doing one of my favourite things – namely indexing large quantities of data then exploring it.

On this project we are interested in the PubMed Central Open Access subset, and more specifically, we are interested in what we can do with the citation data contained within the records that are in that subset – because, as they are open access, that citation data is public and freely available.

We are building a pipeline that will enable us to easily import data from the PMC OA and from other sources such as arXiv, so that we can do great things with it like explore it in a facetview, manage and edit it in a bibserver, visualise it, and stick it in the rather cool related-work prototype software. We are building on the earlier work of both the original Open Citations project, and of the Open Bibliography projects.

Work done so far

We have spent a few weeks getting to understand the original project software and clarifying some of the goals the project should achieve; we have put together a design for a processing pipeline to get the data from source right through to where we need it, in the shape that we need it. In the case of facetview / bibserver work, this means getting it into a wonderful elasticsearch index.

While Martyn continues work on the bits and pieces for managing the pipeline as a whole and pulling data from arXiv, I have built an automated and threadable toolchain for unpacking data out of the compressed file format it arrives in from the US National Institutes of Health, parsing the XML file format and converting it into BibJSON, and then bulk loading it into an elasticsearch index. This has gone quite well.

To fully browse what we have so far, check out http://occ.cottagelabs.com.

For the code: https://github.com/opencitations/OpenCitationsCorpus/tree/master/pipeline.

The indexing process

Whilst the toolchain is capable of running threaded, the server we are using only has 2 cores and I was not sure to what extent they would be utilised, so I ran the process singular. It took five hours and ten minutes to build an index of the PMC OA subset, and we now have over 500,000 records. We can full-text search them and facet browse them.

Some things of particular interest that I learnt – I have an article in the PMC OA! And also PMIDs are not always 8 digits long – they appear in fact to be incremental from 1.

What next

At the moment there is no effort made to create record objects for the citations we find within these records, however plugging that into the toolchain is relatively straightforward now.

The full pipeline is of course still in progress, and so this work will need a wee bit of wiring into it.

Improve parsing. There are probably improvements to the parsing that we can make too, and so one of the next tasks will be to look at a few choice records and decide how better to parse them. The best way to get a look at the records for now is to use a browser like Firefox or Chrome and install the JSONview plugin, then go to occ.cottagelabs.com and have a bit of a search, then click the small blue arrows at the start of a record you are interested in to see it in full JSON straight from the index. Some further analysis on a few of these records would be a great next step, and should allow for improvements to both the data we can parse and to our representation of it.

Finish visualisations. Now that we have a good test dataset to work with, the various bits and pieces of visualisation work will be pulled together and put up on display somewhere soon. These, in addition to the search functionality already available, will enable us to answer the questions set as representative of project goals earlier in January (thanks David for those).

Postscript

David Shotton writes: This productive collaboration between Cottage Labs and the Open Citations Corpus came to an end when Jisc funding ran out.  The corpus has more recently been given a new lease of life, as described here, with a new instantiation named OpenCitations hosted at the Department of Computer Science and Engineering of the University of Bologna, with Silvio Peroni as Co-Director.

Open Citations developments

Following the decision to merge the planned development of the Open Citations Corpus with the Related Work Project, described in the previous blog post, we proposed to develop one underlying data store, one data model and one RDF representation.

Infrastructure

We are presently creating a new OpenStack server environment on one of our Zoology Department Web servers, will host the data storage system, to be called the Open Citations Corpus Datastore (OCCD), the reference processing software, and the web server providing web sites, blog hosting and the Related Work user interfaces to the underlying data within the new enhanced Open Citations Corpus.  This will be available for use shortly.

Input data

We anticipate three primary data streams into the new expanded Open Citations Corpus, from PubMed Central, from ArXiV and from CrossRef.

PubMed Central has more than doubled the number of its Open Access full-text papers since they were last harvested into the Open Citations Corpus early in 2011.  There are now over 530,000 articles in the Open Access Subset, the majority of which are new for the Open Citations Corpus.  These contain approx. 16 million references to approx. 9 million individual cited papers.  The PubMed Central articles have rich metadata, marked up using the National Library of Medicine DTD v3.  The current Open Access Corpus from PubMed Central has now been downloaded, prior to processing.

ArXiV has approximately 750,000 papers, containing references to approx. 16 million papers, about 2 million of which are within ArXiV itself.  These are held in LaTeX format, and are accompanied by basic Dublin Core elements metadata.  The ArXiV corpus was downloaded in October, and will be updated in the near future.

CrossRef holds the reference lists for approx. 18.8 million articles provided by publishers participating in the CrossRef ‘CitedBy Linking’ service.  These are marked up in XML according to the CrossRef data schema that differs from the NLM DTD.  Access to these reference lists via CrossRef will require specific permission from the publishers concerned, that is currently being sought, as detailed in a recent blog post.  Trial downloads of CrossRef-derived reference lists will begin shortly.

New data model

The metadata requirements for the revised Open Citations Corpus, closely based on the data model originally developed by Alex Dutton during the initial JISC Open Citations Project, is now available online at http://goo.gl/eeDe8.

One task that we have already undertaken is to determine from each of the anticipated data input streams those metadata entities (e.g. title) that are already clearly marked up and can be recognisable automatically in the input data from each source – these we call ‘raw’ metadata terms – and those (e.g. author affiliation) that will have to be deduced or parsed out from the text of the input source – referred to as ‘derived’ metadata terms. A spreadsheet showing these mapping is available at http://goo.gl/XTlQ1.

Data format

The native format chosen for storing the reference data within is BibJSON, and the mapping from BibJSON to RDF will be achieved using an RDF metadata mapping that closely reflects that already in use for the original Open Citations Corpus, currently being finalized.  BibServer will be used as the underlying storage technology for the OCCD, which handles BibJSON natively.

Data processing

The existing data processing pipeline developed for the Open Citations Corpus is currently being reviewed to handle the new data, and to modularize it and fit it into a messaging system, which will allow simultaneous execution and status controlling of the various tasks that have to be performed at import.

Technical developments on the unified data store and the unified processing pipeline will be undertaken by intimate collaboration between Cottage Labs and Related Work staff, that has already commenced.  These tasks include:

  1. Zip extraction.
  2. Checking the incoming data for each article from each source for completeness and correctness according to the supplier’s data schema.
  3. Extraction of the article’s bibliographic metadata and its reference list.
  4. Conversion from the input format into the common BibJSON format.
  5. Data clean-up – correction of errors within the references.
  6. Data augmentation, by use of Web services accessing information from PubMed, CrossRef and other authorities for bibliographic records and DOI identifiers. (Well over half of references in published reference lists lack DOIs – if these exist, they will be found and added).
  7. Matching of citation targets: many papers cite the same target article in slightly different ways.  We will determine whether or not references are to a unique target, and then ensure that the correct bibliographic record for that cited article is used.
  8. Conversion of the BibJSON records to RDF using an XSLT transformation.
  9. Authoritative versions of semantically described bibliographic records for the citing and cited articles, together with the citation links, will then be held in both BibJSON and RDF formats in the Open Citations Corpus Datastore, to be used by front-end Related Work services.

Data indexing

The bibliographic reference information will then be indexed and held in an Elastic Search system, which deals natively with JSON data.  A variety of indexes will be built, to provide rapid response times for the user-facing services.

User interfaces

The interface provided to data in the original Open Citations Corpus was limited in functionality.

During the short Open Citations Extension Project, which extends into 2013, we will experiment with two sets of user interfaces, that will be developed and maintained over the new data, to permit a ‘show and tell’ comparison at the end of the project, before deciding on which of these demonstration interfaces we will retain for the on-going service.

First, the BibServer software developed to use Elastic Search by Cottage Labs during the JISC Open Bibliography Project will be employed to provide faceted search and browse over the data.

Additionally, the user interfaces currently under development for Related Work will be employed, using additional text-mining activities ‘under the hood’.  These will have an emphasis on social media interactions, with personalized auto-completion and instant filtering of reference lists.  This will permit the user to move from a selected paper to semantically related papers, to find other papers by the same author, to look at co-authorship and self-citation patterns, etc.

Finally, we hope to develop some time-line visualizations of citation networks.

We aim to provide a clean separation between the underlying data and the services built on top, using RESTful APIs, providing maximum flexibility for future development.

Web site

For the time being, until our new combined project server becomes available, the existing Open Citations Corpus and Related Work web sites will be kept active on their existing separate servers, but will eventually be phased out.

Blogs

At present, each project (Open Citations and Related Work) has its own blog.  This will change. I will open this blog to contributions from other members of the Open Citations and Related Works team, and will rename it Open Citations and Related Work. In addition, I will create a new blog, named Semantic Publishing and available at http://semanticpublishing.wordpress.com/, and will use that to host all new posts concerning semantic publishing and related topics that do not specifically relate to our joint open citations work. Previous semantic publishing posts within this blog will also be copied to the new Semantic Publishing blog.

Further plans

Further development of Related Work and the Open Citations Corpus will be dependent upon additional funding, for which we will apply early this year.

There are two developments that seem particularly beneficial to pursue in the near future:

  1. Improvements of the reference string matching for the bibliographic reference strings that appear in reference lists, for which DOIs or other identifiers are not given, so that the correct bibliographic record for the cited paper can be associated with the reference string.  At present, only two million out of the 16 million reference targets in the ArXiV data are correctly matched in this way.
  2. User generated annotations, particularly provision of CiTO-based citation typing to characterize why one paper cites another.  Tanya Gray has already created a plug-in to Chrome that permits this over native PubMed Central articles, and we are seeking to adapt this to use as a Related Work service with any browser.

Future scope

While our initial data sources have been in the physical and biomedical sciences, respectively, we hope to expand our services for people in the social sciences, humanities and arts, and provide broad coverage in these other disciplines, if we can tap into open sources of bibliographic reference lists from publications in these areas.