Tutorial: how to process COCI’s zipped CSV dump without decompressing it

Blog post by Ivan Heibi (Universiy of Bologna) and Arcangelo Massari (University of Bologna).

OpenCitations publishes the COCI dataset after each new release in three main formats: CSV, N-Triples, and Scholix (see https://opencitations.net/download#coci). The CSV format is the most popular and downloaded one due to its comprehensive data organization (i.e. tabular format) and smaller size (compared to the other formats provided). Therefore, this is also the format we suggest using for a local process of the entire COCI dataset. 

The CSV dumps of COCI are uploaded on Figshare. You can check and download the last dump released from https://doi.org/10.6084/m9.figshare.6741422. The dump consists of one main ZIP file, including other smaller ZIP archives (one for each release) containing the actual CSV files (Figure 1).

Figure 1. The contents of the COCI CSV dataset (after the August 2022 release)

It is possible to process this data without unzipping the internal archives, thus saving a lot of disk space. In this tutorial, we will see how to achieve this in Python. Same process could be done in other programming languages.

Processing the COCI dump using Python

Step 1) Downloading the COCI dump

First, you need to download the last CSV dump release of COCI from https://doi.org/10.6084/m9.figshare.6741422 and decompress only the external archive. After this operation, you should have a folder containing the internal ZIP files such as in Figure 1.

Note: It is beneficial to decompress the external archive because doing so does not increase the space occupied on the disk (compressing archives results in a compression rate of 0%) and because working on nested archives would significantly increase RAM requirements. 

Step 2) Working with the ZIP files

Python provides the built-in zipfile module, whose ZipFile class allows you to create, read, write, edit and list the contents of a ZIP file. Given as input the path of the root directory containing all the ZIP files (FOLDER_PATH), the process elaborates each of these files on a different iteration. Each cycle initializes a ZipFile object by specifying the path to the ZIP file (archive_path).

from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):        
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:     # ...

Step 3) Accessing the ZIP files

Use the namelist() method to return the list of CSV files contained in each archive. Then to open the inner CSV files, simply cycle through the list of names and feed them to the open() method of the ZipFile instance, i.e. archive in the example below.

from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:       # ...

Step 4) Reading the CSVs

The .open() method returns a buffer. To read the CSV file as a list of dictionaries (i.e. represent each row of the CSV in dictionary format, e.g., {“column1″:”val1”, “column2″:”val2”}) we need to transform the buffer using the TextIOWrapper class and read it using the DictReader class of csv. Then we convert the result of DictReader into a list. 

from io import TextIOWrapper
from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# ...

Step 5) Processing the CSVs content

Now you can go through each row of the list and process the citation data as you want. The following example prints the citing and cited entity of each citation in the dump. 

from io import TextIOWrapper
from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# Process the CSV here
for r in rows:
print("Citing entity:",r["citing"])
print("Cited entity:",r["cited"])

Cite this article as: Arcangelo Massari, "Tutorial: how to process COCI’s zipped CSV dump without decompressing it," in OpenCitations blog, 30/09/2022, https://opencitations.hypotheses.org/2940.

 

Three publications describing the Open Citations Corpus

Last September, I attended the Fifth Annual Conference on Open Access Scholarly Publishing, held in Riga, at which I had been invited to give a paper entitled The Open Citations Corpus – freeing scholarly citation data.  A recording of my talk is available here, and my PowerPoint presentation is separately available here.  My own reflections on the major themes of the conference are given in a separate Semantic Publishing Blog post.

While in Riga preparing to give that talk about the importance of open citation data, I received an invitation from Sara Abdulla, Chief Commissioning Editor at Nature, to write a Comment piece for their forthcoming special issue on Impact.  My immediate reaction was that this should be on the same theme, an idea to which Sara readily agreed.  The deadline for delivery of the article was 10 days later!

As soon as the Riga conference was over, I first assembled all the material I had to hand that could be relevant to describing the Open Citations Corpus (OCC) in the context of conventional access to academic citation data from commercial sources.  That gave me a raw manuscript of some five thousand words, from which I had to distil an article of less than 1,300 words.  I then started editing, and asked my colleagues Silvio Peroni and Tanya Gray for their comments.

The end result, enriched by some imaginative art work by the Nature team, was published a couple of weeks later on 16th October [1], and presents both the intellectual argument for open citation data, and the practical obstacles to be overcome in achieving the goal of a substantial corpus of such data, as well as giving a general description of the Open Citations Corpus itself and of the development work we have planned for it.

Because of the drastic editing required to reduce the original draft to about a quarter of its size, all material not crucial to the central theme had to be cut.  I thus had the idea of developing the original draft subsequently into a full journal article that would include these additional themes, particularly Silvio’s work on the SPAR ontologies described in this Semantic Publishing Blog post [2], Tanya’s work on the CiTO Reference Annotation Tools described in this Semantic Publishing Blog post, and a wonderful analogy between the scholarly citation network and Venice devised by Silvio.  I also wanted to give authorship credit to Alex Dutton, who had undertaken almost all of the original software development work for the OCC.  For this reason, instead of assigning copyright to Nature for the Comment piece, I gave them a license to publish, retaining copyright to myself so I could re-use the text.  I am pleased to say that they accepted this without comment.

Silvio and I then set to work to develop the draft into a proper article.  The result was a ten-thousand word paper submitted to the Journal of Documentation a week before Christmas [3].  We await the referees’ comments!

 References

[1]     Shotton D. (2013).  Open citations.  Nature 502: 295–297. http://www.nature.com/news/publishing-open-citations-1.13937. doi:10.1038/502295a.

[2]     Peroni S and Shotton D (2012). FaBiO and CiTO: ontologies for describing bibliographic resources and citations. Web Semantics: Science, Services and Agents on the World Wide Web. 17: 33-34. doi:10.1016/j.websem.2012.08.001.

[3]    Silvio Peroni, Alexander Dutton, Tanya Gray, David Shotton (2015). Setting our bibliographic references free: towards open citation data. Journal of Documentation, 71 (2): 253-277. http://dx.doi.org/10.1108/JD-12-2013-0166; OA at http://speroni.web.cs.unibo.it/publications/peroni-2015-setting-bibliographic-references.pdf

This is the main article about OpenCitations, which includes several background information and the main ideas and works supporting the whole project, the Corpus, and some possible future developments in terms of new kinds of data to be included, e.g. citation functions.

Cite this article as: davidshotton, "Three publications describing the Open Citations Corpus," in OpenCitations blog, 24/01/2014, https://opencitations.hypotheses.org/639.

 

Open letter to publishers

[The text of this post was updated on 27-09-2013 and 04-04-2017 to reflect a new CrossRef metadata best practice document and a change in their URI.]

Today I wrote an open letter to all scholarly journal publishers, available online here, entitled:

Open your article reference lists for inclusion in the Open Citations Corpus.

In this letter, I request that publishers open the bibliographic citation data in their journal article reference lists.  There is a growing movement to make such bibliographic citation data open – for example, Nature Publishing Group’s open Linked Data Platform now includes citation metadata for all published article references.

Provided a publisher is already depositing article references with CrossRef as part of the CrossRef CitedBy Linking service, all the publisher need to do is to inform CrossRef that it is willing for CrossRef to freely distribute these reference, for example in response to queries against the CrossRef XML API.  We will then harvest them from CrossRef and incorporate them as open linked data in the Open Citations Corpus.

Nature Publishing Group, Taylor & Francis, the American Association for the Advancement of Science (who publish Science) and Oxford University Press, as well as a number of open-access publishers, have already given their consent to CrossRef to do this for some or all of their journals.

If not already a subscriber to the CrossRef CitedBy Linking service, a publisher can register for this useful service free of charge.  Having done so, there is nothing further the publisher needs to do to ‘open’ its reference data, other than to give its consent to CrossRef.  This can be done automatically, in the submitted article metadata, or (for back numbers) by informing CrossRef directly.

Even Open Access publishers, publishing articles under a CC-By open license, need to give this specific permission to CrossRef for this to occur, because CrossRef policy is that all publishers, including open access publishers, have to opt in to any distribution of references that CrossRef makes.

For new submissions, publishers should follow instructions detailed in the CrossRef blog at https://www.crossref.org/blog/distributing-references-via-crossref/, which contains the following key instruction:

“In order for publishers to distribute references along with standard bibliographic metadata, publishers need to set the <reference_distribution_opt> metadata element to “any" for each DOI deposit where they want to make references openly available.”

In this way, publishers can choose to open the reference lists for all their journals, or to do so on a journal-by-journal or on an article-by-article basis (useful for ‘hybrid’ subscription-access journals in which only some articles are open access).

To open reference lists for back numbers, publisher needs to e-mail CrossRef to express their intent,using the template shown at the foot of this post, as detailed in my Open Letter to Publishers.

I have copied this open letter to the CEOs of the Open Access Scholarly Publishers Association (OASPA), of the Association of Learned and Professional Society Publishers (ALPSP), and of the International Association of Scientific, Technical & Medical Publishers (STM), asking them to distribute it to their members, perhaps in association with their next Members News Letter, as CrossRef itself is planning to do later this month.

Please spread the word about this, particularly to publishers who may not be members of these professional associations.  Thanks.

= = =

Template for an e-mail to CrossRef expressing willingness to open reference lists in previously published and future journal articles.

To support@crossref.org

I am writing on behalf of *** [name of publisher] to confirm that *** [name of publisher] is willing for the bibliographic reference lists within the articles in [delete as necessary:] all our journals [or] the attached list of journals be made freely available by CrossRef, for inclusion in the Open Citations Corpus. These journals are associated with the following DOI prefix(es): 10.**** [Please complete DOI prefix(es) – see footnote].
Yours sincerely [name, position, date]
= = =
Footnote: Publisher’s DOI prefixes are listed at http://www.crossref.org/06members/50go-live.html by name of publisher.

Cite this article as: davidshotton, "Open letter to publishers," in OpenCitations blog, 03/01/2013, https://opencitations.hypotheses.org/465.

 

Taylor & Francis to open article reference lists

I am very pleased to announce that last year Ian Bannerman, Managing Director for Journals at Taylor & Francis, confirmed this publisher’s willingness to pilot the opening of the reference lists from articles in 29 of their subscription access journals, as well as from all of their current list of 15 Open Access journals, for inclusion in the Open Citations Corpus.  The reference lists for these journals are already being supplied to CrossRef as part of the CrossRef CitedBy Linking service, and will be made available publicly via the CrossRef XML query API.

Taylor & Francis is a major international publisher of over 1,000 academic journals and more than 1,800 books per year, incorporating well-known publishing names including Bios Scientific Publishing, CRC Press, Garland Science, Marcel Decker and Routledge.  It is the largest publisher of subscription access journals yet to agree to make reference lists available from its journal articles, and I welcome them warmly into the Open Citations fold.

Incorporation of new reference data from Taylor and Frances journals into the Open Citations Corpus will commence in the near future.

Cite this article as: davidshotton, "Taylor & Francis to open article reference lists," in OpenCitations blog, 03/01/2013, https://opencitations.hypotheses.org/467.

Nature to open its reference list data

Bibliographic references are the links that knit together independent scholarly endeavours.  I am thus delighted to announce that Nature Publishing Group, publisher of Nature, Nature Genetics and many other leading journals, has agreed to open its articles’ reference lists, initially for a selected number of NPG journals, and contribute the bibliographic citations contained in these lists as open linked data to an expanded Open Citations Corpus, where they will be freely available for everyone to use in whatever manner they choose.  Preparations to expand the corpus in this manner, by integration with the reference processing pipeline of the CrossRef Cited-By Linking service, will be undertaken over the next six months of this year, and incorporation of the references from the selected NPG journals into the expanded Open Citations Corpus is planned to commence in the first half of 2013.

As the first subscription-access publisher to opening its reference lists in this way, Nature Publishing Group is further demonstrating its commitment to ‘lead from the front’ in its embrace of new semantic publishing technologies.  Only two months ago, this publisher announced its decision to open up the bibliographic records of its journal articles as open linked data.  On 4th April, NPG’s Linked Data Platform press release read:

“Nature Publishing Group (NPG) today is pleased to join the linked data community by opening up access to its publication data via a linked data platform. NPG’s Linked Data Platform is available at http://data.nature.com.      The platform includes more than 20 million Resource Description Framework (RDF) statements, including primary metadata for more than 450,000 articles published by NPG since 1869. In this first release, the datasets include basic bibliographic information (title, author, publication date, etc) as well as NPG-specific ontology terms. These datasets are being released under an open metadata license, Creative Commons Zero (CC0), which permits maximal use/re-use of this data.      NPG’s platform allows for easy querying, exploration and extraction of data and relationships about articles, contributors, publications, and subjects. Users can run web-standard SPARQL Protocol and RDF Query Language (SPARQL) queries to obtain and manipulate data stored as RDF. The platform uses standard vocabularies such as Dublin Core, FOAF, PRISM, BIBO and OWL, and the data is integrated with existing public datasets including CrossRef and PubMed.      More information about NPG’s Linked Data Platform is available at http://developers.nature.com/docs. Sample queries can be found at http://data.nature.com/query. ”

We very much hope that NPG’s example will encourage other subscription-access publishers to open their own journal article reference lists, and become early adopters of Open Citations.  Reasons why subscription-access publishers should willingly join NPG and open their citation data are given in my previous blog post.  While at first coverage among subscription-access publishers will be incomplete, this expanded Open Citation Corpus will, I am sure, draw in increasing numbers of publishers, and in true Web 2.0 style will become more useful the more publishers participate, resulting in value-added bibliographic and bibliometic services being created over the open data. Other subscription-access publishers who would like to contribute their journal article references to the Open Citations Corpus should contact me at <david.shotton@zoo.ox.ac.uk>.

Cite this article as: davidshotton, "Nature to open its reference list data," in OpenCitations blog, 11/06/2012, https://opencitations.hypotheses.org/353.

Five Stars Ontology

To accompany today’s publication in D-Lib Magazine of the article The Five Stars of Online Journal Articles – a framework for article evaluation highlighted in the previous post, I have today also published The Five Stars Ontology, a simple ontology written in OWL 2 DL that forms part of SPAR, a suite of Semantic Publishing and Referencing Ontologies. It is intended for use by publishers and others wishing to encode Five Stars ratings, such as those exemplified in the D-Lib article, in machine-readable form, so they can accompany other machine-readable metadata for the article.  To exemplify this, the following RDF graph, shown in turtle notation, gives the Five Stars ratings for the D-Lib article itself:

<http://dx.doi.org/10.1045/january2012-shotton>
     fivestars:hasPeerReviewRating “3”^^xsd:nonNegativeInteger ;
     fivestars:peerReviewRatingComment “Post-publication responsive
          peer review of the preprint.” ;
     fivestars:hasOpenAccessRating “4”^^xsd:nonNegativeInteger ;
     fivestars:openAccessRatingComment “Gold/libre open access
          without author fee!” ;
     fivestars:hasEnhancedContentRating “1”^^xsd:nonNegativeInteger ;
     fivestars:enhancedContentRatingComment “Plentiful Web links in
          text and to all references. No additional semantic
          enhancement of text.” ;
     fivestars:hasAvailableDatasetsRating “0”^^xsd:nonNegativeInteger ;
     fivestars:availableDatasetsRatingComment “Not applicable.” ;
     fivestars:hasMachine-readableMetadataRating “1”^^xsd:nonNegativeInteger ;
     fivestars:machine-readableMetadataRatingComment “Structural
          markup in HTML only.” ;
     fivestars:hasOverallFiveStarsRating “9”^^xsd:nonNegativeInteger ;
     fivestars:overallFiveStarsRatingComment “The nature of this
          article, being a position paper rather than a research
          paper with primary research data, has influenced the
          overall rating obtained.” .
Cite this article as: davidshotton, "Five Stars Ontology," in OpenCitations blog, 16/01/2012, https://opencitations.hypotheses.org/317.

2011 retrospective: meetings on the future of research communications

What a year it has been!  Four key meetings were held during 2011, bringing together academics, computer scientists and scholarly publishers to discuss the future of scholarly communication.  The first of these, a workshop entitled Beyond the PDF, organized and hosted in January 2011 by Philip Bourne at the University of California, San Diego, itself built on an earlier HyPER workshop organized in May 2010 in Amsterdam by Anita de Waard of Elsevier Labs (de Waard et al. 2009, [1]).  It was followed by a meeting entitled Beyond Impact, organized by Cameron Neylon of STFC at the Wellcome Trust headquarters in London in May 2011, that considered alternative metrics to the journal impact factor for the evaluation of research – and particularly researcher – merit.  In August 2011, a further meeting on The Future of Research Communication, organized by Phil Bourne of UCSD, Tim Clark of Harvard University, Robert Dale of Macquarie University, Anita de Waard of Elsevier Labs, Ivan Herman of the W3C, Eduard Hovy of the University of Southern California and myself, was held in Germany as a Schloss Dagstuhl Perspectives Workshop.  This led to the formation of the Force11 Community dedicated to the improvement of research communication and e-scholarship, and to the publication in October 2011 of the Force11 White Paper (Bourne et al., 2011), that was submitted as evidence both to the Royal Society’s Science as a Public Enterprise project and to the UK Cabinet Office’s public consultation Making Open Data Real.  Finally, in October 2011, Microsoft Research and Harvard University jointly hosted a meeting in Cambridge, Massachusetts, entitled Transforming Scholarly Communication, that took these ideas forward.

Additionally, the benefits of Semantic Web technologies for libraries were recently discussed at the 2011 annual Semantic Web in Libraries meeting entitled Scholarly Communication in the Web of Data, held in Hamburg, Germany, in November 2011.

The thinking undertaken at these meetings contributed significantly to the formulation of my paper entitled The Five Stars of Online Journal Articles described in a previous post, and significantly advanced the community’s thinking as a whole both on semantic publishing of journal articles and on the issues surrounding research data publication.

I’m looking forward to what 2012 brings!

[1]  de Waard A, Buckingham Shum S, Carusi A, Park J, Samwald M, and Sándor Á (2009). Hypotheses, Evidence and Relationships: The HypER Approach for Representing Scientific Knowledge Claims. In: Proceedings 8th International Semantic Web Conference, Workshop on Semantic Web Applications in Scientific Discourse (26 Oct 2009, Washington DC.). Lecture Notes in Computer Science, Springer Verlag: Berlin.  http://oro.open.ac.uk/18563/.

Cite this article as: davidshotton, "2011 retrospective: meetings on the future of research communications," in OpenCitations blog, 10/01/2012, https://opencitations.hypotheses.org/306.

Search OpenEdition Search

You will be redirected to OpenEdition Search