Tutorial: how to process COCI’s zipped CSV dump without decompressing it

Blog post by Ivan Heibi (Universiy of Bologna) and Arcangelo Massari (University of Bologna).

OpenCitations publishes the COCI dataset after each new release in three main formats: CSV, N-Triples, and Scholix (see https://opencitations.net/download#coci). The CSV format is the most popular and downloaded one due to its comprehensive data organization (i.e. tabular format) and smaller size (compared to the other formats provided). Therefore, this is also the format we suggest using for a local process of the entire COCI dataset. 

The CSV dumps of COCI are uploaded on Figshare. You can check and download the last dump released from https://doi.org/10.6084/m9.figshare.6741422. The dump consists of one main ZIP file, including other smaller ZIP archives (one for each release) containing the actual CSV files (Figure 1).

Figure 1. The contents of the COCI CSV dataset (after the August 2022 release)

It is possible to process this data without unzipping the internal archives, thus saving a lot of disk space. In this tutorial, we will see how to achieve this in Python. Same process could be done in other programming languages.

Processing the COCI dump using Python

Step 1) Downloading the COCI dump

First, you need to download the last CSV dump release of COCI from https://doi.org/10.6084/m9.figshare.6741422 and decompress only the external archive. After this operation, you should have a folder containing the internal ZIP files such as in Figure 1.

Note: It is beneficial to decompress the external archive because doing so does not increase the space occupied on the disk (compressing archives results in a compression rate of 0%) and because working on nested archives would significantly increase RAM requirements. 

Step 2) Working with the ZIP files

Python provides the built-in zipfile module, whose ZipFile class allows you to create, read, write, edit and list the contents of a ZIP file. Given as input the path of the root directory containing all the ZIP files (FOLDER_PATH), the process elaborates each of these files on a different iteration. Each cycle initializes a ZipFile object by specifying the path to the ZIP file (archive_path).

from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):        
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:     # ...

Step 3) Accessing the ZIP files

Use the namelist() method to return the list of CSV files contained in each archive. Then to open the inner CSV files, simply cycle through the list of names and feed them to the open() method of the ZipFile instance, i.e. archive in the example below.

from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:       # ...

Step 4) Reading the CSVs

The .open() method returns a buffer. To read the CSV file as a list of dictionaries (i.e. represent each row of the CSV in dictionary format, e.g., {“column1″:”val1”, “column2″:”val2”}) we need to transform the buffer using the TextIOWrapper class and read it using the DictReader class of csv. Then we convert the result of DictReader into a list. 

from io import TextIOWrapper
from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# ...

Step 5) Processing the CSVs content

Now you can go through each row of the list and process the citation data as you want. The following example prints the citing and cited entity of each citation in the dump. 

from io import TextIOWrapper
from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# Process the CSV here
for r in rows:
print("Citing entity:",r["citing"])
print("Cited entity:",r["cited"])

Cite this article as: Arcangelo Massari, "Tutorial: how to process COCI’s zipped CSV dump without decompressing it," in OpenCitations blog, 30/09/2022, https://opencitations.hypotheses.org/2940.

 

New documents that present OpenCitations’ mission, unique benefits, present status and future plans

Posted on August 10th 2022 by Chiara Di Giambattista

More than a year ago, Ginny Hendricks, Director of Member & Community Outreach for Crossref, and a valued member of the OpenCitations International Advisory Board, published on the Crossref blog the post “The road ahead: our strategy through 2025”. In order to describe all Crossref’s principles and activities, Ginny presented the Crossref strategic planning framework as a diagram summarizing Crossref’s statements, key messages and truths. The clarity and immediacy of the diagram were such that we adapted it to present  OpenCitations’ own statements and goals. The resulting poster “OpenCitations – what does the future hold?” was presented by our Director David Shotton at the OASPA2021 conference, and can be found in this blog post.

Although the poster offered a wide overview of OpenCitations values, unique traits, benefits and plans, it differed slightly from Ginny’s original diagram, in particular because it lacked a “Mission Statement”, scattering the relevant information within the “Values” and “Principles” boxes. Indeed, at that time (September 2021), we didn’t have a clearly defined Mission Statement.

Nevertheless, the creation of that poster was crucial in helping us start to articulate more clearly the purpose and meaning of OpenCitations. As David underlined in his post “From little acorns…a retrospective on OpenCitations”, since 2018 OpenCitations activities have progressively increased and, with them, the number of related journal articles, conference papers and technical definitions. OpenCitations’ involvement in international networks and collaborations (such as SCOSS and the OpenAIRE-Nexus project), together with our need of identifying and reaching out to new stakeholders to assure OpenCitations’ development and sustainability, has made it necessary to publicly define OpenCitations’ mission, unique strengths and next developmental steps.

After numerous revisions, aided by wise advice from members of the OpenCitations Advisory Board members, we’re now happy to publish the following three OpenCitations documents:

OpenCitations Mission Statement,

The Uniqueness of OpenCitations   and

OpenCitations – Present Status and Future Plans,

which together provide a summary of why we exist and where we are heading.

We are particularly proud of the definition of OpenCitations’ primary mission, namely

to harvest and openly publish accurate and comprehensive metadata describing the world’s academic publications and the scholarly citations that link them, and to preserve ongoing access to this information by secure archiving.

The Mission Statement also presents brief descriptions of the OpenCitations context, our vision, our value proposition and our relationship with the community and stakeholders.

The Uniqueness of OpenCitations provides the answer to the question ‘Why choose to use OpenCitations?’, and is a detailed presentation of OpenCitations’ benefits.

OpenCitations – Present Status and Future Plans summarizes OpenCitations’ ongoing activities, that can be quickly visualized on our public roadmap. It also introduces the OpenCitations Working Groups, served by the members of the OpenCitations International Advisory Board, which are currently working on the themes of governance evolution and community building, with the common purpose of driving OpenCitations along the path from being a ‘sustainable infrastructure’ (in POSI terms) to being an enduring community led and financially sustained infrastructure.

In fulfilling our mission and reaching our goals, the support and vital interest of our community members is fundamental. We request that you, as a member of our community, provide us with feedback on these documents and the ideas they contain, or indeed to ask for clarifications, to help us improving our mission and our communications to explain it. You can reach us here: contact@opencitations.net.

Thank you!

Cite this article as: Chiara Di Giambattista, "New documents that present OpenCitations’ mission, unique benefits, present status and future plans," in OpenCitations blog, 10/08/2022, https://opencitations.hypotheses.org/2498.

The OpenCitations Roadmap is now publicly available on Trello

Want to keep yourself updated about the ongoing activities of OpenCitations? We have now publicly released the OpenCitations Roadmap, available on Trello.com:

https://trello.com/b/RprHYoKL/opencitations

The OpenCitations Roadmap consists of a board fulfilled with colour-labelled cards which present the goals so far reached, the present projects and activities, and the future plans. By clicking on the cards, it is possible to visualize a description for each activity, the progress state, and who in the OpenCitations team is working on it.

The OpenCitations Roadmap covers all kinds of activities divided according to the scope, identified by the coloured labels, in particular:

  • light blue for the technical development, such as the development of the software for the creation of the new database OpenCitations Meta and of DOCI, the OpenCitations Index of DataCite open DOI-to-DOI citations, and the re-engineering of the infrastructure and the website;
  • green for the data model implementation;
  • yellow for the data development, such as the bi-monthly COCI releases;
  • purple for the events and outreach activities.

The cards also highlight the activities related to the two EC-funded projects OpenCitations is involved in, OpenAIRE Nexus (blue label) and RISIS2 (orange label). We thank the OpenAIRE team for the help and suggestions during the Roadmap review process.

The OpenCitations Roadmap is an open work in progress that will reflect the developments and growth of OpenCitations. At OpenCitations, we don’t want this Roadmap to be just an online ‘showcase’, but a room in which to share ideas and opinions. We invite you – the members of our community, our stakeholders, the other Open Science actors, researchers, and librarians, and anyone who is interested in OpenCitations activities – to add a comment or a question in the ‘Leave feedback‘ card. This will help us to better understand our strong and weak points, and to stay in touch with the needs and thoughts of the community.

In this way, supplementing the conventional communications channels of email and the social platforms (our blog, Twitter, LinkedIn), the OpenCitations Roadmap will become a new virtual place for dialogue, where you can directly contribute to improve OpenCitations.

Cite this article as: Chiara Di Giambattista, "The OpenCitations Roadmap is now publicly available on Trello," in OpenCitations blog, 29/03/2022, https://opencitations.hypotheses.org/1519.

Reflections on the global citation graph

In his call for open citations, Dario Taraborelli hailed the scholarly citation graph (in which the nodes (vertices) are individual academic publications and the links (edges) represent bibliographic citations from one publication to another) as one of humankind’s most important intellectual achievements.

We all understand that the inclusion within our own academic publications of bibliographic references to the works of others is one of the most explicit ways of acknowledging the thoughts, discoveries, achievements and influences of other scholars, and their contributions to our own work. Not only does what we gain from their publications enable us to make intellectual progress, by “standing on the shoulders of giants” as Newton once famously observed [1], but the influence of these publications extends forward in time across the entire intellectual landscape, like gigantic shadows cast at sunset, whether or not those influenced by these publications have occasion to reference them in their own works.

A bibliographic citation is not only “a conceptual directional link from a citing entity to a cited entity, created by a human performative act of making a citation”, but it is additionally both enduring and retrospective. Enduring, because once made it persists for ever within the global corpus of scholarly literature, and retrospective because (with the exception of occasional contemporaneous citations) the cited publication predates the citing publication.

At the anterior margin of a crawling cell, cellular protrusive extension (for example of a pseudopodium) is achieved by the catalysed polymerization of new filaments of the cytoskeletal protein actin from attachment sites on an existing stationary actin filament network, pushing the cell margin forward [2]. The scholarly citation network (or citation graph, the two terms here being used interchangeably) is similarly dynamic and temporally directional, being extended forward as new works of scholarship are published. Extension of knowledge is achieved by the catalytic inspiration provided by existing academic publications, themselves temporally stationary within the expanding citation network, leading to the publication of new works of scholarship that cite these previous publications and thus extend the citation network further into the future. The citation graph is thus not just an acyclic directed graph, but an acyclic temporally directed graph. Indeed, it is this temporal aspect of the citation network that is one of its most important features.

To use another analogy, the human genealogical tree is inherently multidimensional and difficult to represent pictorially in its entirety, because each new birth brings together the family trees of the child’s two parents. However, unless the parents are seriously promiscuous, the resulting genealogical tree is not impossibly complex. In contrast, the scholarly citation network is much more highly interlinked, since each new publication cites not just two but many preceding (‘parent’) publications, which themselves may beget many other citations.

Visualization of the global scholarly citation graph, or portions of it, is thus inherently difficult, and the important temporal aspect of the graph is the one ignored by almost every method used for visualizing aspects of that graph. Existing methods may take the broad view, showing the links, and the strength of those links, between one scholarly domain and another, thus visualizing the ‘structure of science’. Alternatively, they may take a more detailed view of a small section of the graph, visualize the proximity of individual publications to one another. Often a radial display is chosen for this, that shows in closest proximity those papers directly referenced by the selected publication in the centre, then at a greater radius those papers referenced by the cited papers shown in the inner circle, and so on. Because of the graph’s complexity, such displays quickly looses intelligibility after two citation links.

Among a small number of visualization applications that do not ignore the temporal aspect of the graph is Citeology, a temporally based citation network visualization tool developed some years ago by Justin Matejka and colleagues at the design software company Autodesk [3]. Unfortunately, this innovative software prototype was not central to that company’s mission, development ceased, and the Citeology Java app is no longer available. However, in his last email to me, Justin Matejka kindly offered to help others re-create this application.

There is thus an urgent need for innovative new open-source visualization tools that will clearly and dynamically display portions of the global citation graph, for example the direct and indirect citation connections between any two publications or any two individuals, along the temporal axis of publication date. Developers within the open science community please step forward!

References

[1] Isaac Newton, in a 1675 letter to Robert Hooke, wrote “If I have seen further it is by standing on the shoulders of Giants.” https://discover.hsp.org/Record/dc-9792/

[2] Bruce Alberts et al. (2014). Molecular Biology of the Cell. 6th Edition. Garland Science. Chapter 16, The Cytoskeleton.

[3] Justin Matejka, Tovi Grossman, George Fitzmaurice (2012). Citeology: Visualizing Paper Genealogy. ACM Extended Abstracts on Human Factors in Computing Systems. https://www.autodesk.com/research/publications/citeology https://d2f99xq7vri1nk.cloudfront.net/CiteologyVideo.mp4

Cite this article as: davidshotton, "Reflections on the global citation graph," in OpenCitations blog, 11/08/2021, https://opencitations.hypotheses.org/1289.

The Sloan Foundation funds OpenCitations

The OpenCitations Enhancement Project funded by Sloan

The Alfred P. Sloan Foundation, which funds research and education in science, technology, engineering, mathematics and economics, including a number of key technology projects relating to scholarly communication, has agreed to fund The OpenCitations Enhancement Project, a new project to develop and enhance the OpenCitations Corpus.

As readers of this blog will know, the OpenCitations Corpus is an open scholarly citation database that freely and legally makes available accurate citation data (academic references) to assist scholars with their academic studies, and to serve knowledge to the wider public.

Objectives

The OpenCitations Enhancement Project, funded by the Sloan Foundation for 18 months from May 2017, will make the OpenCitations Corpus (OCC) more useful to the academic community both by significantly expanding the volume of citation data held within the Corpus, and by developing novel data visualizations and query services over the stored data.

At OpenCitations, we will achieve these objectives in the following ways:

(a) By establishing a new powerful physical server to handle the Corpus data and offer adequate performance for query services.

(b) By increasing the rate of data ingest into the Corpus, by integrating with server 30 small data-ingest computers, Raspberry Pi 3Bs, working in parallel to harvest references, thus increasing the current rate of corpus data ingest some thirty-fold to about half a million citation links per day.

(c) By employing a post-doctoral computer science research engineer specifically to develop information visualisation interfaces and sense-making tools that will both provide smart ways of envisaging and comprehending the citation data stored within the OpenCitations Corpus, and will also ease the task of manual curation of the OCC.

Personnel

This post-doctoral appointment will start in the autumn of 2017, once the new hardware has been commissioned and programmed. We seek a highly intelligent, skilled and motivated individual who is an expert in Web Interface Design and Information Visualization, and who can demonstrate a commitment to increasing the openness of scholarly information. A formal advertisement for this post, which will be held at the University of Bologna in Italy under the supervision of Dr Silvio Peroni, will be published in the near future. In the mean time, individuals with the relevant skills and background who would like to express early interest in joining the OpenCitations team in this role should contact him by e-mail to <silvio.peroni@opencitations.net>.

Expected Outcomes

By the end of the OpenCitations Enhancement Project, we will have harvested approximately 190 million citation links obtained from the reference lists of about 4.4 million scholarly articles (~15% of Web of Science’s coverage). In this way, in a significant initial step towards the comprehensive literature coverage we seek for the OCC, we will establish the OpenCitations Corpus as a valuable and persistent free-to-use global scholarly on-line Linked Open Data service.

In so doing, we aim at empower the global community by liberating scholarly citation data from their current commercial shackles, publishing such data with a Creative Commons CC0 Public Domain Dedication that will enable novel third-party services to be built over them.

Cite this article as: davidshotton, "The Sloan Foundation funds OpenCitations," in OpenCitations blog, 15/05/2017, https://opencitations.hypotheses.org/754.

Search OpenEdition Search

You will be redirected to OpenEdition Search