Blog post by Ivan Heibi (Universiy of Bologna) and Arcangelo Massari (University of Bologna).
OpenCitations publishes the COCI dataset after each new release in three main formats: CSV, N-Triples, and Scholix (see https://opencitations.net/download#coci). The CSV format is the most popular and downloaded one due to its comprehensive data organization (i.e. tabular format) and smaller size (compared to the other formats provided). Therefore, this is also the format we suggest using for a local process of the entire COCI dataset.
The CSV dumps of COCI are uploaded on Figshare. You can check and download the last dump released from https://doi.org/10.6084/m9.figshare.6741422. The dump consists of one main ZIP file, including other smaller ZIP archives (one for each release) containing the actual CSV files (Figure 1).
Figure 1. The contents of the COCI CSV dataset (after the August 2022 release)
It is possible to process this data without unzipping the internal archives, thus saving a lot of disk space. In this tutorial, we will see how to achieve this in Python. Same process could be done in other programming languages.
Processing the COCI dump using Python
Step 1) Downloading the COCI dump
First, you need to download the last CSV dump release of COCI from https://doi.org/10.6084/m9.figshare.6741422 and decompress only the external archive. After this operation, you should have a folder containing the internal ZIP files such as in Figure 1.
Note: It is beneficial to decompress the external archive because doing so does not increase the space occupied on the disk (compressing archives results in a compression rate of 0%) and because working on nested archives would significantly increase RAM requirements.
Step 2) Working with the ZIP files
Python provides the built-in zipfile module, whose ZipFile class allows you to create, read, write, edit and list the contents of a ZIP file. Given as input the path of the root directory containing all the ZIP files (FOLDER_PATH), the process elaborates each of these files on a different iteration. Each cycle initializes a ZipFile object by specifying the path to the ZIP file (archive_path).
from zipfile import ZipFile
import os
for archive_name in os.listdir(FOLDER_PATH):
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:
# ...
Step 3) Accessing the ZIP files
Use the namelist() method to return the list of CSV files contained in each archive. Then to open the inner CSV files, simply cycle through the list of names and feed them to the open() method of the ZipFile instance, i.e. archive in the example below.
from zipfile import ZipFile
import os
for archive_name in os.listdir(FOLDER_PATH):
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:
for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
# ...
Step 4) Reading the CSVs
The .open() method returns a buffer. To read the CSV file as a list of dictionaries (i.e. represent each row of the CSV in dictionary format, e.g., {“column1″:”val1”, “column2″:”val2”}) we need to transform the buffer using the TextIOWrapper class and read it using the DictReader class of csv. Then we convert the result of DictReader into a list.
from io import TextIOWrapper
from zipfile import ZipFile
import os
for archive_name in os.listdir(FOLDER_PATH):
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:
for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# ...
Step 5) Processing the CSVs content
Now you can go through each row of the list and process the citation data as you want. The following example prints the citing and cited entity of each citation in the dump.
from io import TextIOWrapper
from zipfile import ZipFile
import os
for archive_name in os.listdir(FOLDER_PATH):
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:
for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# Process the CSV here
for r in rows:
print("Citing entity:",r["citing"])
print("Cited entity:",r["cited"])
