The four supported infrastructures (OpenCitations, the DOAB, LA Referencia and ROR) “were evaluated by the jury composed by SCOSS, then according to the exemplary criteria of the Committee for the Open Science, which notably guarantee transparency and the participation of the scientific communities in their governance”.
Since 2020, the FNSO has acknowledged OpenCitations as an infrastructure worth its financial support, thanks to its mission of disseminating bibliographic and citation metadata in open access with a level of quality and coverage, thus providing a workable, free and open alternative to the academic community’s current dependency on proprietary tools. OpenCitations’ work therefore frees up citation analysis, promotes the evolution of bibliometric indicators and the broadening knowledge of science.
The FNSO is now contributing to OpenCitations with recurring funding for 2023, 2024 and 2025 for an annual amount of €75,000. This generous support will be crucial in sustaining the maintenance and development of OpenCitations’ technical infrastructure, and in supporting the future activities of the OpenCitations team, that are publicly displayed in the OpenCitations Roadmap.
We are extremely honoured and grateful to the French National Fund for Open Science for renewing the pledge of such a portion of its open science budget to support our work.
We are pleased to announce that, after two online editions, the 2023 edition of the Workshop on Open Citations and Open Scholarly Metadata will come back to Bologna, on 26-27 October 2023. – https://workshop-oc.github.io.
During the two-day event, researchers, policy-makers and scholarly publishers will gather in the historical location of the University of Bologna to take part in the plenary presentations and discussions around this year’s theme:
The value of open scholarly metadata for research assessment purposes.
Participation
If you wish to participate in the Workshop on Open Citations and Open Scholarly Metadata 2023 as an attendee, please fill in the form by providing the requested information and a short bio to apply for the event. You will receive a notification of acceptance by 25 June. In case of oversubscriptions, the organisers will select attendees from among those who have applied.
If you wish to talk in the Workshop of Open Citations and Open Scholarly Metadata 2023, please fill in both the first part and the second part of the application form to submit a contribution (max 500 words) that fits this year’s theme “The value of open scholarly metadata for research assessment purposes”. The organizing committee will decide if the selected submissions will involve either a talk of 10-minute duration, with a 5-minute discussion following, or a poster presentation.
Application deadline: 21 May 2023 11 June 2023 ( h. 23:59 AoE ).
Fees and expenses
For the accepted attendees and contributors, we request a small registration fee (VAT included) of €100 to cover expenses (€25 for students). We will provide lunches and refreshments on all days, and a free workshop dinner on Thursday 26 October.
Those attending will be expected to cover the cost of their own travel and accommodation, and a list of recommended hotels and guest houses is available here.
Details of how to pay will be sent with the notification of acceptance together with information to finalize the registration.
Please circulate this invitation to any colleagues that you think might be interested in attending this workshop.
Venue
Bologna is home to numerous prestigious cultural, economic and political institutions as well as one of the most impressive trade fair districts in Europe. In 2000 it was declared the European capital of culture, and in 2006, a UNESCO “city of music”. Bologna’s porticoes have been listed on the UNESCO World Heritage List in 2021.
The University of Bologna is the oldest university in the western world, and one of the largest universities in Italy (with about 90,000 enrolled students).
The workshop will take place at the University of Bologna in the heart of the city – the Department of History, Cultures, and Civilization (Italian translation: Dipartimento di Storia, Culture e Civiltà), Piazza S. Giovanni in Monte, 2, Room “Aula Prodi”.
The event will be also live-streamed.
List of organizers
Giovanni Colavizza (University of Amsterdam; ScholarIndex)
Chiara Di Giambattista (University of Bologna; OpenCitations)
Ivan Heibi (University of Bologna; OpenCitations)
Philipp Mayr (GESIS – Leibniz-Institute for the Social Sciences; EXCite/OUTCite)
Silvio Peroni (University of Bologna; OpenCitations)
Matteo Romanello (Swiss Federal Institute of Technology in Lausanne; ScholarIndex)
David Shotton (University of Oxford; OpenCitations)
Ludo Waltman (Centre for Science and Technology Studies, Leiden University; VOSViewer)
2023 has arrived, and it’s the time to put aside the old agendas and to fill the freshly printed organizers with new tasks. However, it is also worth looking back over the achievements of the past year.
So, before consigning 2022 to the archives, we want to take a moment to recall and celebrate all that OpenCitations has achieved over the past year, in terms of technical developments, community building, and our own internal organization.
Clarifying our objectives
Since its foundation, OpenCitations has been presented to the scholarly community as a infrastructure organization guided by the values of openness, equity and accessibility. In 2022, we publicly clarified our mission, our aims and our unique points by publishing these three documents:
In addition, we continued to monitor OpenCitations’ compliance with the Principles of Open Scholarly Infrastructure. Because one area in which we were still falling short involved stakeholder involvement in OpenCitations’ governance, in 2022 OpenCitations started reflecting on its organizational structure by creating a Governance Evolution Working Group. In addition, we explored the further development of our stakeholder community by establishing a parallel Community Building Working Group. These two small working groups, involving members of the OpenCitations International Advisory Board, will continue to meet throughout 2023 to develop strategic plans concerning OpenCitations open governance structure and outreach.
More people, more tools
In 2022, OpenCitations welcomed two new full-time developers to its team: Arianna Moretti as software developer, and Ivan Heibi as the person responsible for OpenCitations’ technical infrastructure. As of January 2023, the OpenCitations team consists of our two directors Silvio Peroni and David Shotton, our administrator and research manager Claudio Fabbri, myself as our communications director and community development manager, and our four developers: Giuseppe Grieco (software and systems developer), Arcangelo Massari (developer of OpenCitations Meta), Arianna Moretti (software and indexes developer), and Ivan Heibi (technical infrastructure manager). This team meets weekly at the Research Centre for Open Scholarly Metadata within the University of Bologna.
This enlarged technical workforce has been crucial in the development of the two new OpenCitations indexes, DOCI (citations from DataCite) and POCI (citations from PubMed), and our new database of bibliographic metadata, OpenCitations Meta, all of which were released in December 2022:
DOCI, the OpenCitations Index of DataCite open DOI-to-DOI citations, is an RDF dataset containing details of all the citations that are specified in the most recent dump of DataCite metadata. The citations available in DOCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. Its first release (December 2022) contains information on 169,822,752 citations from 358,277 bibliographic resources.
POCI, the OpenCitations Index of PubMed open PMID-to-PMID citations, is an RDF dataset containing details of all the citations from publications bearing PubMed Identifiers (PMIDs) to other PMID-identified publications, harvested from the National Institutes of Health Open Citations Collection (NIH-OCC). As for DOCI, the citations available in POCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. Its first release contains 717,654,703 citations from 26,024,862 bibliographic resources.
OpenCitations Meta database stores and delivers bibliographic metadata for all the publications involved as citing or cited publications in the citations recorded in the OpenCitations Indexes. For each publication, the metadata exposed by OpenCitations Meta includes the publication’s title, type, venue (e.g. journal name), volume number, issue number, page numbers, publication date, and identifiers such as Digital Object Identifiers (DOIs) and PubMed Identifiers (PMIDs). In addition, OpenCitations Meta includes details of the main actors involved in the document’s publication, i.e., the names of the authors, editors, and publishers, each with its own additional metadata and identifier (e.g. ORCID). Its first release (December 2022) contains information on 87,321,593 bibliographic entities, 277,750,235 authors and 2,359,301 editors (counted by their roles, without disambiguating individuals), 710,226 publication venues, and 17,268 publishers.
As can be seen on OpenCitations roadmap, we have many tasks already scheduled for 2023, and for this reason OpenCitations team needs to expand further: we now seek applicants for a new Research Fellow position to work on the further development of the OpenCitations technical infrastructure from April 2023.
Our role in the community
Throughout 2022, the OpenCitations team has participated both online and in person in a number of international events, presenting details of OpenCitations to potential stakeholders and to Open Science experts, and connecting with similar organizations.
In October, OpenCitations organized our third workshop, the Workshop on Open Citations and Open Scholarly Metadata 2023, as a three-hour online event, to gather together and connect researchers, publishers and policy makers interested in the creation, reuse and improvement of open citation data and open scholarly metadata. The invited speakers provided a broad overview of the current state and future development of open bibliographic metadata and the scholarly record, and the role of OpenCitations in research assessment.
We have participated in two major European research and development project: In the context of the EC-funded project RISIS2we released the new database OpenCitations Meta and the index DOCI, while within the OpenAIRE-Nexus project network we continue to provide open bibliographic citations as part of the open data components of OpenAIRE and the European Open Science Cloud. During 2022, the OpenAIRE team supported our activities in many areas, with a particular attention to communication and dissemination, as we explained in our blog post OpenCitations and EC funding: OpenAIRE Nexus and RISIS2.
The end of 2022 brought with it two new exciting collaborations that will be developed during the coming year: the inclusion of OpenCitations in the Jisc Collections, thereby permitting UK institutions to support OpenCitations through JISC; and our involvement in the GraspOS project (starting date January 1, 2023) as an open and trusted federated infrastructure for next-generation research metrics and indicators.
Continuing support
At OpenCitations we are deeply grateful to all our members and donors that have supported us financially so far, thus making it possible to enlarge our team, maintain our technical infrastructure, and work on our new data releases.
Much of our success has been due to our selection as an infrastructure worthy for inclusion in the SCOSS second pledging round in 2019. SCOSS helped us get in touch with a wider community, and enabled OpenCitations to have an increasingly significant role in the research environment. OpenCitations is now part of the SCOSS family, the community of all SCOSS-supported infrastructures, that provides a framework of mutual support for the the Infrastructures during their growth and a safety net should troubles occur along the path.
However, OpenCitations still has a long way to go along the path from being a sustainable open infrastructure to becoming a sustained open infrastructure providing reliable and invaluable services to our global stakeholder community. There are still many long-planned activities that we hope to initiate in the near future, given sufficient resources. This is why OpenCitations requires ongoing support from the scholarly community as our three-year period of SCOSS sponsorship draws to a close. All the information you may require to start helping us financially is available on the OpenCitations website: https://opencitations.net/membership.
2023 has just started. With your support, you can help us in making it an impactful year for OpenCitations and the Open Science community as a whole.
OpenCitations is seeking applicants for a one-year Research Fellow position to be held from April 2023, for which the application closing deadline is 28 February 2023.
Currently, the amount and complexity of the data made available by OpenCitations opens up several issues related to the improvement, scalability and optimisation of its infrastructure. Therefore, it is crucial to study and customise recent approaches for managing Big Data to create a knowledge base to build decentralised, scalable, long-lasting and easily-customizable services.
The goal of the Research Fellowship is
to study the current limitations and possible improvements to introduce in the OpenCitations infrastructure
to reengineer the current infrastructure to make it more scalable, reliable, interoperable, reusable and distributed.
Thus, the Research Fellow is expected to address issues related to the provision of Web services, the management of distributed and heterogeneous databases and services, data conversion and ingestion processes, parallel computing and security. Particular attention will be given to data preservation and the long-term maintenance and update of the infrastructure.
Requirements
The minimal formal requirement for this position is a Master’s degree in Computer Science, Computer Science and Engineering, Telecommunications Engineering, Digital Humanities or equivalent.
Moreover, applicants are expected to have
sysadmin skills
computer programming skills
the ability to read, write and speak in good English
It is highly desirable for the applicants to have
demonstrable expertise in using and managing Linux servers (especially Debian and Raspbian), container orchestrator systems (e.g. Kubernetes), infrastructure-as-code systems (e.g. Terraform), Web servers implementing FastGCI (especially lighttpd), graph databases (especially Blazegraph), Python programming and Python Web Application Frameworks (especially web.py and Flask), Git-based version-control systems (especially GitHub);
a strong and demonstrable commitment to open science
demonstrable knowledge of Semantic Web technologies, Linked Data and Web technologies
Teamwork abilities and a good knowledge of Italian would also be very beneficial.
How to apply
The Research Fellowship application (written either in Italian or in English) should minimally comprise the following:
a cover letter of application, motivating why the candidate is applying for this position;
a full Curriculum Vitae containing information about prior scientific and professional jobs and the scientific production of the candidate;
a public portfolio of software, projects and activities undertaken within the past ten years, with a clear indication of the applicant’s role in each.
The Call for Applications (in Italian and in English) is available online on the website of the University of Bologna. It also includes an attachment with a description of OpenCitations and of the activities related to the position. The position has a net salary (exempt from income tax, after deduction of social security contributions) in excess of 23K euros per year. As indicated in the Call for Applications, candidates need to apply exclusively through the University of Bologna web portal.
We’re happy to announcePOCI, the OpenCitations Index of PubMed open PMID-to-PMID citations, an RDF dataset containing details of all the citations from publications bearing PubMed Identifiers (PMIDs) to other PMID-identified publications, harvested from the National Institutes of Health Open Citations Collection (NIH-OCC). The citations available in POCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model.
Each citation (i.e. an individual of the class cito:Citation) is identified by an URL structured as follows:
https://w3id.org/oc/index/poci/ci/[[OCI]].
Open Citation Identifiers
Each Open Citation Identifier [[OCI]] has a simple structure: the lower-case letters “oci” followed by a colon, followed by two numbers separated by a dash (e.g. https://w3id.org/oc/index/poci/ci/01600102060800080706-016002060909030401), in which the first number identifies the citing work and the second number identifies the cited work.
For citations in which the citing and cited works are identified by PMIDs, which includes all the POCI citations, the OCI is created in the following manner, as explained more fully here. Each converted numeral part of OCI is prefixed by a 0160, which indicates that NIH is the supplier of the original metadata of the citation (as indicated at http://opencitations.net/oci).
are available as dumps on Figshare in CSV, N-Triples and Scholix.
What is an Open Citation Index?
A citation index is a bibliographic index recording citations between publications, allowing the user to establish which later documents cite earlier documents. The current indexes available in OpenCitations are:
This blog post is the first of a series dedicated to the description and promotion of OpenCitations Meta.
In addition to OpenCitations’ Citation Indexes, OpenCitations is pleased to announce a new service: OpenCitations Meta, a database which stores and delivers bibliographic metadata for all publications involved in the OpenCitations citation indexes.
For each publication, the metadata exposed by OpenCitations Meta includes the publication’s title, type, venue (e.g. journal name), volume number, issue number, page numbers, publication date, and identifiers such as Digital Object Identifiers (DOIs) and PubMed Identifiers (PMIDs). In addition, OpenCitations Meta includes details of the main actors involved in the publication of the document, i.e., the names of the authors, editor, and publisher, each with its own additional metadata and identifier (e.g. ORCID).
Why OpenCitations Meta?
OpenCitations Meta represents a major infrastructural advance for OpenCitations, since it addresses and overcomes some of the limitations previously experienced, providing three major benefits:
1. It permits OpenCitations to respond more quickly to user requests for information.
The OpenCitations Citation Indexes contain citation-related metadata, such as the identifiers for the citing and cited publications, the citation timespan, and other citation-specific metadata, for example whether or not the citation is a self-citation. However, metadata for the citing and cited publications were hitherto not stored by OpenCitations, but rather obtained by on-the-fly API calls to external services such as Crossref, Datacite and ORCID. This dependence on responses from external services significantly slowed the delivery of the required information to users. Since OpenCitations Meta stores in-house the same bibliographic metadata for the citing and cited publications as was previously obtained by those external API calls, the former temporal limitation is now alleviated, thereby accelerating response times for requests made to the OpenCitations REST API services, and thus better supporting our users.
2. It permits OpenCitations to index citations involving publications that lack DOIs.
The second problem that OpenCitations Meta addresses goes beyond the OpenCitations infrastructure and addresses a more general issue of digital publishing, particularly in the Humanities, namely that many scholarly publications lack persistent identifiers such as DOIs and PMIDs. The citations indexed by OpenCitations are themselves treated as first-class data entities, each bearing a unique persistent Open Citation Identifier. According to the Open Citation Identifier definition [1], the entities linked by an indexed citation need to be identified by a persistent identifier (PID) from a recognised scheme such as the Digital Object Identifier scheme. For example, COCI [2] records the DOIs of the citing and cited entities. However, the association of DOIs with published documents is not uniformly spread across all scholarly fields. Indeed, in Scopus and the Web of Science Core Collection, only 50% of the current publications in the Art and Humanities are associated with a DOI [3]. In addition, in many fields, particularly in the Humanities, references are required to pre-digital historical primary sources that lack PIDs. OpenCitations Meta solves this problem by assigning a unique OpenCitations Meta Identifier (OMID) to each document. In this way, it can record the metadata of publications lacking an external identifier and store the citations involving them in one of the OpenCitations Indexes.
3. Use of the OpenCitations Meta Identifier permits de-duplication of records of bibliographic resources having more than one external identifier, such as a DOI and a PMID.
While many scholarly publications lack external PIDs, many others will have more than one external PID, for example a DOI for the record of that publication in Crossref, and a PMID for the record of that same publication in PubMed. This is perfectly legitimate. Since OpenCitations Meta assigns a unique OpenCitations Meta Identifier (OMID) to every entry, and in addition records all external identifiers already assigned to that publication, for example its DOI and its PMID, this permits the automatic de-duplication of such publications within OpenCitations Meta.
The data model
OpenCitations uses Semantic Web technologies to encode all its data, including the bibliographic metadata now stored in OpenCitations Meta. For this purpose, the OpenCitations Data Model (OCDM) [4] uses classes and properties defined in the SPAR Ontologies to represent:
bibliographic entities (fabio:Expression), e.g. a journal article
identifiers (datacite:Identifier), e.g. a DOI
agent roles (pro:RoleInTime), e.g. an author
responsible agents (foaf:Agent), e.g. a person
publication format details (fabio:Manifestation), e.g. first and last pages
Figure 1 depicts relationships between some of the various entities.
Figure 1. A small subset of the OCDM used in OC Meta. Yellow rectangles represent classes, green polygons represent data types, while blue and green arrows represent object properties and data properties, respectively
In addition to storing data, great importance is given to recording provenance and change-tracking of OpenCitations Meta entries, including the agent who processed a specific entity, i.e. creating, deleting, modifying or merging it, the date and time on which this action was performed, and the primary source of the data.
The data
At the release date, 20th December 2022, OpenCitations Meta is using Crossref as its primary source, with the purpose of hosting the metadata describing all citing and cited publications involved in the 1,392,036,835 citations within COCI [2]. From a quantitative point of view, OpenCitations Meta presently contains metadata describing 87,321,593 bibliographic entities, 277,750,235 authors, 2,359,301 editors, 17,268 publishers, and 710,226 publication venues (e.g. journal names). Editors and authors have been counted by their roles, without at this stage disambiguating individuals with the same name. Conversely, bibliographic entities, publishers, and publication venues were counted by their OMIDs. In total, the OpenCitations Meta triplestore presently contains 3,298,335,581 triples.
In the following days, OpenCitations Meta will be expanded to include bibliographic metadata for all citing and cited publications involved in the 169,822,752 citations from DataCite within DOCI, released on 13th December 2022.
Table 1 lists the various types of bibliographic resource represented in OpenCitations Meta. The current dataset contains mostly journal articles (60,683,841), which exceed the number of book chapters (6,288,351) by about ten times and the number of conference proceedings articles (4,917,568) by about twelve times.
OpenCitations Meta allows the users to explore the contained metadata either via SPARQL (https://opencitations.net/meta/sparql) or via the OpenCitations Meta REST API (https://opencitations.net/meta/api/v1). In particular, the API will retrieve a list of bibliographic resources and related metadata starting from one or more identifiers including an OpenCitations Meta Identifier (OMID), an author’s ORCID, or an editor’s ORCID.
Finally, all the metadata and provenance information present in OpenCitations Meta are available as dumps in RDF (JSON-LD) or CSV format under a CC0 licence. Every time a citation index is updated, we will update OpenCitations Meta.
Conclusion
OpenCitations Meta makes search operations on the metadata of scholarly publications involved in the citations indexed in the OpenCitations Citation Indexes faster, since the metadata are now held in-house. The use of OpenCitations Meta Identifiers (OMIDs) permits OpenCitations Meta to act as a mapping hub to disambiguate and unify information on single publications identified by more than one external identifier, and makes it possible to characterise citations involving resources lacking an external identifier.
In the future, we will introduce a new API operation to perform text searches. Moreover, OpenCitations Meta will incorporate metadata from other sources in addition to Crossref. As mentioned above, work is already under way to ingest bibliographic metadata from Datacite for the publications involved in the citations newly indexed in DOCI. This will soon be followed with similar OpenCitations Meta entries relating to a new citation index, POCI, containing citations from PubMed, to be released later this year. PubMed metadata will be made available by February 2023.
Heibi, I., Peroni, S. & Shotton, D. Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. Scientometrics 121, 1213–1228 (2019). https://doi.org/10.1007/s11192-019-03217-6
Gorraiz, J., Melero-Fuentes, D., Gumpenberger, C., & Valderrama-Zurián, J.-C. (2016). Availability of digital object identifiers (DOIs) in Web of Science and Scopus. Journal of Informetrics, 10 (1), 98–10. https://doi.org/10.1016/j.joi.2015.11.008
We’re excited to introduce DOCI, the OpenCitations Index of Datacite open DOI-to-DOI citations, a new tool containing citations derived from publications bearing DataCite DOIs to other DOI-identified publications, harvested from DataCite. The citations available in DOCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model.
For citations in which the citing and cited works are identified by DOIs, which includes all the DOCI citations, the OCI is created in the following manner, as explained more fully here. Each case-insensitive DOI is first normalized to lower case letters. Then, after omitting the initial doi:10. prefix, the alphanumeric string of the DOI is converted reversibly to a pure numerical string using the simple two-numeral lookup table for numerals, lower case letters and other characters presented at https://github.com/opencitations/oci/blob/master/lookup.csv. Finally, each converted numeral is prefixes by a 080, which indicates that DataCite is the supplier of the original metadata of the citation (as indicated at http://opencitations.net/oci).
A citation index is a bibliographic index recording citations between publications, allowing the user to establish which later documents cite earlier documents. The current indexes available in OpenCitations are:
The Wikipedia entry for OpenCitations is woefully out of date, inaccurate and brief. As Directors of OpenCitations, Silvio and I are unable to improve this situation because of Wikipedia’s proper conflict-of-interest restriction on self-promotion.
OpenCitations is actively seeking greater involvement from members of the global academic community, as explained in our Mission Statement. One way in which such individuals, particularly those who are both existing Wikipedia editors and already know about OpenCitations, can help OpenCitations, while at the same time supporting Wikipedia in its quest for accurate information, is by revising and expanding the present Wikipedia entry on OpenCitations to reflect the infrastructure’s current activities and data holdings, while maintaining perspective and a neutral point of view. This will increase the availability of reliable knowledge about OpenCitations and its place in the ecosystem of Open Science infrastructures.
Background information to assist in this task is available in the cannonical paper describing OpenCitations:
Silvio Peroni and David Shotton (2020). OpenCitations, an infrastructure organization for open scholarship. Quantitative Science Studies 1: 428-444. https://doi.org/10.1162/qss_a_00023;
in previous posts in the OpenCitations Blog and in the documents to which those posts link, particularly in:
OpenCitations has happily joined the open-source social media platform joinmastodon.org.
Mastodon is “a free and open-source software developed by a non-profit organization”, with the aim of favouring interoperability and bringing social media interaction “back in the hands of the people”.
We look forward to recreating there our wide network of connections, and getting in touch with new people, projects and institutions in a different virtual environment.
Blog post by Ivan Heibi (Universiy of Bologna) and Arcangelo Massari (University of Bologna).
OpenCitations publishes the COCI dataset after each new release in three main formats: CSV, N-Triples, and Scholix (see https://opencitations.net/download#coci). The CSV format is the most popular and downloaded one due to its comprehensive data organization (i.e. tabular format) and smaller size (compared to the other formats provided). Therefore, this is also the format we suggest using for a local process of the entire COCI dataset.
The CSV dumps of COCI are uploaded on Figshare. You can check and download the last dump released from https://doi.org/10.6084/m9.figshare.6741422. The dump consists of one main ZIP file, including other smaller ZIP archives (one for each release) containing the actual CSV files (Figure 1).
Figure 1. The contents of the COCI CSV dataset (after the August 2022 release)
It is possible to process this data without unzipping the internal archives, thus saving a lot of disk space. In this tutorial, we will see how to achieve this in Python. Same process could be done in other programming languages.
Processing the COCI dump using Python
Step 1) Downloading the COCI dump
First, you need to download the last CSV dump release of COCI from https://doi.org/10.6084/m9.figshare.6741422 and decompress only the external archive. After this operation, you should have a folder containing the internal ZIP files such as in Figure 1.
Note: It is beneficial to decompress the external archive because doing so does not increase the space occupied on the disk (compressing archives results in a compression rate of 0%) and because working on nested archives would significantly increase RAM requirements.
Step 2) Working with the ZIP files
Python provides the built-in zipfile module, whose ZipFile class allows you to create, read, write, edit and list the contents of a ZIP file. Given as input the path of the root directory containing all the ZIP files (FOLDER_PATH), the process elaborates each of these files on a different iteration. Each cycle initializes a ZipFile object by specifying the path to the ZIP file (archive_path).
from zipfile import ZipFile
import os
for archive_name in os.listdir(FOLDER_PATH): archive_path = os.path.join(FOLDER_PATH, archive_name) with ZipFile(archive_path) as archive:
# ...
Step 3) Accessing the ZIP files
Use the namelist() method to return the list of CSV files contained in each archive. Then to open the inner CSV files, simply cycle through the list of names and feed them to the open() method of the ZipFile instance, i.e. archive in the example below.
from zipfile import ZipFile
import os
for archive_name in os.listdir(FOLDER_PATH):
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:
for csv_name in archive.namelist(): with archive.open(csv_name) as csv_file:
# ...
Step 4) Reading the CSVs
The .open() method returns a buffer. To read the CSV file as a list of dictionaries (i.e. represent each row of the CSV in dictionary format, e.g., {“column1″:”val1”, “column2″:”val2”}) we need to transform the buffer using the TextIOWrapper class and read it using the DictReader class of csv. Then we convert the result of DictReader into a list.
from io import TextIOWrapper
from zipfile import ZipFile
import os
for archive_name in os.listdir(FOLDER_PATH):
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:
for csv_name in archive.namelist(): with archive.open(csv_name) as csv_file: reader = csv.DictReader(io.TextIOWrapper(csv_file)) rows = list(reader) # ...
Step 5) Processing the CSVs content
Now you can go through each row of the list and process the citation data as you want. The following example prints the citing and cited entity of each citation in the dump.
from io import TextIOWrapper
from zipfile import ZipFile
import os
for archive_name in os.listdir(FOLDER_PATH):
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:
for csv_name in archive.namelist(): with archive.open(csv_name) as csv_file: reader = csv.DictReader(io.TextIOWrapper(csv_file)) rows = list(reader) # Process the CSV here for r in rows: print("Citing entity:",r["citing"]) print("Cited entity:",r["cited"])
