Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

A new revolutionary workflow for a unified collection of citations: say hello to the OpenCitations Index

Blog post by Ivan Heibi (University of Bologna), Arianna Moretti (University of Bologna) and Chiara Di Giambattista (University of Bologna).

In the past five years, the OpenCitations data has been enriched with numerous new indexes of open citation data from different sources. However, the quantity and diversification of the ingested information have raised several issues, which recently made it essential to conduct a complete revision of the ingestion workflow. The result was a revolution in the way OpenCitations data is delivered. In this blog post, we will explain the context and challenges raised by the old procedure. Then, we will present the new ingestion workflow, designed to produce just two comprehensive collections: OpenCitations Index, collecting open citation data, and OpenCitations Meta, for the open bibliographical metadata. 

Once upon a time, there were five OpenCitations indexes…

In 2018, OpenCitations released the kickoff version of its first citation index, COCI (citations from Crossref), which contained around 300 million citation links derived from the subset of the reference lists in the Crossref database, where citing and cited entities were identified using Digital Object Identifiers (DOIs). COCI gathered citations with associated metadata in compliance with the recommendations from the Initiative for Open Citations (I4OC) that citation data should be structured, separable, and open, thus marking a turning point by providing a disruptive and free and open alternative to earlier sources such as Google Scholar, which provided freely accessible data although not downloadable, and Web of Science or Scopus, which demanded paid access. 

In a short time, COCI became a competitive and trusted index of citation data, used by numerous institutional repositories, including B!son and Optimeta. In 2021, COCI was taken into account in a comparative study with the most relevant sources in the landscape, including the proprietary ones, which showed its coverage approaching parity with those of the other sources involved in the analysis (Microsoft Academic, Scopus, Dimensions, and Web of Science). At the time of its most recent update in January 2023, COCI counted more than 1.4 billion citations. The reason behind this outstanding number lies in several factors, including Elsevier’s endorsement of the Declaration on Research Assessment (DORA) in December 2020, leading to the open release via Crossref of the reference lists of the articles published in all its journals, and confirming the value of initiatives such as the Initiative for Open Citations (I4OC)

However, before this change of heart, in 2019 OpenCitations had tried to narrow the open citations coverage gap by launching its second index, the Crowdsourced Open Citations Index (CROCI). This index allowed publishers and scholars to contribute directly by uploading crowdsourced open citations into the OpenCitations infrastructure.

In December 2022, a new concrete step towards a factual plurality of OpenCitations indexes was taken by the ingestion of new data sources into the infrastructure, with the publication of the inaugural dumps of DOCI (citations from DataCite) and POCI (citations from PubMed). In June 2023, the first version of the OROCI (citations from OpenAIRE) dump was released too, and JOCI (citations from JALC) is expected to be available by the end of November 2023, for a total of five collections from different sources. 

Why a new workflow? The issues with multiple sources management and new challenges

While having such a variety and richness of indexes helped present the extent of OpenCitations sources, the recent increment in the number of sources and the diversification of data integrated led to two primary issues:

    1. the necessity to handle the ingestion of new identifier types in a DOI-based software infrastructure, and
    2. the consequent possibility of encountering the same citation expressed by several sources with different identifiers.

Moreover, it soon became evident the need to optimize the reuse of the already developed software components to facilitate the metadata crosswalk processes between the new sources’ data models and the OpenCitations Data Model, with the aim to define a functional and easily extendable workflow to be easily reused when it comes to incorporating new data sources, which should be: 

    1. sufficiently generic to establish a globally unique procedure; 
    2. customizable enough to capture the necessary information within each of the specific data models and formats. 

As a solution, we decided to use OpenCitations Meta, the new OpenCitations database and tool for managing bibliographic data related to the publications involved in the citations. OpenCitations Meta makes it possible to assign each entity involved in a citation an internal identifier, nominally the OpenCitations Meta Identifier (OMID), to which all the associated persistent identifiers of the same publication are redirected.

As a result, the allocation of an OMID for each bibliographic resource also enabled the unambiguous identification of each citation, regardless of the persistent identifier schema originally used by the data source to identify the resources. This approach allowed us to perform data deduplication and finally make all the sources’ contributions converge into a unified index containing all the unique citations managed by OpenCitations, expressed as OMID to OMID citation links.

The revised workflow

The new workflow is based on three main components with the benefit of optimizing the process both in terms of computational cost and in terms of flexibility. As shown in Fig. 1, in a preliminary step, source-specific software converts the input dataset – structured according to the source data model – to extract two OpenCitations Data Model compliant data collections in tabular format for bibliographic metadata and citation data, respectively.

The following steps are common to the process of each dataset.  

STEP 1: The bibliographic metadata collection is used as input for the META software. At this stage, it is checked whether or not the bibliographic entities have been previously integrated into our infrastructure (coming from other data sources). If so, the existing OMID is linked also to the new alternative identifiers of the new bibliographic resources. New metadata values, if any, are also integrated. A new OMID identifier is produced for entities never previously encountered, uniquely representing the bibliographic resource in OpenCitations. The outputs of the process are: (I) an updated version of the OpenCitations Meta collection that also includes the metadata of the bibliographic entities provided by the new source, and (II) a collection of provenance data. An internal database is constantly refreshed to preserve correspondence between IDs and the associated internal OMIDs.

STEP 2: Starting from the collection of citations expressed as directional links between identifiers of potentially any type (e.g., DOI-DOI, PMID-PMID, PMC-PMID, etc.), the INDEX software queries the internal database mapping IDs to OMIDs to produce an updated version of the OpenCitations Index: unique citations expressed as OMID-OMID links in different formats, accompanied by their corresponding provenance data.

Fig. 1: An overview of the data ingestion workflow, starting from the data source-specific conversion and production of citations and bibliographic metadata tables, progressing through the META process and the assignation of an OMID identifier to each bibliographic record involved in a citation, and culminating with the exposition of the OpenCitations Index collection of OMID-OMID unique citations.

What we have now: The OpenCitations Index 

From now on, OpenCitations will no longer display an index of citation data for each source. Instead, we will publish a single collection of citations into which the contributions from each of the sources will flow, which we will simply call ‘The OpenCitations Index‘. The first version of this unified index of OMID-OMID citations is posted on Figshare. It was produced in RDF, CSV, and SCHOLIX formats, together with a collection of its provenance information, provided in RDF and CSV formats. For each citation, it is possible to trace the source of the information by consulting the Provenance data collection, thanks to the http://www.w3.org/ns/prov#atLocation property, which defines the location of each citation.

This new solution has the benefit of simplifying the consultation of the data maintained by our infrastructure without reducing the information content. In addition, by including efficient handling of the deduplication problem, the new Index not only provides accurate data on the exact number of unique citations exposed by the framework but also verifies the individual contribution of each source, as well as their overlapping data (Fig. 2).

Fig. 2: An overview of the number of citations stored in the OpenCitations Index as of October 31, 2023. The diagonal cells in the table (highlighted in yellow) show the unique contribution of each collection to the OpenCitations Index, while the other cells represent the citations that are shared between the collections. More in detail, the green cells show the overall input of each source, while the pink cells represent the number of overlapping citations between two data sources.

Currently, the Index contains almost 2 billion unique citations. By the end of November, a new version of the collection will be published, including the contribution of the new Japan Link Centre (JaLC) source. 

How to access the OpenCitations Index data

To maximize the reuse of the exposed information and to ensure the greatest possible interoperability, the collection will always be published on Figshare in all formats listed above. In addition, the data will be accessible via an API, a SPARQL endpoint, and a web interface.

The redesign of the ingestion workflow marks a fundamental step for OpenCitations towards a more intuitive and simple access to our services while always preserving and improving the quality of our data. If you need further information on how the new workflow works, please visit our website, contact us at contact@opencitations.net  or leave feedback and/or suggestions in the dedicated card on our public roadmap to help us improve our services and communications. Thank you!

The French National Fund for Open Science renews its support to OpenCitations

We are delighted to announce that the French National Fund for Open Science (FNSO) has renewed its commitment to sustaining the activities of four SCOSS-selected infrastructures, including OpenCitations. 

The four supported infrastructures (OpenCitations, the DOAB, LA Referencia and ROR) “were evaluated by the jury composed by SCOSS, then according to the exemplary criteria of the Committee for the Open Science, which notably guarantee transparency and the participation of the scientific communities in their governance”.

Since 2020, the FNSO has acknowledged OpenCitations as an infrastructure worth its financial support, thanks to its mission of disseminating bibliographic and citation metadata in open access with a level of quality and coverage, thus providing a workable, free and open alternative to the academic community’s current dependency on proprietary tools. OpenCitations’ work therefore frees up citation analysis, promotes the evolution of bibliometric indicators and the broadening knowledge of science.

The FNSO is now contributing to OpenCitations with recurring funding for 2023, 2024 and 2025 for an annual amount of €75,000. This generous support will be crucial in sustaining the maintenance and development of OpenCitations’ technical infrastructure, and in supporting the future activities of the OpenCitations team, that are publicly displayed in the OpenCitations Roadmap

We are extremely honoured and grateful to the French National Fund for Open Science for renewing the pledge of such a portion of its open science budget to support our work. 

Discover POCI, the index of open citations from PubMed 

We’re happy to announce POCI, the OpenCitations Index of PubMed open PMID-to-PMID citations, an RDF dataset containing details of all the citations from publications bearing PubMed Identifiers (PMIDs) to other PMID-identified publications, harvested from the National Institutes of Health Open Citations Collection (NIH-OCC). The citations available in POCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. 

Currently, POCI’s December 2022 release contains 717,654,703 citations from 26,024,862 bibliographic resources, and is based on the dump of NIH Open Citation Collection dated November 2022. 

Citation URLs

Each citation (i.e. an individual of the class cito:Citation) is identified by an URL structured as follows:

https://w3id.org/oc/index/poci/ci/[[OCI]].

Open Citation Identifiers

Each Open Citation Identifier [[OCI]] has a simple structure: the lower-case letters “oci” followed by a colon, followed by two numbers separated by a dash (e.g. https://w3id.org/oc/index/poci/ci/01600102060800080706-016002060909030401), in which the first number identifies the citing work and the second number identifies the cited work.

For citations in which the citing and cited works are identified by PMIDs, which includes all the POCI citations, the OCI is created in the following manner, as explained more fully here. Each converted numeral part of OCI is prefixed by a 0160, which indicates that NIH is the supplier of the original metadata of the citation (as indicated at http://opencitations.net/oci).

OCIs can be resolved using the OpenCitations OCI Resolution Service.

Access to POCI data

All the data in POCI:

What is an Open Citation Index?

A citation index is a bibliographic index recording citations between publications, allowing the user to establish which later documents cite earlier documents. The current indexes available in OpenCitations are:  

All the OpenCitations Indexes have six characteristics in common, summarized here: https://opencitations.net/index   

 

Discover DOCI, the index of open citations from DataCite

We’re excited to introduce DOCI, the OpenCitations Index of Datacite open DOI-to-DOI citations, a new tool containing citations derived from publications bearing DataCite DOIs to other DOI-identified publications, harvested from DataCite. The citations available in DOCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model

Currently, DOCI’s December 2022 release contains 169,822,752 citations from 1,753,860  citing resources, and is based on the last dump of DataCite dated 22 October 2021 provided by the Internet Archive

Citation URLs

Each citation (i.e. an individual of the class cito:Citation) is identified by an URL structured as follows:

 https://w3id.org/oc/index/doci/ci/[[OCI]].

Open Citation Identifiers

Each Open Citation Identifier [[OCI]] has a simple structure: the lower-case letters “oci” followed by a colon, followed by two numbers separated by a dash (e.g. https://opencitations.net/index/doci/ci/080010504060836132137200707121027-080010504060836161221130313.html), in which the first number identifies the citing work and the second number identifies the cited work.

For citations in which the citing and cited works are identified by DOIs, which includes all the DOCI citations, the OCI is created in the following manner, as explained more fully here. Each case-insensitive DOI is first normalized to lower case letters. Then, after omitting the initial doi:10. prefix, the alphanumeric string of the DOI is converted reversibly to a pure numerical string using the simple two-numeral lookup table for numerals, lower case letters and other characters presented at https://github.com/opencitations/oci/blob/master/lookup.csv. Finally, each converted numeral is prefixes by a 080, which indicates that DataCite is the supplier of the original metadata of the citation (as indicated at http://opencitations.net/oci).

OCIs can be resolved using the OpenCitations OCI Resolution Service.

Access to DOCI data

All the data in DOCI:

More information is available at https://opencitations.net/index/doci. 

What is an Open Citation Index? 

A citation index is a bibliographic index recording citations between publications, allowing the user to establish which later documents cite earlier documents. The current indexes available in OpenCitations are: 

All the OpenCitations Indexes have six characteristics in common, summarized here: https://opencitations.net/index  

Follow OpenCitations on Mastodon

OpenCitations has happily joined the open-source social media platform joinmastodon.org.

Mastodon is “a free and open-source software developed by a non-profit organization”, with the aim of favouring interoperability and bringing social media interaction “back in the hands of the people”.

We look forward to recreating there our wide network of connections, and getting in touch with new people, projects and institutions in a different virtual environment.

Follow us at https://scicomm.xyz/@opencitations !

Tutorial: how to process COCI’s zipped CSV dump without decompressing it

Blog post by Ivan Heibi (Universiy of Bologna) and Arcangelo Massari (University of Bologna).

OpenCitations publishes the COCI dataset after each new release in three main formats: CSV, N-Triples, and Scholix (see https://opencitations.net/download#coci). The CSV format is the most popular and downloaded one due to its comprehensive data organization (i.e. tabular format) and smaller size (compared to the other formats provided). Therefore, this is also the format we suggest using for a local process of the entire COCI dataset. 

The CSV dumps of COCI are uploaded on Figshare. You can check and download the last dump released from https://doi.org/10.6084/m9.figshare.6741422. The dump consists of one main ZIP file, including other smaller ZIP archives (one for each release) containing the actual CSV files (Figure 1).

Figure 1. The contents of the COCI CSV dataset (after the August 2022 release)

It is possible to process this data without unzipping the internal archives, thus saving a lot of disk space. In this tutorial, we will see how to achieve this in Python. Same process could be done in other programming languages.

Processing the COCI dump using Python

Step 1) Downloading the COCI dump

First, you need to download the last CSV dump release of COCI from https://doi.org/10.6084/m9.figshare.6741422 and decompress only the external archive. After this operation, you should have a folder containing the internal ZIP files such as in Figure 1.

Note: It is beneficial to decompress the external archive because doing so does not increase the space occupied on the disk (compressing archives results in a compression rate of 0%) and because working on nested archives would significantly increase RAM requirements. 

Step 2) Working with the ZIP files

Python provides the built-in zipfile module, whose ZipFile class allows you to create, read, write, edit and list the contents of a ZIP file. Given as input the path of the root directory containing all the ZIP files (FOLDER_PATH), the process elaborates each of these files on a different iteration. Each cycle initializes a ZipFile object by specifying the path to the ZIP file (archive_path).

from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):        
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:     # ...

Step 3) Accessing the ZIP files

Use the namelist() method to return the list of CSV files contained in each archive. Then to open the inner CSV files, simply cycle through the list of names and feed them to the open() method of the ZipFile instance, i.e. archive in the example below.

from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:       # ...

Step 4) Reading the CSVs

The .open() method returns a buffer. To read the CSV file as a list of dictionaries (i.e. represent each row of the CSV in dictionary format, e.g., {“column1″:”val1”, “column2″:”val2”}) we need to transform the buffer using the TextIOWrapper class and read it using the DictReader class of csv. Then we convert the result of DictReader into a list. 

from io import TextIOWrapper
from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# ...

Step 5) Processing the CSVs content

Now you can go through each row of the list and process the citation data as you want. The following example prints the citing and cited entity of each citation in the dump. 

from io import TextIOWrapper
from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# Process the CSV here
for r in rows:
print("Citing entity:",r["citing"])
print("Cited entity:",r["cited"])

 

Additional 48 million citations in COCI, including references from IEEE 

We announce the August 2022 release of COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations, which is based on open references to works with DOIs within the Crossref dump dated August 2022. This new release extends COCI with more than 48 million additional citations, giving a total number of more than 1.36 billion DOI-to-DOI citation links. 

This release includes citations from the articles published over the last four years by IEEE, whose bibliographic references were opened in June 2022. 

A fundamental role in pushing the commercial publishers to open their citation data was played by Crossref’s recent announcement to change its reference distribution policy, by making all its metadata open.  

Besides IEEE, COCI already includes the citation data derived from Elsevier (open via Crossref since December 2020) and from the last articles published by the American Chemical Society (whose references were opened in February 2021) 

You can find more information about COCI in our open-access article  

Ivan Heibi, Silvio Peroni & David Shotton (2019). Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. Scientometrics, 121 (2): 1213-1228. DOI: https://doi.org/10.1007/s11192-019-03217-6    

Finally, just a reminder that the bibliographic and citation data in COCI:  

    • can be queried using the OpenCitations Indexes SPARQL endpoint;  
    • can be retrieved by using the COCI REST API;  
    • can be searched by using the OpenCitations Indexes Search Interface;  
    • are also available as dumps on Figshare in CSV, N-Triples, and Scholix; and  
    • can be freely re-used for any purpose.

New documents that present OpenCitations’ mission, unique benefits, present status and future plans

Posted on August 10th 2022 by Chiara Di Giambattista

More than a year ago, Ginny Hendricks, Director of Member & Community Outreach for Crossref, and a valued member of the OpenCitations International Advisory Board, published on the Crossref blog the post “The road ahead: our strategy through 2025”. In order to describe all Crossref’s principles and activities, Ginny presented the Crossref strategic planning framework as a diagram summarizing Crossref’s statements, key messages and truths. The clarity and immediacy of the diagram were such that we adapted it to present  OpenCitations’ own statements and goals. The resulting poster “OpenCitations – what does the future hold?” was presented by our Director David Shotton at the OASPA2021 conference, and can be found in this blog post.

Although the poster offered a wide overview of OpenCitations values, unique traits, benefits and plans, it differed slightly from Ginny’s original diagram, in particular because it lacked a “Mission Statement”, scattering the relevant information within the “Values” and “Principles” boxes. Indeed, at that time (September 2021), we didn’t have a clearly defined Mission Statement.

Nevertheless, the creation of that poster was crucial in helping us start to articulate more clearly the purpose and meaning of OpenCitations. As David underlined in his post “From little acorns…a retrospective on OpenCitations”, since 2018 OpenCitations activities have progressively increased and, with them, the number of related journal articles, conference papers and technical definitions. OpenCitations’ involvement in international networks and collaborations (such as SCOSS and the OpenAIRE-Nexus project), together with our need of identifying and reaching out to new stakeholders to assure OpenCitations’ development and sustainability, has made it necessary to publicly define OpenCitations’ mission, unique strengths and next developmental steps.

After numerous revisions, aided by wise advice from members of the OpenCitations Advisory Board members, we’re now happy to publish the following three OpenCitations documents:

OpenCitations Mission Statement,

The Uniqueness of OpenCitations   and

OpenCitations – Present Status and Future Plans,

which together provide a summary of why we exist and where we are heading.

We are particularly proud of the definition of OpenCitations’ primary mission, namely

to harvest and openly publish accurate and comprehensive metadata describing the world’s academic publications and the scholarly citations that link them, and to preserve ongoing access to this information by secure archiving.

The Mission Statement also presents brief descriptions of the OpenCitations context, our vision, our value proposition and our relationship with the community and stakeholders.

The Uniqueness of OpenCitations provides the answer to the question ‘Why choose to use OpenCitations?’, and is a detailed presentation of OpenCitations’ benefits.

OpenCitations – Present Status and Future Plans summarizes OpenCitations’ ongoing activities, that can be quickly visualized on our public roadmap. It also introduces the OpenCitations Working Groups, served by the members of the OpenCitations International Advisory Board, which are currently working on the themes of governance evolution and community building, with the common purpose of driving OpenCitations along the path from being a ‘sustainable infrastructure’ (in POSI terms) to being an enduring community led and financially sustained infrastructure.

In fulfilling our mission and reaching our goals, the support and vital interest of our community members is fundamental. We request that you, as a member of our community, provide us with feedback on these documents and the ideas they contain, or indeed to ask for clarifications, to help us improving our mission and our communications to explain it. You can reach us here: contact@opencitations.net.

Thank you!