A new revolutionary workflow for a unified collection of citations: say hello to the OpenCitations Index

Blog post by Ivan Heibi (University of Bologna), Arianna Moretti (University of Bologna) and Chiara Di Giambattista (University of Bologna).

In the past five years, the OpenCitations data has been enriched with numerous new indexes of open citation data from different sources. However, the quantity and diversification of the ingested information have raised several issues, which recently made it essential to conduct a complete revision of the ingestion workflow. The result was a revolution in the way OpenCitations data is delivered. In this blog post, we will explain the context and challenges raised by the old procedure. Then, we will present the new ingestion workflow, designed to produce just two comprehensive collections: OpenCitations Index, collecting open citation data, and OpenCitations Meta, for the open bibliographical metadata. 

Once upon a time, there were five OpenCitations indexes…

In 2018, OpenCitations released the kickoff version of its first citation index, COCI (citations from Crossref), which contained around 300 million citation links derived from the subset of the reference lists in the Crossref database, where citing and cited entities were identified using Digital Object Identifiers (DOIs). COCI gathered citations with associated metadata in compliance with the recommendations from the Initiative for Open Citations (I4OC) that citation data should be structured, separable, and open, thus marking a turning point by providing a disruptive and free and open alternative to earlier sources such as Google Scholar, which provided freely accessible data although not downloadable, and Web of Science or Scopus, which demanded paid access. 

In a short time, COCI became a competitive and trusted index of citation data, used by numerous institutional repositories, including B!son and Optimeta. In 2021, COCI was taken into account in a comparative study with the most relevant sources in the landscape, including the proprietary ones, which showed its coverage approaching parity with those of the other sources involved in the analysis (Microsoft Academic, Scopus, Dimensions, and Web of Science). At the time of its most recent update in January 2023, COCI counted more than 1.4 billion citations. The reason behind this outstanding number lies in several factors, including Elsevier’s endorsement of the Declaration on Research Assessment (DORA) in December 2020, leading to the open release via Crossref of the reference lists of the articles published in all its journals, and confirming the value of initiatives such as the Initiative for Open Citations (I4OC)

However, before this change of heart, in 2019 OpenCitations had tried to narrow the open citations coverage gap by launching its second index, the Crowdsourced Open Citations Index (CROCI). This index allowed publishers and scholars to contribute directly by uploading crowdsourced open citations into the OpenCitations infrastructure.

In December 2022, a new concrete step towards a factual plurality of OpenCitations indexes was taken by the ingestion of new data sources into the infrastructure, with the publication of the inaugural dumps of DOCI (citations from DataCite) and POCI (citations from PubMed). In June 2023, the first version of the OROCI (citations from OpenAIRE) dump was released too, and JOCI (citations from JALC) is expected to be available by the end of November 2023, for a total of five collections from different sources. 

Why a new workflow? The issues with multiple sources management and new challenges

While having such a variety and richness of indexes helped present the extent of OpenCitations sources, the recent increment in the number of sources and the diversification of data integrated led to two primary issues:

    1. the necessity to handle the ingestion of new identifier types in a DOI-based software infrastructure, and
    2. the consequent possibility of encountering the same citation expressed by several sources with different identifiers.

Moreover, it soon became evident the need to optimize the reuse of the already developed software components to facilitate the metadata crosswalk processes between the new sources’ data models and the OpenCitations Data Model, with the aim to define a functional and easily extendable workflow to be easily reused when it comes to incorporating new data sources, which should be: 

    1. sufficiently generic to establish a globally unique procedure; 
    2. customizable enough to capture the necessary information within each of the specific data models and formats. 

As a solution, we decided to use OpenCitations Meta, the new OpenCitations database and tool for managing bibliographic data related to the publications involved in the citations. OpenCitations Meta makes it possible to assign each entity involved in a citation an internal identifier, nominally the OpenCitations Meta Identifier (OMID), to which all the associated persistent identifiers of the same publication are redirected.

As a result, the allocation of an OMID for each bibliographic resource also enabled the unambiguous identification of each citation, regardless of the persistent identifier schema originally used by the data source to identify the resources. This approach allowed us to perform data deduplication and finally make all the sources’ contributions converge into a unified index containing all the unique citations managed by OpenCitations, expressed as OMID to OMID citation links.

The revised workflow

The new workflow is based on three main components with the benefit of optimizing the process both in terms of computational cost and in terms of flexibility. As shown in Fig. 1, in a preliminary step, source-specific software converts the input dataset – structured according to the source data model – to extract two OpenCitations Data Model compliant data collections in tabular format for bibliographic metadata and citation data, respectively.

The following steps are common to the process of each dataset.  

STEP 1: The bibliographic metadata collection is used as input for the META software. At this stage, it is checked whether or not the bibliographic entities have been previously integrated into our infrastructure (coming from other data sources). If so, the existing OMID is linked also to the new alternative identifiers of the new bibliographic resources. New metadata values, if any, are also integrated. A new OMID identifier is produced for entities never previously encountered, uniquely representing the bibliographic resource in OpenCitations. The outputs of the process are: (I) an updated version of the OpenCitations Meta collection that also includes the metadata of the bibliographic entities provided by the new source, and (II) a collection of provenance data. An internal database is constantly refreshed to preserve correspondence between IDs and the associated internal OMIDs.

STEP 2: Starting from the collection of citations expressed as directional links between identifiers of potentially any type (e.g., DOI-DOI, PMID-PMID, PMC-PMID, etc.), the INDEX software queries the internal database mapping IDs to OMIDs to produce an updated version of the OpenCitations Index: unique citations expressed as OMID-OMID links in different formats, accompanied by their corresponding provenance data.

Fig. 1: An overview of the data ingestion workflow, starting from the data source-specific conversion and production of citations and bibliographic metadata tables, progressing through the META process and the assignation of an OMID identifier to each bibliographic record involved in a citation, and culminating with the exposition of the OpenCitations Index collection of OMID-OMID unique citations.

What we have now: The OpenCitations Index 

From now on, OpenCitations will no longer display an index of citation data for each source. Instead, we will publish a single collection of citations into which the contributions from each of the sources will flow, which we will simply call ‘The OpenCitations Index‘. The first version of this unified index of OMID-OMID citations is posted on Figshare. It was produced in RDF, CSV, and SCHOLIX formats, together with a collection of its provenance information, provided in RDF and CSV formats. For each citation, it is possible to trace the source of the information by consulting the Provenance data collection, thanks to the http://www.w3.org/ns/prov#atLocation property, which defines the location of each citation.

This new solution has the benefit of simplifying the consultation of the data maintained by our infrastructure without reducing the information content. In addition, by including efficient handling of the deduplication problem, the new Index not only provides accurate data on the exact number of unique citations exposed by the framework but also verifies the individual contribution of each source, as well as their overlapping data (Fig. 2).

Fig. 2: An overview of the number of citations stored in the OpenCitations Index as of October 31, 2023. The diagonal cells in the table (highlighted in yellow) show the unique contribution of each collection to the OpenCitations Index, while the other cells represent the citations that are shared between the collections. More in detail, the green cells show the overall input of each source, while the pink cells represent the number of overlapping citations between two data sources.

Currently, the Index contains almost 2 billion unique citations. By the end of November, a new version of the collection will be published, including the contribution of the new Japan Link Centre (JaLC) source. 

How to access the OpenCitations Index data

To maximize the reuse of the exposed information and to ensure the greatest possible interoperability, the collection will always be published on Figshare in all formats listed above. In addition, the data will be accessible via an API, a SPARQL endpoint, and a web interface.

The redesign of the ingestion workflow marks a fundamental step for OpenCitations towards a more intuitive and simple access to our services while always preserving and improving the quality of our data. If you need further information on how the new workflow works, please visit our website, contact us at contact@opencitations.net  or leave feedback and/or suggestions in the dedicated card on our public roadmap to help us improve our services and communications. Thank you!

Cite this article as: Chiara Di Giambattista, "A new revolutionary workflow for a unified collection of citations: say hello to the OpenCitations Index," in OpenCitations blog, 27/11/2023, https://opencitations.hypotheses.org/3499.

OpenCitations’ 2022: one year, numerous outcomes

2023 has arrived, and it’s the time to put aside the old agendas and to fill the freshly printed organizers with new tasks. However, it is also worth looking back over the achievements of the past year.

So, before consigning 2022 to the archives, we want to take a moment to recall and celebrate all that OpenCitations has achieved over the past year, in terms of technical developments, community building, and our own internal organization.

Clarifying our objectives

Since its foundation, OpenCitations has been presented to the scholarly community as a infrastructure organization guided by the values of openness, equity and accessibility. In 2022, we publicly clarified our mission, our aims and our unique points by publishing these three documents:

In addition, we continued to monitor OpenCitations’ compliance with the Principles of Open Scholarly Infrastructure. Because one area in which we were still falling short involved stakeholder involvement in OpenCitations’ governance, in 2022 OpenCitations started reflecting on its organizational structure by creating a Governance Evolution Working Group. In addition, we explored the further development of our stakeholder community by establishing a parallel Community Building Working Group. These two small working groups, involving members of the OpenCitations International Advisory Board, will continue to meet throughout 2023 to develop strategic plans concerning OpenCitations open governance structure and outreach.

More people, more tools

In 2022, OpenCitations welcomed two new full-time developers to its team: Arianna Moretti as software developer, and Ivan Heibi as the person responsible for OpenCitations’ technical infrastructure. As of January 2023, the OpenCitations team consists of our two directors Silvio Peroni and David Shotton, our administrator and research manager Claudio Fabbri, myself as our communications director and community development manager, and our four developers: Giuseppe Grieco (software and systems developer), Arcangelo Massari (developer of OpenCitations Meta), Arianna Moretti (software and indexes developer), and Ivan Heibi (technical infrastructure manager). This team meets weekly at the Research Centre for Open Scholarly Metadata within the University of Bologna.

This enlarged technical workforce has been crucial in the development of the two new OpenCitations indexes, DOCI (citations from DataCite) and POCI (citations from PubMed), and our new database of bibliographic metadata, OpenCitations Meta, all of which were released in December 2022:

  • DOCI, the OpenCitations Index of DataCite open DOI-to-DOI citations, is an RDF dataset containing details of all the citations that are specified in the most recent dump of DataCite metadata. The citations available in DOCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. Its first release (December 2022) contains information on 169,822,752 citations from 358,277 bibliographic resources.
  • POCI, the OpenCitations Index of PubMed open PMID-to-PMID citations, is an RDF dataset containing details of all the citations from publications bearing PubMed Identifiers (PMIDs) to other PMID-identified publications, harvested from the National Institutes of Health Open Citations Collection (NIH-OCC). As for DOCI, the citations available in POCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. Its first release contains 717,654,703 citations from 26,024,862 bibliographic resources.
  • OpenCitations Meta database stores and delivers bibliographic metadata for all the publications involved as citing or cited publications in the citations recorded in the OpenCitations Indexes. For each publication, the metadata exposed by OpenCitations Meta includes the publication’s title, type, venue (e.g. journal name), volume number, issue number, page numbers, publication date, and identifiers such as Digital Object Identifiers (DOIs) and PubMed Identifiers (PMIDs). In addition, OpenCitations Meta includes details of the main actors involved in the document’s publication, i.e., the names of the authors, editors, and publishers, each with its own additional metadata and identifier (e.g. ORCID). Its first release (December 2022) contains information on 87,321,593 bibliographic entities, 277,750,235 authors and 2,359,301 editors (counted by their roles, without disambiguating individuals), 710,226 publication venues, and 17,268 publishers.

As can be seen on OpenCitations roadmap, we have many tasks already scheduled for 2023, and for this reason OpenCitations team needs to expand further: we now seek applicants for a new Research Fellow position to work on the further development of the OpenCitations technical infrastructure from April 2023.

Our role in the community

Throughout 2022, the OpenCitations team has participated both online and in person in a number of international events, presenting details of OpenCitations to potential stakeholders and to Open Science experts, and connecting with similar organizations.

In October, OpenCitations organized our third workshop, the Workshop on Open Citations and Open Scholarly Metadata 2023, as a three-hour online event, to gather together and connect researchers, publishers and policy makers interested in the creation, reuse and improvement of open citation data and open scholarly metadata. The invited speakers provided a broad overview of the current state and future development of open bibliographic metadata and the scholarly record, and the role of OpenCitations in research assessment.

We have participated in two major European research and development project: In the context of the EC-funded project RISIS2 we released the new database OpenCitations Meta and the index DOCI, while within the OpenAIRE-Nexus project network we continue to provide open bibliographic citations as part of the open data components of OpenAIRE and the European Open Science Cloud. During 2022, the OpenAIRE team supported our activities in many areas, with a particular attention to communication and dissemination, as we explained in our blog post OpenCitations and EC funding: OpenAIRE Nexus and RISIS2.

The end of 2022 brought with it two new exciting collaborations that will be developed during the coming year: the inclusion of OpenCitations in the Jisc Collections, thereby permitting UK institutions to support OpenCitations through JISC; and our involvement in the GraspOS project (starting date January 1, 2023) as an open and trusted federated infrastructure for next-generation research metrics and indicators.

Continuing support

At OpenCitations we are deeply grateful to all our members and donors that have supported us financially so far, thus making it possible to enlarge our team, maintain our technical infrastructure, and work on our new data releases.

Much of our success has been due to our selection as an infrastructure worthy for inclusion in the SCOSS second pledging round in 2019. SCOSS helped us get in touch with a wider community, and enabled OpenCitations to have an increasingly significant role in the research environment. OpenCitations is now part of the SCOSS family, the community of all SCOSS-supported infrastructures, that provides a framework of mutual support for the the Infrastructures during their growth and a safety net should troubles occur along the path.

However, OpenCitations still has a long way to go along the path from being a sustainable open infrastructure to becoming a sustained open infrastructure providing reliable and invaluable services to our global stakeholder community. There are still many long-planned activities that we hope to initiate in the near future, given sufficient resources. This is why OpenCitations requires ongoing support from the scholarly community as our three-year period of SCOSS sponsorship draws to a close. All the information you may require to start helping us financially is available on the OpenCitations website: https://opencitations.net/membership.

2023 has just started. With your support, you can help us in making it an impactful year for OpenCitations and the Open Science community as a whole.

Cite this article as: Chiara Di Giambattista, "OpenCitations’ 2022: one year, numerous outcomes," in OpenCitations blog, 28/01/2023, https://opencitations.hypotheses.org/3320.

The OpenCitations Roadmap is now publicly available on Trello

Want to keep yourself updated about the ongoing activities of OpenCitations? We have now publicly released the OpenCitations Roadmap, available on Trello.com:

https://trello.com/b/RprHYoKL/opencitations

The OpenCitations Roadmap consists of a board fulfilled with colour-labelled cards which present the goals so far reached, the present projects and activities, and the future plans. By clicking on the cards, it is possible to visualize a description for each activity, the progress state, and who in the OpenCitations team is working on it.

The OpenCitations Roadmap covers all kinds of activities divided according to the scope, identified by the coloured labels, in particular:

  • light blue for the technical development, such as the development of the software for the creation of the new database OpenCitations Meta and of DOCI, the OpenCitations Index of DataCite open DOI-to-DOI citations, and the re-engineering of the infrastructure and the website;
  • green for the data model implementation;
  • yellow for the data development, such as the bi-monthly COCI releases;
  • purple for the events and outreach activities.

The cards also highlight the activities related to the two EC-funded projects OpenCitations is involved in, OpenAIRE Nexus (blue label) and RISIS2 (orange label). We thank the OpenAIRE team for the help and suggestions during the Roadmap review process.

The OpenCitations Roadmap is an open work in progress that will reflect the developments and growth of OpenCitations. At OpenCitations, we don’t want this Roadmap to be just an online ‘showcase’, but a room in which to share ideas and opinions. We invite you – the members of our community, our stakeholders, the other Open Science actors, researchers, and librarians, and anyone who is interested in OpenCitations activities – to add a comment or a question in the ‘Leave feedback‘ card. This will help us to better understand our strong and weak points, and to stay in touch with the needs and thoughts of the community.

In this way, supplementing the conventional communications channels of email and the social platforms (our blog, Twitter, LinkedIn), the OpenCitations Roadmap will become a new virtual place for dialogue, where you can directly contribute to improve OpenCitations.

Cite this article as: Chiara Di Giambattista, "The OpenCitations Roadmap is now publicly available on Trello," in OpenCitations blog, 29/03/2022, https://opencitations.hypotheses.org/1519.

OpenCitations and EC funding: OpenAIRE Nexus and RISIS2

The incentives for new OpenCitations innovative solutions

Two years ago, in their canonical 2020 QSS paper on OpenCitations, Silvio Peroni and David Shotton anticipated the creation of the new database, OpenCitations Meta, able to “offer a faster and richer service” by storing bibliographic metadata “in house”. Meta would “avoid duplication of data by efficiently permitting us to keep […] a single copy of the metadata for each of the bibliographic entities involved as citing or cited entities in the different OpenCitations’ citation indexes”, would remove the requirement for potentially slow API calls to external metadata sources such as Crossref and ORCID, and would enable us to index citations involving entities lacking DOIs.

Important synergies to achieve goals

Today, thanks to the recent involvement of OpenCitations in two EC-funded projects, the OpenAIRE-Nexus Project (Horizon 2020 EU funded project, GA: 101017452) and the RISIS2 Project (Horizon 2020 EU funded project, GA: 824091), the development of OpenCitations Meta has commenced, with a planned release date later in 2022.

The OpenAIRE-Nexus project started in January 2021 to embrace and expand the operation of a portfolio of thirteen services, provided by OpenAIRE infrastructure, public institutions, organisations and universities, classified into three portfolios entitled PUBLISH, MONITOR, and DISCOVER. The OpenAIRE-Nexus portfolios focus on the demands of the three main categories of the research lifecycle.  Therefore, OpenAIRE-Nexus makes sure such services are integrated to provide a uniform Open Science Scholarly Communication package for the European Open Science Cloud (EOSC). Within the OpenAIRE Nexus project there is scope for producing not only support materials (factsheet, guides, video tutorials, demos) but also training sessions where the services in the three portfolios will be showcased, anticipating the EOSC onboarding process. The role of OpenCitations in the project is to provide open bibliographic citations, and interconnect and integrate (and vice versa) functionalities with the  OpenAIRE Research Graph and more OpenAIRE-Nexus services such as EpiSciences, OpenAIRE MONITOR) the core component of OpenAIRE infrastructure and services and of the EOSC Resource Catalogue. 

Additionally, we are happy to announce our recent involvement in the RISIS2 Project. The Research Infrastructure for Science and Innovation Policy Studies (RISIS) is a project funded by the European Union under a Horizon2020 Research and Innovation Programme. RISIS2 involves 18 partners working together to create and maintain a research infrastructure for the field of Science, Technology, and Industry (STI) Studies, and to build an advanced research community in this field. OpenCitations’ contributions to RISIS2 will include not only the creation of OpenCitations Meta but also the development of a new citation index of open references, the OpenCitations Index of DataCite Open Citations (DOCI), which will be based on the open reference holdings of DataCite and, together with COCI, will be cross-searchable through our unified OpenCitations API.

Lessons learnt so far

A year into the OpenAIRE-Nexus project, we have found that one of the most significant benefits for OpenCitations is our involvement with this wide cooperative network of European research infrastructures, services, and communities, within which we can exchange experiences, ideas, and knowledge, and discuss any challenges and outcomes with our colleagues. More importantly, OpenCitations becomes positioned within the Open Science ecosystem, as a valuable innovative infrastructure with strong proof of integration and interoperable operations. Being part of the OpenAIRE-Nexus team has opened up more future challenges and expectations, and raised the bar for the inclusion of more functionalities of value. Thanks to the dedication of its efficient communication team, OpenAIRE is also helping us by communicating OpenCitations services to additional users and stakeholders, by inclusion within the comprehensive OpenAIRE services catalogue, by releasing an OpenCitations factsheet and by permitting us to present the latest information on OpenCitations through established events (i.e. Open Science FAIR 2022). FAIR and openness of information is our motto, and we strongly promote this through all our activities.

Expanding our team

As announced in our previous blog post “Five reasons why 2021 has been a great year for OpenCitations”, the support we receive from the EU as part of OpenAIRE-Nexus has enabled our recent appointment of Arcangelo Massari, a software developer who is now playing a crucial role in the creation and development of OpenCitations Meta.

As the year 2022 progresses, we look forward to bringing you further information about other new goals for OpenCitations, made possible by the support we receive from our numerous partnerships.

Cite this article as: Chiara Di Giambattista, "OpenCitations and EC funding: OpenAIRE Nexus and RISIS2," in OpenCitations blog, 08/03/2022, https://opencitations.hypotheses.org/1510.
Search OpenEdition Search

You will be redirected to OpenEdition Search