In OpenCitations, we like to define our infrastructure organization as “community-based” and “community-driven”, and we really mean it. The support coming from the number of academic libraries and consortia coming after OpenCitations’ involvement in the 2nd SCOSS funding cycle has made it possible, starting from 2020, to make OpenCitations develop from a small university project based on time-limited grant incomes to being an open infrastructure globally recognized for the provision of open citation data and bibliographical metadata. We want to thank all our members and donors, for trusting our mission and sustaining OpenCitations activities with their continuous and generous support, despite the pandemic and post-pandemic times.
While retracing our work in the last three years, we are astonished by the achievements our team has accomplished, and by how in such a limited time frame OpenCitations has approached David Shotton’s initial vision (who is one of the co-directors of OpenCitations), when he first shaped the project back in 2010. Here are just some of the technical developments that have marked the last years:
in 2022, we released the two new OpenCitations indexes of open citations, DOCI (citations from DataCite) and POCI (citations from PubMed);
we expanded our collection besides the citation data by releasing OpenCitations Meta, a database storing and delivering bibliographic metadata for all the publications in the OpenCitations Indexes, including the publication’s title, type, venue (e.g. journal name), volume number, issue number, page numbers, publication date, identifiers and details of the main actors involved in the document’s publication (the names of the authors, editors, and publishers);
as of October 2023, OpenCitations Indexes contain information on 1.82 billionunique open citations.
However, the most significant achievements for OpenCiations in the last years have been the creation of a prolific network of collaborations with other Open Science projects, such as OpenAIRE-Nexus, RISIS2 and GraspOS, and the establishment of a structured team, involving young researchers and PhD students, whose work at the University of Bologna has made it possible to work on the technical developments day by day.
Our results are a strong indicator of the growing sensibility on the theme of the open provision of bibliographical metadata and citation data, of which Open Citations is at the forefront as a founding member of the Initiative for Open Citations (I4OC) and the Initiative for Open Abstracts (I4OA). The effort and awareness campaigns led by these initiatives, by DORA and the Open Science community as a whole, have led more and more publishers to a change of heart and to open their reference lists. OpenCitations is an integral part of an ongoing process of transformation of the research environment, and we have collected and interpreted some of the needs of the academic community to plan our future activities and developments. We still need your help and support to make it possible to maintain and improve our infrastructure and to sustain the team working at OpenCitations.
If you believe in
the importance of open bibliographic data for the creation of reproducible metrics for research assessment exercises
the power of the scholarly community to change existing practice by reclaiming ownership of its own data– and you want to become an active part of this change
2023 has arrived, and it’s the time to put aside the old agendas and to fill the freshly printed organizers with new tasks. However, it is also worth looking back over the achievements of the past year.
So, before consigning 2022 to the archives, we want to take a moment to recall and celebrate all that OpenCitations has achieved over the past year, in terms of technical developments, community building, and our own internal organization.
Clarifying our objectives
Since its foundation, OpenCitations has been presented to the scholarly community as a infrastructure organization guided by the values of openness, equity and accessibility. In 2022, we publicly clarified our mission, our aims and our unique points by publishing these three documents:
In addition, we continued to monitor OpenCitations’ compliance with the Principles of Open Scholarly Infrastructure. Because one area in which we were still falling short involved stakeholder involvement in OpenCitations’ governance, in 2022 OpenCitations started reflecting on its organizational structure by creating a Governance Evolution Working Group. In addition, we explored the further development of our stakeholder community by establishing a parallel Community Building Working Group. These two small working groups, involving members of the OpenCitations International Advisory Board, will continue to meet throughout 2023 to develop strategic plans concerning OpenCitations open governance structure and outreach.
More people, more tools
In 2022, OpenCitations welcomed two new full-time developers to its team: Arianna Moretti as software developer, and Ivan Heibi as the person responsible for OpenCitations’ technical infrastructure. As of January 2023, the OpenCitations team consists of our two directors Silvio Peroni and David Shotton, our administrator and research manager Claudio Fabbri, myself as our communications director and community development manager, and our four developers: Giuseppe Grieco (software and systems developer), Arcangelo Massari (developer of OpenCitations Meta), Arianna Moretti (software and indexes developer), and Ivan Heibi (technical infrastructure manager). This team meets weekly at the Research Centre for Open Scholarly Metadata within the University of Bologna.
This enlarged technical workforce has been crucial in the development of the two new OpenCitations indexes, DOCI (citations from DataCite) and POCI (citations from PubMed), and our new database of bibliographic metadata, OpenCitations Meta, all of which were released in December 2022:
DOCI, the OpenCitations Index of DataCite open DOI-to-DOI citations, is an RDF dataset containing details of all the citations that are specified in the most recent dump of DataCite metadata. The citations available in DOCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. Its first release (December 2022) contains information on 169,822,752 citations from 358,277 bibliographic resources.
POCI, the OpenCitations Index of PubMed open PMID-to-PMID citations, is an RDF dataset containing details of all the citations from publications bearing PubMed Identifiers (PMIDs) to other PMID-identified publications, harvested from the National Institutes of Health Open Citations Collection (NIH-OCC). As for DOCI, the citations available in POCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. Its first release contains 717,654,703 citations from 26,024,862 bibliographic resources.
OpenCitations Meta database stores and delivers bibliographic metadata for all the publications involved as citing or cited publications in the citations recorded in the OpenCitations Indexes. For each publication, the metadata exposed by OpenCitations Meta includes the publication’s title, type, venue (e.g. journal name), volume number, issue number, page numbers, publication date, and identifiers such as Digital Object Identifiers (DOIs) and PubMed Identifiers (PMIDs). In addition, OpenCitations Meta includes details of the main actors involved in the document’s publication, i.e., the names of the authors, editors, and publishers, each with its own additional metadata and identifier (e.g. ORCID). Its first release (December 2022) contains information on 87,321,593 bibliographic entities, 277,750,235 authors and 2,359,301 editors (counted by their roles, without disambiguating individuals), 710,226 publication venues, and 17,268 publishers.
As can be seen on OpenCitations roadmap, we have many tasks already scheduled for 2023, and for this reason OpenCitations team needs to expand further: we now seek applicants for a new Research Fellow position to work on the further development of the OpenCitations technical infrastructure from April 2023.
Our role in the community
Throughout 2022, the OpenCitations team has participated both online and in person in a number of international events, presenting details of OpenCitations to potential stakeholders and to Open Science experts, and connecting with similar organizations.
In October, OpenCitations organized our third workshop, the Workshop on Open Citations and Open Scholarly Metadata 2023, as a three-hour online event, to gather together and connect researchers, publishers and policy makers interested in the creation, reuse and improvement of open citation data and open scholarly metadata. The invited speakers provided a broad overview of the current state and future development of open bibliographic metadata and the scholarly record, and the role of OpenCitations in research assessment.
We have participated in two major European research and development project: In the context of the EC-funded project RISIS2we released the new database OpenCitations Meta and the index DOCI, while within the OpenAIRE-Nexus project network we continue to provide open bibliographic citations as part of the open data components of OpenAIRE and the European Open Science Cloud. During 2022, the OpenAIRE team supported our activities in many areas, with a particular attention to communication and dissemination, as we explained in our blog post OpenCitations and EC funding: OpenAIRE Nexus and RISIS2.
The end of 2022 brought with it two new exciting collaborations that will be developed during the coming year: the inclusion of OpenCitations in the Jisc Collections, thereby permitting UK institutions to support OpenCitations through JISC; and our involvement in the GraspOS project (starting date January 1, 2023) as an open and trusted federated infrastructure for next-generation research metrics and indicators.
At OpenCitations we are deeply grateful to all our members and donors that have supported us financially so far, thus making it possible to enlarge our team, maintain our technical infrastructure, and work on our new data releases.
Much of our success has been due to our selection as an infrastructure worthy for inclusion in the SCOSS second pledging round in 2019. SCOSS helped us get in touch with a wider community, and enabled OpenCitations to have an increasingly significant role in the research environment. OpenCitations is now part of the SCOSS family, the community of all SCOSS-supported infrastructures, that provides a framework of mutual support for the the Infrastructures during their growth and a safety net should troubles occur along the path.
However, OpenCitations still has a long way to go along the path from being a sustainable open infrastructure to becoming a sustained open infrastructure providing reliable and invaluable services to our global stakeholder community. There are still many long-planned activities that we hope to initiate in the near future, given sufficient resources. This is why OpenCitations requires ongoing support from the scholarly community as our three-year period of SCOSS sponsorship draws to a close. All the information you may require to start helping us financially is available on the OpenCitations website: https://opencitations.net/membership.
2023 has just started. With your support, you can help us in making it an impactful year for OpenCitations and the Open Science community as a whole.
We’re happy to announcePOCI, the OpenCitations Index of PubMed open PMID-to-PMID citations, an RDF dataset containing details of all the citations from publications bearing PubMed Identifiers (PMIDs) to other PMID-identified publications, harvested from the National Institutes of Health Open Citations Collection (NIH-OCC). The citations available in POCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model.
For citations in which the citing and cited works are identified by PMIDs, which includes all the POCI citations, the OCI is created in the following manner, as explained more fully here. Each converted numeral part of OCI is prefixed by a 0160, which indicates that NIH is the supplier of the original metadata of the citation (as indicated at http://opencitations.net/oci).
are available as dumps on Figshare in CSV, N-Triples and Scholix.
What is an Open Citation Index?
A citation index is a bibliographic index recording citations between publications, allowing the user to establish which later documents cite earlier documents. The current indexes available in OpenCitations are:
We’re excited to introduce DOCI, the OpenCitations Index of Datacite open DOI-to-DOI citations, a new tool containing citations derived from publications bearing DataCite DOIs to other DOI-identified publications, harvested from DataCite. The citations available in DOCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model.
For citations in which the citing and cited works are identified by DOIs, which includes all the DOCI citations, the OCI is created in the following manner, as explained more fully here. Each case-insensitive DOI is first normalized to lower case letters. Then, after omitting the initial doi:10. prefix, the alphanumeric string of the DOI is converted reversibly to a pure numerical string using the simple two-numeral lookup table for numerals, lower case letters and other characters presented at https://github.com/opencitations/oci/blob/master/lookup.csv. Finally, each converted numeral is prefixes by a 080, which indicates that DataCite is the supplier of the original metadata of the citation (as indicated at http://opencitations.net/oci).
A citation index is a bibliographic index recording citations between publications, allowing the user to establish which later documents cite earlier documents. The current indexes available in OpenCitations are:
No-one is quite sure of the total number of scholarly publications within the global corpus. Indeed that number will be strongly influenced by the degree to which, in addition to books and journal articles, one includes within the definition of scholarly publications ‘grey literature’ such as reports published by official bodies, patents, etc. Consequentially, the total number of scholarly references within those publications is also unknown, and this number too will vary according to the inclusion criteria chosen. Furthermore, Crossref Event Data and similar datasets recording social media mentions of journal articles in blog posts and tweets extends the concept of a reference beyond that used in ‘conventional’ citation indexes such as COCI.
We celebrate the fact that well over one billion bibliographic citations are now openly available under CC0 waivers in NIH OCC (the National Institutes of Health Open Citation Collection) [1,2] and COCI (the OpenCitations Index of Crossref DOI-to-DOI Citations) . Despite present gaps in their coverage, they include references to all the most important publications within the global corpus, because these will all have been cited multiple times.
Open references available from Crossref and other aggregators and indexes
Crossref, with over 1.6 billion open references, is the largest single source of such bibliographic metadata. Significant numbers of references are also available in a variety of other databases, repositories and indexes.
NIH OCC (the National Institutes of Health Open Citation Collection) is a merger of several citation databases, drawing on PubMed for crucial article metadata, and augmenting this with information from full-text articles that have been made freely available on the internet . The CiteSeerX database, the arXiv preprint repository, and the Dryad data repository are examples of different types of infrastructure that also publish open bibliographic references, while there is further availability of article references from open aggregators such as DataCite and Wikidata. These may either use their own DOIs, DOIs from the Crossref DOI registration agency, or no DOIs at all. Either way, those references will not appear in Crossref.
What is lacking is semantic coherence and interoperability between these sources, permitting federated queries across them. This makes difficult the task of obtaining a comprehensive overview of the availability of open bibliographic references.
However, there are even more citations that are not yet freely and easily available anywhere in bulk, relating to the reference lists within publications of a number of distinct types. This blog post explores academia’s missing references – those that have not yet been documented within open freely accessible citation indexes – and what might be done to bring these into the public domain.
1 References that are closed at Crossref
Eight years ago, I wrote
“In this open-access age, it is a scandal that reference lists from journal articles — core elements of scholarly communication that permit the attribution of credit and integrate our independent research endeavours — are not readily and freely available for use by all scholars.” 
I stand by that statement, and, through OpenCitations , I have been working with colleagues to rectify the situation, as I described in my previous post. The Initiative for Open Citations (I4OC) can rightly be applauded for its part in encouraging almost all the major academic publishers who deposit references at Crossref to make them open.
The only major scholarly publisher not to be listed as an I4OC Participating Publisher is the Institute of Electrical and Electronics Engineers (IEEE), that, having deposited at Crossref reference lists for 58% of its preprints and publications, persists in keeping these deposited references closed and unavailable for indexing and re-use. It is to be hoped that IEEE will now realize that what it looses by not having references openly available outweighs any benefits it might have received from keeping them closed, and will join its fellow publishers in ensuring that its Crossref-deposited references are made open, both for its current issues and for its back-number journal articles. I thus call again upon IEEE to change its present position, as both Elsevier and the American Chemical Society had the courage to do recently, and to instruct Crossref to open all IEEE references. A single email to Crossref Support, with the instruction “Open all references”, is all it would take!
References in Crossref are either open, ‘limited’ or closed. Limited references are available to those subscribing to Crossref Metadata Plus, which includes OpenCitations, but not to the general public. Closed references are not freely available to anyone. The following table shows the number of Crossref works in each category, and the number of references within those categories.
Average references per work
70,477,843 (55.7% of total)
56,149,775 (44.3% of total)
49,274,155 (38.9% of total)
2,933,323 (2.3% of total)
3,942,297 (3.1% of total)
Table 1. Numbers of works and their references recorded in Crossref on 31 July 2021.
2 References not deposited at Crossref for publications with Crossref DOIs
The number of works with Crossref DOIs that lack submitted references is surprisingly large. As of 31 July 2021, 70,477,843 publications (55.7% of all works recorded at Crossref) lacked deposited references (Table 1).
Crossref classifies all types of journal content, including editorials, book reviews and letters, as “journal articles”, thus some of these works without deposited references genuinely lack them. However, the majority are conventional journal articles and books with reference lists that the publishers have simply not deposited at Crossref along with the other metadata for these works.
The average number of references per Crossref work with deposited references is 30.9 (Table 1). If, to make allowance for those works that genuinely lack references, we assume a conservative average of 25 references per work for the 70,477,843 works lacking deposited references, this means that there are over 1.75 billion references within these works that have not been deposited at Crossref, and thus are not conveniently available for indexing and reuse.
These missing references relate both large publishers that have submitted references for only some of their publications, and small publishers that perhaps lack, or think they lack, the resources to deposit any reference lists in addition to the other metadata they are already sending to Crossref for each of their DOIs. However, there are several easy methods for depositing reference lists, as detailed by Crossref here. So I encourage all publishers who are supportive of Open Science to update their procedures and commence or complete the deposition of their publication reference lists at Crossref, starting with their current issues. Crossref Support will provide assistance if required. Note that a publisher does not have to subscribe to the Crossref Cited-by service to deposit its references!
3 Citations missing in COCI: open references in Crossref to publications lacking DOIs
COCI is the OpenCitations Index of Crossref DOI-to-DOI Citations, and, as the name suggests, it indexes Crossref open references from works with Crossref DOIs to other works that have DOIs . It therefore does not index open references in Crossref to works that, for whatever reason, lack DOIs.
The most recent (September 2021) release of COCI, based on the August Crossref dump, contains 1,186,958,898 citations between 69,074,291 unique work, comprising 51,103,720 citing bibliographic resources bearing Crossref DOIs and 56,105,783 cited bibliographic resources. Of the cited bibliographic resources, 38,135,212 bear a DOI issued by Crossref and have open or limited references (thus also being COCI citing resources), while 17,970,571 either have a Crossref DOI but lack open or limited references or have a DOI issued by another DOI registration agency such as DataCite (thus not being COCI citing resources).
Note that in Crossref, the ratio of works with open or limited references to works without open or limited references is 0.7:1 (Table 1). However, in COCI, the ratio of cited works with Crossref DOIs containing open or limited references to all other cited works is 2.1:1. Thus works with Crossref DOIs containing open or limited references are three times more likely to be cited than works that either have a Crossref DOI but lacking open or limited references or have a DOI issued by another DOI registration agency. This is most likely because the most important journals from almost all the larger publishers now have open references. However, it is still a remarkable ratio.
Because references to works lacking DOIs are not included in COCI, the average number of bibliographic references per citing article in COCI is only 23.2, in contrast to the numbers of references to works of all types given in Table 1.
From these data, it can be seen that there are over 480 million open Crossref references to a wide variety of works lacking DOIs that OpenCitations does not index in COCI. This is because of an intentional and fundamental limitation in the structure of the Open Citation Identifier (OCI) , requiring both citing and cited publications to have identifiers of the same type, that lies at the heart of the functionality of our OpenCitations Indexes.
OpenCitations is currently developing a solution that, without compromising that intentional design limitation in OCIs, will nevertheless permit us to index and publish these ‘missing’ references as Linked Open Data. We will report on this development in due course.
Crossref Event Data is a Crossref service / database that records mentions of publications bearing Crossref DOIs in social media including tweets and blog posts, and in other non-traditional citation sources such as news items and Wikipedia articles. From today (Thursday 23rd September 2021), Crossref Event Data will start to include in its holdings open references from publications bearing Crossref DOIs to other publications bearing DOIs. Limited and closed references will not be included. Initially, open references from current publications will be included in Crossref Event Data, with open references from older works with DOIs being added later. In that respect it will come to resemble COCI, except that COCI also included ‘limited’ references, treats citations as first-class data entities with their own identifiers, and makes its citation data available in RDF as Linked Open Data, as well as via a REST API. Subsequently, Crossref Event Data will also record references to publications with other forms of identifier, as OpenCitations also plans to do.
4 References available on publishers’ web sites
Many publishers, particularly those of Open Access works, already make their publication reference lists openly available on their own web sites. While this is commendable, it is not sufficient, since, if scholarly references are not made available in a centralized aggregator such as Crossref from which they can be conveniently harvested in bulk for analysis and re-use, they are much more difficult to access.
Scraping references from the HTML of individual web sites is difficult, time-consuming and liable to be incomplete. While Microsoft Academic achieved considerable success in scraping references from publishers’ web sites, possible because of the special relationships these publishers have with the Microsoft search engine Bing, this service will unfortunately soon no longer be available, illustrating a problem inherent with scholarly infrastructures provided by commercial companies that do not adopt the Principles of Open Scholarly Infrastructures.
A consequence is that such publications will become increasingly ‘invisible’, as bibliographic and analytical services come to rely more and more on centrally available data.
6 References within PDFs of scholarly works lacking DOIs
There is a large but unknown quantity of reference-containing books, academic reports, patents and journal articles from publishers that, for their own good reasons, choose not to use DOIs. The text of many of these publications is already available in a marked-up machine-readable format such as JATS, used in preparation for publication, from which the reference lists could easily be extracted. Other publications are only available as PDFs, both as published Versions of Record, or as preprints deposited in a variety of preprint repositories such as arXiv or CORE. Mining reference lists out of PDFs required expertise in text mining and AI technologies, and is labour-intensive, since it usually required the tuning of extraction algorithms to handle the particular styles and formatting of individual journals, one at a time. Two stages are involved: first, the recognition and extraction of the text of the individual references from the PDF, and second the parsing of each text string into the component parts of the reference (author names, title, publication year, etc.) A considerable number of the citations within NIH-OCC have been obtained in this manner , commercial companies such as Lexical Intelligence specialize in this area, and publicly available software such as GROBID is available for the purpose. However, the overall task of extracting ‘missing’ academic references from the global PDF corpus is daunting in magnitude and would require a well-funded organization.
The correct way to proceed would be for each publisher to take responsibility for liberating the references of their own publications, whether or not the publications themselves are open access, and whether or not these references are already available in a marked-up machine-readable format or only within PDF documents. Then, if the publisher still chose not to use DOIs and to submit these metadata to Crossref, these references could be submitted directly to OpenCitations for aggregation and publication as Linked Open Data.
From the foregoing discussion it is clear that the academic community has a long way to go before the majority of scholarly citations, the products of their own labours, are openly available for analysis and re-use. We at OpenCitations are working to address these issues and to publish more of these missing citations. However, completion of the task will require a coordinated collaborative international effort.
 Ivan Heibi, Silvio Peroni, David Shotton (2019). Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. Scientometrics121 (2): 1213-1228. https://doi.org/10.1007/s11192-019-03217-6
The fourth release of COCI contains more than 655 million DOI-to-DOI citation links between more than 55 million bibliographic entities. The additional 31 million citations added in the new release come from the reprocessing of previous dumps of Crossref data. In particular, we retrieved all the citations that involve references in citing articles that were in the Crossref ‘Limited’ set when we downloaded it in October 2018. Such citing articles currently appear in the Crossref ‘Closed’ dataset due to more recent restrictive policy decisions taken by their publishers.
Finally, we wish to remind you that all the bibliographic and citation data in COCI: