A few months ago, I was invited to have a talk at the European Computer Science Symposium on an aspect of my research I particularly care about, that of open citations. What I tried to address during the presentation concerned the current status of open citation availability in a particular domain, Informatics, by using two open datasets, i.e. DBLP for gathering bibliographic metadata about relevant publications and OpenCitations’ COCI for identifying citations where such publications are involved. This post briefly introduces the preliminaries and results obtained from the material used to prepare the talk.
Open citations and where to find them
A citation is a conceptual directional link between a citing entity and a cited entity which is defined by means of specific textual devices contained in the text of the citing entity, e.g. a bibliographic reference denoted by an in-text reference pointer (e.g. “” or “(Doe et al, 2021)”). While reasons for citing may vary, citations are used in academia for acknowledging others’ work and enabling building trails of relations defining how science evolves in time.
The data needed to describe a citation should include, at least, a representation of such a conceptual link and the basic bibliographic metadata to identify the citing and cited entities, i.e. those typically used for defining bibliographic references such as authors’ names, year of publication, the title of the work, venue of publication, pages, identifiers, etc. We say that a citation is open when these citation data are in the public domain and can be retrieved freely (via the HTTP protocol) in a structured and machine-readable format (e.g. JSON or RDF) without accessing the source of citing article defining it, which, potentially, could be behind a paywall.
OpenCitations [full disclosure: I am one of its directors] is one of the founders of the Initiative for Open Citations (I4OC) and one of the open scholarly infrastructures providing open citation data through several channels (REST APIs, SPARQL endpoints, Web interfaces, full dumps in different formats). As of 31 December 2021, it makes available more than 1.2 billion open DOI-to-DOI citation links between more than 69.5 million bibliographic resources, which are mainly journal articles but also include books, book chapters, datasets, and other DOI-identified resources. The entities involved in such citations come from different domains, spanning from Medicine articles to Humanities publications, and have recently approached parity with those included in well-known proprietary services such as Web of Science and Scopus.
What about Informatics
Such a huge mass of open citations available enables us to analyse citation coverage in different scholarly disciplines, e.g. to understand which publishers contributed to the availability of open citation data in a discipline and to check what are the citation trails between different disciplines. However, to compute such citation coverage, we need to have some information that allows us to identify when a particular bibliographic resource involved in a citation belongs to the particular discipline we want to analyse. We can use information about the subject categories of publications (e.g. that of Web of Science), if included in citation indexes, to identify the discipline(s) of a given bibliographic resource. Unluckily, OpenCitations does not provide this information and, as such, we need to rely on external repositories for gathering subject categories of publications, e.g. collections of bibliographic metadata of disciplinary publications.
In the context of Informatics, there is at least one well-known resource gathering and exposing bibliographic metadata of a large part of Computer Science publications, i.e. DBLP. As of 30 December 2021, DBLP contains more than 5.9 million publications published in 1,781 journals and in the proceedings of 5,621 conferences, involving more than 2.9 million authors that are manually curated (and disambiguated) by the DBLP team.
DBLP can be used as a proxy to understand if a particular publication belongs to the Computer Science subject category. Through it, it is possible to understand how many citations in OpenCitations involve Computer Science publications by comparing the DOIs of citing and cited entities with those available in DBLP. In particular, using the OpenCitations’ COCI September 2021 dump and DBLP October dump, I found that more than 80 million citations in COCI involved at least one of the 4,637,865 entities in DBLP (considering only journal articles, conference proceedings papers, books and book chapters). As shown in Figure 1, only 39% of these citations are between citing and cited entities both included in DBLP, while the rest of them either come from or go to publications not listed in DBLP – that, potentially, could not be Computer Science publications.
Additional information about the publishers of such DBLP entities, retrieved by querying the Crossref API and the DataCite API with entities’ DOIs, are shown in Table 1. IEEE is the publisher with the biggest number of entities of those considered for this study, and its entities are involved in more than 18.9 million incoming and 21.5 million outgoing citations. The other bigger publishers, in terms of entities and citations, are Springer, Elsevier, ACM and Wiley. It is worth mentioning that the two publishers responsible for publishing mainly Computer Science journals and a relatively low number of conference proceedings (if any), i.e. Elsevier and Wiley, are those providing the highest number of openly-available references per publication (on average, around 29 and 37 cited works for each publication respectively).
COCI incoming citations
COCI outgoing citations
Table 1. The DBLP entities retrieved in the study grouped by their publisher and their incoming and outgoing citations according to COCI.
Of course, this study does not provide full coverage of open citations in Computer Science but just a preliminary insight. First, as anticipated below, DBLP does not have the complete coverage of all CS-related publications since there are some venues that are not listed there (yet). Thus, some relevant open citations could not be extracted from COCI if these involve as citing and cited entity non-DBLP publications that belong to the CS domain. However, it is worth mentioning that no bibliographic and citation database (including commercial and proprietary ones) has a full disciplinary coverage anyway and DBLP is, probably, the most comprehensive collection of Computer Science publications metadata (something that could be assessed in future analysis).
Along the same lines, the index of open citations used, i.e. COCI, does not contain all the citations defined in CS publications, but only DOI-to-DOI citations as retrievable from Crossref data. Although Crossref is the biggest DOI provider and it is used by the majority of the big publishers,citations defined in publications with a non-Crossref DOI (e.g. DataCite) and those not having any DOI assigned (e.g. the papers published in CEUR Workshop Proceedings) are not included in COCI and, consequently, have not been used in the analysis. However, OpenCitations plans to extend its data coverage adding more sources in the next years. Thus, it would be interesting to replicate the same analysis in the future to see if and how much the coverage increase, at least in the context of Computer Science publications.
Still about coverage, currently (i.e. 31 December 2021) the only publisher of those included in Table 1 which is not providing open references through Crossref is IEEE. Indeed, while COCI includes several citations involving IEEE publications as citing entities, there is no availability of such citation after October 2018, when IEEE decided not to allow anymore Crossref Metadata Plus users to access these reference data.
Finally, analysing the preliminary results of this study, it would be interesting to understand which are the main subject categories of non-DBLP publications included in the 61% of citations shown in Figure 1 (e.g. by using the Scimago Journal and Country Rank database to retrieve their subject categories) to understand what are the citation dynamics between Informatics and other disciplines. However, I will leave the answer to this question to future analysis.
A final remark on reproducibility
Since several of the suggestions provided above start from the idea of either replicating or extending this study with additional materials and insights, it is important that all data and software used to perform the analysis are available online to permit its reproducibility. To this end, I have published both the software and the data retrieved online with open licenses to enable anyone to reuse it freely for any purpose.
Guest blog post by Alberto Martín-Martín, Facultad de Comunicación y Documentación, Universidad de Granada, Spain <firstname.lastname@example.org>
In this post, as a contribution to Open Access Week, Alberto Martín-Martín shares his comparative analysis of COCI and other sources of open citation data with those from subscription services, and comments on their relative coverage.
Comprehensive bibliographic metadata is essential for the development of effective understanding and analysis across all phases of the research workflow. Commercial actors have historically filled the role of infrastructure providers of bibliographic and citation data, but their choice of subscription-based business models and/or restrictive user licenses has significantly limited how users and other parties can access, build upon, and redistribute the information available on those platforms. Locking bibliographic and citation metadata behind these barriers is problematic, as it hinders innovation and is an obstacle to reproducibility.
Fortunately, the process of digital transformation that scientific communication is currently undergoing is providing us with the tools to get closer to the ideal of science as a public good. One of the most successful initiatives in this area is Crossref, arguably the single most critical piece of research metadata infrastructure currently in existence. I consider the best thing about it to be its commitment to openness. Not only is Crossref responsible for minting many of the DOIs that are assigned to academic publications, but it also publishes metadata about these publications (for over 120+ million records in their latest public data file) without imposing any access or reuse limitations.
Crossref metadata has already boosted innovation in a variety of academic-oriented tools. New discovery services such as Dimensions, The Lens, and Scilit all take advantage of Crossref metadata to keep their indexes up to date with the latest publications. The open-source reference manager Zotero is able to pull metadata associated with a given DOI from Crossref’s servers, providing an easy way to populate one’s personal reference collection that is more reliable than using Google Scholar. The Unpaywall database uses Crossref metadata (among other data sources) to keep track of which documents are Open Access, and this data is in turn used by Unsub, a service that helps libraries make more informed decisions about their journal subscriptions.
Historically, citation indexing has been a functionality available only from a few subscription-based data sources (most notably Web of Science and Scopus), or from free but largely restricted sources (e.g., Google Scholar). In recent years, however, commercial exclusivity over citation data has been waning. Digital publishing workflows make it easier for publishers to deposit the list of cited references along with the rest of the metadata when they register a new document in Crossref, and many are already doing it. Crossref’s policy is to make these lists of references publicly available by default, although publishers can elect to prevent their public release. From this, it follows that if most publishers deposited their reference lists in Crossref and consented to make them open, a comprehensive open citation index, one that is free of the restrictions present in traditional platforms, could be built.
The Initiative for Open Citations (I4OC) is an advocacy group that has been working since 2017 to achieve this precise goal, and it has already managed to convince a large number publishers (over two thousand) to open the references they deposit in CrossRef. In the first half of 2021, Elsevier, the American Chemical Society, and Wolters Kluwer joined this group, so that today all the major scholarly publishers now support I4OC and have open references at Crossref, with the exception of IEEE (the Institute of Electrical and Electronics Engineers). Thanks to the efforts of I4OC and the collaboration of publishers, 88% of the publications for which publishers have deposited references in CrossRef are now open. This has allowed organizations such as OpenCitations (one of the founding members of I4OC) to create a non-proprietary citation index using these data, namely COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. Other open citation indexes such as the NIH Open Citation Collection (NIH-OCC) and Refcat have also been recently released.
How do such open citation indexes compare to long-established indexes? In 2019, I set out with colleagues to analyze the coverage of citations contained within the most widely used academic bibliographic data sources (Web of Science, Scopus, and Google Scholar) to a selected corpus of 2,515 highly-cited English-language documents published in 2006 from 252 subject categories, and to compare this to the coverage provided by some of the more recent data sources (Microsoft Academic, Dimensions, and COCI). At that time, COCI was the smallest of the six indexes, containing only 28% of all citations. For comparison, Web of Science contained 52%, and Scopus contained 57%.
There are a number of reasons for those differences: first, at that point some of the larger commercial publishers including Elsevier, IEEE, and ACS, which routinely deposit references in Crossref, hadnot yet opened them. Second, many smaller publishers still do not deposit their reference lists in Crossref. Third, COCI only captures citation relationships between documents that have DOIs, thus missing citations to publications that lack them. Finally, while for our study data collection from all sources was carried out during May/June of 2019, COCI at that time had not been updated since November 2018, which increased its disadvantage when compared to other data sources with more frequent updates.
Since Elsevier is the largest academic publisher in the world, its recent opening of references at Crossref resulted in a significant increase in the total number of openly available Crossref references. The most recent version of COCI (dated 3 September 2021, and based on open references to works with DOIs within the Crossref dump dated August 2021) now contains both the processed references from Elsevier, and the references in the most recently published articles by ACS (the complete backfile of ACS references will appear in future versions of COCI).
Given these significant developments, how much has the picture changed? To find this out, I updated our 2019 analysis using the version of COCI released on September 3rd 2021 and the NIH-OCC dataset released in the same month. To carry out a reasonably fair comparison while reusing the data extracted in 2019 from the other sources, I employed the same corpus of target documents, and only used citations in which the citing document was published before the end of June 2019. The intention was to learn how much the coverage of open citation data has grown as a result of the subsequent opening of reference lists in Crossref that were not public in 2019, and similar efforts.
The combination of COCI’s and NIH-OCC’s September 2021 releases contained more than 1.62 million citations to our sample corpus of documents from all areas, a 91% increase over the 0.85 million citations that we were able to recover in 2019 from COCI alone. Considering the citations available in all data sources, 53% of all citations are now available from these two open sources under CC0 waivers, up from the 28% we found in 2019. This coverage now surpasses the 52% found by Web of Science, and is much closer to the 54% found by Dimensions, and the 57% covered by Scopus. The relative overlap between COCI and the other data sources has also significantly increased: in 2019 COCI found 47% of the citations available in Web of Science, whereas now open citation data sources find 87% of the WoS citations. In the case of Scopus, in 2019 COCI found 44% of the citations available in Scopus: the percentage available from open sources has now increased to 81%. The number of citations found by COCI but not present in the other data sources has also widened slightly. These data are presented graphically in Figure 1.
Fig. 1. Percentage of citations found by each database, relative to all citations (first row), and relative to the number of citations found by the other databases (subsequent rows).
Where are these new citations coming from? Well, as we might expect, references from articles published in Elsevier journals comprise the lion’s share of the newly found citations in open data sources (close to half of all new citations), as shown in Figure 2. But there are also some IEEE citations here. This is because until recently reference lists from IEEE publications were available in the ‘limited’ Crossref category to members of Crossref Metadata Plus, a paid-for service that provides a few additional advantages over the free services Crossref provides. As a member of Crossref Metadata Plus, OpenCitations obtained these reference lists while they were available and included them in COCI. Subsequently, IEEE decided to make their references completely closed, explaining why references from more recent IEEE publications are not included in COCI.
Fig 2. The increases between 2019 and 2021 of citations indexed by open sources (COCI + NIH-OCC) from the articles of different publishers
There can be no doubt that open citation data is of benefit to the entire academic community. Thanks to COCI, NIH-OCC, and similarinitiatives, and despite some setbacks, we are already witnessing how open infrastructure can help us develop models and practices that are better aligned with the opportunities that our current digital environment offers and the challenges that our society faces.
Conclusion: The coverage of citation data available under CC0 waivers from open sources is now comparable to that from subscription sources such as Web of Science and Scopus, offering a viable alternative upon which to base open and reproducible metrics of academic performance.
“The competitive benefits of closing access to citation data diminish with each new citation released to the public domain, but the benefits of open data remain. Going forward, citation data is almost completely public domain”.
With these words, from the article “A tipping point for open citationsdata” (July 15, 2021), Ian Hutchins celebrated the threshold crossing of one billion citations on public-domain databases in February 2021.
Now, a new significant milestone has been reached. We are enthusiastic to announce that COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations has just been extended with 334 million additional citations. Its most recent release, the COCI July 2021 release, now contains a total of 1.09 billion DOI-to-DOI citation links derived from open references within Crossref,which includes the references of articles deposited or opened in Crossref between November 2020 and January 2021.
These numbers make us proud, and confirm the essential value of the Initiative for Open Citations (I4OC). Since 2018, the mission of I4OC has been to persuade publishers to provide open citation data by means of the Crossref platform. The I4OC untiring commitment has led the major academic publishers to a progressive change of heart regarding open citations, and the scholarly community to a deeper interest in this openness.
These factors contributed to the creation of COCI in 2018, the first open citation index created by OpenCitations, in which we applied the concept of citations as first-class data entities (Heibi I., Peroni S., Shotton D., 2019). Over the last three years, COCI has been extended in a series of releases, by harvesting citations mostly from Crossref data dumps, starting from an initial coverage of 300 million citations (First release).
A crucial event that preceded (and delayed!) this latest COCI release was Elsevier’s endorsement in the DORA Declaration on Research Assessment in December 2020, thereby making “reference lists for all articles published in Elsevier journals openly available via Crossref so they can be available for reuse. This means other important initiatives like I4OC can draw on this metadata”. As described in our previous post, Elsevier’s welcome commitment led to the opening of many previously closed references from its numerous academic journals submitted to Crossref. Now, after an extended period of data ingestion and processing, all these newly opened Elsevier references are available at OpenCitations within COCI.
Elsevier’s involvement has both an effective and a symbolical value. Even if publishing more than one billion citations is a thrilling achievement, and – as Hutchins wrote – we are now at a tipping point with regard to open citations data, this milestone is not the last stop. Together with the other organizations and projects that participate in the Initiative for Open Citations, we will keep claiming the urgency for the remaining academic publishers to join our cause, and sharing our values with the whole academic community to make all existing citations data freely open and accessible. Recalling what Dario Taraborelli wrote in the conclusion of his article “The citation graph is one of humankind’s most important intellectual achievements“, “the world is waiting for the citation graph to become a public good”.
Cite this article as: Chiara Di Giambattista, "Crossing a significant threshold: more than one billion citations now available in COCI!," in OpenCitations blog, 04/08/2021, https://opencitations.hypotheses.org/1253.
The citations added in this the fifth release of COCI came from the most recent Crossref dump downloaded on 22 April 2020, which includes the references of the articles deposited in Crossref between 4 October 2019 and 4 April 2020. Such updates to COCI will now occur regularly at bimonthly intervals.
As a consequence, COCI now contains 702,772,530 citations, and also includes publications about the COVID-19 pandemic. We will use this new release of COCI to update the Coronavirus Open Citations Dataset, the second release of which will include details about these additional references and publications.
The memorable date 20/02/2020 saw the publication by MIT Press of the first issue of Volume One of a new journal, Quantitative Science Studies (QSS), the official open access journal of the International Society for Scientometrics and Informetrics (ISSI). QSS’s Editor in Chief is Ludo Waltman (CWTS, University of Leiden, Netherlands), Vincent Larivière (Université de Montréal, Montreal, Quebec, Canada) and Staša Milojević (Indiana University Bloomington, Bloomington, Indiana, USA) are its Associate Editors, and it has a large and distinguished editorial board.
What makes the launch of this new journal remarkable is the story of how it came into being. In 2019, the entire editorial team of the Journal of Informetrics (JOI), a leading journal in this field published by Elsevier, resigned en masse and decided to start an alternative journal, QSS, both because of Elsevier’s position on open citations, and because, in their opinion, the financial model used by Elsevier violates the scientific ethos.
Reproducibility in the field of scientometrics requires scientific metadata that are both of high-quality and open, particularly those relating to bibliographic citations. The JOI editorial board was deeply concerned by the refusal of Elsevier to join almost all other large scholarly publishers in supporting the Initiative for Open Citations (I4OC). As we have previously reported on this blog, Elsevier is the largest contributor of bibliographic references to Crossref, but insists that these data should be kept closed.
Elsevier’s position, driven by commercial interests (since it sells access to citation data through Scopus), flies in the face of the scientific community’s clear move towards open science, with hundreds of scientometricians having signed an ISSI open letter urging scholarly publishers to support I4OC.
Science is a self-governing system, and the editorial team held the view that the ultimate responsibility for a scholarly journal should fall with the scientific community, who serve as the gatekeepers, producers, and consumers of scientific content.
The editorial team also believed Elsevier’s subscription fees to be excessive, and its article processing charges (APCs) for open access publishing to be unfairly high, thus limiting both those who can afford to read Elsevier journals and those who can afford to publish in them, so that publishing with Elsevier inevitably places major limits on scholarship, harming both science and society. It was for all these reasons that they forsook JOI and started QSS.
We at OpenCitations congratulate the editorial team for their courage in deciding to make this journal flip, and wish them, together with the ISSI and MIT Press, every success for this important new journal. We also commend the Technische Informationsbibliothek (TIB) – Leibniz Information Centre for Science and Technology and the Communication, Information, Media Centre (KIM) of the University of Konstanz, who, in collaboration with the Fair Open Access Alliance (FOAA), have generously agreed to cover APCs for the first three years of the QSS journal.
Copyright notice: This work is licensed under a Creative Commons Attribution 4.0 International License. You are free to share (i.e. copy and redistribute the material in any medium or format) and adapt (e.g. remix, transform, and build upon the material) for any purpose, even commercially, under the following terms: attribution, i.e. you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. The licensor cannot revoke these freedoms as long as you follow the license terms.
In this paper, we present COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations (http://opencitations.net/index/coci). COCI is the first open citation index created by OpenCitations, in which we have applied the concept of citations as first-class data entities, and it contains more than 445 million DOI-to-DOI citation links derived from the data available in Crossref. These citations are described in RDF by means of the new extended version of the OpenCitations Data Model (OCDM). We introduce the workflow we have developed for creating these data, and also show the additional services that facilitate the access to and querying of these data by means of different access points: a SPARQL endpoint, a REST API, bulk downloads, Web interfaces, and direct access to the citations via HTTP content negotiation. Finally, we present statistics regarding the use of COCI citation data, and we introduce several projects that have already started to use COCI data for different purposes.
The availability of open scholarly citations  is a public good, of significant value to the academic community and the general public. In fact, citations not only serve as an acknowledgment medium , but also can be characterised topologically (by defining the connected graph between citing and cited entities and its evolution over time ), sociologically (such as for identifying odd conduct within or elitist access paths to scientific research ), quantitatively by creating citation-based metrics for evaluating the impact of an idea or a person , and financially by defining the scholarly value of a researcher within his/her own academic community . The Initiative for Open Citations (I4OC, https://i4oc.org) has dedicated the past two years to persuading publishers to provide open citation data by means of the Crossref platform (https://crossref.org), obtaining the release of the reference lists of more than 43 million articles (as of February 2019), and it is this change of behaviour by the majority of academic publishers that has permitted COCI to be created.
OpenCitations (http://opencitations.net) is a scholarly infrastructure organization dedicated to open scholarship and the publication of open bibliographic and citation data by the use of Semantic Web (Linked Data) technologies, and is a founding member of I4OC. It has created and maintains the SPAR (Semantic Publishing and Referencing) Ontologies (http://www.sparontologies.net)  for encoding scholarly bibliographic and citation data in RDF, and has previously developed the OpenCitations Corpus (OCC) of open downloadable bibliographic and citation data recorded in RDF .
In this paper, we introduce a new dataset made available a few months ago by OpenCitations, namely COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations (https://w3id.org/oc/index/coci). This dataset, launched in July 2018, is the first of the indexes proposed by OpenCitations (https://w3id.org/oc/index), in which citations are exposed as first-class data entities with accompanying properties (i.e. individuals of the class cito:Citation as defined in CiTO ) instead of being defined simply as relations among two bibliographic resources (via the property cito:cites). Currently COCI, contains more than 445 million DOI-to-DOI citation links made available under a Creative Commons CC0 public domain waiver, that can be accessed and queried through a SPARQL endpoint, an HTTP REST API, by means of searching/browsing Web interfaces, by bulk download in different formats (CSV and N-Triples), or by direct access via HTTP content negotiation.
The rest of the paper is organized as follows. In Section 2 we introduce some of the main RDF datasets containing scholarly bibliographic metadata and citations. In Section 3, we provide some details on the rationale and the technologies used to describe citations as first-class data entities, which are the main foundation of the development of COCI. In Section 4, we present COCI, including the workflow process developed for ingesting and exposing the open citation data available and other tools used for accessing these data. In Section 5, we show the scale of the community uptake of COCI since its launch by means of quantitative statistics on the use of its related services and by listing existing projects that are using it for specific purposes. Finally, in Section 6, we conclude the paper sketching out related and upcoming projects.
We have noticed a recent growing interest within the Semantic Web community for creating and making available RDF datasets concerning the metadata of scholarly resources, particularly bibliographic resources. In this section, we briefly introduce some of the most relevant ones.
ScholarlyData (http://www.scholarlydata.org)  is a project that refactors the Semantic Web Dog Food so as to keep the dataset growing in good health. It uses the Conference Ontology, an improvement version of the Semantic Web Conference Ontology, to describe metadata of documents (5,415, as of March 31, 2019), people (more than 1,100), and data about academic events (592) where such documents have been presented.
Another important source of bibliographic data in RDF is OpenAIRE (https://www.openaire.eu) . Created by funding from the European Union, its RDF dataset makes available data for around 34 million research products created in the context of around 2.5 million research projects.
While important, these aforementioned datasets do not provide citation links between publications as part of their RDF data. In contrast, the following datasets do include citation data as part of the information they make available.
In 2017, Springer Nature announced SciGraph (https://scigraph.springernature.com) , a Linked Open Data platform aggregating data sources from Springer Nature and other key partners managing scholarly domain data. It contains data about journal articles (around 8 millions, as of March 31, 2019) and book chapters (around 4.5 millions), including their related citations, and information on around 7 million people involved in the publishing process.
The OpenCitations Corpus (OCC, https://w3id.org/oc/corpus)  is a collection of open bibliographic and citation data created by ourselves, harvested from the open access literature available in PubMed Central. As of March 31, 2019, it contains information about almost 14 million citation links to more than 7.5 million cited bibliographic resources.
WikiCite (https://meta.wikimedia.org/wiki/WikiCite) is a proposal, with a related series of workshops, which aims at building a bibliographic database in Wikidata  to serve all Wikimedia projects. Currently Wikidata hosts (as of March 29, 2019) more than 170 million citations.
Biotea (https://biotea.github.io)  is an RDF datasets containing information about some of the articles available in the Open Access subset of PubMed Central, that have been enhanced with specialized annotation pipelines. The last released dataset includes information extracted from 2,811 articles, including data on their citations.
Finally, Semantic Lancet  proposes to build a dataset of scholarly publication metadata and citations (including the specification of the citation functions) starting from articles published by Elsevier. To date it includes bibliographic metadata, abstract and citations of 291 articles published in the Journal of Web Semantics.
Indexing citations as first-class data entities
Citations are normally defined simply as links between published entities (from a citing entity to a cited entity). However, an alternative richer view is to regard each citation as a data entity in its own right, as illustrated in Figure 1. This alternative approach permits us to endow a citation with descriptive properties, such as those ones introduced in Table 11.
The advantages of treating citations as first-class data entities are:
all the information regarding each citation is available in one place, since such information is defined as attributes of the citation itself;
citations become easier to describe, distinguish, count and process, and it becomes possible to distinguish separate citations within the citing entity to the cited entity, enabling one to count how many times, from which sections of the citing entity, and (in principle) for what purposes a particular cited entity is cited within the source paper;
if available in aggregate, citations described in this manner are easier to analyse using bibliometric methods, for example to determine how citation time spans vary by discipline.
We have appropriately extended the OpenCitations Data Model (OCDM, http://opencitations.net/model)  so as to define each citation as a first-class entity in machine-readable manner. In particular, we have used the class cito:Citation defined in the revised and expanded Citation Typing Ontology (CiTO, http://purl.org/spar/cito) , which is part of the SPAR Ontologies . This class allows us to define a permanent conceptual directional link from the citing bibliographic entity to a cited bibliographic entity, that can be accompanied by additional ontological terms for defining specific attributes, as introduced in Table 1.
The bibliographic entity which acts as source for the citation.
Object property cito:hasCitingEntity.
The bibliographic entity which acts as target for the citation.
Object property cito:hasCitedEntity.
citation creation date
The date on which the citation was created. This has the same numerical value as the publication date of the citing bibliographic resource, but is a property of the citation itself. When combined with the citation time span, it permits that citation to be located in history.
Data property cito:hasCitationCreationDate, one of xsd:date, xsd:gYearMonth, or xsd:gYear as datatype value.
The temporal characteristic of a citation, namely the interval between the publication date of the cited entity and the publication date of the citing entity.
Data property cito:hasCitationTimespan, xsd:duration as datatype value.
A classification of the citation according to particular dimensions, e.g. whether or not it is a self-citation.
Property rdf:type associated with one or more subclasses of cito:Citation – in particular, for example cito:AuthorSelfCitation (i.e. citing and the cited entities have at least one author in common) and cito:JournalSelfCitation (i.e. citing and the cited entities are published in the same journal).
Table 1. List of characteristics that can be associated with a citation when it is described as first-class data entity, using the properties and classes available in CiTO for their definition in RDF.
So as to identify each citation precisely, when described as first-class data entity and included in an open dataset, we have also developed the Open Citation Identifier (OCI) , which is a new globally unique persistent identifier for citations. OCIs are registered in the Identifiers.org platform (https://identifiers.org/oci) and recognized as persistent identifiers for citations by the EU FREYA Project (https://www.project-freya.eu) . Each OCI has a simple structure: the lower-case letters oci followed by a colon, followed by two sequences of numerals separated by a dash, where the first sequence is the identifier for the citing bibliographic resource and the second sequence is the identifier for the cited bibliographic resource. For example, oci:0301-03018 is a valid OCI for a citation defined within the OpenCitations Corpus, while oci:02001010806360107050663080702026306630509-02001010806360107050663080702026305630301 is a valid OCI for a citation included in Crossref. It is worth mentioning that OCIs are not opaque identifiers, since they explicitly encode directional relationships between identified citing and cited entities, the provenance of the citation, i.e. the database that contains it, and the type of identifiers used in that database to identify the citing and cited entities. In addition, we have created the Open Citation Identifier Resolution Service (http://opencitations.net/oci), which is a resolution service for OCIs based on the Python application oci.py available at https://github.com/opencitations/oci. Given a valid OCI as input, this resolution service is able to retrieve citation data in RDF (either as RDF/XML, Turtle or JSON-LD), or in Scholix, JSON or CSV formats. A more detailed explanation of OCIs and related material is available in .
At OpenCitations, we define an open citation index as a dataset containing citations that complies with the following requirements:
the citations contained are all open, according to the definition provided in ;
the citations are all treated as first-class data entities;
each citation is identified by an Open Citation Identifier (OCI) ;
the citation data are recorded in RDF according to the OpenCitations Data Model (OCDM) , where the OCI of a citation is embedded in the IRI defining it in RDF;
each citation defines the attributes shown in Table 1.
COCI: ingestion workflow, data, and services
COCI, the OpenCitations Index of Crossref open DOI-to-DOI references, is the first citation index to be published by OpenCitations, in which we have applied the concept of citations as first-class data entities, introduced in the previous section, to index the contents of one of the major open databases of scholarly citation information, namely Crossref (https://crossref.org), and to render and make available this information in machine-readable RDF under a CC0 waiver. Crossref contains metadata about publications (mainly academic journal articles) that are identified using Digital Object Identifiers (DOIs). Out of more than 100 million publications recorded in Crossref, Crossref also stores the reference lists of more than 43 million publications deposited by the publishers. Many of these references are to other publications bearing DOIs that are also described in Crossref, while others are to publications that lack DOIs and do not have Crossref descriptions. Crossref organises such publications with associated reference lists according to three categories: closed, limited and open. These categories to publications for which the reference lists are not visible to anyone outside the Crossref Cited-by membership, are visible only to them and to Crossref Metadata Plus members, or are visible to all users, respectively2.
Followed the first release of COCI on June 4, 2018, the most recent version of COCI, released on November 12, 2018, contains more that 445 million DOI-to-DOI citations included in the open and the limited datasets of Crossref reference data3. All the citation data in COCI and their provenance information, described according the Graffoo diagram  presented in Figure 2, are included in two distinct graphs – https://w3id.org/oc/index/coci/ and https://w3id.org/oc/index/coci/prov/ respectively – released under a CC0 waiver, and compliant with the FAIR data principles .
An example of a citation included in COCI is shown in the following excerpt (in Turtle), where the OCI is embedded as part of the IRI of the citation (without the oci: prefix) after the ci/ (meaning citation according to the OpenCitations Data Model ):
In the following subsections we introduce the ingestion workflow developed for creating COCI, we provides some figures on the citations it contains, and we list the resources and services we have made available to permit access to and querying of the dataset.
We processed all the data included in the October 2018 JSON dump of Crossref data, available to all the Crossref Metadata Plus members. The ingestion workflow, summarised in Figure 3, was organised in four distinct phases, and all the related scripts developed and used are released as open source code according to the ISC License and downloadable from the official GitHub repository of COCI at https://github.com/opencitations/coci.
Phase 1: global data generation. We parse and process the entire Crossref bibliographic database to extract all the publications having a DOI and their available list of references. Through this process three datasets are generated, which are used in the next phase:
Dates, the publication dates of all the bibliographic entities in Crossref and of all their references if they explicitly specify a DOI and a publication date as structured data – e.g. see the fields DOI and year in the array reference in https://api.crossref.org/works/10.1007/978-3-030-00668-6_8. Where the same DOI is encountered multiple times, e.g. as a proper item indexed in Crossref and also as a reference in the reference list of another article deposited in the Crossref, we use the full publication date defined in the indexed item.
ISSN: the ISSN (if any) and publication type (journal-article, book-chapter, etc.) of each bibliographic entity identified by a DOI indexed in Crossref.
ORCID: the ORCIDs (if any) associated with the authors of each bibliographic entity identified by a DOI indexed in Crossref.
Phase 2: CSV generation. We generate a CSV file such that each row represents a particular citation between a citing entity and a cited entity according to the data available in the Crossref dump, by looking at the DOI identifying the citing entity and all the DOIs specified in the reference list of such a citing entity according to the Crossref data. In particular, we execute the following four steps for each citation identified:
We generate the OCI for the citation by encoding the DOIs of the citing and cited entities into numerical sequences using the lookup table available at https://github.com/opencitations/oci/blob/master/lookup.csv, which are prefixed by the supplier prefix 020 to indicate Crossref as the source of the citation.
We retrieve the publication date of the citing entity from the Dates dataset and assign it as citation creation date.
We retrieve the publication date of the cited entity (from the Dates dataset) and we use it, together with the publication date of the citing entity retrieved in the previous step, to calculate the citation timespan.
We use the data contained in the ISSN and ORCID datasets to establish whether the citing and cited entity have been published in the same journal and/or have at least one author in common, and in these cases we assign the appropriate self-citation type(s) to the citation.
Simultaneously with the creation of the CSV file of citation data, we generate a second CSV file containing the provenance information for each citation (identified by its OCI generated in the aforementioned Step 1). These provenance data include the agent responsible for the generation of the citation, the Crossref API call that refers to the data of the citing bibliographic entity containing the reference used to create the citation, and the creation date of the citation.
Phase 3: converting into RDF. The CSV files generated in the previous phase are then converted into RDF according to the N-Triples format, following the OWL model introduced in Figure 2, where the DOIs of the citing and cited entities become DOI URLs starting with http://dx.doi.org/4, while the IRI of the citation includes its OCI (without the oci: prefix), as illustrated in the example given in the previous section.
Phase 4: updating the triplestore. The final RDF files generated in Phase 3 are used to update the triplestore used for the OpenCitations Indexes.
COCI was first created and released on July 4, 2018, and most recently updated on November 12, 2018. Currently, it contains 445,826,118 citations between 46,534,705 bibliographic entities. These are stored by means of 2,259,134,894 RDF statements (around 5 RDF statements per citation) for describing the citation data, and 1,337,478,354 RDF statements (3 statements per citation) for describing the related provenance information. Of the citations stored, 29,755,045 (6.7%) are journal self-citations, while 250,991 (0.06%) are author self-citations. The number of identified author self-citations, based on author ORCIDs, is a significant underestimate of the true number, mainly due to the sparsity of the data concerning the ORCID author identifiers within the Crossref dump. Journal entities (i.e. journals, volumes, issues, and articles) are the type of the bibliographic entities that are mostly cited, with over 420 million citations.
We also classify the cited documents according to their publishers – Table 2 shows the ten top publishers of citing and cited documents, calculated by looking at the DOI prefixes of the entities involved in each citation. As we can see, Elsevier is by far the publisher having the majority of cited documents. It is also the largest publisher that is not participating in the Initiative for Open Citations by making its publications’ reference lists open at Crossref – which is highlighted by the very limited amount of outgoing citations recorded in COCI. Its present refusal to open its article reference lists in Crossref, contrary to the practice of most of the major scholarly publishers, is contributing significantly to the invisibility of Elsevier’s own publications within the corpora of open citation data such as COCI that are increasingly being used by the scholarly community for discovery, citation network visualization and bibliometric analysis, as we introduce below in the section entitled Section 5.
Informa UK Limited
Institute of Electrical and Electronics Engineers (IEEE)
American Physical Society (APS)
Ovid Technologies (Wolters Kluwer Health)
Oxford University Press (OUP)
Table 2. A classification of the COCI citations according to the publishers of the cited (incoming citations) and citing (outgoing citations) documents. The table shows the top ten publishers by the overall amount of incoming and outgoing to/from their published works. Those publishers shown in italics are not participating in the Initiative for Open Citations by making their publications’ reference lists open at Crossref – see https://i4oc.org for additional information.
Resources and services
The citation data in COCI can be accessed in a variety of convenient ways, listed as follows.
Open Citation Index SPARQL endpoint. We have made available a SPARQL endpoint for all the indexes released by OpenCitations, including COCI, which is available at https://w3id.org/oc/index/sparql. When accessed with a browser, it shows a SPARQL endpoint editor GUI generated with YASGUI . Of course, this SPARQL endpoint can additionally be queried using the REST HTTP protocol, e.g. via curl. In order to access to COCI data, the graph https://w3id.org/oc/index/coci/ must be specified in the SPARQL query.
COCI REST API. Citation data in COCI can be retrieved by using the COCI REST API, available at https://w3id.org/oc/index/coci/api/v1. The rationale of making a REST API available in addition to the SPARQL endpoint was to provide convenient access to the the citation data included in COCI for Web developers and users who are not necessarily experts in Semantic Web technologies. This REST API, as are all the other REST APIs made available by OpenCitations, has been implemented by means of RAMOSE, the Restful API Manager Over SPARQL Endpoints (https://github.com/opencitations/ramose), which is a Python application that allows one to simply create a REST API over any SPARQL endpoint by means of a simple configuration file that execute a SPARQL query dependently of the particular API call specified. The configuration file for the COCI API is available at https://github.com/opencitations/api/blob/master/coci_v1.hf. Currently, the COCI REST API makes available four operations, that will retrieve either (a) the citation data for all the references of a given DOI (operation: references), or (b) the citation data for all the citations received by a given DOI (operation: citations), or (c) the citation data for the citation identified by an OCI (operation: citation), or (d) the metadata for the articles identified by the specified DOIs (operation: metadata). It is worth mentioning that the latter operation strictly depends on live API calls to external services, namely the Crossref API (https://api.crossref.org), the DataCite API (https://api.datacite.org), and the Unpaywall API (http://api.unpaywall.org), to gather the metadata of the requested articles, such as the title, the authors, and the journal name, that are not explicitly included within the OpenCitations Index triplestore.
Data dumps. All the citation data and provenance information in COCI are available as dumps stored in Figshare (https://figshare.com) in both CSV and N-Triples formats, while a dump of the whole triplestore is available on The Internet Archive (https://archive.org). The links to these dumps are available on the download page of the OpenCitations website (http://opencitations.net/download#coci).
In the past months, we have monitored the accesses to COCI data since its launch in July 2018. The statistics and graphics we show in this section highlight two different aspects: the quantification of the use of COCI data – and related services – and the community uptake, i.e. the use of COCI data for specific reuses within cross-community projects and studies. All the data of the charts described in this section are freely available for download from Figshare .
Figure 4 shows the number of accesses made between July 2018 and February 2019 (inclusive) to the various COCI services described above – the search/browse interfaces, the REST API, SPARQL queries, and others (e.g. direct HTTP access to particular citations and visits to COCI webpages in the OpenCitations website). We have excluded from all these counts all accesses made by automated agents and bots. As shown, the REST API is, by far, the most used service, with extensive usage recorded in the last four months, following the announcement of the second release of COCI. This is reasonable, considering that the REST API has been developed exactly for accommodating the needs of generic Web users and developers, including (and in particular) those who are not expert in Semantic Web technologies. There is just one exception in November 2018, where the SPARQL endpoint was used to retrieve quite a large amount of citation data. After further investigation, we noticed a large proportion of the SPARQL calls were coming from a single source (according to the IP data stored in our log), which probably collected citation data for a specific set of entities.
Figure 5 shows a particular cut of the figures given in Figure 4, which focuses on the REST API accesses only. In particular, we analysed which operations of the API were used the most. According to these figures, the most used operation is metadata (which was first introduced in the API in August 2018) which allows one to retrieve all the metadata describing certain publications. In contrast to the other API operations, this metadata search accepts one or more DOIs as input. The least used operation was citation, which allows one to retrieve citation data given an OCI, which should not be surprising, considering the currently limited knowledge of this new identifier system for citations.
In addition, we have also retrieved data about the views and downloads (as of March 29, 2019) of all the dumps uploaded to Figshare and to the Internet Archive. The CSV data dump received 1,321 views and 454 downloads, followed by the N-Triples data dump with 316 views and 93 downloads. The CSV provenance information dump has 166 and 127 downloads, while the N-Triples provenance information dump had 95 views and 34 downloads. Finally, the least accessed dump was that of the entire triplestore available in the Internet Archive, uploaded for the very first time in November 2018, that had only 3 views.
The data in COCI has been already used in various projects and initiatives. In this section, we list all the tools and studies doing this of which we are aware.
VOSviewer (http://www.vosviewer.com)  is a software tool, developed at the Leiden University’s Centre for Science and Technology Studies (CWTS), for constructing and visualizing bibliometric networks, which may include journals, researchers, or individual publications, and may be constructed based on citation, bibliographic coupling, co-citation, and co-authorship relations. Starting from version 1.6.10 (released on January 10, 2019), VOSviewer can now directly use citation data stored in COCI, retrieved by means of the COCI REST API.
Citation Gecko (http://citationgecko.com) is a novel literature mapping tool that allows one to map a research citation network using some initial seed articles. Citation Gecko is able to leverage citation links between seed papers and other papers to highlight papers of possible interest to the user, for which it uses COCI data (accessed via the REST API) to generate the citation network.
OCI Graphe (https://dossier-ng.univ-st-etienne.fr/scd/www/oci/OCI_graphe_accueil.html) is a Web tool that allows one to search articles by means of the COCI REST API, that are then visualised in a graph showing citations to the retrieved articles. It enriches this visualisation by adding additional information about the publication venues, publication dates, and other related metadata.
Zotero  is a free, easy-to-use tool to help users collect, organize, cite, and share research. Recently, the Open Citations Plugin for Zotero (https://github.com/zuphilip/zotero-open-citations) has been released, which allows users to retrieve open citation data extracted from COCI (via its REST API) for one or more articles included in a Zotero library.
COCI data, downloaded from the CSV dump available on Figshare, have been also used in at least two bibliometric studies. In particular, during the LIS Bibliometrics 2019 Event, Stephen Pearson presented a study (https://blog.research-plus.library.manchester.ac.uk/2019/03/04/using-open-citation-data-to-identify-new-research-opportunities/) run on publications by scholars at the University of Manchester which used COCI to retrieve citations between these publications so as to investigate possible cross-discipline and cross-department potential collaborations. Similarly, COCI data were used to conduct an experiment on the latest Italian Scientific Habilitation  (the national exercise that evaluates whether a scholar is appropriate to receive an Associate/Full Professorship position in an Italian university), which aimed at trying to replicate part of the outcomes of this evaluation exercise for the Computer Science research field by using only open scholarly data, including the citations available in COCI, rather than citation data from subscription services.
In this paper, we have introduced COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. After an initial introduction of the notion of citations as first-class data entities, we have presented the ingestion workflow that has been implemented to create COCI, have detailed the data COCI contains, and have described the various services and resources that we have made available to access COCI data. Finally, we have presented some statistics about the use of COCI data, and have mentioned the tools and studies that have adopted COCI in recent months.
COCI is just the first open citations index that OpenCitations will make available. Using the experience we have gathered by creating it, we now plan the release of additional indexes, so as to extend the coverage of open citations available through the OpenCitations infrastructure. The first of these, recently released, is CROCI (https://w3id.org/oc/index/croci) , the Crowdsourced Open Citations Index, which contains citations deposited by individuals. CROCI is designed to permit scholars proactively to fill the open citations gap in COCI resulting from four causes: (a) the failure of many publishers using Crossref DOIs to deposit reference lists of their publications at Crossref, (b) the failure of some publishers that do deposit their reference lists to make these reference lists open, in accordance with the recommendations of the Initiative for Open Citations; (c) the absence from ~11% of Crossref reference metadata of the DOIs for cited articles which in fact have been assigned DOIs (https://www.crossref.org/blog/underreporting-of-matched-references-in-crossref-metadata/), a problem that Crossref are currently working hard to rectify; and (d) the existence of citations to published entities that lack Crossref DOIs. In the near future, we plan to extend the number of indexes by harvesting citations from other open datasets including Wikidata (https://www.wikidata.org), DataCite (https://datacite.org), and Dryad (https://datadryad.org). In addition, we plan to extend and generalise the current software developed for COCI, so as to facilitate most frequent updates of the indexes.
We gratefully acknowledge the financial support provided to us by the Alfred P. Sloan Foundation for the OpenCitations Enhancement Project (grant number G‐2017‐9800).
Nuzzolese, A. G., Gentile, A. L., Presutti, V., Gangemi, A. (2016). Conference Linked Data: The ScholarlyData project. In Proceedings of the 15th International Semantic Web Conference (ISWC 2015): 150-158. DOI: https://doi.org/10.1007/978-3-319-46547-0_16
Hammond, T., Pasin, M., & Theodoridis, E. (2017). Data integration and disintegration: Managing Springer Nature SciGraph with SHACL and OWL. In International Semantic Web Conference (Posters, Demos & Industry Tracks). http://ceur-ws.org/Vol-1963/paper493.pdf
Alexiou, G., Vahdati, S., Lange, C., Papastefanatos, G., Lohmann, S. (2016). OpenAIRE LOD services: scholarly communication data as linked data. In Semantics, Analytics, Visualization. Enhancing Scholarly Data: 45-50. DOI: https://doi.org/10.1007/978-3-319-53637-8_6
Peroni, S., Shotton, D., Vitali, F. (2017). One year of the OpenCitations Corpus – releasing RDF-based scholarly citation data into the public domain. In Proceedings of the 16th International Semantic Web Conference (ISWC 2017): 184-192. DOI: https://doi.org/10.1007/978-3-319-68204-4_19
Garcia, A., Lopez, F., Garcia, L., Giraldo, O., Bucheli, V., Dumontier, M. (2018). Biotea: semantics for Pubmed Central. PeerJ, 6: e4201. DOI: https://doi.org/10.7717/peerj.4201
Bagnacani, A., Ciancarini, P., Di Iorio, A., Nuzzolese, A. G., Peroni, S., Vitali, F. (2014). The Semantic Lancet Project: A Linked Open Dataset for Scholarly Publishing. In EKAW 2014 Satellite Events: 101-105. DOI: https://doi.org/10.1007/978-3-319-17966-7_10
Erxleben, F., Günther, M., Krötzsch, M., Mendez, J., Vrandečić, D. (2014). Introducing Wikidata to the linked data web. In Proceedings of the 13th International Semantic Web Conference (ISWC 2013): 50-65. DOI: https://doi.org/10.1007/978-3-319-11964-9_4
Di Iorio, A., Peroni, S., Poggi, F. (2019). Open data to evaluate academic researchers: an experiment with the Italian Scientific Habilitation. (To appear) Proceedings of the 17th International Conference on Scientometrics and Informetrics (ISSI 2019). https://arxiv.org/abs/1902.03287
Heibi, I., Peroni, S., Shotton, D. (2019). Crowdsourcing open citations with CROCI – An analysis of the current status of open citations, and a proposal. (To appear) Proceedings of the 17th International Conference on Scientometrics and Informetrics (ISSI 2019). https://arxiv.org/abs/1902.02534
Sugimoto, C. R., Waltman, L., Larivière, V., van Eck, N. J, Boyack, K. W., Wouters, P., de Rijcke, S. (2017). Open citations: A letter from the scientometric community to scholarly publishers. ISSI Society. http://issi-society.org/open-citations-letter (last visited 23 March 2019)
Ferguson, C., McEntrye, J., Bunakov, V., Lambert, S., van der Sandt, S., Kotarski, R., … McCafferty, S. (2018). Survey of Current PID Services Landscape (Deliverable No. D3.1). Retrieved from FREYA project (EC Grant Agreement No 777523) website: https://www.project-freya.eu/en/deliverables/freya_d3-1.pdf
Wilkinson, M. D., Dumontier, M., Aalbersberg, Ij. J., Appleton, G., Axton, M., Baak, A., … Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018. DOI: https://doi.org/10.1038/sdata.2016.18
1. An in-depth description about the definition and use of citations as first-class data entities can be found at https://opencitations.hypotheses.org/816. [back]2. Additional information on this classification of Crossref reference lists is available at https://www.crossref.org/reference-distribution/.[back]3. We have access to the limited dataset since we are members of the Crossref Metadata Plus plan.[back]4. We are aware that the current practice for DOI URLs is to use the base https://doi.org/ instead of http://dx.doi.org/. However, when one tries to resolve a DOI URL owned by Crossref by specifying an RDF format (e.g. Turtle) in the accept header of the request, the bibliographic entity is actually defined using the old URL structure starting with http://dx.doi.org/. For this reason, since COCI is derived entirely from Crossref data, we decided to stay with the approach currently used by Crossref.[back]
Keywords: Open citations; COCI; CROCI; Crossref; I4OC; OpenCitations
Copyright notice: This work is licensed under a Creative Commons Attribution 4.0 International License. You are free to share (i.e. copy and redistribute the material in any medium or format) and adapt (e.g. remix, transform, and build upon the material) for any purpose, even commercially, under the following terms: attribution, i.e. you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. The licensor cannot revoke these freedoms as long as you follow the license terms.
Notes: submitted as research-in-progress paper to the 17th International Conference on Scientomentrics and Informetrics (ISSI 2019, https://www.issi2019.org). PDF version available on arXiv.
In this paper, we analyse the current availability of open citations data in one particular dataset, namely COCI (the OpenCitations Index of Crossref open DOI-to-DOI citations; http://opencitations.net/index/coci) provided by OpenCitations. The results of these analyses show a persistent gap in the coverage of the currently available open citation data. In order to address this specific issue, we propose a strategy whereby the community (e.g. scholars and publishers) can directly involve themselves in crowdsourcing open citations, by uploading their citation data via the OpenCitations infrastructure into our new index, CROCI, the Crowdsourced Open Citations Index.
The availability of open scholarly citations is a public good, which is of intrinsic value to the academic world as a whole (Shotton, 2013; Peroni et al. 2015; Shotton, 2018), and is particularly crucial for the scientometrics and informetrics community, since it supports reproducibility (Sugimoto et al., 2017) and enables fairness in research by removing such citation data from behind commercial paywalls (Schiermeier, 2017). Despite the positive early outcome of the Initiative for Open Citations (I4OC, https://i4oc.org), namely that almost all major scholarly publishers now release their publication reference lists, with the result that more than 500 million citations are now open via the Crossref API (https://api.crossref.org), and despite the related ongoing efforts of sister infrastructures and initiatives such as OpenCitations (http://opencitations.net) and WikiCite/Wikidata (https://www.wikidata.org), many scholarly citations are not freely available. While these initiatives have the potential to disrupt the traditional landscape of citation availability, which for the past half-century has been dominated by commercial interests, the present incomplete coverage of open citation data is one of the most significant impediments to open scholarship (van Eck et al., 2018).
In this work, we analyse the current availability of open citations data (Peroni and Shotton, 2018) within one particular dataset, namely COCI (the OpenCitations Index of Crossref open DOI-to-DOI citations; http://opencitations.net/index/coci). This dataset is provided by OpenCitations, a scholarly infrastructure organization dedicated to open scholarship and the publication of open bibliographic and citation data by the use of Semantic Web (Linked Data) technologies. Launched in July 2018, COCI is the first of the Indexes proposed by OpenCitations (http://opencitations.net/index) in which citations are exposed as first-class data entities with accompanying properties. It has already seen widespread usage (over nine hundred thousands API calls since launch, with half of these in January 2019), and has been adopted by external services such as VOSviewer (van Eck and Waltman, 2010).
In particular, in this paper we address the following research questions (RQs):
What is the ratio between open citations vs. closed citations within each category of scholarly entities included in COCI (i.e. journals, books, proceedings, datasets, and others)?
Which are the top twenty publishers in terms of the number of open citations received by their own publications, according to the citation data available in COCI?
To what degree are the publishers highlighted in the previous analysis themselves contributing to the open citations movement, according to the data available in Crossref?
The results of these analyses show a persistent gap in the coverage of the currently available open citation data. To address this specific issue, we have developed a novel strategy whereby members of the community of scholars, authors, editors and publishers can directly involve themselves in crowdsourcing open citations, by uploading their citation data via the OpenCitations infrastructure into our new index, CROCI, the Crowdsourced Open Citations Index.
Methods and material
To answer the RQs mentioned above, we used open data and technologies coming from various parties. Specifically, the open CC0 citation data we used came from the CSV dump of most recent release of COCI dated 12 November 2018 (OpenCitations, 2018), which contains 449,840,503 DOI-to-DOI citation links between 46,534,705 distinct bibliographic entities. The Crossref dump we used for the production of this most recent version of COCI was dated 3 October 2018, and included all the Crossref citation data available at that time in both the ‘open’ dataset (accessible by all) and the ‘limited’ dataset (accessible only to users of the Crossref Cited-by service and to Metadata Plus members of Crossref, of which OpenCitations is one – for details, see https://www.crossref.org/reference-distribution/).
We additionally extracted information about the number of closed citations to each of the 99,444,883 DOI-identified entities available in the October Crossref dump. This number was calculated by subtracting the number of open citations to each entity available within COCI from the value “is-referenced-by-count” available in the Crossref metadata for that particular cited entity, which reports all the DOI-to-DOI citation links that point to the cited entity from within the whole Crossref database (including those present in the Crossref ‘closed’ dataset).
Furthermore, we extracted the particular publication type of each entity, so as to identify it either as a journal article, or as a book chapter, etc. We determined these publication types for all the DOI-identified entities available in the Crossref dump we used. We then identified the publisher of each entity, by querying the Crossref API using the entity’s DOI prefix. This allowed us to group the number of open citations and closed citations to the articles published by that particular publisher, and to determine the top twenty publishers in terms of the number of open citations that their own publications had received.
Finally, we again queried the Crossref API, this time using the DOI prefixes of the citing entities, to check the participation of these top twenty publishers in terms of the number of open citations they were themselves publishing in response to the open citation movement sponsored by I4OC. Details of all these analyses are available online in CC0 (Heibi et al., 2019).
First (RQ1) we determined the numbers of open citations and closed citations received by the entities in the Crossref dump. All the entity types retrieved from Crossref were aligned to one of following five categories: journal, book, proceedings, dataset, other, as illustrated in (Heibi et al., 2019). The outcomes are summarised in Figure 1, where it is evident that the number of open citations available in COCI is always greater than the number of closed citations to these entities to which COCI does not have access, for each of the publication categories considered, with the categories proceedings and dataset having the largest ratios.
Analysis of the Crossref data show that there are in total ~4.1 million DOIs that have received no open citations and at least one closed citation. Conversely, there are ~10.7 million DOIs that have received no closed citations and at least one open citation in COCI. Most of the papers in both these categories have received very few citations.
The outcome of the second analysis (RQ2) shows which publishers are receiving the most open citations. To this end, we considered all the open citations recorded in COCI, and compared them with the number of closed citations to these same entities recorded in Crossref. Figure 2 shows the top twenty publishers that received the greatest number of open citations. Elsevier is the first publisher according to this ranking, but it also records the highest number of closed citations received (~97M vs. ~105.5M). The highest ratio in terms of open citations vs. closed citations was recorded by IEEE publications (ratio 6.25 to 1), while the lowest ratio was for the American Chemical Society (ratio 0.73 to 1).
Considering the twenty publishers listed in Figure 2, we wanted additionally to know their current support for the open citation movement (RQ3). The results of this analysis (made by querying the Crossref API on 24 January 2019) are shown in Figure 3. Among the top ten publishers shown in Figure 2, i.e. those who themselves received the largest numbers of open citations, only five, namely Springer Nature, Wiley, the American Physical Society, Informa UK Limited, and Oxford University Press, are participating actively in the open publication of their own citations through Crossref.
It is noteworthy that JSTOR contributes very few references to Crossref, while the many citations directed towards its own holdings place JSTOR twelfth in the list of publishers receiving open citations (Figure 2). However, as the last column of Figure 3 shows, all the major publishers listed here are failing to submit reference lists to Crossref for a large number of the publications for which they submit metadata, that number being the difference between the value in the last column for that publisher and the combined values in the preceding three columns. JSTOR is the worst in this regard, submitting references with only 0.53% of its deposits to Crossref, while the American Physical Society is the best, submitting references with 96.54% of its publications recorded in Crossref.
It should be stressed that a very large number of potentially open citations are totally missing in the Crossref database, and consequently from COCI, for the simple reason that many publishers, particularly smaller ones with limited technical and financial resources, but also all the large ones shown in Figure 3 and most of the others, are simply not depositing with Crossref the reference lists for any or all of their publications.
According to the data retrieved, the open DOI-to-DOI citations available in COCI exceed the number of closed DOI-to-DOI citations recorded in Crossref for every publication category, as shown in Figure 1. The journal category is the one receiving the most open citations overall, as expected considering the historical and present importance of journals in most areas of the scholarly ecosystem. However, the number of closed citations to journal articles within Crossref is also of great significance, since these 322 million closed citations represent 43% of the total.
It is important to note that about one third of these closed citations to journal articles (according to Figure 2) are references to entities published by Elsevier, and that references from within Elsevier’s own publications constitute the largest proportion of these closed citations, since Elsevier is the largest publisher of journal articles. Thus Elsevier’s present refusal to open its article references is contributing significantly to the invisibility of Elsevier’s own publications within the corpus of open citation data that is being increasingly used by the scholarly community for discovery, citation network visualization and bibliometric analysis.
It is also worth mentioning the discrepancy between the citations available in COCI, which comes from the data contained in the open and limited Crossref datasets as of 3 October 2018, and those available within those same Crossref datasets as of 24 January 2019. The most significant difference relates to IEEE. While the citations present in COCI include those from IEEE publications to other entities prior to November 2018 (since in October 2018 its article metadata with references were present within the Crossref limited dataset), in November 2019 this scholarly society decided to close the main part of its Crossref references, and thus from that moment they became unavailable to Crossref Metadata Plus members such as OpenCitations, as highlighted in Figure 3. Thus IEEE citations from articles whose metadata was submitted to Crossref after the date of this switch to closed will no longer be automatically ingested into COCI.
To date, the majority of the citations present in Crossref that are not available in COCI comes from just three publishers: Elsevier, the American Chemical Society and University of Chicago Press (Figure 3). In fact, considering the average value of 18.6 DOI-to-DOI citation links for each citing entity – calculated by dividing the total number of citations in COCI by the number of citing entities in the same dataset – these three publishers are holding more than 214 million DOI-to-DOI citations that could potentially be opened. (The IEEE citation data which was in the Crossref ‘limited’ category as of October 2018 are actually included in COCI, although those from that organization’s more recent publications will no longer be, as mentioned above).
We think it is deeply regrettable and almost incomprehensible that any professional organization, learned society or university press, whose primary mission is to serve the interests of the practitioners, scholars and readers it represents, should choose not open all its publications’ reference lists as a public good, whatever secondary added-value services it chooses to build on top of the citations that those reference lists contain.
CROCI, the Crowdsourced Open Citations Index
The results of the Initiative for Open Citations (I4OC) have been remarkable, since its efforts have led to the liberation of millions of citations in a relatively short time. However, many more citations, the lifeblood of the scholarly communication, are still not available to the general public, as mentioned in the previous section. Some researchers and journal editors, in particular, have recently started to interact with publishers that are not participating in I4OC, in attempts to convince them to release their citation data. Remarkable examples of these activities are the petition promoted by Egon Willighagen (https://tinyurl.com/acs-petition) addressed to the American Chemical Society, and the several unsuccessful requests made to Elsevier by the Editorial Board of the Journal of Informetrics, which eventually resulted in the resignation of the entire Editorial Board on 10 January 2019 in response to Elsevier’s refusal to address their issues (http://www.issi-society.org/media/1380/resignation_final.pdf).
To provide a pragmatic alternative that would permit the harvesting of currently closed citations, so that they could then be made available to the public, we at OpenCitations have created a new OpenCitations Index: CROCI, the Crowdsourced Open Citations Index, into which individuals identified by ORCiD identifiers may deposit citation information that they have a legal right to submit, and within which these submitted citation data will be published under a CC0 public domain waiver to emphasize and ensure their openness for every kind of reuse without limitation. Since citations are statements of fact about relationships between publications (resembling statements of fact about marriages between individual persons), they are not subject to copyright, although their specific textual arrangements within the reference lists of particular publications may be. Thus the citations from which the reference list of an author’s publication has been composed may legally be submitted to CROCI, although the formatted reference list cannot be. Similarly, citations extracted from within an individual’s electronic reference management system and presented in the requested format may be legally submitted to CROCI, irrespective of the original sources of these citations.
To populate CROCI, we ask researchers, authors, editors and publishers to provide us with their citation data organised in a simple four-column CSV file (“citing_id”, “citing_publication_date”, “cited_id”, “cited_publication_date”), where each row depicts a citation from the citing entity (“citing_id”, giving the DOI of the cited entity) published on a certain date (“citing_publication_date”, with the date value expressed in ISO format “yyyy-mm-dd”), to the cited entity (“cited_id”, giving the DOI of the cited entity) published on a certain date (“cited_publication_date”, again with the date value expressed in ISO format “yyyy-mm-dd”). The submitted dataset may contain an individual citation, groups of citations (for example those derived from the reference lists of one or more publications), or entire citation collections. Should any of the submitted citations be already present within CROCI, these duplicates will be automatically detected and ignored.
The date information given for each citation should be as complete as possible, and minimally should be the publication years of the citing and cited entities. However, if such date information is unavailable, we will try to retrieve it automatically using OpenCitations technology already available. DOIs may be expressed in any of a variety of valid alternative formats, e.g. “https://doi.org/10.1038/502295a”, “http://dx.doi.org/10.1038/502295a”, “doi: 10.1038/502295a”, “doi:10.1038/502295a”, or simply “10.1038/502295a”.
Submission of such a citation dataset in CSV or Scholix format should be made as a file upload either to Figshare (https://figshare.com) or to Zenodo (https://zenodo.org). For provenance purposes, the ORCID personal identifier of the submitter of these citation data should be explicitly provided in the metadata or in the description of the Figshare/Zenodo object. Once such a citation data file upload has been made, the submitter should inform OpenCitations of this fact by adding an new issue to the GitHub issue tracker of the CROCI repository (https://github.com/opencitations/croci/issues).
OpenCitations will then process each submitted citation dataset and ingest the new citation information into CROCI. CROCI citations will be available at http://opencitations.net/index/croci using an appropriate REST API and SPARQL endpoint, and will additionally be published as periodic data dumps in Figshare, all releases being under CC0 waivers. We propose in future to enable combined searches over all the OpenCitations indexes, including COCI and CROCI.
We are confident that the community will respond positively to this proposal of a simple method by which the number of open citations available to the academic community can be increased, in particular since the data files to be uploaded have a very simple structure and thus should be easy to prepare. In particular, we hope for submissions of citations from within the reference lists of authors’ green OA versions of papers published by Elsevier, IEEE, ACS and UCP, and from publishers not already submitting publication metadata to Crossref, so as to address existing gaps in open citations availability. We look forward to your active engagement in this initiative to further increase the availability of open scholarly citations.
On the 5th February 2019, after these analyses had been concluded and the text of this paper had been finalized, Crossref published an announcement that DOIs were missing from approximately 11% of its references (https://www.crossref.org/blog/underreporting-of-matched-references-in-crossref-metadata/), because of an historical fault in the manner in which Crossref automatically processes references upon deposit by publishers. This fault means that the absolute numbers of the open and closed DOI-to-DOI citations given in this paper are significantly lower than they should be. While this does not invalidate the comparisons we have reported here, it is clearly regrettable. In April 2019, after Crossref have completed the re-processing of their reference data to include the missing DOIs, we will create an updated version of COCI, and will then recompute and republish the data presented here to include those citations for which Crossref is presently failing to correctly assigned DOIs to the cited entities.
Heibi, I., Peroni, S. & Shotton, D. (2019). Types, open citations, closed citations, publishers, and participation reports of Crossref entities. Version 1. Zenodo. DOI: https://doi.org/10.5281/zenodo.2558257
Peroni, S., Dutton, A., Gray, T. & Shotton, D. (2015). Setting our bibliographic references free: towards open citation data. Journal of Documentation, 71: 253-277. DOI: https://doi.org/10.1108/JD-12-2013-0166
Sugimoto, C. R., Waltman, L., Larivière, V., van Eck, N. J., Boyack, K. W., Wouters, P. & de Rijcke, S. (2017). Open citations: A letter from the scientometric community to scholarly publishers. ISSI. http://www.issi-society.org/open-citations-letter/ (last visited 26 January 2018)
van Eck, N.J., Waltman, L., Larivière, V. & Sugimoto, C. R. (2018). Crossref as a new source of citation data: A comparison with Web of Science and Scopus. CWTS Blog. https://www.cwts.nl/blog?article=n-r2s234 (last visited 26 January 2018)
As a result of the Initiative for Open Citations (I4OC), launched on April 6 last year, almost all the major scholarly publishers now open the reference lists they submit to Crossref, resulting in more than half a billion references being openly available via the Crossref API.
It is therefore time to think carefully about how citations are treated, and how they might be better handled as part of the Linked Open Data Web.
Citations are normally treated simply as the links between published entities.
However, an alternative richer view is to regard a citation as a data entity in its own right.
This permits us to endow a citation with descriptive properties, such as
has citation creation date: 3rd March 2015
has citation time span: 6 years, 5 months and 23 days
has type: Self-citation
has identifier: oci:7295288-3962641
[Note: a later blog post entitled “Open Citation Identifiers” will include an explanation of the identifier shown here.]
Advantages of treating citations as First-Class Data Entities
All the information regarding each citation is available in one place.
Citations become easier to describe, distinguish, count and process.
If available in aggregate, citations described in this manner are easier to analyze using bibliometric methods, for example to determine how citation time spans vary by discipline.
Requirements for citations to be treated as First-Class Data Entities
They must be definable in a machine-readable manner as a member of the class “Citation”, and describable using appropriate ontology terms.
They must have metadata structured using a generic yet appropriately detailed data model.
They must be storable, searchable and retrievable in an open database designed for bibliographic citations.
They must be identifiable using a global persistent identifier scheme.
There must be a Web-based identifier resolution service that takes the citation identifier as input and returns a description of the citation.
Blog post detailing how these requirements are met
Subsequent blog posts will describe how we at OpenCitations have satisfied these requirements, permitting citations to indeed be treated as First-Class Data Entities:
Some folk are confused, but OpenCitations and the Initiative for Open Citations, despite the similarity of their names, are two distinct organizations.
OpenCitations (http://opencitations.net) is an open scholarly infrastructure organization directed by Silvio Peroni and myself, and its primary purpose is to host and build the OpenCitations Corpus (OCC), an RDF database of scholarly citation data that now contains almost 13 million citation links.
In contrast, the Initiative for Open Citations (I4OC; https://i4oc.org) is a separate and independent organization, whose founding was spearheaded by Dario Taraborelli of the WikiMedia Foundation. OpenCitations was just one of several organizations that founded the Initiative for Open Citations, as documented at https://i4oc.org/#founders.
I4OC is a collaboration between scholarly publishers, researchers, and other interested parties to promote the unrestricted availability of scholarly citation data, but it does not itself host citation data.
Within a short space of time, I4OC has persuaded most of the major scholarly publishers to open their reference lists submitted to Crossref, so that the proportion of all references submitted to Crossref that are now open has risen from 1% to over 50%.
These references are now available for OpenCitations to harvest into the OpenCitations Corpus and publish in RDF, as well as for others to harvest and use as they wish.
Good news! Today, on January 16th 2018, Oxford University Press (OUP) announced its participation in the Initiative for Open Citations, and requested Crossref to turn on reference sharing for all OUP deposited references from more than half a million publications. Oxford University Press is the largest university press in the world, publishing in 70 languages and 190 countries.