Discover POCI, the index of open citations from PubMed 

We’re happy to announce POCI, the OpenCitations Index of PubMed open PMID-to-PMID citations, an RDF dataset containing details of all the citations from publications bearing PubMed Identifiers (PMIDs) to other PMID-identified publications, harvested from the National Institutes of Health Open Citations Collection (NIH-OCC). The citations available in POCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. 

Currently, POCI’s December 2022 release contains 717,654,703 citations from 26,024,862 bibliographic resources, and is based on the dump of NIH Open Citation Collection dated November 2022. 

Citation URLs

Each citation (i.e. an individual of the class cito:Citation) is identified by an URL structured as follows:

https://w3id.org/oc/index/poci/ci/[[OCI]].

Open Citation Identifiers

Each Open Citation Identifier [[OCI]] has a simple structure: the lower-case letters “oci” followed by a colon, followed by two numbers separated by a dash (e.g. https://w3id.org/oc/index/poci/ci/01600102060800080706-016002060909030401), in which the first number identifies the citing work and the second number identifies the cited work.

For citations in which the citing and cited works are identified by PMIDs, which includes all the POCI citations, the OCI is created in the following manner, as explained more fully here. Each converted numeral part of OCI is prefixed by a 0160, which indicates that NIH is the supplier of the original metadata of the citation (as indicated at http://opencitations.net/oci).

OCIs can be resolved using the OpenCitations OCI Resolution Service.

Access to POCI data

All the data in POCI:

What is an Open Citation Index?

A citation index is a bibliographic index recording citations between publications, allowing the user to establish which later documents cite earlier documents. The current indexes available in OpenCitations are:  

All the OpenCitations Indexes have six characteristics in common, summarized here: https://opencitations.net/index   

Cite this article as: Chiara Di Giambattista, "Discover POCI, the index of open citations from PubMed ," in OpenCitations blog, 27/12/2022, https://opencitations.hypotheses.org/3246.

 

Discover DOCI, the index of open citations from DataCite

We’re excited to introduce DOCI, the OpenCitations Index of Datacite open DOI-to-DOI citations, a new tool containing citations derived from publications bearing DataCite DOIs to other DOI-identified publications, harvested from DataCite. The citations available in DOCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model

Currently, DOCI’s December 2022 release contains 169,822,752 citations from 1,753,860  citing resources, and is based on the last dump of DataCite dated 22 October 2021 provided by the Internet Archive

Citation URLs

Each citation (i.e. an individual of the class cito:Citation) is identified by an URL structured as follows:

 https://w3id.org/oc/index/doci/ci/[[OCI]].

Open Citation Identifiers

Each Open Citation Identifier [[OCI]] has a simple structure: the lower-case letters “oci” followed by a colon, followed by two numbers separated by a dash (e.g. https://opencitations.net/index/doci/ci/080010504060836132137200707121027-080010504060836161221130313.html), in which the first number identifies the citing work and the second number identifies the cited work.

For citations in which the citing and cited works are identified by DOIs, which includes all the DOCI citations, the OCI is created in the following manner, as explained more fully here. Each case-insensitive DOI is first normalized to lower case letters. Then, after omitting the initial doi:10. prefix, the alphanumeric string of the DOI is converted reversibly to a pure numerical string using the simple two-numeral lookup table for numerals, lower case letters and other characters presented at https://github.com/opencitations/oci/blob/master/lookup.csv. Finally, each converted numeral is prefixes by a 080, which indicates that DataCite is the supplier of the original metadata of the citation (as indicated at http://opencitations.net/oci).

OCIs can be resolved using the OpenCitations OCI Resolution Service.

Access to DOCI data

All the data in DOCI:

More information is available at https://opencitations.net/index/doci. 

What is an Open Citation Index? 

A citation index is a bibliographic index recording citations between publications, allowing the user to establish which later documents cite earlier documents. The current indexes available in OpenCitations are: 

All the OpenCitations Indexes have six characteristics in common, summarized here: https://opencitations.net/index  

Follow OpenCitations on Mastodon

OpenCitations has happily joined the open-source social media platform joinmastodon.org.

Mastodon is “a free and open-source software developed by a non-profit organization”, with the aim of favouring interoperability and bringing social media interaction “back in the hands of the people”.

We look forward to recreating there our wide network of connections, and getting in touch with new people, projects and institutions in a different virtual environment.

Follow us at https://scicomm.xyz/@opencitations !

Tutorial: how to process COCI’s zipped CSV dump without decompressing it

Blog post by Ivan Heibi (Universiy of Bologna) and Arcangelo Massari (University of Bologna).

OpenCitations publishes the COCI dataset after each new release in three main formats: CSV, N-Triples, and Scholix (see https://opencitations.net/download#coci). The CSV format is the most popular and downloaded one due to its comprehensive data organization (i.e. tabular format) and smaller size (compared to the other formats provided). Therefore, this is also the format we suggest using for a local process of the entire COCI dataset. 

The CSV dumps of COCI are uploaded on Figshare. You can check and download the last dump released from https://doi.org/10.6084/m9.figshare.6741422. The dump consists of one main ZIP file, including other smaller ZIP archives (one for each release) containing the actual CSV files (Figure 1).

Figure 1. The contents of the COCI CSV dataset (after the August 2022 release)

It is possible to process this data without unzipping the internal archives, thus saving a lot of disk space. In this tutorial, we will see how to achieve this in Python. Same process could be done in other programming languages.

Processing the COCI dump using Python

Step 1) Downloading the COCI dump

First, you need to download the last CSV dump release of COCI from https://doi.org/10.6084/m9.figshare.6741422 and decompress only the external archive. After this operation, you should have a folder containing the internal ZIP files such as in Figure 1.

Note: It is beneficial to decompress the external archive because doing so does not increase the space occupied on the disk (compressing archives results in a compression rate of 0%) and because working on nested archives would significantly increase RAM requirements. 

Step 2) Working with the ZIP files

Python provides the built-in zipfile module, whose ZipFile class allows you to create, read, write, edit and list the contents of a ZIP file. Given as input the path of the root directory containing all the ZIP files (FOLDER_PATH), the process elaborates each of these files on a different iteration. Each cycle initializes a ZipFile object by specifying the path to the ZIP file (archive_path).

from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):        
archive_path = os.path.join(FOLDER_PATH, archive_name)
with ZipFile(archive_path) as archive:     # ...

Step 3) Accessing the ZIP files

Use the namelist() method to return the list of CSV files contained in each archive. Then to open the inner CSV files, simply cycle through the list of names and feed them to the open() method of the ZipFile instance, i.e. archive in the example below.

from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:       # ...

Step 4) Reading the CSVs

The .open() method returns a buffer. To read the CSV file as a list of dictionaries (i.e. represent each row of the CSV in dictionary format, e.g., {“column1″:”val1”, “column2″:”val2”}) we need to transform the buffer using the TextIOWrapper class and read it using the DictReader class of csv. Then we convert the result of DictReader into a list. 

from io import TextIOWrapper
from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# ...

Step 5) Processing the CSVs content

Now you can go through each row of the list and process the citation data as you want. The following example prints the citing and cited entity of each citation in the dump. 

from io import TextIOWrapper
from zipfile import ZipFile
import os

for archive_name in os.listdir(FOLDER_PATH):
    archive_path = os.path.join(FOLDER_PATH, archive_name)
    with ZipFile(archive_path) as archive:
        for csv_name in archive.namelist():
with archive.open(csv_name) as csv_file:
reader = csv.DictReader(io.TextIOWrapper(csv_file))
rows = list(reader)
# Process the CSV here
for r in rows:
print("Citing entity:",r["citing"])
print("Cited entity:",r["cited"])

Cite this article as: Arcangelo Massari, "Tutorial: how to process COCI’s zipped CSV dump without decompressing it," in OpenCitations blog, 30/09/2022, https://opencitations.hypotheses.org/2940.

 

Transparency meets open citations

This post was first published on QUERTY: musings from the rabbit holea blog by Silvio Peroni

In the scholarly ecosystem, a bibliographic citation is a conceptual directional link from a citing entity to a cited entity, used to acknowledge or ascribe credit for the contribution made by the author(s) of the cited entity. Citations are one of the core elements of scholarly communication. They enable the integration of our independent research endeavours into a global graph of relationships that can be used, for instance, to analyse how scholarly knowledge develops over time, assess scholars’ influence, and make wise decisions about research investment.

A copyrighted fact

However, as citation data, i.e. pieces of factual information aiming at identifying entities and relationships among them, are of great value to the scholarly community, it has been a “scandal” that they have not been recognised as part of the commons. Indeed, only recently we have seen some efforts – such as the Initiative for Open Citations (I4OC) – that have tried to change the behind-the-paywall status quo enforced by the companies controlling the major citation indexes used worldwide, convincing scholarly publishers to support the unrestricted availability of scholarly citation data by publishing them in suitable open infrastructures, such as Crossref and DataCite.

Of course, as for many other kinds of data, putting bibliographic and citation data behind a paywall is a thread to enabling the full reproducibility of research studies based on them (e.g. in bibliometrics, scientometrics, and science of science domains), even when such studies are published in open access articles. For instance, the results of a recent open access article published on Digital Scholarship in the Humanities, which aimed to analyse the citation behaviour of Digital Humanities (DH) research across different proprietary and open citation databases, are not fully reproducible since the majority of the databases used – namely Scopus, Web of Science, and Dimensions – do not make their bibliographic and citation data openly available.

In addition, the coverage of publications and related citations in specific disciplines, particularly those within the Social Sciences and the Humanities (SSH), is inadequate compared to other fields. Usually, this is due to the limited availability of born-digital publications accompanied by a wide variety of publication languages, publication types (e.g. monographs), and complex referencing practices that may limit their automatic processing and citation extraction. As a side effect, such a partial coverage may result in a considerable bias when analysing SSH disciplines compared to STEM disciplines that usually have better coverage in existing citation databases.

Reforming research assessment

All these scenarios have at least another negative effect on the area strictly concerned with the research assessment, which often uses quantitative metrics based on citation data to evaluate articles, people, and institutions. Indeed, the unavailability and partial coverage of bibliographic and citation data create an artificial barrier to the transparency of the processes used to decide the careers of scholars in terms of research, funding, and promotions.

In the past years, several initiatives around the world have highlighted the importance of reforming research assessment exercises, such as those summarised in the following figure: the Frech National Plan for Open Science, the San Francisco Declaration on Research Assessment, the Leiden Manifesto for Research Metrics, and the recent proposal for a reform of the research assessment system by the European Commission. All these initiatives agree on a few essential characteristics necessary for having a trustful assessment system: 

  • to be open and transparent by providing machine-readableunrestricted and reusable data and methods for calculating the metrics used in research assessment exercises, and 
  • to leave to the research community, instead of commercial players, the control and ownership of the crucial infrastructures and tools used to retrieve, use and analyse such data within research assessment systems. 

Thus, the leading guideline that can be abstracted is to follow Open Science practices even when assessing research and not only when performing research.

Some initiatives pushing for reforming the principles behind research assessment systems.
Some initiatives pushing for reforming the principles behind research assessment systems.

Introducing OpenCitations

Within this context, OpenCitations (full disclosure: I am one of its directors) plays an important role, acting as a key infrastructure component for global Open Science, and pushes for actively involving universities, scholarly libraries and publishers, infrastructures, governments and international organisations, research funders, developers, academic policy-makers, independent scholars and ordinary citizens. The mission of OpenCitations is to harvest and openly publish accurate and comprehensive metadata describing the world’s academic publications and the scholarly citations that link them, with the greatest possible global coverage and subject scope, encompassing both traditional and non-traditional publications, and with a breadth and depth that surpasses existing sources of such metadata, while maintaining the highest standards of accuracy and accompanying all its records with rich provenance information, and providing this information, both in human-readable form and in interoperable machine-readable Linked Open Data formats, under open licenses at zero cost and without restriction for third-party analysis and re-use.

For OpenCitations, open is the crucial value and the final purpose. It is the distinctive mark and founding principle that everything OpenCitations provide – data, services and software – is open and free and will always remain so. OpenCitations fully espouses the aims and vision of the UNESCO Recommendations on Open Science, complies with the FAIR data principles, and promotes and practises the Initiative for Open Citations recommendation that citation data, in particular, should be Structured, Separable, and Open. 

The most important collection of such open citation data is COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. The last release, dated August 2022, contains more than 1.36 billion citation links between more than 75 million bibliographic entities that can be accessed programmatically using its REST API, queried via the related SPARQL endpoint, and downloaded in full as dumps in different formats (CSV, JSON, and RDF). 

Collaborations between OpenCitations and other Open Science infrastructures and services.
Collaborations between OpenCitations and other Open Science infrastructures and services.

In addition to the publication of citation data, a considerable effort has been dedicated to collaborating with other Open Science infrastructures working in the scholarly ecosystem, as summarised in the figure above. Since 2020, OpenCitations has significantly benefited from the scholarly community that resulted from the 2019 selection by the Global Sustainability Coalition for Open Science Services (SCOSS) of OpenCitations as a scholarly infrastructure worthy of financial support. The community funding permitted the appointments of people dedicated to the administration, communication, community development, and maintenance and improvement of the OpenCitations software and the computational infrastructure on which it runs. In addition, OpenCitations started its involvement with OpenAIRE and the European Open Science Cloud (EOSC), and it is collaborating with other funded projects project such as RISIS2OutCiteOPTIMETA and B!SON.

While OpenCitations is currently providing a good set of citation data, which is already approaching parity with other commercial citation databases and that has already been used in a few studies for research purposes, there is still a margin for improvement. Currently, the citations included in the OpenCitations Indexes come mainly from Crossref data, one of the biggest open reference providers. However, Crossref does not cover all the publishers of DOI-based resources. Indeed, other DOI providers, in some cases, expose citation relations in their metadata, such as DataCite. In addition, DOI-based publications represent just a limited set of all the bibliographic entities published in the scholarly ecosystem. Other identifier schemas have been used to identify bibliographic entities – and, for some publications, there not exist identifiers at all!

Thus, to address these two issues, OpenCitations is working on expanding its coverage according to two different directions. On the one hand, OpenCitations is developing two new citation indexes of open references based on the holdings of DataCite and the National Institute of Health Open Citation Collection, which, together with COCI, will be cross-searchable through the Unifying OpenCitations REST API

On the other hand, OpenCitations has started working to create a new database entitled OpenCitations Meta, which will provide three major benefits. First, it will permit storing in-house bibliographic metadata for the citing and cited entities involved in all OpenCitations Indexes, including author identifiers using ORCID and VIAF identifier schemes where available. Second, it will provide better query performance than the present API system, which obtains bibliographic metadata on-the-fly by live API calls to external services, such as Crossref and DataCite APIs. Finally, it will permit indexing citations involving entities lacking DOIs, by providing them OpenCitations Meta Identifiers.

This last collection, combined with automatic tools for citation extraction from digital formats, is crucial for increasing the coverage of underrepresented disciplines and fields in bibliographic databases, such as SSH publications. One of the OpenCitations’ goals is to reduce this gap in citation coverage by setting up crowdsourcing workflows for ingesting missing citation data from the scholarly community (e.g. libraries and publishers). In the future, another contribution will be to set up tools for automatic extraction of citations that can also support small and local publishers, crucial assets for SSH research, that may find difficulties in carrying out citation extraction tasks on their own since using and maintaining a tool (or paying a company addressing those tasks on behalf of the publisher) requires extra costs beyond publishers’ finances.

To conclude: OpenCitations is one piece of a puzzle that is working to change existing scholarly practices to create an open and inclusive future for science and research in which the scholarly community owns and is responsible for its own data.

Disclaimer: This text was created during the TRIPLE Booksprint “The role of open metadata in the SSH scholarly communication” and is intended to be a contribution to the “Guidelines on the research data in the humanities” deliverable (8.5) of the TRIPLE project which will be made publicly available under the CC-BY 4.0 license.

Cite this article as: Silvio Peroni, "Transparency meets open citations," in OpenCitations blog, 27/09/2022, https://opencitations.hypotheses.org/2890.

OpenCitations Access Tokens: how they work and why they are important

Since its inauguration in 2010, OpenCitations has always granted free access to its services to users throughout the world, with no requirement for registration or sign-up. Programmatic access to OpenCitations data can be obtained either via our SPARQL endpoints and our REST APIs. In addition, OpenCitations data – available in CSV, Scholix, and RDF formats – can be downloaded from data dumps made periodically and stored on Figshare, so as to enable large-scale analyses using the whole content of the data sets, and also be obtained via our user-friendly text-based search and browsing interfaces.

One of OpenCitations’ priorities is (and will always be) to keep its data globally open and available at zero cost and without restriction for third-party analysis and re-use. As a matter of sustainability, OpenCitations relies on financial support from the scholarly community, which includes those institutions that use OpenCitations data. However, OpenCitations has not so far had in place a proper system to monitor its users, and the main evidence of the impact of OpenCitations in different academic fields and countries has been incompletely obtained from direct contacts with our members and donors across the world, our collaborations with international projects, and the interactions on our social platforms (Twitter and LinkedIn).

We would now like to institute a system that enables us to follow the usage and assess the impact of OpenCitations more reliably. For this purpose, we are now happy to announce the launch of the OpenCitations Access Token System for access to the OpenCitations data and services.

An OpenCitations Access Token is an opaque character string that anonymously identifies a unique user of the OpenCitations APIs. OpenCitations assigns an access token only if authorized to do so by each user, who can request a token by inserting his/her email address into the access form and clicking “Get token”. Upon submission of such a request, each user will automatically receive a personal access token by email. Users can save their personal access token and reuse it every time calling the APIs of OpenCitations, by passing it as a value for the key access-token in the header of each API call. 

Obtaining and using an OpenCitations Access Token is thus easy. It only requires a simple form request, and then the insertion of your personal token into the API call header when using OpenCitations REST APIs. OpenCitations will not store users’ email addresses or any personal information, so that the users’ privacy will be totally safeguarded. The token system just provides a simple mechanism for identifying unique users, for which the use of IP addresses is insufficient.

Obtaining an OpenCitations Access Token will take the user only a few seconds and needs to happen only once. You can request your OpenCitations Access Token here

https://opencitations.net/accesstoken 

Use of an OpenCitations Access Token is not compulsory. However, token use will help OpenCitations incredibly, by enabling us to monitor the number of the unique users accessing our data and services, providing objective anonymized evidence of the number of institutions and researchers accessing our data either occasionally or on a regular basis, which we can then employ to demonstrate the usefulness of OpenCitations in the research environment. While the token system will initially be employed just for API calls (the most used service we offer), it will subsequently be extended to our other forms of data access.

OpenCitations exists for the people that use its data for research purposes every day, and thanks to their support. This is why obtaining precise knowledge of how many researchers and institutions are accessing our services is essential to us, since it will enable us to present the uniqueness and value of OpenCitations to new communities of stakeholders, and thus to make it possible to enlarge the already enthusiastic and diverse group of people and institutions supporting and using our Open Science Infrastructure.

To summarize: Getting and using an OpenCitations Access Token is voluntary, easy, and does not cost you anything. However, it will help OpenCitations a great deal. Please get your own token now, and use it next time you access OpenCitations. Thank you very much!

Cite this article as: Chiara Di Giambattista, "OpenCitations Access Tokens: how they work and why they are important," in OpenCitations blog, 02/09/2022, https://opencitations.hypotheses.org/1535.

 

Additional 48 million citations in COCI, including references from IEEE 

We announce the August 2022 release of COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations, which is based on open references to works with DOIs within the Crossref dump dated August 2022. This new release extends COCI with more than 48 million additional citations, giving a total number of more than 1.36 billion DOI-to-DOI citation links. 

This release includes citations from the articles published over the last four years by IEEE, whose bibliographic references were opened in June 2022. 

A fundamental role in pushing the commercial publishers to open their citation data was played by Crossref’s recent announcement to change its reference distribution policy, by making all its metadata open.  

Besides IEEE, COCI already includes the citation data derived from Elsevier (open via Crossref since December 2020) and from the last articles published by the American Chemical Society (whose references were opened in February 2021) 

You can find more information about COCI in our open-access article  

Ivan Heibi, Silvio Peroni & David Shotton (2019). Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. Scientometrics, 121 (2): 1213-1228. DOI: https://doi.org/10.1007/s11192-019-03217-6    

Finally, just a reminder that the bibliographic and citation data in COCI:  

    • can be queried using the OpenCitations Indexes SPARQL endpoint;  
    • can be retrieved by using the COCI REST API;  
    • can be searched by using the OpenCitations Indexes Search Interface;  
    • are also available as dumps on Figshare in CSV, N-Triples, and Scholix; and  
    • can be freely re-used for any purpose.

      Cite this article as: Chiara Di Giambattista, "Additional 48 million citations in COCI, including references from IEEE ," in OpenCitations blog, 31/08/2022, https://opencitations.hypotheses.org/2732.
Search OpenEdition Search

You will be redirected to OpenEdition Search