The International Advisory Board for OpenCitations is one of the four organizational bodies which define OpenCitations’s governance structure (Director; Council; Advisory Board; Management Team). The Board is a committee of the Research Centre for Open Scholarly Metadata at the University of Bologna, and is made up of Open Infrastrucutures and Open Science experts elected by the Council of OpenCitations among a plethora of candidates. The role of the International Advisory Board is to give strategic advice, and guide OpenCitations development by providing an outsider’s point of view on our activities.
The first structure of the Advisory Board was defined by the election, back in 2020, of eight members selected by the Council within the main scholarly stakeholder communities of relevance to OpenCitations (librarians, scientometricians, academics, publishers, funders, data service providers, etc.). These members have served OpenCitations for the last four years, and have accompanied and addressed some of the major changes in OpenCitations. Indeed, in the previous four years, OpenCitations has grown in team composition, community collaborations and data provision. In these years, our coverage approached that of the major databases, we have revolutionized our ingestion workflow to manage a growing number of data sources, and thanks to the involvement of some members of the Advisory Board in two internal working groups, we have been working on the stakeholder identification and governance evolution.
It was right in the Governance Evolution working group that, in the last year, the need to renovate the International Advisory Board composition has emerged, intending to make the renewal process recurring every three years after the pilot experience with the first International Advisory Board. As an initial action resulting from the Governance Evolution Working group, we held the elections of the members of the new International Board on OpenCitations in the first week of December. The voting council selected eight members among ten candidates who strongly committed to open scholarship.
We are now happy to announce the new members of the International Advisory Board, and while thanking them for their enthusiastic involvement in the Board, we also want to express our gratitude to the former Board members who have accompanied us in the last four years. The new International Advisory Board will also welcome one representative from OpenCitations’ Strategic Member (Scientific Interest Group of the French National Fund for Open Science) as a non-elected Board member.
The new members of the International Board for OpenCitations are:
Marcel R. Ackermann (Team Lead of dblp Computer Science Bibliography)
Maria Gould (Product Director at DataCite and Director of ROR)
Catriona MacCallum (Director of Open Science, Hindawi Open Access Publisher)
Philippe Mongeon (Associate Professor and Chair of the Department of Information Science at University of Montreal)
Cameron Neylon (Professor of Research Communications, Centre for Culture and Technology, Curtin University)
Iratxe Puebla(Director of Make Data Count)
Dominika Tkaczyk (Director of Data Science at CrossRef)
Didier Torny (on behalf of the French Open Science Committee)
Ludo Waltman (Professor of Quantitative Science Studies and Deputy Director, Centre for Science and Technology Studies, Leiden University)
The International Advisory Board’s activities will start early in 2025, picking up the threads of the previous Board’s work. We look forward to working with relevant actors in the Open Science community.
The first month of the new year has almost come to an end, and we at OpenCitations have dedicated these weeks after the holiday season to retrace the progress we reached as an open infrastructure throughout 2023, an activity that has become a tradition in the past few years. Looking back to the achievements has proven to be a great practice for our team not only to honour and celebrate the success, but also to visualize the points to focus on to best serve our community when planning future activities. During 2023 we have reached numerous milestones, which can be summarized under the two main topics: technical developments and new connections.
Technical Developments
1. New sources
At the end of 2022, we had enriched our collection with two more sources, PubMed and DataCite, which were added to Crossref. During 2023 we completed the integration between OpenCitations and the OpenAIRE Graph as concerns open bibliographic and citation data, and in December we included the DOI-to-DOI citation links derived from the Japan Link Center (JaLC). As a result, by the end of 2023, OpenCitations counted almost 2 BILLION open citations available in its collection.
2. A brand-new workflow for an easier consultation
The progressive integration of new sources is one of OpenCitations’ main objectives. However, the recent source increase made it necessary to design a new ingestion workflow to facilitate their management. In November 2023 OpenCitations launched the new workflow, designed to produce just two comprehensive collections: OpenCitations Index, collecting open citation data, and OpenCitations Meta, containing open bibliographic metadata of citing and cited entities involved in citations. This evolution is making it simpler for OpenCitations users to consult the data maintained by our infrastructure without reducing the information content and accuracy.
New connections
3. A great event for the community
In October 2023, OpenCitations hosted the Workshop on Open Citations and Open Scholarly Metadata 2023 (WOOC 2023) in the historical location of the University of Bologna. After two online editions, it was exciting to meet in person the researchers, policy makers and service providers who in the past few years have collaborated with OpenCitations in many ways. For two days, almost 80 people from all around the world gathered to discuss the theme “The value of open scholarly metadata for Research Assessment purposes”. The event gave us a picture of a young and enthusiastic research community, aiming to collaborate among different countries to spread the message “Open Science is Better Science”. If you couldn’t attend the workshop, but are interested in the theme, you can find all the presentations from WOOC 2023 on Zenodo. Stay tuned on the WOOC X account and website, we will announce the 2025 edition later in 2024!
4. New and old partnerships
OpenCitations’ main activities, besides the technical development, are tied to its participation in R&D and funded projects. Since 2019, we have started building a network of collaborations all around the world with institutions which share the same Open Science values. In 2023, our involvement in the SCOSS 2nd funding cycle has come to an end. In the last four years SCOSS has worked as an international showcase for OpenCitations, making it possible for us to get known by many institutions on an international level. We are now proud to keep on collaborating with SCOSS within the SCOSS Family network, a community of practice gathering all the OS infrastructure involved in the SCOSS funding cycles. During 2023, the SCOSS Family was involved in four internal sprints dedicated to planning four actions – collectively advocating the importance of open infrastructure, communications kit for ‘the post-SCOSS phase’, collective self-assessment towards POSI, and map open science infrastructure ecosystem including SCOSS infrastructures – and the initial outcomes were presented in the Scoss Family Public Webinar 2023. The SCOSS family will continue its activities throughout 2024.
In 2023, we also celebrated the update of the Principles of Open Scholarly Infrastructure (POSI), as a result of the work conducted since 2022 with the “POSI-Adopters” group to make the principles clearer and more inclusive. In June, the OpenAIRE-Nexus project, started back in 2021, formally came to an end, with the release of the index of citations from OpenAIRE. OpenCitations is part of the OpenAIRE Catalogue, which also links to the exhaustive and engaging materials we produced within the project. However, we continue to meet the familiar and friendly faces from the OpenAIRE team in the meetings of GraspOS, a project which started in January 2023 involving several European Open Science institutions, services and infrastructures. In GraspOS, OpenCitations is involved as one of the Data Asset Sources, which provides both the raw and the structured data that can be used to directly support OS-aware RRA processes or to calculate indicators or collect evidence that can be of value in these processes. The researchers from OpenCitations and the University of Bologna are contributing to GraspOS by developing a tool for the extraction of citation data from PDFs, enriched with additional information on the reasons for citing (because the citing entity reuses a method described in the cited entity, for agreeing with the claims of the cited entity, etc.) through the implementation of machine-learning systems. Also, together with GraspOS, OpenAIRE, and many other Coalition members, the OpenCitations personnel at the University of Bologna is among the participants of the CoARA Working Group Towards Open Infrastructures for Responsible Research Assessment (OI4RRA), to work on enabling the research community to be in full control of the data and indicators it relies on in assessments. OpenCitations is always seeking interest from new institutions, and we are proud that our visibility is boasted by some reliable catalogues including the OpenAIRE catalogue, the EOSC Marketplace and, from 2023, also the JISC Catalogue. And soon you’ll find us in a new tool – stay tuned to find out!
5. Our way towards sustainability
OpenCitations is increasingly gaining a prominent role in the community, establishing itself as a key player in changing research evaluation practices towards greater openness and equity. In recent years, OpenCitations has grown a lot both technically and in popularity, and for this we want to thank the number of members and donors who, even in the last year, have decided to support us and believe in our infrastructure. During 2023, our two Internal Working Groups “Governance Evolution” and “Community Building”, thanks to the participation of some members of the International Advisory Board for OpenCitations, reflected on the sustainability of OpenCitations, discussing strategies both at the management and communication level to ensure the integration of new actors into its subsistence. The groups will continue to meet throughout 2024, and some of the results will be released in the coming months. However, there is still a long way to go towards sustainability, and OpenCitations’ existence still relies on community funding, which is why OpenCitations still needs support: if you are looking for more information on why and how to support OpenCitations, you can read our recent call for action here. Thank you!
6. Enlarging our team
It is thanks to the support received that OpenCitations was able to develop its yearly activities, and we are happy to announce that in 2023 a new software developer, Elia Rizzetto, and a new PhD student, Francesca Cappelli, have joined our team to make this progress possible. The OpenCitations team now has a total of 12 members, including research fellows, PhD students, MA students, a Reseach Manager, an Assistant Professor and OpenCitations’ Directors. The number is expected to grow soon, as OpenCitations has recently released a call for the hiring of a System Administrator for technical infrastructure management.
As can be seen from this brief (and detailed) report of some of the developments that have marked our 2023, OpenCitations is increasingly emerging as a community-based and community-guided infrastructure organization, in which there is an indissoluble link between technical progress and the network of people who guide the planning, work on implementation, and use or promote our services. Throughout 2024, OpenCitations aims to focus even more on harmonizing the relationship between the technical infrastructure and the community, developing more intuitive ways of accessing services and communications, and continuing to weave relationships with new research groups. We look forward to seeing what the year ahead will bring with it, and we hope you will continue to be with us.
2023 has arrived, and it’s the time to put aside the old agendas and to fill the freshly printed organizers with new tasks. However, it is also worth looking back over the achievements of the past year.
So, before consigning 2022 to the archives, we want to take a moment to recall and celebrate all that OpenCitations has achieved over the past year, in terms of technical developments, community building, and our own internal organization.
Clarifying our objectives
Since its foundation, OpenCitations has been presented to the scholarly community as a infrastructure organization guided by the values of openness, equity and accessibility. In 2022, we publicly clarified our mission, our aims and our unique points by publishing these three documents:
In addition, we continued to monitor OpenCitations’ compliance with the Principles of Open Scholarly Infrastructure. Because one area in which we were still falling short involved stakeholder involvement in OpenCitations’ governance, in 2022 OpenCitations started reflecting on its organizational structure by creating a Governance Evolution Working Group. In addition, we explored the further development of our stakeholder community by establishing a parallel Community Building Working Group. These two small working groups, involving members of the OpenCitations International Advisory Board, will continue to meet throughout 2023 to develop strategic plans concerning OpenCitations open governance structure and outreach.
More people, more tools
In 2022, OpenCitations welcomed two new full-time developers to its team: Arianna Moretti as software developer, and Ivan Heibi as the person responsible for OpenCitations’ technical infrastructure. As of January 2023, the OpenCitations team consists of our two directors Silvio Peroni and David Shotton, our administrator and research manager Claudio Fabbri, myself as our communications director and community development manager, and our four developers: Giuseppe Grieco (software and systems developer), Arcangelo Massari (developer of OpenCitations Meta), Arianna Moretti (software and indexes developer), and Ivan Heibi (technical infrastructure manager). This team meets weekly at the Research Centre for Open Scholarly Metadata within the University of Bologna.
This enlarged technical workforce has been crucial in the development of the two new OpenCitations indexes, DOCI (citations from DataCite) and POCI (citations from PubMed), and our new database of bibliographic metadata, OpenCitations Meta, all of which were released in December 2022:
DOCI, the OpenCitations Index of DataCite open DOI-to-DOI citations, is an RDF dataset containing details of all the citations that are specified in the most recent dump of DataCite metadata. The citations available in DOCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. Its first release (December 2022) contains information on 169,822,752 citations from 358,277 bibliographic resources.
POCI, the OpenCitations Index of PubMed open PMID-to-PMID citations, is an RDF dataset containing details of all the citations from publications bearing PubMed Identifiers (PMIDs) to other PMID-identified publications, harvested from the National Institutes of Health Open Citations Collection (NIH-OCC). As for DOCI, the citations available in POCI are treated as first-class data entities, with accompanying properties including the citations timespan, modelled according to the OpenCitations Data Model. Its first release contains 717,654,703 citations from 26,024,862 bibliographic resources.
OpenCitations Meta database stores and delivers bibliographic metadata for all the publications involved as citing or cited publications in the citations recorded in the OpenCitations Indexes. For each publication, the metadata exposed by OpenCitations Meta includes the publication’s title, type, venue (e.g. journal name), volume number, issue number, page numbers, publication date, and identifiers such as Digital Object Identifiers (DOIs) and PubMed Identifiers (PMIDs). In addition, OpenCitations Meta includes details of the main actors involved in the document’s publication, i.e., the names of the authors, editors, and publishers, each with its own additional metadata and identifier (e.g. ORCID). Its first release (December 2022) contains information on 87,321,593 bibliographic entities, 277,750,235 authors and 2,359,301 editors (counted by their roles, without disambiguating individuals), 710,226 publication venues, and 17,268 publishers.
As can be seen on OpenCitations roadmap, we have many tasks already scheduled for 2023, and for this reason OpenCitations team needs to expand further: we now seek applicants for a new Research Fellow position to work on the further development of the OpenCitations technical infrastructure from April 2023.
Our role in the community
Throughout 2022, the OpenCitations team has participated both online and in person in a number of international events, presenting details of OpenCitations to potential stakeholders and to Open Science experts, and connecting with similar organizations.
In October, OpenCitations organized our third workshop, the Workshop on Open Citations and Open Scholarly Metadata 2023, as a three-hour online event, to gather together and connect researchers, publishers and policy makers interested in the creation, reuse and improvement of open citation data and open scholarly metadata. The invited speakers provided a broad overview of the current state and future development of open bibliographic metadata and the scholarly record, and the role of OpenCitations in research assessment.
We have participated in two major European research and development project: In the context of the EC-funded project RISIS2we released the new database OpenCitations Meta and the index DOCI, while within the OpenAIRE-Nexus project network we continue to provide open bibliographic citations as part of the open data components of OpenAIRE and the European Open Science Cloud. During 2022, the OpenAIRE team supported our activities in many areas, with a particular attention to communication and dissemination, as we explained in our blog post OpenCitations and EC funding: OpenAIRE Nexus and RISIS2.
The end of 2022 brought with it two new exciting collaborations that will be developed during the coming year: the inclusion of OpenCitations in the Jisc Collections, thereby permitting UK institutions to support OpenCitations through JISC; and our involvement in the GraspOS project (starting date January 1, 2023) as an open and trusted federated infrastructure for next-generation research metrics and indicators.
Continuing support
At OpenCitations we are deeply grateful to all our members and donors that have supported us financially so far, thus making it possible to enlarge our team, maintain our technical infrastructure, and work on our new data releases.
Much of our success has been due to our selection as an infrastructure worthy for inclusion in the SCOSS second pledging round in 2019. SCOSS helped us get in touch with a wider community, and enabled OpenCitations to have an increasingly significant role in the research environment. OpenCitations is now part of the SCOSS family, the community of all SCOSS-supported infrastructures, that provides a framework of mutual support for the the Infrastructures during their growth and a safety net should troubles occur along the path.
However, OpenCitations still has a long way to go along the path from being a sustainable open infrastructure to becoming a sustained open infrastructure providing reliable and invaluable services to our global stakeholder community. There are still many long-planned activities that we hope to initiate in the near future, given sufficient resources. This is why OpenCitations requires ongoing support from the scholarly community as our three-year period of SCOSS sponsorship draws to a close. All the information you may require to start helping us financially is available on the OpenCitations website: https://opencitations.net/membership.
2023 has just started. With your support, you can help us in making it an impactful year for OpenCitations and the Open Science community as a whole.
The Wikipedia entry for OpenCitations is woefully out of date, inaccurate and brief. As Directors of OpenCitations, Silvio and I are unable to improve this situation because of Wikipedia’s proper conflict-of-interest restriction on self-promotion.
OpenCitations is actively seeking greater involvement from members of the global academic community, as explained in our Mission Statement. One way in which such individuals, particularly those who are both existing Wikipedia editors and already know about OpenCitations, can help OpenCitations, while at the same time supporting Wikipedia in its quest for accurate information, is by revising and expanding the present Wikipedia entry on OpenCitations to reflect the infrastructure’s current activities and data holdings, while maintaining perspective and a neutral point of view. This will increase the availability of reliable knowledge about OpenCitations and its place in the ecosystem of Open Science infrastructures.
Background information to assist in this task is available in the cannonical paper describing OpenCitations:
Silvio Peroni and David Shotton (2020). OpenCitations, an infrastructure organization for open scholarship. Quantitative Science Studies 1: 428-444. https://doi.org/10.1162/qss_a_00023;
in previous posts in the OpenCitations Blog and in the documents to which those posts link, particularly in:
OpenCitations has happily joined the open-source social media platform joinmastodon.org.
Mastodon is “a free and open-source software developed by a non-profit organization”, with the aim of favouring interoperability and bringing social media interaction “back in the hands of the people”.
We look forward to recreating there our wide network of connections, and getting in touch with new people, projects and institutions in a different virtual environment.
In the scholarly ecosystem, a bibliographic citation is a conceptual directional link from a citing entity to a cited entity, used to acknowledge or ascribe credit for the contribution made by the author(s) of the cited entity. Citations are one of the core elements of scholarly communication. They enable the integration of our independent research endeavours into a global graph of relationships that can be used, for instance, to analyse how scholarly knowledge develops over time, assess scholars’ influence, and make wise decisions about research investment.
A copyrighted fact
However, as citation data, i.e. pieces of factual information aiming at identifying entities and relationships among them, are of great value to the scholarly community, it has been a “scandal” that they have not been recognised as part of the commons. Indeed, only recently we have seen some efforts – such as the Initiative for Open Citations (I4OC) – that have tried to change the behind-the-paywall status quo enforced by the companies controlling the major citation indexes used worldwide, convincing scholarly publishers to support the unrestricted availability of scholarly citation data by publishing them in suitable open infrastructures, such as Crossref and DataCite.
Of course, as for many other kinds of data, putting bibliographic and citation data behind a paywall is a thread to enabling the full reproducibility of research studies based on them (e.g. in bibliometrics, scientometrics, and science of science domains), even when such studies are published in open access articles. For instance, the results of a recent open access article published on Digital Scholarship in the Humanities, which aimed to analyse the citation behaviour of Digital Humanities (DH) research across different proprietary and open citation databases, are not fully reproducible since the majority of the databases used – namely Scopus, Web of Science, and Dimensions – do not make their bibliographic and citation data openly available.
In addition, the coverage of publications and related citations in specific disciplines, particularly those within the Social Sciences and the Humanities (SSH), is inadequatecompared tootherfields. Usually, this is due to the limited availability of born-digital publications accompanied by a wide variety of publication languages, publication types (e.g. monographs), and complex referencing practices that may limit their automatic processing and citation extraction. As a side effect, such a partial coverage may result in a considerable bias when analysing SSH disciplines compared to STEM disciplines that usually have better coverage in existing citation databases.
Reforming research assessment
All these scenarios have at least another negative effect on the area strictly concerned with the research assessment, which often uses quantitative metrics based on citation data to evaluate articles, people, and institutions. Indeed, the unavailability and partial coverage of bibliographic and citation data create an artificial barrier to the transparency of the processes used to decide the careers of scholars in terms of research, funding, and promotions.
to be open and transparent by providing machine-readable, unrestricted and reusable data and methods for calculating the metrics used in research assessment exercises, and
to leave to the research community, instead of commercial players, the control and ownership of the crucial infrastructures and tools used to retrieve, use and analyse such data within research assessment systems.
Thus, the leading guideline that can be abstracted is to follow Open Science practices even when assessing research and not only when performing research.
Some initiatives pushing for reforming the principles behind research assessment systems.
Introducing OpenCitations
Within this context, OpenCitations (full disclosure: I am one of its directors) plays an important role, acting as a key infrastructure component for global Open Science, and pushes for actively involving universities, scholarly libraries and publishers, infrastructures, governments and international organisations, research funders, developers, academic policy-makers, independent scholars and ordinary citizens. The mission of OpenCitations is to harvest and openly publish accurate and comprehensive metadata describing the world’s academic publications and the scholarly citations that link them, with the greatest possible global coverage and subject scope, encompassing both traditional and non-traditional publications, and with a breadth and depth that surpasses existing sources of such metadata, while maintaining the highest standards of accuracy and accompanying all its records with rich provenance information, and providing this information, both in human-readable form and in interoperable machine-readable Linked Open Data formats, under open licenses at zero cost and without restriction for third-party analysis and re-use.
For OpenCitations, open is the crucial value and the final purpose. It is the distinctive mark and founding principle that everything OpenCitations provide – data, services and software – is open and free and will always remain so. OpenCitations fully espouses the aims and vision of the UNESCO Recommendations on Open Science, complies with the FAIR data principles, and promotes and practises the Initiative for Open Citations recommendation that citation data, in particular, should be Structured, Separable, and Open.
The most important collection of such open citation data is COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. The last release, dated August 2022, contains more than 1.36 billion citation links between more than 75 million bibliographic entities that can be accessed programmatically using its REST API, queried via the related SPARQL endpoint, and downloaded in full as dumps in different formats (CSV, JSON, and RDF).
Collaborations between OpenCitations and other Open Science infrastructures and services.
In addition to the publication of citation data, a considerable effort has been dedicated to collaborating with other Open Science infrastructures working in the scholarly ecosystem, as summarised in the figure above. Since 2020, OpenCitations has significantly benefited from the scholarly community that resulted from the 2019 selection by the Global Sustainability Coalition for Open Science Services (SCOSS) of OpenCitations as a scholarly infrastructure worthy of financial support. The community funding permitted the appointments of people dedicated to the administration, communication, community development, and maintenance and improvement of the OpenCitations software and the computational infrastructure on which it runs. In addition, OpenCitations started its involvement with OpenAIRE and the European Open Science Cloud (EOSC), and it is collaborating with other funded projects project such as RISIS2, OutCite, OPTIMETA and B!SON.
While OpenCitations is currently providing a good set of citation data, which is already approaching parity with other commercial citation databases and that has already been used in a few studies for research purposes, there is still a margin for improvement. Currently, the citations included in the OpenCitations Indexes come mainly from Crossref data, one of the biggest open reference providers. However, Crossref does not cover all the publishers of DOI-based resources. Indeed, other DOI providers, in some cases, expose citation relations in their metadata, such as DataCite. In addition, DOI-based publications represent just a limited set of all the bibliographic entities published in the scholarly ecosystem. Other identifier schemas have been used to identify bibliographic entities – and, for some publications, there not exist identifiers at all!
Thus, to address these two issues, OpenCitations is working on expanding its coverage according to two different directions. On the one hand, OpenCitations is developing two new citation indexes of open references based on the holdings of DataCite and the National Institute of Health Open Citation Collection, which, together with COCI, will be cross-searchable through the Unifying OpenCitations REST API.
On the other hand, OpenCitations has started working to create a new database entitled OpenCitations Meta, which will provide three major benefits. First, it will permit storing in-house bibliographic metadata for the citing and cited entities involved in all OpenCitations Indexes, including author identifiers using ORCID and VIAF identifier schemes where available. Second, it will provide better query performance than the present API system, which obtains bibliographic metadata on-the-fly by live API calls to external services, such as Crossref and DataCite APIs. Finally, it will permit indexing citations involving entities lacking DOIs, by providing them OpenCitations Meta Identifiers.
This last collection, combined with automatic tools for citation extraction from digital formats, is crucial for increasing the coverage of underrepresented disciplines and fields in bibliographic databases, such as SSH publications. One of the OpenCitations’ goals is to reduce this gap in citation coverage by setting up crowdsourcing workflows for ingesting missing citation data from the scholarly community (e.g. libraries and publishers). In the future, another contribution will be to set up tools for automatic extraction of citations that can also support small and local publishers, crucial assets for SSH research, that may find difficulties in carrying out citation extraction tasks on their own since using and maintaining a tool (or paying a company addressing those tasks on behalf of the publisher) requires extra costs beyond publishers’ finances.
To conclude: OpenCitations is one piece of a puzzle that is working to change existing scholarly practices to create an open and inclusive future for science and research in which the scholarly community owns and is responsible for its own data.
Posted on August 10th 2022 by Chiara Di Giambattista
More than a year ago, Ginny Hendricks, Director of Member & Community Outreach for Crossref, and a valued member of the OpenCitations International Advisory Board, published on the Crossref blog the post “The road ahead: our strategy through 2025”. In order to describe all Crossref’s principles and activities, Ginny presented the Crossref strategic planning framework as a diagram summarizing Crossref’s statements, key messages and truths. The clarity and immediacy of the diagram were such that we adapted it to present OpenCitations’ own statements and goals. The resulting poster “OpenCitations – what does the future hold?” was presented by our Director David Shotton at the OASPA2021 conference, and can be found in this blog post.
Although the poster offered a wide overview of OpenCitations values, unique traits, benefits and plans, it differed slightly from Ginny’s original diagram, in particular because it lacked a “Mission Statement”, scattering the relevant information within the “Values” and “Principles” boxes. Indeed, at that time (September 2021), we didn’t have a clearly defined Mission Statement.
Nevertheless, the creation of that poster was crucial in helping us start to articulate more clearly the purpose and meaning of OpenCitations. As David underlined in his post “From little acorns…a retrospective on OpenCitations”, since 2018 OpenCitations activities have progressively increased and, with them, the number of related journal articles, conference papers and technical definitions. OpenCitations’ involvement in international networks and collaborations (such as SCOSS and the OpenAIRE-Nexus project), together with our need of identifying and reaching out to new stakeholders to assure OpenCitations’ development and sustainability, has made it necessary to publicly define OpenCitations’ mission, unique strengths and next developmental steps.
After numerous revisions, aided by wise advice from members of the OpenCitations Advisory Board members, we’re now happy to publish the following three OpenCitations documents:
which together provide a summary of why we exist and where we are heading.
We are particularly proud of the definition of OpenCitations’ primary mission, namely
“to harvest and openly publish accurate and comprehensive metadata describing the world’s academic publications and the scholarly citations that link them, and to preserve ongoing access to this information by secure archiving.”
The Mission Statement also presents brief descriptions of the OpenCitations context, our vision, our value proposition and our relationship with the community and stakeholders.
The Uniqueness of OpenCitations provides the answer to the question ‘Why choose to use OpenCitations?’, and is a detailed presentation of OpenCitations’ benefits.
OpenCitations – Present Status and Future Plans summarizes OpenCitations’ ongoing activities, that can be quickly visualized on our public roadmap. It also introduces the OpenCitations Working Groups, served by the members of the OpenCitations International Advisory Board, which are currently working on the themes of governance evolution and community building, with the common purpose of driving OpenCitations along the path from being a ‘sustainable infrastructure’ (in POSI terms) to being an enduring community led andfinancially sustained infrastructure.
In fulfilling our mission and reaching our goals, the support and vital interest of our community members is fundamental. We request that you, as a member of our community, provide us with feedback on these documents and the ideas they contain, or indeed to ask for clarifications, to help us improving our mission and our communications to explain it. You can reach us here: contact@opencitations.net.
2021 is just behind us. Since January is “the Monday of the months”, as F. Scott Fitzgerald once wrote[1], it’s a good time to take stock of what happened at OpenCitations during the past year.
Among the numerous events, achievements and challenges that 2021 brought with it, we want to highlight five milestones which make us proud to look back:
1. We extended our coverage to well over one billion citations
During 2021, OpenCitations’ largest index COCI (the OpenCitations Index of Crossref open DOI-to-DOI citations) was able to include for the first time the citation links involving references that had been opened at Crossref by Elsevier and the American Chemical Society, thereby greatly expanding its coverage. The last release of COCI (November 2021) is based on open references to works with DOIs within the Crossref dump dated October 2021, and, as a result, COCI now contains information on more than 1.23 billion citations involving almost 70 million publications. A recent analysis by Alberto Martìn-Martìn (Facultad de Comunicación y Documentación, Universidad de Granada, Spain), published on the OpenCitations Blog in October, shows that the citation coverage provided by OpenCitations is approaching parity with that of the leading commercial citation indexes, Web of Science and Scopus, offering a viable alternative upon which to base open and reproducible metrics of academic performance.
2. OpenCitations team grew
Last summer, we appointed Claudio Fabbri as our Administrator and Research Manager to take responsibility for the day-to-day administrative and financial activities of OpenCitations; Chiara Di Giambattista as Communications Director and Community Development Manager to take care of all communications and community interactions made on behalf of OpenCitations; and Giuseppe Grieco as our new Software and Systems Developer to take charge of technical development related to the OpenCitations services.
Thanks to the support from the OpenAIRE Nexus project, the team has also recently welcomed Arcangelo Massari as our new Software Developer to take care of the development of the new database OpenCitations Meta. We anticipate further appointments during 2022!
Our International Advisory Board met in November, and we thank its members for the valuable advice they provided. The Board will meet again later this month.
3. We participated in many international meetings
During the past year, OpenCitations’ directors Silvio Peroni and David Shotton took part in numerous international conferences, webinars and workshops, including the LIBER Annual Conference 2021, the OS Fair 2021, OASPA 2021 and FORCE2021. These provided excellent opportunities to describe and promote OpenCitations, to reach out to new potential stakeholders, and to discuss with other experts the main themes of our activities and plans as they relate to Open Science.
The year ended with a bang, with the announcement during the closing session of FORCE2021 that the 2021 Open Publishing Award for Open Data had been awarded to OpenCitations.
4. We received a world of support
In 2021, thanks to our involvement in the SCOSS funding campaign and to our commitment to reaching out to the libraries and universities potentially interested in OpenCitations, we gathereda wide international community of stakeholders and supporters around us. We are deeply thankful to the 6 consortia and 56 institutions across the globe which are now supporting us financially, thus making it possible for us to enhance our services and expand our team. You can find the full list of our supporters on the OpenCitations website and in this recent Thank You video:
Additionally, in January 2021, we started our involvement in the EC-funded OpenAIRE Nexus project, bringing us into closer collaboration with our European colleagues and infrastructures, including OpenAIRE. The main aim of the project is to create a framework of services for assisting in publishing research, monitoring its impact, helping promote its discovery, and integrating it into the European Open Science Cloud (EOSC) “for the benefit of the open science community worldwide”. In OpenCitations, we’re thrilled to be part of this collaborative project by providing open bibliographic citations as part of the open data components of OpenAIRE and the EOSC.
5. We set the stage for future developments
Thanks to the research grants and the support and endorsement we have received from the international scholarly community, we are now working on a variety of new services, thus setting our goals for the coming years. In particular, we want to enhance OpenCitations partnerships and dialogue with the scholarly community; to collaborate with colleagues to develop new services that will expand our citation coverage, including new OpenCitations indexes of NIH-OCC, of DataCite and of other sources of open references, that will all be searchable through a single API; and to create OpenCitations Meta, our new database that will hold comprehensive bibliographic metadata of the publications involved in our indexes citations, thereby enabling faster query responses and the ability to host citations involving publications lacking DOIs[2].
A few months ago, I was invited to have a talk at the European Computer Science Symposium on an aspect of my research I particularly care about, that of open citations. What I tried to address during the presentation concerned the current status of open citation availability in a particular domain, Informatics, by using two open datasets, i.e. DBLP for gathering bibliographic metadata about relevant publications and OpenCitations’ COCI for identifying citations where such publications are involved. This post briefly introduces the preliminaries and results obtained from the material used to prepare the talk.
Open citations and where to find them
A citation is a conceptual directional link between a citing entity and a cited entity which is defined by means of specific textual devices contained in the text of the citing entity, e.g. a bibliographic reference denoted by an in-text reference pointer (e.g. “[3]” or “(Doe et al, 2021)”). While reasons for citing may vary, citations are used in academia for acknowledging others’ work and enabling building trails of relations defining how science evolves in time.
The data needed to describe a citation should include, at least, a representation of such a conceptual link and the basic bibliographic metadata to identify the citing and cited entities, i.e. those typically used for defining bibliographic references such as authors’ names, year of publication, the title of the work, venue of publication, pages, identifiers, etc. We say that a citation is open when these citation data are in the public domain and can be retrieved freely (via the HTTP protocol) in a structured and machine-readable format (e.g. JSON or RDF) without accessing the source of citing article defining it, which, potentially, could be behind a paywall.
OpenCitations [full disclosure: I am one of its directors] is one of the founders of the Initiative for Open Citations (I4OC) and one of the open scholarly infrastructures providing open citation data through several channels (REST APIs, SPARQL endpoints, Web interfaces, full dumps in different formats). As of 31 December 2021, it makes available more than 1.2 billion open DOI-to-DOI citation links between more than 69.5 million bibliographic resources, which are mainly journal articles but also include books, book chapters, datasets, and other DOI-identified resources. The entities involved in such citations come from different domains, spanning from Medicine articles to Humanities publications, and have recently approached parity with those included in well-known proprietary services such as Web of Science and Scopus.
What about Informatics
Such a huge mass of open citations available enables us to analyse citation coverage in different scholarly disciplines, e.g. to understand which publishers contributed to the availability of open citation data in a discipline and to check what are the citation trails between different disciplines. However, to compute such citation coverage, we need to have some information that allows us to identify when a particular bibliographic resource involved in a citation belongs to the particular discipline we want to analyse. We can use information about the subject categories of publications (e.g. that of Web of Science), if included in citation indexes, to identify the discipline(s) of a given bibliographic resource. Unluckily, OpenCitations does not provide this information and, as such, we need to rely on external repositories for gathering subject categories of publications, e.g. collections of bibliographic metadata of disciplinary publications.
In the context of Informatics, there is at least one well-known resource gathering and exposing bibliographic metadata of a large part of Computer Science publications, i.e. DBLP. As of 30 December 2021, DBLP contains more than 5.9 million publications published in 1,781 journals and in the proceedings of 5,621 conferences, involving more than 2.9 million authors that are manually curated (and disambiguated) by the DBLP team.
DBLP can be used as a proxy to understand if a particular publication belongs to the Computer Science subject category. Through it, it is possible to understand how many citations in OpenCitations involve Computer Science publications by comparing the DOIs of citing and cited entities with those available in DBLP. In particular, using the OpenCitations’ COCI September 2021 dump and DBLP October dump, I found that more than 80 million citations in COCI involved at least one of the 4,637,865 entities in DBLP (considering only journal articles, conference proceedings papers, books and book chapters). As shown in Figure 1, only 39% of these citations are between citing and cited entities both included in DBLP, while the rest of them either come from or go to publications not listed in DBLP – that, potentially, could not be Computer Science publications.
Figure 1. A Venn diagram showing how many citations involving Computer Science publications (obtained from DBLP) are included in OpenCitations.
Additional information about the publishers of such DBLP entities, retrieved by querying the Crossref API and the DataCite API with entities’ DOIs, are shown in Table 1. IEEE is the publisher with the biggest number of entities of those considered for this study, and its entities are involved in more than 18.9 million incoming and 21.5 million outgoing citations. The other bigger publishers, in terms of entities and citations, are Springer, Elsevier, ACM and Wiley. It is worth mentioning that the two publishers responsible for publishing mainly Computer Science journals and a relatively low number of conference proceedings (if any), i.e. Elsevier and Wiley, are those providing the highest number of openly-available references per publication (on average, around 29 and 37 cited works for each publication respectively).
Publisher
DBLP entities
COCI incoming citations
COCI outgoing citations
IEEE
1,730,485
18,930,055
21,582,093
Springer
1,012,534
18,482,132
11,179,566
Elsevier
574,860
15,536,207
17,019,716
ACM
433,188
3,695,255
6,050,342
Wiley
89,662
3,350,183
3,357,065
Table 1. The DBLP entities retrieved in the study grouped by their publisher and their incoming and outgoing citations according to COCI.
Future developments
Of course, this study does not provide full coverage of open citations in Computer Science but just a preliminary insight. First, as anticipated below, DBLP does not have the complete coverage of all CS-related publications since there are some venues that are not listed there (yet). Thus, some relevant open citations could not be extracted from COCI if these involve as citing and cited entity non-DBLP publications that belong to the CS domain. However, it is worth mentioning that no bibliographic and citation database (including commercial and proprietary ones) has a full disciplinary coverage anyway and DBLP is, probably, the most comprehensive collection of Computer Science publications metadata (something that could be assessed in future analysis).
Along the same lines, the index of open citations used, i.e. COCI, does not contain all the citations defined in CS publications, but only DOI-to-DOI citations as retrievable from Crossref data. Although Crossref is the biggest DOI provider and it is used by the majority of the big publishers,citations defined in publications with a non-Crossref DOI (e.g. DataCite) and those not having any DOI assigned (e.g. the papers published in CEUR Workshop Proceedings) are not included in COCI and, consequently, have not been used in the analysis. However, OpenCitations plans to extend its data coverage adding more sources in the next years. Thus, it would be interesting to replicate the same analysis in the future to see if and how much the coverage increase, at least in the context of Computer Science publications.
Still about coverage, currently (i.e. 31 December 2021) the only publisher of those included in Table 1 which is not providing open references through Crossref is IEEE. Indeed, while COCI includes several citations involving IEEE publications as citing entities, there is no availability of such citation after October 2018, when IEEE decided not to allow anymore Crossref Metadata Plus users to access these reference data.
Finally, analysing the preliminary results of this study, it would be interesting to understand which are the main subject categories of non-DBLP publications included in the 61% of citations shown in Figure 1 (e.g. by using the Scimago Journal and Country Rank database to retrieve their subject categories) to understand what are the citation dynamics between Informatics and other disciplines. However, I will leave the answer to this question to future analysis.
A final remark on reproducibility
Since several of the suggestions provided above start from the idea of either replicating or extending this study with additional materials and insights, it is important that all data and software used to perform the analysis are available online to permit its reproducibility. To this end, I have published both the software and the data retrieved online with open licenses to enable anyone to reuse it freely for any purpose.
Guest post by Arcangelo Massari, University of Bologna
In this post, Arcangelo Massari, who recently graduated in Digital Humanities and Digital Knowledge under Professor Silvio Peroni at the University of Bologna, shares the results of his master thesis.
A particular problem in information retrieval is that of obtaining data from an evolving dataset, independent of the time at which that item of data was added, changed or removed. To permit such time-independent queries to be performed over evolving RDF datasets, I have developed two new pieces of open source software, time-agnostic-library [1] and time-agnostic-browser [2], that are now available from the OpenCitations GitHub repository.
The time-agnostic-library is a Python library to perform live time-traversal queries on RDF datasets. Time-traversal means being agnostic about time: a SPARQL query that is not run on the current state of the collection but over its entire history or over a specified timespan of that history [3]. This tool allows materializations – obtaining all versions of an entity over time, or its status at a given time. Furthermore, SPARQL queries can be performed to get the delta between two or more versions of one or more resources. Thereby, the time-agnostic-library realizes all the retrieval functionalities described in the taxonomy by Fernández et al. [3].
To complement this query software, the time-agnostic-browser is a web application built on top of the time-agnostic-library to achieve the same results via a graphical user interface.
The primary purpose of these developments is to offer a system for browsing the provenance [4] of RDF statements across time: who produced them, when, where the information was taken from, and what changes were made compared to the previous state of the resource. Knowledge of such information is essential because data changes over time, either because of the natural evolution of concepts or due to the correction of mistakes. Indeed, the latest version of knowledge may not be the most accurate. Such phenomena are particularly tangible in the Web of Data, as highlighted in a study by the Dynamic Linked Data Observatory, which noted the modification of about 38% of the nearly 90,000 RDF documents monitored for 29 weeks, and the permanent disappearance of 5% of them [5] (Figure 1).
Figure 1. Donut chart showing the results of the study conducted by the Dynamic Linked Data Observatory on the evolution of RDF documents [5].
Additionally, the truthfulness of data cannot be assessed without provenance records and a system to query them. In fact, the truth value of an assertion on the Web is never absolute, as demonstrated by Wikipedia, which in its official policy on the subject states: “The threshold for inclusion in Wikipedia is verifiability, not truth.” [6]. The Semantic Web does not alter that condition, and trustworthiness has to be evaluated by each application by probing the context of the statements [7]. It is a challenging task and thus, in the Semantic Web Stack, trust is the highest and most complex level to satisfy, subsuming all the previous ones (Figure 2).
Figure 2.The Semantic Web layers [7]. Trust is the uppermost level of the stack, subsuming all the others.
Notwithstanding these premises, at present the most extensive RDF datasets – DBPedia [8], Wikidata [9], Yago [10], and the Dynamic Linked Data Observatory [11] – do not use RDF to track changes and record the provenance of such changes. Instead, they all adopt backup-based archiving policies. Some of them, such as Yago 4, record provenance but not changes. As far as citation databases are concerned, OpenCitations is the only infrastructure to implement change-tracking mechanisms and to record full RDF provenance records for each data entity. Among the leading players in this field, neither Web of Science nor Scopus have adopted similar solutions.
In accordance with the OpenCitations Data Model (OCDM) [12],a provenancesnapshot is generated by OpenCitations every time a bibliographical entity is created or modified. Each snapshot (prov:Entity) records the responsible agent (prov:wasAttributedTo), the generation time (prov:generatedAtTime), the invalidation time (prov:invalidatedAtTime), the primary source (prov:hadPrimarySource), and a link to the previous snapshot (prov:wasDerivedFrom), using terms from the Provenance Ontology. In addition, OCDM introduced a system to simplify restoring an entity’s status at a given time, by saving the delta between two versions as a SPARQL update query (prov:hasUpdateQuery) [13] (Figure 3). This approach enables one to restore an entity to a specific timepoint (snapshot) in a straightforward way by applying the inverse operations, i.e., deletions instead of additions, etc.
Figure 3. Provenance in the OpenCitations Data Model.
This solution is concretely used in all the datasets related to the OpenCitations infrastructure, such as COCI, an open index containing almost 1.2 billion DOI-to-DOI citation links derived from the open reference data available in Crossref [14]. It is important to note that this OpenCitations provenance model is generic and reusable in any other context. Since the time-agnostic-library leverages OCDM, it too is generic and can be used for any RDF dataset that tracks changes and provenance as OpenCitations does.
The time-agnostic-library is released under the ISC license and is downloadable through pip [1]. Test-driven development was adopted as a software development process during its creation [15]. It makes three main classes available to the user: AgnosticEntity, VersionQuery, and DeltaQuery, for materializations, version queries, and delta queries, respectively (Listing 1).
Listing 1. Code template to achieve materializations, time-traversal queries, and delta queries.
All three operations can be performed over the entire available history of the dataset, or by specifying a time interval via a tuple in the form (START, END).
The time-agnostic-browser [2] is also released under the ISC license and can be run as a Flask application. It is organized into two macro-sections: “Explore” and “Query”. In the former, a text input accepts a URI. By submitting it, the entire history of the corresponding resource is displayed. In the latter, a text area receives a SPARQL query, which is resolved on all dataset states. Its main added value is hiding the triples and the complexity of the underlying RDF model: predicate URIs, as well as subjects and objects, appear in a human-readable format. Moreover, all the entities are displayed as links, providing shortcuts to reconstruct the history of the related resources (Figure 4).
Figure 4. Graphical user interface of an entity history reconstruction through the time-agnostic-browser.
The efficiency of time-agnostic-library was measured with two types of benchmarks [16], one on execution times and the other on the amount of computer memory (RAM) required by ten different use cases, each repeated ten times to produce significant results and avoid outliers. In light of these benchmarks, time-agnostic-library has proven effective for any materialization. Regarding structured queries, they are swift if all subjects are known or deductible. On the other hand, the presence of unknown subjects in the user’s SPARQL query involves the identification of all present and past entities that satisfy that pattern, and so requires a more significant amount of time and resources. Specifically, all materializations and the cross-version structured query with known subjects required about half a second and about 50 MB of RAM; conversely, with unknown subjects, 581 seconds and 519 MB of RAM are required. It can be concluded that the proposed software can be used effectively in all cases where the subject is known, that is, for any materialization or formulated SPARQL queries without isolated triple patterns containing unknown subjects.
Other software solutions for such problems have been proposed. Table 1 shows the list of available software to perform materializations and time-traversal queries on RDF datasets. As can be observed, time-agnostic-library is the only one to support all retrieval functionalities without requiring pre-indexing processes. This feature makes it particularly suitable for use in scenarios with large amounts of data that often change over time. Moreover, compared to the approach of Im, Lee and Kim [17] and OSTRICH [18], the OpenCitations Data Model only requires storing the current state of the dataset, rather than the original one, allowing one to query the latest version, without additional computational effort to first re-create the original version.
Table 1. Comparative between time-agnostic-library and preexisting software to achieve materializations and time traversal queries on RDF datasets. (Scroll right to see Columns 6-8).
The OpenCitations Data Model and the time-agnostic-library software are the pre-requisites that will allow OpenCitations to involve third parties, for example members of staff in academic libraries, in the submission, curation and updating of OpenCitations bibliographic and citation data. At this stage, all entities in COCI have a single snapshot — the one made at the time of creation. However, since these entities may become modified, corrected or enriched over time, it is imperative to have appropriate software tools available for use by curators. With the time-agnostic-library software and its associated time-agnostic-browser, it will be possible for a curator to explore the entire history of the changes within an RDF dataset, to know when they were made, based on which source, and by which responsible agent, thus ensuring the reliability and verifiability of data, and facilitating any necessary further changes.
[3] J. D. Fernández, A. Polleres, and J. Umbrich, ‘Towards Efficient Archiving of Dynamic Linked’, in DIACRON@ESWC, Portorož, Slovenia: Computer Science, 2015, pp. 34–49.
[5] T. Käfer, A. Abdelrahman, J. Umbrich, P. O’Byrne, and A. Hogan, ‘Observing Linked Data Dynamics’, in The Semantic Web: Semantics and Big Data, vol. 7882, P. Cimiano, O. Corcho, V. Presutti, L. Hollink, and S. Rudolph, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 213–227. doi: 10.1007/978-3-642-38288-8_15
[8] F. Orlandi and A. Passant, ‘Modelling provenance of DBpedia resources using Wikipedia contributions’, Journal of Web Semantics, vol. 9, no. 2, pp. 149–164, Jul. 2011, doi: 10.1016/j.websem.2011.03.002.
[9] P. Dooley and B. Božić, ‘Towards Linked Data for Wikidata Revisions and Twitter Trending Hashtags’, in Proceedings of the 21st International Conference on Information Integration and Web-based Applications & Services, Munich Germany, Dec. 2019, pp. 166–175. doi: 10.1145/3366030.3366048.
[11] J. Umbrich, M. Hausenblas, A. Hogan, A. Polleres, and S. Decker, ‘Towards Dataset Dynamics: Change Frequency of Linked Open Data Sources’, in Proceedings of the WWW2010 Workshop on Linked Data on the Web, Raleigh, USA, 2010. Available: http://ceur-ws.org/Vol-628/ldow2010_paper12.pdf
[12] M. Daquino, S. Peroni, and D. Shotton, ‘The OpenCitations Data Model’, p. 836876 Bytes, 2020, doi: 10.6084/M9.FIGSHARE.3443876.V7.
[13] S. Peroni, D. Shotton, and F. Vitali, ‘A Document-inspired Way for Tracking Changes of RDF Data’, in Detection, Representation and Management of Concept Drift in Linked Open Data, Bologna, 2016, pp. 26–33. Available: http://ceur-ws.org/Vol-1799/Drift-a-LOD2016_paper_4.pdf
[14] I. Heibi, S. Peroni, and D. Shotton, ‘Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations’, Scientometrics, vol. 121, no. 2, pp. 1213–1228, Nov. 2019, doi: 10.1007/s11192-019-03217-6.
[15] K. Beck, Test-driven development: by example. Boston: Addison-Wesley, 2003.
[16] A. Massari, ‘time-agnostic-library: benchmark results on execution times and RAM’. Zenodo, Oct. 05, 2021. doi: 10.5281/ZENODO.5549648.
[17] D.-H. Im, S.-W. Lee, and H.-J. Kim, ‘A Version Management Framework for RDF Triple Stores’, Int. J. Softw. Eng. Knowl. Eng., vol. 22, pp. 85–106, 2012.
[18] R. Taelman, M. V. Sande, and R. Verborgh, ‘OSTRICH: Versioned Random-Access Triple Store’, in Companion Proceedings of the Web Conference 2018, 2018, pp. 127–130. Available: https://core.ac.uk/download/pdf/157574975.pdf
[19] N. F. Noy and M. A. Musen, ‘Promptdiff: A Fixed-Point Algorithm for Comparing Ontology Versions’, in Proc. of IAAI, 2002, pp. 744–750.
[20] M. Völkel, W. Winkler, Y. Sure, S. Kruk, and M. Synak, ‘SemVersion: A Versioning System for RDF and Ontologies’, 2005.
[21] M. V. Sande, P. Colpaert, R. Verborgh, S. Coppens, E. Mannens, and R. V. Walle, ‘R&Wbase: Git for triples’, 2013.
[22] T. Neumann and G. Weikum, ‘x-RDF-3X: Fast Querying, High Update Rates, and Consistency for RDF Databases’, Proceedings of the VLDB Endowment, vol. 3, pp. 256–263, 2010.
[23] A. Cerdeira-Pena, A. Farina, J. D. Fernandez, and M. A. Martinez-Prieto, ‘Self-Indexing RDF Archives’, in 2016 Data Compression Conference (DCC), Snowbird, UT, USA, Mar. 2016, pp. 526–535. doi: 10.1109/DCC.2016.40.
[24] T. Pellissier Tanon and F. Suchanek, ‘Querying the Edit History of Wikidata’, in The Semantic Web: ESWC 2019 Satellite Events, vol. 11762, P. Hitzler, S. Kirrane, O. Hartig, V. de Boer, M.-E. Vidal, M. Maleshkova, S. Schlobach, K. Hammar, N. Lasierra, S. Stadtmüller, K. Hose, and R. Verborgh, Eds. Cham: Springer International Publishing, 2019, pp. 161–166. doi: https://doi.org/10.1007/978-3-030-32327-1_32.