Add Reproducibility Composite Confidence Index (RCCI) #93
Conversation
Introduced the Reproducibility Composite Confidence Index (RCCI) as a metric for assessing research artefacts. Added detailed sections on metrics, methodologies, and data sources related to RCCI.
vtraag
left a comment
vtraag
left a comment
There was a problem hiding this comment.
This looks quite good, thanks! Nice to see some of the other indicators integrated. I've gong through the indicator, and have some comments here and there. I think it shouldn't be too much work to fix.
| RCCI = FWCI \times FWRI \times FI \times RCI | ||
| $$ | ||
|
|
||
| A value greater than 1 (after scaling) suggests that artefacts are impactful, widely reused, FAIR-compliant, and positively regarded in the scientific community. |
There was a problem hiding this comment.
Could you explain this? For FWCI and FWRI I personally understand it, but it should perhaps be explained. But for FI and RCI I don't understand. Are these also somehow "normalised" so that 1 corresponds to the average or something? Or how should I think about this?
There was a problem hiding this comment.
Here is the info I took directly from the TIER2 Reproducibility Monitoring Dashboard Documentation document:
The FAIR Index (FI) is a score ranging from 0 to 1, where 1 represents adherence to the FAIR principles.
It is based on the presence and completeness of key metadata elements associated with the artefact, including:
- Name: The artefact’s name.
- Version: The version number or identifier of the artefact.
- License: Information about the licensing under which the artefact is made available.
- URL: A web link providing access to the artefact.
Formula:
FAIR Index = Number of Valid Metadata Elements / 4
If all four elements (Name, Version, License, URL) are present and valid, the artefact scores 1.0, indicating it fully meets the FAIR criteria.
Repro Confidence Index (RCI) considers the number of positive (supporting), neutral, and negative (refuting) citations a research artefact receives.
Each type of citation is weighted to reflect its influence on the perceived reproducibility of the artefact:
Repro Confidence Index =
(1 * Positive Citations + 0.5 * Neutral Citations - 1 * Negative Citations) / Total Citations
A higher Repro Confidence Index indicates that the artefact is generally viewed as reproducible, based on the feedback it has received.
There was a problem hiding this comment.
So 1 is not similar to average or something like that, but just a threshold that seems reasonable somehow, right?
If you could explain this (in a perhaps slight simpler terms) in the document itself, that would be wonderful.
Thanks!
| - $\overline{Citations}_{f,y}$ = the mean number of citations for all publications in the same field $f$ and year $y$. | ||
|
|
||
| **Interpretation:** | ||
| - FWCI = 1 → the publication/artefact is cited at the world average for its field and year. |
There was a problem hiding this comment.
Perhaps it would be better to also use LaTeX for these types of inline formulas?
There was a problem hiding this comment.
Just to clarify: the same comment goes for various places of course, but I won't all point them out explicitly.
| #### 2. Field-Weighted Reusability Index (FWRI) | ||
|
|
||
| **Definition:** | ||
| The Field-Weighted Reusability Index (FWRI) measures how often a research artefact (dataset, code, software) is **reused** compared to the average reuse rate of artefacts in the **same Field of Science (FoS Level 3)** and within a **comparable publication window (e.g. 3 years after release)**. |
There was a problem hiding this comment.
This specifically refers to SciNoBo fields, I guess, but that need not necessarily be the case, right?
There was a problem hiding this comment.
That is, it seems very specific here, but less so for the FWCI.
There was a problem hiding this comment.
In addition, I'm very curious how you define the Field of Science for a research artefact, since that is far from trivial.
There was a problem hiding this comment.
Yes the FoS Level 3 is a SciNoBo field (https://github.com/iNoBo/scinobo-fos-classification), because in TIER2 we used SciNoBo to calculate the FWCI score.
Here I can either add the above link to the SciNoBo FoS Classification to make it more specific, or I can remove the "Level 3" to let it more generic.
The FoS for a research artefact is proxied though the FoS of the publication that it was created in. For example if the paper that BERT was introduced is classified by SciNoBo as "Artificial Intelligence & Image Processing" in Level 3, then BERT inherits this FoS.
There was a problem hiding this comment.
I think it would be better to leave the field definition implicit, but perhaps note that it is a challenge in itself (and perhaps link to SciNiBo?)
|
|
||
| ### Measurement | ||
|
|
||
| #### 1. Field-Weighted Citation Impact (FWCI) |
There was a problem hiding this comment.
It might be better if we stick to MNCS, instead of FWCI, since that aligns better with the rest of the handbook, most notably the citation impact indicator.
There was a problem hiding this comment.
I now see that in the impact of code/data, we use NCI. Also there, I think it's useful to actually use the same terminology. There are now three different names for the same thing, we should clean that up a bit.
There was a problem hiding this comment.
In TIER2 we used the FWCI terminology, but I get your point, for consistency within the handbook I can use the NCI terminology.
| $$ | ||
|
|
||
| Where: | ||
| - $Citations_{i}$ = the number of citations received by publication or artefact *i*. |
There was a problem hiding this comment.
Why not use LaTeX here?
| - $Citations_{i}$ = the number of citations received by publication or artefact *i*. | |
| - $Citations_{i}$ = the number of citations received by publication or artefact $i$. |
|
|
||
| The [SciNoBo Toolkit](https://scinobo.ilsp.gr/toolkit) has implemented and operationalised the RCCI and its component indicators into a **working monitoring dashboard**. | ||
|
|
||
| - In the **TIER2 project**, SciNoBo was used to extract artefacts from project deliverables and publications, link them to citation and reuse data, and compute FWCI, FWRI, FI, RCI, and RCCI. |
There was a problem hiding this comment.
Could you provide a URL for TIER2?
There was a problem hiding this comment.
yes, I'll add https://tier2-project.eu/ here
| While SciNoBo currently offers the most complete implementation, other methodologies and tools can be used to compute individual RCCI components: | ||
|
|
||
| - **Citation normalisation** | ||
| FWCI can be derived using normalisation approaches described in the [Citation Impact](../2_academic_impact/citation_impact.qmd) indicator, based on expected citation counts per field and year. This methodology is implemented in bibliometric databases such as Web of Science/InCites (CNCI), Scopus (FWCI), and Dimensions (FCR). |
There was a problem hiding this comment.
Most notably, it is also implemented in OpenAlex, which should also be included given its open character.
|
Ping @PetrosStav. Could you perhaps address the outstanding comments? I could then merge this. |
|
Hi @vtraag! I addressed your comments and made the required changes, please check them. |
|
Thanks @PetrosStav ! Could you also resolve the relevant comments if you believe you resolved them? I'll take a closer look at the newer changes then shortly. |
2 participants
I added the Reproducibility Composite Confidence Index (RCCI) indicator, which we introduced in our work within the TIER2 project pilot, where we developed a Reproducibility Dashboard for funders and RPOs. This indicator has already been presented, reviewed, and discussed with stakeholders (funders and RPOs) through two webinars and presentations in the TIER2 project. I included it here, as we discussed before the summer, since it is a good way to highlight the synergy between the PathOS and TIER2 projects.