Responsible use of metrics

Responsible metrics capture fairly the richness, diversity, and complexity of research and researchers.  Humility, robust data, transparency, and reflexivity are also key components of responsible metrics. 

For a sense of what responsible metrics look like in practice, watch this short video.  The video summarises the Leiden Manifesto for Research Metrics, a document that has proved highly influential in the worldwide move towards responsible metrics.

Both here in the UK and internationally, there is a growing urgency to encourage more responsible use of metrics.  In the UK, responsibility for this falls formally under the remit of the Forum for Responsible Metrics.  The Forum works with universities, funders, and the research community to support and promote responsible metrics.


Why worry about responsible metrics?

Much as big data now plays a role in the commercial sector, the availability of large datasets and better computing power has lead to a growth in the use of metrics in Higher Education. Used responsibly, metrics can offer a lot to our sector, including valuable strategic insight and new perspectives.

Yet, with their proliferation has come growing concern from both the academic community and the metrics profession about the ways in which metrics are being used and understood—and, indeed, misused and misunderstood

How does this affect me and my career?

Concern is growing, both amongst academics and the metrics community, as the use of metrics in practice now often shifts its focus on to the individual, a level of scale at which many metrics were never intended to be used. This shift combined with the ready availability of metrics generated through easily accessible high-performance computing tools means that just about anyone can now 'do metrics'.  

But, because of their potential power, metrics inherently carry ethical considerations that place a heavy responsibility on all users, whether experienced or not. Today, perhaps more than ever before, there is the need for balanced, informed, and expert practice around the use of metrics.


How are responsible metrics being promoted?

The international metrics community has responded to this need with formal statements that set out the principles of responsible metrics. Two key documents in this process are:

*  The San Francisco Declaration on Research Assessment (DORA) 2012  

*  The Leiden Manifesto 2015

In the UK, this work continues under the umbrella of University UK's Forum for Responsible Metrics.

How do I know if metrics are responsible?

The UK Forum defines responsible metrics as following five key principles:
Robustness – basing metrics on the best possible data in terms of accuracy and scope

Humility – recognising that quantitative evaluation should support, but not supplant, qualitative, expert assessment

Transparency –that those being evaluated can test and verify the results

Diversity – accounting for variation by research field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system

Reflexivity – recognising and anticipating the systemic and potential effects of indicators, and updating them in response

Working to these five principles is s good start, but there is still much debate and discussion needed around the responsible use of metrics.


What about here at Surrey?

Here, bibliometric and altmetric work falls under the remit of Library and Learning Support Services (LLSS). We conduct bibliometric and altmetric studies at various level of scale (institutional, faculty, departmental, group, and individual) for a variety of internal stakeholders. Stakeholders then apply the outcomes of these studies in a range of ways, including for strategic intelligence, decision making, evaluation, and recruitment.
Our work is carried out in line with the ethics, principles, and values of responsible metrics as set out by the international and UK metrics communities. Through a combination of leading by example and awareness raising, we strive to engage the Surrey community with responsible metrics and to advocate their application at all times in any metrics related work undertaken at the University.
These are the actions we take to put responsible metrics into practice:

Cherish human judgement

 Use metrics to support—not supplant—peer review and expert knowledge.
 Recognise that research and its outputs are rich, complex, and varied. In contrast, metrics are usually one dimensional. Be realistic about the limits of what they can tell you.
 Foster humility in your approach to metrics and take care not to imbue them with undue power.
 Understand that most metrics are designed to work at larger scales. At the level of the individual, prefer qualitative information.

Know your data and their limitations

 Use data that are of sufficient accuracy and scope.
 Where such data are unavailable, the usefulness of metrics diminishes. At best, results will be incomplete. At worst, they may be inaccurate and misleading.
 Weigh up the pros, cons, and risks of proceeding under such circumstances.

If you can’t normalise, then don’t compare

 Recognise that almost all uses of metrics involve comparison of some sort, no matter how small or inconsequential that comparison may seem.
 Ensure that only appropriately normalised indicators are used when comparisons are made, even in cases where comparisons are deemed to be 'within the same field' or for the same person.
 Recognise that some questions can only be addressed using metrics that cannot be systematically normalised.
 Only use these non-normalised metrics in non-comparative, and ideally, narrative contexts.

Respect the difference between indicators and measures

 Recognise that most metrics are indicators of potential effects or states.
 Indications are informed inferences—not indisputable facts!
 Avoid misplaced precision in interpreting indicators.
 Accompany point estimates with stability (confidence) intervals, where possible.

Specify what you are asking and why

 Align metrics work transparently with mission or strategy.
 Undertake bibliometric and altmetric studies with a stated purpose in mind.
 Define questions and aims in sufficient detail, and only then, choose methods and indicators.
 Ensure that the metrics selected capture the concept probed by the question asked.

Come at it from various angles

 Most bibliometric and altmetric studies explore more than one aspect of research.
 Check that the metrics you select capture the concepts you think they do
 To look at multiple aspects of research, you need to use multiple metrics.
 Even if looking at a single aspect, capture this with several metrics if you can.

Heed limitations

 Recognise that all metrics have limitations. Use these limitations to frame your results.
 Accept that in some contexts, metrics aren't useful at all.

Take a step back

 Never make vital decisions on metrics alone.
 Consider not only the positive impacts of using metrics, but also the negative ones.
 Don’t overuse metrics.
 Appreciate that where metrics are applied to individuals, careers can be at stake.

Practice integrity

 Insist on reproducibility and transparency in data collection and analysis.
 Be on the lookout for bias. Seek to reduce it and acknowledge its effects.
 Cherish objectivity and neutrality.
 Be fair, mindful, and considerate of all parties involved.

Engage and explain

 Choose the most straight-forward approach sufficient for addressing the question asked.
 Provide plain English, non-technical explanations of bibliometric or altmetric methods used.
 Raise awareness and increase understanding of metrics.
 Listen to concerns that are raised.
 Encourage questions and thoughtful debate and discussion.

Anticipate and adapt

 Keep pace, expect change, and adjust metrics and processes accordingly. Areas of work related to both research and metrics are rapidly developing and evolving.
 Recognise that metrics establish incentives. These can change human behaviours and the research ecosystem.
 Look out for unintended consequences. When you spot them, determine their effect.
 Manage the goal displacement and gaming that can arise in response to the use of metrics.

Where we are

University Library, George Edwards Building,  University of Surrey,  Guildford,  Surrey GU2 7XH,  United Kingdom T: +44 (0)1483 689235 F: +44 (0)1483 689500

More contact details
Back to top

Page Owner: rb0043
Page Created: Tuesday 17 October 2017 14:37:55 by rb0043
Last Modified: Friday 19 January 2018 12:40:13 by rc0059
Expiry Date: Thursday 17 January 2019 14:37:29
Assembly date: Thu Jan 25 11:07:08 GMT 2018
Content ID: 171940
Revision: 6
Community: 1308

Rhythmyx folder: //Sites/ (NEW)/researchn/openresearch/understandingmetrics
Content type: rx:Generic