Bibliometric and altmetric studies
Here at the University, bibliometric and altmetric work falls under the remit of the Open Research team. We conduct bibliometric and altmetric studies at various levels of scale (institutional, faculty, departmental, group, and individual) for a variety of internal stakeholders. Stakeholders then apply the outcomes of these studies in a range of ways, including for strategic intelligence, decision making, evaluation, and recruitment.
Our work is carried out in line with the ethics, principles, and values of responsible metrics as set out by the international and UK metrics communities. Through a combination of leading by example and awareness raising, we strive to engage the Surrey community with responsible metrics and to advocate their application at all times in any metrics related work undertaken at the University.
Get in contact
If you have any questions about research metrics or the tools we use here then please email our Open Research team.
Metrics tools used
Metrics tools used at Surrey are:
Citation indices supply the data that underlie bibliometrics. There are two subscription citation indices:
- Clarivate's Web of Science
- Elsevier's Scopus
Both are accessible through the Library at Surrey.
Scival provides easy access to bibliometric profiles and is the bibliometrics tool we use at SurreyScival is a subscription bibliometrics tool that uses data from Scopus.
On campus access is available to all staff with a Surrey email address.
For altmetric data, Surrey subscribes to the Altmetric Explorer for Institutions (EFI).
Responsible use of metrics
Responsible metrics capture fairly the richness, diversity, and complexity of research and researchers. Humility, robust data, transparency, and reflexivity are also key components of responsible metrics.
Both here in the UK and internationally, there is a growing urgency to encourage more responsible use of metrics. In the UK, responsibility for this falls formally under the remit of the Forum for Responsible Metrics. The Forum works with universities, funders, and the research community to support and promote responsible metrics.
As much as big data now plays a role in the commercial sector, the availability of large datasets and better computing power has led to a growth in the use of metrics in higher education. Used responsibly, metrics can offer a lot to our sector, including valuable strategic insight and new perspectives.
Yet, with their proliferation has come growing concern from both the academic community and the metrics profession about the ways in which metrics are being used and understood, and indeed, misused and misunderstood.
Concern is growing, both amongst academics and the metrics community, as the use of metrics in practice now often shifts its focus on to the individual, a level of scale at which many metrics were never intended to be used. This shift combined with the ready availability of metrics generated through easily accessible high-performance computing tools means that just about anyone can now 'do metrics'.
But, because of their potential power, metrics inherently carry ethical considerations that place a heavy responsibility on all users, whether experienced or not. Today, perhaps more than ever before, there is the need for balanced, informed, and expert practice around the use of metrics.
The international metrics community has responded to this need with formal statements that set out the principles of responsible metrics. Two key documents in this process are:
In the UK, this work continues under the umbrella of University UK's Forum for Responsible Metrics.
The UK Forum defines responsible metrics as following five key principles:
- Robustness – basing metrics on the best possible data in terms of accuracy and scope
- Humility – recognising that quantitative evaluation should support, but not supplant, qualitative, expert assessment
- Transparency –that those being evaluated can test and verify the results
- Diversity – accounting for variation by research field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system
- Reflexivity – recognising and anticipating the systemic and potential effects of indicators, and updating them in response.
Working to these five principles is a good start, but there is still much debate and discussion needed around the responsible use of metrics.
Here, bibliometric and altmetric work falls under the remit of Library and Learning Support Services. We conduct bibliometric and altmetric studies at various level of scale (institutional, faculty, departmental, group, and individual) for a variety of internal stakeholders.
Stakeholders then apply the outcomes of these studies in a range of ways, including for strategic intelligence, decision making, evaluation, and recruitment.
Actions we are taking
Our work within Open Research, and across the University, is carried out in line with the ethics, principles, and values of responsible metrics as set out by the international and UK metrics communities, our funders, and the sector statements outlined above (including the principles of DORA, the Leiden Manifesto, and the UK Forum for Responsible Use of Metrics).
The University of Surrey has developed its own statement of responsible metrics principles. A new implementation plan to support the practical application of responsible metrics, including resources, the development of evaluation, and recognition of the value of all relevant research outputs, has been created from the academic year 2020-21. Progress against this plan will be actively monitored.
The responsible metrics principles the University promotes are:
- Use metrics to support—not supplant—peer review and expert knowledge.
- Recognise that research and its outputs are rich, complex, and varied. In contrast, metrics are usually one dimensional. Be realistic about the limits of what they can tell you.
- Foster humility in your approach to metrics and take care not to imbue them with undue power.
- Understand that most metrics are designed to work at larger scales. At the level of the individual, prefer qualitative information.
- Use data that are of sufficient accuracy and scope.
- Where such data are unavailable, the usefulness of metrics diminishes. At best, results will be incomplete. At worst, they may be inaccurate and misleading.
- Weigh up the pros, cons, and risks of proceeding under such circumstances.
- Recognise that almost all uses of metrics involve comparison of some sort, no matter how small or inconsequential that comparison may seem.
- Ensure that only appropriately normalised indicators are used when comparisons are made, even in cases where comparisons are deemed to be 'within the same field' or for the same person.
- Recognise that some questions can only be addressed using metrics that cannot be systematically normalised.
- Only use these non-normalised metrics in non-comparative, and ideally, narrative contexts.
- Recognise that most metrics are indicators of potential effects or states.
- Indications are informed inferences—not indisputable facts!
- Avoid misplaced precision in interpreting indicators.
- Accompany point estimates with stability (confidence) intervals, where possible.
- Align metrics work transparently with mission or strategy.
- Undertake bibliometric and altmetric studies with a stated purpose in mind.
- Define questions and aims in sufficient detail, and only then, choose methods and indicators.
- Ensure that the metrics selected capture the concept probed by the question asked.
- Most bibliometric and altmetric studies explore more than one aspect of research.
- Check that the metrics you select capture the concepts you think they do.
- To look at multiple aspects of research, you need to use multiple metrics.
- Even if looking at a single aspect, capture this with several metrics if you can.
- Recognise that all metrics have limitations. Use these limitations to frame your results.
- Accept that in some contexts, metrics aren't useful at all.
- Never make vital decisions on metrics alone.
- Consider not only the positive impacts of using metrics, but also the negative ones.
- Don’t overuse metrics.
- Appreciate that where metrics are applied to individuals, careers can be at stake.
- Insist on reproducibility and transparency in data collection and analysis.
- Be on the lookout for bias. Seek to reduce it and acknowledge its effects.
- Cherish objectivity and neutrality.
- Be fair, mindful, and considerate of all parties involved.
- Choose the most straight-forward approach sufficient for addressing the question asked.
- Provide plain English, non-technical explanations of bibliometric or altmetric methods used.
- Raise awareness and increase understanding of metrics.
- Listen to concerns that are raised.
- Encourage questions and thoughtful debate and discussion.
- Keep pace, expect change, and adjust metrics and processes accordingly. Areas of work related to both research and metrics are rapidly developing and evolving.
- Recognise that metrics establish incentives. These can change human behaviours and the research ecosystem.
- Look out for unintended consequences. When you spot them, determine their effect.
- Manage the goal displacement and gaming that can arise in response to the use of metrics.