Research metrics are measures used to assess the influence or impact of scholarly outputs. They can be both qualitative and quantitative. Qualitative methods will include peer review of the actual research not just the publications, the amount of funding achieved, awards and patents obtained and the amount of the material that is available as open access. However, quantitative methods are more generally used to answer questions such as how the work has influenced future work, who are the leading researchers in a field, what are the emerging areas of research, who are potential collaborators/competitors and to assess the impact of an individual’s/group research output. The most common methods here are bibliometrics (methods to analyse and track scholarly publications), citation analysis (who is citing who) and almetrics (a relatively new way of tracking and analysing scholarly works online).
Bibliometrics refers to the traditional quantitative analysis of scholarly publications. Bibliometrics should be used in addition to qualitative, expert assessment to provide more robust evidence of academic impact.
Metrics can be used by individual researchers or research administrators can use them to measure their own or their institutions citation impact. It I crucially important that bibliometrics are used responsibly and appropriately, in a fair and transparent way.
Bibliometric data can be used to track and monitor academic impact when used responsibly. However, bibliometrics have restrictions and limitations and as such can present very narrow and limited information and should not be solely used in isolation to demonstrate impact or to assess research. A more responsible and appropriate approach towards research evaluation would be to combine multiple different metrics and to include informed expert, qualitative peer review and assessment. Research evaluation should go beyond tracking citation numbers linked to publications and journals etc and should include a wide range of research outputs and activities, such peer review, supervision and mentorship, editorial roles, contribution to broader society, economy and policy making etc.
Article/Book Citation Impact: The academic impact of specific publications, including journal articles, books, and conference proceedings, can be measured by the number of times they are cited in other publications.
Journal Citation impact: The impact of academic journals can be measured by the number of times their articles receive citations and where they are cited.
Researcher Citation impact: The number of outputs a researcher has published and the number of citations these publications accrued can be used in part to evaluate the academic impact of an individual researcher.
It is crucial to acknowledge that while bibliometric indicators can be a useful source of information they are also restricted and limited measures that can be gamed both in your favour and to your detriment if they are not used responsibly and appropriately. For example over relying on one singular metric to demonstrate the overall impact of a piece of research, an individual researcher or group of researchers would be highly irresponsible. It is best practice to combine multiple metrics along with expert qualitative peer review and assessment where possible to provide a more contextualised representation of the impact made. Additionally traditional bibliometric indicators only reflect a portion of the impact of publications solely and not everything that goes into the day to day work of research that should be recognised as a contribution to research culture and impact holistically. This includes but is not limited to things such as the development of others through supervision or mentorship, their involvement with open research practices, contribution to peer review, serving on committees, communities of practice, editorial roles, economic, societal and policy impact, public engagements, honours and awards achieved and so on.
Some limitation include:
Quality: high citation counts may not necessarily indicate quality, in some instances a paper may simply be cited for its inaccuracies or shortcomings.
Approaches between discipline's vary significantly: some research areas cite papers more than others so higher citation rates are to be expected as the norm in some disciplines and this will not be the case in others. As such it is imperative to only compare like with like!
Early Career Researchers or people who come to a career in research later in life or who may have taken a career break may be disadvantaged as citations accrue over time. As such stage of career must be taken into account. PlumX may be useful in this instance.
Data coverage varies significantly: the tools and databases used to collect and calculate bibliometric data does not cover all research areas or index all publications. As such the results available will only reflect the percentage of publications that are indexed within those tools and databases. This will vary depending on the tool/database you use and within each discipline.
This work is licensed under CC BY-NC-SA 4.0
