Getting started with Bibliometrics and Altmetrics

Table of contents

• Using metrics responsibly
• What are bibliometrics and altmetrics
• How to find and use bibliometrics and altmetrics
• Finding and using key author, article and journal bibliometrics and altmetrics
• Key author metrics – what, why, where?
• Key author metrics – how?
  • Web of Science – how to
  • Scopus – how to
  • Google Scholar – how to
• Journal metrics – what, why, where?
• Journal metrics – how?
  • Journal Citation Reports – Web of Science
  • How to use CiteScore in Scopus
  • How to use journal metrics in Google Scholar
• Field Weighted Citation Impact (FWCI) What, why, where, how?
• Altmetrics – what, why, where, how?
• More support for bibliometrics and altmetrics

Using metrics responsibly

DORA

The University is a signatory to the Declaration on Research Assessment (DORA). DORA recognises the need to improve the ways in which the outputs of scholarly research are evaluated. Principles for the responsible use of research metrics at the University of Bradford have now been agreed. Limitations and best practice approaches to using metrics are discussed below.

What are bibliometrics and altmetrics and why should I be interested?

Bibliometrics, a definition:

"Bibliometrics is the quantitative analysis of written documents. It is frequently used to analyse scientific and scholarly publications." (CWTS 2018:3)

Bibliometrics is intended to help in evaluating research outputs and provide an indication of their impact. Traditionally, metrics are based on citation accounts - the number of times a research paper is cited by other papers since publication - but there are now other measures of impact available.

Altmetrics, a definition:

Altmetrics (alternative metrics) are data that can explain both the volume and nature of attention that research receives online. They are complementary to traditional citation-based metrics, e.g. citation counts and journal impact factors, and do not replace these. They measure overall reach and engagement of research outputs with social media, blogs and in the news.

Benefits of bibliometrics and altmetrics 

• They provide evidence of the impact of your work, or that of your research group, or of a university. You could use this evidence when applying for a job, promotion or for research funding. It could be included in a CV.
• They help in identifying which journals to publish in, and finding new areas of research.
• They are used in some university rankings and the Research Excellence Framework.
• The Metrics Toolkit provides further use cases. 

Try our Reading list of materials on bibliometrics and altmetrics to find out more. This includes books, articles and websites to introduce you to bibliometrics, and a section on responsible metrics.

A quick reference guide to bibliometrics, produced by Elsevier, is available to view online.

How to find and use bibliometrics and altmetrics

Elsevier produced a webinar video for the University of Bradford in August 2020 which gives an excellent overview of Bibliometrics and Altmetrics.

The Metrics Toolkit is a resource for researchers that provides guidance for demonstrating and evaluating claims of research impact. It is an excellent starting point to help in quickly understanding what a metric means; how it is calculated; and if it is a good match for an impact question. It covers both traditional metrics and altmetrics.

The toolkit is used to find evidence-based information about research metrics across disciplines, including an explanation of how each metric is calculated, where you can find it, and how each should - and should not - be applied.

Key author metrics - what, why, where?

What?

Author metrics are used to track how often an author's work is cited; and to demonstrate the reach and impact of their work.

The author citation information in Scopus, Web of Science and Google Scholar also includes the h-index (Hirsch index) score. This is designed to measure an author's productivity and impact. It measures a researcher's impact, based on the number of citations to their work.

An author has index h, if h of their papers has been cited at least h times. For example, an author has published 20 papers which have been cited 20 times, giving them an h-index 20. "Hirsch reckons that after 20 years of research, an h index of 20 is good, 40 is outstanding, and 60 is truly exceptional." (Oswald N, 2009)

Authors who do not do well in the h-index:

• Those who write large numbers of low citation papers.
• Those who have only one highly cited paper.
• Younger authors.

The advantages and disadvantages of the h-index, and how to calculate it are covered in the Web of Science: h-index information guide.

Web of Science Author Impact Beamplots are a newer way of visualizing the citation performance of an author. A Beamplot shows the volume and citation impact of a researcher’s publication portfolio through time. Each paper’s citation count is normalized (i.e., benchmarked against other similar publications from the same discipline) and measured as a percentile.

Author Impact Beamplot or H Index?

An Author Impact Beamplot was developed as an alternative to the H Index and tries to combat criticisms of the H Index, notably that the H Index is biased in favour of senior researchers and against researchers who have taken a career break or published less at any given time.

Why?

Author metrics provide evidence of the impact of your work, or that of your research group, or of a university. This evidence can be used when applying for a job, promotion or research funding. It could also be included in a CV. They are also used to discover key researchers in a field, track the work of colleagues and identify potential collaborators.

Where?

Web of Science, Scopus and Google Scholar can be used to find citations for your own articles; an author you are interested in; or your h-index. NB! There is no single citation analysis tool that collects all publications and their cited references. The University of Michigan have explored the benefits and limitations of using Google Scholar for citations; and The Centre for Science and Technology Studies at Leiden University have flagged up some advantages and limitations of using Google Scholar. Google Scholar's own help pages on metrics are also available.

Author Impact Beamplots for an individual author are now shown on the Web of Science new interface.

Key author metrics - how?

NB! These step-by-step instructions are correct on 1 Aug 2020. Contact your subject librarian for further guidance.

How to find your Author Impact Beamplot

1. Go to Summon and search for "Web of Science", and follow the link to access it.
2. When you go to Web of Science, you will see the message “The new Web of Science is here” – click on the Check it Out button, and then search by author.
3. The Beamplot summary for the author is shown. Click on View Full Beamplot for full details.

The Author Impact Beamplot Summary displays three points:

• Minimum publication percentile
• Overall median publication percentile (indicated by a green circle)
• Maximum publication percentile

Advice on interpreting a beamplot is available or watch this short beamplot overview video to find out more. A more detailed look at Beamplots is also available. 

Journal metrics - what, why, where?

What?

These metrics look at the impact of individual journal titles. They measure the frequency with which an average article in a journal has been cited in a particular year. Impact factors vary in different subject disciplines.

Why?

Journal metrics can help identify which journals to publish in, and find new areas of research.

Where?

To find journal metrics, such as journal impact factors, use Web of Science, Scopus and Google Scholar. NB! There is no single journal metrics analysis tool that covers all publications.

Journal metrics - how?

NB! These step-by-step instructions are correct on 1 Aug 2020. Contact your subject librarian for further guidance.

How to use journal metrics in Google Scholar

Google Scholar lets you browse the top 100 publications in several languages, ordered by their five-year h-index and h-median metrics. To see which articles in a publication were cited the most and who cited them, Click on its h-index number to view the article as well as the citations underlying the metrics. You can also explore publications in research areas of your interest.

To find out more, Google Scholar offer an overview on journal metrics. 

Field Weighted Citation Impact (FWCI) What, why, where, how?

What?

FWCI shows how an article's citation count compares to similar articles in the same field and timeframe. It measures the number. of citations received by an article divided by the expected no. of citations for similar articles. A FWCI of 1 means that the output performs as expected for the global average; an FWCI of 1.44 means 44% more citated than expected.

Why?

FWCI measures the citation impact of the output itself, not the journal in which it is published. Because the FWCI includes field normalization, theoretically, the score should be a better indicator of performance than a raw citation count. Find out more about Field Weighted Citation Impact. 

Where?

You will find FWCI in Scopus.

How?

Try this short online article metrics tutorial by Scopus which includes a section on FWCI.

Limitations of metrics

To ensure fair assessment of research, metrics should not be used on their own. In particular, use journal and author metrics with caution and understand the key limitations outlined below.

High citation counts

High citation counts can be used to show the influence of an article, but you should not assume they are an indicator of quality and validity. A controversial article may be cited frequently, because others are arguing against its findings. A fraudulent paper may get a high number of negative citations. Older material will tend to be more highly cited as it has had more time to be cited. Citation-based metrics should not be interpreted as a direct measure of research quality.

Example

The 1998 article by Wakefield et al, which fraudulently reported an association between vaccination and autism, has a very high citation countand continues to accumulate citations even after it was retracted.

Wakefield AJ, Murch SH, Anthony A et al. Ileal-lymphoid-nodular hypeplasia, non-specific colitis, and pervasive developmental disorder in children [retracted in: Lancet. 2010; 375(9713): 445]. Lancet. 1998; 351(637-641. doi:10.1016/S0140-6736(97)11096-0.

Comparing different researchers

Do not use the author metric h-index to compare the output of different researchers. Raw citation numbers are not comparable from person to person, as metrics increase over time and are more likely to be higher for an experiences researcher who has published more papers over a long period of time than an early career researcher, so will discriminate against early career researchers on the basis of the length of their career and nothing else.

Comparing different disciplines

Do take care when using journal metrics to compare different disciplines which have different citation behaviour and where that behaviour may vary over different periods of time. Some research areas, such as medicine, cite papers more than others and different fields of research publish at different rates. Publications in different disciplines will also be cited at a varying pace. There are metrics available which will try to take citation behaviour within a field into account, to provide a reasonable comparison. The table below, provided by Scopus (2020), shows which key metrics are field weighted.

Using different sources and more than one source

It is important to look at more than one source of metrics analysis, as different sources of metrics will yield different results. Resources such as Scopus and Web of Science cover different journals, so will give you distinct results for metric analysis. Using Altmetrics will give you a broader picture. Be careful of sources like Google Scholar where it is less clear what is covered.

Using journal impact factors

A high impact factor does not tell you anything about the quality of the individual articles within the journal. The content of a paper is more important than where it has been published. DORA's first recommendation is not to use journal based metrics such as Journal Impact Factors as a surrogate measure of the quality of individual research articles.

Best practice - 3 dos and don'ts

Now that you have read some of the limitations of bibliometrics, watch this short video on How to use bibliometrics effectively, and follow these three best practice principles:

1. Do ensure you are clear and transparent in the metric methodology you are using, and consider whether the metric you chose is appropriate for the aspect of research performance you are looking at.

   This generally means using relative measures that give context and more accurate means of comparison. These include metrics related to publications such as article-based metrics, e.g. field weighted citation ration, rather than absolute measures or concrete totals, such as journal-based metrics, e.g. Journal Impact Factor, SJR, SNIP or author metrics, e.g. h-index. 

   Below we have made available a document for more information on the pros and cons of different metrics.

2. Do use quantitative alongside qualitative input such as peer-review and expert opinion.
3. Do not use one quantitative metric in isolation - research performance should not be evaluated against a number of metrics used together. For quantitative metrics use at least two distinct metrics sources to reduce bias.

More support for bibliometrics and altmetrics

• The library offers an Introduction to Bibliometrics workshop.
• Talk to your Faculty Librarian.
• Learn more about bibliometrics and how to use them responsibly through our recommended books, reports and articles.