Exploring the Impact of Altmetrics in Relation to Citation Count and SCImago Journal Rank (SJR)

Altmetrics are considered complementary to traditional, citation-based metrics. They include the number of article downloads, citation of research in online news/social media sources, Mendeley and non-traditional forms of scholarship. Altmetrics are fast, using public APIs to gather data in days or weeks.^[1] It measures the impact of research from non-traditional sources which categorize the social web.^[2] According to Torres-Salinas et al.^[3] the sources that are analysed include-micro-blogging (Twitter), social networking sites (Face-book), blogs (Word-Press, Blogger), social bookmarking networks (Delicious), academic bookmarking platforms (CiteULike, Mendeley), etc. This new alternative metrics serves the purpose of being a measure of consumption of the research.^[4] Broadly speaking, it tracks the online mentions of research outputs from various sources, mainly social media like social networking sites, blogs, etc., for impact measurement.^[5]

Citation counts-which measure the number of times a publication has been cited by other publications in the database- have long been used to gauge the influence of scientific works. But these metrics are slower, narrow and a work’s first citation can take years. This is a main problem as people who are familiar with a field would require to be acquainted with the most recently published work, that remains uncited except in unusual cases. Other limitations of citation metrics in present times include-Widespread self-publishing (via blogging, microblogging, and comments or annotations on existing work).

Sharing of “raw science” like datasets, code, and experimental designs with articles.

Semantic publishing or “nano publication,” where the citable unit is an argument or passage rather than the entire article.^[2,6]

Altmetrics is regarded as an alternative to such citation base metrics like h-index and journal impact factor and is determined by the count of article views, saves, downloads, cites, mentions etc., in social media,^[7] with a main aim of measuring the scholar interaction from the online social media tools which are used to share the research out-put, like bookmarking, blog post, twitter, Facebook post, etc. This alternative measure, thus, mainly focuses on the social use and impact of scholarly content which is measured by the indicators like shares, saves, views, reviews, downloads, etc.^[8] Usually, the altmetric score assigned to a particular attention indicator, which the research output has gained on different sites, is given as a weighted count in order to reflect the relative reach of each type of site or source.^[9]

Since, Altmetrics measure diverse impacts and accumulate faster than citations, it is usually regarded as being complementary to citations. According to Priem et al.^[10] “Citation and Altmetrics indicators track related but distinct impacts, with neither able to describe the complete picture of scholarly use alone.” Therefore, comparing the altmetric scores of publications with the citation counts would help to gauge the degree of association between the two.

Since, Altmetrics and citations, both measure the impact and influence of research output, the role of citation analysis while remains limited to the academic community, Altmetrics gauges the overall usage and social-impact.^[11] Altmetrics determines the reach of scholarly output beyond academia. Since the inception of this new metric, many studies have been conducted pertaining to its various dimensions.

Thelwall^[12] analyzed publications of various branches of social sciences, arts and humanities field, produced during 2013, to gauge altmetrics scores from blogs, news, twitter, reddit, face-book, Pinterest, Wikipedia, Google plus, etc. The study found a low percentage of articles, in all the fields, with non-zero altmetric score after five years of publication, while some with high score were found due to individual activities or self-publicity rather than public discussion.

Didegah et al.^[13] conducted a survey on Finnish articles to study the impact of range of factors on the citation and altmetric score of publications. The study found that high impact journals tend to have high altmetrics scores and the scores were found significant from Mendeley, twitter, Facebook, blogs and news posts. Similarly, publications with high collaboration rates-which increase the visibility of a work, demonstrated high count on most of the social media sites. Publications from top ranked institutions though score high on citations but altmetrics scores were found to have no relation with the institutional ranks. Research funding was found to have great impact on the citation scores to the extent that funded publications were having almost 40% more citations than the other, while for altmetrics also the research funding was found to have a positive impact with increased counts for Mendeley readers, tweets and face-book posts while blog and news-posts depicted no change.

Peters et al.^[14] examined the impact of cited research data on the altmetric scores and gauged the relationship between the citation counts and altmetrics. The data for the study, which included cited research publications, were retrieved from database-Data Citation Index (DCI) and Altmetrics analyzed from PlumX, ImpactStory and Altmetric.com. The findings reveal that the significant amount of cited research data has no altmetric score. Further, citation score and altmetric score were found to have no correlation. Ezema and Ugwu^[15] undertook a study to analyze the relationship between citation scores and Altmetrics in the field of Library and Information Science (LIS). The findings depicted that-citations data from the databases-Web of Science and Scopus, were traced to have the moderate relation with the altmetric scores of the LIS journals as analyzed from the altmetric explorer site while citation data from Google scholar was having high correlation with their altmetric attention. Similarly, Jabur^[8] in a study investigated the correlation between the bibliometric indicators and altmetric indicators. The citations as bibliometric indicators were collected from the article publications from Scopus database and altmetric indicators were collected from the social media like Research gate, Twitter, Mendeley, Facebook, etc., for the years 2012-2014 in the subject areas of social sciences and humanities. On calculation of Pearson’s correlation, the study concluded that the two variables showed a high correlation for year 2012 but for 2013 altmetric attention for the publications elevated resulting in the diminished correlation between the two. Wee and Chia^[16] analyzed 720 articles, which included 360 highly cited publications from 18 subject areas in the Web of Science database and 360 publications with high altmetric count, to examine if highly cited articles from Web of Science database have elevated altmetric scores too and vice versa. The study traced negative correlation between altmetric scores and citations of the publications in all subject areas excluding medicine. Wang et al.^[17] studied the relationship between metrics like article views, citations, downloads, social attention, etc., using data of 63, 805 PLOS articles. The study found a positive correlation between the citations and article views with significant correlation between citations and downloads. Though, social attention too was found to have a high correlation with the article views but low correlation with the citations.

Barnes^[4] while assessing research impact of the altmetric data concluded that altmetrics are better at measuring the public consumption of knowledge than the long-term practical impact of research. He concluded that “altmetrics are an extremely imperfect tool for predicting article performance in terms of future citations”. In a similar type of study, Thelwall and Nevill^[18] assessed the impact of altmetric.com scores in predicting the citation counts. Among all the altmetric indicators, Mendeley readers were found to have a strong correlation with the citations while face-book posts and tweets had no correlation with the future citations. This study contradicted the Barnes study and confirmed that early Altmetric.com scores can be helpful in predicting citation counts, though the power was found stronger for the journal impact factor. Besides, the employment of both the Altmetric.com scores and the journal impact factors was found to be the “optimal strategy” for this purpose.

Nuzzolese et al.^[19] conducted a study on Altmetrics to reveal the related or similar indicators and to gauge the effectiveness of indicators for assessing research quality while utilizing altmetric indicators from PlumX and citation count from Scopus. The data, which comprised bibliographic details of sample groups of 43184 and 41442 articles for full professor and associate professor rank, respectively, were collected from the Italian National Scientific Qualification (NSQ), 2016. The study found that the pairs of- citation counts and captures, and social media posts and blog-mentions, correlate better than other indicator pairs. Number of readers-a captured metric-depicted a moderately high and statistically significant relation with the citation counts while tweets and face-book posts from social media and mentions from blogs also demonstrated a significant relation. Thus, number of readers on Mendeley was found to be an alternative metrics correlating efficiently with traditional metrics i.e., citation count (article level) and h-index (author level) and these Mendeley reads were also traced as an accurate representation for predicting scientist qualification.

Several performance indicators have been used to evaluate research for some times now. These traditional indicators such as bibliometrics, citation analysis and journal impact factors gauge only the research works that are considered significant and theoretically relevant from the abundance of scholarly literature produced.^[20] The novel indicator- Altmetrics- which measures the attention, use and dissemination of scientific articles^[10] from social-media mentions is often considered complementary or supplementary to these traditional indicators.

Therefore, the current study aims to gauge the correlation or association between Altmetrics with citation score of the articles and Altmetrics with the SJR score of the corresponding publications.

The scope of this study was limited to the articles in the field of Biological Sciences, Earth and Environmental Science, History and Archaeology and Studies in Human Society featuring in Top 100 articles as listed by altmetric.com during 2014-17. The reason behind the selection of these four fields is that the most publications among top 100 belonged to these fields and also these cover different spheres of study (like science, arts and humanities). To ensure sufficient data and conduct meaningful correlation analysis, papers published in these fields were selected for the sample.

To assess the relation between Altmetrics scores with citation counts of publications.

To gauge the relation between Altmetrics and journal rank of the publications (SJR).

To analyse the trends in these relations over the four years period (2014-17).

Various altmetric aggregator sites like Altmetric, ImpactStory, Plum Analytics, etc., help to track the altmetric scores of the research output. The current study employed the aggregator Altmetric (altmetric.com) which provides the impact details utilizing the digital object identifier or other identifier. Besides, it freely provides the yearly ‘top 100 articles’, from year 2013 onwards, based on Altmetric score or rank. These articles are categorized into different subject fields based on the “Fields of Research classification codes” and these have received major attention on various online platforms. Further, Altmetrics has recently added citation data- harvested from Dimensions- a research insights database, providing information about where and to what extent scholarly publications have been cited by other academic works.^[21] Therefore, this service from aggregator Altmetric was utilized for the current study and the altmetric scores and citation scores were recorded. SCImago Journal and Country Rank (scimagojr.com) were used as a source for gathering data regarding the prestige of journals (using SJR indicator) in which the selected articles were published.

From the Altmetric.com “Top 100 articles” for years 2014, 2015, 2016 and 2017 were harvested.

Among “Top 100 articles”, only the articles belonging to the discipline of Biological Sciences, Earth and Environmental Science, History and Archaeology and Studies in Human Society were selected for the study. Once going through the data, 33, 42, 38 and 42 numbers of research papers from year 2014, 2015, 2016 and 2017 respectively, were identified in the selected fields and subsequently harvested. Then the data regarding select papers with altmetric score and citation count were recorded.

Further, data regarding the SJR of the journals in which the corresponding articles were published were also harvested from the official web portal of SCImago Journal Rank (scimagojr.com). The SCImagoJR (SJR) is a size-independent prestige indicator that ranks journals by their ‘average prestige per article’. It is based on the idea that ‘all citations are not created equal’. It is the only indicator that highlighting the fact that journals do not depend exclusively on the number of endorsements, as citations, they receive from other journals, but rather on a combination of the number of endorsements and the importance of each one of these endorsements. Thus, SJR was selected for the study. It is expressed as the average number of weighted citations received in the selected year by the documents published in the selected journal in the three previous years.^[22,23]

Moreover, spearman’s correlations were calculated for determining the association/relation between the altmetric score with citation count and altmetric score with SJR. Spearman rank correlation is a non-parametric test that is used to measure the degree of association between two variables. The Spearman rank correlation test does not carry any assumptions about the distribution of the data and is the appropriate correlation analysis when the variables are measured on a scale that is at least ordinal. Spearman’s correlation determines the strength and direction of the monotonic relationship between two variables. A monotonic relationship is a relationship that does one of the following:

As the value of one variable increases, so does the value of the other variable; or
As the value of one variable increases, the other variable value decreases.^[24] The following formula is used to calculate the Spearman rank correlation:

ρ= Spearman rank correlation; di= the difference between the ranks of corresponding variables; n= number of observations.

Usually, Spearman instead of Pearson correlation is used because metrics data is typically too skewed for the assumption of normal distribution of a Pearson test and has too many zero values to be transformed into a normal distribution.^[2,25]

Finally, the calculated data was analyzed and findings/conclusion were drawn on that basis.

Spearman correlations were calculated as a basic measure of association between pairs of variables. They are useful to assess whether there could be an underlying relationship that would allow a claim that the indicators reflected a common type of impact.^[2]

Spearman correlations were calculated for determining the association between altmetrics score and citation count of the selected articles. Most of the articles in Biological Sciences have weak positive correlations with each other ranging from 0.21-0.29 (Table 1) while for year 2016, negative correlation (-0.1) is found.

As far the earth and environmental science is concerned, the Altmetrics and citation scores mostly show strong positive correlation except for 2014 that shows very weak positive correlation (0.1). Year 2017 shows ‘very strong’ positive correlation (0.88).

In history and archeology also, the publications mostly show strong positive correlations between the two variables for study period except year 2016 for which the correlation is positive but weak (0.25) while the publications of the year 2014 show the strongest correlation (1). Table 1 presents the detailed view. Thus, the analysis shows an overall significant positive association between the Altmetrics and citation scores in History and Archeology.

The correlation between the two variables i.e., Altmetrics and citation scores in the discipline of studies in human society range from negative to positive, as presented in Table 1. This depicts a weak association between the two indicators in the said field.

As it is evident from the above data that the relation between altmetrics score and Citation count has changed in the past four years and it has been ranging from negative to positive. But for the more recent articles i.e., those falling under year 2017, all the selected disciplines depict the positive relationship. This may be attributed to the factor that altmetrics is an emerging field and use of social media is now a daily routine of every person including the scholarly world, leading to the tremendous popularity of altmetrics with every passing day, thus, making it an important metrics for the evaluation of research publications in future. Besides, correlation seems profound in the discipline of earth and environmental science which suggest that the type of discipline influences the degree of correlation.

Spearman correlations were also calculated for the altmetric score of select articles in biological sciences with SJR score of journals in which these articles were published (Table 2). Positive correlation was found among the two variables except during 2017 where the negative correlation was found among the variables.

Contrary to the significant association between Altmetrics and citation scores of publications in the discipline of earth and environmental science, association between Altmetrics and SJR in the selected articles ranges from negative to positive, thus not depicting any particular trend.

The association between Altmetrics and SJR in History and Archaeology mostly displayed significant positive association as presented in the Table 2. During 2014 the strongest positive association (1) is observed. The correlations are almost in accordance with the results of Altmetrics and citation scores in the said discipline.

In the category of Studies in human society, correlation between Altmetrics of publications and SJR score of the journals turned to be negative during all the years under study.

The relation between Altmetrics and SJR is ranging from negative to positive in all the selected fields as depicted from data in Table 2. As per the recent data (i.e., 2017), the History and Archaeology and earth and environmental science are depicting the positive relationship. But other two disciplines (Biological Sciences and Studies in human society) show negative correlation.

Thus, it can be concluded that no uniform trend of correlation is seen within the science and arts subject fields.

Altmetrics differs from traditional citation analysis in two main factors.

1. It furnishes the data pertaining the use and consumption of research from outside the realm of academia i.e., from the general public.
2. It gives the opportunity of assessing the impact of other forms of research outputs like blogs, datasets, etc. which citation-base measures fail to acknowledge.

Therefore, while assessing their associations statistically, the findings of the current study reveal that Altmetrics and citation scores differ/vary in different disciplines. The association between the two metrics range from positive to negative. In some subject fields, like earth and environmental science and history and archaeology, the association seems significant. Therefore, in such fields Altmetrics can predict future citation-based impact of the articles. The reason for such a significant correlation between Altmetrics scores and citation scores for disciplines like environmental science, history and archaeology may be because the “non-academic-peer-networks, primarily populated by members of the general public, are much less likely to be interested in esoteric fields of research than in research that connects to popular topics of discussion like climate change”.^[26]

Further, the analysis of the relationship between Altmetrics and citations for recently published articles (i.e., 2017), in all the studied disciplines, confirms to have positive correlations but is relatively weak in biological sciences and human society. This supports the idea that Altmetrics do not reflect the same kind of impact as citations. Thus, Altmetrics, though furnishes a broader view of the research impact, cannot replace citation analysis as a scientific metric, but both employed in combination would enhance the worth of the research evaluation.

Altmetric scores depict a similar trend with SJR as noted with article citations which confirm that prestige of a journal also has a weak correlation with the Altmetrics. But, the presence of Altmetrics is gaining momentum over time and could play an important role by complementing the evaluation of scientific publications.

Thus, this study makes it imperative for the academic community, administrators and policy-makers that Altmetrics should not be employed to replace any existing metrics, rather, it should be utilized in expansion of the traditional metrics to assess and analyze science efficiently. They need to take into consideration that Altmetrics over promotion should not lead to its goal-displacement.^[27] He describes it using Campbell’s law, which states that- “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”