Quality Research Follows the Power Law

Research organizations often use publications and citations as part of their quantitative scholarship assessment. While publications measure the productivity of a researcher, citations reflect the research impact of the scholar.^[1] Publications and citations are tracked in various databases such as Web of Science, SciFinder, Scopus, and Google Scholar. There are various reasons to cite a publication, e.g., to inspire and support a statement, to introduce the field and progress, to credit the contributions, among others. Since these reasons are difficult to distinguish, citations are usually treated as a lump sum popularity score received from peers. Both the number of publications (N) and citations (Nc) have played significant roles in many evaluation procedures including hiring and promotion of researchers, recruiting of graduate students and postdocs, and distribution of awards and grants. Therefore, it becomes important to prudently use these publication and citation data by fully understanding their properties and limitations.

In 2005, h-index was suggested by Jorge E. Hirsch as a unified tool to evaluate both productivity and impact of theoretical physicists.^[2] It was designed to balance the productivity (i.e., number of publications, N) and the impact of the publications (i.e., number of citations received, Nc) by finding the number of papers (H) each of which is cited no less than H times. Due to its simplicity and inclusiveness for both research productivity and impact, it has been quickly adopted in scientific communities beyond physics.

Data show that h-index is an acceptable global index for a research community but has problems to evaluate research of individuals.^[3] As an extensive variable, h-index favors researchers with more publications. Figure 1A shows a thought experiment of four scholars. Scholar c has consistently higher citation per paper than scholar a or b. However, this fact is not reflected by the h-index as c has fewer publications. Even more problematic is for scholar d, who has a much smaller h-index but may have made a breakthrough contribution to the field. If one uses h-index as a threshold to group scholars, d has a higher chance to be excluded from the group. To address these problems, we envisage indices with an intensive property, such as Nc/N or H/N, should be used to evaluate research quality. An analogy for intensive and extensive variables can be illustrated by comparing water and copper. For example, 10 kg water vs 1 kg copper are extensive values that can be changed by adding or removing materials, but their densities 1 kg/L and 9 kg/L are intensive values that should not change after adding or removing materials.

Figure 1:
(A) Thought distributions of four scholars. The x-axis is the publications of four scholars (a-d). The dashed line is where the value of the citations equals the number in the sorted publications. The intersection value between the dashed line and the citation line is the H-index for each scholar. (B) H vs Nc/N for 179,597 scholars from the world’s top 2% researchers in the Scopus database.^[20] The dashed line depicts maximum H value. (C) Histogram of correlation coefficients between H and Nc/N for 100,000 randomly selected pairs from all the top 2% scholars. (D) H vs N, a power law fitting (red), and H = N curve (dashed black linear line). The lower panel is the relative divergence of the data (H/H_N, see Eqn. 1 for H_N) from the fitting where the red line is the linear fitting. (E) H vs Nc, a power law fitting (red), and the H = (Nc)^0.5 curve (dashed, which represents the maximum H one can obtain for a specific Nc). The lower panel is the relative divergence (H/H_C, see Eqn. 3 for H_C) where the dashed red line is the fitted results normalized to one.

There have been efforts to modify and use h-index.^[4] Based on power-law dependence observed between variables such as h-index, citations, and number of publications for individual scholars, institutes, journals, and countries,^[4–19] correction factors of the h-indices for scholars in different fields have been proposed,^[13] and new indices are proposed.^[19] However, all these corrections still use extensive parameters, which varies significantly for a scholar in different career stages and different research fields. In this manuscript, we found that the H-indices of a large group of scholars in all fields also demonstrated the power law distributions against total citations and/or number of publications. Based on this correlation, we propose ratiometric, intensive indices that have the potential to be used to rank scholars in different fields and different stages of their careers.

We analyzed 179,597 researchers (estimated ~2% of total researchers) summarized by John Ioannidis and coworkers.^[20] We found the relatively poor correlation between h-index and the research quality represented as Nc/N (Figure 1B, indicated by the red arrow). As a result, if we randomly choose two scholars from the list, ~31% of time their h-index and average publication quality are oppositely correlated (Figure 1C). Similar result of weak to no correlation between productivity (such as publications) and quality (such as citation per publication) has been observed in the literature.^[1,3] Interestingly, we found that H indices are maximized with ~100 citations/paper among top 2% scientists (Figure 1B). Because the absolute value of average citation per publication of each scholar spans four orders of magnitude for the 2% scholars (Figure 1B) in a typical subfield, there is a need to come up with simpler indices to measure the research quality.

h-index follows a power law relationship with the number of publications or citations. To search for a quality index with a simple and intensive property to measure the research quality, we first investigated the relationship of H with published papers (productivity N) or citation numbers (impact, N_C). h-index is positively correlated with the productivity (Figure 1D) or impact (Figure 1E). The distributions show a power-law like Pareto front^[21] following the scattered data points.

After we plotted the h-index vs number of publications (N) for the top 2% researchers^[20] from all scientific disciplines, we found that h-index vs N can be fit with a power law using a minimalist fitting without free parameters (Figure 1D),

This equation is conceptually different from the correlation Hirsch has proposed in his original paper, H_N=a (N)^1/2, where a is an empirically determined fraction to correct for the maximal reachable H_N,max = (N)^1/2 (similar to dashed curve in Figure 1E).^[2] For all top 2% scientists, n = 0.68. The goodness of the fit R² = 0.36, and the relative divergence of the data to the fitting varies from 0 to 3. We can use the divergence as a parameter (Q_N) to measure the distance of a particular scholar to the fitted “average” line of the top 2% scholars with the same N values,

Q_N is ratiometric and intensive, single-valued with no ambiguity, and positively correlated to H/Nⁿ (SI Figure S10) where n=0.5 or 1. The distribution of Q_N of all top 2% scholars can be fitted with three Gaussian distributions (Figure 2A). The main peak is centered at 1.14 with σ = 0.32.

Figure 2:
(A) Histogram of Q_N in top 2% scientists fitted with three Gaussian peaks. Inset shows H = N curve. (B) The distribution of Q_C fitted with three Gaussian peaks. Insets show the scattered plot of Q_C vs H (left) and the H = (Nc)^0.5 curve (the maximum H value for a particular Nc, right).

Interestingly, h-index vs number of citations (N_C) can be fit with a better power law expression (Figure 1E),

For all top 2% scientists, c = 0.42. The goodness of the fit R² = 0.86, and the relative divergence varies from 0 to 1.5. As discussed under eqn (1), we stress again that this equation is conceptually different from the empirical correlation Hirsch has proposed between H and Nc, i.e., H_C = a(N_C)^1/2, where (N_C)^1/2 is the maximum Hc (see the dashed black line (the Pareto front) in Figure 1E).^[2] The early parts of the two models for H<50 overlap but the rest diverges.

We can also create a parameter (Q_C) based on the divergence by referencing the “average” h-index of the scholars with the same Nc values,

Like Q_N, Q_C is an intensive variable. The distribution of Q_C of all top 2% scholars can be fitted with three Gaussian distributions as well (Figure 2B). The main peak centers at 1.07 with sigma 0.08.

It is significant that the asymmetric distributions in Q_N and Q_C have a mirror pattern (Figure 2). While Q_N shows a tail towards larger values, Q_C depicts a bias for smaller numbers, suggesting that the number of publications should bear less importance (H_N underestimated with respect to h-index) than the number of citations (H_C overestimated relative to h-index) in the h-index calculation. In addition, the asymmetric distribution patterns in Q_N or Q_C make it difficult to quickly evaluate research performance using these two variables.

Since h-index is defined in such a way that both N and Nc contribute to its value, it is not surprising that the h-index somehow increases with either N or Nc. The fact that the fitting is much better in Nc than N suggests it is the citation, instead of the publication number, that contributes more significantly to the h-index, which is consistent with the asymmetric Q_N and Q_C distributions discussed above. If we consider N represents research output that is a function of total research investment and H represents the output of quality work, then, the power-law relationship between H and N implies that many heterogenous pathways are involved to spend the total research investment (will discuss more in the later sections).^[22] By the same token, the power law relationship between H and Nc also suggests heterogenous output for the Nc received. It is conceivable that not all citations positively contribute to the quality work. For example, negative citations may reduce the impact of cited work in the long run by alarming peer scientists. In addition, self-citations may also contribute to the heterogeneous citation distributions that lead to the power law behavior of H vs Nc.

In the simplest form, the average research quality can be represented by citations per published paper (N_C/N) for a given researcher. When we scrutinized Q_N and Q_C, we found Q_N correlates with N_C/N reasonably well (Figure 3A) but bend to anticorrelation for scholars with large N_C/N values (e.g., >500). Q_C carries significant anti-correlation with N_C/N which is inherited from the anti-correlation between H and N_C/N (Figure 3B, indicated by the red circle and arrow). We further propose that a dynamic research quality may be calculated by newly received citations per newly published paper each year for the researcher. This calculation, however, does not distinguish the academic stages of the researcher, nor does it offer a perspective on a comparable basis (e.g., scientists with similar publications and/or citations). As a result, senior researchers gain advantages as new citations received each year are cumulative over all their prior publications.

Figure 3:
(A) Correlation between N_C/N and Q_N, and (B) N_C/N vs Q_C of 179,597 scholars among the top 2% scientists.^[20] The red arrows indicate the anti-correlated regions. (C) The positive correlation between Q₁ and N_C/N. (D) Distribution of Q₁ among all top 2% scholars. (E) The 3D plot of N, Nc, and H with a fitting using eqn. 6. The colored plane surface represents the fitted H_NC results. The lighter blue is above the fitting plane and the darker blue is below. The red arrow indicates the group of scholars with anticorrelated N_C and H values. See SI Figure S2 for views from other angles. (F) H/H_NC distribution fitted by two Gaussian peaks. (G) Distribution of Q₂ and (H) its relationship with N and N_C among all top 2% scholars. See SI Figure S3 for views from other angles.

To address this problem, we introduced a ratiometric research quality index, Q₁ to reflect the reduced quality per paper with respect to that of the “average” from the top 2% scholars,

Here, Q₁ depicts the reduced Nc/N with respect to the global power law distribution constants n and c among the top 2% scholars. The distribution of Q₁ among all top 2% scholars can also be fitted with three Gaussian peaks with the average centered ~1.0 for all top 2% scholars. Compared to H, Q₁ has a better linear relationship with Nc/N (compare Figure 1B and Figure 3C). Further correlation analysis confirms that Q_N or Q₁ correlates with Nc/N (each has ~83% correlation coefficients) better than h-index (~70%) (SI Figure S1 a-c).

However, histogram of the Q₁ from all top 2% scientists in Figure 3D also shows that Q₁ does not follow a perfect Gaussian distribution, which is not surprising as the Q₁ calculation (Eqn. 5) synergizes the unsymmetric biases in Q_N and Q_C (Figure 2) by taking their ratio. To quest for a quality research index with symmetric Gaussian distribution, we reasoned that a unified equation combining Eqns. 1 and 3 may give a more accurate representation of H to both N and N_C. To this end, we arranged the N, Nc, and H in a 3D plot with the 3D fitting using the following minimalist power law expression without free parameters,

For all top 2% scientists, a = 0.12, b = 0.35 (Figure 3E). This correlation agrees with Schubert-Glänzel’s finding in which the journal h-index (H_J) is related to the number of papers (N) in the journal and its impact factor (IF) (H_J ∝ N^1/3 IF^2/3).⁶ Our plot demonstrates the power law relationship among H, Nc, and N with the goodness of the fit R² = 0.89 for the published data among top 2% scholars. Such a fitting is better than that of the power law shown in Eqn. 1 or Eqn. 3. Close inspection on the fitting reveals that the scholars on the left side of the plot are below the plane (indicated by the red arrow in Figure 3E), indicating their h-indices are smaller than the fitted H_NC. This may represent type c and d scholars with high quality publications (Figure 1A), which are not identified by the h-index method. In addition, the relatively narrow Gaussian distributions of the divergence and the improvement in distribution symmetry (Figure 3F) provide a promising direction to generate a more accurate index with more symmetric Gaussian distribution.

Since H can be represented by the power laws of both N and N_C, it is convenient to express an average H_NC′ by taking the geometric mean of H_N and H_C to keep the same units, H_NC′=(H_NH_C)^1/2. From Q_N, Q_C, and H_NC, we then propose another composite ratiometric variable,

to evaluate the quality research of a scientist with respect to the average H_NC among top 2% scientists with similar publication and citation records.

It is clear that Q₂ is an almost perfect Gaussian distribution for the top 2% scholars (Figure 3G). The average values among peers are again designed to ~1.0. The distribution of Q₂ over N and N_C is bent comparing to H (Figure 3H). Unlike H that goes up with the increasing N, the peak distribution Q₂ now is seen at N ~ 100 (Figure 3H and SI Figure S3), which is consistent with the finding that quality research is not correlated with the number of publications.^[1,3] The anti-correlation between quality and quantity becomes obvious when analyzing the relationship between Q₂ and N (anti-correlation: 56%), as well as between Q₁ and N (anti-correlation: 71%) (SI Figure S8), both are higher than that between N and N_C/N for randomly selected two scholars among the top 2% scholars (anti-correlation: 54%). Interestingly, Q indices are found to be positively correlated with H/N ratios (Figure S10). Therefore, the readily accessible H over N values can be used to quickly quantify research qualities.

Since these analyses have been performed on 179597 (top 2%) scientists, it is questionable whether the power laws persist in the rest of the population or in a population with fewer people. To test whether power laws are valid in scientists ranked with different percentiles, we separated the top 2% into two groups of top 1% and 1-2% populations using either the composite score (C-score),^[23] the h-index, the N, or the N_C ranking.^[20] We found that while power law relationships are still valid for populations with different percentiles, the power exponents of n (Eqn. 1) and c (Eqn. 3) do decrease with decreasing percentiles (SI Figure S4). In addition, we also evaluated all 18 disciplines with number of researchers ranging from 113 (Visual Arts and Performance) to 55693 (Clinical Medicine) (Table 1), the power laws are valid for H vs N as well as H vs N_C relationships (SI Figures S5, S6).

Therefore, with these power-law derived Q indices, it is possible to evaluate the publication quality of any researcher with reference to the top 2% scientists. The same quality indices can also be used to evaluate any scientific community as exemplified by the analyses of the 18 disciplines summarized in Table 1.

According to our definition, the Q indices are integrated functions that reflect the quality research (h-index) with respect to the average H obtained from a population. This value is expected to evolve over time for a group of selected scientists with certain age or academia stage, or for individual research fields. However, it does not measure the dynamic research quality needed to interrogate the Q-indices over time for individual researchers. One way to achieve this task is to plot the Q-indices vs time for a researcher. We analyzed the four Q indices over time for an exemplary researcher, Dr. Roger Y. Tsien, in Figure 4, after referencing him against the top 2% scientists in 2021. We observed an increase in all four Q-indices for Dr. Tsien over the years, indicating a stable growth of impacts with respective to the top 2% scientists. Q_N and Q₁ clearly indicate his breakthrough years (development of Green Fluorescence Protein (GFP^[24–26]) when the values climb up to 1.7-1.8, which are significantly above the average values (~1.0) among the world top 2% scientists on the 2021 list (at ~10% percentile of the top 2%). Both Q_N and Q₁ are more stable and indicative than H, N, Nc, or Nc/N, or their respective slopes (SI Figure S9), in distinguishing Dr. Tsien’s research stages. As the geometric mean between Q_N and Q_C, the Q₂ values grow from below the average of the top 2% scientists, quickly to slightly above the average when Dr. Tsien became an independent researcher, which continues to grow over the rest of his career. When Dr. Tsien won the Nobel Prize of Chemistry in 2008, his Q₂ score was 1.6, which places him within top 0.3% percentile of the top 2% scholars in 2021 (600/179697). Between the two Q₁ and Q₂ indices defined by Eqn. 5 and Eqn. 7, respectively, Q₁ is more sensitive to the average quality of the publications whereas Q₂ is more sensitive to the total quality of the publications.

Figure 4:
Exemplary Q_N, Qc, Q₁, and Q₂ analyses of Dr. Roger Y. Tsien’s (Nobel laureate of Chemistry 2008) citation data over time.

The power law behavior of H vs Nc and/or N resembles that of the photoluminescent lifetime of individual quantum dots.^[22] When a single quantum dot gets excited, it will take one of many possible decay pathways to either emit a visible photon (radiative decays) or phonons (nonradiative decays). As a result, the emission lifetime (proportional to the quantum yield) of each particle will follow a distribution with weakly ergodicity feature that can be described with the power-law distribution.^[22] The ensemble behavior of many particles is also power-law dependent rather than a Maxwell-Boltzmann or black-body distribution for well-behaved ergodic particles.^[22,27] These observations prompted us to propose a multi-state, multi-pathway model to delineate the relationship between resource input and research output (Figure 5).

Figure 5:
The M-C diagram of publication states. N and H depict the number of publications and h-index, respectively. “I” represents the total input whereas “a” depicts a proportionality factor.

When a researcher in resting state receives resources (I), such as research funding, space, personnel, materials, etc., they rise to the loaded state from the resting state. The relaxation pathway (attrition) follows from the loaded state, which includes training of personnel, preparation for experiments, consumable expenditures (space, instrument, electricity), etc., before the output state is reached. From the output state, at least three heterogenous relaxation pathways exist to produce a total of N publications. In the first pathway, effort is futile, resulting in no publications. In the second pathway, (N-H) publications are produced for a regular output. In the third pathway, extra effort must be spent to reach the quality state from the output state to generate quality research product (H). Overall, H=Nⁿ, where N is dependent on the function of resource input as N=f(I). Just like the relaxation of excited quantum dots via multiple pathways,^[22] heterogeneous pathways are involved in the relaxation of the loaded state back to the resting state of researchers. Therefore, we anticipate f(I) follows the power law as well.

After analyses on the research record of top 2% scientists, we have revealed that quality research represented by h-index follows a power law relationship with the number of publications and/ or citations. Such a behavior is akin to the Pareto distribution originated from the heterogeneity in the system.^[22,28–30] We have therefore proposed that high quality publication is a result of the heterogeneous relaxation pathways from the loaded state after a scientist has obtained research resources. Such a power law dependency between h-index and publications/citations allows us to propose indices (Qs) with the intensive property to evaluate the quality research (using h-index as a virtual state indicator) of individual scientists with respect to that predicted from the reference group (e.g., top 2% scientists) with the same citation or publication records. These indices address the problem of H-index, which, due to its extensive nature, favors senior scientists with more publications over juniors. We propose to use readily available H/(N)ⁿ or H/(Nc)^c values (where n and c may be approximated to 2/3 and 1/2, respectively, for all scientists) to quickly estimate these Q indices.

Data treatment and analysis method have been included in the manuscript’s main text. The analysis can be done using Excel and MATLAB in this manuscript. The Q values for Dr. Tsien are analyzed from the raw citation data downloaded from Scopus using a home-written MATLAB code to sort publication citation data from year to year to calculate the accumulated citation and publication numbers. These numbers are used to calculate the indices with the equations described in the main text.

We thank Dr. Yuening Zhang and Dr. Tao Xu for beneficial discussion; Korie Sell and Albert Green for the disclosure preparation.