Predicting Reviewers’ Decisions in Scientific Submissions through Linguistic Analysis

Predicting the decisions of reviewers in scientific submissions based on the textual content of their reviews presents a novel challenge in the domain of computational linguistics and data science. This task involves understanding the nuanced and subjective elements of review texts to anticipate the final verdict of acceptance or rejection. There is, however, a need for a comprehensive model that integrates advanced text analysis techniques with machine learning algorithms to predict the decisions of scientific reviewers. By focusing on the linguistic and semantic features inherent in review texts, we endeavor to explore the complex interplay of factors guiding reviewers’ decisions, aiming to contribute to a more transparent, efficient and fair peer review process. Despite the inherent complexity due to various factors such as the multidimensional criteria used by reviewers, subjectivity, bias and the lack of standardized evaluation metrics, recent advances in machine learning and natural language processing have opened new avenues for exploration in this field.

Existing studies have laid the groundwork by focusing on aspects such as identifying the factors predicting the quality of peer reviews,^[1] the influence of biases^[2] and the exploration of methods to enhance the fairness and reliability of the peer review process.^[3,4] However, these studies primarily concentrate on the broader aspects of peer review quality and the systemic features of the peer review process, rather than directly predicting reviewers’ decisions from text reviews.

Recent efforts have shifted towards leveraging textual analysis and machine learning algorithms to predict the helpfulness of reviews, a proxy to understanding the underlying decision-making process of reviewers.^[5,6] These approaches demonstrate the potential of computational methods in extracting valuable insights from the text, highlighting the feasibility of predicting reviewers’ decisions in scientific submissions.

This paper investigates the application of computational models and Natural Language Processing (NLP) techniques to predict the outcomes of peer review based on the textual content of reviews. Utilizing Word Space Models and the Linguistic Inquiry and Word Count (LIWC) dictionary, the study effectively categorizes reviews into binary outcomes (accept or reject). The research showcases a significant improvement in predictive accuracy over simpler baseline models, demonstrating the potential for these tools to streamline editorial decision-making by quickly filtering out likely rejections.

Furthermore, the analysis provides deeper insights into the linguistic patterns that correlate with reviewer decisions, identifying specific words and phrases that are predictive of outcomes. This contributes to the broader fields of computational linguistics and Explainable AI (XAI) by offering methods to objectively analyze subjective content and by simplifying complex decision processes into understandable models. Such advancements not only aid in enhancing the efficiency of editorial workflows but also in training new reviewers and in guiding authors on how to better tailor their submissions to increase the likelihood of acceptance.

Predicting decisions from texts is an interdisciplinary field that merges machine learning and Natural Language Processing (NLP) techniques to analyze and predict outcomes from textual data. This area has seen significant applications across various domains such as judicial decisions, medicine and text analytics.^[7–10]

In the judicial domain, researchers have explored the use of Support Vector Machines (SVMs) to predict the decisions of the European Court of Human Rights, demonstrating the potential of machine learning in understanding complex legal texts and outcomes.^[7] Similarly, the work by Visentin et al.^[8] introduced a new ensemble classifier that provides a statistically rigorous approach for predicting judicial decisions, emphasizing the importance of combining multiple models for enhanced prediction accuracy.

The integration and classification of legal documents have also been explored, with studies focusing on specific court cases, such as those of the French Supreme Court^[9] and the US.^[10] This indicates a growing interest in applying text analysis techniques to legal documents for predictive purposes.

The study “Does Reviewer Training Improve the Quality of Peer Review? A Randomized Controlled Trial”^[11] examines if training enhances peer review quality in biomedical research. Through a randomized controlled trial, it found that training interventions, including workshops and feedback, did not significantly improve review quality as judged by editors and authors but increased reviewers’ confidence. This suggests formal training boosts confidence but has a limited effect on the actual quality of peer reviews.

A study by Li et al.^[12] investigated the impact of reviewers’ words on their helpfulness ratings, developing a novel prediction model to address this issue. Similarly, Kavousi and Saadatmand^[13] analyzed review texts to evaluate and predict ratings, focusing on the influence of lexical and sentimental features on prediction accuracy. These studies highlight the potential of text analysis in predicting reviewer helpfulness, setting a foundation for further exploration in predicting reviewer decisions for scientific work submissions.

Historically, the focus has been on improving the efficiency and fairness of the peer review process, with less attention given to understanding the predictive elements of reviewer decisions based on textual analysis. Studies in related fields have demonstrated the feasibility of using machine learning and Natural Language Processing (NLP) techniques to glean insights from text data, suggesting a promising avenue for enhancing the peer review process.^[14]

The challenge, however, lies in the nuanced nature of scientific reviews. Unlike medical decisions, which are often based on historical data and well-established guidelines, reviewer decisions are influenced by a complex interplay of factors including the novelty, rigor and relevance of the work, as well as the reviewers’ own expertise and biases. This complexity necessitates a sophisticated approach to predict reviewer decisions, one that can account for the varied and intricate nature of review texts.

This research aims to bridge the gap in the literature by exploring the potential of text analysis and machine learning models to predict the decisions of scientific reviewers. By analyzing the textual content of reviews, we seek to identify patterns and features that correlate with reviewer decisions, thereby offering insights that could streamline the review process and enhance the overall quality of scientific publications. In doing so, we contribute to the broader discourse on improving the transparency and efficacy of peer review, a critical mechanism in the advancement of scientific knowledge.

The most commonly used machine learning techniques for analyzing text data and classifying it into decisions include:

Naive Bayes: A probabilistic classifier that uses Bayes’ Theorem to calculate the probability of each tag for a given text, predicting the tag with the highest probability.^[15,16]

Logistic Regression: A supervised learning method that uses logistic functions to predict the probability of a text belonging to a specific class.^[17]

Random Forest: An ensemble learning technique that combines several decision trees to improve the accuracy of predictions.^[17]

Deep Learning: A family of algorithms that use neural networks to learn representations of text data and make predictions.^[18]

These algorithms are often used in conjunction with Natural Language Processing (NLP) techniques such as TF-IDF vectorization, word embeddings and named entity recognition to transform text into a format that can be understood by machine learning models.^[19]

To analyze the reviews, we employed the Linguistic Inquiry and Word Count (LIWC) tool,^[20] categorizing words into psychologically meaningful groups. Table 1 is a summary of key LIWC characteristics.

The algorithm for feature reduction using subset selection, as detailed in,^[21] was utilized to condense the sets of attributes. This supervised method searches for optimal subsets of attributes that yield effective classification performance. The process of subset search is conducted using the BestFirst strategy.

The BestFirst approach begins with an initial empty attribute set and sequentially incorporates attributes. It also marks backtracking checkpoints to revert to a previous state if the Addition of new attributes fails to enhance classification results.

In our implementation, the algorithm was configured to consider adding up to five nodes before it backtracked to a previously saved point.

For our experiments, we used review data from the MICAI 2023 conference, organized by the Mexican Society for Artificial Intelligence (SMIA). This conference is a prominent international gathering in the field of Artificial Intelligence (AI). Its significance lies in its focus on both theoretical and practical aspects of AI, providing a platform for researchers, practitioners and educators to present and discuss the latest advancements, trends and challenges in the field. The conference covers a broad range of AI topics, including machine learning, computer vision, robotics, natural language processing and AI applications in various domains.

The 2023 edition of MICAI received 117 submissions, from which 60 were accepted. Each work had an average of 2.17 reviews. There were 80 reviewers and each reviewer wrote an average of 3.1 reviews. In total, he has 248 reviews. Each review had the following structure:

Q1 (Please write a short summary of the paper).
Q2 (Describe this paper’s strengths).
Q3 (Describe areas to improve in this paper).
Q4 (General comments for authors).
Q5 (Please select your overall recommendation).

We concatenated all of these questions in a string and used it as the input to our classifier to predict one out of 5 possible outcomes (per reviewer). Table 2 shows the class distribution. See the Appendix for examples of reviews.

Now we aim to characterize the reviews attached to each one of the possible outcomes. First, we will analyze the possible classification into 5 classes. Then, we will reduce our number of classes to two, mapping Accept and Weak Accept to Accept (class 1) and Borderline, Weak Reject and Reject to Reject (class 0). Column ‘Two classes’ in Table 2 shows the distribution of classes when the 5 classes are reduced to 2.

Table 3 shows the results of classification into 5 classes (weak accept, accept, weak reject, borderline and reject) given the input of reviews, represented as a WSM vector of 2445 different words. All evaluations were performed using a 10-Fold Cross Validation. Results are only slightly above the baseline, which is assigning all results to the largest class (weak accept), resulting in a recall of 0.323; whereas the greatest recall is obtained by Naive Bayes, improving only 3.2% on the baseline-recall only, as other measures do not apply for the baseline. Note that Borderline seems difficult to characterize, as both Naive Bayes and Random Forest completely fail to classify its reviews.

Reducing the feature set, as described in Section “Feature selection” in “Preliminaries”, we obtain 8 features that can be used to classify reviews into 5 classes. Selected features are: 2023, not, position, publication, since, aids, fails and had.

Results are presented in Table 4. In terms of recall, there is an improvement compared to the baseline (0.323). It is important to note, however, that the Weak Reject and Borderline classes are predominantly unaddressed by all classifiers.

In order to see the effect of each of these features on each class, Table 5 shows the coefficients calculated by Logistic Regression for each class, while Figure 1 shows the distribution of words’ values associated to each class. The word ‘not’, for example, is positively associated with class reject, while negatively associated with accept.

5-Class classification is possible; however, performance is rather low. That is why we decided to group 5 classes into 2: rejected (0) and accepted (1), as it was shown in Table 2. Next sections detail results obtained using two classes.

Figure 1:
Feature analysis for WSM reduced set for 5-Class classification.

Table 6 shows result for the two-class classification using WSM features. A vocabulary of 1007 different tokens was used.

The algorithm for feature reduction using subset selection, as detailed in Section “Feature selection” of “Preliminaries”, was utilized to condense the set of 1007 WSM attributes. Attributes were reduced to 25. Results of classification based on these attributes are shown in Table 7.

As introduced in Section LIWC characteristics analysis, we used LIWC to characterize each one of the reviews. Table 8 shows a fragment of the input data to the classifier after the LIWC analysis.

Using the LIWC features alone, was not powerful alone to represent reviews for classification, as can be seen in Table 9. All tests were performed using 10-fold cross validation.

The algorithm for feature reduction described in Section “Feature selection” in “Preliminaries” Surprisingly, for this dataset, the 70 LIWC features were reduced to only 2:

Negation words (no, not, never, …) 57 in dictionary.
Exclusive words (but, without, exclude, …) 17 in dictionary.

With this information, obtained results are shown in Table 10.

The Logistic Regression coefficients for an accepting review were: -0.49+0.02 * [negation words]+0.07*[exclusive words].

Figure 2 shows the Gaussian curves corresponding to the selected features for the classes 0 (reject) and 1 (accept). It can be seen that fewer negation words and fewer exclusive words likely conform an accepting review.

Given the previous results, we applied the feature selection procedure described in Section “Feature selection” in “Preliminaries” to the 1007+70 features (WSM+LIWC), resulting in 18 features (Table 11).

Figure 2:
Gaussian curves for selected LIWC features to classify reviews.

It is interesting to note that only 1 LIWC feature was selected (Exclusion words), whereas the other words are specific words (from WSM). Results are shown in Table 12.

Table 13 summarizes the best results for the different feature sets and classifiers used to classify reviews in two classes. This might yield the idea that the best results are obtained by using the reduced set of 25 words; however, it is important to consider that these words were selected considering the whole dataset and thus, if new information is added, it is probable that there will be OOV (out of vocabulary) words. Perhaps a more stable solution would be using LIWC, as unseen words can be still be covered by the LIWC dictionary.

For comparison purposes, we have implemented BERT and RoBERTa classifiers as well, based on the string input of reviews and two output classes. BERT was tested with several epochs and, despite reaching 0.1753 losses, classification results are around 0.6 and 0.62, while RoBERTa reached values around 0.66. Please note that this is not a direct comparison, as the rest of classifiers were evaluated using a 10-fold cross-validation strategy, while BERT was evaluated against a randomly selected 20% of the dataset.

However, we found consistently similar values after several runs (around 10).

This study systematically examines the efficacy of various classifiers and feature sets in the context of review classification, initially across five defined classes and subsequently within a binary framework. The investigation commenced with an analysis of a Word Significance Measure (WSM) vector encompassing 2445 distinct terms, culminating in performance metrics modestly surpassing a baseline established by assigning all outcomes to the most populous class, ‘weak accept’, which achieved a recall of 0.323. Notably, the Naive Bayes classifier demonstrated a slight improvement over this baseline, enhancing recall by 3.2%. This increment, albeit minimal, underscores the inherent difficulty in accurately classifying reviews, particularly within the ‘Borderline’ category, a task at which both the Naive Bayes and Random Forest classifiers were unsuccessful.

Further analysis entailed reducing the feature set to eight significant terms, resulting in a marginal increase in recall, thereby indicating the challenges in classifying ‘Weak Reject’ and ‘Borderline’ reviews remain substantial. This outcome suggests that both extensive and significantly reduced feature sets may not capture the nuanced sentiment embedded within review texts.

Adopting a binary classification approach, which consolidates the initial five categories into ‘accept’ and ‘reject’, streamlines the classification process and aligns with the practical requirements of review analysis. The exploration of Linguistic Inquiry and Word Count (LIWC) features, though limited in isolation, revealed a nuanced dimension when combined with WSM attributes, leading to a refined set of 18 predictive features. This feature set, particularly emphasizing negation and exclusive words, provides a framework for dissecting review sentiments, as delineated by logistic regression coefficients and Gaussian distribution analyses. Moreover, the comparison of traditional classifiers with advanced neural network-based models, specifically BERT and RoBERTa, highlights the advancements in text classification methodologies. Despite inherent variabilities in review datasets, these models demonstrated recall metrics approximately 0.62 for BERT and 0.66 for RoBERTa, indicating their robust- ness and adaptability. It is important to note, however, that direct comparisons between these advanced models and traditional classifiers are complicated by differences in evaluation methodologies. Despite the apparent simplicity of the utilized features, the classification outcomes, particularly within a binary framework, demonstrated a discernible improvement over the established baseline. This enhancement, however, was modest in the context of a five- class classification system, underlining the inherent challenges in employing such simplified methodologies for nuanced text analysis.

The endeavor to encapsulate the diverse perspectives and writing styles of 80 individuals underscores a significant challenge in review classification. The variability in individual criteria and modes of expression complicates the development of a universally applicable model. Nevertheless, achieving an improvement of 6.5% over the baseline represents a non-trivial advancement in this domain, suggesting that further refinement and exploration of these methods are warranted. Particularly, the reduction to a mere two features from the extensive suite provided by LIWC offers intriguing insights into the efficacy of focused linguistic features in review classification.

Furthermore, this research undertook an exploratory analysis within the realm of Explainable Artificial Intelligence (XAI), focusing on linguistic characteristics. This analysis illuminated a set of categories and specific lexemes that significantly contribute to the classification process. Notably, the words “not,” “since,” and “had” emerged as particularly salient, each offering unique insights into the reviewers’ evaluative processes. For instance, the frequent use of “had” may indicate reviewers’ emphasis on unmet requirements or actions necessary for acceptance.

These findings not only highlight the potential of simple, dictionary-based approaches for text classification but also underscore the complexity of capturing the multifaceted nature of human thought and expression. The modest yet significant improvements observed in this study advocate for continued investigation into linguistic feature-based models. Such efforts could further refine our understanding of textual analysis and enhance the accuracy and interpretability of review classification models, aligning with the broader objectives of XAI in providing transparent and understandable decision-making processes.

The authors wish to thank the support of the Instituto Polite’ cnico Nacional (COFAA, SIP- IPN, Grant SIP 20240610) and the Mexican Government (CONAHCyT, SNI).