Quartile Prediction and Journal Recommendation Using Deep Learning Models for Artificial Intelligence Articles

The development of Deep Learning models has profoundly impacted different subareas of Natural Language Processing (NLP), particularly since the introduction of the Transformer architecture.^[1–3] In addition, Recommendation Systems have also been significantly influenced by recent advances in NLP.^[4,5]

In particular, the dual task of Journal Recommendation and Quartile Prediction for journals, as part of Natural Language Processing, might also be solved by leveraging Transformers. A Journal Recommendation system can guide users in correctly inferring where to submit a completed research paper, while Quartile Prediction assesses the quality of the paper. Quartiles serve as bibliographic metrics of journal quality, based on citations. Although this hypothesis is debatable, it assumes that better papers receive more citations than others. Even if this hypothesis is not universally true, predicting the quartile of a research paper indirectly predicts its future citations, which a is key feature in research.

In this article, we aim to predict both the quartile and journal of a given research paper in the field of Artificial Intelligence. We used three crucial features: the title, the abstract and the keywords. While the second subtask (Journal Recommendation) has been extensively explored with various models (see Section

2), the Quartile Prediction task remains understudied, at least to our knowledge. The relationship between the two tasks lies in the fact that a good journal predictor is also a competent quartile predictor, or at least performs comparably to the quartile classifier. However, can this generalization hold for unseen data?

This paper is structured as follows. Section 2 is devoted to the literature research. Section 3 presents the Methodology. Results are shown in Section 4, while the Discussion appears in Section 5.

To our knowledge, while the development of journal recommendation systems is an active and rich research area, the formulation of the more specific topic of quartile prediction systems remains unexplored. Nevertheless, our literature research indicates that in the context of journal recommendation, most work has been conducted using classical Natural Language Processing (NLP) techniques, with Deep Learning methods being scarcely implemented.

Beel et al.^[6] conducted a review until 2016, revealing that over 200 articles had been published on journal recommendation. Most of these studies relied on traditional Machine Learning techniques, such as TF·IDF. However, the emergence of the Transformer architecture shifted the landscape. The proposal of the Transformer architecture in 2017 by Vaswani et al.^[1] rapidly became the state-of-the-art for several NLP tasks. Consequently, journal recommendation systems should consider incorporating Deep Learning techniques, especially Transformers.

Feng et al.^[7] implemented a Convolutional Neural Network (ConvNet) to classify 1130 biomedical journals, achieving superior results compared to most commercial journal finders. Notably, they also explored the incorporation of Long Short-Term Memories (LSTMs), albeit with slightly slower performance. It is worth noting that the utilization of journal identifiers, such as the LocatorPlusID, may not be advisable due to potential issues related to data leakage.

Nguyen et al.^[8] also compared the performance of an LSTM architecture and ConvNets, incorporating text embeddings. They proposed an ensemble approach, combining ConvNet with the Scientific Submission Recommendation System for Computer Science (S2RSCS) method, achieving top results in their data. Their technique, named S2CFT, utilized inputs from the title, abstract and keywords. The S2RSCS comprises Logistic Regression and a shallow neural network.^[9] The optimal results were obtained by leveraging a combination of abstracts and keywords, underscoring the relevance of title information. Subsequent refinements have been detailed in their follow-up publication.^[10]

Recent developments have considered the usage of Transformers. For instance, Michail et al.^[11] achieved superior results using the DistilBERT architecture compared to both the BERT architecture and Doc2vec. Liu et al.^[12] also implemented BERT as a classifier and the combination of BERT with an AutoEncoder architecture yielded better results than classical models. Additionally, Hassan et al.^[13] compared BERT, SciBERT and the USE transformers, with the last architecture showing the best results. Hybrid approaches have also been explored. For example, Zhao et al.^[14] implemented a BERT architecture and a Bidirectional Gated Recurrent Unit (BiGRU), achieving better results than classical methods.

In spite of the comparison with the classical approaches, Barolli et al.,^[15] which focused its research on COVID-19-related papers, defends the use of techniques such as TF·IDF and graph-based approaches since their representation are similar to the Transformers (BERT and MiniLM).

Considering the literature review, we fine-tuned the following Transformer architectures:

BERT-base.^[2]
DistilBERT-base.^[16]
SciBERT-base.^[17]
RoBERTa-base.^[18]
XLM-RoBERTa-base.^[19]
GPT-2 medium.^[3]

We used Adam optimizer with a learning rate of 3×10^-5 and Sparse Categorical Cross Entropy as a loss function. In this case, we selected a low learning rate because we were implementing fine-tuning. The rest of the hyperparameters are default in Hugging Face. The training process considered two initial epochs for training and the training continues until decay on the F1-score macro in validation. The rest of the hyperparameters were not changed in the context because we prioritized evaluating different models rather than different configurations of the same models.

For the purpose of comparison, we implemented some classical machine learning experiments. In these experiments, we employed Support Vector Machines (SVM) with a linear kernel. The maximum number of iterations was set to 1×10⁹, with a regularization parameter of 0.9. The data underwent preprocessing, including the removal of non-ASCII characters, alphanumeric words and HTML entities and the conversion of all letters to lowercase.

Text encoding was performed using the TF·IDF technique, employing N-gram ranges from 1 to 5 for both title and abstract SVMs. For the SVM trained with keywords, N-gram ranges from 1 to 3 were utilized. The maximum document frequency was set at 0.7, 0.8 and 1.0 for SVMs trained with titles, abstracts and keywords, respectively. The parameters for the TF·IDF vectorizer and the SVM model were selected after several experiments with different parameters were computed, with this parameter the SVM model had the best performance on the validation set.

For the model ensemble, we opted for a majority voting system. In the event that all three models (title, keywords, abstract) voted for the same class, the global class was assigned. If two of these models voted for the same class, the majority class was assigned. If each model voted for a different class, then preference was given to the model that had performed best during validation.

The acquisition processing was performed by using web scrapping using Elsevier’s Abstract Retrieval API. Additional refinements of the dataset were performed manually. We considered all journals with quartile in the Journal Citation Reports 2022 list with the tag COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE-SCIE.

However, as a result of the web scrapping process, only a few papers are correctly labeled with the abstract and keywords. This process led to three different configurations of the dataset, depending on the specific subtask.

In the first experimental setup, we aimed to predict the quartile by using the title. As a result, we got 54749 papers for the training set, 6844 for the validation set and 6844 for the testing set. In total, 17526 were devoted to the Q1 class, 16721 for the Q2 class, 16921 for Q3 and 17269 for Q4, showing a low imbalance. This dataset was studied with most of the models. In this configuration, the split process was completely random. Each journal has around 466 papers. We considered 147 journals in total. Subtask 1B:

We considered a reduced dataset with the full information of the articles, which are the title, abstract and keywords, using the same scheme of.^[8] In total, the reduced dataset used seven journals. To test the generalization capability of the quartile prediction scheme, we used 5 were devoted to training, one for validation and one for testing. The classes present balance as shown in Figure 1.

Figure 1:
Distribution of the reduced dataset with the quartils as labels.

In this case, the reduced dataset was split differently, using a random scheme. In total, we got 9540 articles in the training set, 1193 in the validation set and 1193 for testing. In this case, the task aimed to predict the journal, given the information of the article, creating a recommender system. It is also worth mentioning that the dataset is slightly unbalanced, as shown in Figure 2.

Figure 2:
Distribution of the reduced dataset with the journals as labels.

Subtask 1A was employed as a preliminary exploration for subsequent experiments. In this context, all considered deep learning models were utilized. Each model underwent training for two Epochs (Ep). However, for F1-macro (F1-m) values surpassing 0.49, a retraining process was implemented until the validation F1-macro demonstrated a decrease. The corresponding results are presented in Table 1.

In the table, we observe that cased models were generally slightly better, or at least as good as their uncased counterparts. Contrary to intuition, BERT-cased, DistilBERT-cased and SciBERT-scivocab-cased achieved superior comparative results, even outperforming GPT-2, XLM-RoBERTa and RoBERTa. We conducted evaluations on the testing set with the best model (DistilBERT-base-cased), resulting in an F1-score macro of 0.48397, accuracy (Acc) of 0.48758 and a weighted F1-score (F1-w) of 0.48422. For the prediction of two classes (Q1-Q2 versus Q3-Q4), we obtained an F1-macro of 0.69022, an accuracy of 0.74883 and a weighted F1-score of 0.75582. The confusion matrix for the testing set is provided in Figure 3.

Figure 3:
Confusion matrix of the best model (DistilBERT-base-cased) with the testing set.

In the subsequent experiments, we exclusively considered the best models from Sub-task 1A (BERT, DistilBERT, SciBERT) while introducing the use of Machine Learning algorithms. For the Quartile prediction task with full information, it’s important to note that we kept journals separate in the training, validation and testing sets, focusing on studying the generalization capabilities of the predictor. Unfortunately, due to VRAM limitations on the A100, we were only able to evaluate abstracts with the smaller models (DistilBERTs and Machine Learning algorithms). In this scenario, we measured the F1-score₂ performance, addressing the binary classification problem by merging classes Q1 with Q2 and Q3 with Q4. Table 2 summarizes the results of the models using the title as input. It is noteworthy that DistilBERT-base-uncased achieved the best performance in F1-macro, consistently delivering good results in the binary classification problem. However, all the studied models performed worse than random in the multi-class problem, a challenge persisting throughout the entire evaluation.

In Table 3, when using keywords as inputs, DistilBERT-base-cased demonstrated better results in the F1-macro for the multi-class scenario, while DistilBERT-base-uncased exhibited superior results in the F1-macro for the binary classification. In Table 4, the reverse situation arises when utilizing abstracts as inputs: DistilBERT-base- cased presents better results in the binary problem, whereas DistilBERT-base-uncased showed superior results in the multi-class problem. In general, we observed a trade-off between an increase in binary metrics and a decrease in multi-class metrics.

Results of the ensemble are presented in Table 5. In this case, it is evident that only the SVM method achieved better results than random. However, it exhibits slightly lower performance in the binary problem. All ensembles performed worse than the best base models.

Table 6 displays the application of the best models on the validation set, based on two indicators: the best model in the multi-class problem (SVM trained with abstracts) and the best model in the binary problem (DistilBERT-base-cased trained with abstracts). The quantitative results reveal a dramatically different scenario compared to the validation set, despite the identical data generation process. In this case, performance is notably worse than random in the multi-class problem but demonstrates results similar to random in the binary problem. This suggests that the best models struggle to generalize effectively.

In the case of the Journal recommendation, we utilized similar models to those employed in Subtask 1B. We conducted evaluations using three different inputs: title,

keywords and abstract. For the title input, the best-performing model was BERT- cased with an F1-macro of 0.54883, a Top-5 accuracy (Acc₅) of 0.90277 and a Top-10 accuracy (Acc₁₀) of 0.96982, exhibiting slightly better performance than DistilBERT, which was the top-performer in Subtask 1A (see Table 7). However, when using key- words, DistilBERT-cased outperformed other models (see Table 8). In evaluations with abstracts, DistilBERT-uncased demonstrated superior results and emerged as the overall best model (Table 9).

In this specific case, the ensemble proved beneficial for the SVM, although it did not enhance the performance of the Deep Learning algorithms (see Table 10).

In the testing set (Table 11), we saw a similar scenario than in the validation set. In this case, DistilBERT, in general, presents healthy metrics and just slightly worse than in the validation set, indicating that the model is capable of correctly classifying unseen papers to the corresponding journal. This result differs from the results of Subtask 1B. The corresponding confusion matrix is presented in Figure 4.

Figure 4:
Confusion matrix of the best model applied in the testing set (DistilBERT-base-cased).

In the testing set (see Table 11), we observed a similar scenario to that of the validation set. In this case, DistilBERT generally exhibits robust metrics, slightly inferior to those in the validation set, suggesting the model’s capability to accurately classify unseen papers into the corresponding journals. This result notably differs from the outcomes of Subtask 1B. The corresponding confusion matrix is presented in Figure 4.

In general, the application of different Machine Learning algorithms and recent Deep Learning models showed two relevant conclusions about the studied subtask:Model Ep F1-m F1-w Acc Acc₅ Acc₁₀ DistilBERT-base-cased 4 0.5548 0.54894 0.56245 0.86421 0.95809

The journal classification process is plausible, even with a high number of classes.

The quartile classification process remains open.

Various approaches have been attempted for journal prediction, but quartile classification has not been extensively explored in the literature. This may be due to the apparent challenge of predicting quartiles based on current technologies. Why did all models fail in a different task with the same data and fewer classes? It is intriguing to note that, in some cases, the trained models even performed worse than random ones, providing potential insights into the challenges of generalization.

As Subtask 1A demonstrates, it is indeed possible to predict the quartile of journals that appeared in the training set. In this regard, the confusion matrix of the best model for this subtask (see Figure 3) shows that similar quartiles are predicted similarly, apparently showing that the classifiers fail to distinguish similar quartiles. Nevertheless, considering the results of Subtask 1B, perhaps this confusion was produced by the presence of similar journals in similar quartiles, but this idea is debatable.

Generalizing to unseen journals proves to be more challenging. None of the studied classifiers produced good performance in the complete subtask with the four quartiles, although the distinction between Q1-Q2 and Q3-Q4, was possible in the validation set. However, the best classical Machine Learning classifier and the best Deep Learning algorithm failed to generalize to the testing set, which did not include the journals of the training set. In general, since most methods performed similarly, the main problem may rely on the insufficiency of the information to accurately predict the quartile without depending on predicting the journal. By applying sentence embeddings with the all-MiniLM-L6-v2 model^[20] and Principal Component Analysis, we visualize the cluster of data points labeled with quartiles (Figure 5 in the validation set and Figure 6 in the testing set) and with journals (Figure 7). It is evident that there is a lack of structure in both quartile distributions, but some clusters emerge when labeled by the journal. In Figure 7 we can see that some classes are clusterized in different regions, although we can comprehend that there are many outliers. For example, a journal devoted to Natural Language Processing (for instance, Natural Language Engineering) would contain most journals with titles semantically similar to the main topic, but this does not exclude the possibility that one NLP article cannot be found in a very different journal, due to the possibility of finding special issues, the presence of interdisciplinarity, or the existence of very general journals like Artificial Intelligence. This might explain why the metrics in Subtask 2 are low in comparison with other NLP tasks.

Figure 5:
Scatter plot of different journals labeled with the journal quartile using the validation set. Orange dots represent the Q1 class, red dots represent the Q2 class, violet dots represent the Q3 class, while blue dots stand for the Q4 class.

Figure 6:
Scatter plot of different journals labeled with the journal quartile using the testing set. Orange dots represent the Q1 class, red dots represent the Q2 class, violet dots represent the Q3 class, while blue dots stand for the Q4 class.

Figure 7:
Scatter plot of different journals labeled with 14 different journals.

In the case of quartile prediction, the problem is even bigger. If we have a perfect journal recommendation system, we do have a perfect quartile prediction algorithm, but it might not be able to generalize for unseen journals. Figures 5 and 6 depicts the situation. Although there are some areas with one color, dots from different classes can be found in the same regions. In both figures, we see a similar distribution of the classes, but the quantitive results suggest otherwise. In general terms, the task of predicting the quartile might need more information than the one provided for this problem.

Another important explanation for this phenomenon can be deduced from the metrics. As observed, some models performed better for the binary classification problem while others excelled in the multi-class problem. Moreover, clear results lower than random or trivial algorithms surfaced. For this reason, it can be concluded that the training process biased the classifiers and, in all cases, quartile prediction is an indirect form of journal prediction.

In this article, we explored two closely related text classification problems: quartile prediction and journal classification for a given paper. Despite the apparent connection, these two tasks are more intertwined than initially perceived. As demonstrated in the literature, the subtask of journal estimation has been reasonably well-executed. Comparing various approaches, we implemented recent Large Language Models, including GPT-2. Surprisingly, more modest models outperformed larger ones and classical approaches also exhibited relatively good performance when compared with Transformer architectures.

In the task of quartile prediction, the results are controversial. The initial experiment (Subtask 1A) showed that it is feasible to train Machine and Deep Learning models for this specific subtask. In this instance, we combined journals in the training, validation and testing sets, achieving relatively good results, at least better than random. However, upon separating journals (Subtask 1B), we observed that quartile prediction becomes challenging. This suggests that the quartile prediction process indirectly predicts the journal to classify the article’s quartile. However, this process struggles to generalize, or at least, it is not achievable with the studied methods. Despite employing several state-of-the-art models, the results appear inconclusive.

It seems that current classifiers face challenges in predicting quartiles effectively, potentially due to insufficient data or indistinguishable classes. Nonetheless, we assert that this topic warrants further research, offering the potential to pave the way for improvements in the automated evaluation of articles.

The authors wish to thank the support of the Instituto Politécnico Nacional (COFAA, SIP-IPN, Grant SIP 20240610) and the Mexican Government (CONAHCyT, SNII).