Comparison of Stanford Encyclopedia of Philosophy and Wikipedia Articles’ References: In Search of Evidence for Wikipedia Credibility

The encyclopedia is a written summary of knowledge that in the form of an article, describes all topics of human interest briefly and comprehensively.^[1] The first encyclopedias were prepared for constant study by a single individual and were provided in a textual form for educational purposes therefore they were different from today’s encyclopedias, which are mainly used as a source of information and are the result of teamwork. The new encyclopedias were largely the result of a cultural movement known as the Enlightenment and the desire to produce and distribute those encyclopedias worldwide.^[2] New encyclopedias were often printed, but the development of computers and the Internet in the late twentieth century led to the spread of electronic publishing and consequently, the emergence of electronic encyclopedias. Electronic encyclopedias became popular thanks to their interactive, multimedia, updating, search, and access capabilities, which made the situation difficult for printed encyclopedias. There are many e-encyclopedias, most of which are available over the Internet for free or shared. Wikipedia is a type of online electronic encyclopedia that was formally launched in 2001 by Jimmy Wales and Larry Sanger.^[3] This encyclopedia can be edited by anyone and with its rapid and fundamental growth, it has become a widely used resource.^{[4, 5]}

Wikipedia is based on a new Web technology called Wiki, a service software that allows users to create or modify content on web pages using web browsers. Until March of 2021, Wikipedia contains more than 55 million articles in 319 different languages, of which more than 6.3 million are in English.^[3] In addition to the increasing usage of Wikipedia by the general public, its popularity is growing among the academic communities, and citing Wikipedia in academic research confirms this case.^[6–10] For example, the results of a study by JamielNiak and Aibar (2016) showed that the number of citations to Wikipedia in the first year of its start-up in 2001, was only one in Scopus and Web of Science, but in 2014 the number of citations to Wikipedia in the indexed documents of Scopus and Web of Science reached by 12798 and 1805 respectively.^[10] Despite this popularity, there are concerns about the reliability and credibility of Wikipedia’s content,^[11] and examining the credibility of these contents has always been a topic of academic research. The reason for these concerns is probably related to the way by which Wikipedia content is created. Both experts and non-experts can participate in the preparation and editing of Wikipedia articles.^[12] Although the articles must be written within the framework of the Wikipedia statute and should respect the principles of neutrality, the lack of professional supervision and peer review system which is common in modern encyclopedias increases the concerns about the reliability and credibility of Wikipedia’s content. Sensitivity to content credibility increases, especially when they are going to be cited in academic researches. This study aims to examine the credibility of Wikipedia content by using a bibliometric technique called bibliographic coupling. This technique is proposed by Kessler (1963) and states that whenever two documents cite one or more common documents in their source list, the two documents form a bibliographic coupling.^[13] The more references the two documents have in common, the greater the bibliographic link and similarity of their content. The similarity of Wikipedia’s content with the specialized encyclopedias can be considered as one of the factors in examining the credibility of Wikipedia. Accordingly, this study intends to compare the overlap of references of Stanford’s encyclopedia entries with Wikipedia, as well as to analyze the references of two encyclopedias from different aspects, to examine the credibility of Wikipedia’s specialized content. In this regard, the questions are as follows:

What is the frequency of articles’ references in Wikipedia and Stanford Encyclopedia of Philosophy?
How much is the degree of overlap between articles’ references of Wikipedia and Stanford Encyclopedia of Philosophy?
How long is the half-life of articles’ references of Wikipedia and Stanford Encyclopedia of Philosophy?
What is the type of articles’ references of Wikipedia and Stanford Encyclopedia of Philosophy?
How is the language distribution of articles’ references of Wikipedia and Stanford Encyclopedia of Philosophy?
How is the accessibility status of articles’ references of Wikipedia and Stanford Encyclopedia of Philosophy?
What is the frequency of citations to articles’ references of Wikipedia and Stanford Encyclopedia of Philosophy?

Wikipedia has been the subject of researches for decades. A study by Park (2011) on researches about Wikipedia, as well as the number of citations to Wikipedia based on documents indexed in the Web of Science and Scopus databases, showed that since the startup of Wikipedia in 2001 the number of researches done on Wikipedia as well as the citation to Wikipedia has been increased. Also, according to the results of that study, academic researchers are the main participants in conducting research on Wikipedia as well the most people who cite Wikipedia. Most of the contributors to Wikipedia research are computer, information, and math scientists, and most of the researchers who have cited Wikipedia are computer, information, and social scientists.^[7] Mesgari et al. (2015) systematically reviewed 110 studies related to Wikipedia content. In addition to identifying the main process of research, research area, research design, source, and data collection methods in the conducted researches, they found that two important and main processes of these researches are Wikipedia content quality (including comprehensiveness, accuracy, readability, and capability) and the size of Wikipedia.^[14] The credibility of Wikipedia is one of the main processes of research on Wikipedia. Claes and Tramullas (2021) suggest that credibility of Wikipedia has received special attention in studies on credibility of information on the web. Their systematic review revealed that various models and analysis techniques have been used in study of Wikipedia credibility. They found that assessments of credibility are influenced by users’ social and educational contexts.^[15] Below are studies which have examined Wikipedia credibility from different aspects.

Some of the researchers conducted research in response to concerns about Wikipedia’s accuracy. These studies compared Wikipedia entries with those of credited encyclopedias and concluded that the accuracy of Wikipedia was comparable with major encyclopedias.^[4,16-18] Giles (2005) Compared 42 Wikipedia and the Britannica encyclopedia entries based on the judges’ opinions to find out the validity of Wikipedia and answer the question that whether free Wikipedia encyclopedia can be as valid as the Encyclopedia of Britannica? In this study, each expert examined the entries of a single topic in these Encyclopedias. it was found that the average number of incorrect scientific entries was 4 in Wikipedia and about 3 in Britannica, which showed that Wikipedia was comparable with reputable encyclopedias like Britannica.^[4] Chesney (2006) asked 258 researchers, including Ph.D. students and research assistants, to determine the credibility of authors and their articles by reading Wikipedia articles. The 69 researchers who agreed to participate in the study were divided into two groups, one of which was given a random article and the other group was given an article in their profession. In general, only 13% of the articles contained errors and there was no difference between the two groups in terms of the author’s credibility, but the group that evaluated the specialized articles assured that articles were more valid.^[16] It should be noted that these results are not sufficient to support Wikipedia as a reliable source, as the sample was small. Rector (2008) used a content analysis method to compare nine Wikipedia articles with those of Britannica and the American Historical Dictionary and the American National Biography. His goal was to compare the accuracy of Wikipedia with three other sources. The results showed that the accuracy of Wikipedia content was 80% and the accuracy of the other two sources was about 95-96%. The results of this study do not support the claim that Wikipedia is less credible than other sources.^[17] Flanagin and Metzger (2011) had a different approach to examine users’ perceptions of Wikipedia’s credibility as opposed to the online version of the Britannica. The survey, which was conducted between two groups of people aged 11-18 and adults 18 and older, showed that although Wikipedia is common and people trust Wikipedia as a source of information, they doubt its validity. The results also showed that Wikipedia content is more believable if it is presented on the Britannica platform than when it is presented on the Wikipedia website itself.^[19] Recently, Sun, Yang, and Zheng (2021) compared the quality and updateness of information on liver disease in Wikipedia (in English) and the Baidu Baike Encyclopedia (in Chinese). The results showed that the quality of articles and the reliability of Wikipedia information on liver disease are better than information on Baidu Baike, however, the quality of treatment options provided in both encyclopedias is not desirable. Also, the interval for updating entries in Baidu was significantly longer than in Wikipedia.^[20]

Other researches have focused on citing Wikipedia articles in academic publications, which could be a sign that the academic community allocated credit to Wikipedia’s articles. Brazzeal (2011) examined the number and manner of citations to Wikipedia in chemistry journals from three major publishers over a period of five years and the results of this study showed that most articles had more than one citation to Wikipedia.^[6] Tohidinasab and Jamali Mahmoui (2013) studied 602 samples of English-language articles indexed on the Scopus database in 2007 and 2012, citing Wikipedia articles. Findings showed that the most important motivation to cite Wikipedia articles was to provide general information, definition, statistics, and concepts. The number of citations to Wikipedia in 2012 was higher than in 2007 and citations was often to introduction, main text, materials and methods, and theoretical sections. The cited articles were often research type and topics cited in Wikipedia were related to the computer, internet, and chemistry. They concluded that the number of citations to Wikipedia is increasing in quantity and variety and there is a difference between scientific domains in terms of motivation and place of citation.^[11] According to the data from the Scopus and Web of Science databases, Bould et al. (2014) examined the citations of scientific medical journals to Wikipedia. The results showed that 1433 full-text articles from 1008 medical journals indexed in Medline, PubMed, and Embase had a total of 2049 citations to Wikipedia. The frequency of Wikipedia citations has increased over time and citations have occurred since December 2010. More than half of the citations were definitions and descriptions and citations were not limited to low-impact journals, but also high-impact journals were cited to Wikipedia.^[8] Tomaszowski and McDonald (2016) also used the Web of Science database to examine the process of citing Wikipedia in scientific journals between 2002 and 2015. The results showed that citations to Wikipedia in scientific journals have been increased since 2002. They also found that Wikipedia was cited not only by open access journals, low-quality journals, low-quality academic institutions, and low-income economies but also by high-impact journals and researchers from leading scientific institutions. Scientists indeed confirm Wikipedia credibility by citing Wikipedia. Editors and reviewers of journals that accredit citations to Wikipedia, implicitly consider Wikipedia to be a credible source.^[9] Singh et al. (2020) took a different approach and provide a comprehensive dataset of Wikipedia references, consisting of 29.3 million citations in 6.1 million English Wikipedia articles. These references were classified into three categories: books, articles, and web content. 4 million references were made to scientific publications with recognized and valid identifiers such as DOI, PMC, PMID, ISBN, and 261,000 of them were to publications with DOI from Crossref. Only 2% of references were made to documents which had DOI and were indexed on Web of Science. Their research is significant because it shows that a large proportion of references are made by valid scientific sources, which can prove the credibility of Wikipedia articles.^[21] Li, Thellwal, and Mohammadi (2021) compared the citations of Scopus-indexed documents to four encyclopedias, including Wikipedia, Baidu Baike, Britannica and Scholarpedia from 2002 to 2020. The first two is crowdsource and the second two are expert-based encyclopedias. The results showed that Wikipedia was by the far the most cited among the four studied encyclopedias. However, the results showed that Wikipedia citations have been declining since 2010, while the Britannica and the Scholarpedia encyclopedia experienced an increase in citations by 2020. They suggest this as evidence for declining of crowdsource encyclopedias’ popularity.^[22]

Studying the validity of drug information on Wikipedia has been the subject of other researches. Clauson et al. (2008) compared Wikipedia and MDR (Medscape Drug Reference) in terms of scope, completeness, and accuracy of drug information. The findings of this study showed that the MDR database is wider and more complete than Wikipedia and has fewer unintentional errors. Therefore, Wikipedia may be a good database for consumers, but it is not credible and should be used a s a complementary source to pharmaceutical information.^[23] Lavsa et al. (2011) evaluated several Wikipedia pharmaceutical article’s accuracies, completeness, references, and also, they analyzed the information categories which are commonly found in drug brochures. The study, which was based on comparisons with brochures and reputable drug databases, found that no articles contained all the information categories, the articles had poor citation and some had no citations, and the information in some aspects of medicine was incorrect. According to the findings, researchers stated that Wikipedia does not provide accurate and complete references on drug information so they recommended resources with more reliability to gain drug information.^[24] Koopen et al. (2015) reviewed references of drug-related articles in the Food and Drug Administration Medwatch at Wikipedia and Lexicomp. The results showed that Wikipedia relies more on review journal (49.2%) and news articles (12%). The average time interval between the issuance of drug warnings on Medwatch and the inclusion of this information in Wikipedia articles was 5.9 days and the access to the main resource in Wikipedia articles was much lower than Lexicamp articles. This study also concludes that caution should be exercised when using Wikipedia for drug information.^[25] Finally, Hunter and Persaud (2018) compared two databases of Wikipedia and Lexicomp in terms of content (dosage and instructions, usage, side effects, or warnings) and the supportive articles about adverse reactions. This study showed that Lexicomp is a more complete online source for providing pharmaceutical information than Wikipedia. Also, compared to Wikipedia, LexiCamp content relies more on judged literature in adverse reactions.^[26]

Literature review shows that Wikipedia’s credibility has been examined from different aspects. Research shows that Wikipedia is not much different from general encyclopedias in terms of information quality and credibility, but in special fields such as pharmaceutical information, it is not reliable. Also, the increase in citations to Wikipedia in reputable academic publications can be considered as a sign of the academic community’s trust in Wikipedia and its implicit credibility. This study intends to examine the credibility of Wikipedia with a new approach. Accordingly, it analyzes and compares the references of articles in Wikipedia and the Stanford Encyclopedia of Philosophy.

This study analyzes articles’ references of Stanford Encyclopedia of Philosophy and Wikipedia using the citation analysis method. Also, the degree of overlap in these references is analyzed based on the theory of bibliographic coupling. The study population is the Stanford Encyclopedia of Philosophy entries and their equivalent in the Wikipedia. Based on a search with the keyword encyclopedia which took place in the Cited Reference Search of Web of Science and analysis of the retrieved results, it was found that the Stanford Encyclopedia of Philosophy was the most cited encyclopedia of that database. Because of that, it was chosen as it was able to be a reliable resource among the academic community. The statistical sample of this research was 5% of the 1685 entries of Stanford Encyclopedia of Philosophy and their equivalent in Wikipedia. The samples selected randomly and systematically. The entry was selected as sample if its equivalent was in the English Wikipedia. When the entry was not found in Wikipedia, an entry before or after it was selected. The statistical sample of this research was 84 entries, the complete list of which is given in Appendix 1. References of these 84 entries were analyzed in the Stanford Encyclopedia of Philosophy and Wikipedia, which was 9,245 sources. A worksheet was created for each entry in Excel software, then in order to compare the references of both encyclopedia, references of each entry in Stanford Encyclopedia of Philosophy were entered in one worksheet column and the references of equivalent entry in Wikipedia were entered in the other column of the same worksheet. Thus, data such as common references of encyclopedias in each entry, unique references of each entry in both encyclopedia, document type and publication year of references, number of citations received by references (if available in Google Scholar), language and accessibility status of references were collected.

The 84 entries selected from the Stanford Encyclopedia of Philosophy and Wikipedia had a total of 9245 references. In 75 entries the number of articles’ references of Stanford Encyclopedia of Philosophy was more than Wikipedia and in only 9 entries, the number of articles’ references of Wikipedia was more Stanford Encyclopedia of Philosophy. According to Table 1, the frequency of Stanford Encyclopedia of Philosophy articles’ references is approximately 3.5 times more than Wikipedia. To compare the average frequency of articles’ references in two encyclopedias in general and due to the non-normality of the data, Mann-Whitney test was conducted. The results of the test showed that the difference between the number of articles’ references of two encyclopedias at the alpha level of 0.05 was significant and as a result, the number of articles’ references of Stanford Encyclopedia of Philosophy were significantly higher than Wikipedia references.

The overlap of articles’ references of Wikipedia and Stanford Encyclopedia of Philosophy as well as the number of unique references in their articles, is given in Table 2. Of 9245 references, 229 references were common between the Stanford Encyclopedia of Philosophy and Wikipedia, which is 2.47% of the total references. accordingly, the number of unique references of the two encyclopedias was high, as 7083 (98.35%) of Stanford Encyclopedia of Philosophy and 1933 (94.61%) of Wikipedia references were unique.

The half-life of references is the length of time that half of the articles’ references occurred during that time. The half-life of references in each encyclopedia is equal to the average half-life of each entry in that encyclopedia. According to the findings of this study, the average half-life of references in the Stanford encyclopedia of philosophy and Wikipedia were 29.96 and 20.08, respectively. In 62 entries, Stanford encyclopedia references half-life was more than Wikipedia, and in 21 entries, Wikipedia references half-life was more than Stanford encyclopedia of philosophy, and just in 1 entry the half-life of references was equal in both encyclopedias. In other words, the references of the Stanford Encyclopedia of Philosophy were spread out over a longer period and had older references. Mann-Whitney test showed that the average half-life of two encyclopedia references was significantly different at the alpha level of 0.05, so the half-life of articles’ references of Stanford encyclopedia of Philosophy was significantly longer than Wikipedia.

document type of articles’ references in Wikipedia and Stanford Encyclopedia of Philosophy is given in Table 3. In both encyclopedias, the main sources are books, magazines, and websites. Compared to Stanford Encyclopedia of Philosophy, websites have a higher percentage in Wikipedia’s articles’ references. Comparing the average percentage of citations to different types of documents in two encyclopedias showed that there is a significant difference between them in term of citation to books, journals, and websites (Table 4). This means that citation to books and journals in Stanford Encyclopedia of Philosophy is higher than in Wikipedia, while the citation to websites in Wikipedia is higher than Stanford Encyclopedia of Philosophy.

Table 5 shows the language distribution of articles’ references in Stanford Encyclopedia of Philosophy and Wikipedia encyclopedias. Most references in the Stanford encyclopedia and Wikipedia are in English, as out of 7,202 references, 5937 references (82.43%) in the Stanford Encyclopedia of Philosophy and out of 2,043 references, 1,625 references in Wikipedia (79.54%) are in English language. The language distribution of other references is almost the same for each encyclopedia, except that in Stanford Encyclopedia of Philosophy, German is the second while in Wikipedia French is the second language. Other languages include a wide range of languages including Chinese, Polish, Portuguese, Italian, Greek, Latin, etc. Comparison of the average percentage of references to different languages showed that there was no significant difference between the two encyclopedias in terms of reference to English, French, German, Japanese, Spanish, and other languages (Table 6).

The accessibility status (open access, non-open access, in-accessible) to articles’ references in Stanford Encyclopedia of Philosophy and Wikipedia is summarized in Table 7. Compared to the Stanford Encyclopedia of Philosophy, a higher percentage of Wikipedia references were open access, but the percentage of references that were not accessible was higher in Wikipedia. Comparing the average percentage of the accessibility status of references in two encyclopedias showed that the aforementioned differences were statistically significant (Table 8). Thus, the percentage of open access and inaccessible resources in Wikipedia was higher than Stanford Encyclopedia of Philosophy, but the percentage of non-open access resources in the Stanford Encyclopedia of Philosophy was higher than Wikipedia.

In this section of the research, the citation rate of articles’ references of Stanford Encyclopedia of Philosophy and Wikipedia was examined. Out of 7202 references of Stanford Encyclopedia of Philosophy, citations of 1486 references were unknown, the rest 5716 references have received 3960086 citations in total. Accordingly, the average citation per reference in Stanford Encyclopedia of Philosophy was 692.8. Out of 2043 references of Wikipedia, the citations of 982 references were unknown, the rest 1061 references have received 853713 citations in total. thus, the average citation per reference in Wikipedia was 804 (Table 9). This result indicates that Wikipedia references received more citations than Stanford Encyclopedia of Philosophy.

Mann-Whitney test was used to compare the citations received by articles’ references of two encyclopedias. The test results showed that the number of citations received by articles’ references of two encyclopedias is significantly different at the alpha level of 0.05, so the citations received by Wikipedia articles’ references were significantly higher than Stanford Encyclopedia of Philosophy.

The results showed that the frequency of references in the Stanford Encyclopedia of Philosophy is higher than Wikipedia, which indicates that the content of the Stanford Encyclopedia of Philosophy is based on a larger volume of sources. Theoretically, references are evidence of information content and the more the evidence of information content, the more it can be relied. In this respect, the Stanford Encyclopedia of Philosophy is superior to Wikipedia. On the other hand, Stanford Encyclopedia of Philosophy references, which are more than Wikipedia, had a longer half-life. This means that references in Stanford Encyclopedia of Philosophy have been spread over a wider period than Wikipedia, and thus had older references. In addition, the overlap of references in two encyclopedias was very low (2.47%). This shows that two encyclopedias had different sources of information, so it is not possible to judge the similarity of their content by relying on the overlap of references.

In both encyclopedias, the main resources of articles were: books, journals, and websites. Singh et al. (2020) research which led to the creation of a comprehensive dataset of Wikipedia references, also showed that Wikipedia references fall into three general categories: books, journals, and Web contents.^[21] According to the results of this research, citations to books and journals in the Stanford Encyclopedia of Philosophy were higher than Wikipedia, and citations to websites in Wikipedia were higher than Stanford Encyclopedia of Philosophy. Based on this, it can be concluded that the Stanford Encyclopedia of Philosophy has paid more attention to traditional academic resources such as books and journals, whereas Wikipedia has paid more attention to Internet resources. Regarding that Wikipedia originated from the internet and web, paying more attention to internet resources can be justified. Most references in Stanford Encyclopedia of Philosophy and Wikipedia were in English, and citations to other language resources in the two encyclopedias were almost similar. According to the Wikipedia homepage, English Wikipedia has the highest number of articles. English is also the pioneer language of science and most of the world’s scientific works are published in this language. The frequency of English-language references in both encyclopedias can be explained from this perspective. German is the second most widely used language in the Stanford Encyclopedia of Philosophy and the number of references to German-language works in this encyclopedia is higher than Wikipedia. Since there are important philosophical works in German, the second place of this language in the references of Stanford Encyclopedia of philosophy indicates the attention of the authors to the resources of the original language. According to the findings, the percentage of open access and inaccessible resources in Wikipedia was higher than Stanford Encyclopedia of Philosophy, and the percentage of non-open access resources in the Stanford Encyclopedia of Philosophy was higher than Wikipedia. This finding suggests that the authors of Wikipedia articles have relied more on unsustainable resources, resources that have not been accessible over time. They also refer more to resources that do not require payment to access. On the other hand, the authors of the Stanford Encyclopedia of Philosophy refer to non-open access resources that are published by commercial publishers and are recognized by the academic community. Finally, a comparison of the citations received by articles’ references of two encyclopedias showed that the citations received by Wikipedia references were significantly higher than Stanford Encyclopedia of Philosophy. Regarding the increasing usage of Wikipedia and the increasing number of web users visits to its articles, users refer to Wikipedia references for more and complete information. In other words, increase in the usage of Wikipedia leads to more citations to its references. The high rate of citations to Wikipedia references may also be justified by the fact that researchers refuse to refer directly to Wikipedia (due to doubts on its validity) and directly refer to its entries. In this regard, Harris (2017) states that Wikipedia should not be used as a primary resource and researchers should find the information, they need by following the resources at the end of each Wikipedia article.^[27]

The Stanford Encyclopedia of Philosophy is a specialized encyclopedia that has received significant citations from reputable scientific publications. The more similar the content of Wikipedia to this encyclopedia, the more we can rely on Wikipedia as a credible resource for finding specialized philosophical information. This study compared and analyzed their references to examine the similarities between the two encyclopedias. According to the findings, it can be concluded that despite the similarities in the referencing pattern of the two encyclopedias, their information content comes from different resources and comparison article references of Wikipedia with Stanford Encyclopedia of Philosophy provide no evidence for Wikipedia’s content similarity and credibility. A promising point for the specialized content of Wikipedia in the field of philosophy is that the references in the field of philosophy have received considerable citations from other academic works, the point that indicates the quality of resources and consequently the quality of the philosophy articles in Wikipedia. It should be noted that this study was conducted on a limited number of entries as well as articles related to a special subject. To have a clear view, it is necessary to conduct research on a larger scale and in other specialized fields. It is also recommended to use text analyzing techniques to check the similarity of Wikipedia content with specialized encyclopedias. The results of these researches can help in deciding to use Wikipedia as an alternative to specialized encyclopedias.