Indicators for the Evaluation of Science, Technology and Innovation Activities: A Systematized Review

The processes of generation, exploitation and dissemination of knowledge are considered essential for the economic growth, development and well-being of nations.^{[1, 2]} In this sense, many countries promote the development of Science, Technology and Innovation (STI) as a strategy for economic and social development.

In order for countries to advance in their growth and have sustainable development, it is essential to assume STI Activities (ASTI) as a critical factor for the structural transformation of markets and ecosystems through the use of knowledge generation, technological development and innovation.^[3]

The development, expansion and consolidation of STI Systems (SSTI) have led to the emergence of new needs that emerge from society and from scientific policies themselves, and that make evaluation a key tool for assigning or distributing information. material or financial resources, the definition of new incentives and the validation of results in certain scientific areas in relation to national needs. For these reasons, the construction of indicators that reflect the convergence of ASTI with social development becomes a particularly important need, especially for developing countries.

STI indicators refer to a series of metrics that are used to measure and evaluate the level of development and progress of a society in terms of scientific research, technology, and innovation. Therefore, there is a close relationship between STI indicators, STI systems and social innovation. On the one hand, STI indicators can be used as an indicator of the level of development of a society in terms of its ability to generate new solutions to social challenges. On the other hand, STI systems can be an important catalyst for social innovation by providing the necessary resources and infrastructure to carry out research, development and application of new solutions.

STI indicators are considered a reflection of a country’s development.^[4] In general, a country with high values in its social and economic indices and indicators also presents high investments in these spheres, adequate capacities and trained human resources, and an industrial sector that takes advantage of said capacities and obtains benefits from the derivation of knowledge in products and services. Therefore, the STI indicators constitute a tool for decision-making, as quantitative representations of the processes and parameters that define the status and dynamics of the SSTI.

In this context, it is necessary for the actors of the SSTI to know the progress in the execution of the ASTI in order to determine what are the capacities that are available and understand the dynamics of research, development and innovation (R+D+i), in order to have elements to plan actions, make decisions and direct processes.

STI measurements are carried out by organizations that have undertaken the task of building and calculating indicators, based on reference manuals^[1,5-8] to monitor ASTI. Among the organizations that lead these evaluation processes are the National Science and Technology Organizations (NSYTO) such as ministries, secretariats, state agencies, administrative departments, entities delegated to keep national statistics, STI observatories, organizations in charge of generating knowledge (universities, research centers) and researchers.^[9] It is important to highlight that, based on the statistics generated by said organizations, governments, institutions and organizations make decisions aimed at directing actions and strategies to strengthen STI processes, as well as the selection of strategic areas and the resource allocation, among others.

Previous research conducted systematic reviews of the scientific literature on specific types of indicators, such as citation and impact indicators^[10] and some approaches to understanding and measuring interdisciplinary scientific research.^[11] Other research analyzes the performance of STI indicators in Indonesia and provides recommendations towards the applied indicators to measure efforts to achieve Indonesia’s 2045 target.^[12] A Bibliometric Analysis about Science, Technology and Innovation in National Innovation System offers a conceptual and theoretical mapping of STI.^[13] Aspects related to STI indicators are also addressed in a study related to evaluation of the Indonesia national strategic policy of science and technology development.^[14]

The scientific literature about STI indicators has focused mostly on bibliometric indicators, with special emphasis on citation impact indicators.^[10] Due to this, systematic reviews of the scientific literature related to these indicators have proliferated. This is evidenced by systematic reviews of the scientific literature on the h-index and related indicators.^[15–20] However, there is a lack of a review that integrates the indicators with the greatest representation in the scientific literature and that enables a comprehensive description and discussion of the diversity of STI indicators used in publications related to ASTI evaluation.

In this order of ideas, this review has a broader scope, incorporating into its analysis the diversity of STI indicators, with greater coverage in terms of type of publications and coverage in various scientific databases. This article presents an in-depth review of the literature in order to serve both researchers who study STI indicators, as well as STI professionals, managers and evaluators who work with these indicators.

Given the importance and implications of the evaluation of the STI and the development of indicators that incorporate other aspects of the ASTI not contemplated in the existing measurement instruments at the international, regional and local level, this research is carried out in response to the following questions: what aspects of STI are measured by the indicators proposed in the scientific literature and what is their level of updating? Consequently, the present investigation is carried out with the objective of identifying, grouping and characterizing the indicators for the evaluation of ASTI, through a systematic review of the scientific literature.

An investigation was carried out with a double qualitative/ quantitative approach. Qualitative research aims at complex and particular situations, making it possible to analyze the problem, its interactions, processes and a deeper understanding of the object of study.^[21] Although the qualitative-interpretative analysis constitutes the essential dimension of the work, a descriptive and inferential statistical treatment is incorporated to complete the knowledge about the object of study.^[22] The information extracted from the qualitative analysis constitutes the core of the work, while the statistical results complement the approach to the analyzed reality.

Regarding the procedures, a systematic review of the scientific literature was carried out, through the use of scientific databases with wide-ranging content. Systematized reviews provide a rigorous and systematic framework, with a methodology that makes it possible to locate existing studies, select and evaluate their contributions, synthesizing data in order to provide conclusions on a specific topic. Being systematic helps to reduce the likelihood of bias and is a way of ensuring that a full body of knowledge is identified on the subject under review.^[23]

For this particular study, the framework to carry out systematized reviews called SysteRe-HSS, an acronym for Systematized Reviews in Human and Social Sciences,^[24] was adopted as a methodology. The methodology is based on the work of^[23] for the set of their contributions and for the idea of systematic approaches, as well as on the terminology of the seminal analysis of^[25] for the expression systematized reviews. It is divided into 4 fundamental phases: I. Search; II. Evaluation; III. Analysis; and IV. Synthesis.

For data recovery, the Main Collection of the Web of Science, the Scopus database and Google Scholar were used as information sources. The selection of these data sources was based on the criteria that they cover a wide range of topics related to ASTI and provide structured information on the scientific literature.

A comprehensive search was carried out in the databases on January 10, 2022. In order to analyze the scientific publications of the last 12 years, the search strategy was limited to the period 2010-2021. As a step prior to the selection of the search terms, several titles and abstracts related to the topic were analyzed, based on an approximate search in several databases. In order to avoid the recovery of irrelevant records, the search was carried out in the title field of the databases, since the search through the title, keywords and abstract fields presented profuse documentary noise. Next, the publications in Spanish and English were selected for analysis and as a result, a total of 228 publications were retrieved.

The keywords used in the search were: indicators, metrics, evaluation, measure, measurement, science, technology, innovation, research and their equivalents in Spanish. Boolean operators were used to correlate the selected terms and the search equation was adapted to the language and characteristics of each database used (Table 1).

The records were exported to the EndNote X9 bibliographic reference manager, where they were subjected to a metadata normalization process. In addition, filtering procedures were adopted that consist of discarding duplicate publications and analyzing the content of the publications, due to the possibility of including works not related to the topic in question in the sample. As a result of this process, 91 duplicate articles were eliminated, leaving 137 for further content analysis, considering the following criteria:

Work directly related to indicators or metrics for the evaluation of the ASTI.
Full text studies, downloaded and incorporated into the database designed for analysis.

Papers that deal with indicators or metrics for the evaluation of activities that are not directly related to STI.
Publications in languages other than Spanish and English.

After filtering, 96 articles met the defined criteria, which is why they made up the final sample of this research. Figure 1 shows the methodological process carried out to obtain the final sample.

A series of analyzes was applied to the set of publications selected as the final sample for this study. The main criteria that were considered were: type of publication, informative summary, type of investigation, methods of obtaining data, main contributions, value, originality and most relevant aspects, comments and evaluative synthesis, keywords and proposed STI indicators.

In order to identify the main terms used in the publications, the map visualization technique based on term co-occurrence networks was used, using the VOSviewer v1.6.17 software (www. vosviewer.com). After the metadata normalization process in the EndNote X9 reference manager, the unified file of the three databases was exported in RIS format, for analysis in VOSviewer. The pre-processing of the corpus (keywords) yielded 486 words, as a result, terms susceptible to normalization were identified and, correspondingly, 411 descriptors were processed with the software.

In addition, the fractional counting method was used to calculate the weight of the terms.^[26] Microsoft Excel 2019 was used for data processing, the generation of frequency distribution lists, tables and figures.

As a complement to the results presented in the research and in accordance with the international principles of Open Science, the list of identified indicators is published and available in open access.^[27]

The analysis by period, considering the total number of documents published (n = 96), shows a limited growth in the number of publications on indicators for the evaluation of ASTI, with the periods of greatest publication being 2016-2018 (n = 41) and 2013-2015 (n = 23). The lowest levels of scientific production correspond to the last period 2019-2021 (n = 20) and the initial period 2010-2012 (n = 12; see Figure 2).

Regarding the type of publications, there was a predominance of original articles. 59.4% of the papers offer theoretical frameworks that serve as a reference to address the subject. 25% use a methodology with a qualitative approach, 28.1% with a quantitative approach and 46.9% with a mixed approach.

Figure 1:
Methodological process of analysis and selection of the sample for the study

Figure 2:
Annual scientific production

Of the sources analyzed, the one with the greatest scientific production is Research Evaluation, with 5 publications (Figure 3). It is an international peer-reviewed interdisciplinary journal published at the University of Oxford. It focuses mainly on the evaluation of activities related to scientific research, technological development and innovation. The articles collected in the sample that were published in the journal address the issue of research evaluation, with the support of the use of indicators, especially in the area of social and human sciences.

The analysis of co-occurrence of terms, after the normalization process and the frequency threshold (n>2) was reduced to 55 terms, which were represented by colors in 9 clusters with a total of 405 relationships and an association strength of 196.50 (Figure 4). The publication topics represented by the two-dimensional map show the appearance weights by binary count by means of the size of each term. The density of the term labels was proportional to the frequency of occurrences and their weight.

The cluster located in the center of the map indicates a high interrelationship of the terms that comprise it, while the clusters located at the edges of the map indicate a lower interrelationship. As a result of the visualization, the thematic clusters that configured the main emerging focuses of research on STI indicators stand out:

Address the evaluation of research, particularly innovation, through patents and inventions and their application in the industrial sector. In addition, it examines the dynamics of innovation evaluation systems and their role in decision making. In these clusters, terms such as “industry”, “patent”, “inventions”, “innovation”, “research evaluation”, “prediction”, “decision-making”, “dynamics”, “innovation system” and “technological innovation” stand out.

Figure 3:
Scientific production by journals

Examines the evaluation of science from metrics based on scientific collaboration and the impact of research. Predominant terms such as “scientific evaluation”, “metrics”, “impact”, “collaboration” and “responsible metrics” are appreciated.

Focus on scientometrics and bibliometrics as tools for the evaluation of STI. Among the main terms positioned in this conglomerate are “evaluation”, “scientometric”, “bibliometric”, “scientific production”, “scientific indicators”, “scientific journal”, “impact factor”, “altmetrics” and “h-index”.

Examines the development of STI indicators, especially indicators for the measurement of social innovation and its importance in the formulation of scientific policies. In this conglomerate, terms such as “indicators”, “innovation”, “measurement”, “social innovation” and “policy” stand out.

For the grouping of the identified STI indicators, the multiple types of R&D&I indicators available and in constant development were reviewed.^{[1,5–8,28–31]} Therefore, the existing STI indicator typologies are taken into consideration and other typologies that represent more current indicators are added. The scientific literature analyzed shows that there is a predominance of indicators related to the evaluation of innovation activities (36.5%), applied to different innovation projects and sectors (Figure 5). A set of Science and Technology indicators (S&T; 26%) related to human resources allocated to STI activity, financial resources and investments in R+D+i are positioned in second place.

The third and fourth most frequent groups of indicators are those related to Bibliometrics and Scientometrics (14.6% and 7.3%, respectively), focused on analysis of the scientific production of researchers, organizations, countries, and regions, as well as the impact of scientific research through publications. These four groups of dominant indicators accumulate 84.4% of the published papers. A group of indicators related to social innovation appear in fifth place (5.2%), which link knowledge of existing innovation measurement approaches with the essential characteristics of social innovation.

The rest of the indicators, although they appear in a much lower percentage, are related to highly topical and significant issues, such as the evaluation of technological production and its impact through patents, the measurement of the impact of practices of social appropriation of science and technology, and next generation metrics, with proposals for responsible metrics and evaluation for Open Science (OS), as well as alternative indicators for evaluating the social impact of research on web 2.0.

Figure 4:
Thematic co-occurrence network.

Based on the grouping of indicators, a frequency count (n >5) was performed to identify those with the highest prevalence in the scientific literature. Table 2 shows that there is a predominance of indicators related to R&D activity that are focused on fundamentally measuring the quantity of new or transformed products or technological processes and services, as well as the investment of financial resources allocated to the activity. Within the set of indicators on Science and Technology, there is a greater representation of indicators focused on measuring the number of people involved in R+D+i and the expenditure made, both by the public sector and by the private sector.

In relation to the bibliometric and scientometric indicators, there is a predominance of indicators related to the measurement of the number of scientific publications and their impact; a similar behavior is reported by the patentometric indicators, with a greater representation of works that measure the number and impact of patents. Another preponderant indicator, although less frequently, is related to the understanding and intervention of the relations between science, technology and society, based on the active participation of the various social groups that generate knowledge and other indicators that measure the economic and S&T social. In contrast, indicators related to social innovation and new generation metrics have a low frequency.

The introduction of metrics in the evaluation of research has made it possible to increase the performance of the evaluated entities, optimizing management and funds, and helping in decision-making. These metrics have been shown to be more accurate, independent, unbiased, and reproducible when used carefully.^[32] They offer quantitative data to successfully express an assessment view as accurately as possible.

Taking into consideration that the evaluation processes of the ASTI are systematic and require new and constant transformations, it is possible to infer that the scientific production in the last 12 years is low. This review on measurement proposals confirms what^[33] asserts, when he mentions that the literature on science and technology indicators is mainly based on lists and chronological accounts of the introduction of such indicators in the reports of the public institutions that apply.

Figure 5:
Papers published by group of indicators

The volume of articles focused on the measurement of innovation is significantly higher than the other indicators identified. At the international level, there are many investigations carried out to measure and evaluate innovation activity in countries. And it is that innovation programs are vital tools for economic growth, knowledge and technology transfer.^[34] For these reasons, this analysis is carried out in various sectors.

Several studies have shown the need for indicators for innovation management, in order to more efficiently control investments in R&D.^[35–38] However, the literature review suggests that the debate on the subject is still insufficient, since innovation activities are very complex and therefore difficult to measure.^[39] In this sense, the Gault Manual^[40] is a must-read to understand innovation indicators and their application in policy formulation. Its exhaustive coverage and discussions make it an important contribution to the literature on this topic.

The need to investigate new measures of innovation is evident, the indicators generated from innovation surveys based on a measurement approach at the subject or company level were the center of attention. These surveys and the resulting indicators provide an important new data resource for understanding innovation performance within and across economies, and for telling a relevant story for policy development, monitoring and evaluation.^[41] However, despite the wide availability of published survey indicators, they lack crucial information content and are subject to several shortcomings that require further understanding and improvement. This is evidenced, in part, by low buy-in by policymakers and broader measurement exercises. This background reveals three key requirements for indicator development. First, they must be provided at the sectoral level, in order to capture the diversity of the industry as the basis of the domains of knowledge, technologies and capacities involved in the production of innovations. Second, they need to reflect firm heterogeneity and differences in firm size demographics, because they shape the challenges and strategies around innovation performance. Third, they must reflect the specificities of the relevant innovation system.

The innovation indicators must represent the economic and conditioning factors of the business, highlighting the market, its growth capacity and entrepreneurial decision-making, emphasizing operational and innovation practices.^[42]

As alternatives to encourage innovation programs in different countries, whether developed or developing,^[34] propose the creation of an innovation data bank for best practices; build a global innovation database with annual innovation reports from different countries to facilitate the exchange of information; promote innovation to strengthen the modern knowledge-based economy and, finally, focus on entrepreneurship as the most important element in generating innovation and economic growth, including new firms, job growth and small businesses.

Other proposals are related to the design of indicators to evaluate eco-innovation projects, with the purpose of measuring the development and introduction of new products, processes and services that reduce the general negative impact on the environment, uniting business and innovation to create sustainable solutions.^{[43, 44]}

In general, recent approaches to assess innovation in countries have adopted composite indicators, where the active participation of companies, universities, and the public sector is essential.^[45]

Social innovation constitutes another necessary concept within innovation and which is complex to measure due to the few studies that have been carried out in this regard. However, there is a growing interest in social innovation, a wide variety of qualitative studies of social innovation and several quantitative innovation indices have been carried out that try to inductively elaborate the theoretical concept and measure the social innovation potential of organizational units.^{[46, 47]}

To the basic dimensions used in the existing innovation metrics: financial resources, knowledge, knowledge protection and patents, collaboration and networks, entrepreneurial activities and culture of innovation, social indicators were incorporated in selected social fields (for example, well-being, sustainability) to consider the social component of social innovation. The sum of the social innovation indicators, therefore, arises from bringing together innovation dimensions used in existing innovation metrics and indicators from non-innovation-based measurement systems to adequately cover the social dimension.

The framework of the social climate, which includes the attitudes of citizens towards novelty, their level of participation in social needs and civic movements, is of enormous importance. The potential for social innovation largely depends on the attitudes of specific social groups and their willingness to contribute to overcoming societal challenges. Without contributions from the social side, social innovations cannot be effective.^[46]

The advancement of new technologies makes possible the appearance of innovative, more dynamic, open and transparent channels of citizen participation. In this order of ideas, a crucial aspect in the evaluation of the ASTI is to respond to the demands of society in recent years, that the Public Administrations work with efficiency and transparency and that they face their actions with a management approach open to evaluation and accountability. Alfaro and Gómez^[48] present an approximation to a system of indicators for measurement, evaluation, innovation and participation oriented to the Public Administration. The system is focused on a collaborative model that encourages citizen participation and promotes social innovation.

Krasnopolskaya and Korneeva^[47] expose a series of elements that make the study of this particular area difficult, which are the lack of statistics for any estimate of the size and scope of the social innovation sector, the precarious reliability of the measures of self-reported innovation and the absence of a quantitative instrument to identify social innovations. In their research, the authors determine certain limitations that hinder the study of social innovation today, clearly revealing the absence of established measures of social innovation by organizations and the inability of quantitative metrics to cover a process of social innovation.

One point to highlight is the set of variables of the indicator of scientific culture and innovation, oriented towards the collection and measurement of data related to the perception of the importance of science and the social appropriation of science and technology.^[49–51] The inclusion of indicators of this nature is very important, since a culture focused on innovation and knowledge will undoubtedly be a facilitator of integrated and shared action issues.

The growth of scientific production, combined with the development of technologies for the dissemination of scientific results, contributed to the increase in the amount of data and information available to society on research results and activities. Other authors^[52] highlight that the organization of data and information on science and technology indicators based on concepts from the information visualization area can help to maximize the understanding of the results of scientific and technological initiatives, as well as guide decisions regarding the development and implementation of more rational and sustainable science and technology policies, providing information on possible differences in scientific research cultures.

As a result, the adoption and use of scientometric assessments as a performance tool in research policy has been increasing and the culture in which these scientometric assessments are useful for research policy has become incredibly widespread.^[32]

In this sense, bibliometric indicators have frequently been used to support the level of scientific development reached by a certain discipline, institution or country, a practice that has led to proposals for scientific and information policies based on the recognition of high levels of productivity, impact, visibility and growth of the scientific literature generated. However, this has meant that those disciplines, sources of information, institutions and countries that are in the so-called main current or main stream (by its name in English) and whose group is made up of it, are better represented by this type of indicators. basically, the countries of the “center” or developed economies.^[53]

The previous behavior has raised controversy about the validity and usefulness of this type of indicators in the evaluation of science, which is why the study, recognition and definition of bibliometric indicators in the specialized literature is extensive. In this regard, new proposals and metrics based on databases with greater scope such as Google Scholar Metrics and Dialnet Metrics have been explored.^{[54, 55]}

In this order of ideas, publications reflect on the center-periphery binomial in science, the division between main and peripheral science, and the use of indicators for the evaluation of peripheral spaces.^[56] In this sense, the limited scope of the metrics of the main science to evaluate the peripheries and the need to adapt the indicators to the fields and contexts in which the phenomena occur, with the recognition of the objectives of the STI systems, are discussed.

The increasing use of bibliometric exercises that have real consequences for the entities subjected to them also increases the importance that such exercises are valid and that the results of bibliometric investigations are not overinterpreted. To favor these aspects, standardized citation indicators are proposed for scientific mapping and research impact assessment.^[57–59]

Other investigations pose as a challenge for bibliometric purposes the evaluation of the academic performance of scientists in social sciences and humanities; since the predominant publication channels in these fields are not well covered by large bibliometric databases such as Web of Science or Scopus.^{[60, 61]} However, the need to find criteria unanimously accepted by the international scientific community has led to the adoption of international bibliographic and bibliometric indexes and bases as the only quality indicators in scientific evaluation.^[62]

In this sense, there are innumerable scientific results that point to the limitations of these indicators to evaluate scientific production and, consequently, the inconvenience of their use.^[63] To resolve this situation, different products have been created to be used in national contexts, since they cover certain aspects not present in the more international indices, such as Diffusion and Editorial Quality of the Spanish Journals of Humanities, Social Sciences and Law-DICE.^[64] Brazilian Journal Evaluation System-QUALIS,^[65] UCRindex of Costa Rica,^[66] Mexican Journal Evaluation Index,^[67] Red Iberoamericana de Innovación y Conocimiento Científico-REDIB^[68] and Dialnet Métricas.^[55] All of them contribute to the visibility of national and regional magazines.

The measurement of webometric indicators is also addressed as one of the most important indicators to reflect the international presence of universities.^[69] Special emphasis is placed on content and technical indicators that influence the promotion of webometric rankings.

Another recurring aspect in the scientific literature is related to patentometry. Through patent analysis, one can understand the technical expertise of each company and investigate the technical development path and business disposition of a specific industry. Studies are based on patent indicators and take into consideration the patent application, the place of publication and the profit. Patent indicators, including the number of patents, patent growth rate, citations per patent, current impact index, technology strength, and technology life cycle, to assess innovation orientation and business competition.^[70]

Other research highlights the need to move towards a next generation metric in the context of Open Science (OS) that promotes the removal of barriers, the development of research infrastructures and enables OS to be inserted into society.^{[71, 72]} The foregoing constitutes a crucial aspect, taking into consideration that as the evaluation of research has become more relevant, the evaluation of social impacts has gained in importance along with that of the qualities of the research and, consequently, a system of OS should make both goals more viable.

With the advent of Web 2.0 technologies, new possibilities have appeared to assess the “impact” of scientific publications, not only periodicals, but also books, reports, data, and other types of non-traditional publications.^[73] From this approach, alternative metrics (Altmetrics) have become a means to measure the broader social impacts of scientific research and can enrich the evaluation of current research by adding new perspectives (visibility, social impact).

In this order of ideas, the metrics of use, capture, mention and social networks measure the number of views, downloads, markings, mentions and interactions with a scientific publication, representing many potential users (students, legislators, interested public) who read publications or use their results.

The main research fronts on indicators for the evaluation of ASTI are related to the evaluation of science, the metrics to measure innovation and its relationship with the industrial sector, the role of indicators in decision-making and development. of public and scientific policies. Other aspects of scientific production are related to scientometric, bibliometric, patentometric, webometric and altmetric indicators, and the role they play in measuring the academic and social impact of scientific research. Other indicators are related to the measurement of social innovation and the development of public policies.

The need for further evaluation of innovation survey indicators is highlighted, on a large scale, since they collect data at the company level and go beyond traditional R+D+i approaches by providing a direct and thematic approach for the measurement. In general, the studies have shown the need for indicators for the management of innovation, in order to more efficiently control investments in R+D+i.

Other publications point out the main challenges of scientific evaluation in peripheral spaces, emphasizing the creation of the most representative data sources of science in these countries and the search for more inclusive indicators, with a plural and contextual approach capable of representing more broadly peripheral science. Proposals for responsible metrics that allow the transition to OS are also presented.

The results presented by this study constitute a contribution for ASTI evaluators, decision makers and researchers in general and can be considered as the basis for broader studies on research in relation to indicators for the evaluation of ASTI. In addition, they can be complemented with other data sources such as reports, manuals, these and other publications.

The times are of changes in science and scientific communication, driven by technological advances, by the emphasis on collaborative processes and by open access to knowledge. Like the ASTI, the STI indicators are in a period of rapid evolution. In the coming years, the efforts of evaluators, researchers, politicians and scientists will have to focus on the new emerging conceptual and methodological difficulties, as well as on improving data collection and standardization systems.

Practices and evaluation systems must value scientific results oriented to local and regional needs, even if they are more difficult to measure. This reinforces the need to propose new indicators and use multiple indicators that reveal in a broader way the value of the science produced in these spaces.

The future of the evaluation of research results must include a more balanced approach that considers both quantitative and qualitative evaluation and emphasizes the quality of the research, not just the quantity.

In addition, it is necessary to propose new indicators for the analysis of peripheral science, which cover aspects discovered by traditional scientific metrics and also by altmetrics, such as the social use of scientific results and the impact of the findings on social and economic development. of the science of the peripheries.

There is a growing need for public or private, governmental or civil society entities involved in the development, financing or management of projects based on social technologies, to monitor and evaluate with a view to verifying compliance with previously defined objectives. Greater efforts are required to develop a methodology that enables the diagnosis, monitoring, and evaluation of social technologies, as well as social intervention projects based on them, considering their complexity, multidimensionality, and interdisciplinarity.

Future research should also aim to develop a more holistic model of country technology level assessment that integrates patent and document indicators through additional comparative analysis of a variety of industries.

It is also considered important to analyze environmental innovation as a whole in future research in order to have a solid method that makes it possible to measure the level of eco-innovation and includes the four dimensions of eco-innovation (product, process, organizational and marketing) and a combination list of indicators in each of these dimensions.

Other indicators must assess emerging technologies, which have a great influence on the economy and society, such as information and communication technologies, nanotechnology and the indicators of the information society.