Leibniz-Institute for Psychology Information (ZPID) University of Trier, Trier, Germany
Scientific communications—including criticisms, comments, and replies—are a significant foundation of scientific progress.
To give an overview on the frequency of written professional comments and replies published in the subdisciplines of psychology till 2015.
Scientometric analyses refer to the psychological databases PsycINFO and PSYNDEX.
Firstly, the results show that 2.8% of PsycINFO and 2.2% of PSYNDEX documents refer to scientific discourse. However, time trends were different, which increased (up to 3.6% at the millennium) and then decreased (2.4% in 2013-2015) in PsycINFO, with an up-and-down trajectory in PSYNDEX (decreasing from 3.5% before 1982 to 2.2% in the 1990s, an increase up to 3.1% at the millennium, and a continuous decrease to 0.9% in 2013-2015). Secondly, distinct differences were observed between the subdisciplines of psychology and with reference to both databases: psychological/health personnel issues, psychology & the humanities, clinical psychology, history & systems, and personality psychology received the most comments and replies in PsycINFO, and educational psychology, industrial/organizational psychology, and intelligent systems the least. Most comments and replies related to PSYNDEX were found in sport psychology/leisure, personality psychology, consumer psychology, and experimental psychology, and the least are found in publications on intelligent systems, animal/comparative psychology, history & systems of psychology, and military psychology.
The results are collectively discussed (1) with respect to other indicators of scientific discourse in psychology and other sciences and (2) with respect to the different cultures of scientific discourse between the subdisciplines of psychology in the Anglo-American vs. the German-speaking countries.
Keywords: Scientific discourse, Subdisciplines of psychology, Scientific communication, Psychologists, History of psychology.
open-access license: This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0), a copy of which is available at: (https://creativecommons.org/licenses/by/4.0/legalcode). This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
* Address correspondence to this author at Department of Psychology, ZPID, University of Trier/Germany
University of Trier, ZPID, D-54286 Trier, Germany; Fax: 0049-651-2012961; Tel: 0049-651-38323; E-mail: firstname.lastname@example.org
Scientific communications—including argumentations, criticisms, plaudits, professional comments, and replies— are a significant foundation of scientific progress and inventions. Since the beginning of science in the ancient world, such scientific discourse exists and will continue to exist within and between research teams, in teaching, at scientific meetings and congresses, and in scientific publications. The latter, i.e. the written comments and replies, are somewhat more important, because they potentially reach a larger audience and—even more—they are enduring. However, only little is known about the frequency of professional comments and replies in the field of psychology and its different sub-disciplines, and there appears to be no information available about the possible time trends in the frequency of professional comments and replies.
Very little is known about the frequency of professional criticisms refering to the results of scientometric citation analyses. In a stratified random sample of 90 psychological journal articles published in three English- and three German-language journals in the publication years 1985, 1990, and 1995 (sub-disciplines: developmental, personality, and social psychology), it was shown that the most citations referred to empirical results (30%), theoretical considerations (20%), and methods (9%) in the cited original articles—or they are very general without specific content (25%; e.g., “see, for an overview” or “see, for example “). Only 0.8% of all citations referred to theoretical and/or methodological criticisms in the cited publication [1Krampen G, Becker R, Wahner U, Montada L. On the validity of citation counting in science evaluation: content analyses of references and citations in psychological publications. Scientometrics 2007; 71(2): 191-202. [http://dx.doi.org/10.1007/s11192-007-1659-2] ]. In addition, there were no significant differences in this low occurrence of critical citations between the three psychology sub-disciplines, the two publication languages, and the three publication years included in the analysis. However, in comparison with other scientific disciplines, this proportion is much less. For physics, sociology, and demographics [2Chubin DE, Moitra SD. Content analysis of references. Soc Stud Sci 1975; 5: 423-41. [http://dx.doi.org/10.1177/030631277500500403] -4Peritz BC. A classification of citation roles for the social sciences and related fields. Scientometrics 1983; 5: 303-12. [http://dx.doi.org/10.1007/BF02147226] ], the reported percentages of critical (“negative”) citations were between 5% and 14%, far beyond 1% scarce rate observed in psychology [1Krampen G, Becker R, Wahner U, Montada L. On the validity of citation counting in science evaluation: content analyses of references and citations in psychological publications. Scientometrics 2007; 71(2): 191-202. [http://dx.doi.org/10.1007/s11192-007-1659-2] ]. This is somewhat astonishing because psychology is frequently described as a theoretically and methodologically competitive and heterogeneous (positively formulated: a pluralistic) science. Scheerer [5Scheerer E. Psychology. In: Ritter J, Gründer K, Eds. Historisches Wörterbuch der Philosophie 1989 Basel. Switzerland: Schwabe 1989; Vol. 7: pp. 1599-653.] prototypically states:
“(Psychology) shows the characteristics of the ‘normal state’ of a science described by Th. S. Kuhn, however, psychology does not possess a, for the ‘normal state,’ constitutive consensually accepted paradigm. Rather, psychology includes a rash of competing theory-centered research programs, each appearing with the claim to incorporate psychology in its totality—therefore, the possibility must be assumed that the definition of the scope of psychology is determined by each particular program itself.” Scheerer ([5Scheerer E. Psychology. In: Ritter J, Gründer K, Eds. Historisches Wörterbuch der Philosophie 1989 Basel. Switzerland: Schwabe 1989; Vol. 7: pp. 1599-653.], p. 1648; translation by author). Thus, the situation of having only few critical citations in psychology may actually be an indicator of a very cooperative and very loyal professional scientific communication culture that avoids intra-psychology debates—or, instead, may be an indicator of mutual ignorance of the proponents of different theory-centered research programs.
In short, more empirical results on the frequency of professional comments and replies in psychology would be meaningful to analyze and delineate the scientific discourse practice. We extended this research question (1) to the analyses of possible differences between different psychology sub-disciplines, (2) to time trend analyses up to the publication year 2015, and (3) to comparisons between psychology publications from the Anglo-American versus the German-speaking countries (as an example for Europe). Thus, we referred to written professional comments and replies, which are more public and enduring than verbal ones. English- and German-language psychology publications from the German-speaking countries were used as an example for Europe because they are exhaustively documented in the database PSYNDEX, and the documentation accords with the guidelines used for the dominant Anglo-American database PsycINFO.
MATERIAL AND METHOD
All data were reported in the following form based on the psychological databases PsycINFO® and PSYNDEX®. The American Psychological Association (APA, Washington, DC) produces PsycINFO and features it as an international database of publications in psychology and the behavioral and social sciences, which dates back to 1806. The majority of publications recorded in PsycINFO are English-language publications from Anglo-American countries (>90 percent whereas less than 10% are from the “rest of the world” including 2-3% English- and German-language publications from the German-speaking countries [6Krampen G. Introduction and some ideas as well as visions on an Open Access European Psychology Publication Platform. Psychol Sci Q 2009; 51(Suppl. 1): 3-18., 7Krampen G. Scientometric trend analyses of publications on the history of psychology: Is psychology becoming an unhistorical science? Scientometrics 2016; 106: 1217-38. [http://dx.doi.org/10.1007/s11192-016-1834-4] [PMID: 26924864] ]), and its coverage of psychological publications greatly improved but only after the emergence of digitalization in the late 1970s [7Krampen G. Scientometric trend analyses of publications on the history of psychology: Is psychology becoming an unhistorical science? Scientometrics 2016; 106: 1217-38. [http://dx.doi.org/10.1007/s11192-016-1834-4] [PMID: 26924864] ]. As in late 2016, there were more than 4 million documents recorded in PsycINFO (which can be retrieved, e.g., from http://www.apa.org/pubs/ databases/ psycinfo/index.aspx). The search routine described below resulted in 4,154,511 PsycINFO documentations of psychological publications.
PSYNDEX, developed and hosted by the Leibniz Institute for Psychology Information (ZPID; Trier, Germany), is the complementary, comprehensive database containing German- and English-language publications in psychology and related disciplines from the German-speaking countries (described by the acronym DACHLL: D = Germany, A = Austria, CH = Switzerland, first L = Liechtenstein, second L = Luxembourg; note: for two countries, Switzerland and Luxembourg, German is one of three widely used official languages). Documentation in PSYNDEX began in 1977, for German psychological tests going back to manuals since 1945. In late 2016, more than 320,000 documents were recorded in PSYNDEX which can be retrieved, for example, from www.zpid.de, www.MEDPILOT.de, or www.pubpsych.eu. The search routine outlined below resulted in 323,689 PSYNDEX documentations of psychological publications. Thus, in absolute numbers, PsycINFO contains almost 13 times more documents than its European counterpart, PSYNDEX.
Our searches focused on the search field Document Type (DT) “comment/reply” for all records listed in all specified Classification Codes (CC) in the Thesaurus of Psychological Index Terms ([8Gallagher TL. Thesaurus of psychological index terms. Washington, DC: American Psychological Association 2007. Available from: http://www.apa.org/pubs/databases/psycinfo/index.aspx] retrieval, e.g., from http://www.apa.org/pubs/databases/psycinfo/index.aspx). In addition, the document field “Year of Publication” (YR) and the specific Classification Codes (CC*) for all the sub-disciplines of psychology and other related issues (e.g., for publications on professional psychological & health personnel issues) are used. Time trend analyses focused on publications published before 1980 and to three-year publication intervals from 1981 up to 2015. Identical search routines were conducted in November 2016 for PsycINFO and PSYNDEX.
Because of the very well documented vast increase in the overall absolute number of psychological publications in the 20th century (which can be modeled by way of exponential smoothing [9Krampen G, von Eye A, Schui G. Forecasting trends of development of psychology from a bibliometric perspective. Scientometrics 2011; 87: 687-94. [http://dx.doi.org/10.1007/s11192-011-0357-2] ]), the absolute frequencies found for professional comments and replies must be relativized by the total number of publications documented in the database for each year of publication. This is similar to findings in the other sciences: Behrens and Luksch [10Behrens H, Luksch P. Mathematics 1868-2008: a bibliometric analysis. Scientometrics 2011; 86: 179-94. [http://dx.doi.org/10.1007/s11192-010-0249-x] ], for example, showed a similar increase in literature published in the field of mathematics between 1868 and 2010, which can be modeled by exponential or linear functions. These increased rates reflect the growth of the research communities and resources and have been—in addition—strongly intensified in the last decades by digitalization technologies that enable more efficient submission, communication, and publication systems via the Internet, and shorter production times.
Frequency of professional comments and replies in psychology up to 2015. Table 1 presents the absolute and relative frequencies of professional comments and replies referring to all publications documented in PsycINFO and PSYNDEX. The absolute number of comments and replies in PsycINFO is 16 times greater than that in PSYNDEX. This exceeds the value of 13 times more for PsycINFO for the total number of documents in comparison with PSYNDEX. Thus, the percentage of all comments and replies documented in PsycINFO (2.8%) is higher than in PSYNDEX (2.2%). In general, written scientific discourse practice is more pronounced in psychological publications from the Anglo-American than from German-speaking countries. Furthermore, time trends point to very different developments (see Fig. 1).
Table 1 Absolute (f) and relative frequencies (%)a of professional comments and replies referring to all psychological publications documented in PsycINFOa and PSYNDEX up to 2015b.
Percent (%) of comments and replies referring to all publications documented in PsycINFO and PSYNDEX up to 2015.
Only a small percentage of psychology literature published before 1981 and documented in PsycINFO refer to professional comments and replies (see Table 1). This clearly increased in the early 1980s, followed by a slight drop in the late 1980s, and increased up to the millennium to a maximum of 3.6% (see Fig. 1). After 2000, the percentage decreased by approximately 0.5% and remained relatively stable with professional comments and replies adding up to about 3% of all PsycINFO records in the last 10 years of publications from the Anglo-American countries.
Time trend for the relative frequency of comments and replies in psychology publications from the German-speaking countries, documented in PSYNDEX, shows a completely different picture (see Table 1 and Fig. 1): 3-4% of all psychological publications before 1981 and in the early 1980s refer to professional comments and replies. This can be explained by the high proportion of test criticisms published after World War II, which refers mainly to older tests (published in the 1930s to 1940s, some of them republished up to the 1970s and are still in use in applied psychology until then). In the 1980s—with some precursors in the 1970s—a renewed focus on test development was prominent in the German-speaking countries accompanied by a shift in the direction of internationally used psychometric methodologies. Thus, the maximum rates of professional comments and replies in PSYNDEX before and at the beginning of 1980s reflect the critical discussion and reflection of tests used before, during, and up to two decades after World War II. This was a sign of the modernization of psychology, especially in Germany and Austria, as well as of the efforts to catch up once again to international psychology developments. Concurrently, these professional criticisms were significant precedent of modern test construction in the German-speaking countries.
After the early 1980s, the relative frequency of professional comments and replies in PSYNDEX records dropped to 2%. There was a slight increase around the millennium (maximum: 3%), followed by a very obvious, continuous drop up to 2015 with only 1% scarcity in the publication years 2013-2015 (see Table 1 and Fig. 1).
Frequency of professional comments and replies in the subdisciplines of psychology up to 2015. Table 2 includes the absolute and relative frequencies of professional comments and replies referring to publications on the different subdisciplines of psychology and some other issues—with Classification Codes (CC) in the Thesaurus of Psychological Index Terms [8Gallagher TL. Thesaurus of psychological index terms. Washington, DC: American Psychological Association 2007. Available from: http://www.apa.org/pubs/databases/psycinfo/index.aspx]—documented in PsycINFO and PSYNDEX. Firstly, there were significant differences observed between PsycINFO and PSYNDEX. All 22 different CCs were examined in this study (including their subcategories this number, which increased to 156), and only two had rather similar rates of comments and replies in PsycINFO and PSYNDEX (CC = 27*: communications systems; CC = 40*: engineering & environmental psychology; see Table 2. The differences, significant in some cases, form the majority of the comparisons between critical publications from the Anglo-American and German-speaking countries. For PSYNDEX, the relative frequencies were observed to be only 7 times larger than in PsycINFO. For PsycINFO, the relative frequencies were 13 times larger than in PSYNDEX including most classical subdisciplines of psychology with only a few exceptions, of which the differences for human experimental psychology and educational psychology were the largest.
Table 2 Absolute (f) and relative frequencies (%)a of professional comments and replies referring to all psychological publications documented in PsycINFOa and PSYNDEX up to 2015b.
Thus, the written scientific discourse cultures are very different not only between the subdisciplines, but also in the psychological publications originating from the Anglo-American research community versus the German-speaking countries. Most discursive subdisciplines in the German-speaking countries are (> 3% comments and replies in PSYNDEX; see (Table 2); ordinal order) (1) sport psychology and leisure, (2) personality psychology, (3) consumer psychology, (4) human experimental psychology, (5) psychometrics, statistics, methodology, (6) educational psychology, and (7) social psychology. This is a mixture of large subdisciplines with high publication numbers (e.g., experimental, educational, and social psychology) and rather small subdisciplines with lower publication base rates (e.g., sport and leisure psychology, consumer psychology). Less discursive subdisciplines (< 1% comments and replies in PSYNDEX) include (1) intelligent systems, (2) military psychology, (3) animal experimental and comparative psychology, and (4) history, systems and introductions. Especially the last is striking, because one would expect history and systems (i.e., theories) of psychology to be discursive. However, along with the other three less discursive subdisciplines, these issues have rather low absolute publication numbers in psychology from the German-speaking countries in common.
The rank order for the most discursive subdisciplines in Anglo-American psychology is almost completely different (> 3% comments and replies in PsycINFO; see (Table 2): (1) psychology and health personnel issues, (2) psychology and the humanities, (3) history, systems, introductions, (4) clinical psychology: treatment and prevention, (5) clinical psychology: disorders, (6) personality psychology, and (7) forensic psychology and legal issues. This is a rather plausible rank order, because the scope of these issues or subdisciplines seems to be a priori more discursive. The least discursive subdiscipline (1.02% comments and replies in PsycINFO) is (1) educational psychology, and following in the ordinal order (but with 1.4% - 1.6% a somewhat higher percentage of comments and replies in PsycINFO) are (2) engineering and environmental psychology, (3) intelligent systems, (4) industrial and organizational psychology, and (5) consumer psychology (see (Table 2).
For reasons of parsimony, I chose to forego a visual presentations of time trends depicting the rates of comments and replies in the subdisciplines, and not only because of the large number of necessary figures (22 subdisciplines x 2 databases = 44 figures). The time trends are diverse and could not be clustered to more or less similar curves. The presentation of selected “prototypical” time trends is not meaningful, because there are no such prototypical trends with the exception of the general time trend of the percentage of all comments and replies presented in Figure 1 (see above), to which most of the subdiscipline-specific trends are more or less similar.
DISCUSSION AND CONCLUSION
Integration of the presented results to the research on scientific discourse practice is difficult, because this state of research is sparse. However, first of all, it should be emphasized that, in contrast to other sciences, very low percentage of critical citations in psychological publications (less than 1% [1Krampen G, Becker R, Wahner U, Montada L. On the validity of citation counting in science evaluation: content analyses of references and citations in psychological publications. Scientometrics 2007; 71(2): 191-202. [http://dx.doi.org/10.1007/s11192-007-1659-2] ]) is not confirmed by the percentage of publications classified as written scientific comments and replies, in fact it is almost tripled (PsycINFO: 2.8%) or doubled (PSYNDEX: 2.2%), respectively. Unfortunately, these rates for psychological publications cannot be compared with other sciences up till now, because there are neither data nor databases which allow the kind of scientometric analyses with comparable thesaurus index terms presented here.
Of course, we frequently find critical comments and discursive statements in the sections “Introduction” and “Conclusions and Discussion” of many published articles. However, separately written and published professional comments and replies are much more salient and public, therefore potentially more influential on research development. Search and identification of these publications are easy, because many of them contain terms like “criticism,” “comment,” “reply,” or something similar in their titles. Moreover, professional criticism and comments can be the forerunners of significant changes and innovations in science: for instance, this is the case for test criticisms published after World War II in the German-speaking countries. These types of publications initiated the renewal of test construction and modernization of psychology in Germany and Austria thus allowing them to again achieve the international level developments in psychology. A more recent example may be the presentation of “positive psychology” as an approach for the new millennium [10Behrens H, Luksch P. Mathematics 1868-2008: a bibliometric analysis. Scientometrics 2011; 86: 179-94. [http://dx.doi.org/10.1007/s11192-010-0249-x] -12Seligman ME, Csikszentmihalyi M. Positive psychology: An introduction. Am Psychol 2001; 56: 89-90. [http://dx.doi.org/10.1037/0003-066X.56.1.89] [PMID: 11392865] ]. This has provoked a series of professional criticisms, replies, and comments—not only, but very public in the international psychology community—in the leading journal American Psychologist and lasting up to now [13Heintzelman SJ, King LA. Self-reports of meaning in life matter. Am Psychol 2015; 70(6): 575-6. [http://dx.doi.org/10.1037/a0039483] [PMID: 26348348] -15Seligman ME, Fowler RD. Comprehensive Soldier Fitness and the future of psychology. Am Psychol 2011; 66(1): 82-6. [http://dx.doi.org/10.1037/a0021898] [PMID: 21219053] ].
What may be the reasons for relatively few professional comments and replies in psychology? Of course, publications of primary empirical, theoretical, and methodological research results are more valuable for scientific progress and innovations. Such publications have an especially strong intra-individual and social reinforcement value, because they are crucial for the professional careers of scientists and motivate them to continue moving forward in their work. However, these primary research results should and must be scientifically reflected and discussed within the research community. Yet the publication of such papers may be risky as professional criticisms may have a lower reinforcement value and evaluative status. They include the possibility of negative effects on the professional career, on job appointments, and the danger of making “enemies” and losing friends and support from and within the scientific community.
Of course, data presented cannot clear the question, whether the combination of few critical citations [1Krampen G, Becker R, Wahner U, Montada L. On the validity of citation counting in science evaluation: content analyses of references and citations in psychological publications. Scientometrics 2007; 71(2): 191-202. [http://dx.doi.org/10.1007/s11192-007-1659-2] ] and somewhat more, but nonetheless relatively few professional comments and replies in psychology are the indicators of a very cooperative and very loyal professional scientific communication culture that avoids intra-psychology debates or, instead, are the indicators of mutual ignorance of the proponents of different theory-centered research programs [5Scheerer E. Psychology. In: Ritter J, Gründer K, Eds. Historisches Wörterbuch der Philosophie 1989 Basel. Switzerland: Schwabe 1989; Vol. 7: pp. 1599-653.]. It must be kept in mind that both—critical citations and professional comments/replies—are, in addition to other factors (e.g., discourses within and between research teams, in teaching, at scientific meetings and congresses, etc.), only the two indicators of discourse practice in the field of science. However, they are two indicators with maximum scientific readership and durability.
The scientometric results demonstrating the considerable differences between the percentages of comments and replies in publications (1) from different subdisciplines and some other issues of psychology and (2) from the Anglo-American research community and that of the German-speaking countries point to strongly varying scientific discourse cultures in psychology. The rank orders of rates between PsycINFO and PSYNDEX are almost completely different. The rates of comments and replies are markedly higher in classic subdisciplines like educational, social, personality, and experimental human psychology in publications from the German-speaking countries, whereas the rates are higher in publications from Anglo-American psychology on history and systems of psychology, psychology and the humanities, clinical psychology, forensic psychology and legal issues, and—most pronounced—on psychological and health personnel issues. Low percentages of comments and replies covary sometimes with lower publication numbers in the subdiscipline (e.g., intelligent systems, engineering and environmental psychology, animal and comparative psychology), sometimes not (e.g., educational psychology in PsycINFO). Some percentages may be explained by lasting debates in the subdiscipline (e.g., the fragile status of educational psychology in the German-speaking countries, because of the increasing popularity of applied clinical as well as organizational and business psychology for psychology students’ specializations since the 1980s). Altogether, there is a kind of mutable picture of results, for which each subdiscipline must be analyzed separately for publications from the Anglo-American and German-speaking countries.
For scientometric analyses, it is of some interest that “professional criticism” and “professional criticism reply” disappeared in the 1990s from the Subject Headings (SH; also known as “Descriptors” or “Index Terms”) in the APA Thesaurus of Psychological Index Terms [8Gallagher TL. Thesaurus of psychological index terms. Washington, DC: American Psychological Association 2007. Available from: http://www.apa.org/pubs/databases/psycinfo/index.aspx]. Until then, both were sub-classified under “scientific communication,” but were substituted, somewhat hidden in the nonstandard search strategies, in the documentation and search field Document Type (DT) by the term “comment/reply.” This more neutral term may be a compromise due to a somewhat mainstream-like political correct language usage in psychology, at least in the US, but with international effects because the APA Thesaurus is used internationally and therefore sets international standards in psychology.
CONFLICT OF INTEREST
The author confirms that this article content has no conflict of interest.
I thank Ms. Lisa Trierweiler and Ms. Hiba Qureshi very much for the professional copyediting.
Krampen G, Becker R, Wahner U, Montada L. On the validity of citation counting in science evaluation: content analyses of references and citations in psychological publications. Scientometrics 2007; 71(2): 191-202. [http://dx.doi.org/10.1007/s11192-007-1659-2]