Nima Orazani's Article “Towards Transparency: Corruption in Psychology, a Critique from Within” – Part One

Towards Transparency: Corruption in Psychology, a Critique from Within

Nima Orazani (Researcher and Lecturer at the Australian National University)

The author's aim in the essay “Towards Transparency: Corruption in Psychology, a Critique from Within” is to briefly demonstrate, through several arguments, why one should initially be skeptical of psychological findings. This critique finds its meaning within the tradition of mainstream psychology, meaning that the author accepts the assumption that psychology is a science in its modern sense (the author does not have much sympathy for such a conception of psychology). From this perspective, psychology as a science is value-free, or at least it is supposed to be. Moreover, the political leanings and personal beliefs of the researcher have no place in it and do not influence its results. And perhaps most importantly, psychology, like the hard sciences, seeks to discover (and not fabricate) causal relationships between phenomena, and by this virtue, it reveals the truth of matters as they are. In this particular case, the “truth of matters” consists of the psychological processes through which we humans (in the most general sense) understand and manage ourselves and others. All of these goals are achieved through an ideal methodology: the experimental method. Since the author intends to critique psychology from within this very tradition (i.e., the positivist tradition), he accepts these presuppositions as required by the discussion.

And now the main question: why should one initially be skeptical of the findings of scientific research conducted in psychology? This claim is supported by the following four reasons.

1. The Replication Crisis (Repligate): Replicability is one of the foundations of science. As its name suggests, replicability or reproducibility is the ability of other researchers to reproduce the results of previously conducted research by resorting to the same methods that produced the initial results. The inability to replicate or reproduce previous research results means that those results are not reliable, and it was merely chance or factors other than those claimed by the researchers that produced the initial results.

In May 2014, the journal Social Psychology, in a special issue whose guest editors were Brian Nosek from the University of Virginia and Daniël Lakens from Eindhoven University of Technology, published the results of an ambitious project aimed at replicating 13 classic, well-known effects published in reputable psychology journals. A little over a year later, in August 2015, the journal Science published the results of a similar but larger project, also led by Brian Nosek. In this project, 100 psychological effects published in 2008 in three of the most prestigious psychology journals – Psychological Science, Journal of Personality and Social Psychology, and Journal of Experimental Psychology: Learning, Memory, and Cognition – were put to the test of replicability. Of the 13 effects selected in the first project, 10 were replicated, and of the 100 effects examined in the second project, only 36 percent of the results were replicated. These results left the community of psychologists in shock and astonishment. The reactions, using a simplified categorization, can be placed into two camps. In the first camp were those psychologists who more or less acknowledged the replication crisis in psychology, and in the second camp were those psychologists who vehemently opposed these results and adopted a vindictive reaction. Examples include Daniel Kahneman, who won the Nobel Prize in Economics for his research in social psychology, and Daniel Gilbert from Harvard University.

After the controversies subsided and arguments were presented by the two aforementioned camps, the majority of psychologists acknowledged the replication crisis. This is why the mere publication of an article in the most prestigious psychology journals (and, of course, other sciences) does not in itself signify the validity of the hypothesis proposed by the authors. Of course, given that the theories of the Austrian-British philosopher of science Karl Popper are the common currency in psychology, one cannot speak of the "validity of a theory" in the strict sense, because theories are always subject to falsification. In any case, the message the replication crisis holds for the audience of psychology is that upon seeing an article in a reputable journal, the article—and, more precisely, its authors—must first answer several questions, and only if those answers are convincing can the published results be accepted, with the limitations that exist regarding generalizability (refer to the third reason). One of these questions is replicability. Have the results of the published article been replicated? As I will explain in other parts of this essay, the problem with psychology as an academic discipline is that the vast majority of articles published in journals are those with novel topics, not articles attempting to replicate the results of other articles. Hence, most published articles have only been replicated to the extent of the few studies that were published, and no further.

2_ The Innovation Mania and Positive Results (Positive results). But if replicability is one of the foundations of science and, consequently, psychology, why has the number of experiments conducted with the aim of replication in psychology been so low that it ultimately confronted psychology with a replication crisis? The answer must be found in a kind of sociology of psychology. For instance, one must examine how psychology, as a discipline, is realized through what procedures and practices. To understand such a matter, the following questions must be answered: What criteria do psychology journals have for publishing scientific articles? What are the evaluation criteria for professors working in psychology departments, and by what criteria do these professors receive promotions? What value does the community of psychologists place on new ideas and theories compared to articles that deal with replicability? Which activities are encouraged and which are dismissed within the community of psychologists?

The psychology community obsessively emphasizes novelty. For this reason, articles that design and empirically test a new hypothesis or theory are published far more easily than articles that attempt to replicate the results of previous research. Articles that replicate prior findings are judged as uncreative. The prevailing assumption is that these articles add nothing to science, and that the psychologists who conduct such experiments are not intelligent enough to generate new theories and hypotheses. Consequently, the attitude of the psychology community and journal reviewers toward the authors of such articles is, if not negative, by no means positive. This is how psychology journals prioritize publishing articles that demonstrate novelty in their work. Although the principle of novelty as an idea is defensible in general, in this case it results in no one being eager to replicate the results of completed studies. This is why, as Chris Ferguson and Moritz Heene put it, psychology has become “a graveyard full of theories that are like the walking dead.” Some of these journals even explicitly cite novelty as their publication policy. “The journal Nature states that for an article to be considered for review, its results must be ‘novel’ and ‘interesting.’ The journal Cortex notes that empirical research reports must ‘report important and novel material.’ The journal Brain warns authors that ‘some [articles] are rejected without being reviewed because they lack the necessary novelty,’ and Cerebral Cortex goes a step further, noting that even after review, ‘final acceptance depends not only on technical merit but also on the reviewer’s subjective ranking of the article’s novelty’” (Chambers, 2017). We thus witness pressure from psychology journals on authors to publish novel articles and, consequently, the rejection of articles that attempt to replicate the results of previous research.

On the other hand, professors working in psychology departments need to publish as many articles as possible, especially in reputable scientific journals, for career advancement. There is a well-known saying among professors and doctoral students in psychology: “publish or perish.” In fact, a psychologist’s academic life depends on publishing articles. Income, academic and social standing, career advancement, the likelihood of securing research grants, being invited to speak at scientific conferences, and receiving offers to conduct research from private and government organizations—which are also lucrative—all depend on the number of articles they publish. It is these institutional norms that pressure researchers to avoid replicating previous findings and instead constantly engage in producing and publishing new hypotheses and theories that have not passed the test of replicability.

Another problem is that even these novel hypotheses and theories are only published if they are statistically significant. Simply put, if I conduct a study on the difference between two teaching methods in, say, Persian literature, and conclude that there is no difference between the two methods (negative results), or in technical terms, if I fail to reject my null hypothesis (H0), even if my hypothesis is novel, reputable scientific journals show no interest in publishing the results of such a study. This norm is so pervasive that psychologists have internalized it in advance and no longer bother submitting negative results to journals. This process is called “publication bias” or the “file-drawer effect,” meaning the highly disproportionate publication of studies that have produced positive results (i.e., rejection of the null hypothesis) and the disregard for publishing studies that have yielded no results. This is how even reputable psychology journals are filled with positive (and not negative) results. The results in these journals can be summarized at an abstract level as follows: as the researchers predicted and hypothesized, a statistically significant difference was observed between the experimental group and the control group (rejection of the null hypothesis). The consequence of this bias is that, for example, as a researcher, I test various hypotheses until I finally obtain the desired result (the ability to reject one of the hypotheses), and only under these conditions do I publish my results. This is how readers of scientific journals have no access to the conditions that lead to negative results and only deal with positive findings.

Daniele Fanelli from the University of Edinburgh collected a random sample of over 2,000 published articles from a wide range of sciences, from space sciences to physics and chemistry, biology, psychology, and psychiatry. The results were striking. Across all sciences, the publication of positive results was more prevalent than negative results. Even in the case of space sciences, where the highest percentage of negative results was published, 70 percent of the sample articles supported the proposed hypothesis. A very important point is that this bias was greater in psychology than in all other sciences, at more than 91 percent.

3_ Participants in Research. Let us assume that the problems I have raised so far are all incorrect. The problem of sampling and generalization, however, remains. Henrich, Heine, and Norenzayan (2010), in an influential article, addressed the problem of the WEIRD sample. According to the argument of these authors, which was also accompanied by empirical data, because most of the samples used in psychological research come from countries that are Western/White, Educated in the educational systems of Western countries, Industrialized, Rich, and have more or less Democratic political systems, their findings cannot be generalized to countries that do not possess such characteristics. It is clear that the results of a study can only be generalized to the population from which the sample was drawn. This is while it is usually assumed that if a research result is obtained using American samples, that effect also applies to Iranian samples, which is, of course, an unscientific conclusion. In this regard, Nick Brown from the University of Groningen provides a statistic that is very thought-provoking. 96 percent of participants in psychological research are from Western countries, while Western countries constitute only 12 percent of the world's population. 68 percent of all participants in psychological research are from the United States, which constitutes only 5 percent of the world's population. Of this 68 percent, 77 percent are white, and 67 percent are studying psychology. Furthermore, undergraduate psychology students in the United States are 4,000 times more likely to participate in a psychological study compared to the rest of the population. Do you think studies conducted with such samples can be generalized to a population other than the one from which these samples come? I leave the answer to you.

اما حدس بزنید برای اینکه بتوانیم نتایج داده‌ها را به جوامع دیگر تعمیم بدهیم چه باید بکنیم. درست است: توسل به تکرارپذیری. این در حالی است که در قسمت قبل توضیح دادیم که چرا انجام پژوهش‌های تکرارپذیری کار ساده‌ای نیست.

۴_ عدم شفافیت. این مشکل در ارتباط با مشکل دومی است که در بالا توضیح دادم یعنی میل وصف‌ناپذیر مجلات و به تبع آن پژوهشگران به انتشار نتایج ایجابی. ازآنجا که انتشار پی در پی مقاله در مجلات معتبر مهم است و سبب شهرت و ارتقاء شغلی روانشناسان می‌شود بنابراین پژوهشگران خودآگاه و ناخودآگاه دست‌اندرکار دستکاری نتایج پژوهش هستند تا به معناداری آماری برسند. از جمله فرایندهایی که احتمال چنین کاری را افزایش می‌دهد عدم شفافیت در فرایند پژوهش است. ما روانشناسان پیش از آنکه آغاز به کار پژوهشی بکنیم در هیچ جایی فرضیه‌های اصلی و فرعی، متغیرهای کنترل (covariates)، متغیرهای تعدیل‌کننده (moderators)، متغیرهای واسط (mediators)، روش‌های تحلیل آماری، معیارهای حذف داده‌ها، تعداد نمونه، شرایطی که در صورت بروزْ تعداد نمونه را افزایش می‌دهیم و ... را با شفافیت ثبت نمی‌کنیم. به همین دلیل از آنچه «درجه آزادی پژوهشگر» (researcher degree of freedom) نام دارد (سوء)استفاده می‌کنیم تا نتایج آماری معنادار به دست آوریم. وقتی پژوهشگر از پیش فرایندهای پژوهشی را ثبت نکرده باشد برای به دست آوردن نتیجه آماری معنادار جای مانور بسیاری دارد. با امتحان کردن انواع و اقسام تکنیک‌های آماری، حذف بخشی از داده‌ها و یا اضافه کردن تعداد آزمودنی در نهایت به نتیجه دلخواه می‌رسد و مقاله‌ای برای انتشار تولید می‌کند. به این کار به اصطلاح p-hacking می‌گویند. به بیان جورج ایستربروک «اعداد را شکنجه کن، به هرآنچه بخواهی اعتراف خواهند کرد.» به دلیل همین درجه آزادی پژوهشگر و البته بحران تکرارپذیری و p-hacking بود که برایان نوزک به همراه همکارانش جنبش علم شفاف (Open Science Movement) را آغاز کردند. در این جنبش که اکنون تقریبا مراحل ابتدایی خود را طی می‌کند روانشناسان تشویق می‌شوند تا از الف یا یای پروژه‌های پژوهشی خود را پیش از آغاز یا دست‌کم پیش از تحلیل داده‌ها به صورت علنی در اختیار عموم قرار دهند تا امکان p-hacking وجود نداشته باشد. مجلات روانشناسی نیز امروزه از پژوهشگر می‌خواهند تا اگر ملاحظات اخلاقی مانع نمی‌شوند داده‌ها را به صورت علنی منتشر کند تا هر که اراده کرد بتواند داده‌ها را تحلیل کند و به نتایج مشابه برسد (شاید هم نرسد!). بعضی از مجلات نیز فرایند داوری را در همین مرحله انجام می‌دهند و نه پس از انجام پژوهش، به این معنا که پژوهشگر پس از ثبت آنلاین فرایند کامل پژوهش آن را به مجله ارسال می‌کند و مجله آن را داوری می‌کند. پس از پذیرفته شدن پژوهش، پژوهشگر آغاز به کار می‌کند و با توجه به اینکه مقاله‌اش از پیش داوری شده و برای چاپ پذیرفته شده است صرف‌نظر از اینکه نتایجش به لحاظ آماری معناداری هستند یا نه پژوهش‌اش را در انتهای کار در مجله مورد نظر منتشر می‌کند. اینگونه دیگر انگیزه‌ای برای توسل به درجه آزدی پژوهشگر برای دستکاری داده‌ها به منظور گرفتن نتایج دلخواه وجود نخواهد داشت.

مشکل دیگری که به دلیل عدم شفافیت بروز می‌کند HARKing (فرضیه‌سازی پس از دیدن نتایج) نام دارد. این اصطلاح را روانشناس اجتماعی نوربرت کر (Norbert Kerr) ابداع کرد. از آنجا که داوران مجلات از پیش نسبت به فرضیه پژوهشگران آگاه نیستند بنابراین پژوهشگران این امکان را دارند تا پس از تحلیل‌های آماری فرضیه خود را عوض کنند (در صورتی که فرضیه اولیه‌شان تایید نشده باشد اما به نتیجه ایجابی دیگری رسیده باشند) و آنگاه مقدمه مقاله را با توجه به آن فرضیه که فرضیه‌ای پساتجربی (Ad hoc hypothesis) است بنویسند. بدین منظور پس از اینکه فرضیه را در هماهنگی با نتایج ایجابی عوض کردند به سراغ پیشینه پژوهشی می‌روند تا مقالاتی هم در تایید آن نتیجه که از ابتدا انتظارش را نداشتند پیدا کنند. طبیعی است که فرضیه‌ پساتجربی را نمی‌توان ابطال کرد.

There are other critiques of psychology and other scientific findings based on empirical experiments, but to avoid prolixity I will defer them to another article. Those who believe in the scientific nature of psychology must take such critiques seriously, especially in our country, Iran, where ideological processes also influence the manipulation of findings. For example, I heard from a friend that if research results contradict or conflict with accepted religious beliefs, some professors encourage students to manipulate data so that the results of empirical research align with religious beliefs. Iran's scientific community, especially the psychology community, must join the transparency movement in psychology. But the reader may ask why we should be so concerned about findings published in journals that most people, except psychologists themselves, do not read. The answer is that we psychologists, based on the findings published in these very journals, advise our clients, counsel them, and policymakers, based on these same findings, spend public funds and design policies that affect the health of people's minds and psyches. The community of psychologists must, more than ever, think about a radical critique of their discipline instead of promoting it. This article is an invitation to the psychology community, including the Psychology and Counseling Organization, the Iranian Association of Social Psychology, the Iranian Psychological Association, and other institutions responsible for this discipline, to join the transparency movement and engage in the process of critiquing psychology as an institution.

It should be noted that this critique finds meaning within the positivist interpretation of science. Critical approaches to psychology critique, in a far more fundamental way, the scientific nature, neutrality, truth-discovering capacity, and generally the positivist foundations of science upon which psychology is built, and offer an entirely different narrative of psychology. In another article, I will address the critique of psychology from this perspective.