Diederik Stapel and the frequency of scientific shenanigans

On August 27, two junior researchers working with the Dutch social psychologist Diederik Stapel at Tilburg University contacted a university administrator with suspicions that their senior colleague was using faked data.  As one of the worst forms of academic shenanigans that fall under the broad umbrella of "academic misconduct", an allegation of data fabrication was quite serious.  This is especially true because Diederik Stapel was in the early stages of a prolific scientific career; he served on the editorial board of six different academic journals and had received the 2007 "Early Career Award" from the International Society for Self and Identity (ISSI).  He had also published many articles that received generous press attention, including one in Science that claimed that messy environments promote discrimination.

Nonetheless, a little over a week and one university investigation later, Stapel admitted to making up data and was sacked from Tilburg University.

The revelation that Stapel committed data fabrication has sent shockwaves through academic psychology.  Beyond tarnishing Stapel's own work, the news also threatens to tarnish both the work of his colleagues and the journals with which Stapel was affiliated.  This has led to some vociferous distancing from Stapel's colleagues and the disappearance of all mentions of Stapel (outside the news of his sacking) from both the Tilburg University website and the ISSI website, as shown below:

Selected Tilburg University professor editorial activities,
before and just after Stapel admitted to data fabrication.
"Before" image taken from Google cache on 9/19/2011

ISSI Early Career Award recipients before and just after Stapel admitted to data fabrication.
"Before" image taken from Google cache on 9/19/2011

Stapel's swift, precipitous diminution in internet presence illustrates the "contagion principle" of academic misconduct: when it is uncovered, it threatens not only the credibility of the scientist who committed it, but also the credibility of all those associated with it.  Of course, the contagion principle is actually quite well-founded -- fundamentally, science rests on the trust of the public and the greater scientific community.

If Stapel were the only "bad apple" in a basket of otherwise delicious scientific fruit, the contagion principle might only apply in a limited way, to Stapel's immediate colleagues.  Unfortunately, Stapel is not alone in his shenanigans.  Just a year ago, the prominent psychology researcher Marc Hauser also admitted to misconduct, and was eventually forced to resign his post from Harvard.  By themselves, even two cases might not spoil psychology's credibility, but because psychology is typically not taken as seriously as other sciences (perhaps with good reason), it simply does not enjoy a large margin for error.

This leads to an obvious (but no less important) question:  how frequent is data fabrication (and other forms of misconduct) in psychology?

This is a difficult question to answer.  The most cost-effective way to obtain an answer is through a random-sample survey, but surveys have the fundamental problem that research scientists are probably pretty damn unwilling to admit to that they have made up data, even under conditions of anonymity.  Researchers can partially address this problem by comparing admission rates of personal misconduct to admission rates of witnessing the misconduct of others (though reports of witnessing misconduct might themselves be biased by factors such as professional jealousy).  Unfortunately, no such survey of research psychologists has been conducted, so the best we can do for now is draw inferences from surveys conducted in other sciences.

Enter a paper in PLoS ONE by Daniele Fanelli.  This paper aggregates the results from 18 anonymous academic misconduct surveys of randomly-sampled research scientists.  It also compares the results of surveys that asked for admission of personal academic misconduct to the results of surveys that asked admission of witnessing the misconduct of others, allowing one to indirectly assess the extent to which surveys that ask about personal misconduct are tainted by reporting bias.

The Fanelli paper is full of interesting insights.  The first that I'll highlight is from Figure 2, which shows the rate of personal admission of making up data (the worst form of misconduct) from the various surveys analyzed by the paper, as well as the overall mean estimate across the surveys.  The overall estimated rate of admission might seem low (1.97%), but remember that this is the rate of admission for the most severe form of misconduct. The estimated rate of admission when other, less severe forms of misconduct were included was much higher (9.54%).

Figure 2 from Fanelli (2009). Admission rates for personally
committing data fabrication.  Lines are 95% confidence intervals.

Compare the results regarding personal admission of data fabrication to the results regarding witnessing others commit data fabrication (below).   Here the estimated rate is 14.12%; this jumped to 28.53% when other, less severe forms of misconduct were included in the estimate.  Scary.

Figure 4 from Fanelli (2009).  Admission rates for witnessing others
commit data fabrication.  Lines are 95% confidence intervals.

What are the implications of these data for scientific psychology?  First, a large psychological society needs to step up by conducting its own randomized survey of research psychologists.  As I mentioned above, an academic misconduct survey of psychologists has yet to be conducted, which leaves both research psychologists and the public in the dark about the extent of the misconduct problem.  Second, if the problem is widespread, research psychologists need to do a better job of policing themselves.  If they don't, they will either be policed from outside -- or, more likely, they'll simply be unfunded.


Reference:

Fanelli, D. (2009). How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data. PLoS ONE 4(5): e5738. doi:10.1371/journal.pone.0005738

The quest for (social) scientific truth

Every now and then when I tell a stranger that I study scientific psychology, I get a reaction that is perhaps best summarized by this comic from xkcd:

Click for a larger image

Essentially, the reaction is that, as a social science, psychology isn't a "real" science like biology, chemistry, or physics.  I encounter this reaction often enough, even among otherwise scientifically savvy people, that I am often tempted to do silly things like shout, punch things, or at the very least launch into an ill-advised rant.

Fortunately, I do none of these things.  As I have discussed previously, the reaction that social sciences are pseudoscientific is rooted partly in the knee-jerk intuition that understanding people is easier than understanding "hard science" topics like physics and chemistry, and therefore that social scientific truths are less valuable than truths discovered in other sciences. Therefore, I consider it part of my duty as a social scientist to be a force of enlightenment, and consequently grit my teeth and explain as gracefully as possible that the defining features of science are rooted in method rather than content.  Science is defined by an interplay between evidence and theory, not the particulars of the content studied.

Deep in my heart, though, I know that these reactions are rooted not only in human biases, but also in the very slow, non-cumulative character of the advance of the social sciences.  The "harder" sciences are responsible for large numbers of inventions and increases in human comfort.  The social sciences do not have as impressive a track record.  For another source of evidence, consider the following figure, taken from a paper in PLoS One by Daniele Fanelli:

Figure 1 from Fanelli (2010)

The above figure shows, across a variety of disciplines, the proportion of papers claiming to have found support for the authors' predictions.  Science being what it is, we should expect that regardless of the discipline, sometimes scientists get it wrong -- the results of their experiments do not support their original predictions.  Disciplines that have a low proportion of papers reporting these sorts of experiments either have theories so powerful as to make experimental predictions a trivial exercise, or, perhaps more likely, have theories so flexible and imprecise that they are nigh infalsifiable.  And, of course, the discipline with the highest proportion of papers claiming to support the original experimental predictions is psychology.

Consider a second figure, this one taken from John Ioannidis' provocatively named paper, "Why most published research findings are false":

Table 4 from Ioannidis (2005)

Ioannidis constructed a variety of models designed to show the probability that a research finding is true (what he calls the "positive predictive value", or PPV) under various research conditions.  Through his model, Ioannidis identified three factors that decrease the probability that a research finding is true:

1.  An inadequate number of observations (or, for the statistics nerds among my readers, inadequate statistical power, which Ioannidis labels 1 - β).  Fewer observations decrease the chances of detecting true relationships that actually exist in a given study, and thus decrease your confidence that the reported relationships in a study are actually true.

2.  A large number of tested relationships in a given study.  As the number of tested relationships goes up, the ratio of true to false relationships in the study (which Ioannidis labels R) typically goes down.  Thus, your confidence in a study's reported relationships should also go down.

3.  A large number of factors, such as financial incentives, researcher ideology, flexible research designs, that increase researcher bias (which Ioannidis labels u).  The presence of these factors straightforwardly decrease your confidence in a study's reported relationships.

Unfortunately, low power, a large number of tested relationships, and researcher bias are all characteristics that typify social science research.  By my own (admittedly speculative) estimation, row 6 in Ioannidis' Table 4 best represents the typical psychology experiment: low power, a large number of tested relationships, and a moderate amount of researcher bias due to either a flexible research design and / or a researcher attempting to support his or her "pet theory".  Ioannidis gives the results of that study a .12 probability of being true.

Given the above evidence, is the quest for social scientific truth entirely quixotic?  Not necessarily.  Social science (psychology among them) is a difficult enterprise.  Difficulty, though, is not the same as impossibility.  In the face of difficulty, it is absolutely incumbent on social scientists to conduct their research in ways that increase confidence in their findings.  Ioannidis pointed out some of these factors -- large numbers of observations, a focus on a few variables, and strong research designs that are independent of researcher ideology and financial interests.  As more social scientists design their studies in these ways, perhaps we can hold out hope that the social sciences will develop more of the cumulative character of the harder sciences.


References:

Fanelli, D.  (2010).  “Positive” results increase down the hierarchy of the sciences. PLoS ONE 5(4): e10068. doi:10.1371/journal.pone.0010068.

Ioannidis, J.  (2005).  Why most published research findings are false. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124.

A glimpse into the abyss of psychology prelims

Over the past month, I endured the crucible of the so-called "preliminary exams", or as they are more affectionately called, "prelims".  These exams go by different names in different areas ("qualifying exams" or "quals", "comprehensive exams" or "comps"), but across institutions, the intent is the same: complete an exam (or more rarely, write a paper) to prove your mastery of a body of knowledge.  Following prelims, graduate students are allowed to begin their dissertation research and, eventually, their PhD.

Needless to say, taking prelims is an intense and exhausting process.  While the specifics vary from place to place, it usually involves studying for months, followed by a multi-day exam with a strict deadline.  My own prelims consisted of a five-hour in-class test, followed by a six-day period in which I wrote four six-page essays.  I studied for my own exams for around five months and, according to my prelims Google Notebook, I read some 75 papers and book chapters.  By the end of prelims I felt like I was leaking social psychology out the ears.

For me, one of the frustrating aspects of prelims is that despite the long hours of study and the intense testing process, the papers you write are largely useless.  They can't be published (though sometimes the ideas can make their way into other papers) and they don't give you practice with the practical aspects of academic life, such as obtaining grant money, submitting papers to journals, or navigating nasty departmental politics.  So, it is with the vain hope that my experiences will be useful to someone that I am publishing what I think is my best prelims essay on this blog.  It will give you, my readers, a taste of what a prelims essay is like, and perhaps it will even interest the more masochistic and nerdy among you.  I can always hope.

Here is the prompt:

For almost three decades, social psychologists have argued that humans have surprisingly little insight into the underlying causes of their behavior. More recent research has gone so far as to argue that human will or volition is an “illusion”.  Please provide an overview of the bases for these arguments and critically examine the empirical evidence used to support them. Link these basic assumptions of control versus automaticity to other phenomena in social psychology (e.g., stereotyping and prejudice, persuasion, etc.).  Finally, give us your opinion on the notion of control and automaticity.

And here is my response:

The long arm of control:

Volition and the long-term regulation of behavior

            One of the longest-standing debates in Western philosophy is that of the nature of free will.  Although early psychologists limited themselves to speculation about it (James, 1884), advances in social-cognitive and neuroscientific methods have enabled psychologists to study the nature of free will more directly.  These advances in method have generated a flurry of activity, and prominent researchers are divided over whether people possess volition (e.g., Ryan & Deci, 2006; Baumeister, Bratslavky, Muraven, & Tice, 1998) or whether volition is illusory (Wegner, 2002; Bargh & Chartrand, 1999).[1]  I will argue that the division about the existence of volition stems in part from the failure to distinguish between the regulation of behavior in the moment from the regulation of one’s long-term behavior.  I will then discuss the implications of this insight for dual-process theories in social cognition.

Control, automaticity, and the components of volitional behavior

            Because of the difficulties in measuring volition, researchers interested in volition have inferred its properties by observing broad differences in large classes of behavior.  For example, people’s behavior differs in the extent to which environmental cues inevitably give rise to a particular behavioral response.  Sometimes, people mindlessly respond to environmental cues (e.g., Langer, Blank, & Chanowitz, 1978), while at other times, people are relatively flexible in their responses (Wheeler & Fiske, 2005; Fleming, Darley, Hilton, & Kojetin, 1990).  Another broad difference in classes of behavior is the extent to which behavior is intended.  Although all behavior may ultimately serve some motive (Kenrick, Griskevicius, Neuberg, & Schaller, 2010; Maslow, 1943), some behavior serves an explicitly formulated prior goal (Ajzen, 1991), whereas other behavior is more reactive, a response to rapidly changing situational circumstances (Bargh, 1994).  Behavior also differs in how effortful it is (Baumeister et al., 1998); in fact, researchers have recently found that some behavior rapidly uses energy in the form of blood glucose, whereas other behavior does not (Gailliot et al., 2007).  Finally, behavior differs in whether it is accompanied by the perception of control, a perception that gives rise to distinct feelings of volition (Deci & Ryan, 2000; Wegner, 2002).

            The above distinctions in behavior (inevitability versus flexibility in responses to environmental cues, intentional versus unintentional behavior, effortful versus effortless responding, and perceptions of control) often co-occur into two distinct clusters, leading theorists to dub flexible, intentional, effortful behavior that is experienced as volitional controlled and inevitable, unintentional, effortless behavior that is not experienced as volitional automatic.  Moreover, many theorists have reasoned that categorically different processes (controlled processes versus automatic processes) must be responsible for the two types of behavior (Shiffrin & Schneider, 1977; Schneider & Shiffrin, 1977).  These two processes are assumed to be mutually exclusive, in that behavior is either dominated by one process or the other.  The distinction between automatic and controlled processes has proven popular and has given rise to a large number of so-called dual-process theories, which specify, within a certain domain, the conditions that give rise to controlled versus automatic behavior (see Chaiken & Trope, 1999).  However, because dual-process theories have been formulated based on broad, co-occurring differences between classes of behavior, the specific roles and meanings of automatic and controlled processes have become unclear as researchers have provided evidence that effort and the perception of control do not always co-occur with flexible, intentional behavior.

The dissociation between effort and flexible, intentional behavior

            The primary evidence that flexible, intentional behavior can be produced in the absence of effort stems from work on implementation intentions, which are simple, consciously-formed plans that specify a triggering situation and a response (i.e., if x, then y; Gollwitzer, 1999).  Importantly, the effort in forming implementation intentions occurs before their execution; once formed, implementation intentions produce relatively effortless behavior once a person is placed in a relevant situational context.  Implementation intentions can support behavioral responses despite conflicting prepotent responses, such as are present in the Stroop task (Gollwitzer & Schaal, 1998), and can successfully improve performance on tasks that supposedly preclude strategic responding, such as shooter tasks with response deadlines of 630 ms (Mendoza, Gollwitzer, & Amodio, 2010).  However, the behavior produced by implementation intentions supports a specific goal (Gollwitzer, 1993), and remains flexible (Gollwitzer, Parks-Stamm, Jaudas, & Sheeran, 2007); implementation intentions that do not support a goal for which they are formulated do not inhibit performance (Gollwitzer et al., 2007).  Thus, research on implementation intentions provides evidence of behavior that is flexible and intentional, but not effortful.

The dissociation between the experience of control and flexible, intentional behavior

            The experience of control is associated with attention to one’s responses and the attribution that one is the cause of a certain outcome.  Two lines of work suggest that these experiences are dissociated from flexible behavior in the service of an intention.  First, work on “auto-motives” suggests that flexible, intentional behavior can occur even when people believe that they are not acting towards a particular goal (Bargh, 1990).  In this work, goal-relevant knowledge is made accessible, which, as long as the goal has been made chronically accessible in the past, triggers goal pursuit in the absence of a conscious intention to pursue the goal (Bargh, Raymond, Pryor, & Strack, 1995).  The effects of these goals on behavior are similar to the effects of goals that are consciously pursued; for example, participants who are primed with achievement-relevant information prior to performing a word search task find more words than participants who are given neutral primes, and find as many words as when the achievement goal is consciously pursued.  Additionally, participants in goal-priming conditions tend to show other goal-directed effects – they persist in behavior directed towards the goal when they are interrupted and inhibit behavioral alternatives that compete for attention (Bargh et al., 2001).

            A separate line of work suggests that people can experience the feeling of volition, even if the action causally attributed to the self was objectively caused by another person (Wegner, 2002; Wegner, 2003).  In order for a person to attribute their actions to their own thoughts and therefore experience volition, a thought must occur just prior to the action, must be consistent with the action, and must be produced in the absence of other plausible causes (Wegner, 2002).  Using these principles, Wegner and Wheatley (1999) have experimentally produced the illusion of volition for actions, such as movements of a computer mouse, that were objectively caused by a confederate.  Together, the work of Wegner and Bargh suggests that the experience of volition, and therefore the perception that one’s actions are produced by oneself, can be dissociated from volitional behavior (for similar arguments, see Nisbett & Wilson, 1977).

Long- and short-term behavioral regulation

            The above analysis suggests that, at least in the moment that a behavior occurs, neither the exertion of effort nor the experience of control are necessary to produce behavior that is flexible and intentional.  Some theorists have interpreted this evidence to mean that volition does not exist (e.g., Bargh & Chartrand, 1999).  However, a more nuanced view is that behavior stemming from automatic goal activation is still intentional, in that it promotes the achievement or avoidance of a given outcome.  According to this interpretation, the effects of both implementation intentions and auto-motives occur due to the relevance of a given situation to one’s ongoing concerns; hence, apparently “automatic” behavior exhibited in these experiments occurs at least partially in the service of control.  This argument provides a potential resolution to the volition debate; while some theorists take an immediate approach to the analysis of behavior, and thus come to the conclusion that behavior stems from mainly automatic sources (Bargh & Chartrand, 1999), others take a longer-term view, arguing that behavior stems from controlled plans (Ryan & Deci, 2006).

            Another implication of the above analysis is that the mechanisms that promote long-term behavioral regulation are distinct from the mechanisms that promote in-the-moment behavioral regulation.  Thus, feelings of volition and effort, while unnecessary for in-the-moment behavioral regulation, may be crucial determinants of the formulation of a long-term goal (Ajzen, 1991; Deci & Ryan, 2000; Bandura, 1986).  Once a plan has been effortfully formulated, various processes, including the formulation of sub-plans (Carver & Scheier, 1998), and the perception that one’s behavior is at odds with one’s overarching goal (Devine, Monteith, Zuwerink, & Elliot, 1991), may be necessary to connect the larger goal to the ongoing maintenance of behavior.  In the moment, mechanisms such as implementation intentions may, in a pseudo-controlled way, translate into the expression of flexible, intentional behavior (Gollwitzer, 1999).  The distinction between short- and long-term regulation may also shed some light on why long-term plans sometimes fail and why short-term interventions often do not have long-term effects.  Formulating a long-term plan does not guarantee that behavior will be successfully regulated in-the-moment; likewise, producing a change through relatively automatic processes does not guarantee that those changes will contribute to the long-term regulation of behavior (Devine, Forscher, Austen, & Cox, under review).
Implications for dual-process theories in social psychology
            On the basis of the assumed distinction between behavior that is flexible, intentional, effortful, and accompanied by feelings of control versus inevitable, unintentional, effortless, and not accompanied by feelings of control, social psychological dual-process theories (e.g., Devine, 1989; Brewer, 1988; Fiske & Neuberg, 1990; Fazio & Towles-Schwen, 1999) have typically assumed that in the moment, either a controlled process or an automatic process dominates behavior.  However, the fact that the above four dimensions of behavior do not necessarily neatly co-occur casts doubt on this assumption.  Although some controlled behavior may possess all of the above characteristics, this analysis suggests a more differentiated notion of control that focuses on whether the behavior in question fits a person’s short-term or long-term goals.  Likewise, the focus of dual-process theorists on a fine-grained analysis of behavior within a specific moment has been productive, but the above analysis suggests that dual-process theorists have paid insufficient attention to the ways in which one’s momentary behavior is connected to one’s long-term goals.

References

Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179-211.

Bandura, A. (1986). The explanatory and predictive scope of self-efficacy theory. Journal of Social and Clinical Psychology, 4, 359-373.

Bargh, J. A., Gollwitzer, Peter M., Lee-Chai, A., Barndollar, K., & Trötschel, R. (2001). The automated will: Nonconscious activation and pursuit of behavioral goals. Journal of Personality and Social Psychology, 81, 1014-1027.

Bargh, J. A. (1990). Auto-motives: Preconscious determinants of social interaction. Handbook of motivation and cognition: Foundations of social behavior (pp. 93-130). New York, NY: Guilford Press.

Bargh, J. A. (1994). The four horsemen of automaticity: Awareness, intention, efficiency, and control in social cognition. Handbook of social cognition (2nd ed.). Hillsdale, NJ: Erlbaum.

Bargh, J. A., & Ferguson, M. J. (2000). Beyond behaviorism: On the automaticity of higher mental processes. Psychological Bulletin, 126, 925-945.

Bargh, J. A., Raymond, P., Pryor, J. B., & Strack, F. (1995). Attractiveness of the underling: An automatic power → sex association and its consequences for sexual harassment and aggression. Journal of Personality and Social Psychology, 68, 768-781.

Baumeister, R. F., Bratslavsky, E., Muraven, M., & Tice, D. M. (1998). Ego depletion: Is the active self a limited resource? Journal of Personality and Social Psychology, 74, 1252-1265.

Brewer, M. (1988). A dual process model of impression formation. Advances in social cognition (pp. 1-36). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Carver, C., & Scheier, M. (1998). On the self-regulation of behavior. New York, NY: Cambridge University Press.

Chaiken, S., & Trope, Y. (1999). Dual-process theories in social psychology. New York: Guilford Press.

Deci, E., & Ryan, R. (2000). The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11, 227-268.

Devine, P. G. (1989). Stereotypes and prejudice: Their automatic and controlled components. Journal of Personality and Social Psychology, 56, 5-18.

Devine, P. G., Monteith, M. J., Zuwerink, J. R., & Elliot, A. J. (1991). Prejudice with and without compunction. Journal of Personality and Social Psychology, 60, 817-830.

Fazio, R., & Towles-Schwen, T. (1999). The MODE model of attitude-behavior processes. Dual process theories in social psychology (pp. 97-116). New York, NY: Guilford Press.

Fisk, S., & Neuberg, S. (1990). A continuum of impression formation, from category-based to individuating processes: Influences of information and motivation on attention and interpretation. Advances in Experimental Social Psychology (Vol. 23, pp. 1-74). Elsevier.

Fleming, J. H., Darley, J. M., Hilton, J. L., & Kojetin, B. A. (1990). Multiple audience problem: A strategic communication perspective on social perception. Journal of Personality and Social Psychology, 58, 593-609.

Gailliot, M. T., Baumeister, R. F., DeWall, C. N., Maner, J. K., Plant, E. A., Tice, D. M., Brewer, L. E., et al. (2007). Self-control relies on glucose as a limited energy source: Willpower is more than a metaphor. Journal of Personality and Social Psychology, 92, 325-336.

Gollwitzer, P. M. (1993). Goal achievement: The role of intentions. European Review of Social Psychology, 4, 141-185.

Gollwitzer, P. M., Parks-Stamm, E., Jaudas, A., & Sheeran, P. (2008). Flexible tenacity in goal pursuit. Handbook of motivation science (pp. 325-341). New York: Guilford Press.

Gollwitzer, Peter M. (1999). Implementation intentions: Strong effects of simple plans. American Psychologist, 54, 493-503.

Gollwitzer, Peter M., & Schaal, B. (1998). Metacognition in action: The importance of implementation intentions. Personality and Social Psychology Review, 2, 124-136.

Kenrick, D. T., Griskevicius, V., Neuberg, S. L., & Schaller, M. (2010). Renovating the pyramid of needs: Contemporary extensions built upon ancient foundations. Perspectives on Psychological Science, 5, 292-314.

Langer, E. J., Blank, A., & Chanowitz, B. (1978). The mindlessness of ostensibly thoughtful action: The role of “placebic” information in interpersonal interaction. Journal of Personality and Social Psychology, 36, 635-642.

Maslow, A. H. (1943). A theory of human motivation. Psychological Review, 50, 370-396.

Mendoza, S. A., Gollwitzer, P. M., & Amodio, D. M. (2010). Reducing the expression of implicit stereotypes: Reflexive control through implementation intentions. Personality and Social Psychology Bulletin, 36, 512-523.

Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.

Ryan, R. M., & Deci, E. L. (2006). Self-regulation and the problem of human autonomy: Does psychology need choice, self-determination, and will? Journal of Personality, 74, 1557-1586.

Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychological Review, 84, 1-66.

Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. Psychological Review, 84, 127-190.

Wegner, D. (2002). The illusion of conscious will. Cambridge, MA: MIT Press.

Wegner, D. M. (2003). The mind’s best trick: How we experience conscious will. Trends in Cognitive Sciences, 7, 65-69.

Wegner, D. M., & Wheatley, T. (1999). Apparent mental causation: Sources of the experience of will. American Psychologist, 54, 480-492.

Wheeler, M. E., & Fiske, S. T. (2005). Controlling racial prejudice: Social-cognitive goals affect amygdala and stereotype activation. Psychological Science, 16, 56-63.

---

[1] Part of the volition debate stems from a definition of volition as a quality that makes one’s behavior uncaused by external forces (Bargh & Ferguson, 2000).  However, because psychology assumes that one can at least probabilistically determine the causes of people’s behavior, this definition is not discussed further in this paper.

Six graphs answer questions about the PhD labor market

In my experience, getting an honest, straightforward answer about the post-PhD labor market from most professors in graduate school is about as easy as extracting teeth from the mouth of a sparrow.  Even when an answer is forthcoming, it is too often clouded by unrealistic expectations about the types of careers graduate students want and / or attempts to boost morale in order to increase research productivity.

Fortunately, no less an authority than the National Science Foundation has been conducting rigorous, nationally representative surveys on the US PhD labor market since 1993.  On top of that, the NSF has made the results from its biannual surveys open to the public, both in the form of raw data and in the form of summary statistics.  For a quantitative geek like me, the data are a little slice of heaven.

Thus, both to satisfy my own curiosity and for the benefit of other people who want hard data about PhD labor outcomes, I created the following six graphs with the goal of answering common questions about the STEM PhD labor market.  I will structure my graphs around three questions in particular:  (1) How successful are PhDs at finding the jobs they want?  (2) Where do PhDs go for their jobs and what do they do on the job?  (3) How well compensated are PhDs for the work they do?

The source data come the 2006 survey, which is the most recent dataset made available from the NSF.  A total of 42,955 people were surveyed, or about 5.5% of the total 2006 population of PhDs.  The response rate was 77.9%, which is typical for nationally representative surveys.

And now, on to the questions and graphs!!


Question 1:  How successful are PhDs at getting the jobs they want?

Graph #1: The estimated unemployment rate and rate of people forced to involuntarily take jobs outside their field broken down by PhD category.

Click for a larger image

As you can see from the above graph, the unemployment rates and involuntary out-of-field (IOF) rates varied considerably by PhD category in 2006.  Fortunately for STEM PhDs, the unemployment rates were all relatively low; considerably below the 4.6% for the labor force as a whole in August 2006.  The IOF rates ranged slightly higher than the unemployment rates, peaking at 5.4% for the physical sciences.  Fortunately for me and for my colleagues in psychology, the unemployment and IOF rates in psychology were actually quite low, at 1.3% and 1% each.

Graph #2: Percentage of employed PhDs at differing years since PhD by PhD category.

Click for a larger image

The above graph is an interesting counterpoint to the unemployment and IOF graph in that it suggests that the very fields that have higher unemployment and IOF rates were also relatively “topheavy” – they were fields that had relatively high proportions of people who earned their PhDs a long time ago and relatively low proportions of newly-minted PhDs.  In contrast, the fast-growing fields of computer science and health had proportions of newly minted PhDs that are quite high, at 26.8% and 27.9% each.

In sum, based on the data from 2006, the job prospects for STEM PhDs appear to be relatively bright.  However, job prospects are best for fields like computer science and health that appear to be enjoying strong growth in the labor market.


Question #2: Where do PhDs go for their jobs and what do they do on the job?

Graph #3: Percentage of employed PhDs in various job sectors by PhD category.

Click for a larger image

As shown in the above graph, STEM PhDs do a lot of different things once they are done with graduate school.  Although a large proportion the PhDs surveyed were in either a 4-year academic institution (43.7% of those surveyed in 2006) or another academic institution like a 2-year college (3.4%), 47.1% were in a job outside academia.  The high rates of non-academic jobs might come as a surprise to graduate students who, like me, are immersed in departmental cultures that are highly focused on academia.

Graph #4: Percentage of employed PhDs who reported working that the activity was one of two on which they spent the most time, broken down by PhD category.

Click for a larger image

The above graph illustrates yet again that, based on the 2006 survey, PhDs do a lot of different things once they earn their PhD.  With the exception of Psychology, research was the most frequently listed on-the-job activity across the disciplines; however, even in the most extreme of the research-focused disciplines, survey respondents listed other activities, such as management and administration, as taking a large portion of their time.

In sum, STEM PhDs go to many different employment sectors (both academic and non-academic) and, once there, engage in both research and non-research activities.


Question #3: How well compensated are PhDs for the work they do?

Graph #5: Median reported salary across three selected employment sectors (4-year university, private for-profit, self-employed), broken down by PhD category.

Click for a larger image

The overall median salary for all PhDs was $85,900 in 2006; as a basis of comparison, the median household income in 2006 was $48,201.  However, according to the above graph, the compensation a PhD receives varies at least in part with the job sector the PhD enters.  In particular, across all the STEM disciplines, taking an academic job is equivalent to accepting a (sometimes substantial) pay cut.  The size of this pay cut varies somewhat with the particular PhD category, but across the various PhD categories, PhDs in 4-year universities earn 67% of the salaries of their peers in the private for-profit sector.

Graph #6: Median reported salary by job activity and PhD category.

Click for a larger image

The above graph reinforces the point that as a PhD, your compensation depends on the specifics of your job. In particular, PhDs who listed management and administration as one of the activities on which they spent the most time were better compensated than their peers across the disciplines.  In contrast, PhDs who listed teaching as one of the activities on which they spent the most time consistently worse compensated than their peers across the disciplines.

In sum, based on the 2006 data, most PhDs in STEM disciplines are well-compensated, but the amount of compensation varies considerably with employment sector and the type of work the PhD does.


Hopefully the above graphs helped shed some light on the PhD labor market.  The situation is not all bleak, particularly if one is willing to explore options outside the traditional academic research career.  The trick seems to be figuring out how to translate the skills one earns over the course of the PhD into skills one can market to potential employers.

EDIT: Fixed a broken link.

Unraveling the "obviousness" bias in psychology

In my last post, I argued that the pseudoscience of parapsychology (and in particular the publication of a paper in the Journal of Personality and Social Psychology claiming to provide evidence of precognition) hurts the perception of psychology as a science. This may seem like an obvious argument to make; Bem's paper was published in the flagship journal of social psychology, so it is easy to make the logical jump that this article is representative of the kind of research most social psychologists do. Therefore, the reasoning goes, social psychology is not a real science at all; to quote one comment on the media coverage of the Bem article,

Psychology is such a joke. A demonstration of future events influencing present events would be one of the most important (if not *the* most important) findings in the history of mankind. Yet this demonstration doesn't end up in Science or Nature, but is published in the Journal of Personality and Social Psychology? And some wonder why psychology is still considered pseudoscience....

However, all sciences have their occasional crackpots (in physics, John Baez humorously proposed the Crackpot index to deal with the profusion of physics cranks on Usenet forums). Why should psychology be any more affected by pseudoscientific claims for precognition than physics is by pseudoscientific claims for perpetual motion?

According to a recent paper by Keil, Lockhart and Schlegel, there may be a very good reason that psychology as a field is affected more than physics. Keil and his colleagues argue that psychology findings are afflicted by what I'm calling an "obviousness bias" -- they are viewed as intrinically less difficult to figure out than the findings of other sciences.

Keil and his colleagues took questions from physics, chemistry, biology, psychology, and economics. These questions were pre-tested with adults to be equally difficult and span a wide range of topics (see below).

The questions used in Keil, Lockhart, and Schlegel (2010) study 1.

The authors then asked people of various age-groups, from kindergarten up through adulthood, to rate the questions on difficulty from 1 (very easy to understand) to 5 (very hard to understand). The key finding is that while kindergartners viewed the questions from the various sciences as equivalently difficult to understand, children from 2nd to 8th grade viewed psychology problems as much easier to understand than the other problems. This bias disappeared in adulthood (see graph below).

A graph from Keil, Lockhart, and Schlegel of the mean difficulty score assigned to the natural and social sciences across the various age groups.

Note that children have absolutely no basis on which to judge how easy it is to understand why cooked eggs go from liquid to solid or why it is hard to understand two people talking at once. Children have no experience on which to draw when judging the difficulty of these questions and therefore, presumably, rely on heuristics to answer these questions. Thus, the fact that younger children rated psychology questions as less difficult than natural science questions points to a heuristic that makes psychology facts seem more obvious than other sorts of facts.

The authors found an identical pattern of judgments of questions within the various sub-disciplines within psychology; children judged questions from "harder" sub-disciplines, such as neuroscience and cognitive psychology, as more difficult to understand than questions from "softer" sub-disciplines, such as social psychology and personality psychology. Moreover, when the authors asked adults how easy it would be to figure the questions out just by living and watching things, the adults showed the bias as well -- they rated the psychology questions as easier to figure out than the natural science questions.

While the authors do not explore directly whether the obviousness bias translates into psychological findings being taken less seriously than the findings from other disciplines, the authors do show that people assume they can figure out psychological findings on their own. This could very easily lead to a perception that less scientific rigor is required for research psychology than for other sciences -- after all, anyone can do psychology just by experiencing life. This means that anything that reflects on the integrity of psychology as a scientific discipline -- like pseudoscientific work on precognition -- is likely to be especially damaging for psychology.

Reference:

Keil, F. C., Lockhart, K. L., & Schlegel, E. (2010). A bump on a bump? Emerging intuitions concerning the relative difficulty of the sciences. Journal of Experimental Psychology: General, 139, 1 - 15.