Help! There's an Epidemic of Anxiety! (Part II)

18.30 wsn

In my last-post-but-one I slammed the claim that the British are suffering from an epidemic of anxiety disorders. I declared it a myth pushed by the Mental Health Foundation and echoed uncritically by British newspapers (although The Economist has since run a kind-of skeptical piece on it.) But I also promised that there are important lessons to be learned here. So, here we go:

The Mental Health Foundation produced a report, In The Face of Fear, which contains various interesting thoughts about the role of fear in public debates. Here's just one:

Individually we experience both rational and irrational fears that drive our behaviour and fear also drives communities and social policies... Excessive fear poses an enormous burden on our society directly through anxiety related illness, which can be physical as well as mental, and indirectly through inappropriate behaviours such as excessive supervision of children or failure to invest. It also paralyses long term rational planning to deal with key future threats such as global warming by diverting attention to more immediate but less important fears.
This is true. Everyone should be scared of global warming. Most people aren't. They're scared of... well, it varies. Cervical cancer was scary a few weeks ago, before that it was the crisis in child protection services, right now it's the Mexican swine flu crisis - not to mention the economic crisis, the knife crime "crisis" - and that's just England.

I'm not saying that we shouldn't care about these things. I'm worried about Mexican swine flu, and so should you be. Especially you, Simon "The Armchair Virologist" Jenkins. But in the face of crisis after crisis after crisis, it becomes hard to take the really crucial crises, such as global warming, seriously. There's a temptation to see every apparant crisis as just another piece of overblown nonsense in need of "debunking" as Ben Goldacre has just discovered. One could call this "crisis fatigue", but that's not exactly right. We're too fond of crises. There are just too many of them.

This is why the MHF felt the need to be so "creative" with the data. As I explained in Part I of this post, the best available figures show that the prevalence of anxiety disorders in Britain has remained boringly level since at least 2000. The MHF simply ignored those numbers in order to make it look as though we're currently facing an epidemic of anxiety. A crisis.

I wish they hadn't. But I don't really blame them for what they did. They did it because they knew that if they didn't, no-one would care about anything they had to say. In an ideal world they would have said: Although British anxiety and depression levels are probably not rising, and although they're not as high as in some countries, they're still higher than in other countries, so we can and should try harder to reduce them. That's the truth. But the truth doesn't involve a crisis, so it wouldn't have made the headlines, or if it did, no-one would have cared. Thus it is that a report warning (inter alia) about the dangers of scaremongering ended up becoming a prime example of scaremongering.

This is the point where, conventionally, one blames "the media" for only publishing "sensationalist" stories in order to "sell papers". Well, that's all true. But the media don't behave that way just for fun. A sensational story is a good story. People want sensationalist stories. Nothing wrong with that, as such. And there's nothing wrong with caring more about a crisis than about a mere problem. A crisis, by definition, is something that deserves urgent attention.

But the result of this is that today, in order to get attention, a problem has to be a crisis - something which is bad and getting worse, fast. Just being a problem in need of a solution isn't enough. There are too many problems - no-one can possibly care about them all. Whereas if something is a crisis, it might just get a little attention. Hence why the MHF had to do what they did. They needed a crisis, so they created one.

If I were a humanities graduate, I would now start explaining how it's all the fault of our postmodern, "post-historical" condition in which there are no grand narratives or central moral authorities to tell us what to care about, leaving every political or moral cause (and organization) to fend for itself in a Darwinian (or market) struggle for attention (and money) in which the only way to survive is to adopt the language of panic, crisis, and emergency thereby devauling that very discourse in a cultural tragedy-of-the-commons. But I'm a science graduate, so I wouldn't dream of doing that.

[BPSDB]

Posted in , ,
Diposting oleh wsn

More Brain Voodoo, and This Time, It's Not Just fMRI

05.28 wsn

Ed Vul et al recently created a splash with their paper, Puzzlingly high correlations in fMRI studies of emotion, personality and social cognition (better known by its previous title, Voodoo Correlations in Social Neuroscience.) Vul et al accused a large proportion of the published studies in a certain field of neuroimaging of committing a statistical mistake. The problem, which they call the "non-independence error", may well have made the results of these experiments seem much more impressive than they should have been. Although there was no suggestion that the error was anything other than an honest mistake, the accusations still sparked a heated and ongoing debate. I did my best to explain the issue in layman's terms in a previous post.

Now, like the aftershock following an earthquake, a second paper has appeared, from a different set of authors, making essentially the same accusations. But this time, they've cast their net even more widely. Vul et al focused on only a small sub-set of experiments using fMRI to examine correlations between brain activity and personality traits. But they implied that the problem went far beyond this niche field. The new paper extends the argument to encompass papers from across much of modern neuroscience.

The article, Circular analysis in systems neuroscience: the dangers of double dipping, appears in the extremely prestigious Nature Neuroscience journal. The lead author, Dr. Nikolaus Kriegeskorte, is a postdoc in the Section on Functional Imaging Methods at the National Institutes of Health (NIH).

Kriegeskorte et al's essential point is the same as Vul et al's. They call the error in question "circular analysis" or "double-dipping", but it is the same thing as Vul et al's "non-independent analysis". As they put it, the error could occur whenever

data are first analyzed to select a subset and then the subset is reanalyzed to obtain the results.
and it will be a problem whenever the selection criteria in the first step are not independent of the reanalysis criteria in the second step. If the two sets of criteria are independent, there is no problem.


Suppose that I have some eggs. I want to know whether any of the eggs are rotten. So I put all the eggs in some water, because I know that rotten eggs float. Some of the eggs do float, so I suspect that they're rotten. But then I decide that I also want to know the average weight of my eggs . So I take a handful of eggs within easy reach - the ones that happen to be floating - and weigh them.

Obviously, I've made a mistake. I've selected the eggs that weigh the least (the rotten ones) and then weighed them. They're not representative of all my eggs. Obviously, they will be lighter than the average. Obviously. But in the case of neuroscience data analysis, the same mistake may be much less obvious. And the worst thing about the error is that it makes data look better, i.e. more worth publishing:
Distortions arising from selection tend to make results look more consistent with the selection criteria, which often reflect the hypothesis being tested. Circularity is therefore the error that beautifies results, rendering them more attractive to authors, reviewers and editors, and thus more competitive for publication. These implicit incentives may create a preference for circular practices so long as the community condones them.
To try to establish how prevalent the error is, Kriegeskorte et al reviewed all of the 134 fMRI papers published in the highly regarded journals Science, Nature, Nature Neuroscience, Neuron and the Journal of Neuroscience during 2008. Of these, they say, 42% contained at least one non-independent analysis, and another 14% may have done. That leaves 44% which were definitely "clean". Unfortunately, unlike Vul et al who did a similar review, they don't list the "good" and the "bad" papers.

They then go on to present the results of two simulated fMRI experiments in which seemingly exciting results emerge out of pure random noise, all because of the non-independence error. (One of these simulations concerns the use of pattern-classification algorithms to "read minds" from neural activity, a technique which I previously discussed). As they go on to point out, these are extreme cases - in real life situations, the error might only have a small impact. But the point, and it's an extremely important one, is that the error can creep in without being detected if you're not very careful. In both of their examples, the non-independence error is quite subtle and at first glance the methodology is fine. It's only on closer examination that the problem becomes apparent. The price of freedom from the error is eternal vigilance.

But it would be wrong to think that this is a problem with fMRI alone, or even neuroimaging alone. Any neuroscience experiment in which a large amount of data is collected and only some of it makes it into the final analysis is equally at risk. For example, many neuroscientists use electrodes to record the electrical activity in the brain. It's increasingly common to use not just one electrode but a whole array of them to record activity from more than brain one cell at once. This is a very powerful technique, but it raises the risk the non-independence error, because there is a temptation to only analyze the data from those electrodes where there is the "right signal", as the author's point out:
In single-cell recording, for example, it is common to select neurons according to some criterion (for example, visual responsiveness or selectivity) before applying
further analyses to the selected subset. If the selection is based on the same dataset as is used for selective analysis, biases will arise for any statistic not inherently independent of the selection criterion.
In fact, Kriegeskorte et al praise fMRI for being, in some ways, rather good at avoiding the problem:
To its great credit, neuroimaging has developed rigorous methods for statistical mapping from its beginning. Note that mapping the whole measurement volume avoids selection altogether; we can analyze and report results for all locations equally, while accounting for the multiple tests performed across locations..
With any luck, the publication of this paper and Vul's so close together will force the neuroscience community to seriously confront this error and related statistical weaknesses in modern neuroscience data analysis. Neuroscience can only emerge stronger from the debate.

ResearchBlogging.orgKriegeskorte, N., Simmons, W., Bellgowan, P., & Baker, C. (2009). Circular analysis in systems neuroscience: the dangers of double dipping Nature Neuroscience DOI: 10.1038/nn.2303

Posted in , , , ,
Diposting oleh wsn

Help! There's an Epidemic of Anxiety! (Part I)

01.32 wsn

All British journalists are psychotic. Pathologically obsessed with "mental health issues", and suffering from grandiose delusions of their competence to discuss them, these demented maniacs...

Sorry. I got a bit carried away there. But you'll forgive me, because I was just following the example of seemingly everyone in the British media these past couple of weeks. If you believe the headlines, we're in the grip of an epidemic of anxiety:

BBC: UK society 'increasingly fearful'
The Telegraph: Britons 'living in fear' as record numbers suffer from anxiety

The Independent: Britain is becoming a more fearful place – and the economy is paying the price. The Indie also ran a comment by Janet Street-Porter - "The main reason people feel anxious is loneliness.", thanks Janet, qualifications: none, career path: fashion journalist - and a piece by a clinically anxious person - "I reckon a root cause of my anxiety is the modern notion that we can do away with risk by anticipating every imaginable danger."
It all started with a report by the Mental Health Foundation called In The Face Of Fear. The Mental Health Foundation are a perfectly decent charity organization, although they have a prior history of endorsing slightly dodgy research. One of their previous reports, Feeding Minds: The Impact of Food on Mental Health, presented a simplistic and overblown account of the effects of nutrition upon mood and drew heavily on the "work" of Patrick Holford, vitamin pill peddler and well-documented crank. Parts of the present report are, unfortunately, dodgy as well, as you'll see below.

In The Face of Fear is actually quite thought-provoking piece of writing, but you wouldn't know that from reading the newspapers. The headlines are all about the supposed surge in anxiety amongst the British population. This, however, is the dodgiest part of the report. Firstly, the report's authors surveyed 2246 British adults in January 2009. 37% said that they get frightened or anxious more often than they used to, 28% disagreed, and 33% neither agreed nor disagreed.

That's it. That's the finding. It's really not very impressive, because quite apart from anything else, it relies upon the respondent's ability to remember how anxious they were in the past. You just can't trust people to do hard stuff like that. I know exactly what I'm worried about today - I can't remember very well what I worried about ten years ago - so I must be more worried today! Of course, this could also work in reverse, and people might forget their past lack of anxiety and wrongly say that they are less anxious today.

The survey also found that 77% of people said that "people in general" are more anxious than they used to be, while just 3% disagreed. But remember that only (at most) 37 out of those 77% said that they themselves were actually more anxious. Hmm. So the real finding here seems to be that there is a widespread perception that other people are becoming more anxious, though it's anyone's guess whether this is in fact true. The report itself does note that
more than twice as many of us agree that people in general and the world itself are becoming more frightened and frightening as agree that they themselves are more frightened and anxious
This was rather too subtle for the newspapers, though, who reported... that people are becoming more anxious.

In The Face of Fear also cites a government study on the mental health of the British population, the Adult Psychiatric Morbidity Survey. Their use of this data, however, is selective to the point of being deception. This was a household survey of a weighted sample of the British population. That section of the population who live in houses and don't mind being interviewed about their mental health, that is. Diagnoses were made on the basis of the CIS-R interview, which scores each person on a number of symptoms (including "worry", "fatigue", and "depressive ideas"). Each person is then given a total score; a total score of 12 or more is (arbitrarily) designated to indicate a "neurotic disorder".

This was done in 1993, 2000 and 2007. The 2007 report notes that overall, levels of neurotic disorders increased between 1993 and 2000, but then stayed level in 2007. In terms of anxiety disorders, there was a very small increase in "generalized anxiety disorder" (from 4.4% to 4.7%), which mostly happened between 1993 and 2000; there was an increase in phobias, from 1993 2.2% to 2007 2.6%, but rates peaked at 2.8% in 2000; and "mixed anxiety and depressive disorder" increased from 7.5% in 1993 to 9.4% in 2000 to 9.7% in 2007.

What to make of that? It's hard to know, but it's clear that any worsening in anxiety levels occured some time between 1993 and 2000. Mysteriously, while the Mental Health Foundation report cites the 1993 and the 2007 figures, and makes much of the increase, it simply ignores and does not mention the 2000 figures, which show that any increase has long since stopped. It's history, not current events. Back in 2000, you might recall, the twin towers were still standing, The Simpsons was still funny, and Who Let The Dogs Out was top of the charts.

Overall, the evidence that people in Britian are actually feeling more and more anxious is extremely thin. In fact, I would say that it's a myth. It's a very popular myth, however: 77% of the population believe it. Why? Well, the fact that the Mental Health Foundation seem determined to make the data fit that story can't be helping matters. The newspapers, not to be outdone, focussed entirely on the scariest and most pesimissitic aspects of the report.

A poor show all round, but - as always on Neuroskeptic - there are some important lessons here about how we think about threats, social change, and "crisis". Stay tuned for the good stuff next post.

[BPSDB]

Posted in , ,
Diposting oleh wsn

The Voodoo Strikes Back

16.50 wsn

Just when you thought it was safe to compute a correlatation between a behavioural measure and a cluster mean BOLD change...

The fMRI voodoo correlations controversy isn't over. Ed Vul and collegues have just responded to their critics in a new article (pdf). The critics appear to have scored at least one victory, however, since the original paper has now been renamed. So it's goodbye to "Voodoo Correlations in Social Neuroscience" - now it's "Puzzlingly high correlations in fMRI studies of emotion, personality and social cognition" by Vul et. al. 2009. Not quite as catchy, but then, that's the point...

Just in case you need reminding of the story so far: A couple of months ago, MIT grad student Ed Vul and co-authors released a pre-publication manuscript, then titled Voodoo Correlations in Social Neuroscience. This paper reviewed the findings of a number of fMRI studies which reported linear correlations between regional brain activity and some kind of measure of personality. Vul et. al. argued that many (but by no means all) of these correlations were in fact erroneous, with the reported correlations being much higher than the true ones. Vul et. al. alleged that the problem arose due to a flaw in the statistical analysis used, the "non-independence error". For my non-technical explanation of the issue, see my previous post, or go read the original paper (it really doesn't require much knowledge of statistics).

Vul's paper attracted a lot of praise and also a lot of criticism, both in the blogosphere and in the academic literature. Many complained that it was sensationalistic and anti-fMRI. Others embraced it for the same reasons. My view was that while the paper's style was certainly journalistic, and while many of those who praised the paper did so for the wrong reasons, the core argument was both valid and important. While not representing a radical challenge to social neuroscience or fMRI in general, Vul et. al. draws attention to a widespread and potentially serious technical issue with the analysis of fMRI data, one which all neuroscientists should be aware of.

That's still my opinion. Vul et. al.'s response to their critics is a clearly worded and convincing defense. Interestingly, their defense is in many ways just a clarificiation of the argument. This is appropriate, because I think the argument is pretty much just common sense once it is correctly understood. As far as I can see the only valid defence against it is to say that a particular paper did not in fact commit the error - while not disputing that the error itself is a problem. Vul et. al. say that to their knowledge no accused papers have turned out to be innocent - although I'm sure we haven't heard the last of that.

Vul et. al. also now make explicit something which wasn't very clear in their original paper, namely that the original paper made accusations of two completely seperate errors. One, the non-independence error, is common but probably less serious than the second, the "Forman error", which is pretty much fatal. Fortunately, so far, only two papers are known to have fallen prey to the Forman error - although there could be more. Go read the article for more details on what could be Vul's next bombshell...

ResearchBlogging.orgEDWARD VUL, CHRISTINE HARRIS, PIOTR WINKIELMAN, AND, & HAROLD PASHLER (2009). Reply to comments on “Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition” Perspectives in Psychological Science

Posted in , , ,
Diposting oleh wsn
Langganan: Postingan (Atom)
 
powered by Blogger