An Acupunture Meta-Analysis


A recent meta-analysis of acupuncture studies for chronic pain by Vickers et al is getting a great deal of press. The authors’ conclusions are:

Acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo. However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture.

News reports generally reflect this conclusion – acupuncture works, but mostly (although not entirely) through placebo effect, but that’s OK.

I took a close look at the study and find that the authors display considerable pro-acupuncture bias in their analysis and discussion. They clearly want acupuncture to work. That aside, the data are simply not compelling, and the authors, in my opinion, grossly overcall the results, which are compatible with the conclusion that there are no specific effects to acupuncture beyond placebo.

The meta-analysis looked at 29 randomized clinical trials of acupuncture in back pain, neck pain, headache, and osteoarthritis, involving both sham and no-acupuncture controls. The differences between acupuncture and no-acupuncture were large, reflecting an absolute reduction in pain of about a 30% (50% relative reduction). However, the authors acknowledge:

Because the comparisons between acupuncture and no-acupuncture cannot be blinded, both performance and response bias are possible.

In other words – the unblinded comparison between acupuncture and no acupuncture is entirely overwhelmed by bias and completely useless. The no acupuncture control groups involved patients continuing to receive usual care (whatever they were already receiving that was not effective, or sometimes just being told not to get acupuncture). This was not a comparison to any specific medical intervention. In other words, the subjects were aware they were receiving no treatment.

Written By: Steven Novella
continue to source article at


  1. Meta-analysis?

    Danger signals flash every time I read or hear that.

    Funnel plot problems.

    File drawer problems. ( of two types )

    Agenda bias problems.

    Anyone with a bit of statistics will know what I am talking about. Or, just Google meta-analysis–problems.

    Seems Steven Novella has got this studies number also!

  2. Although I have a mostly good knowledge of statistics, I don’t know much about meta-analysis methods. But it seems to me not only possible but frequent that a null hypothesis which few if any studies undermine somehow gets the thumbs down after the pooling is done. Why do I say that? Because the least plausible claims which somehow get alleged statistical support always get purely meta support; this acupuncture case is just 1 example.  Aspirin is a painkiller; statistics prove that. Did we believe it because of a meta-analysis? Was any medical truth established that way?

  3. I have always wondered…: how is the intensity (or reduction) of pain measured “objectively”? Are there biological markers that can be measured, or is it a subjective “how do you feel today”?

    Please excuse my ignorance…

  4. For humans, their own testimony is the “gold standard”. There is the Mosby Scale, typically used in ER triage settings, whereby the patient is asked on a scale of 0 – 10 (or facial pictures showing a range of emotion) to describe the intensity of their pain. Objective measurements to corroborate pain could include increased rates of respiration, pulse, or changes of activity level among other signs, but those are only helpful aids, not necessarily evidence of pain because other factors could be responsible. Brain activity imaging is another modality to assess pain.

    Recognition and alleviation of pain in laboratory animals:

    And here are a few other examples for humans:


    P.S. Perhaps an expert can chime in.

  5. It may be unwarranted, but nevertheless I always think of the phrase, “the plural of anecdote is not data” when I hear the word meta-analysis. Your question, “was any medical truth established that way?” is a good one.


  6. I would have though a simple test where both parties accept that they are receiving acupuncture but only one group is, while the other receives a mere prick in the back. If they respond the same—placebo.

  7. I had acupuncture administered by a Physiotherapist who was overseeing my recovery from a cartilage operation.

    She said she couldn’t tell whether it works or not, but on balance thought it didn’t do any harm, and might have a beneficial placebo effect; and of course, I wasn’t paying for it, she was an NHS nurse.

    I do know that on one ocassion when I caught my ankle bone on a table edge, by the time I pulled up my trouser leg to see the damage – a few seconds at most – a bump had formed; in other words the white blood cells were doing their work immediately. So perhaps when a needle enters the skin the same thing happens.

    That’s a form of healing, but only after damage has been deliberately done by puncturing the flesh in the first place.

    One thing is certain though, this practice wasn’t being carried thousands of years ago in China; the technology for making such fine needles didn’t exist until about a hundred and fifty years ago.  

  8. Personally, I’ve never been swayed by the “can’t harm argument.” First, in the case of acupuncture, that’s not necessarily true as poor sterile technique carries the risk of infection. But something else that is overlooked is the economic harm by therapies in that “can’t harm” column. Your taxes are paying for it when certainly such funds could be better spent on science based research.

    In the US, the National Center For Complimentary and Alternative Medicine is a 40 billion dollar cow last I checked. Granted, the NCCAM researches more than just acupuncture. There’s homeopathy for one…


    P.S. Spot on about Chinese technology.

  9.  The “can’t harm” argument is indeed rather pathetic. Homeopaths tend to use it. It surely doesn’t say much for a “medicine” if the best claim made for it is “it won’t poison you”, which invites the reply “neither will plain water”.

  10. If I recall correctly, recent studies on the efficacy of acupuncture have used very cleverly designed placebo needles that not even the the applicator can determine whether they are the real needles or not.  From what I recall, there was no difference detected between the use of real needles and placebo needles, just as there was no difference between application of needles at so-called meridian points and needles applied to random locations.  These recent studies (if I am recalling them correctly) strongly indicate that acupuncture is nothing more than the placebo effect.

  11. Homeopathy does harm our reputations as intelligent animals who are capable of scientific inquiry.

  12. I am not an expert (chronic pain management is not the major focus of my work) but I do manage acute pain on a near-daily basis – a few thoughts if I may:

    Jos Gibbons, I don’t know if any medical truth (where truth = as close to scientific ‘theory’ as we can get) can be established by meta-analysis. The necessary precursor is good systematic review of high-quality (generally double-blind randomised control trial) studies which themselves should be at least hinting towards the eventual ‘truth’ such as ‘aspirin relieves pain’. If the finding of interest is not one of the primary/secondary endpoints, you could certainly question the study’s validity for showing it and hence its inclusion in the meta-analysis.  It can certainly help confirm or reinforce the inkling that we’re on to something not entirely intuitive e.g. the use of beta-blockers in congestive heart failure. Not too long ago, you might have been chided for suggesting a drug that reduced the contractility of an already failing heart was good idea. But since the late 90’s it had been noted during trials of other cardiac medications that those already on beta-blockers had reduced rate of death over those who were just on the trial drug. Today, it is a mainstay of treatment (with other medications).

    Here is an idiots (doctors who hate actual statistics) guide to meta-analysis from the Oxford Biomedical Science Division… Probably too basic for you but it handily includes a forest plot of some of the studies on beta-blockers on page 5 (although it does not label it as such – just as an illustration of what a forest plot looks like). As Sample inferred, it can reinforce anecdotal wishful-thinking if not done well and garbage in-garbage out applies.

    Mancino, I believe, in the case of pain, human testimony is unfortunately all we have to go on – certainly in the clinical setting, due to its inherently subjective nature. It is possible to use pain scales that are validated to at least differentiate between whether is it mild/quite bad/very bad and to control for fear/anxiety/the actual illness. Try: http://pediatrics.aappublicati… – I pick this small study just as an example of how the authors have tried to get around the problem of just asking – such as differing the orientation of the scales between patients, asking those not in pain to score themselves and seeing if both scales score to the same degree and direction. Certainly one could argue that how the patient feels is all that matters – there are many conditions (e.g. types of neuropathic pain and compartment syndrome) where pain (initially) out of proportion to  other physical symptoms and signs is expected.

    Sample, some interesting attempts have been made to be objective. In neonates (babies up to 28 days or premature infants up to the equivalent post-due-date), 24% sugar syrup is dropped on the tongue routinely before unpleasant procedures (taking blood, putting in venous access lines) – anecdotally, they stop crying and screwing their faces up and look like they’re happily sucking on sugar. It has been questioned whether or not this is the same as not being in as much pain. A study in the Lancet… looked into this by measuring EEG (pain-specific ‘brain-wave’ activity) and EMG of reflex withdrawal to a ‘heel-prick’ blood-sampling procedure. They showed that the brain activity and withdrawal was not reduced in the sugar group even though the analogue observer   ‘pain rating ‘ score was. They argue that babies may not show they are in pain in the way we expect with older kids and adults. However, this does not mean they aren’t distracted in some way from the noxious stimulus and hence, ‘feeling’ (overall) less pain. Of course this doesn’t answer the question of what we should do instead…

  13. As someone else wrote here. Why do we need Meta-analysis? That seems to suggest we can’t trust the independent studies on their own right. Correct me if I’m wrong, but I have never heard the term meta-analysis being used with regard to medical research in general…

  14. Fascinating post. The Lancet study seems counter-intuitive to me (but what do I know). By giving the sugar first, couldn’t that risk conditioning the patient to anticipate subsequent pain? Your point about distraction is understood.

    As an aside, the reverse is practiced in the veterinary setting (perhaps not with primate vets though!). Non-human animals (particularly dogs) are often given copious praise/treats after noxious stimuli (viscous vaccinations, etc.). Post event rewards do appear to work (by work I mean remaining still for future injections) though I’m unaware of any official study supporting this observation.


  15. Nunbeliever, I suppose you don’t need meta-analysis per se. If cause and effect is barn-door or all-or-nothing (extreme example – reattaching a severed finger leads to better restoration of function than not reattaching it), you might not even need the trial.

    Possibly a bit unfair to say we can’t trust studies (singular) because studies (combined) happen to give different answers. The point is not just to combine studies into a ‘mega-study’ equivalent and get a ‘better’ overall answer. It can show if a particular study is an ‘outlier’ in how much it may over or understate (any) treatment benefit. A meta-analysis can also control for bias in individual studies. It also forces you to lay out your reasoning for including/interpreting studies, good or bad.

    Part of the problem, which has been pointed out in the acupuncture analysis above, is that deciding to call a 5% difference between treatment groups ‘significant’ is often arbitrary and does not mean this translates to a clinical difference between patients. Sometimes it can be tempting to pick a positive finding when you find ‘statistics’ to back this up. It works the other way though – one of my old lecturers in clinical pharmacology pointed out that a major study showing the effectiveness of aspirin in heart attacks (forgot which one) statistically shows no benefit if you’re a Sagittarius 😛

    There are academic networks such as the Cochrane Collaboration ( whose purpose is to put together the combined evidence for specific healthcare ‘questions’ and often use meta-analysis in their reasoning. For those trying to set treatment policy they can be very useful.

    (EDIT: in the 3rd paragraph, by 5% difference, I of course meant a p-value of 0.05 to decide if a result is significant or not. This may not be anything like a 5% difference. Naughty Docjitters – slap wrist)

  16.  Well if not counter-intuitive, potentially disturbing. Whilst there’s no getting away from needing to do bloods tests occasionally, nobody wants to think that inside the child might be suffering…

    I suspect that the sugar thing is simply a distraction with a pleasant stimulus as sometimes it is done whilst a baby is feeding on milk. There are a few studies to show it doesn’t work if you put the sugar straight into the stomach via nasogastric tube.

    The reverse (concurrent/subsequent reward) is also very much practised in little humans ( – usually in the form of blown bubbles, stickers, cuddles, iPhone apps that make animal noises and parentally-promised bars of chocolate). Sometimes they’ll hold still if they need serial tests knowing a reward is on the way.

    The reason I chose the neonatal sucrose study is that they’re thought to be too young to anticipate reward which controls for possible bias there. Of course, conversely, many older children do learn that the bubbles come out at blood-testing time and subsequently object, usually loudly.

    I’m sure you appreciate that sometimes you have to do what works, even if you don’t know why it does (though probably not acupuncture)…

  17. Stafford, nature has been making such fine needles for millions of years. Pyracantha angustifolia for example has long sharp spiny thorns which are excellent for puncturing the skin, as my scars can testify after much hard-pruning experience.

     It was introduced to Europe from China. 

  18. This is not related to the post…
    I just wanna thank all of you the commentators.
    And the RDF people for such a great site. Every time I come here is so enlightening to read all the notes.My favorite is to read you guys elaborating, and giving opinions is so awesome. Some of us don’t have PHD’s, but try to keep up and learn from you guys.
    I always feel at home here. I have bought most of Richards books and all the other horsemen. Big Fan of Krauss and self proclaimed biggest fan of Christopher Hitchens. When he passed away I felt devastated (sigh). This is sincerely an oasis of knowledge. Just wanted to show my gratitude for helping me grow mentally. I’m a Latin American male from a country called El Salvador ( Spanish for the savior ). I have been an atheist for over 8 years, and Richard’s work has kept me grounded on this freedom, that has lifted such a burden carried for many years. Thanx Mods you guys Rule I love you 🙂
    Gracias y mucho cariño para ustedes.

Leave a Reply