Which scientists can you trust?

Suggested listening: [Morcheeba – Who Can You Trust?]

How do you sort the "good" scientists or the "bad" without actually going to check out their labs? This was part of the question asked by Daily Mail science editor Michael Hanlon at the Goldacre vs Lord Drayson debate at the Royal Institution last week.

This is an important question in light of the over-blown MRSA scares only a few years ago. Ben Goldacre has written a multitude of articles and blogs about this, exposing the so-called “expert” used by newspapers Dr Christopher Malyszewicz as a fraudster with a lack of proper qualifications, who “got false positive results from his garden shed laboratory”.

Actually checking out people’s labs is something that Dr James Logan PhD says is impractical and can be misleading:

“A trained scientist could even find it hard to tell what quality of research is going on in a lab just by having a look around – after all, in my area we get some high quality research coming out of labs in Africa and most of their labs are very run down and poorly resourced, yet the quality of the science that they are able to do there is not necessarily compromised.’

So what can journalists do to ensure that their sources are bona fide proper scientists? Of course, the quality of the scientific method of an individual research paper and its peer-reviewed context are the most important things to look at, rather than its author (more on this if you read on).

“Obviously though, reputation and previous can influence opinion, but in an ideal world a reporter should be only assessing the quality of the work,” says Adam Rutherford, editor at Nature, “Does it affect the quality of Goodfellas that Age of Innocence was pony?”

However, chatting to Vaughan Bell, blogger for Mindhacks.com in the aftermath of the Godacre vs Drayson debate (which was a re-enactment of their previous widely-publicised stances, read more here), many questions were raised in my mind about the scientists themselves.

Especially in the light of research that Goldacre highlighted in his talk about how people, and indeed scientists, are influenced by the popularity of a science story in the MSM. In the case of the scientists, they tended to cite studies with more frequency if they are covered in the media. See here for details of the study, in which academics were influenced by coverage in the NY Times. (Phillips DP et al. N Engl J Med. 1991;325:1180-3.)

Also worrying is this recent article from the Guardian about scientist selling their signatures to big pharma research. 

Another reason that quantity of citations is not a good indicator of a good scientist or the quality of a paper is that studies can be cited as examples of controversial or bad research that the paper wishes to contradict.

Bell says using Google Scholar to see how many times studies have been cited can be a fairly good indicator of relevance – but whether this is for good of bad reasons, we must bear in mind its popularity is also tempered by the influence of the media.

All in the rep

So how else can we assess if a scientist is “good” or “bad”? A lot rides on reputation. Dr Logan says that “Reputation is probably a good indicator, and would be assessed by talking to peers.” This kind of snooping around can prevent another Dr Christopher Malyszewicz-style disaster.

Of course, reputation becomes a lot more relevant when you are not looking at a research paper, but something else as a basis of a report. This could be notes from a conference plus an abstract of the research, the result of a meeting, press conference or other less- or un-scrutinised channels.

“Then assessing credibility becomes much more relevant,” says Rutherford, “At this point I think looking at publication record can be a useful way of assessing the quality, impact and import of a researcher.”

Of course, we have to be prepared that we might be surprised by a scientist deviating from their track record – “As happens with reputation in every other walk of life,” says Bell, “How do you know Tom Cruise isn’t going to make a good Hamlet?”

Even if a scientist’s reputation is good, it doesn’t necessarily follow that all of their claims are founded in good science. Mark Henderson, science editor at The Times, nudges me in the direction of his blog on Tracy Alloway’s controversial statements about Twitter dulling the memory – which she then admitted wasn’t substantiated with any research whatsoever. Just a hypothesis, then.

Would now be an appropriate time to mention Ida? 

Rutherford comments: “Some of the members of the team behind the world’s greatest ever scientific discovery in the world ever that changed the universe for ever aka Ida the lemur, made claims that were not supported by the peer reviewed evidence. Any further publications from those team members should be treated with the same levels of scrutiny as any other academic peer reviewed paper.”

Peer pressure?

Tricky stuff; at least with a peer-reviewed paper you know that claims about it are based on research that has been given the stamp of approval by reviewers. But can the hallowed tradition of peer review also be susceptible to bias?

In theory, no. But in reality, almost certainly. Occasionally, “bits of “science junk” slip through [peer review] – even in journals like Nature, Science and PNAS,” says Dr Logan. See this analysis of retraction of research papers by Nature.

And here is an example of  dodgy science in PNAS (via Ed Yong).

Dr Logan also says that this is not down to lack of rigour in the peer review process, in his experience. So that leaves the shifty-looking candidates of unscrupulous editing, or nepotism amongst reviewers. Of course the opposite can happen: “Competing scientists can slam a manuscript simply because they don’t like the authors or because they want to be the first people to publish that work,” says Dr Logan.

One scientist, a biological physicist I chat to over Twitter, Dr Ian Hopkinson PhD, agrees that there is politics operating within the scientific community.

SmallCasserole@christineottery Yes - politics exist. Diversity of sources probably best. Many sources should give equally good answers.

@SmallCasserole Ooh thanks for that. So majority rule?

SmallCasserole@christineottery Unless you know your scientist well, I'd say "yes". Scientist tend to be a bit protective of their turf

Outside the box

This leads me to pondering about one so-called maverick scientist that I have recently heard speak at a Skeptics in the Pub meeting: Aubrey de Grey.

De Grey claims that “The first 1,000-year-old is 20 years younger than the first 150-year-old.” And guess what? He thinks they’re alive now.

His idea for increasing longevity is to combat disease in the body by creating therapies to treat the onset of aging, such as de-fuzzing arteries of cholesterol. He proposes this can be done by isolating the genes in certain strains of bacteria used to break down cholesterol, for example, to formulate as part of an injected panacea.

Not having read any of his research papers, I couldn’t comment on the quality of the science. However, I was fascinated by De Grey as a character, and it seems he has both his opponents and, er, non-opponents in the scientific community. I was curious whether he was too marginal and outlandish to be important or whether he was in fact some kind of genius visionary. Only the rigorous testing and verification of his theories will prove this one way or another.

Rutherford makes the point that with someone like De Grey, even a sidelined position can grow to appear more mainstream then it is within the media. This is probably because it is interesting, as in unorthodox, and also because hunger of the press to regurgitate pithy soundbites. Rutherford says: “A good journalist would only reiterate this... if the argument was sound or the evidence compelling.”

Mavericks are shown over time and testing to belong to one of two camps. One: Their “heretical” ideas are supported by evidence and embraced by conventional science, such as Lynn Margolis on endosymbiosis (a theory that parts of cells originally came from bacteria) or James Lovelock’s Gaia theory.

Two: Or, their maverick status becomes “self-selecting” – in other words their work is not proven and they therefore chose to be regarded as anti-establishment. For an example of this see this piece by Rutherford on Rupert Sheldrake.  

Ahead of such a time when a scientists' theories are proven or dismissed, how should the media handle mavericks?

Henderson tells me it is fine to write up a maverick’s untested views as long as the caveats about lack of proof and peer criticisms about their work are high up in the article. He warns: “Ask yourself, though, if the hypothesis is plausible, and whether it contravenes things that are thought to be well-established. If so, it's worth applying the maxim of extraordinary claims needing extraordinary evidence.”

Scientist-journalist relations 

These are worth forging as a journalist. Time to get connected.

Henderson suggests that beyond ringing the authors and independent scientists for their views on the research, it can be useful putting papers you are reading through “a kind of informal peer-review” by running them past scientists you know in the relevant field to get their comments.

Dr Logan agrees, and would like to see more scientists’ views represented in articles. He says: “It is rare that the journalist will change the piece to be more balanced when they have a skepticism from another scientists – after all that will ruin a perfectly good and interesting story! It’s usually tagged on the end.”  (BTW, Dr Logan big-ups the New Scientist for “giving a fair and balanced review of controversial research”).

From Dr Logan’s experience working with journalists on a skills exchange initiative run by the Wellcome Trust and Documentary Filmmakers Group, he would like to see more media training on “how science works and how scientists think”.

But this is a two-way street. Dr Logan also reckons that scientists need to buck up their ideas and engage more in communication with the media. He says: “A lot of scientists are too scared to communicate in case their reputation is compromised, which is a shame.” Perhaps understanding this, as a journalist, is one small teeny tiny step in the right direction for scientist-journalist relations.

NB Had to interrupt my science literacy series to blog this, part two coming v soon, promise.

Science literacy – getting more people into science, innit

Part one of a two-part blog on science literacy: Education

Let’s start with the basics. What is science literacy? Why is it important? What is the government doing about it? And so on.

And who better to speak to than the science tsar of the realm, Professor John Holman? He advises the government on the science curriculum and has been the director of the STEM (Science, Technology, Engineering and Mathematics) programme, a national drive to consolidate action for better science teaching and a more comprehensive curriculum, since October 2006.

The aim of STEM is to enthuse enough children in school with science to feed into universities and thence to dribble into R&D. This new generation of scientists will fuel our flagging economy, help the UK become the world-leader in science R&D and generally save the world.

Also, it’s about ensuring that each and every kid grows into an adult with a fully functional sense of ‘How Science Works’, as the GSCE topic is called. It was rolled out across the GSCE science curriculum in September 2006, and forms a large part of the single ‘core’ science GCSE, and part of the double (now called “additional science”) or triple science.

Why is ‘How Science Works’ important?

Prof Holman says: “It is about making sure that the population have sufficient scientific understanding to serve them well for the rest of the lives. So this is about not only understanding the basic laws of science, like what a gene is and what electromagnetism is, but is also about understanding the methods of science. It’s about how scientists do experiments; how they manage to draw conclusions, even if their results are uncertain; and things like peer review, sample size.”

This kind of understanding of scientific method can be valuable when putting science journalism into a sensible critical perspective, as science correspondent for the Guardian Alok Jha suggested in my last blog post, but also any source of information about science. And in a world threatened by climate change, tricky health choices and inundated with ever speedier and more sophisticated technologies, including biotech, that is NO BAD THING.

For example, if people can understand what science is about, they are probably more likely to make better-informed judgements about important things like health and cutting their carbon emissions.

Is the STEM programme delivering?

There have been some quibbles about this year’s set of core science questions by Ofqual, for having too many multiple choices, and that some of them were too easy to answer without having actual scientific knowledge.

In response, Professor Holman says: “Now, this wasn’t true about all of them, it was true about some of the questions on some of the papers. That’s an area where we’ve got to sharpen up, we’ve got to tighten up the standard of question setting for ‘How Science Works’.”

But it seems that it’s not only Ofqual taking issue with the science curriculum.

I spoke to a mother of a 14-year-old boy, just about to start his GSCEs. She said that science isn’t made relevant enough to kids these days and their lives, or exciting enough.

Ex-science teacher and author Clare Dudman agrees that science must be made more engaging by having a “more hands-on experience in small classes where pupils discover for themselves the fun of science”. One of the problems limiting this is health and safety measure that she says are “inappropriately restrictive”. Although Clare admits it’s a while since she’s been at “chalkface”, this is exactly the same argument made by the Science Minister Lord Drayson in yesterday’s Guardian, see here.

Meanwhile, award-winning science blogger Ed Yong is concerned that science education in the UK, which he declaims as “sorely deficient” despite the recent reforms to include science literacy, is suffering under “target-driven style of modern syllabuses and exams that encourage people to learn from checklists”. Yong would like to see children developing “reasoning skills, inquisitiveness or skeptical empiricism.”

Similar criticism could (and has been) be levelled at the whole state education system. Teachers have been accused of palm-feeding children to pass exams, head teachers accused of pressurising teachers to meet targets, and the government accused because they set the targets in the first place.

This is far from the ideal scenario for nurturing curiosity and sprouting understanding in children’s minds.

For example, podcast editor at Nature and TV presenter Adam Rutherford says that in biology syllabuses evolution is covered too late and not in enough depth, something he discovered researching a programme for Teachers TV. He explains:

“An old awesome Russian dude once stated, quite correctly, that “nothing makes sense in biology except for in the light of evolution”, so it’s daft not to approach biology with this in mind.”

On the uptake

Despite these various critiques of the science curriculum and its contents, there has recently been a significant upwards shift in the numbers of GCSE and A-level students picking to study science, pointing to the fact that the STEM drive seems to be working – at least to some extent.

For GCSE students, more of them are choosing to study science in depth, with a 20 per cent growth in those opting for all three sciences.

Professor Holman tells me that at A-level there has been a 10 per cent rise in those picking science subjects this year. He says: “The subject that has been most endangered was physics, now A level physics numbers were up 4.5 % this year… after many years, several decades of decline.” The 70s and 80s were the lowest point in the popularity the sciences, especially physics.

Why the sudden renaissance?

The STEM programme set out to improve science teaching. Regional, and one national, Science Learning Centres have been instrumental in training and re-training teachers. Good teachers make the subject more attractive.

Professor Holman says that there is sense of economic realism among students that subjects like physics, maths, chemistry, and biology make you employable than if you take “softer” [his quote marks] subjects.

“The evidence is very clear that if you chose maths of science subjects, you will have higher lifetime earnings that if you don’t,” he says.

Also, Professor Holman reckons that there is more of an air of optimism about science in this country than there was in decades past, that science is now perceived as more of a solution, rather than a cause, regarding the environment. For example, geoengineering.

“There are also some very interesting developments going on in technology, such as stem cell research, and this will add to the feeling that science is dynamic and has a lot of good to do to improve people’s lives,” he adds.

Extra points for STEM?

What about the Conservative’s proposal to weight sciences and maths with more points towards university admission? And, are they harder?

According to Professor Holman, it’s not the actual difficulty that is important (and it is hard to measure) but the perception of it.

He suggests a market solution to counter-act this instead of the points system, which he thinks would be very difficult to calibrate:

“I think a better way, another way, anyway, would be to say to universities, you must say very clearly which A-level subjects you prefer. Cambridge and LSE have already done this. They have two groups of subjects: these are the ones we like and these are the ones we’re not so interested in. And as you can imagine subjects like maths and physics and French and history are in the group they like.”

Getting kids sparked up to science

Rutherford points out a big contradiction between “the public’s adoration of natural history on TV presented by Britain’s most trusted man [David Attenborough, as if he needs any introduction], and relating it to scientific thinking. But this love for the natural world is also an opportunity to learn, if it is tapped into properly and from a young age. He suggests: “Simple things like counting species as measures of biodiversity, or basic taxonomy, or collecting fossils on beaches. Darwin himself used to do this with his brood, and the evolution chest sent out by the Wellcome Trust earlier this year included lots of those types of exercises.”

Lord Drayson, in the Guardian article in mentioned earlier, said that the solution for getting children more into science is for them to meet scientists and see what they do, such as at the Centre of the Cell, a new children’s science education centre.

Other ways for children to meet scientists are at festivals: music ones, such as Latitude, where science talks take place, or specialist sceincey ones, notably the British Science Festival, taking place every September (hurry, it ends tomorrow!)

There are talks from scientists for children and ‘Family Fun’ days exploring topics from biodiversity to AIDs at The Royal Institution.

Or, at roadshows, such as the Bang Goes the Theory series. There’s a few more dates left on the tour.

The government’s science literacy drive ‘Science, So what? So Everything’ had tours over the summer with interactive beach science, and this may be repeated next summer. 

Then, of course there are plenty of museums to visit: The Science Museum and the Natural History Museum are the obvious ones. There is a Science and Industry Museum in Swindon. But also check out the Wellcome Trust events and exhibitions, especially with older children, although one of their current exhibitions, Exquisite Bodies, is not recommended for the under-18s.

Please comment with any other suggestions for interactive stuff, other museums or exhibits or anything parents might find useful to enthuse their offspring with the joys of science. Cheers! 

In a land far, far away: the future of science journalism

So far I’ve discussed some of the problems facing science and health journalism. Here is one more that I’ve touched on but not gone into depth on, and it’s pretty crucial.

Alok Jha says: “Unfortunately science is basically the direct opposite of news, which is short, sharp, to the point, and science is incredibly the opposite of that, if you invert all those values. Now, which one is correct? Which one is wrong? Well, neither are. But to be honest, when you’re trying to shoehorn science into the news, it’s not the happiest of fits.”

According to Nick Davies, the particular problems faced by science journalism are that science is complicated and subtle, and the ‘news factory’ likes simple and clear stories. So, it struggles to get in the news, especially versus popular sports and celeb stories. Stories can get ‘distorted’ in the process of making them simple and sexing them up with an angle for the readers.

So science and news aren’t a match made in heaven. That’s a given. But, always liking to leave things on a positive note, I asked myself, and a whole host of very busy journalists, what possible solutions to the challenges faces by science journalism are?

Nick Davies pinged over: “In principle, it would help to have more specialist science writers, who would have a better grip on the underlying material and, therefore, be better placed to convert it into journalism. The trouble is that that costs money, which is precisely what is missing in newsrooms at the moment...” That’s why people such as myself are forking out a truckload of cash to get the training off our own backs.

Anyway, sounds pretty depressing. Not into feeling depressed, so here are some rays of light:

Critics, we like you

Criticism of science journalism is very healthy at the moment, which can maybe keep journalists and editors on their toes. For example, I do think for the most part coverage of swine flu has been pretty responsible. Of course there have been disagreements over that, but it’s just my general impression.

Blogging and tweeting

Jha says that some science blogs are brilliant, such as Ed Yong’s, who was poached to write for the Guardian’s science blog: “He’s just very measured, and his blog is great. He’s good at critical analyses. He complains now and then [about media coverage of science] but I don’t think he feels particularly superior to anyone.”

Jha says that one crucial difference between journalism and blogging is that bloggers tell the story they want to tell and journalists write with the audience in mind, and tell other people’s stories.  “Neither approach is necessarily better than the other,” he says. “They're just different. We're going to need both.” An example of the two can work in synthesis is how blogs informing journalism by putting their very specific and specialist knowledge out there, rather than just commenting on reporter’s groundwork – although criticism is sometimes vital, too, as I mentioned above.

As well as the internet, social media is finding a place in the dissemination of science journalism. Rowan Hooper, news editor of Newscientist.com, says that New Scientist uses Twitter (@newscientist) to alert over 28,000 followers of stories on the website, and also as a “news gathering-tool… being able to see, real-time, what’s going on in the world.”

How can you get followed by @newscientist? Probably by being a scientist. Hooper would like to see more scientists tweeting, and says: “It would get really interesting because if scientists can talk directly to people they don’t go through journalists or even journals in the way that they used to – and then the game really changes. You’re not going to get much data out there in 140 characters, but they can make assertions, for sure. We’d check those out like we would any statement from scientists. I think it’s all healthy and interesting and a good way of getting information out there”.

Streamlining

Pondering on our chat about the future of not just science journalism but journalism as a whole, Jha tells me that he thinks that the role of the journalist will be streamlined as a reaction to bloggers and citizen journalists. “We'll be a leaner industry, probably more specialised and perhaps more professional. When you have a million sources of information online, how d’you know which to trust? Perhaps journalists will be the trusted guides, weighing things up, checking sources, guiding us to the most valuable information.” 

I can see his point. With the proliferation of all kinds of information sources it is tricky to sift the nourishing stuff from the useless, baseless, beige stuff - this is why good journalism in any form will become even more important in the future. However, ideally, journalists should already be doing this job, in my opinion, but I can see how that doesn’t reflect reality. I wonder how this specialisation will come to pass while journalists continue to churn and all the other multitude of journalistic sins? And where will the money come from? 

Or perhaps it is just a case of the cream rises to the top of the bloodbath, with the best journalists naturally being the best journalists because they specialise - and they're the ones that have work and so have a voice. 

Non-profit agencies

In Flat Earth News, Nick Davies talks about a non-profit investigative news service in the US that is successful. I don’t remember its name right now but here is a list. But, surely this model could be replicated for other specialist, time-consuming and butt-kicking work such as a specialist science agency?

Specialist sources

We’ve been talking mainly about the science journalism in the mainstream media. Hooper reckons that: “The way newspapers work it's often not the science journalists, who set the tone of a story so much as the subeditors and news editors who take a story and put their own spin on it. It makes a strong case for relying on specialist outlets such as New Scientist!” Good plug. But I do love the New Scientist, so I’ll let it go.

Another option, less to do with journalism, but more a kind of raising of science consciousness among the masses, is this:

A science literate society

Journalism cannot be held solely to blame for people’s misinformation. Jha says that journalism’s purpose is not only to educate, but to be interesting and entertaining and relevant, so people should rely other sources. For example, their education, and state information such as the swine flu website (when it’s not busy crashing). The pressured state of our newsrooms also shifts some responsibility for learning critical analysis onto the audience. Good luck with that, everybody.

Jha says that on that fundamental basis he agrees with Goldacre: ‘He [Goldacre] rails on journalists because they are a soft target, I think, when really what he wants is a more science-literate society.’

*** Next blog on how we achieve a more science literate society!