BBC Radio 4 programme "More or Less" on 29th January 2004 discussed the vital "regression to the mean" issue with regard to speed cameras.
In particular they referred to the DfT "report of the two year pilot" and asks if the headline claim of 35% fewer killed or seriously injured at camera sites" was justified in the light of a possible regression to the mean benefit illusion.
Many thanks to Maureen Patterson for preparing the transcript.
Presenter: Andrew Dilnot
Presenter: If you were in charge of road safety in your county and you had the money to pay for just one speed camera where would you put it? Surely the answer's obvious - in the spot with the most speed related accidents? And if the number of accidents then fell dramatically wouldn't that be proof that the speed camera had improved safety? Well some people don't think so.
Safe Speed: My name's Paul Smith. I'm the founder of the Safe Speed campaign. If you come and put up a speed camera where accidents have been at a high level and accidents reduce you can't actually tell whether the speed camera was the cause of the reduction or whether, because accidents at that location were at a high level, they would have reduced naturally anyway. It's been said that you might get the same effect by placing a garden gnome at the site.
Presenter: Is that true? We decided to find out. The phenomenon he describes is known by statisticians as regression to the mean. That basically means that if the occurrence of something like an accident rate is abnormally high, or abnormally low, simply because of random variation, the next time you measure it the statistical chances are that it will be much closer to the average. Now is this something that the people in charge of safety on our roads really understand?
I've come to meet Geoff Barrell, Oxfordshire County Council's Chief Road Safety Engineer. He's in charge of identifying the county's accident black spots and then implementing engineering measures, things like road humps, signs and road markings, that might help to cut the accident rate. He's brought me to a former black spot at a bypass near the village of Dorchester 5 miles outside Oxford where a change to the road layout looks as if it's produced results.
Interviewer: So Geoff, we're standing now at one end of Dorchester Bypass. Since you've done this have you been confident that you've seen a reduction in the number of accidents? How sure can you be that it's helped?
Barrell: We wouldn't give it a formal clean bill of health yet because we want to monitor for at least 5 years after. Five years before and 5 years after, but we have monitored for 18 months afterwards and in that period the accidents have reduced by 75% and the target accidents, i.e. the overtaking ones, they've gone down by 45%. So far it looks very good.
Interviewer: And why do you want to wait for as long as 5 years? A member of the public might say “Come on Geoff, there were 13 accidents in the previous 5 years; so far in this 18 month period you've only had one. You must be able to say that it's worked?”
Barrell: Because of the random nature of accidents there are so many factors involved that come together to make an accident including bad luck and driver error and things like that, so we find that 5 years gives a reasonable period to get an overall, average view.
Interviewer: And do you have a clear view of how big the change in numbers has to be even over a 5 year period for you to be sure this isn't just the normal pattern of any numerical behaviour?
Barrell: We are very aware of that regression to mean possibility yes, so we have to look at long term trends as the only way to get a true picture and if we are investing a lot of money we want to make sure it is invested effectively and a long-term picture is the best way to do that. But of course people want results quickly and that's what we can't give.
Presenter: So far as these engineering measures like getting rid of an overtaking lane at this bypass are concerned, it seems as if the people in charge of the roads are taking into account the possibility that a fall in the number of accidents at a so-called black spot might just be down to chance. But what about that more controversial accident prevention measure that we mentioned to start with – speed cameras? There's a bit more suspicion surrounding speed cameras because the fines they generate are seen as an easy source of revenue for the authorities. And we all know they sometimes produce some pretty odd driving behaviour. But that aside, do they cut accidents? Well, according to a report published by the Department for Transport, they do, by an average of 36%. But unlike Oxfordshire County Council's engineers the authors of the Department for Transport's study didn't look at the accident record over a 5 year period either side of the cameras being put in but for just 3 years before and 2 years afterwards. And did the report take into account this possibility of regression to the mean - the idea that you'd expect the number of accidents to fall in a lot of these places with an abnormally high number of crashes? I asked Professor John Adams, an expert in transport and risk management, from the Geography Department at University College London for his assessment.
Adams: I have had a quick look at this Department for Transport study and it does appear to me to indeed run the risk of claiming too much credit for speed cameras because they explicitly choose the areas for installing their test cameras on sections of road that have had a higher number than expected of accidents in the previous 3 years and that's exactly the sort of case [laughs] where you would look for the regression to mean effect.
Interviewer: Have there been studies in the past that have tried to estimate how large this regression to mean effect might be?
Adams: There was a famous study done in Sweden of over 2,500 unsignalled junctions and they discovered that the accident black spots without any treatment at all in the ‘after’ period had decreased by over 50% Now that's the sort of order of magnitude of increased safety that is often claimed for black spot treatments.
Interviewer: And in fact in the Department for Transport study the order of magnitude is about 35% so if the regression to mean effect is random and statistical effect might account for a 50% change then in fact one could end up thinking that speed cameras have no effect at all.
Adams: Well [laughs], I would like to read in more detail exactly what they did but that order of magnitude falls well within what has been demonstrated to occur by chance.
Presenter: Just a few hundred yards and a couple of traffic humps away from John Adams's office is the Centre for Transport Studies, also at University College London, where one of the authors of the Department for Transport study on speed cameras works – Professor Benjamin Heydecker. I went to ask him whether the dramatic reduction he's attributed to speed cameras could, at least in part, have been due to chance…
Heydecker: Certainly if the number of accidents were the only criterion for selection, we would expect there to be an appreciable effect as you describe.
Interviewer: But if it's any part of the criteria for selection, then there will be some effect.
Heydecker: Perhaps. But it had to be clear for the sites to be treated with a safety camera that speed was a contributory factor to the accidents that had occurred there. Sites where there were many accidents but there was no causal relationship between speed and the accidents that occurred were not eligible for entry into the programme. And secondly, if there were a statistical phenomenon as you describe then we'd expect - in that case – we'd expect to see accident numbers falling at the sites that were treated and increasing for exactly the same reason elsewhere in the participating regions. Whereas we saw casualties decrease elsewhere. So for those reasons we are confident in our results.
Interviewer: So it sounds as if you were clearly aware of this problem, but in the numbers that you produced there's no adjustment for it is there? So the 36% reduction in accidents – isn't it rather likely that some of that would have occurred simply as a result of random behaviour of accidents?
Heydecker: Well, that is certainly possible, but we did not attempt to make any correction for that at the sites – there is a published correction that is available, but in fact it is known to reduce the accuracy of the estimate because the correction is less accurate than the original figure.
Presenter: Perhaps the answer would have been to compare the fall in accidents at the sites with speed cameras with the accident record of similar sites which didn't have speed cameras. That would have involved doing a much larger and more expensive study though. So is there any good reason for speed cameras? Round the corner to John Adams again.
Interviewer: So John, have you yet seen any evidence that persuades you that speed cameras really do reduce accidents?
Adams: I've yet to see any clear evidence that persuades me that speed cameras can claim credit for any significant reduction in accidents but on the other hand I'm in favour of speed cameras because they slow traffic down and I think that's a highly desirable thing to do, especially in residential areas which are suffering from high speed traffic: children aren't allowed to cross the road; old people are afraid to cross the road; pedestrian adults cross it quickly and carefully which cuts the neighbourhood in half.
Presenter: I'm fairly sure that speed cameras in the right places do help to reduce accidents but the Department for Transport study really should have taken the regression to the mean problem seriously as other academic studies have done and still found reductions in accidents. It would have reduced the size of the effects they found so the study might have attracted less attention but at least they couldn't have been accused of bias.
new: Another letter to Heydecker, following the third year report. (click here)
At least it has been confirmed that regression to the mean errors are usually present, and that no compensation has been applied in the report.
Calling for real road safety, based on truth
Created 29/01/2004. Last update 21/06/2004