50 times more cost effective
Department for Transport figures provided to the House of Commons Transport Select Committee led to massively erroneous conclusions about the relative cost effectiveness of speed cameras and vehicle activated signs.
While the Transport Committeee claimed that speed cameras were marginally more cost-effective than vehicle activated signs, the truth is that vehicle activated signs are around 50 times more cost effective than speed cameras.
Mr Idris Francis noticed the discrepancy and set about:
- Finding our where the errors had come
This page provides the story and the document archive for examining the unfolding story in detail.
15 March 06 Dr. Stephen Ladyman says in oral evidence to the Transport Select Committee
“One of the things we wanted to deal with was indeed public perception. Our rationale for changing the rules was that it was clear to us that, in certain areas, partnerships had formed which might be minded to look first for a road camera based solution rather than a better and perhaps more cost effective solution.“ (pg EV112 Q345. Committee Report)
18th October 06. The report (Pg EV156 Para 18, Q345) quoted the above and the question the Committee had put to Dr. Ladyman in response:
“Please provide details of the evaluation of different techniques for cutting speed-related road deaths and injuries. Please identify those techniques which have been proven to be more effective in reducing speed-related deaths and injuries than speed cameras. Please include details of cost-benefit analysis to identify which techniques provide the greatest value for money in reducing road deaths and injuries; and a comparison of results from speed cameras and other techniques.”
The report then set out Dr. Ladyman’s detailed reply to these questions. In summary, and in the context only of the comparison between cameras and vehicle activated signs, the reply prepared by the DfT and authorised by Dr. Ladyman:
(a) failed even to mention the existence of TRL548, “a large scale trial into vehicle activated signs” which the DfT had itself commissioned and published on 1st Jan 2003.
(b) claimed that,
“For the reasons set out above it is difficult to undertake a direct comparison of the effectiveness of speed reduction measures. Furthermore whilst a number of techniques have been individually evaluated over recent years, these assessments have not generally looked at value for money or cost benefit matters.
In order to do make any attempt at a
meaningful comparison it is necessary to compare like with like. It
This is of course utter twaddle, because (i) it is never possible to establish precise comparisons between what would have happened had cameras or signs not been introduced at particular sites or (ii) there was however ample evidence available to establish approximate comparisons of benefits of cameras and signs (iii) that direct comparisons of cost effectiveness had not previously been done in no way meant that they could not be done now that they the Committee had asked for them and (iv) when the approximate comparisons show, as they do, that signs are massively more cost effective than cameras, there is no need whatever for precise figures.
(Note at this point how very embarrassed the Dr. Ladyman and the DfT would have been I their reply to the Committee’s direct question had revealed, as it could and should have done, that they had been aware at least since January 2003 that signs are indeed massively more cost effective than cameras, but had continued the expansion of the camera scheme while largely ignoring signs.)
(c) having claimed that no comparison was possible, then provided one (pg. EV157 onwards) that
(1) was based on statistically meaningless low single figure accident reductions of 2.2 and 3.1 pa., despite having data from 1000s of camera sites and at least 61 sign sites.
(2) used the absolute reduction figures of 2.2 and 3.1 instead of the % reductions – 38% and 100%, thus failing to compensate for the higher accident levels at the camera site.
(3) quoted the cost of a camera as £7,500 pa, when the real average figure is £50,000 pa (see FoI confirmation below)
(4) quoted the cost of a sign as £14,000 pa when the real average figure is £1,000 pa (see FOI confirmation below.
(5) claimed a 12 to 10.6 cost effectiveness advantage for cameras over signs when the real comparison is at least 50 to 1 the other way (see below)
Quoting this bogus comparison, the Committee then stated in its Report (Pg 40 para 117)
“In terms of the value for money, however, the speed camera was shown to be the most cost-effective (the first year rate of return was 12 times the cost, compared to 0.8 and 10.6 respectively”
“A more cost effective measure for reducing speeds and casualties has yet to be introduced. An increase in safety camera coverage would be supported by evidence, as well as public opinion. There are many more sites which meet the existing camera guidelines and more funding should be made available to enable better coverage.”
8th November 06 Having been aware of TRL548 and the enormous cost effectiveness of signs over cameras, Idris Francis wrote to every member of the Committee (URL?), Dr, Ladyman and Robert Gifford of PACTS, the special adviser to the Committee pointing out that the reports’ findings were,
“grotesquely at odds with the findings of TRL548 which shows the cost effectiveness of signs to be massively greater than speed cameras”
and having set out in detail the extraordinary errors in Dr. Ladyman’s figures, concluded by saying that,
“I bring these facts to your attention in the hope that you will reconsider the report”
On the same day Mr. Francis sent a Freedom of Information request to the DfT seeking to find out where the data in Dr. Ladyman’s reply had come from and who had prepared the nonsensical figures.
4th December 06 The chairman of the Committee, the Hon Gwynneth Dunwoody replied (LINK) claiming that,
“the Committee does not believe that it has been misled by the Government over the relative cost effectiveness of speed cameras and flashing signs.”
and effectively told Mr. Francis not to bother to write again.
5th December 06 A reply was received from the DfT to the Freedom of Information request confirming that,
“The supplementary memorandum was drafted by officials and cleared by Stephen Ladyman, Minister of State for Transport. This element of the memorandum was drafted by Department for Transport officials responsible for speed management”
“Whilst a number of techniques have been individually evaluated over recent years, including TRL report 548, Vehicle Activated Signs – a large scale evaluation, those evaluations do not make comparisons with other speed management measures. Neither do they generally consider value for money or cost benefits. In order for an attempt to be made at comparing the techniques, officials used Appendix A of the Department's A Road Safety Good Practice Guide as the source document.”
We see again hear the ludicrous proposition
that because no prior attempt had been made to compare cost effectiveness
on the basis of large amounts of data that had long been available, it
could not be done and that the comparison should instead be made of the
statistically meaningless data for only one site!
5th December 07 Not satisfied with the level of detail provided in the above DfT reply Mr. Francis wrote again (LINK), pointing out the scale of discrepancies and seeking more information.
17th January 07 After a 6 week delay and checking by their lawyers, the DfT then replied again, making Appendix A available for the first time on their web site, but stating that,
“The Department does not accept that misleading information was contained in the memorandum” (to the Committee) “and does not therefore intend to take any action of the sort you suggest”
24 January 07 Having finally been able to access Appendix A and find the data quoted by Dr. Ladyman’s submission to the Committee Mr. Francis then sent Freedom of Information requests (27th January 07) to Oxfordshire County Council in respect of the speed camera site and Norfolk County Council in respect of the flashing sign site.
26 January 07 Norfolk County Council replied (LINK) stating that:
“The Felthorpe signs were more straightforward with no weather/temp detection and the cost of £14,000 was for the pair of signs. Again this figure excludes provision of an electricity supply. Running costs of the signs are low; this is particularly true for modern VAS as LEDs have very long life (100,000hrs) and very low power consumption.”
This reply confirms that the £14,000 quoted by Dr.Ladyman was for two signs, not one.
29 January 07 In response to a further query (LINK) Norfoll County Council confirmed that,
“The average budget figure that I currently use for a complete VAS installation at a selected site is £6,000. This comprises £4000 for the sign, £1500 for pole & elec supply and £500 for staff time (we now have staff with a lot of expertise in the siting………..If a commuted sum to cover 10 yrs maintenance and electricity were to be added I would suggest a figure of £2000. Some of the newer smaller VAS mounted on an existing lighting column can cost as little as £3000 total”.
This reply confirms that the amortised cost of running a sign over 10 years is rather less than £1,000 pa, not the wholly ludicrous £14,000 implied by Dr. Ladyman.
21 February 07 Mr. Francis emailed the Parliamentary Ombudsman to ask whether he has jurisdiction on this improper conduct by MPs and others (reply received 2nd April, see below)
24 February Oxfordshire County Council replied (LINK) regarding the speed camera site, stating that,
“The cost of £ 7,500 indicated in the Department of Transport’s Good Practise Guide for Road Safety was for the camera housing plus setting up a power supply. During the 1990s the housings for the cameras cost approximately £ 6,000……. In 1998 the cameras themselves cost approximately £ 32,000”
This reply alone confirms that Dr. Ladyman’s figure of £7,500, and therefore the overall cost effectiveness comparison was wrong by a factor of at least 6 and probably by a factor of 10 allowing for other costs such as police time, fine administration etc. The council has since confirmed that the error occurred at the DfT, not in the figures Oxfordshire provided to the DfT.
The real cost of a flashing sign, sensibly amortised over 10 years being less than £1,000 pa and the real cost of speed cameras being more than £50,000 pa (LINK), Fourth annual report into speed camera partnerships, it is clear that Dr. Ladyman’s comparison, £7,500 for a camera but £14,000 for signs is wrong by a factor APPROACHING 100 TIMES.
19th March 07 Having accumulated this conclusive proof that Dr. Ladyman’s comparison, repeated by the Committee in their report, was wildly wrong Mr. Francis wrote again to the Committee, copying that evidence.
20th March 07 The Committee Chairman send a single paragraph reply, (LINK) making no comment whatever, saying only that a copy had been sent to Clerk to the Committee for their records.
25th March 07 Mr. Francis sent a formal complaint (LINK) to New Scotland Yard of misfeasance and/or malfeasance in public office against the DfT, Dr, Ladyman, every member of the Committee and Robert Gifford, their special adviser.
02 April 07 Parliamentary Ombudsman replies (LINK) stating that Mr. Francis’ complaint is “undergoing initial investigation” on whether it is within his jurisdiction.
03 April 07 Mr. Francis sent a letter (LINK) to the Committee advising them that in view of their collective failure to act, formal complaints had been laid against them.
At no time has Dr. Ladyman responded at all to my letters copied to him.
More detail if needed!
15 March 06 Ladyman gave oral evidence, and referred to "possibly lower cost alternatives" to speed cameras.
Unspecified date when the Committee asked Q345, for Ladyman to provide costs effectiveness comparisons.
Unspecified date when Ladyman submitted the DfT figures in reply
18th Oct 06 Committee report published, claiming that speed cameras are the most cost effective method, recommending more and included pgs EV157 onwards, the misleading data.
8th November - Idris Francis first letter to the Committee, Ladyman and Gifford pointing out gross errors
7th November - 1st FoI request to DfT
4th December - after several reminders - Dunwoody replies that the Committee has not been misled and in effect not to bother them again.
5th Dec first FoI reply from DfT
5th Dec second FoI request to DfT, followed shortly by the 3rd, extra one.
17th Jan 07 second reply from DfT, delayed while lawyers checked it, I was told by Magee!
21 Jan 07 first FoI to Norfolk
24 Jan 07 FoI to Oxfordshire, as DfT had identified them (you do not have a copy because it is effectively contained in their reply, below)
25 Jan 07 approx second FoI to Norfolk (you do not have a copy because it is effectively contained in their reply, below)
29 Jan 07 second foI reply (same pdf file as first)
26 Jan first FoI reply from Norfolk confirming £6,000, and that £14,000 was for two signs
21 Feb FoI reply from Oxfordshire confirming £7,500 for housing and power, £32,000 for camera.
21st Feb asked Ombudsman of they can intervene.
early March - drafted and agreed 3 page supplementary guidance for SCPs, finally circulated 26 March
19th March 2007, my second letter to the Committee pointing out irrefutable evidence that the data was misleading
20th March 2007 Dunwoody reply saying nothing except my letter filed.
23 March Gifford replies, evading every issue
26th March 2007 New Scotland Yard received by post my formal complaint of misconduct etc against the DfT, Ladyman, Dunwoody and the rest of the Cttte and Gifford. (not yet acknowledged)
26th March 2007 - circulated all SCPs, police forces and Magistrates Association advising of problem.
2nd April 2007 Parlimentary Ombudsman replies, complaint being assessed for admissibility
2nd April 2nd letter to Gifford calling for resignation
Document A - letter to Transport Committee
dated 8th November 2006 (word document)
Safe Speed PR outlining the situation (click)
|Relative Cost Effectiveness
of Speed Cameras and Vehicle Activated Signs (Document J)
In January 2003 the Department for Transport (DfT) published TRL548, “a large-scale evaluation of Vehicle-activated signs” (VAS) by the Transport Research Laboratory (TRL). The accident reduction figures provided in the report were at least comparable to and arguably better than those routinely claimed for speed cameras while installation and operating costs were orders of magnitude lower than those of cameras, with each VAS costing about £5,000 compared to well established figures of £30,000 or more for each camera.
Despite the remarkable improvement in cost-effectiveness shown in TRL548 (which the DfT had itself commissioned) it’s findings appears been largely ignored by the DfT, while the speed camera programme continued to expand and operating costs rising to £96m pa in 2005, according to the 4th Annual Review.
On January 31st 2007 the DfT issued their “Guidance for Camera Partnerships under the new rules from 1st April 2007”. This recommended inter alia that all those involved in deciding what methods and equipment to use under the new rules should read TRL548 - yet signally fail to point out the massive cost effectiveness advantage of VAS.
Stranger still, in response to a direct question from the Commons Select Committee on Road Transport in the summer of 2006 on relative cost effectiveness of alternatives to speed cameras, the DfT failed to mention TRL548 at all! Having first claimed that no such comparison was possible, they then submitted one – ludicrously, based on low single figure – and therefore statistically meaningless - accident numbers from just one camera site and one VAS site. Using those numbers and costs figures that would have embarrassed Enron in it’s prime, the DfT purported to show a 12% cost effectiveness advantage for cameras – a comparison that was wrong at least by a factor of 50! The Committee’s acceptance of that assessment, and on which it relied at least to some extent for it’s subsequent call for more speed cameras, was no less astonishing, as indeed was it’s dismissal of vas on the basis of a mere 12% difference.
The remainder of this document sets out how that data submitted by the DfT to the Committee, stage by stage, skewed what should have been a massive advantage for VAS into a marginal one in favour of speed cameras. Note that what follows sets out only how these figures and calculations were incorrect and/or misleading – we make no attempt to explain why they were or why the these grotesque errors were neither spotted nor, even worse, acknowledged by the DfT or Select Committee after they had been pointed out. In the few places where we set out how such calculations should have been done, we make clear that these are our opinions, albeit opinions that are surely irrefutable.
The Details, Step by Step
Please Refer to House of Commons Transport
Committee report “Roads Policing and Technology: Getting
In the context of the change to Camera Partnership rules from 1st April 2007 the Select Committee asked the Home Office and DfT to provide information on alternatives to speed cameras, and their relative cost effectiveness. The information provided by Dr. Stephen Ladyman for the DfT is set out on Page EV156 of the Select Committee report, par. 18 onwards. Only the figures for cameras and VAS are relevant to this analysis.
(A) The first significant point is that the DfT failed to mention to the Committee even the existence of TRL548 – despite having published it in January 2003 and it having been “a large scale evaluation” of precisely the sort of alternative to cameras that the Committee had asked about.
(B) Secondly, the DfT claimed that “For the reasons set out above it is difficult to undertake a direct comparison of the effectiveness of speed reduction measures. Furthermore whilst a number of techniques have been individually evaluated over recent years, these assessments have not generally looked at value for money or cost benefit matters. In order to do make any attempt at a meaningful comparison it is necessary to compare like with like. It is not therefore possible to compare the effectiveness of a national programme, such as the national safety camera programme, with a 20 mph zone or vehicle activated sign.”
In our view this is nonsense for several reasons:
This overall comparison claimed a 12 to
10.6 advantage for cameras, was, astonishingly, accepted by the Select
Committee despite the prima facie nonsense of the cost figure for a VAS
being almost double that of a camera. We summarise below how both the data
and the methods used to calculate this ratio were seriously flawed and
therefore utterly misleading,
“2 signs at Felthorpe in February 1998. Both were early prototypes of vehicle activated signs (VAS) ……..(and) used fibre-optic technology whereas VAS now are manufactured (in significant numbers) using LEDs which are both cheaper and more reliable. The cost of these 2 schemes is not therefore a good guide for any similar scheme today……If a commuted sum to cover 10yrs maintenance and electricity were to be added I would suggest a figure of £2000. Some of the newer smaller VAS mounted on an existing lighting column can cost as little as £3000 total”.
“The cost of £ 7,500 indicated
in the Department of Transport’s Good Practise Guide for Road Safety was
for the camera housing plus setting up a power supply. During the 1990s
the housings for the cameras cost approximately £ 6,000……….. In 1998
the cameras themselves cost approximately £ 32,000.”
The DfT, having been asked by the Select Committee to provide comparisons in cost effectiveness of alternatives to speed cameras:
(a) failed to disclose the existence of TRL548.
It is also, incidentally, rather surprising that neither the Committee in their 264 page report nor Dr. Ladyman in the various DfT submissions ever mentioned the possibility that within months a verdict in favour of the applicants to the European Court of Human Rights over the right to silence in O’Halloran and Francis v UK (Application Nos: 15809/02 and 25624/02) may render S172 1988 RTA and the entire speed camera system inoperable, given that vas provide a complete solution to the problems the authorities would then face.
While it is necessary in this context to compare on a like-for-like basis the reductions in speed, accidents and casualties at camera and sign sites, as reported by the DfT, TRL and Camera Partnerships, nothing in this note should be read as necessarily agreeing that all of those reductions are in reality due to the presence of those cameras or signs, rather than to regression to the mean, pre-existing downward trend or other factors covered in Appendix H of the 4th annual report on Camera Partnerships However it is clear that the greater the extent to which the notional benefits of cameras or signs should be reduced – sensibly by similar proportions - to allow for such factors, the more important it becomes that the costs involved be minimised.
Idris Francis email@example.com (ECHR
False assumptions won't save lives
© SafeSpeed 2007
Created 10/04/2007. Last update 15/06/2007