Department for Transport figures provided to the House of Commons Transport Select Committee led to massively erroneous conclusions about the relative cost effectiveness of speed cameras and vehicle activated signs.

While the Transport Committeee claimed that speed cameras were marginally more cost-effective than vehicle activated signs, the truth is that vehicle activated signs are around 50 times more cost effective than speed cameras.

Mr Idris Francis noticed the discrepancy and set about:

- Finding our where the errors had come from
- Ensuring that the Transport Committee was aware of the problem
- Discovering that the errors were all originated by Department for Transport
- Discovering that the Transport Committee were, at best, unconcerned.
- Finally issuing formal complaints because the errors will inevitably cause scarce life saving resources to be misdirected with consequent loss of life.

This page provides the story and the document archive for examining the unfolding story in detail.

15 March 06 Dr. Stephen Ladyman says in oral evidence to the Transport Select Committee

“One of the things we wanted to deal with was indeed public perception. Our rationale for changing the rules was that it was clear to us that, in certain areas, partnerships had formed which might be minded to look first for a road camera based solution rather than a better and perhaps more cost effective solution.“ (pg EV112 Q345. Committee Report)

18th October 06. The report (Pg EV156 Para 18, Q345) quoted the above and the question the Committee had put to Dr. Ladyman in response: 

“Please provide details of the evaluation of different techniques for cutting speed-related road deaths and injuries. Please identify those techniques which have been proven to be more effective in reducing speed-related deaths and injuries than speed cameras. Please include details of cost-benefit analysis to identify which techniques provide the greatest value for money in reducing road deaths and injuries; and a comparison of results from speed cameras and other techniques.”

The report then set out Dr. Ladyman’s detailed reply to these questions. In summary, and in the context only of the comparison between cameras and vehicle activated signs, the reply prepared by the DfT and authorised by Dr. Ladyman:

(a) failed even to mention the existence of TRL548, “a large scale trial into vehicle activated signs” which the DfT had itself commissioned and published on 1st Jan 2003.

(b) claimed that, 

“For the reasons set out above it is difficult to undertake a direct comparison of the effectiveness of speed reduction measures. Furthermore whilst a number of techniques have been individually evaluated over recent years, these assessments have not generally looked at value for money or cost benefit matters.

In order to do make any attempt at a meaningful comparison it is necessary to compare like with like. It
is not therefore possible to compare the effectiveness of a national programme, such as the national safety camera programme, with a 20 mph zone or vehicle activated sign.”

This is of course utter twaddle, because (i) it is never possible to establish precise comparisons between what would have happened had cameras or signs not been introduced at particular sites or (ii) there was however ample evidence available to establish approximate comparisons of benefits of cameras and signs (iii) that direct comparisons of cost effectiveness had not previously been done in no way meant that they could not be done now that they the Committee had asked for them and (iv) when the approximate comparisons show, as they do, that signs are massively more cost effective than cameras, there is no need whatever for precise figures.

(Note at this point how very embarrassed the Dr. Ladyman and the DfT would have been I their reply to the Committee’s direct question had revealed, as it could and should have done, that they had been aware at least since January 2003 that signs are indeed massively more cost effective than cameras, but had continued the expansion of the camera scheme while largely ignoring signs.)

(c) having claimed that no comparison was possible, then provided one (pg. EV157 onwards) that

(1) was based on statistically meaningless low single figure accident reductions of 2.2 and 3.1 pa., despite having data from 1000s of camera sites and at least 61 sign sites.

(2) used the absolute reduction figures of 2.2 and 3.1 instead of the % reductions – 38% and 100%, thus failing to compensate for the higher accident levels at the camera site.

(3) quoted the cost of a camera as £7,500 pa, when the real average figure is £50,000 pa (see FoI confirmation below)

(4) quoted the cost of a sign as £14,000 pa when the real average figure is £1,000 pa (see FOI confirmation below.

(5) claimed a 12 to 10.6 cost effectiveness advantage for cameras over signs when the real comparison is at least 50 to 1 the other way (see below)

Quoting this bogus comparison, the Committee then stated in its Report (Pg 40 para 117)

“In terms of the value for money, however, the speed camera was shown to be the most cost-effective (the first year rate of return was 12 times the cost, compared to 0.8 and 10.6 respectively”

and

“A more cost effective measure for reducing speeds and casualties has yet to be introduced. An increase in safety camera coverage would be supported by evidence, as well as public opinion. There are many more sites which meet the existing camera guidelines and more funding should be made available to enable better coverage.”

8th November 06 Having been aware of TRL548 and the enormous cost effectiveness of signs over cameras, Idris Francis wrote to every member of the Committee (URL?), Dr, Ladyman and Robert Gifford of PACTS, the special adviser to the Committee pointing out that the reports’ findings were,

“grotesquely at odds with the findings of TRL548 which shows the cost effectiveness of signs to be massively greater than speed cameras”

and having set out in detail the extraordinary errors in Dr. Ladyman’s figures, concluded by saying that,

“I bring these facts to your attention in the hope that you will reconsider the report”

On the same day Mr. Francis sent a Freedom of Information request to the DfT seeking to find out where the data in Dr. Ladyman’s reply had come from and who had prepared the nonsensical figures.

4th December 06 The chairman of the Committee, the Hon Gwynneth Dunwoody replied (LINK) claiming that,

“the Committee does not believe that it has been misled by the Government over the relative cost effectiveness of speed cameras and flashing signs.”

and effectively told Mr. Francis not to bother to write again. 

5th December 06  A reply was received from the DfT to the Freedom of Information request confirming that,

“The supplementary memorandum was drafted by officials and cleared by Stephen Ladyman, Minister of State for Transport. This element of the memorandum was drafted by Department for Transport officials responsible for speed management”

and

“Whilst a number of techniques have been individually evaluated over recent years, including TRL report 548, Vehicle Activated Signs – a large scale evaluation, those evaluations do not make comparisons with other speed management measures. Neither do they generally consider value for money or cost benefits. In order for an attempt to be made at comparing the techniques, officials used Appendix A of the Department's A Road Safety Good Practice Guide as the source document.”

We see again hear the ludicrous proposition that because no prior attempt had been made to compare cost effectiveness on the basis of large amounts of data that had long been available, it could not be done and that the comparison should instead be made of the statistically meaningless data for only one site!
 

5th December 07   Not satisfied with the level of detail provided in the above DfT reply Mr. Francis wrote again (LINK), pointing out the scale of discrepancies and seeking more information. 

17th January 07    After a 6 week delay and checking by their lawyers, the DfT then replied again, making Appendix A available for the first time on their web site, but stating that,

“The Department does not accept that misleading information was contained in the memorandum” (to the Committee) “and does not therefore intend to take any action of the sort you suggest”

24 January 07 Having finally been able to access Appendix A and find the data quoted by Dr. Ladyman’s submission to the Committee Mr. Francis then sent Freedom of Information requests (27th January 07) to Oxfordshire County Council in respect of the speed camera site and Norfolk County Council in respect of the flashing sign site.

26 January 07  Norfolk County Council replied (LINK) stating that:

“The Felthorpe signs were more straightforward with no weather/temp detection and the cost of £14,000 was for the pair of signs. Again this figure excludes provision of an electricity supply. Running costs of the signs are low; this is particularly true for modern VAS as LEDs have very long life (100,000hrs) and very low power consumption.”

This reply confirms that the £14,000 quoted by Dr.Ladyman was for two signs, not one.

29 January 07  In response to a further query (LINK) Norfoll County Council confirmed that,

“The average budget figure that I currently use for a complete VAS installation at a selected site is £6,000. This comprises £4000 for the sign, £1500 for pole & elec supply and £500 for staff time (we now have staff with a lot of expertise in the siting………..If a commuted sum to cover 10 yrs maintenance and electricity were to be added I would suggest a figure of £2000. Some of the newer smaller VAS mounted on an existing lighting column can cost as little as £3000 total”.

This reply confirms that the amortised cost of running a sign over 10 years is rather less than £1,000 pa, not the wholly ludicrous £14,000 implied by Dr. Ladyman.

21 February 07  Mr. Francis emailed the Parliamentary Ombudsman to ask whether he has jurisdiction on this improper conduct by MPs and others (reply received 2nd April, see below)

24 February   Oxfordshire County Council replied (LINK) regarding the speed camera site, stating that,

“The cost of £ 7,500 indicated in the Department of Transport’s Good Practise Guide for Road Safety was for the camera housing plus setting up a power supply. During the 1990s the housings for the cameras cost approximately £ 6,000……. In 1998 the cameras themselves cost approximately £ 32,000”

This reply alone confirms that Dr. Ladyman’s figure of £7,500, and therefore the overall cost effectiveness comparison was wrong by a factor of at least 6 and probably by a factor of 10 allowing for other costs such as police time, fine administration etc. The council has since confirmed that the error occurred at the DfT, not in the figures Oxfordshire provided to the DfT.

The real cost of a flashing sign, sensibly amortised over 10 years being less than £1,000 pa and the real cost of speed cameras being more than £50,000 pa (LINK), Fourth annual report into speed camera partnerships, it is clear that Dr. Ladyman’s comparison, £7,500 for a camera but  £14,000 for signs is wrong by a factor APPROACHING 100 TIMES. 

19th March 07   Having accumulated this conclusive proof that Dr. Ladyman’s comparison, repeated by the Committee in their report, was wildly wrong Mr. Francis wrote again to the Committee, copying that evidence.

20th March 07  The Committee Chairman send a single paragraph reply, (LINK) making no comment whatever, saying only that a copy had been sent to Clerk to the Committee for their records. 

25th March 07  Mr. Francis sent a formal complaint (LINK) to New Scotland Yard of misfeasance and/or malfeasance in public office against the DfT, Dr, Ladyman, every member of the Committee and Robert Gifford, their special adviser. 

02 April 07   Parliamentary Ombudsman replies (LINK) stating that Mr. Francis’ complaint is “undergoing initial investigation” on whether it is within his jurisdiction.

03 April 07  Mr. Francis sent a letter (LINK) to the Committee advising them that in view of their collective failure to act, formal complaints had been laid against them.

At no time has Dr. Ladyman responded at all to my letters copied to him.

More detail if needed!

Idris Francis

15 March 06  Ladyman gave oral evidence, and referred to "possibly lower cost alternatives" to speed cameras.

Unspecified date when the Committee asked Q345, for Ladyman to provide costs effectiveness comparisons.

Unspecified date when Ladyman submitted the DfT figures in reply

18th Oct 06   Committee report published, claiming that speed cameras are the most cost effective method, recommending more and included pgs EV157 onwards, the misleading data.

8th November - Idris Francis first letter to the Committee, Ladyman and Gifford pointing out gross errors

7th November - 1st FoI request to DfT

4th December - after several reminders - Dunwoody replies that the Committee has not been misled and in effect not to bother them again.

5th Dec  first FoI reply from DfT

5th Dec  second FoI request to DfT, followed shortly by the 3rd, extra one.

17th Jan 07 second reply from DfT, delayed while lawyers checked it, I was told by Magee!

21  Jan 07 first FoI to Norfolk

24 Jan 07 FoI to Oxfordshire, as DfT had identified them (you do not have a copy because it is effectively contained in their reply, below)

25 Jan 07 approx second FoI to Norfolk (you do not have a copy because it is effectively contained in their reply, below)

29 Jan 07 second foI reply (same pdf file as first)

26 Jan first FoI reply from Norfolk confirming £6,000, and that £14,000 was for two signs

21 Feb FoI reply from Oxfordshire confirming £7,500 for housing and power, £32,000 for camera.

21st Feb  asked Ombudsman of they can intervene.

early March - drafted and agreed 3 page supplementary guidance for SCPs, finally circulated 26 March

19th March 2007, my second letter to the Committee pointing out irrefutable evidence that the data was misleading

20th March 2007  Dunwoody reply saying nothing except my letter filed.

23 March  Gifford replies, evading every issue

26th March 2007 New Scotland Yard received by post my formal complaint of misconduct etc against the DfT, Ladyman, Dunwoody and the rest of the Cttte and Gifford. (not yet acknowledged)

26th March 2007 - circulated all SCPs, police forces and Magistrates Association advising of problem.

2nd April 2007 Parlimentary Ombudsman replies, complaint being assessed for admissibility

2nd April   2nd letter to Gifford calling for resignation

Document A - letter to Transport Committee dated 8th November 2006 (word document)
Document B - Appendix to the letter above. (word document)
Document C - Reply from Transport Committee dated 4th December 2006 (PDF document)
Document D - Letter to Transport Committee dated 22nd February 2007 (word document)
Document E - FoI request to DfT (email as word document)
Document F - FoI reply from DfT (word document)
Document G - Further FoI requests to DfT (email as word document)
Document H - FoI reply from DfT (PDF document)
Document J - Multiple signatory statement about serious errors. (word document) and below:
Document K - Statement for enforcement authorities. (word document)
Document L - Oxfordshire respond to FoI on camera cost (email as word document)
Document M - Cost effectiveness comparison (pdf document)
Document N - 3rd April 2007 Letter to Transport Committee. (pdf document)
new Document O - Memo from Minister Ladyman to Transport Committee (pdf document)
new Document P - Memo from Transport Committee to Minister Ladyman (pdf document)
new Document P1 - Letter from Mr Francis to Minister Ladyman (word document) 
new Document P2 - Letter from Mr Francis to Transport Committee (word document) 
new Document Q - Letter from Ms Dunwoody to Mr Francis, 5th June 2007 (pdf document)
new Document R - Letter from Mr Francis to Ms Dunwoody, 6th June 2007 (word document)

Safe Speed PR outlining the situation (click)

Introduction

In January 2003 the Department for Transport (DfT) published TRL548, “a large-scale evaluation of Vehicle-activated signs” (VAS) by the Transport Research Laboratory (TRL). The accident reduction figures provided in the report were at least comparable to and arguably better than those routinely claimed for speed cameras while installation and operating costs were orders of magnitude lower than those of cameras, with each VAS costing about £5,000 compared to well established figures of £30,000 or more for each camera.

Despite the remarkable improvement in cost-effectiveness shown in TRL548 (which the DfT had itself commissioned) it’s findings appears been largely ignored by the DfT, while the speed camera programme continued to expand and operating costs rising to £96m pa in 2005, according to the 4th Annual Review.

On January 31st 2007 the DfT issued their “Guidance for Camera Partnerships under the new rules from 1st April 2007”. This  recommended inter alia that all those involved in deciding what methods and equipment to use under the new rules should read TRL548 - yet signally fail to point out the massive cost effectiveness advantage of VAS.

Stranger still, in response to a direct question from the Commons Select Committee on Road Transport in the summer of 2006 on relative cost effectiveness of alternatives to speed cameras, the DfT failed to mention TRL548 at all! Having first claimed that no such comparison was possible, they then submitted one – ludicrously, based on low single figure – and therefore statistically meaningless - accident numbers from just one camera site and one VAS site. Using those numbers and costs figures that would have embarrassed Enron in it’s prime, the DfT purported to show a 12% cost effectiveness advantage for cameras – a comparison that was wrong at least by a factor of 50! The Committee’s acceptance of that assessment, and on which it relied at least to some extent for it’s subsequent call for more speed cameras, was no less astonishing, as indeed was it’s dismissal of vas on the basis of a mere 12% difference.

The remainder of this document sets out how that data submitted by the DfT to the Committee, stage by stage, skewed what should have been a massive advantage for VAS into a marginal one in favour of speed cameras. Note that what follows sets out only how these figures and calculations were incorrect and/or misleading – we make no attempt to explain why they were or why the these grotesque errors were neither spotted nor, even worse, acknowledged by the DfT or Select Committee after they had been pointed out. In the few places where we set out how such calculations should have been done, we make clear that these are our opinions, albeit opinions that are surely irrefutable.

The Details, Step by Step

Please Refer to House of Commons Transport Committee report “Roads Policing and Technology: Getting
The right balance Tenth Report of Session 2005–06” (18 October 2006) and in particular to the sections identified in the text below.

In the context of the change to Camera Partnership rules from 1st April 2007 the Select Committee asked the Home Office and DfT to provide information on alternatives to speed cameras, and their relative cost effectiveness. The information provided by Dr. Stephen Ladyman for the DfT is set out on Page EV156 of the Select Committee report, par. 18 onwards. Only the figures for cameras and VAS are relevant to this analysis.

(A) The first significant point is that the DfT failed to mention to the Committee even the existence of TRL548 – despite having published it in January 2003 and it having been “a large scale evaluation” of precisely the sort of alternative to cameras that the Committee had asked about.

(B) Secondly, the DfT claimed that “For the reasons set out above it is difficult to undertake a direct comparison of the effectiveness of speed reduction measures. Furthermore whilst a number of techniques have been individually evaluated over recent years, these assessments have not generally looked at value for money or cost benefit matters. In order to do make any attempt at a meaningful comparison it is necessary to compare like with like. It is not therefore possible to compare the effectiveness of a national programme, such as the national safety camera programme, with a 20 mph zone or vehicle activated sign.”

In our view this is nonsense for several reasons:

• Precise comparisons of cost effectiveness are indeed impossible, given the variety of systems and sites, and in particular given that the supposed benefit relies on what can only ever be estimated, not measured - what would have happened had the systems not been in place.

• However, that precise comparisons are impossible cannot excuse failure to do the approximate comparisons, based on the mass of data available that were possible. Given that these approximate comparisons show a difference of at least 50 to 1, there is no need whatever for a more accurate figure to allow the only logical decisions to be made.

• That such comparisons had not been done previously cannot excuse failing to do them when asked by the Committee for that information – especially as all the necessary data (Camera Partnership reports and TRL548) were already on file at the DfT. Indeed, it would have been the work of moments to find in TRL 548 that VAS reduce speeds and accidents by rather more than do cameras, but at a small fraction o the cost. Similarly “The national safety camera programme Four-year evaluation report” of December 2005” (pg. 8 par.4) shows operating costs of £96m p.a. for (what it confirms elsewhere as) 4,108 camera sites but only 1,876 cameras – an annual cost per camera of the order of £50,000 compared to £800 for VAS. 

• This £50,000 figure of course includes all the support services as well as hardware, but as cameras would serve no purpose without them, no rational comparison could exclude such costs. Even then however, the figure does exclude substantial legal, time and other costs incurred by motorists.

• VAS in contrast, averaging around £800 p.a. each (£6,000 to install and £200 p.a. to run) involve no such costs whatever, and their use would therefore significantly lower the workload of magistrates courts and the legal profession.

• Thus the DfT already had in its possession more than enough reliable and directly equivalent data to confirm that VAS are substantially more cost effective than speed cameras, by a factor of at least 50 to 1. That they failed to do so is not only inexplicable, but also culpable.

(C) Despite having argued (above) that it was not possible to carry out meaningful comparisons, and the necessity of comparing like with like, the DfT then, astonishingly, submitted to the Committee (EV157 onwards in the Committee’s report) data for just one camera site on a an urban single carriageway road, and just one VAS site at a rural junction, taken from Appendix A of their Road Safety Good Practice Guide.

This overall comparison claimed a 12 to 10.6 advantage for cameras, was, astonishingly, accepted by the Select Committee despite the prima facie nonsense of the cost figure for a VAS being almost double that of a camera. We summarise below how both the data and the methods used to calculate this ratio were seriously flawed and therefore utterly misleading, 
 

• Accidents being what they usually are, the result of the near random coincidence of more than one adverse factor, accident data from single sites are virtually meaningless in statistical terms. We find it impossible to understand why the DfT used accident reduction data in low single figures, from only one site of each type, when they had a great deal of better data to hand. (However to demonstrate the other DfT errors we do need to take these single digit figures as read in what follows).

• The camera site experienced an average of 5.8 accidents p.a. before installation, but the VAS site only 3.1 p.a. (or 1.55 p.a. per sign as there were two signs, facing different ways). Either way, this clearly limited the potential of VAS accident reduction compared to that of the camera. 

• The camera site showed accident reductions from 5.8 p.a. to 3.6 p.a., (down 2.2 or by 38%) and the VAS from 3.1 p.a. to nil  (down 3.1 or 100%). In order to compensate for the widely different “before” accident numbers, the percentage reduction figures should have been carried forward to the cost effectiveness calculations, but for reasons best known to themselves, the DfT chose to use the absolute numbers, thus skewing the comparison by a factor of 2:1 in favour of cameras.

• That there were two signs at the VAS site was unclear in the DfT submission, and hence the impression was given that the cost of a single VAS was £14,000. The real figure per sign was therefore £7,000 and in response to a FoI request, Norfolk County Council, who installed that site, have confirmed in an email message that:

“2 signs at Felthorpe in February 1998. Both were early prototypes of vehicle activated signs (VAS) ……..(and)  used fibre-optic technology whereas VAS now are manufactured (in significant numbers) using LEDs which are both cheaper and more reliable. The cost of these 2 schemes is not therefore a good guide for any similar scheme today……If a commuted sum to cover 10yrs maintenance and electricity were to be added I would suggest a figure of £2000. Some of the newer smaller VAS mounted on an existing lighting column can cost as little as £3000 total”.

• Oxfordshire County Council, in response to a FoI request, have confirmed that: 

 

 
 
 
 
 
 
 

“The cost of £ 7,500 indicated in the Department of Transport’s Good Practise Guide for Road Safety was for the camera housing plus setting up a power supply. During the 1990s the housings for the cameras cost approximately £ 6,000……….. In 1998 the cameras themselves cost approximately £ 32,000.”
 

• Even if these figures had been accurate, it was been totally inappropriate to base cost comparisons of equipment which has a useful life of a decade or more) on first year costs alone. This is especially so when, cameras have high operating costs and vas utterly trivial maintenance costs.

Summary

The DfT, having been asked by the Select Committee to provide comparisons in cost effectiveness of alternatives to speed cameras: 

(a) failed to disclose the existence of TRL548.

(b) claimed that such comparisons were not possible despite having all the necessary information to hand.

(c) then provided a statistically meaningless comparison based on only two, dissimilar sites. 

(d) failed to realise that the £14,000 cost was for two VAS and not representative of current costs 

(e) quoted a bizarre first year cost of £7,500 for a speed camera, when the real figure was £40,000 plus. 

(f) failed to compensate for widely different accident levels at the two sites. 

(g) used wholly inappropriate first year costs as the basis for comparison and 

(h) in all these ways transformed what they should have known since 2003 to be a massive cost effectiveness advantage for VAS into a marginal cost effectiveness for speed cameras.

That Dr. Stephen Ladyman  (as the DfT have confirmed) authorised submission of these figures to the Committee, and that the Committee then accepted them, much too important in terms of road safety policy to be ignored – hence this note, following the refusal of both bodies to acknowledge their errors.

It is also, incidentally, rather surprising that neither the Committee in their 264 page report nor Dr. Ladyman in the various DfT submissions ever mentioned the possibility that within months a verdict in favour of the applicants to the European Court of Human Rights over the right to silence in O’Halloran and Francis v UK (Application Nos: 15809/02 and 25624/02) may render S172 1988 RTA and the entire speed camera system inoperable, given that vas provide a complete solution to the problems the authorities would then face.

Caveat

While it is necessary in this context to compare on a like-for-like basis the reductions in speed, accidents and casualties at camera and sign sites, as reported by the DfT, TRL and Camera Partnerships, nothing in this note should be read as necessarily agreeing that all of those reductions are in reality due to the presence of those cameras or signs, rather than to regression to the mean, pre-existing downward trend or other factors covered in Appendix H of the 4th annual report on Camera Partnerships  However it is clear that the greater the extent to which the notional benefits of cameras or signs should be reduced – sensibly by similar proportions - to allow for such factors, the more important it becomes that the costs involved be minimised.

Idris Francis irfrancis@onetel.com (ECHR applicant)
Paul Smith psmith@safespeed.org.uk (Safe Speed)
Malcolm Heymer malcolm.heymer@btinternet.com (Association of British Drivers)
Mark McArthur-Christie mark@freemanchristie.co.uk (Association of British Drivers)
Paul Biggs pmbbiggsy@yahoo.com. (Association of British Drivers)