AvGen’s concerns mount over Heathrow’s “Fly Quiet” statistics – very opaque how they are arrived at
Heathrow produced figures, intended to show how well airlines that use the airport are performing in terms of noise. The criteria include noise quota/seat, plane Chapter number (noise certification), the NOx emissions/seat, the CAEP standard (engine emissions certification), the Continuous Descent Approach (CDA) violations, the airline’s Track keeping (TK) violations, and early or late movements between 23:30 and 04:30. The group, AvGen, assesses the numbers put out by Heathrow, and finds – every time – that the numbers do not make much sense, and do not even match the stated methodology by Heathrow. Airlines get given numbers of points (it is far from clear how these are measured), and rankings for how “quiet” they are. It appears airlines are bumped up and down the rankings in a fairly random way. Perhaps to make some airlines look good, and gloss over the amount of noise they make? For the Quarter 1 (Q1) results this year, AvGen calculates the figures, using the stated Heathrow methodology, British Airways short haul comes out 4th best (Heathrow put them first); British Airways long haul comes out 14th (Heathrow puts them 6th). Aer Lingus comes out at 10th best (Heathrow has them 4th). And so on. Contact AvGen for the full data.
.
Tweet
AvGen’s concerns mount over Heathrow’s Fly Quiet statistics
From Dave Reid, of AvGen
11th June 2019
Heathrow last week belatedly revealed the results from its Fly Quiet & Green programme for Q1 2019, after having concealed them from public scrutiny behind password protection for almost a month.
For Q1, as with previous quarters, league table scores have again been inflated, this time by a record 46% compared to the results that are produced when Heathrow’s own published methodology and performance rankings are used.
Once again that increase has not been applied uniformly across all of the 50 airlines considered (some of the poorer performing carriers have been awarded more than double the number of points that they merit), with the result that the relative league table positions are significantly distorted.
A more detailed look at the Q1 table shows:
- a) Individual airline scores are inflated by between 9% and 135%, with the poorest performing carriers receiving the biggest unjustified increase in their score. For example, the 797 points score awarded by Heathrow to Jet Airways is 434 points more than the airline actually merits under Heathrow’s own rules, based on its published performance metric rankings.
- b) 48 out of the 50 airlines in Heathrow’s table are awarded more than the correctly calculated average (based on Heathrow’s data and methodology) of 528 points (from a maximum possible 1,000 points).
- c) Jet Airways and Air India are given an unexplained hike up the table, by 17 and 15 places respectively, compared to the positions that their performance merits.
- d) Among the airlines entitled to feel aggrieved with this quarter’s published results include China Southern, relegated 23 places from its rightful position, together with Icelandair shorthaul and Japan Airlines, each unjustifiably demoted by 16 places. El Al, despite meriting 368 points by Heathrow’s own methodology, putting it just above Jet Airways, bizarrely ends up ranked 22 places and 281 points below the Indian carrier.
- e) “RAG” (red/amber/green) classifications are again applied inconsistently; for example Cathay Pacific and BA longhaul, ranked 40th and 41st, respectively, by Heathrow for early/late movements, get an “Amber” for that category while American, ranked 35th for that metric by Heathrow, gets a “Red”.
- f) Finnair’s wide-body types (Airbus A330 and A350) which account for more than 20% of its flights at LHR and should therefore, under Heathrow’s rules, be considered separately, have been lumped in with the narrow-body fleet, thereby invalidating the results for metrics such as Quota Count per seat and CAEP.
- g) BA shorthaul has been propelled into first place (even though SAS, Delta and United actually performed better) and we’re asked to believe that it scored only 43 points short of a “perfect” 1,000 despite coming close to bottom (45th out of 50) for engine emissions and two-thirds of the way down the rankings for Night Quota compliance. It turns out that Heathrow has only docked BA a little over 0.6 of a Fly Quiet point for every place lost across the seven parameters measured (adjusted by the appropriate “weighting”).That would mean, were BA to be the worst-performing airline (i.e. in 50th place) for every parameter, that instead of the zero points that the rules stipulate, it would still be awarded a score of 833 out of 1,000 !
However one positive outcome in this quarter’s results is that, for the first time, Heathrow has successfully managed to correctly determine its 50 busiest airlines (by counting how many times each airline lands on its runways).
Leaving aside the ingenious way the final league table is manipulated, even some of the individual metric rankings (which, up to now, we had been inclined to accept by default) turn out to be suspect when examined closely.
Take noise, for example. Heathrow uses the Quota Count (QC) values (based on ICAO noise certification). So far, so good – an average QC value per flight, or even per seat, would be a good parameter to use for that metric in order to compare different airlines’ fleets.
But rather than do that, Heathrow totals the QC values for all an airline’s movements, divides the sum by (number of flights x average seat size, i.e. aggregate seat capacity) and then divides the answer again by the number of flights. It doesn’t take a genius to work out that the best way to get a good score for this noise metric is simply to have a lot of flights, since the aircraft type involved carries far less weight.
Sure enough, BA longhaul, which operated nearly a quarter of its flights with noisy Boeing 747-400s (QC:4) and 777-200s (QC:2), gets ranked as second-best carrier for noise Quota Count whereas Air India for example, which operated all but a handful of its flights using modern, quiet Boeing 787-8s (QC:0.5), gets ranked Number 20.
As usual, attached is AvGen’s detailed audit of the Q1 results, contrasting Heathrow’s scores and league table placings with those calculated by AvGen using Heathrow’s own published methodology and metric performance rankings. We have copied these to Heathrow.
Also enclosed is an updated copy of our “Fly Quiet Forensics” paper that discusses the two years of the current programme and documents AvGen’s unsuccessful attempts to have a meaningful discussion with Heathrow about the issues.
We will, as usual, invite Heathrow’s comments on our findings, though past experience suggests there will be none. Heathrow could, of course, choose to publish a breakdown of how many points each metric contributed to airlines’ aggregate scores – but that would make its flawed results even more obvious.
Dave Reid
AvGen Limited
Reading, UK
.
There is information about the Heathrow “Fly Quiet and Green” programme at
https://www.heathrow.com/noise/making-heathrow-quieter/fly-quiet-and-green
See earlier:
Consultancy AvGen finds, yet again, Heathrow’s “Fly Quiet & Green” programme comes up with weird, incorrect, results
Heathrow has published the results from its Fly Quiet & Green programme for 2018 Quarter 1. Unfortunately it seems determined to persist with the flaky arithmetic and absence of logic and common sense that characterised the results for previous quarters (which remain unaltered). For Q1, as with previous quarters, league table scores have again been inflated, this time by an average of around 44% compared to the results that are produced when Heathrow’s own published methodology and performance rankings are used. Once again that increase has not been applied uniformly across all 50 airlines (a number of them have been awarded more than double the number of points that they merit), with the result that the relative league table positions are significantly altered. Below are some examples, from consultancy, AvGen, showing the arbitrary results – which do not appear to be based on much logic – of airlines being put into higher and lower rankings, based on their noise and emissions. By contrast with the Heathrow figures, those from AvGen show the greenest airline is Aer Lingus – not Scandinavian. The second greenest is Finnair, not LOT Polish Airlines. Curious that Heathrow does such odd things with the data ….
.
.
.