Heathrow’s Fly Quiet results reach new heights of improbability
Heathrow has this week (22nd March) belatedly published the results from its Fly Quiet & Green programme for Q4 2018. In this scheme Heathrow assesses 7 different aspects of environmental performance, but it only publishes a single, numeric “Fly Quiet points” score for each airline. That published score is the sum of the Fly Quiet points awarded to the airline for each of the 7 metrics. But that part that is far from transparent, with the 7 numbers per airline not made public. The results put out by Heathrow do not make any sense, and do not appear to properly reflect the actual noise. Rather, they appear to be manipulated to make noise levels look lower than they really are. This time around instead of giving the airlines an average score of around 750 out of (optimum) 1000, as with previous quarters’ results (already grossly inflated), Heathrow has hiked the average score by over 8% to 813 points. The expected average (mean and median) score should be around 500. But not content with inflating the scores even more than usual, Heathrow has also inexplicably excluded 5 (China Southern, El Al, Korean Air etc) of its 50 busiest airlines from the results – but added others instead.
.
Tweet
Comment from AvGen on the recent Heathrow noise figures
23.3.2019 (From Dave Reid, at AvGen)
The Fly Quiet & Green programme aims to measure, assess and compare the environmental characteristics and performance of the airlines that serve Heathrow and the aircraft that they use (we fully support that aim, though not the way it has been implemented).
Heathrow assesses seven different aspects of environmental performance, but it only publishes a single, numeric “Fly Quiet points” score for each airline. That published score is the sum of the Fly Quiet points awarded to the airline for each of the seven metrics. That’s the part that is far from transparent and is, AvGen believes, flawed.
Heathrow consistently declines to provide a breakdown of how many of the points that make up an airline’s published total Fly Quiet score have been gained from each of the seven metrics.
That refusal is ridiculous and disingenuous, particularly given that Heathrow identifies which airline is best, second-best, third-best, etc, for each of the metrics and states how those rankings are supposed to translate unambiguously into Fly Quiet points.
Our view is that Heathrow cannot publish the Fly Quiet points breakdown per metric because it would reveal that points are not being awarded in accordance with the stated rules of the scheme.
Specifically, we believe that airlines performing poorly on any of the seven metrics are being overmarked, sometimes significantly, which inflates the resulting Fly Quiet points scores.
Heathrow could, if it chose, address our criticisms by simply publishing the points breakdown for the world to see.
.
Background to the “Fly Quiet and Green” scores
In July 2017, Heathrow published the first of a new series of quarterly Fly Quiet & Green statistics, ranking airlines on several different aspects of their environmental performance.
Shortly after publication, AvGen drew Heathrow’s attention to apparent anomalies in the calculation of the results. The published “league table” did not appear to be reproducible, despite using exactly the methodology and inputs published by Heathrow.
AvGen requested some worked examples of the methodology from Heathrow in an attempt to identify where the discrepancies lay. Heathrow did not respond to this request.
A subsequent request by AvGen elicited the response from Heathrow that it was satisfied with the accuracy of its results, but still without providing any examples to substantiate them.
AvGen then provided Heathrow with a paper highlighting (with specific examples) areas where it believed Heathrow’s analysis was flawed. No response was received to this.
See full details of the AvGen assessment of “Fly Quiet and Green” at
Fly Quiet & Green Forensics V1.3
AvGen says:
We will never know how China Southern, El Al, Korean Air, Kuwait Airways or Pakistan International Airlines are judged to have performed, because Egyptair short/longhaul, Icelandair (ditto) and MEA longhaul (all with fewer flights than any of the above) have been substituted instead. In fact El Al and China Southern had over three times as many flights as MEA longhaul during Q4.
We will, as usual, invite Heathrow’s comments on our findings, though past experience suggests there will be none. Heathrow could, of course, choose to publish a breakdown of how many points each metric contributed to airlines’ aggregate scores – but that would make its flawed results even more obvious.
Incidentally, although Heathrow never acknowledges it when I send Matt Gorman, Richard Norman and Nigel Milton a copy our our quarterly findings, they have taken note of one of the issues I raised in the email. Those have now been fixed on the website..
A more detailed look at the Q4 table shows:
- a) Individual airline scores are inflated by between 17% and 240%, with the poorest performing carriers receiving the biggest unjustified increase in their score. The 544 points score awarded by Heathrow to MEA shorthaul is over 380 points more than the airline actually merits based on its performance and Heathrow’s rules..
- b) 48 out of the 50 airlines in Heathrow’s table are awarded more than the correctly calculated average (based on Heathrow’s data and methodology) of 522.
- c) Turkish Airlines longhaul and Jet Airways are given an unexplained hike up the table, each by 15 places, compared to the positions that their performance merits.
- d) Among the airlines entitled to feel aggrieved with this quarter’s published results include Icelandair shorthaul, relegated 20 places from its rightful position.Turkish Airlines shorthaul, despite meriting 503 points by Heathrow’s own methodology, putting it just above Air Malta, bizarrely ends up ranked 21 places below the Maltese carrier.
- e) “RAG” (red/amber/green) classifications are again applied inconsistently; for example Thai Airways and TAP, ranked 44th and 45th, respectively, by Heathrow for early/late movements, get an “Amber” for that category while Scandinavian, ranked 34th for that metric by Heathrow, gets a “Red”.
- f) For the second successive quarter, 180 flights by Finnair’s A330 and A350 fleets (out of an airline total of 905) appear not to have been taken into account in calculating the results, with only its narrow-body A320 family flights having been counted.
Dave Reid
AvGen Limited
Reading, UK
+44 (0)118 975 7929
dave.reid@avgen.com
Another way of looking at the Fly Quiet anomalies – by AvGen
Consider just one airline: Oman Air.
We’re asked to believe that its environmental performance across the seven Fly Quiet
metrics in Q3 2018 merited an aggregate score of 917 – that’s only 83 points short of the 1,000 “perfect” score !
So how did Heathrow decide to deduct only 83 points ? The airport refuses to explain, probably because it makes no sense.
Leaving aside the 2 metrics where Oman Air ranked Number One and so didn’t lose any points for those (Track-keeping and Early/Late Movements), we are left with the other 5 metrics (Noise Quota Count, Noise Chapter, NOx emissions, CAEP and CDAs) where those 83 points must therefore have been deducted (for 24th, 8th, 34th, 7th, and 20th place, respectively).
We can easily deduce how much those 5 metrics each contributed to the 83 points lost, because Heathrow tells us how many places Oman dropped for each metric and the relative weighting per metric (50%, 50%, 50%, 50% and 150%, respectively).
For Noise Quota, Oman dropped 23 places (47% of the way down), but only lost 15.2 points (17%) from the 89.3 available.
For Noise Chapter, Oman dropped 7 places (14% of the way down), but only lost 4.6 points (5%) from the 89.3 available.
For NOx, Oman dropped 33 places (67% of the way down), but only lost 21.7 points (24%) from the 89.3 available.
For CAEP, Oman dropped 6 places (12% of the way down the table), but only lost 4.0 points (4%) from the 89.3 available. [CAEP means Standard (engine emissions certification)]
For CDAs, Oman dropped 19 places (39% of the way down), but only lost 37.5 points (14%) from the 267.9 available.
BUT the Fly Quiet rules state that, for each metric, the proportion of points deducted from the maximum available score is determined solely by how far down the table the airline is for that metric. So the above scores make no sense at all.
If points deducted by Heathrow are compared with places dropped, then its flawed implementation of its own rules means that Oman, if it came bottom for every metric, would still be awarded 638 points instead of the 0 that the rules specify !
.
See earlier:
Consultancy AvGen finds, yet again, Heathrow’s “Fly Quiet & Green” programme comes up with weird, incorrect, results
Heathrow has published the results from its Fly Quiet & Green programme for 2018 Quarter 1. Unfortunately it seems determined to persist with the flaky arithmetic and absence of logic and common sense that characterised the results for previous quarters (which remain unaltered). For Q1, as with previous quarters, league table scores have again been inflated, this time by an average of around 44% compared to the results that are produced when Heathrow’s own published methodology and performance rankings are used. Once again that increase has not been applied uniformly across all 50 airlines (a number of them have been awarded more than double the number of points that they merit), with the result that the relative league table positions are significantly altered. Below are some examples, from consultancy, AvGen, showing the arbitrary results – which do not appear to be based on much logic – of airlines being put into higher and lower rankings, based on their noise and emissions. By contrast with the Heathrow figures, those from AvGen show the greenest airline is Aer Lingus – not Scandinavian. The second greenest is Finnair, not LOT Polish Airlines. Curious that Heathrow does such odd things with the data ….
.
.
.