KPV Validation

September 20, 2017

This post will introduce the idea of KPVs in aviation safety; their uses in flight safety analysis, their value as a data product (from a data science perspective), and various techniques that can be applied to analyse them.

A KPV is a Key Point Value. From the FlightDataAnalyzer docs:

A Key Point Value is a value of interest measured at a point during a flight. The key attribute of a KPV is the value - which represents a measured value at a point in the flight. A point can be a single instance or the start of a slice eg. duration.

In real terms, a KPV is an “interesting value” - usually sensor output that’s been saved in the flight data recorder, decoded using the appropriate LFL definition1 (each aircraft has a standard data format which is usually unique to its particular model), and then converted to “engineering units” - standard units of measurement. Usually a KPV is recorded at a specific point in the flight (for example, during liftoff or approach/landing). Note that not all KPVs are directly calculated from sensor data - some are derived from other KPVs (usually determined by applying some transformation based on another KPV). An example of this is the transformation from total air temperature to static air temperature (air temperature measurements must take into account the speed of the aircraft because of the effects of friction and air compression).

Basic Bounds Checking

Before applying any statistical validation to KPVs, it’s helpful to remove as much junk data as possible beforehand. In this case, it means that domain knowledge should be harnessed to develop sensible heuristics to get rid of junk data. For example, take the KPV Pitch At Liftoff. This is the angle of the nose of the aircraft as it leaves the ground (specifically, as the radio altimeter reads an altitude of greater than 0). For some flights, this KPV registers a value of less than 0, which doesn’t seem right - the nose of the aircraft is unlikely to be pointing towards the ground during liftoff (unless this metric needs to be adjusted for the attack angle of the wings, flaps, elevators, etc). Upon further analysis, it appears that the negative Pitch At Takeoff value is likely generated by helicopters - which don’t need to change pitch to create lift.

An example of the Pitch At Liftoff KPV - note the yellow values on the

This sort of technique can even be applied to much simpler data, too - even enforcing sensible bounds on timestamps flags a surprising amount of incorrect data (for example, ensuring that data lies is between 1970-01-01 and 2020-01-01). Likewise, it can be assumed that aeroplanes can’t have a negative pitch angle on takeoff, that they can’t have a negative altitude, etc, etc.

Correlation Analysis

Certain KPVs are likely to be correlated with each other - for example, SAT/TAT and altitude. As altitude increases, temperature is expected to decrease (specifically, by about 2 degrees Celsius per 1000ft). Clearly then, if temperature is measured at several different altitudes during the flight, one would expect to see a negative correlation between altitude and air temperature.

Interestingly, if a particular flight doesn’t exhibit this correlation pattern, it is quite likely that the aircraft experienced flew through a weather phenomenon known as a temperature inversion (where the atmosphere gets cooler at lower altitudes - potentially creating challenging conditions for the pilots).

Monitoring Distributions Over Time

As well as filtering out extreme values (and flagging invalid data based on boundaries determined by domain knowledge), it’s also possible to look at how data is distributed. For the most part, if the data in the underlying distribution is being recorded accurately then it can be assumed that these distributions won’t change significantly over time. Therefore, if these changes exceed certain boundaries, then it should be possible to trigger an alert that there is a data problem.

Mean, Standard Deviation

This is fairly self explanatory - standard distribution parameters can be monitored over time (mean, standard deviation, skew and kurtosis). It should also be possible to to characterise distribution types (e.g. exponential/Weibull/Gaussian) based on domain knowledge (i.e. an understanding of the generative model underlying the data) and distribution fitting techniques (e.g. least squares, kernel density estimation, etc).

Number of Modes (Unsupervised Cluster Analysis)

Generally, one would expect the number of modes in a distribution to remain constant over time. Data being created according to the same generative model should retain the same structure - particularly as each distribution is generally unique to each aircraft model and aircraft type (i.e. it’s often possible to identify particular helicopters and aeroplanes based on some of their KPV distributions). In some cases, it might be useful to characterise distributions by the number of clusters in the data - for example, note the plot below containing distributions of Pitch At Liftoff. There is a bimodal distribution for Pitch At Liftoff for the A330 and A340 - this could possibly be caused by some property of these particular aircraft, but it’s likely that there’s a problem with the data - that is, unless there genuinely are aeroplanes taking off pointing directly towards the ground.

An example of unimodal and bimodal

In the case of the A330 and A340 above displaying clear bimodal distributions, it is potentially useful to apply a clustering algorithm to estimate (in an unsupervised manner, at least) the number of clusters. In the case displayed below, mean-shift clustering has been applied - each subsequent cluster has been highlighted with its own colour.

An example of mean-shift clustering applied to bimodal

Probabilistic Filtering Methods

To see how KPVs develop over time, it might be helpful to consider a probabilistic modelling technique such as a Kalman filter. A Kalman filter should be able to very quickly learn the normal variation of a time-series dataset (e.g. monthly KPV data), and make probabilistic estimates of the bounds that the data should lie within. Even more conveniently, this filtering approach should be able to estimate sensible confidence bounds for parameter values (as it is a Bayesian method) - and therefore, trigger sensible automated alerts when this threshold is exceeded.

Identifying the Causes of Problems

Once some issues with data have been identified, it’s often useful to dig into them a little deeper by attempting to find correlated variables. For example, if data from a particular model of aircraft is incorrect, then it might be useful to see if there are particular patterns - for example, if the data problem only appears after a certain date, it’s usually possible to diagnose a faulty LFL update or a software change. Similarly, it is also possible to identify specific aircraft that exhibit this flawed data. Once some common traits have been identified, it’s possible to identify the underlying cause of the faulty data very quickly.

Decision Tree Modelling

One of the easiest explanatory models to understand is the humble decision tree. They’re fast to build, easy to visualise, and, when used in ensemble (e.g. XGBoost or Random Forests), very robust to overfitting. All of these factors allow decision trees to be used to rapidly create diagnostic plots for identifying data problems.

An example of some (anonymised) decision tree rules after applying basic
correlation modelling

  1. An LFL is a “Logical Frame Layout” - a standard data description format associated with the output of the flight data recorder for each particular model or variety of aircraft.