AI Knows When Drivers Are Getting Very Sleepy

“It’s very realistic,” says Arno Eichberger, an engineering professor at Graz and the team leader. “And for the specific study here on drowsiness, it’s even better because this type of monotonous driving is not that difficult to simulate.”

They recreated a nighttime drive on a highway without any traffic. “For some of the drivers, we got a complete micro sleep,” he says.

The team collected data on 92 drivers. Each participant came to the lab twice: once while rested, and once when fatigued. In the fatigued condition, they were required to have been awake for at least 16 hours or to have slept for no more than four hours the previous night. On each visit, they participated in manual- and automated-driving scenarios.

“The basic idea was to create a huge, unique database of drowsy drivers that would also be available for public use,” Eichberger says.

Objectively measuring drowsiness remains a research challenge, but for their study, the Graz researchers created ground truth labels by asking traffic psychologists to watch recordings of the drivers and give their best assessment based on yawning, head nodding, and long blinks. The psychologists provided one of four labels: alert, moderately drowsy, extremely drowsy, and falling asleep. This study combined the last two into extremely drowsy.

Eichberger says it is important to have at least three classes of drowsiness because if you have only two—alert and extremely drowsy—then when a car warns you that you’re extremely drowsy, you’re already in danger.

Deep Sleep

Previous drowsiness detection methods had used manually coded rules to process ECG signals, which arrive as a complex waveform. Each time the heart beats, it produces something in the ECG called an R-peak. Programmers have told their software to look for those R-peaks, measure the length of time between them, and calculate how much those spans vary, producing a statistic called heart-rate variability, which correlates with drowsiness. But these methods might miss other important information hidden in the ECG signal that researchers don’t know to look for. The power of deep learning is its ability to find those subtle patterns, the way humans build intuition through experience.

So how do you apply a convolutional neural network, designed for images, to an ECG signal, which is just a sequence of electrical amplitudes? And why would you do that? To answer the how: You turn the waves into an image.

Sadegh Arefnezhad, the lead author of the paper on the new method, published in Energies, used Wavelet Toolbox™ in MATLAB^® to create wavelet scalograms. Time-series data can be considered the sum of many brief “wavelets” of different frequencies. MATLAB decomposes the wave into these simpler wavelets, with time on the x-axis and frequency on the y-axis. The brightness at each point in the wavelet scalogram represents the amplitude of the wavelet of that frequency at that time.

Why transform a wave into a shaded image before feeding it to a neural network? “The idea is that a time-frequency view of a signal can make relevant characteristics more apparent than in the raw time-domain data,” says Wayne King, a principal software engineer at MathWorks. “Importantly, creating images allowed the researchers to take advantage of convolutional neural nets, which computer scientists have finely honed over the years.”

Neural networks consist of parameters that define the strengths of connections between virtual neurons. These parameters change during training. They also use hyperparameters during training. Researchers set these values to control the learning rate (how much the parameters change in response to feedback) and how much noise to add during training (which affects the network’s robustness). Some people pick hyperparameters based on rules of thumb, and some use brute-force search to try many. Arefnezhad employed Bayesian optimization, which uses probability theory to narrow the search over time.

The team tested the network on images it had not seen and compared its performance to two other machine learning methods, both of which relied on features manually extracted from the ECG data. First, they collected all the intervals between R-peaks. Then, they calculated 11 values, such as the standard deviation within a set of intervals. They fed these values into one of two classifiers, a k-nearest neighbor (KNN) model and a random forest. The best of these baseline methods, the random forest, achieved 62% accuracy when classifying drowsiness in manual driving modes and 64% in automated modes.