Not too long ago, when the instrument panel on a popular car consisted of just five or six instruments and five or six auxiliary (secondary) controls to operate the radio and heating system, the idea of making a hand gesture in a designated space to operate one of these controls would rightly have been seen as an unnecessary extravagance at best.

Today, driver workload is already heavy in modern cars, with an ever-increasing number of vehicles on the road. This is compounded by a constant stream of new auxiliary devices such as navigation systems, active safety systems, nomadic devices (personal digital assistants and mobile telephones, for example), advanced telematics systems and infotainment systems. The potential for distraction can clearly be seen in the comparison between the JaguarMkV Saloon of 1948-51 (Fig.1) and the 2007 model year JaguarXJ (Fig.2).

Consider that a driver is to perform a specific in-car task. In some cases a single sample of the in-car task is sufficient but, in other cases, more than one sample is required. Glance times typically range from 0.6s to 1.6s, with a mean glance time of about 1.2s. A car moving at 48km/h (30mph) will travel over 15m (50ft) in that time, so driver distraction is now one of the major causes of road accidents.

In 1999, The Commission of the European Communities recognised the potential dangers and issued a recommendation on safe and efficient in-vehicle information and communication systems: ‘A European statement of principles on human machine interface, to ensure secondary controls are designed to meet safe and common standards’, which cites driver distraction as one of the reasons for producing the recommendation.

In an attempt to reduce distractions, many car manufacturers have introduced menu-driven systems, controlled by touchscreens or central controllers. Voice recognition control systems have also been introduced, usually in conjunction with touchscreens or central controllers, because voice commands can be used without taking the driver’s eyes off the road. Since each of these existing interface solutions has limitations, there is increasing interest in gesture recognition driver interface systems.

Gesture recognition systems have the potential to offer substantial safety benefits, since commands can be made without taking the driver’s eyes off the road, and gesture recognition systems do not share the same limitations as voice control systems or menu-based systems using touchscreens or central controllers. Several manufactures are now carrying out research to develop new gesture-operated interfaces using camera-based systems that utilise image recognition software. Much of this work is being carried out in conjunction with Tier 1 suppliers and universities. However, this research has identified several inherent difficulties associated with in-car camera-based gesture recognition systems, such as adapting to uncontrolled variations in lighting, maintaining accuracy with dynamic backgrounds, and real time operation. While these challenges are likely to be overcome, and camera-based gesture systems will probably appear on new cars in the not-too-distant future, the systems will probably add technical complexity and cost. Nonetheless, the author believes that there is a reasonable probability that gesture recognition technologies will be in widespread use in numerous automotive HMI applications by 2020.

Engineers at Jaguar Technical Research have considered the potential advantages of gesture recognition systems but are well aware of the limitations of camera-based systems. So instead, the engineers are pursuing an alternative sensor-based approach that has less technical complexity yet can achieve the same safety benefits. Sensor-based gesture recognition technology largely builds on pioneering work undertaken at the Massachusetts Institute of Technology in the USA.

Low-frequency electric field sensors are safe, do not require line-of-sight, offer fast response times and high resolution, consume little power and are low cost. The intervention of a human hand entering the path between the transmit and receive electrodes cause a change in the displaced current measured at the receive electrode; this can be used to transmit control inputs to a HMI.

Sensor-based systems do not suffer from the technical challenges facing vision-based systems, yet they still offer the same safety benefits and HMI opportunities. This supports the suggestion by the author that a sensor-based gesture-recognition HMI may be the best way to achieve a safer, more reliable secondary control interface on cars.

Carnegie Mellon University is developing the iWave gesture recognition system in collaboration with, and funded by, General Motors. The primary objective was to create an innovative human-car gesture interface to support information or entertainment goals without compromising safety. The initial interface was designed and tested in a driving simulator with 18 subjects using one-handed gestures in front of the centre console. A projector was used to project a first-person view of downtown San Francisco and users controlled the driving simulation directly with steering wheel and pedals. A second projector was used to emulate the use of a head-up display (HUD) that displayed the driver menu options for navigation and entertainment, together with the required gesture control icons showing which hand movement should be used to trigger the displayed commands. As the subject made these gestures, an unseen experimenter interpreted the gestures and controlled the interface. The results were then compared with similar tests using a conventional physical interface. The results claim that the subjects made fewer total errors in the gesture mode than in the standard mode, but the difference was not significant. The conclusion claimed that gesture interfaces are a viable option for secondary tasks.

The Institute for Human Machine Communication at the Technical University of Munich has carried out a research study in collaboration with BMW to evaluate differences in driver distraction while controlling different input interfaces. In this study, haptic (touch) and gesture input modes are compared with regard to distraction from a controlling task similar to steering a car. The study was carried out in a driving simulation laboratory using an automated sequence of control tasks.

During the experiment, the steering error, the object recognition performance, and the duration of each user input task was measured. The results showed significant benefits forgesture inputs.

Gesture Panel

Daimler Chrysler, together with Visteon, is also funding the Georgia Institute of Technology to develop a system, called the Gesture Panel, that allows a driver to control secondary devices using simple gestures. The Gesture Panel uses a camera aimed at a grid of infrared light emitting diodes (IR LEDs), and gestures are made between the camera and the grid of IR LEDs. This occludes light from some of them, so gestures can be recognised, based on the various patterns of occlusions through time. Hidden Markov Model techniques are then used to identify each gesture from the light occluded patterns and output the relevant control command.

The Daewoo Motor Company is collaborating with the University of Dundee to develop a non-contact pointing interface for control of non-safety-critical systems inside a vehicle with the aims of improving safety, decreasing manufacturing cost and improving the ease of driver migration between different cars. A driver operates the interface via an on-screen cursor using pointing gestures to be identified by a computer vision system. This research is part of the Active (advanced camera technology in visual ergonomics) project that aims to replace the secondary, non-safety-critical controls, such as radio and climate control, with a virtual implementation displayed in front of the driver. Daewoo featured this system in its Mirae concept car.

Mitsubishi Fuso Truck and Bus Corporation, in partnership with Keio University of Japan, is developing a new camera-based system that will allow drivers to operate secondary controls using hand and finger gestures only; drivers would not have to look at or touch any controls.

Renault Research Department has also collaborated with the Université de Bretagne-Sud in Vannes, France, to evaluate the performance of subjects using a small gesture touchpad interface versus conventional rotary controls to execute given tasks.

Toyota’s Compact Sports and Speciality (CS and S) concept car made its debut at the Frankfurt Motor Show 2003 featuring the company’s ‘Space Touch It’ concept. This is an integrated multimedia interface system operated by a series of holographic projections that the user ‘touches’. Spheres of information appear to float in space but, when touched, they allow the user to operate the vehicle’s secondary controls.

Increasing awareness of the need for a safer driver interface has also led parties outside the automotive industry to suggest that gesture recognition applications could help to achieve this. The UMEA Institute of Design in Sweden has carried out research based on gesture recognition to control a car radio. The theoretical model suggests using an image of the radio reflected onto the windscreen and some form of hand proximity/position sensors for interactive control of a limited number of the radio’s controls. Either an array of ultra violet sensors, or capacitance/resistance switches are suggested.

Alpine Electronics has developed a working prototype automotive gesture control system called ‘Space Commander’, which uses a holographic projector system to display a secondary controls menu. The system uses infrared radiation gesture recognition and an innovative air curtain that provides haptic feedback when a finger gesture input is made to the hologram.

Canesta, of California, USA, has developed a low-cost electronic perception system that is based on time-of-flight technology. This system consists of a modulated light source, an array of pixels (each capable of detecting the phase of the incoming light), and an optical system for focusing the light onto the sensor. The distance is measured and a depth image is produced.

User acceptance

The rate of introduction of any automotive gesture recognition system will be largely dictated by the rate of user acceptability. This will be driven by how fast and widespread gesture recognition becomes established in our everyday lives where human interaction with machines takes place. Touch-based gesture recognition will probably lead the way in gaining user acceptability with personal computer (PC) tablets, PDAs (personal digital assistants), point-of-sale (POS) terminals and, of course, touchscreens for automotive applications and possibly touch control for positioning air vents. The author is aware of a gesture recognition touchscreen from Sony; the natural extension will be lower-cost gesture touchpads that can utilise existing displays within the vehicle. These touchpads could be located in the centre of the steering wheel, and possibly a general touch pad located in the centre console area.

Although touch-based gesture interfaces are likely to gain the first real significant market penetration, it is the non-contact gesture recognition technologies that will ultimately be more widespread. Non-contact gesture recognition systems are more flexible, as they can be used remotely from the sensing technology. Gesture commands can therefore be given from the comfort of your office chair, armchair or car seat without the specific need for a contact-based user interface.

Carl Pickering is with Jaguar and Land Rover Technical Research, Coventry, UK.