It’s been on the drawing boards for years and now the BMW Group is taking the wraps off of what has been highly conceptual work. Augmented reality has been on iPhones since 2008 but the technology has the potential to be much more of a watershed change in the way we drive and interact with technology. Will it come to MINI? As we’ve seen over the years, everything developed in the labs in Munich that sees the light of day in high-end BMWs eventually makes it across the model ranges. When? Think the next decade rather than next year. But before we get to that, lets take a look back.
In 2004, the BMW Group was the first automotive manufacturer to bring out a colour Head-Up Display which projected driving-related information directly in the driver’s line of sight. In early 2011 the latest generation of this system was introduced, featuring full-colour graphics. But already the BMW researchers and developers are working on a new milestone in this technology which will see the Head-Up Display acquiring “contact analogue” functionality. This is a technique whereby virtual “markings” are superimposed on real objects in the external environment, so that navigation information or information from the driver assistance systems can be displayed at exactly the right points on the driver’s view of the road scene. Navigation instructions can be blended into the road, and vehicles or safety-relevant objects can be highlighted or marked in context.
What exactly is augmented reality, or a contact analogue display system??The Head-Up Display was the first step towards in-car implementation of augmented reality. The HUD augments the external scene with additional information and artificially generated objects which react to and adapt to the situation in real time. The Head-Up Display presents useful information such as current speed and navigation information directly in the driver’s line of sight, but the technology has much greater potential than that: its applications can be greatly expanded by the use of contact analogue display technology.
Contact analogue displays are a special form of augmented reality. The displayed information is integrated into the external environment in the correct perspective and at the actual point or points in the scene to which it relates, so that effectively the information appears to be “attached” to the external objects. Contact analogue displays have many advantages. The fact that the information is presented in the driver’s direct line of sight, and that it is overlaid on the objects it is referring to, means that the driver does not have to shift his attention away from the driving scene. His gaze is not distracted and he does not have to change focus, as he does when looking back from an instrument cluster or central information display to the road. This means that information relevant to the driving situation can be scanned more quickly and more directly. At the same time, currently required actions can be displayed in an intuitive form.
“With the contact analogue HUD, we place the information at exactly that point in the driver’s field of view where it belongs and is required. The driver no longer has to correlate abstract information to the concrete driving situation. Since the display is directly congruent with the real world, we can also selectively direct the driver’s attention to specific information or hazards, so that he can respond quickly and in an appropriate manner.” (Dr Bernhard Niedermaier, Head of Human-Machine Interaction at BMW Group Research and Technology)
Contact analogue displays are already a reality.
Ideally, contact analogue displays would make use of the driver’s entire field of vision. From the technical point of view, however, this is not yet possible. Nevertheless, contact analogue display functionality can already be implemented even within considerably smaller display areas, with significant benefits for the customer. Two application scenarios are described below which illustrate the wide-ranging possibilities offered by contact analogue displays.
Contact analogue navigation.
The first scenario shows the possibilities of contact analogue display applications in the field of in-car navigation. Whenever a navigation manoeuvre needs to be performed, such as turning at an intersection, the system presents the information in such a way that it appears to blend with the road itself. The driver can keep his eye on the road throughout, and intuitively drives in the right direction.
“Turning manoeuvres and lane recommendations are shown in such a way that they appear to be directly mapped onto the road. Drivers no longer have to correlate the abstract image from a map to the road in front of them. Instead, the task is delegated to the contact analogue Head-Up Display.” (Robert Hein, Head of Navigation and Data Services of the Future at BMW Group Research and Technology).
Since the driver is better informed, he is able to drive more proactively and also more confidently. The spatial congruence between the displays and the road makes it easier for the driver to absorb information even in complex driving situations. In one of the first contact analogue systems developed by the?BMW Group, the Head-Up Display is approximately four times the size of typical HUDs used in current production vehicles. With such a display, it is already possible to present contact analogue information in the vehicle’s own traffic lane. In an upcoming version, the display area will be further enlarged to include neighbouring lanes as well.
Contact analogue navigation works like this: the navigation system calculates the optimal route based on digital road map information. If lane-level information is available, the route is further refined and the necessary lane manoeuvres are computed. Positioning information continuously supplied by GPS and vehicle sensor systems allows the vehicle to detect the lane it is currently travelling in and compare it with the optimal lane for the given route. If the vehicle is not in the correct lane for an upcoming manoeuvre, the system computes a 3D model of the road situation ahead using the signals from the camera system, and superimposes the instructions congruently on the external scene.
Application in driver assistance systems.
A second application scenario for contact analogue displays is in the field of driver assistance systems. Here the contact analogue functionality provides the driver with a better understanding of what is going on around him and makes it easier to absorb highly specific instructions. For example if the driver activates Active Cruise Control with collision warning, the system not only shows the driver – in the real-world environment – which vehicle is currently serving as the “lead vehicle”, it also shows the preset following distance, superimposed directly onto the road surface. Instructions to the driver to intervene are therefore more easily and more quickly understandable. Other information that could potentially be “projected” in this way includes lane boundaries, lane departure warnings, nighttime highlighting of pedestrians not readily visible to the driver, and even recommendations for evasive manoeuvres into other lanes, complete with marked-out paths.
Outlook and technical challenges.
The bigger the display area, the more applications contact analogue display technologies can offer. Looking further ahead, the developers are therefore already aiming to increase the size of the displays as much as possible. In the first prototypes, they have already been successful in presenting contact analogue information for the vehicle’s own lane, using an area just four times bigger than that of current head-up displays. In a test environment, they have also already implemented wider display areas, capable of presenting information for a number of lanes. A further consideration when integrating driving-related and location-based information into the external scene is that the virtual images need to be projected at a considerably greater apparent distance from the driver than the 2.20 metres most appropriate for current HUDs. Also, the display area must be moved further up towards the horizon, so that the displays are positioned over the driving situation. One of the main challenges the developers face now is to free up the necessary space and to develop the requisite technological solutions for head-up displays of this kind.
“The issue now is to develop new technological solutions for these large projection areas and to integrate them into the vehicle.” (Gunnar Franz, Head of Head-Up Display Development)
Another challenge as far as contact analogue displays are concerned is to ensure precise matching between the virtual and real worlds. If the projected imagery and the external situation are not in sync, this supplementary information can quickly distract or confuse the driver. The developers are therefore using intelligent sensor fusion methods to ensure good matching and a high-quality display experience. To create the impression of convergence between reality and the displays, the system uses the vehicle’s high-performance environment- recognition and sensing technology such as GPS systems, the front camera and the Active Cruise Control radar, further supplemented by advanced digital road map data including lane-level information. All this data is used to compute and display contact analogue information.
The developers see the contact analogue Head-Up Display as the key to a new dimension in display technology. The two application scenarios described above, in navigation and driver assistance systems, are just the first stages in this process. Other potential scenarios are illustrated by the BMW Vision ConnectedDrive concept car, which features contact analogue representation of numerous different types of information.
Testing in the driving simulator.
Initial testing of new display systems, like the contact analogue Head-Up Display, is carried out mainly in the BMW Group driving simulators. Driving simulator testing is more cost-efficient, and modifications can be made more quickly, than using real vehicle prototypes. Different versions can be quickly implemented and tested under identical conditions, within a short time scale and with large numbers of pilot users – without the hazards of road testing. These are particularly important considerations in view of the amount of verification testing required when developing pioneering new display concepts like the contact analogue Head-Up Display.
The BMW Group driving simulators are an optimal tool for highly realistic testing of complex display processes. Up to seven full-HD projectors provide a high-resolution, sharply focused simulation of the driving environment. This is particularly important for detecting and classifying more distant objects and for identifying the direction of the road. The special projectors also provide highly realistic simulation of moving objects. Fluent and sharp imagery is of great importance when simulating urban driving scenes, where large numbers of objects are represented in great detail, in order to ensure that pilot users have a sound basis for judging driving man oeuvres.
The high-performance projectors jointly generate a 240-degree horizontal and a 45-degree vertical field of view around the driver. The projection of driving scenes onto wall screens alongside the car allows intersections and complex turning manoeuvres to be optimally simulated – and optimally assessed. Drivers know exactly where they are in the real-world environment and can therefore perform precise turning manoeuvres just as they would in reality.
As a tool for testing and evaluating new systems, over the past years the BMW Group has developed and refined a simulator-based reproduction of a complex city driving environment. The results of this work are unique, both in form and in quality. It demands a lot of processing power, of course. For example, an image refresh rate of 60 Hertz – i.e. the traffic scenario is updated 60 times per second – is needed to ensure fluent simulation of heavy traffic volumes. The complex intersections and road network in this simulated city environment now provide an ideal basis for testing lane-level navigation and contact analogue displays.
“The closer our simulation comes to reality, the more meaningful the results. These results are very valuable for developing navigation systems, driver assistance systems and control and display systems.” (Martin Strobl, Head of Driving Simulation at BMW Group Research and Technology)
Needless to say, when evaluating their display concepts the BMW Group development team doesn’t only rely on the highly promising results of simulator testing but also carries out road testing with real prototypes. Results obtained in the simulator are applied to the vehicle, where they undergo further refinement. Both types of testing are indispensable for reliable and robust evaluation of control and display systems.