Off-highway leading the way in autonomous driving

Jon Lawson

There is much talk in the media about driverless cars, but when it comes to autonomous vehicles, the off-highway, construction and mining industries have been working quietly on the technology for decades.

Volvo CE investing in autonomous vehicle development

Volvo CE is investing heavily in autonomous technology, and last year completed a successful project at Skanska’s Vikan Kross cement quarry near Gothenburg, Sweden. Volvo’s Ulrich Faß, manager Emerging Technologies, explains what was learned from this: “It was very beneficial to get a customer on board early to test concepts. The ‘Concept Lab’ idea with selected customers to see what is feasible is very different to how machines used to be developed in the past. Just developing products and then showing customers very late on won’t work with the rapid pace of change we have today. It’s an iterative process and we need regular feedback from customers to keep development moving quickly and in the right direction.”

Commercialising autonomous vehicles

Fast-forward to today and there is an ongoing project at Eskilstuna with Telia and Ericsson, where the initial designs have been handed over to the product development team to determine how to commercialise what has been learned so far.

The machine at the heart of the research is the HX2. Faß explains, “HX2 was a proof of concept – much of what we learned will find its way into commercial products in the future. We need to simplify, so instead of merely using the latest tech, we wanted to start using commercial tech that’s more widely available and cheaper. The car industry’s autonomous research will help to make componentry more affordable.”

Autonomous vehicles in construction and mining

“Construction and mining is at the forefront with the automotive industry when it comes to autonomy – off-road equipment is leading in some domains and benefits from the economics of scale in the car and truck industry. Mines and construction sites are not part of the public space – they have used these sensors for many years to cope with their environment – so our industry has seen semi-autonomous and autonomous machines earlier than with passenger cars. That said, the variables are relatively uncomplicated compared to automotive. But progress is super rapid. Years ago, it was single-layer lidar – now it’s 64.”

Sensor technology key to successful automation

“Sensors and computers are the heart of the machine,” confirms Faß. “To be safe around humans, lidars, radars and ultrasonics with cameras and GPS are needed. When humans are not around, then we could use fewer of these sensors. As the electric site was a connected area, with limited access we reduced the amount of sensors to just lidars and radars for the HX2.”

Safety is always uppermost in the research. Faß says, “There’s a lot we can do here. We are working towards zero accidents in the future. We’re doing a lot of research with insurance companies and end users to get proper statistics on what happened to cause an accident. Automation will ultimately bring us the freedom of full awareness during an entire shift. No operator will get tired and careless by, for example, operating a vibrating machine. This builds trust.”

Autonomous mining vehicles

In mining, getting humans completely out of the picture is the goal. Volvo’s customers are already talking about people-free underground mines that one day will run 24/7, 365 days a year.
Volvo is really pushing the boundaries with the research. Faß says, “We are working at the limit of what’s possible - of course - we always will be. 100MP cameras exist today - and with good quality. Price is an issue though; they are too costly. Sadly the compression of the data is not good enough yet. Raw data transfer is not good enough either; we need to improve, but it is happening.”

What’s next in autonomous driving?

So what is the next step? Faß thinks resolution will get better, with high dynamic range (HDR) technology coming to the fore. “When we talk vision now, we talk about an RGB camera with a seamless sensor. HDR offers the ability to capture multiple frames and merge them together. It mimics the way the eye sees things – normal cameras can’t do that. HDR is standard in mobile phones now and we need to use it. And fuse lidar with RGB data – we want manufacturers to take the lead with this.”

“Also we need to take into account the wider environment, so stationary cameras and drone-based cameras will be useful. Don’t just rely on what’s in the machines – but build static cameras and mobile cameras outside the machine. Because of this external vision, being able to see around corners becomes possible. And don’t forget infrared – night vision is almost as good as day vision now. The resolution and price are going up and down respectively. It’s astonishing.”

The beauty of the HX2 project can be seen in what Volvo is learning about adapting to the specifics of obstacles in different environments. “So for mining,” thinks Faß, “when underground it’s harsh, with tight turns, narrow spaces and it’s often wet and dark. Above ground – in quarries, for instance – you have snow, rain, dust and sometimes elevation to contend with. Also, there is a desire for speed.”

“If we broaden this out, we can learn lessons about what needs to be done for other environments, so in forestry we need accurate positioning and for agriculture it’s slippery and we need to avoid running over the crops. The trend is for smaller machines to be used in a fleet, which offers more redundancy and is easier to handle.”

Vehicle automation and machine learning

All that sensor data needs interpretation. Torbjörn Martinsson is a research engineer in Volvo’s Emerging Technologies Team, Sweden. Of processing power, he says, “Machine learning is very popular and has a high resource demand – many tera ops/second. It is moving very fast. 100 tera ops are available now – and it’s affordable. Silicon manufacturers will have a mammal’s thinking capability within a couple of years (neuromorphic processing).”

So which is better, having the brain onboard or acting separately, joined by a grid? Martinsson observes, “5G and networking gives us the possibility to be grid-based, instead of onboard. But it’s too vulnerable for functional safety making it impossible to guarantee 100% safety. So, the mix will be very important. We want grid sharing of data but the onboard safety aspect will be crucial for redundancy if the grid fails. We may change our view on this in the future.

“5G seems like the technology we need, the research shows us this is enough, and Sweden is leading with this. It offers a very small latency, the data moves quickly. Data needs to arrive in the right order and be stable, fast and predictable, which is essential for remote control. WLAN should be a good alternative to 5G. TSN (time-sensitive network) is a comms tech and this looks very interesting. TSN means you know exactly when you are going to get things with microseconds to get data. There are several interesting possibilities, it’s all being researched at the moment.”

Where does AI fit into this? Martinsson says, “It’s a very wide spectrum, from a simple driver controller to act like human intelligence (how do I do something?), to the most complex. We will use AI on several levels from the very simple to bashing big data. Machine learning is part of AI – it’s just much bigger.”

“We have proved we can have an autonomous machine that digs on its own, and learns how to do it better. Reinforced learning will become an important factor, but we need to ensure it is safe. You need the software to understand if it is doing a good job; this is immature now. We need to define what ‘good’ looks like.”

Off-highway electric vehicles

Martinsson sees electrification as crucial for the growth in automation. “An electric drive is better for autonomy as there are fewer moving parts. There is industry-wide interest in solid-state technology for the powerpacks. We’re looking into this. It’s promising but still early. With charging, it’s moving rapidly, and with regeneration it is easier to optimise with autonomous – it’s harder with humans as we don’t want to disturb the operators in their work. Autonomous machines don’t mind.”

“Maintenance is also easier. An automated machine knows what it’s done, how it’s done it and what it must do, and maintenance is predictable. Humans use machines in a non-deterministic way: machines use machines predictably.”

Autonomous mining vehicles on track with Caterpillar

Caterpillar has been involved with autonomy research for decades. Michael Murphy, chief engineer, Mining Technology, is a member of Caterpillar’s original team that developed autonomous mining vehicles, and he offers a quick history lesson: “We started collaborating with Carnegie Mellon University more than 30 years ago, working on GPS for off-road machines – when there were very few satellites in the sky. We knew that this technology would be key to developing autonomous machine capabilities. In fact, we showed our first use of high-precision GPS on our Terrain product at MINExpo in 1996. We also broadcast live to an autonomous truck operating at our Tucson Proving Grounds.

“At the time, however, the mining industry was not ready for autonomous machines, and the technology needed further development. In 1998 we went to work creating the building blocks - the core technologies - needed for autonomy. We launched Terrain to give mining operators productivity information for shovels, drills, wheel loaders and dozers. And we continued to build on monitoring machine health onboard with Cat VIMS, a key building block for autonomy. With no operator onboard, we needed to be able to remotely monitor and understand any problems developing with machines.”

The DARPA challenge

The DARPA (Defence Advanced Research Projects Agency) Challenges provided a step-change for autonomy. “We partnered with Carnegie Mellon to sponsor the award-winning ‘Boss’, an autonomous Chevrolet Tahoe, that won first place in the 2007 DARPA Urban Challenge,” Murphy says. “We were the first company to leverage the technology from the DARPA Challenges. Since then, we believe our machines have travelled twice as many miles as any automotive company’s autonomous vehicles.”

The company’s record is indeed impressive. Currently, there are over 225 autonomous trucks in operation and they have achieved over 1.5 billion tonnes hauled over 50 million kms with zero lost time stemming from injuries.

A closer look at the tech

Elaborating on the technology, Gary Cook, commercial manager, Surface Autonomy, MineStar Solutions, explains, “The trucks have a similar technology stack to what you see on autonomous cars but built for durability in a tougher environment and with additional safety systems.

“Our key advantage is our use of spinning lidar to detect objects. Using lidar is like driving a car with high-beam headlights – where you get improved visibility allowing for travelling at higher speeds safely and with confidence. This strong perception technology allows us to run the truck at maximum speed when conditions allow.”

Cat uses multiple positioning systems (GNSS and an inertial measurement unit) for accuracy. If a satellite signal is lost, autonomous trucks can continue to run for a short period of time.

Taking autonomous driving to a new level

The firm’s philosophy is to have what Cook calls “advanced processing power and decision-making capability, which enables more decisions to be made onboard.” Cloud computing handles non-critical analysis that is not time sensitive, and deeper analytical analysis can be done onboard while events are tracked in the system.

The company has been operating underground for 15 years. Cook thinks productivity improvements are more common with autonomous underground operations. “Machines are more productive, with improved accuracy of tunnel navigation and reduced downtime due to damage from contact with drive walls. And mines that incorporate automation are more productive with fewer people underground. There’s no need for workers to evacuate for blasting or travel long distances to and from the surface.”

The plan is to take the learnings from autonomy into manned machines. The work on collision avoidance is one example. “Collision avoidance is an emerging technology,” says Cook. “Preventing machine movement when conditions dictate are key features coming.”

Leveraging communication technology in autonomous vehicles

Cook notes, “Over the past 20 plus years, we’ve continued to leverage improvements in communication technology. We view 5G as a technology well suited for congested, urban environments. It will have widespread use in consumer and large city settings, where large volumes of data are required for transfer, but that doesn’t necessarily make it suitable for open pit, surface mining at this time given its short transmission distances. As the technology evolves, we’ll continue to monitor how it can be leveraged for improvements in our solutions.”

Behind the scenes, Cat has used VR/AR technologies in the design and manufacture of its machines and associated technologies for more than 10 years. Hardware-in-the-loop and software-in-the-loop testing achieve a more efficient product development process.

Cook explains, “We use knowledge gained from this testing to apply to new models. Moving on from simulation, we have facilities where we install the system on small equipment and automobiles for early field testing. Next, we install the system on actual equipment at our proving grounds where we test in a mining environment running double shifts. All of this happens before we ever install at a customer site.”

As well as the purely autonomous creations, the company is also interested in leveraging this new tech for semi-autonomous and normal driven vehicles. Cook continues, “We have multiple options for dozers ranging from remote control to semi-autonomous operation where a single operator controls four to five machines. Again, allowing the operator to remotely operate the machine in a safe environment is key. Semi-autonomous systems require additional communication bandwidth due to video but can serve as an ideal solution for maximising dozer push operations.”

“It’s dependent on what issue the customer is trying to solve. Machine recovery, unstable footing, aggressive ripping, stockpiles, bench slides and steep slopes are ideal operations for remote control dozer operation. If the customer is trying to remove process variability, autonomy may be the answer.”