Jon Lawson reports on the latest developments in automating mining vehicles.
Fully automating the load and haul sector could offer huge advantages for mine operators with cost savings as well as improvements in safety and emissions.
Volvo CE is investing heavily in new technology here, and last year completed a successful project at Skanska’s Vikan Kross cement quarry near Gothenburg, Sweden. Volvo’s Ulrich Faß, manager Emerging Technologies, explains what was learned from this: “It was very beneficial to get a customer on board early to test concepts. The ‘Concept Lab’ idea with selected customers to see what is feasible is very different to how machines used to be developed in the past. Just developing products and then showing customers very late on won’t work with the rapid pace of change we have today. It’s an iterative process and we need regular feedback from customers to keep development moving quickly and in the right direction.”
Fast-forward to today and there is an ongoing project at Eskilstuna with Telia and Ericsson, where the initial designs have been handed over to the product development team to determine how to commercialise what has been learned so far.
The HX2 Machine
The machine at the heart of the research is the HX2. Faß explains, “HX2 was a proof of concept – much of what we learned will find its way into commercial products in the future. We need to simplify, so instead of merely using the latest tech we wanted to start using commercial tech that’s more widely available and cheaper. The car industry’s autonomous research will help to make componentry more affordable.
“Construction and mining is at the forefront with the automotive industry when it comes to autonomy – off road equipment is leading in some domains and benefits from the economics of scale in the car and truck industry. Mines and construction sites are not part of the public space – they have used these sensors for many years to cope with their environment – so our industry has seen semi-autonomous and autonomous machines earlier than with passenger cars. That said, the variables are relatively uncomplicated compared to automotive. But progress is super rapid. Years ago, it was single layer lidar – now it’s 64.”
Sensor technology is key to successful automation. “Sensors and computers are the heart of the machine,” confirms Faß. “To be safe around humans, lidars, radars and ultra sonics with cameras and GPS are needed. When humans are not around, then we could use fewer of these sensors. As the electric site was a confined area, with limited access we reduced the amount of sensors to just lidars and radars for the HX2.”
Safety In Automation
Safety is always uppermost in the research. Faß says, “There’s a lot we can do here. We are working towards zero accidents in the future. We’re doing a lot of research with insurance companies and end users to get proper statistics on what happened to cause an accident. Automation will ultimately bring us the freedom of full awareness during an entire shift. No operator will get tired and careless by for example operating a vibrating machine. This builds trust.”
Are People-Free Mines The Future?
Getting humans completely out of the picture is the goal, and Volvo’s customers are already talking about totally people-free underground mines that one day will run 24/7 for 365 days a year.
Volvo is really pushing the boundaries with the research. Faß says, “We are working at the limit of what’s possible - of course - we always will be. 100 megapixel cameras exist today - and with good quality. Price is an issue though; they are too costly. Sadly the compression of the data is not good enough yet. Raw data transfer is not good enough either; we need to improve, but it is happening.”
So what is the next step? Faß thinks resolution will get better, with high dynamic range (HDR) technology coming to the fore. “When we talk vision now, we talk about an RGB camera with a seamless sensor. HDR offers the ability to capture multiple frames and merge them together. It mimics the way the eye sees things – normal cameras can’t do that. HDR is standard in mobile phones now and we need to use it. And fuse lidar with RGB data – we want manufacturers to take the lead with this.
“Also we need to take into account the wider environment, so stationary cameras and drone-based cameras will be useful. Don’t just rely on what’s in the machines – but build static cameras and mobile cameras outside the machine. Because of this external vision, being able to see around corners becomes possible. And don’t forget infrared – night vision is almost as good as day vision now. The resolution and price are going up and down respectively. It’s astonishing.”
The beauty of the HX2 project is what Volvo is learning about adapting to the specifics of obstacles in different environments. “So for mining,” thinks Faß, “when underground it’s harsh, with tight turns, narrow spaces and it’s often wet and dark. Above ground in quarries you have snow, rain, dust and sometimes elevation to contend with. Also there is a desire for speed.
“If we broaden this out, we can learn lessons about what needs to be done for other environments, so in forestry we need accurate positioning and for agriculture it’s slippery and we need to avoid running over the crops. The trend is for smaller machines to be used in a fleet, which offers more redundancy and is easier to handle.”
Interpreting The Data
All that sensor data needs interpretation. Torbjörn Martinsson is a research engineer in Volvo’s Emerging Technologies Team, Sweden. Of processing power, he says, “Machine learning is very popular and has a high resource demand – many teraflops/second. This is moving very fast. 100 teraflops are available now – and it’s affordable. Silicon manufacturers will have a mammal’s thinking capability within a couple of years (neuromorphic processing).”
So which is better, having the brain on-board or acting separately, joined by a grid? Martinsson observes, “5G and networking gives us the possibility to be grid-based, instead of onboard. But it’s too vulnerable for functional safety making it impossible to guarantee 100% safety. So, the mix will be very important. We want grid sharing of data but the on-board safety aspect will be crucial for redundancy if the grid fails. We may change our view on this in the future.
“5G seems like the technology we need, the research shows us this is enough, and Sweden is leading with this. It offers a very small latency, the data moves quickly. Data needs to arrive in the right order and be stable, fast and predictable, which is essential for remote control.
“WLAN should be a good alternative to 5G. TSN is a comms tech and this looks very interesting. TSN I (a time sensitive network) means you know exactly when you are going to get things with microseconds to get data. There are several interesting possibilities, it’s all being researched at the moment.”
Where does AI fit into this?
Martinsson says, ‘It’s a very wide spectrum, from a simple driver controller, to act like human intelligence (how do I do something?) to the most complex. We will use AI on several levels from the very simple to bashing big data. Machine learning is part of AI – it’s just much bigger.
“We have proved we can have an autonomous machine that digs on its own, and learns how to do it better. Reinforced learning will become an important factor, but we need to ensure it is safe. You need the software to understand if it is doing a good job; this is immature now. We need to define what ‘good’ looks like.”
Electrification Is Crucial
Martinsson sees electrification as crucial for the growth in automation. “An electric drive is better for autonomy as there are fewer moving parts. There is industry-wide interest in solid-state technology for the powerpacks. We’re looking into this. It’s promising but still early. With charging, it’s moving rapidly, and with regeneration it is easier to optimise with autonomous – it’s harder with humans and as we don’t want to disturb the operator in their work. Autonomous machines don’t mind.
“Maintenance is also easier. An automated machine knows what it’s done, how it’s done it and what it must do, and maintenance is predictable. Humans use machines in a non-deterministic way: machines use machines predictably.”
Caterpillar’s Involvement With Autonomy Research
Caterpillar has been involved with autonomy research for decades. Michael Murphy, chief engineer, Mining Technology, is a member of Caterpillar’s original team that developed autonomous mining vehicles, and he offers a quick history lesson: “We started collaborating with Carnegie Mellon University more than 30 years ago, working on GPS for off-road machines – when there were very few satellites in the sky. We knew that this technology would be key to developing autonomous machine capabilities. In fact, we showed our first use of high-precision GPS on our Terrain product at MINExpo in 1996, 23 years ago. We also broadcast live to MINExpo 1996 an autonomous truck operating at our Tucson Proving Grounds.
“At the time, however, the mining industry was not ready for autonomous machines, and the technology needed further development. In 1998 we went to work creating the building blocks - the core technologies -needed for autonomy. We launched Terrain to give mining operators productivity information for shovels, drills, wheel loaders and dozers. And we continued to build on monitoring machine health off-board with Cat VIMS, a key building block for autonomy. With no operator onboard, we needed to be able to remotely monitor and understand any problems developing with machines.”
The DARPA (Defense Advanced Research Projects Agency) Challenges provided a step change for autonomy. “We partnered with Carnegie Mellon University to sponsor the award-winning ‘Boss’, an autonomous Chevrolet Tahoe, that won first place in the 2007 DARPA Urban Challenge,” Murphy said. “We were the first company to leverage the technology from the DARPA Challenges. Since then, we believe our machines have travelled twice as many miles as any automotive company’s autonomous vehicles.”
The company’s record is indeed impressive. Currently there are over 225 autonomous trucks in operation and they have achieved over 1.5 billion tonnes hauled over 50 million kilometres with zero lost time from injuries.
Elaborating on the technology, Gary Cook, commercial manager, Surface Autonomy, MineStar Solutions, explains, “The trucks have a similar technology stack to what you see on autonomous cars, but built for durability in a tougher environment, and with additional safety systems.
“Our key advantage is our use of spinning lidar to detect objects. Using lidar is like driving a car with high-beam headlights – where you get improved visibility allowing for traveling at higher speeds safely and with confidence. This strong perception technology allows us to run the truck at maximum speed when conditions allow.”
Cat uses multiple positioning systems (GNSS and an inertial measurement unit) for accuracy. If a satellite signal is lost, autonomous trucks can continue to run for a short period of time.
The firm’s philosophy is to have what Cook calls “advanced processing power and decision-making capability, which enables more decisions to be made onboard.” Cloud computing handles non-critical analysis that is not time sensitive, and deeper analytical analysis can be done offboard while events are tracked in the system.
The company has been operating underground for 15 years. Cook thinks productivity improvements are more common with autonomous underground operation. “Machines are more productive, with improved accuracy of tunnel navigation and reduced downtime due to damage from contact with drive walls. And mines that incorporate automation are more productive with fewer people underground. There’s no need for workers to evacuate for blasting or travel long distances to and from the surface.”
Automation In the Real World
The plan is to take the learnings from autonomy into manned machines. The work on collision avoidance is one example. “Collision avoidance is an emerging technology,” says Cook. “Preventing machine movement when conditions dictate are key features coming.”
Cook notes, “Over the past 20 plus years, we’ve continued to leverage improvements in communication technology. We view 5G as a technology well suited for congested, urban environments. It will have widespread use in consumer and large city settings, where large volumes of data are required for transfer, but that doesn’t necessarily make it suitable for open pit, surface mining at this time given its short transmission distances. As the technology evolves, we’ll continue to monitor how it can be leveraged for improvements in our solutions.”
Behind the scenes Cat has used VR/AR technologies in the design and manufacture of its machines and associated technologies for more than 10 years. Hardware-in-the-loop and software-in the-loop testing achieves a more efficient product development process.
Cook explains, “We use knowledge gained from this testing to apply to new models. Moving on from simulation, we have facilities where we install the system on small equipment and automobiles for early field testing. Next, we install the system on actual equipment at our proving grounds where we test in a mining environment running double shifts. All this happens before we ever install at a customer site.”
The Argument For Semi-Autonomous Vehicles
As well as the purely autonomous creations, the company is also interested in leveraging this new tech for semi-autonomous and normal driven vehicles. Cook continues, “We have multiple options for dozers ranging from remote control to semi-autonomous operation where a single operator controls four to five machines. Again, allowing the operator to remotely operate the machine in a safe environment is key. Semi-autonomous systems require additional communication bandwidth due to video but can serve as an ideal solution for maximising dozer push operations.
“It’s dependent on what issue the customer is trying to solve. Machine recovery, unstable footing, aggressive ripping, misfires, stockpiles, bench slides and steep slopes are ideal operations for remote control dozer operation. If the customer is trying to remove process variability, autonomy may be the answer.”
To aid the driver, Cat employs a fatigue detection technology, which works by monitoring eye-closure duration and head pose.
JCB has taken the ‘use-the-tech-to-help-the-driver’ route instead of autonomy and has just announced an expansion in its Uptime Centre scheme, with the latest office opening in Singapore to cover the South East Asia region. Data is collected with LiveLink monitoring and predictive algorithms examine whether there are any likely upcoming maintenance issues. In addition, the company has created a new app for operators to perform tasks such as making the pre-start machine check process electronic and allowing operators to leave comments. Data such as idle time and fuel consumption can be more easily accessed by owners.
So as the technology continues to evolve, operators can look forward to gradual incremental improvements leading to safer, cleaner mines.
