ORLANDO, Fla. — While graphics, computing speed and even hardware are being improved to make virtual and augmented reality training more realistic, the integration of data science and artificial intelligence will take immersive training to the next level, according to Booz Allen Hamilton consultants in charge of the company’s strategic innovation.
The Interservice/Industry Training, Simulation and Education Conference showroom floor was full of virtual training products, but many are still essentially two-dimensional with a little more instructional design and a little more interaction built in.
But Booz Allen sees an opportunity to take the immersive training environment further.
For most, artificial intelligence is a “cool buzzword,” Young Bang, Booz Allen’s senior vice president of data science, AI and immersive concepts, told Defense News in an interview at I/ITSEC.
“But you don’t really see it, it’s not real to them, not yet,” he said.
The over 100-year-old company, in looking to last another century, has become focused on innovation and advancing technology. As a result, it created its innovations group. The group has more than 800 data scientists — more than Google or Amazon — and has since pioneered data science and AI developments.
And when it comes to the U.S. military, Booz Allen sees a direct application for using data science and AI in immersive training, according to Munjeet Singh, the vice president of the Strategic Innovation Group and co-lead of immersive concepts.
Booz Allen started with a project to take dense operational manuals and basic training materials and create more digestible, engaging training. Over the past six years, that effort has morphed from reformatting materials to more immersive experiences.
“One of the big reasons for that, quite frankly, it’s the millennial generation that is starting to come into the U.S. forces and they don’t react well to a 500-page manual being slapped on their desk,” Singh said. “They want an app or a 2-D game.”
Booz Allen hired talent from Nintendo, Disney Digital Studio Services and others in the gaming field — people with no experience in government contracting — who could create engaging training materials in innovative ways, Singh said.
Booz Allen has three augmented reality/virtual reality studios in Seattle, Washington; Austin, Texas; and McLean, Virginia. Seattle and Austin are the two top cities in the country for gaming talent.
The company has about 400 game designers, developers and artists that make up the practice.
Making it look good
Booz Allen has focused on making its immersive training environments as real as possible.
For example, its C-130 gunship trainer is the result of merging thousands of pictures from the inside of an actual C-130, digitally recreated for a headset. The experience “feels so real to the point where when you open up the door, we like to give people a little test and see if they want to step out of the plane and literally like 60 percent of the people just refuse to do it,” Singh said.
The company is also working with the U.S. Navy in the Pacific theater on an immersive experience called OceanLens where the sea floor is digitally recreated using real bathymetric and typographic data, which allows a user to float around in the ocean like a submarine.
Singh said the utility there is the ability to accurately plot out courses and experience major obstacles, not just from the surface but below. “Unless you are actually in the sub tracking someone on the surface in the map, you won’t be able to see there is a ridge” that could block the view when looking from below. The OceanLens experience can provide more accurate information for planning.
And taking the ability to deliver convincing and accurate visuals, Bang said, there is great utility in training across the military. For example: jumpmaster training at Fort Bragg, North Carolina. Bang himself has gone through such training there.
Booz Allen can deliver a real-time picture of the ground at Fort Bragg from the air using light detection and ranging data so someone training to jump can go through all the motions before going up in an actual plane.
“When they go into the airplanes to actually take their tests for jumpmaster, they will already know the terrain, they will know the markers and they will actually have a higher first-time pass rate,” Bang said.
A quick study
Booz Allen is now converging data science and machine intelligence into an already visually striking experience.
Singh said the project he is most proud of is a program that takes 50- to 100-page operation orders — which are given to a soldier when receiving deployment orders — and ingesting it into an immersive environment.
“We use natural language processing and some algorithms that we have created to pore through that data and basically summarize it into a series of mini-booklets that in the immersive experience you can pick up, you can look at what equipment is going to be needed, you can put that away, you can look at what the geography [is] and then put that away,” he said.
The books can fade into the background of the immersive training environment and an actual model of the operational area will pop up, Singh explained. And “you can start war gaming where you are going to move your battalions, where are you going to move your equipment, what the opposing force will do,” he added.
The program, called Tabletop Commander, uses a simple Samsung Odyssey headset and was created in three weeks. A full operations order can be processed into the program using AI in a matter of seconds. And the immersive experience is visually pleasing, realistic and accurate, according to Mike Wikan, creative director at Booz Allen, whose past experience includes work on the famed Nintendo game Donkey Kong.
Gaming the game
AI and data science can also contribute to generating countless scenarios in a virtual training environment, Bang said. This prevents the ability to “game the game,” he added. For instance, Bang said, when learning to play PacMan, “I didn’t learn to play the game, I learned to memorize the screen.”
Right now the cost to change the scenarios up in a training game is “astronomical” because the whole game has to be reprogrammed differently to take predictability out of it, Bang said.
Using AI, “we are actually creating dynamic actions, not programming logic into a character,” he said.
The capability “gives you the ability to have artificial intelligence change the scenario dynamically and start training the soldier on unintended consequences, surprises they might actually see in the battlefield,” Bang said.
With that, “we are just really beginning to touch the tip of the iceberg on how we do dynamic content generation, but we are also looking at how to actually use artificial intelligence to generate code for this whole immersive experience as well [as] which will revolutionize this whole gaming experience,” he added.
According to Singh, the industry is roughly two to three years out from mainstream adoption of a lot of data science and AI integration into immersive training, and “there are still so many barriers right now.”
But there are also a lot of Booz Allen clients who are willing “to push the envelope,” he added.