U.S. Air Force: Project RISE

Next, we wanted to see how different instructors ran these sessions so we could recognize key factors and patterns in flight training. From this intensive observation period we were able to distill out mission-critical controls to carry over into our experience as well as identifying gaps that might be introduced in a virtual instruction session by not having an instructor physically in the same space as the student. Through wireframes and iterative review with AETC, we were able to solidify the toolkit that we needed to offer within the Apple Vision Pro prototype.

Building off our wireframes, our Creative Team began the work of designing the interface within the AR space. Understanding the intended audience and the purpose of this interface lead us to propose a range of clean and focused UI Designs that always kept the mission at the forefront. We proposed our usual "mild to wild" (or in this case Alpha to Delta) that kept the interface minal while offering variations in the overall framing styles, future-forward glows, and widget styling. Overall the AETC team chose our Delta concept, which was very much inspired by the simple, clean style of Apple's existing design language.

A combination of sharp edges, fine lines, and tasteful monochromatic looks is what sets this visual style apart from others. It can sometimes feel drab, but the idea is that the UI yields to the dense information it's delivering.

Our second look builds on the first and embodies a quintessentially “future UI” aesthetic. It uses layering, opacity, dust, and grain to create an immersive space—one that feels as though it was designed by humans operating with a far greater portion of their cognitive capacity than we use today. Hard edges are present once again, but the design often incorporates a few softer elements to balance the look and introduce a more natural, human feel.

Everyone wants their own Iron Man, their own JARVIS, and a chance to play in the MCU. These interfaces are bold and colorful, often styled to match the vibrant costumes of the characters who use them. We see heavy use of 3D extrusion, transparency, blurring, and other design elements Tony Stark himself would include—resulting in visuals that feel like the coolest thing you’ve seen since your first Lamborghini.

Rethinking the idea of "settling" for an Apple-esque interface, this treatment embraces minimalist communication to guide the user toward intuitive understanding. It merges the timeless design philosophy of the GOAT, Dieter Rams, with classic military encyclopedia aesthetics to create something that feels both enduring and highly functional.

The existing simulator was a mix of spartan fighter-jet style graphics and the legacy cockpit simulator readouts that are built more for function over form. Our team looked at each element and designed all of the screens, flight telemetry, and controls used in the simulator to sit in the AVP environment. It was important that we mimicked the layout and format of the existing telemetry and controls so that the interface was familiar to the instructor pilots who had been flying with these for 15+ years, but we knew there was opportunity to modernize these to some degree in the chosen style. Our UI Design team had fun crafting an elegant and futuristic interface with no superfluous pixels.

Beyond the widgets and telemetry, the interface also incorporates a few additional elements to help solve for the virtual vs. physical instructor in the room as well as make use of the Apple Vision Pro's AR capabilities. First, we included a video feed from the VR flight environment that the student was seeing so the instructor could monitor their gaze and head position. We also included a feed from a camera placed over the student's shoulder in the simulator. This allowed the instructor to see where the student was reaching and the stick angle within the cockpit and how they were reacting to the simulated scenarios.

Finally, we designed a 3D representation of the flight path and ground terrain that could be placed in the instructor's "room space" so they could monitor altitude and distance from geographic features and ground targets.

The 3D map design serves as a dynamic and intuitive visual interface that enhances spatial awareness and navigation. Built with precision and attention to user experience, the map integrates real-time data and high-fidelity rendering to provide a seamless, interactive environment. Its layered structure and adaptive scaling allow users to explore complex terrains, maneuvers and other important data streams.

The system is powered by a robust backend architecture, featuring the NOR military simulation platform—developed by Metrea and built on Unreal Engine—which delivers high-fidelity training scenarios. Supporting infrastructure includes Node.js and Express.js web servers that manage data and simulator command APIs, along with WebRTC services for streaming live webcam feeds, cockpit instrument data, and two-way audio communication.

The instructor application is developed using visionOS, integrating SwiftUI, MapKit, and RealityKit to deliver an immersive and interactive experience. Instructors are presented with real-time visualizations of cockpit instrument panels, live video feeds, and a dynamic 3D representation of aircraft movement. This is made possible by leveraging telemetry data to render the aircraft’s position and trajectory across geographic regions, providing a comprehensive situational awareness tool for pilot training and evaluation.

We were honored to support the U.S. Air Force in addressing its shortage of on-site instructors. Although this project was executed over several short sprints to rapidly deliver a prototype, we're proud to have successfully integrated multiple APIs from various sources into a seamless experience within the AVP. This is only the beginning—we've just scratched the surface of what this powerful application can offer. We look forward to completing the full build in the coming months and realizing our creative vision for the next generation of flight simulation and instruction.