zSpace’s Maya plugin
zSpace provides an immersive, interactive, multi-sensory spatial computing experience. Users view virtual objects in 3D through lightweight glasses, examine them in detail from all angles, and manipulate them using a stylus pen. Since the system continuously tracks the zSpace eyewear and knows where the user's head is, the user can view and look around virtual 3d objects. The 6DOF stylus, held like a pen in mid-air, is used to select and manipulate objects in 3d.
Platforms: Desktop VR
Disciplines: Design Strategy, Product Design, Spatial Computing Design, User Research, Usability, Design Thinking, Information Architecture, Wireframing, Prototyping, Design Systems
Tools: Unity, Omnigraffle, Photoshop, Keynote, Maya
Problem Definition - Products, not Toys
To convey zSpace's vision and promise to prospective investors, we created demos highlighting the opportunities provided by the zSpace technology stack. They showcased the unique combination of zSpace's stereoscopic display, head tracking, and 6DOF interaction capabilities.
Hired in June of 2010, I was zSpace's first traditional Experience Designer. I grew the design team to eight, including 3d technical artists, 2d visual designers, and researchers. As Design Lead, I drove the design strategy for the demos, which generated over $60M in VC investment.
These compelling examples of zSpace generated a lot of jaw-dropping moments for users. But, the demos were still simple toys. We had to create a product providing real customer value to justify our VC's investment.
Business Direction - 3d Artists
To create a product, we had core questions to answer. We had to identify a market segment that could bear our product's price point. Second, we needed a user problem that zSpace could solve elegantly and efficiently.
zSpace had explored markets, including mechanical engineering, medical visualization, automotive design, and video games. We decided to focus on 3d modelers and animators. Their work often appears in movies, video games, and commercials. Using specialized software, 3d modelers create the shape and appearance of CGI objects such as household objects, animals, or people. 3d animators embed extra data in the 3d models that describe how models appear over time when moving.
Many specialized tools already existed for these 3d professionals. The most direct path for zSpace was to leverage an existing solution rather than invent a new one. Pixologic's zBrush and Autodesk's Maya, 3ds Max, and Mudbox were some of the most popular of these tools.
We worked with our industry contacts at Autodesk and ILM to choose a 3d software tool to enhance with zSpace. Mudbox or zBrush was our initial target. But, technical and business hurdles made these two products unsatisfactory options. The second option was Maya. It was an industry-standard tool for 3d modelers and animators. Maya's mature plugin architecture provided the features we needed. Our Autodesk contacts could direct us to technical support for roadblocks we might encounter. We would enhance Maya with zSpace.
Evaluation Criteria
With a target user base and software tool identified, we created high-level evaluation criteria for how zSpace+Maya would provide tangible value. We decided that adding zSpace's capabilities to Maya would:
Make modeling and animating a more intuitive experience
Create a more immersive modeling and animating experience
Result in higher-quality models and animations
Make modeling and animating content faster
Identify User Value and Define MVP
After deciding to enhance Maya for 3d artists, I conducted a generative research investigation. The goal was to identify the specific Maya workflows and features to augment with zSpace. While 3d artists completed simple, everyday modeling and animating tasks, software recorded their activity. I analyzed this mouse and keyboard data to learn which Maya features were used most often during these specific workflows.
We learned an incredible amount from this investigation. Dozens of atomic interaction were captured and categorized. The most common category of actions by far was hovering, selecting, moving, rotating, or scaling an object, face, edge, vertex, joint, or curve point. The next most common operations were interactions with 2d UI such as menus, buttons, text input, palettes, and color pickers.
Design, product management, and engineering evaluated these features. We considered each feature's importance and the challenges to design and implement them. In some cases, quick experiments informed our estimates for the challenges presented. Creating zSpace-powered equivalents of Maya's tools for hovering, selecting, moving, rotating, or scaling of core Maya objects became our MVP.
The MVP included zSpace-specific options like volumetric selection. In a 2d GUI, a user can "drag-select" a group of objects by pressing the left mouse button and dragging to create a rectangle that selects everything it touches. In 3d applications where the Z-axis is relevant, the drag-select operation handles Z by assuming the 2d rectangle extends infinitely in Z. With zSpace's 6DOF stylus, the user can create a 3d box that selects anything it touches.
Using zSpace-powered Maya, participants completed class exercises they created to teach students specific tasks. Then, in groups of three-four people, we used design thinking methods to identify the pros, cons, and future opportunities for our product.
Participants provided vital insights to fine-tune our design and prioritization of feature enhancements. Without prompting, they suggested that zSpace capabilities would shine in Mudbox, zBrush, or Unity. On the positive side, they loved the experience's immersive and intuitive nature. For the future, they believed our efforts should include:
Greater stylus tracking precision,
Enhanced ease in interacting with existing 2d UI,
Better emulation of current Maya functionality in zSpace-enabled modes, and
More original 3d UI, including 3d context menus.
Enhancements and Second Usability Study
The team evaluated the research results and prioritized our efforts. Working with the hardware team, we increased the responsiveness and precision of the stylus. These efforts provided some relief for interacting with Maya's existing 2d UI. We fine-tuned many rough edges in Maya's zSpace-enabled features. Original 3d UI was designed but deprioritized due to implementation complexity.
As our work created a more usable and valuable experience, I planned a second usability study. We used nine participants with varying experience in 3d modeling and animation. This study confirmed previous findings. It also exposed areas needing more significant effort to create a product with distinct business value. The positive results included:
The learning curve is very manageable,
Inspecting models in zSpace is much more immersive and useful than standard Maya,
Core tools work well and provide more functionality than the standard Maya tools, and
The enhanced 2d UI work, 3d context menus, and 3d volumetric selection were invaluable.
The negative findings included:
The current zSpace-powered tools needed more enhancements
More tools needed to be zSpace-enabled,
Extreme zooming causes eye strain, and
Task completion was possible, but completing tasks took 3x-8x longer than standard Maya
Final Product Decision
zSpace's Maya plugin generated exuberant reactions like "That is beyond bitchin!" and "I love looking at this thing!" Experts lauded its value for inspecting 3d models and animations. In doing so, it met three of our evaluation criteria for success:
Make modeling and animating a more intuitive experience
Create a more immersive modeling and animating experience
Result in higher-quality models and animations
But, since tasks took 3x-8x longer to complete, it failed to "make modeling and animating content faster."
While we had zSpace-enhanced the most common interactions for modelers and animators, we hadn't enhanced them all. The slowdowns occurred during the transition to and from standard Maya tools. Complete Maya workflows would need to be zSpace-enhanced to overcome these slowdowns.
The product team determined that making a valuable zSpace Maya plugin would be too significant an investment. We used the plugin as an example of zSpace's capabilities inside design tools.
3d UI and Tools
Before and during the development of zSpace's Maya plugin, there were advances in zSpace-centric UI components and interaction techniques. Here, I will discuss:
the "Déjà Vu" approach to stylus selection challenges
3d Menus
3d Color and Material Pickers
Creating a camera path using the spline tool
zSpace's "light+glass" look&feel
Déjà Vu
Our usability participants had consistent difficulty using the stylus to select 2d UI such as buttons and menus. As one subject put it, "I really have to aim.” Maya's UI, designed for keyboard and mouse interactions, had relatively small hit targets. For zSpace's stylus interactions, this would be a problem for any app initially designed for mouse input.
An engineer and I researched this issue. We discovered a subtle ergonomic problem with the stylus' design. As the user pressed the stylus’ button, the pressure caused the stylus point to move slightly. By the time the button’s electronics registered contact, the beam no longer pointed at the original target.
We gathered complex, multivariate timing data for stylus interactions. We were very concerned that the solution would be complicated and brittle. We were happily surprised that the time between hovering the target object, starting to press the button, and the electronic contact occurring was very deterministic at 0.333 seconds.
Our final solution, named "Déjà Vu," was to go back in time. We created a small 0.333-second buffer of transform data for the stylus. When the system registered a button contact, the system would retrieve and use the stylus' transform data from 0.333-seconds ago.
Déjà Vu was invisible to users and so robust that it became the system’s default behavior for all selection operations.
Color and Material Pickers
Selecting colors and materials was identified as a frequent task in early research for zSpace's Maya plugin.
I recognized that picking colors in a spatial computing environment was a perfect match, as many color spaces are 3d constructs. I designed a 3d color picker that leveraged zSpace's 6DOF stylus. A solid cone of colors represented the HSV color space. The stylus controlled a 2d cutting plane that intersects the HSV color cone. A selection point pinpoints the currently-selected color.
A critical aspect of selecting materials is viewing them from different angles. Current material selectors emulate a variety of viewing angles by providing static, 2d renders of 3d objects. With zSpace's capabilities, users select materials by seeing them on rotating 3d objects.
3d Menus
Our research concluded that zSpace-powered context menus would improve the Maya experience. We experimented with pie menus, traditional rectangular menus, and quad rectangular menus. In the end, the best results came from applying our 3d treatment to the Maya-standard menu layout.
light+glass
In the menu and material picker examples above, you saw "light+glass," the look and feel I drove for the zSpace platform. As a new 3d environment, zSpace needed a visual language, interaction model, and core tools. I looked at prior art and domain experts for guidance and inspiration.
Domain experts referenced popular media including Minority Report, Iron Man, Avatar, Tron, and Kinect games for what they expected from 3d spaces. I analyzed these sources, identified themes, and created light+glass' design principles:
Translucency and lighting effects drove the design
Glowing edges on transparent planes defined surfaces
Blue was the primary color
Simple lines and geometric shapes comprising the UI
Colors with different levels of glow and translucency conveyed tool states and modes
Translucent planes revealed the depth of the 3d workspace. Without occluding objects completely, these planes provided spatial reference points so users could localize items in the 3d space. The different glowing colors reinforced the type and modes of tools.