Product Designer

Analytics Redesign Details

Case Study: Data Visualization

Analytics Platform Redesign for Surgical Residents

Role

UI/UX Designer

Timeline

2018-present

collaborators

Backend Software Engineers, Surgical Residents, Residency Programs, Director of Clinical Affairs

Overview

The Analytics Platform Redesign for desktop was created for surgical residents to capture their surgical proficiency to help them understand how to better prepare for future surgical cases.

Achievements

  • Released to surgical residents for evaluation

  • 70% Response rate from study and survey

LaptopRedo_GrayBackground_Updated.png

The OlD

Prior to the redesign of the analytics performance dashboard, a dashboard had been built but rarely seen or used by Osso VR’s customers. Due to requests from surgeons and surgical residents along with the need to harness and visually represent data collected from virtual reality run-throughs, I was placed in charge of identifying and displaying data that could be used by our end users to improve their surgical abilities in preparation for surgical cases in the future.

Old Performance Dashboard

Old Performance Dashboard

The New

Surgical Residents requested four main types of data to help them evaluate and see where to improve their skills on surgical techniques:

  1. Speed of their VR run-through on a given technique

  2. How they compare to expert surgeons on a given technique

  3. Where in the technique they can improve

  4. How they compare to colleagues of the same year


Prioritizing the data

Through user interviews, we got a glimpse of how residents were assessed for surgical proficiency in the operating room. This helped us prioritize the data we wanted to show to both reflect the current assessment’s priorities while incorporating the data that surgical residents requested to see.

Our Findings:

  • Attendings are the only ones evaluating a resident’s competency.

  • Residents are evaluated on a 1-5 scale. 1 being you’ve never picked up a knife to 5 being you’re exceptional at the surgery.

  • The assessment process can be very subjective and depend a lot on how well you work with a given attending.

It’s supposed to be practical but sometimes it’s more like he knows the next instrument or he’s in sync with me on the next step.
— 5th year resident
 

What kind of graphs should be used to visualize this data?

 

I mocked up several different ways to visualize the above data and keeping the original style and surgical assessment in mind. We were trying to show more concrete data than was already being assessed in the OR. I established the idea of performance and proficiency levels to encompass the requested data. I aimed to incorporate ideas from the original dashboard to expand on what we already had built out.

Iteration 1 . All data on one graph. Unfortunately, this is overwhelming and not as informative as it could be to our end users.

Iteration 1. All data on one graph. Unfortunately, this is overwhelming and not as informative as it could be to our end users.

 
Iteration 2.  Separating out the data and experimenting with different types of visual feedback for users to understand in more detail where they requested hinted and need to focus their efforts.

Iteration 2. Separating out the data and experimenting with different types of visual feedback for users to understand in more detail where they requested hinted and need to focus their efforts.

Iteration 3.  To capture all requested information and provide an easy way to compare across data sets, I focused on presenting bar graphs and a line graph to show trends across time for the flow of the procedure.

Iteration 3. To capture all requested information and provide an easy way to compare across data sets, I focused on presenting bar graphs and a line graph to show trends across time for the flow of the procedure.

 

Where and when should this data be presented?

Two Major Use Cases to Consider:

Use Case #1:  User has completed a run-through and wants to immediately see results to know where to focus their time to improve

Use Case #1: User has completed a run-through and wants to immediately see results to know where to focus their time to improve

Use Case #2:  User has logged into their account to review previous run-throughs before continuing VR training

Use Case #2: User has logged into their account to review previous run-throughs before continuing VR training

 
User 1 requires a modal and a performance dashboard to see their results both immediately and in more depth. User 2 on the other hand only requires a performance dashboard to see their results to understand in more depth where to start their VR training

User 1 requires a modal and a performance dashboard to see their results both immediately and in more depth. User 2 on the other hand only requires a performance dashboard to see their results to understand in more depth where to start their VR training

 
 

How do we evaluate the designs so far?

In collaboration with the Director of Clinical Affairs, we organized a beta Study with 2 residency programs to gain feedback on surgical resident overall perception of the dashboard, and to determine whether the data shown was valuable or what other aspects are needed to make the dashboard more useful.

Screen Recording of Prototype in Invision. Data for Procedure and Procedure Description are intentionally missing.

Screen Recording of Prototype in Invision. Data for Procedure and Procedure Description are intentionally missing.

 

What were our findings?

 
  1. The use of stars for proficiency on a procedure was confusing. A number of the users reported associating stars with a rating not proficiency.

  2. Few users pressed on the survey link which made it challenging to initially gather feedback. As a result, I followed up with residents to collect further feedback. Future surveys and feedback needs to be either done in person or more visible.

 

Updated Designs Prepared for Initial Release

TrimmedFinalDashboardDesign.gif

What could have been done differently?

  1. Instead of grouping all residents together, creating a slightly different dashboard for an intern, junior, and senior resident

  2. Scheduling an in person walk through of the prototypes to receive feedback more efficiently and immediately following the experience.