First American's Golden Eagle Metrics (GEM) initiative aimed to establish a standard set of product metrics across the organization, ultimately allowing product performance to be viewed through a centralized dashboard.
Before rolling out the initiative, the organization needed to understand its current measurement landscape. Were teams already using product metrics? How were those metrics influencing decisions? And, most importantly, what could prevent teams from adopting a standardized framework?
My role was to conduct exploratory research to uncover those potential barriers before implementation began.
I owned the research process end-to-end, from defining the research approach and recruiting participants to analyzing the data and presenting findings to senior product leadership.
I began with a survey to understand how product teams currently approached measurement.
The survey was distributed to approximately 80 product managers and designers, receiving 35 responses. It explored what teams measured, how frequently they tracked and used metrics, whether metrics informed decision-making, and challenges they had previously encountered.
I created the survey in Microsoft Forms, analyzed the responses using Excel, and used Python to prepare portions of the data for visualization.
What I found
The results were mixed.
Some teams regularly incorporated metrics into product decisions, while others had limited or inconsistent measurement practices. Rather than providing one clear explanation, the variation raised a more important question:
Why were teams approaching product measurement so differently?
That question shaped the next phase of the research
To understand the context behind the survey results, I conducted approximately 12 semi-structured stakeholder interviews.
But there's a particularly important research decision here that I want to make prominent:
I interviewed the people who didn't answer, too.
Rather than treating survey non-responses as missing data, I saw them as another potential source of information.
I recruited both survey respondents and people who had chosen not to complete the survey. During 30-minute interviews, I revisited themes from the survey but asked participants to share stories about their experiences implementing and using product metrics.
I analyzed interview transcripts in FigJam, grouping observations and recurring patterns into themes.
This should probably be the visual centerpiece of the entire case study.
It wasn't "we don't want to." It was "we can't."
The interviews revealed something that wasn't apparent from the survey alone.
Some teams understood the value of product metrics and weren't opposed to GEM. Instead, their products created technical and structural barriers to implementation.
Products built on legacy code or operating as embedded products, for example, couldn't necessarily be instrumented in the same way as other products.
This challenged an important assumption behind the initiative: that every product team could reasonably implement the same four metrics.
The original GEM direction centered around implementing four standardized metrics across product teams.
My findings showed that a universal approach would be difficult to implement across fundamentally different product environments.
At the end of my internship, I synthesized the quantitative and qualitative findings into a research presentation for the VP of Design & Product and Director of Product.
The research shifted leadership's understanding of where resistance was coming from. What had appeared to be an adoption problem was, in many cases, an implementation problem.
This prompted the initiative to reconsider whether requiring the same four metrics across every product was a feasible approach.
