How do you know what's important in a customer interview?
And, why ChatGPT might struggle to tell you which insights were actionable (no matter how good your prompt was)
A piece of advice you’ll see elsewhere in Deploy Empathy is to dig deeper and ask follow-up questions when the person you’re interviewing says something interesting.
But… how do you know what’s interesting?
In an interview, you’re piecing together the steps of their process and looking for how what you fits into that, and where there might be opportunities for you—to do something new, to do something better, to take some revenue from a competitor to save your customer some money and get some expansion revenue yourself by adding a new feature, and so forth…
But there are a million different directions you could take things in.
You could have two businesses that run a SaaS in the exact same industry and have them interview the same potential customer and have both of them come away with entirely different takeaways for their business. They’ll probably—hopefully—agree on the customer’s underlying process, goals, and experience are, but what they do with that information could be entirely different.
Why?
Strategy and business model.
Let’s say the potential customer describes a complicated, manual process. One company might hear a consulting opportunity on top of their existing SaaS. Another might hear an opportunity to build a new feature to make it less manual. Neither is right or wrong, and but you need to at least a general sense of your business model and strategy in the back of your head to know what’s interesting and where you should dig in during an interview.
For example, before I ran Geocodio, I worked in the personal finance space. The company I worked for was a publisher, not a regulated financial advisor, and therefore couldn’t provide personalized investment advice to people, only general advice.
That meant that when we interviewed people about, say, how they decided to buy stocks for their portfolios, we didn’t go down lines of inquiry about desires to have someone manage their portfolios for them or to receive personalized advice. We instead focused on situations where the company’s general recommendations could be used, how they used them, and so forth. The company’s strategy was to help individual investors succeed through general financial advice, and that guided how we conducted our interviews.
At this point, you might be reading this and saying, “well that’s all well and good if you know what your strategy and business model is, but I have no idea.”
And that would be a very understandable reaction!
Many companies, even large, established ones, don’t have well-articulated strategies or business models.
At a basic level, the kinds of things you want to have a general awareness of are—even if you don’t have a business yet and these are just aspirations on your part:
Why does your business exist? (Beyond being something that pays you.)
What unique value does it bring to the market that the market wants and is willing to pay for?
What key skills and competencies do you have?
How do you acquire customers?
What competitive advantages do you have?
Let’s do an example
Now, I know that all sounds quite business school-ey and perhaps a bit abstract, so let’s break it down.
Let’s say you have a new business idea:* you want to make a plug-in that allows people to export spreadsheets. Specifically, you used to work in the banking industry, and noticed that banks with multi-national customers often need their statements exported with different currency formats, such as periods as a decimal separator vs commas as a decimal separator. Customers would download the export and then manually, one-by-one, replace the commas with periods, or vice versa. High-end customers would ask bank employees to do this for them. This drove you nuts when you worked for the bank, because it could clearly be automated.
Why does your business exist? Because you believe that bank customers deserve to be able to get their statement exports in their preferred format, and that banks can’t provide this themselves. Also, because you were tired of working at the bank, and want to be able to replace your income with your own business.
What unique value does it bring to the market that the market wants and is willing to pay for? In your view, there’s currently no off-the-shelf solution for this, and building it themselves is too costly and time-consuming with the ancient code banks are often running. High-end, multi-national corporate customers had bank employees do this for them, and the bank use those employees for higher-value tasks and could save money by using software instead.
What key skills and competencies do you have? You used to work in a bank, so you know the technical challenges they face.
How do you acquire customers? Given your experience in the industry, you know this is going to be in-person sales, starting with past contacts.
What competitive advantages do you have? Your past contacts mean that you have unique access to customers who can serve as your early users and provide you with referrals (compared to someone who never worked in banking).
If we now apply that to interviews:
It’s clearly a vertical-focused company. Interviewing people outside of banking wouldn’t make sense (at this point).
It’s focused on a particular activity: customers working with, and exporting, their data. If a potential customer starts talking about the deposit process, you’d steer the conversation back to exports.
It’s going to be a high-touch sales model. Things that lead in the direction of self-service probably won’t be interesting (as nice as self-service is).
You’ll probably want to talk to some end-users, too. You have your own experience with this, but need to talk to others to get a diverse, broadened perspective.
You’re focusing on CSV exports. You don’t see as much struggle in the activities around PDF exports. You might decide that for now, PDF exports of statements (like what the bank would mail) aren’t interesting, and steer the conversation away from that.
You’re looking for an opportunity that replaces your salary, not a moonshot. It’s okay if this opportunity is a $1M annual revenue company opportunity. So you’re not looking to replace all of the customer-facing software a bank uses—which would be big money, but also very complicated and hard to sell—but rather just one component.
And so forth. Where you take your interviews depends on your strategy and business model.
*I’m not saying this is a good business idea, but it’s maybe something I had to deal with personally this week.
How to figure out what your strategy and business model are
Even if you’re sitting there saying you have no idea what your strategy or business model is, I bet if you sat down with the right resources, you would surprise yourself with just how much you do know about them.
Going in-depth on strategy and business models is a bit outside the scope of this book, but I’m more that happy to refer you further.
On strategy, I can’t recommend No Bullsh!t Strategy by Alex H. M. Smith highly enough. It’s short and practical—only about 150 pages.
For years, I’ve been looking for a good, digestible, actionable strategy book to recommend to other founders, and a founder friend recommended this one to me a few months ago. (Thanks, Kieran!) I tore through it in one night. It’s so good that I’ve already given a lightning talk on it.
Trust me—No Bullsh!t Strategy is well worth the two hours it will take to read it.
On business models, it might sound a bit stodgy, but nothing beats the Business Model Canvas. It’s a very helpful tool for thinking through the different components of what makes a business work (or what you’ve got that might make it work).
For example, the Business Model Canvas will help you define what your unique skills and competencies are. Each person and business has things they’re better at than other things, and things they’re better at, or more focused on, than other companies. If an interview is leading in the direction of features your company doesn’t have the capabilities to build, it isn’t going to be worth digging deeper, and you might as well steer it towards things that are in scope.
And this is why AI can’t tell you what’s actionable from an interview
Lately, probably like many of you, I’ve been experimenting with asking Claude to summarize and analyze my interviews.
I’m experimenting with a variety of prompts—I still haven’t one that I think is good enough to recommend—and one thing I’ve noticed is that the “Key Insights” are always way off-base. I’ve asked it to highlight opportunities for improved or new features, and it invariably fails to capture things that are roadmap-relevant for my company.
But it’s not because Claude is interpreting the interview wrong. In fact, I’d say Claude gets about 80% of the insights (and that matches Claude’s own self-analysis, too).
Instead, it’s because Claude, or ChatGPT, or whichever LLM you’re using, cannot possibly have as much context on the motivations, capabilities, competitive advantages, and aspirations of the business as you do. Perhaps you could do the (simple and effective) exercises recommended by Alex Smith, upload your roadmap, and plug those into Claude… but even still, I’m going to bet its takeaways are going to miss insights, as well as come up with a bunch of irrelevant ones.
When it comes to using AI in my research process, I think of it like a very smart, very fast, very eager intern who just started yesterday. Not only do they have limited practical experience, they have limited context on the business (even if they’ve read everything they can get their hands on about it), and often need to be steered (and still might often end up wildly in the wrong direction).
As a former overly-eager intern myself, I have a certain sympathy with Claude’s well-intentioned over-achieving way beyond my prompt. But still, its analysis cannot compare to someone with more context.
Perhaps you could spend hours and hours loading it with context. But at that point, you might as well have it create a general summary, and then take a few minutes to write out your own bullet points of the takeaways. There’s value in the process of research, after all. Even if you conducted the interview yourself, reading the transcript and your LLM of choice’s summary might make you think of things you didn’t think of the first time. Aiming for the utmost efficiency and speed doesn’t necessarily lead to the most effective result.
Yet, AI has huge promises for customer research. It’s absolutely amazing that we can get automated transcripts back in a matter of minutes now (rather than waiting days or weeks for a human transcription!), and a summary that’s 80% of the way there. I admit I’ve been a bit of a luddite when it comes to LLMs—admittedly a bit stung by how Meta and possibly ChatGPT used my book as part of their training data—but I’m a convert for using it within clear-headed bounds. It’s an intern, not a seasoned expert.
Anyway, that’s all for now.
Have a good weekend,
Michele
PS. On the topic of the use of AI, I’ve been thinking about this paper a lot this week. There’s this idea in the tech industry that the things we create are morally neutral and whether they’re used for good or evil purposes simply comes down to the user and their use case… and this paper wholly rejects that idea, and instead says that we should be designing to encourage positive uses and protecting against negative ones. Some food for thought as you start your weekend.