Asking the Right Questions
This is the last of four posts to introduce my views on how software development is changing in the agentic era. Here we touch on what is probably the most important skill a leader can have; asking the right questions. Be insightful, not blindly curious. It's a crucial topic, one that we'll definitely revisit.
Ever hear about the parable of the “old man and the hammer” or the “handyman’s invoice”? As a leader in the engineering org, the art of asking questions is the most important skill that’s the same as knowing where to tap on a furnace to fix it. Product development is a game of taking risks and maximizing the chance of success. When you step back for a moment and you think about a typical team - engineers, product managers, designers, data scientists etc. - these minds are put together to ensure the team makes the best possible decisions. The best possible decision is the one with the highest chance of success, and it’s only a chance because it’s not possible to know the future. Knowing where to tap the hammer means having your team put their effort on answering the right questions.
Agents have changed many aspects of engineering but what hasn’t changed is telling engineering to build the wrong thing is still the most expensive mistake you can make. Figuring out what is the right thing to do requires asking questions. But there's an important point to double down on: asking questions is not free.
What’s your product?
Before asking questions, first understand your product. Imagine your team as a small business and you’re the owner. It’s not that far-fetched: a team of seven to ten is a multi-million spend per year; that’s far more than the vast majority of small businesses. If you’re a dev tool team then presume others could have went with different vendors in a build vs buy decision.
Figure out what makes your product a winner. What is your key differentiator? Write your elevator pitch. What do you sell and why are you critical to your customers? If you haven’t found product market fit yet that’s fine too; write down the possibilities or what you hope it’ll be.
Describe why competitors haven’t built what your team ships or will ship. What is hard about it? If you’re the first mover, describe why you’re tackling this green field space in the industry; what differentiates it from the past? For example, you could be building a graph-based database system that is more performant than traditional relational or other solutions.
Figure out what your product is about and what makes it the winner.
What are your product differentiators? Let's draft something up.
Your value proposition is that it's a creator-led environment with users directly funding them with their hard earned money. It's positive; users are there for the content, goods and services by the creators and the platform provides an easy way for creators to interact with their customers. When creators do well, so does the platform.
It's a social media app that respects and recognizes the creators that create the content the users view and share. Creators can offer goods/services through the platform and people pay them directly, the platform just takes a cut. Users get to have the viewing experience they want; no ads.
How are you competing? Other platforms rely on ads, not creator revenue, so you'll get more creators on your platform and therefore better content. The incentives are aligned between the platform and the creators. Users get a better app experience because there's less or no ads.
Got your winning formula ready? Let's move onto the next part.
What needs to be proven to function?
Asking the right questions ultimately translates into the team spending effort on answering them. That’s why it’s crucial to hone your leaders to be able to pinpoint exactly what needs to be proven and how it can be proven. In a head to head competitive environment someone else who can spend a tenth the effort to get to the same point will win, all else being equal.
Work backwards from what you believe needs to be built and ask what are the unproven aspects. The most important part is to differentiate what is a bet and what is work. Bets need to be proven and they’re the key ways the team believes the product will win in the market. Maybe it’s a way for content search in a RAG system to be ten times faster, or it’s a way to process an image and say whether it’s a hot dog or not. Anything you know how to do with certainty? That’s work. Think of building an app, standing up backend services, configuring the databases, ensuring accessibility compliance etc. Just deal with these when you draw up the production plan.
Working backwards from your goal is key. Imagine you’re building a paralegal agent in a software suite meant for law firms. You hope to automate the rote work of pulling up case references (and forgive this author’s lack of expertise in the area of law). Then the key question to ask is whether you can pull up relevant information when a paralegal enters in a description. From there you might ask, what does relevance mean? Or what is a typical kind of description? A team might discuss this and realize that the bigger problem paralegals face is summarizing the content, not pulling up relevant info.
Asking the right questions is the key difference in leadership competency. But remember it’s not done in a vacuum. Teams are full of expertise to help shape and discuss the questions. We’ll talk in the future about process and org dynamics.
On the technical side, any good social media app is about the experience that users have when they're on it. The creator side is usually pretty straightforward; building out ways for users to shop goods/services, content creation tools, post scheduling etc. That leaves engineers to wonder if they can build the right algorithms for populating feeds.
This leads to deeper questions. While any algorithm feed is functionally similar to a two-sided ad auctioning system (supply being slots in a feed and demand being the creator content bidding for a slot), how might you build experiments to understand what features are most important: content diversity, content quality, etc. These can be the focus of the engineers.
On the non-technical side, the value proposition was built around creators making content for the platform; therefore how do you entice creators to join? Are there any market incentives needed? Is there a big marketing push that needs to happen? Do you need to kick it off with a few big names to attract some audiences? This can be the focus of the cross-functional partners.
Asking the right questions will focus the energy of the team on the right items. Wasting energy on the wrong items can be costly; slower time to market, or even confusing the product development.
How can this be proven?
Different kinds of bets need different types of proof. Technical ones imply an engineer’s time in experimenting or prototyping. Maybe you’re testing whether an LLM-based agent can tell the time based on a photo or if you can crawl a website’s content and tag the information retrieved with high accuracy. Non-technical ones imply cross-functional partners such as product managers or designers, experimenting in their own way (eg. customer research or functional design mocks). Maybe it’s not clear what kind of user input might be received in a particular feedback tool.
Figure out the minimum effort needed to prove a bet. We’ve moved on from Agile where teams built a product iteration. Forget complete solutions. Think faster, think cheaper: backtest experiments are completely possible outside of traditional ML systems, prototypes can be vibe coded (but don’t forget that it’s not fit for production), or you can shadow existing systems.
Work upwards from the easiest type of solution. Imagine a thought process like this:
- backtest with existing data
- shadow processing with live data
- prototype of one or more happy paths
- lightweight version of the product
Your goal is to build with confidence. If you correctly identify what your bets are then you can more easily craft the test. Then you can discuss the results as a team. This process moves you into the new world: prove then build.
For a lot of social media apps, testing algorithm changes (or more specifically, changing a particular bid in the ad auctioning system) is a matter of backtesting with existing user data. Change how a particular bid works, run it again, see what answers it comes up to guess at how it might work. Then when you have enough confidence you do it with live data before an A/B test.
But this is an overly well tread area, let's look to another example.
There's plenty of chat transcripts that you can use to test the conversation flow of the leasing agent. What are you trying to ensure it does? Maybe it's the following:
- Records the name and email of the potential renter
- Notes the wish list of features for the apartment they're trying to rent (eg. number of bedrooms, approximate square footage, etc.)
- When they're thinking to move in
So you devise a system of evaluations based on partial pieces of past conversation to see how the agent responds and whether it correctly collects information. For instance, to see if it records the name properly you take transcripts up to the point where the agent asks for the name to see if your leasing agent will do the same.
Later when you gain more confidence it works, you could have the agent shadow existing live conversations and see what response it would have given.
Avoid bikeshedding
Stay focused on the product. Know the bets and prove them out. Pivot if there are difficulties and figure out how else your product can tackle an industry problem. Quickly get to what questions need to be asked and answered before moving onto a production plan.
Avoid bikeshedding - what you can discuss should not determine what you should discuss. A good sign of success is whether the team can come up with a clear list of action items. You don’t need the whole UX to play around with a few flows. You don’t need to know all the edge cases before proving the happy path was well designed - it’s like disproving corollaries before tackling the original theorem.
Ask questions that make a material difference to your decisions. If the answer to a question does not change the product, it’s a waste of resources to ask. Every moment spent on irrelevant questions are eng resources that could have been doing something else.
Once you’ve got questions to answer, go and do it.
The overall look and feel of an app is a big decision. For a social media app, it can be the difference between the next trend and a forgotten attempt. One way to figure out the big picture is to use focus groups to test out a design theme. Then based on the reactions, scrap or fine tune the next round of designs.
Here agents can be a friend. Get it to whip up designs based on vague ideas and turn it into high fidelity output. You can reduce the iteration time for each design you can present to a focus group. Maybe now it only takes a day to come up with five designs to show a focus group instead of a week.
Exact colours, button placement, sizing and all the rest? Don't bikeshed on that yet. That comes after you've decided the overall theme.
Conclusion
The Achilles’ heel of using agents is the siren call of simply generating output. More output is not more business impact. Asking questions isn't free. It takes effort and energy to answer them. If they're not materially changing what you'll build, or worse they misdirect your team onto the wrong path, you shouldn't be asking them.
An org that can ask the right questions can use agents to answer those questions fast. That accelerates product development with high confidence. Getting to the right answer before your competitors can is the key to success.