Stuff I'M Thinking About

Last month I posted Revisiting Media Strategy and a retrospective on Digital Strategy at RNZ from 2005 to 2016. In both cases I mentioned the Jobs To Be Done (JTBD) framework as the pathway to a better understanding of customers. Knowledge of a customer’s ‘Jobs’ can dramatically improve the quality of a strategy, and lead to improvements in product design, operations and marketing.

In this final post on the broad subject of strategy and the media I am going to take a deeper look at how customer research is typically done, and present a powerful tool (JTBD) that can be used to better understand the customer.

At a high level, data and surveys provide information about what the customer did in the past, or what they think they might do or want. The shortcoming is that this information is not particularly  predictive of what a customer might do in the future. That limits its use in designing the future, which is at the heart of creating a great strategy.

JTBD shows us the why, and is inferentially predictive because we understand what the goals of the customer are, and how their decision making process works.

All tools have limitations and a context in which they are useful. The trick is known which tool to use, and when.

JTBD is not a silver bullet, but I have found it a powerful tool to help me understand the customer.

Let’s start by looking at some examples of how markets and audiences are typically understood. I am falling back on some media examples, but the thinking applies to any sector.

Understanding Markets and Audience

The following types of information are typically used by media companies to guide the business choices they make. The data are used as fodder for both strategy and planning sessions, but as I noted in Revisiting Media Strategy we don’t need all this information up-front, and some of it may not even be needed at all. Data and market research should be selectively used (or created) to test the viability of a strategy, not as the sole inspiration of it.

1. Research and Thought Leaders

There has been some important research and thinking in the web space over the years, and I consider this foundational contextual information for any media executive, regardless of the research approach taken. I will cover a few resources that I found useful as an example; your industry will have its own sources.

For anyone doing business online, Mary Meeker’s reports on internet trends were a must-read up until 2019, the year of the last official report.

Those interested in news should always start with the latest Reuters Institute 2023 Digital News Report. The Infinite Dial, from Edison Research gives a good sense of online behaviour in the US and in many countries.

There were also some historical reports that created a lot of buzz at the time—The New York Times Innovation Report was leaked in 2014, and the BBC published their Future of News report in 2015. These were followed by the NYT 2020 report in 2017.

For podcasting there are plenty of sources of historical data, and I would rely on independent sources first. The easiest way to find good source reports is often through listicles like this one, or here. The major podcasting platforms such as Spotify and Apple (summary) also provide information from time to time.

Some of the larger media companies publish their own data, for example Cumulus Media and NPR (requires email), and this can also contain some general market insights.

This information is all useful in understanding the market, and broad trends, but all of it is about the past. It does not tell you how to innovate, or what to do next.

2.  Markets and Segments

All demand-side markets are usually divided into various demographic segments such as age, gender, location, ethnicity or race, religion, generation, or language. As we will see later, Jobs To Be Done spans these demographics.

On the media supply-side we have different content genres options such as news, technology and gadgets, sports, food, travel, personal finance, entertainment, health, and education. Jobs theory shows us that our competition doesn’t always come from obvious places.

Producers of content can choose a mix of supply- and demand-side options in order to position themselves in the marketplace. For example, we are going to do personal finance for people who have just left university. Or how about the hyper-niche of indoor bowls news for speakers of Spanish who live in San Francisco?

We would expect this to be an explicit choice; companies should know what market they are in, or what market they are working on attempting to win in.

3.  Customer Research

I’d like to dive into this area in much more detail, as an understanding of current research techniques is important so we can understand what they are good for, and where they are weak.

An understanding of the customer is a key component of running any business, and most customer insights are usually based on what they have preferred in the past, or what they say they might like in the future. Neither will lead to innovation.

Most companies now have access to vast amounts of data about their customers. What they buy, how long they stay on pages, their path through the website, and so on. They might also have a way to link customers to other data, such as their social media profiles, credit card use, and so on. These days, it is possible to buy much of this information from data brokers. 

As an aside, that is why you’ll sometimes be hit with ads for a product you just bought, or for similar products or services. There are real privacy concerns about the way this data is collected and aggregated, and I think this is just a bit creepy. 

My contention is that while all this data does provide some real insights into the customers and their past behaviour, it does not tell us directly about their motivation and what drives them to consume our product over some other product, or some other piece of content over our content. 

You may be able to see patterns—for example men aged between 40 and 60 read a lot about weather on Thursday nights, but it does not tell us why, or what motivates them and their goals. (They are anticipating being ‘sick’ on Friday to play golf.)

I want to talk in detail about the different forms of data available so that you can understand their limitations, before moving on to explain Jobs To Be Done and how it differs. Once you have the JTBD lens, there is no turning back; it is such a powerful tool it is hard to go back to the old ways of thinking.

4.  Market Surveys

Real surveys are conducted by genuine market research companies, and based on sound statistical methods.

To understand the results of a survey, we need to understand the difference between signal and noise, and how to differentiate between them. We also need to understand that surveys are about the past, and do not predict the future. Just because we performed at a certain level last week, last month, or last quarter, does not mean that this will continue into the future. 

We are measuring a chaotic system, and what this means in practice is that anything could happen.

Surveys use statistical sampling methods to balance accuracy against cost. The aim is to get a value that represents (i.e. is close to) the actual value in the whole population we are interested in. In the simplest implementation, the whole population is randomly sampled (it must be random, with all people having an equal chance of being selected).

For a population of 5 million, a sample of 1100 will give a margin of error of ±3%. To reduce this margin to ±1%, we must increase the sample size to 10,000, all of whom must be randomly selected, hence my comment about balancing cost and accuracy.

Random selection is, of course, difficult. In the days when every household had a phone, and people were more likely to be at home, it was relatively easy. Survey companies now have ways to ensure randomness while accounting for the unreachability of some parts of the population. The sampling design has to account for this, as well as many other factors such as the reluctance of certain groups of people to answer surveys at all. Survey design is something for trained professionals; do not try this at home. 

In order to use survey data correctly, we need to learn what a signal is – that is a statistically significant change or trend.

Most properly run surveys have a margin of error of plus or minus three percent (±3%), which is considered adequate enough for most purposes. If the survey result was 50%, this means that the actual value in the population being surveyed is probably between 47% and 53%. A value outside this range is a signal that there is a high chance that something has changed. The margin of error is a proportion, and is quoted for a result of 50% (in this case), and will decrease for values closer to 100% or 0%.

That means that taking action based on a change within this range is unwise; being within the margin of error, it could indicate no change at all. Put another way, the change is within the statistical noise that stems from the limited sample size. Likewise, thinking that a change you made prior to the survey period has caused the reported change you’ve seen is also not correct when within the margin of error.

With the right sample size we can be about 95% confident that the true value does fall within the margin of error. That means there is a 1 in 20 chance that the true value is NOT in the stated range.

When I was at RNZ they used to call these rogue surveys, but it is important to note that they are ‘wrong’ because of sampling error, not because of a mistake. Over my time (35 years), there were at least a couple of these rogues. In one case there were huge celebrations about a stellar result, with much back-slapping and champagne cork-popping. “Our work is paying off, we need to stay focussed to lock in the result.” The following year, the numbers returned to worse than previous levels, and there was much head-scratching “What did we do wrong? Whose fault is this? Who let the team down?” These responses, while understandable, were not rational given the statistical reality.

I reiterate: the tendency to treat changes within the margin of error as significant, and an indication of progress or failure, is a major trap. Certainly there might have been a change, but we can’t know that. A change between this month/last month, or this year/last year, does not indicate a trend when the second result in the series is within the margin of error. The more data points you have in a series, the more likely it is to confirm a trend. Just not two. Or even three.

Yes, you can draw a line, but this line does not predict where things are going (statistically speaking). The Sharpie is not all powerful!

This is very important to understand. Humans do not naturally discern the difference between a signal and noise, and this results in actions being taken (or celebrations being made) when none are warranted. Humans love to find patterns in things. Burn marks on toast. Pictures in the night sky. We’ve been doing it for millenia. 

The danger of tampering with a process in response to statistical noise is well-known, having been discovered by Walter Shewhart of Bell Labs in the 1920s. His work has since been applied in countless industries, from manufacturing to service industries and medicine.

As an aside, credible news organisations have editorial standards around what can be reported as a survey, and how the results should be reported.

5.  Web Analytics

These are considered a real boon for anyone running a business online—real-time analytics give direct feedback on how a particular piece of content is doing right now, compared with other content. That is useful, to be sure.

To be most useful though, web stats needs to be considered as a package deal. There will be trends over time, and the relative popularity of sections can be determined.

A common mistake is to take action based on limited data, and this is even worse when fuelled by personal biases.

Back in 2014 one manager at RNZ wanted to cut content in the Te Manu Korihi (now Te Ao Māori) section of the site and divert staff to other stories. This was because, according to him, it did not do well compared with mainstream content. I do not like the implications of terms like ‘mainstream’ in this context, but will continue to use it for this example.

Stats did suggest that these stories did worse than other news when initially published. What the data also showed was that while the so-called mainstream stories would do very well the first day they were released, they had almost no views a few days later.

On the other hand, content in the Te Manu Korihi section of the site would continue to accumulate page views for days, weeks and occasionally months, sometimes gaining more total views than the mainstream stories. Visitors also spent much longer on Te Manu Korihi stories, long enough to actually read the whole story!

This is an example of a common mistake, based on a combination of bias and a focus on the wrong metrics. Taken to the extreme, this approach (reacting to what is happening right now) will result in perverse and unwanted outcomes, with the company’s production of content being pushed off course from what the strategy dictates. This issue is well known in statistics, and is called the funnel problem.

It is also important to understand the limitations of web analytics.

Another example: we found that, on average, the amount of time spent on most RNZ news stories was insufficient for the whole story to have been read. The wrong response to that information is to shorten all stories. A better response is to see if there is a pattern—is there a difference between different types of story, does it change based on time of day or location, and so on.

A third: we might assume that a change in headline has caused a jump in traffic 5 minutes ago. It might have, but we really don’t actually know. A correlation between a headline change and a change in traffic does not automatically mean the change caused it. Attempts to boost clicks by changing headlines can actually have a worse effect—people click on a story thinking it is new and find they have already read it. Annoying!

To summarise, web stats needs to be considered as a package, and not over-thought.

6.  Personas

Many companies use personas as a proxy for the customer. A persona is an amalgam of customers’ attributes that staff can use when working on problems. I have used these over the years, but have found them quite unhelpful because they usually get created for use in one narrow context, and suddenly they are being used as the tool to solve problems everywhere else.

We tend towards wanting one-size-fits-all solutions, but the interface presented by ‘the market’ is a complex system that will defy attempts to define it in simple terms. Don’t try.

The Nielson Norman group has an excellent overview of why personas fail, and how to avoid the pitfalls.

7.  Targeted surveys

I get three or four emails a year to participate in online radio surveys and panels. Some of these are to judge if my listening habits have changed—am I listening to different stations at different times, or am I listening to something new. These are similar to the market surveys cited above except they are not random, and may be demographically biassed (they do ask for demographic information). This isn’t a problem as long as we understand who the data represents, and that it may not be representative of the general population.

Regardless, these relatively insensitive surveys can serve as an early warning sign of major problems, or sudden changes in the market.

The other type of targeted survey is one that focussed on a specific issue. I recently saw a survey focussed on business news. It asked questions such as what would I like to hear more of/less of, and how could they do better.

This type of survey is trying to elicit solutions from customers. I want more stock market results! Can this programme please be twice as long (but without any understanding that increasing the length might not increase the amount of content that is actually useful to me)!? And so on.

People are notoriously bad at directly expressing their preferences in ways that are useful, if they can express them at all, and will often ask for more of the same, or something very different. Most responses lack the detail needed to really decide what to do.

What do customers really need? How can we be sure? 

That is where Jobs To Be Done can help.

Rethinking our understanding of the customer

The existential question I outlined in Revisiting Media Strategy was ‘how do we get people to listen/watch/buy more of our stuff?”

There is a terrible assumption in this question: our product is perfect so all we have to do is market it more effectively.

This pushes us down a bad path where we try to find ways to make people use our product, rather than to find out if there is a mismatch between our product and what the consumer needs to make progress towards some goal. Sometimes that gap is big, and our product is not for them. Other times the gap will be small, and we just have to bridge that for the customer.

If we take a supply-side approach to business—that is, we focus on the product and how to sell it—we are really reliant on marketing and other push-based strategies which can only take us so far. Or not very far at all.

How do we gain a better understanding of the customer, and adjust our course to align with their needs? What do consumers want or need on the demand side, and how can we find out?

JTBD strives to help us, framing this in terms of a struggle the customer has, and the context in which that occurs. The customer has a goal they want to make progress towards, and they will ‘hire’ something to help them get this job done. If we can understand that fully, and align with it, then the customer will be ‘pulled’ towards our product; we won’t have to push, push, push, to try to convince them when the fit isn’t really there.

In Jobs theory we use the term ‘make progress’ a lot. At some point a consumer becomes dissatisfied with something in their life, and they start to look for something to hire to help them make progress towards resolving their struggle. The term ‘hire’ can mean buy, hire, borrow, or just use in some way.

A quick example: Our two-door car is fine for the two of us. We add a baby, and suddenly it becomes a problem—the car seat is hard to fit, and flipping the front seat forward to get the baby in and out is a major pain. The recognition of this struggle triggers a search for something that will help us make progress. The context of use has changed from a couple, to parents, and the places we need to go are different. As we consider a replacement, different criteria are now used to assess things we might hire to get the job done.

If we zoom out, the Job itself has nothing to do with the car. The Job is transporting the three of us from one place to another. We could hire many things to get this job done—buy a car, take public transport, use a car service, phone a friend, cycle or walk—and the solution we hire will depend on the context at different times and places. It is sunny, and the distance is short? Cycle. It’s 100 kilometres away, and will be overnight? There is a lot more to take. Is there a bus? Nope, we need a car. 

This is what makes JTBD so powerful, because what we learn is not linked to any particular person or product/solution. Jobs tend to be very stable over time, and getting from A to B is a classic example.

Moving to this type of demand-side thinking seems like only a small change, but it drives huge changes in terms of strategy and the tactical choices it opens up are significant.

How does it work? 

I will provide here a high-level overview of JTBD. This is not exhaustive, but will hopefully give enough information that you can see why it is superior to the approaches outlined above for certain kinds of problems.

There are two parts, the first is the process a consumer goes through as they progress from realisation through to satisfaction.

The moment of realisation is when they first see that they have a struggle. This car is not working for us.

They might not yet understand what could be better until they start doing some research into options. Once most of the research is done, they will form a view on what might be suitable. Something with four doors and a high safety rating. 

From that point they start a more active search, looking at specific solutions and comparing. They might learn things during this active search that alter their selection criteria. Cruise control and proximity alerts seem like good safety features.

At some point a decision is made, although the consumer will also consider any trade offs before they fully commit to their choice.

Once they’ve used the product, we hope that they like it. This is so much better, why did we struggle with the other car for so long? Satisfaction, as last.

In addition to this process, there are four main forces that surround a ‘struggling moment’, the moment when a consumer realises that a solution no longer is a good fit, and that things could be better. There are two forces that drive the consumer towards a change, and two that hold them back.

The first force is the frustration that stems from the question, how can this be better? This force pushes the customer away from the current solution towards something else. The car is too small. It doesn’t carry enough for the three of us. It has a low safety rating.

The second force is the attraction towards something better. This is the hope of improvement, of making progress. We would have so much more flexibility with something different. We would be safer.

The third and forth forces hold back the consumer from change. The third is the habit of the current solution. It’s the comfort of the known, and concerns about the sunk costs of that commitment, or losses from a change. The seats are so comfortable, and I look cool driving it. If we sell the car, we will lose money.

The fourth force is the anxiety associated with making the change. The question ‘what if…’ will be front of mind. What if it costs a lot more? What if I look like a doofus driving it. What if we have missed an important feature?

These four forces block movement towards a solution, but now that we know about them, how do we get information we can use?

We ask the customer, of course.

Interviewing

This is the hardest part of JTBD because we cannot just go up to the average customer and ask them, what is holding you back, why are you anxious, and so on. We have to be more subtle.

What does that mean in practice?

A typical JTBD interview asks the customer to tell us a story of the purchase they made, and as they unpack what happened we ask more detailed questions to elicit the underlying thoughts and feelings. We are looking for evidence of the four forces and to identify the decision points in their journey. Questions like, when did they realise the current solution wasn’t working for them, what pushed them to change, when did they finally decide on a solution and what pulled them towards that choice?

This is quite hard to describe in writing, so I have linked to a video at the end of this post the includes a JTBD interview.

What do we do with this information?

Once we have conducted interviews with people who have recently hired (bought) or fired (replaced) our product, and collated the information (another hard task), we get an understanding of the forces at play. 

In particular, we gain an understanding of what the customer jobs are, and how our product helped them make progress towards their goal (job). We will also discover from the people who ‘fired’ us what caused them to stop using our product.

This is a lot better than simply collecting data about past behaviour, or conducting surveys. Don’t get me wrong, those things are important, but they do not give you any insight into what caused the behaviour you are observing in the data. They also don’t tell you why someone stopped using the product because their data is not in the data set anymore.

Once you have an understanding of the customer through the JTBD lens, everything changes. Marketing copy is no longer about the features of your product. It shifts to being about how the product helps the customer make progress, using the insights you collected.

The way you design features also changes, because you are now oriented around the customers goals and the progress they want to make, and not a list of functional requirements.

JTBD is one of the tools that takes a while to understand fully, but is one of those lenses that once you have it there is no going back, and you cannot unsee what it reveals.

And even if you don’t interview customers in depth—the ideal—an understanding of Jobs can still make a positive impact on your work. 

Closing thoughts

When the only tool you have is a hammer, everything looks like a nail. When the only tool you have is a rear vision mirror, you crash. JTBD is not a silver bullet, but is a powerful tool to use to understand certain problems, and it is certainly worth adding to your repertoire.

If you want to learn more about JTBD, The Re-wired Group has an introduction to the framework, and also run courses from time to time. I also recommend the following two videos.

The first includes an overview of JTBD and an example interview.

The second is an in depth interview with Bob Moesta from Re-wired Group talking about the framework and the content of his book.

That book, Demand-Side Sales 101, is a must-read for anyone wanting to get into JTBD.

* My thanks to:
Liza Bolton, Professional Teaching Fellow in the Department of Statistics at the University of Auckland, for checking, and her suggestions about, the statistical content in this post.
Aylon Herbet, for his feedback on the JTBD section.
Bob Moesta, for permission to use the JTBD graphics in this article.