At Informerly we are focused on one thing: delivering relevance. We want to be the best at getting the right articles in front of the right users. The service that meets the professional information needs of our users and helps them be better at what they do.
The question we obsess over is how to measure our ability to deliver relevance. How do we know that the articles we’re sending are actually useful to the person receiving them.
John Borthwick wrote a seminal piece on engagement in today’s media environment. He gives us idea of the Hill of Wow, that moment in content consumption where you’ve clicked on an article and it delivers on it’s promise. It’s the “right hill” in the reader’s journey below.
The content consumption journey
We want every link we send to live on that Hill of Wow.
The words “content matching and distribution process” don’t exactly make one dance on a table, but we’ve built a unique process to deliver relevant content that we’re pretty excited about. It’s a big step towards getting the right articles in front of the right people. We’ve been using it to power an email newsletter (twice a week, 5 links per email) and we think we’re impressed with the results.
I say “we think” because understanding whether an individual link is relevant to an individual user is a never-ending process. Some headlines are just more clickable, some topics are just more enticing. Did that link help you with your job? Did it take you to the Hill of Wow?
As we’re just an email product on the surface, the topline metric we monitor is of course, the open rate. They’re really good. We see 60%+ open rates on every campaign. The test group is incredibly small by newsletter standards (500), but we’ve chosen a very specific group. We knew it’d be a lot easier to chop up TechCrunch and send a newsletter to a homogeneous audience of twenty-something tech guys in SF.
Instead we’ve managed to put together group composed of everyone from entry-level marketers to CEOs, food delivery to lingerie companies, Amazon to McKinsey to Alibaba, and from 27 different countries. With a group this diverse, high open rates start to convince us that something is working. But we know the open rate is not nearly enough.
Click-through rates are the next convenient metric. Out of all emails sent, 25-30% of people click a link, and out of people who open the email around 45% of people click a link. Again, pretty good, but does it really mean we’re building a brand as a trusted source of business information?
This is where our obsession with tracking everything and the technology currently out there starts to make things interesting. We’ve been using a tool called Litmus that does some pretty amazing things with email tracking.
Read vs. Glance
Litmus has a metric whether someone “read” or “glanced” at your email. 85% of people who open are “reading” the emails, which means the email was open for more than 8 seconds. We’re a bit suspect on how accurate this one actually is.
Multiple Opens + Forwards
Litmus also tells you how many times someone opens the email. This was one of those “holy shit” data points. Each week at least 10% of the overall group open the email more than once. Then there are the ones who open it over and over again.
Which brings us to the beloved email forward. Litmus captures when someone forwards the email (it doesn’t work perfectly). This is an instant indicators of relevance. If the email is so good someone is willing to forward it on, our confidence is high we’re delivering value.
Lifetime User Behavior
Then comes question of habit: do our users open our emails with regularity? This is important because it convinces us there’s far more than subject line magic. We’re still working on how to represent lifetime user engagement in aggregate. For specific high-value users we regularly check their behavior, but en masse, we’re limited to a few high-level metrics. Out of all emails we’ve ever sent: 61% have been opened, 27% have been clicked, 13% have been opened more than once.
For each individual user, we’ve built a profile that shows there lifetime statistics, along with what they’ve clicked on (both headlines and the associated tags). We are madly in love with users who have clicked 100% of emails over the four months we’ve been sending them, but we’re still searching for how to quickly and accurately gauge engagement over time.
So far all the metrics I’ve mentioned relate to user behavior. But what about the content itself?
If Informerly is going to be the company we want to be, we need to be able to get the right content in front of the right user. Is the subject matter of the articles we’re sending relevant to the jobs our users do every day?
Each article that enters our content pool is parsed and tagged. We also preset an interest string for each user based on their sector, title, and region. We monitor what are the most common tags of the articles they’re clicking on. When we get it right, the common tags match the interest string. When they don’t match, we adjust the interest string. It’s an ongoing process trying to understand what users need to read about.
Aggregate Links Clicked
Finally, in trying to understand if the links we’re sending are relevant, we look at the overall number of links clicked per campaign. For a campaign of 500, we’re sending 2500 links. Around 9% of all links sent are clicked on average. We think that’s pretty good.
Beyond all these numbers, there are the things we can’t quantitatively measure that make us smile. When we get an unsolicited email that has the word “relevance” in it, we kind of freak out (in a good way).
“wanted to thank you again for the informerly tip. The newsletter has great quality/relevancy!”
“Just wanted to say I think this service is absolutely brilliant. It also has content that’s relevant and interesting to me.”
“Informerly has become an important, relevant and truly curated source of information for me and my team.”
So what’s the point of all this?
Borthwick concludes, “people who are focussed on building around this second hill are going to end up with stronger businesses.”
The businesses and models playing on the left hill have already been built. We’ve seen people coming up with some pretty creative ways to get millions of people to them, with little regard for what happens after. The long-term opportunity is owning the right hill. That’s where the next generation of media companies will have to play and win. We want to live atop the Hill of Wow.