Quality vs. Quantity: A 6-Month Analysis of the Age-Old Blogging Debate

July 22, 2015 at 02:00PM

The marketing industry has always been obsessed with the quality vs. quantity debate.

Should you create more content of a lower quality or less content of a higher quality?

In an ideal world, the answer would always be less content of a higher quality. You’d spend lots of time research and writing every post, then when you published it, the whole internet would notice. Thanks to this unrivaled quality, every post would generate massive amounts of traffic and leads.

But that’s not how blogging works in real life. To grow a blog, you need to consistently publish content that your readers enjoy reading. Yet exactly how often to publish and what those posts should look like can vary tremendously by blog.

Last year, we ran several tests on HubSpot's Marketing blog and determined that our sweet spot hovered around four posts a day (the articles themselves varied in terms of comprehensiveness).

But lots had changed on the Marketing blog since that test was run. We hired two new people to the team and started accepting more submissions from our partners. We experimented with new editorial formats. We completely redesigned the entire blog. Our dev team rolled out new tools like the Attribution Report that helped us better measure what our team is doing. We implemented a new strategy to update, optimize, and republish old posts to increase organic traffic and lead gen.

And then, late one night in January, my colleagues Kieran and Joe had a Twitter conversation with Rand Fishkin about the HubSpot blog's editorial strategy. It got me thinking: With all these new tools and resources at our disposal, was the Marketing blog's current high volume strategy still best for our readers, thus best for us? Keeping our headcount and resources constant, what kind of posts should we be publishing -- and at what frequency -- to grow faster?

Funny enough, we weren't the only ones interested in finding out this answer. After Joe and Kieran chatted with Rand on Twitter, we reached out to Trevor Klein at Moz to see if he wanted to run a similar experiment with their audience. He did -- and you can head over to their blog to read their findings.

Six months after starting this experiment on the Marketing blog, we've uncovered some compelling insights that will transform how our section operates. Here's what we did to figure out our optimal editorial strategy -- and what we're going to do about it.  

Some Background: Our Former Publishing Strategy

Before we dive into the experiment details, let's begin with a little background on our previous editorial strategy. 

Our Metrics

The Marketing blog is measured on views, net new leads, and subscribers -- so those are the numbers we wanted to test. Here’s how each of those are defined:

• Views: A view is counted every time a blog post with the HubSpot software tracking token is loaded.
• Net New Leads: When someone who’s never filled out a lead gen form on a landing page (example) actually fills out that form, they become a lead.
• Subscribers: These are people who opt in to receive instant, daily, or weekly notifications from the Marketing blog.

To reach goals for these metrics, the Marketing blog has typically published 3-5 blog posts each week day and 1 blog post each weekend day, totaling about 20-25 posts in a given week.

But publishing new posts isn't the only driver of our blog’s success: 92% of our leads and 75% of our traffic in a given month are from posts published prior to that month. So for the purposes of this experiment, we are only looking at the effects of new posts.

The Types of Posts We Publish

Below are the different types of blog posts we publish:

• Tactical: These post make up the bulk of our blog. They teach people how to do something or inform them about a marketing-specific subject. The coverage usually is at a higher level and may not feature data or original quotes. (Example: What Is Multi-Channel Marketing? [FAQs])
• Deep Tactical: These posts are like Tactical posts, but more in-depth. Often, their word count often exceeds 1,500 words -- but that’s not its only defining characteristic. These in-depth posts cover topics using relevant, recent examples, original quotes, and current data (which can be original). (Example: Typography 101: Everything a Beginner Should Know)
• Infographic/SlideShare: These posts let the curated infographic or SlideShare stand on its own. Usually, they feature a few paragraphs of introduction, the embedded media itself, and not much else. (Note: This is abbreviated as IG/SS throughout the rest of this post.) (Example: The Essential Elements of an Excellent Blog Post [Infographic])
• Editorial: These posts cover a trend or issue that pertains to marketers, using original interviews, hard data, and examples. The difference between this and a tactical post is there will often be no concrete takeaway to implement and the issues discussed in these posts may not be directly about marketing. (Example: The Engagement Ring Story: How De Beers Created a Multi-Billion Dollar Industry From the Ground Up)
• Promo: These posts are very short and directly promotional of a gated offer (like an ebook, template, webinar, or download) or tool (like the Blog Topic Generator). (Example: How to Get 100,000 People to Read Your Blog [Free Ebook])
• Syndications: These posts always have an italicized line of text that says "This post originally appeared on ..." (usually from an internal site like Agency or Sales, but could also be an external site). (Example: B2B Businesses Are Adopting a B2C Sales Approach)
• TOFU: These posts are created with broad, top-of-the-funnel traffic in mind (hence, the name). They’re usually related to larger internet trends or business-related topics, and are lighter in nature. There are also very few tactical lessons that can be taken away from them. We tend to publish these posts in an ad hoc basis, often with traffic goals in mind. (Example: 15 of Google's Coolest Doodles)
• Misc. Team Initiatives: We’re blogging to support a business, so these posts support the larger HubSpot goals, which may not relate to visits, lead gen, or subscriber gen. These also include posts that don’t fit into any of the above categories.

Here’s how often we typically publish the previous post formats on the blog. (Note: TOFU is not included as it is not a typical format we rely on.)

Now, on to the good stuff: How we ran our experiment, and what it uncovered. 

Part 1: What’s Our Optimal Editorial Strategy?

In this part of the experiment, we teamed up Moz to find out what each of our optimal editorial strategies was. 

Should we be writing more posts that require more intensive research and writing at a lower frequency, or should we be publishing less intensive posts at a higher volume? And what kind of short-term effects would these strategies have on traffic, leads, and subscribers?

We ran this test for six weeks, and shortly after, Moz ran a similar test on their blog. Here’s what we did.

Methodology

Each of the following editorial strategies was run in two-week phases.

Benchmark: (23 posts per work week) This was our typical frequency and editorial distribution, just to give us a reference point for the following two phases.

Low Volume, High Comprehensiveness (LVHC): (11.5 posts each workweek) This is 50% fewer posts than the Benchmark phase with a higher skew toward more comprehensive posts (ex: Deep Tactical and Editorial) and away from lighter posts (ex: Tactical and Infographic/SlideShare).

High Volume, Low Comprehensiveness (HVLC): (34.5 post in a given workweek) This is 50% more than our Benchmark phase with a higher skew toward the less comprehensive posts (ex: Infographic/SlideShare, Promo, and Tactical).

And when analyzing the success of phase, we looked at:

• Only traffic and leads from new posts. By eliminating all old posts that generate significant long-term traffic, we were better able to identify the effect post format and frequency had on traffic and leads.
• Only weekday results. As a B2B business, our traffic is most consistent during the week, so it’s the best sample to use for this experiment. So traffic and lead generation results were only measured from the time they were published until the last workday of the week (Friday).
• No traffic or leads from paid sources. This way, we’re comparing consistent results across all three phases of the experiment.

Our Results

1) The Benchmark and HVLC phases received almost the same amount of traffic.

During the LVHC phase, we received nearly 32% less traffic than the Benchmark phase, but during the HVLC phase, traffic only increased 5% from our Benchmark phase.

Section Takeaway

Because the traffic from each source was pretty similar in the HVLC and BM phases, there was only one conclusion we could draw: There’s only so much content our readers can consume. Expecting them to read 35 posts in one workweek is just too much.

That being said, they are willing to consume a higher amount of content than the initial tweet from Rand suggested. When the number of posts dips below our Benchmark frequency -- even when the posts are more comprehensive -- traffic also dips.

Digging deeper into where our traffic was coming from, the reason why this phase performs so poorly becomes clearer.

The distribution of traffic sources remained fairly constant across all channels in all phases of the experiment -- except for the two most reliant on quantity: email and social. Both email and social had a dip in traffic during the LVHC phase, but remained steady during the Benchmark and HVLC phases.

The reason these were dipping so much? The fewer posts we publish, the fewer opportunities there are for posts to get clicked on in the inbox and share on social media.

Moral of the story here: Comprehensiveness can’t make up for frequency -- at least when it comes to short-term traffic.

2) The HVLC phase received the largest number of leads -- almost double what we received during the Benchmark phase.

During the LVHC phase, leads only dipped slightly -- we received a just 4% fewer leads than the Benchmark phase. But during the HVLC phase, we generated almost double the Benchmark phase leads. 

Section Takeaway

The HVLC phase seems like a clear winner when it comes to new post lead gen. Though it generates the same average number of leads per post, the overall volume is much higher.

3) Our Instant email list received the highest number of unsubscribes during the HVLC phase.

To see how post volume affected our subscribers, we looked at how many people we lost per phase (subscriber churn) according to their subscription type.

As you can see in the chart below, the Instant list is the only one that seemed to be influenced by the experiment, lowering subscriber churn the LVHC phase and increasing churn the HVLC phase. This makes sense because that’s the only email type that’s really affected by changing volume -- the same number of Daily and Weekly emails were sent all three phases.

Also, we shouldn’t worry about the daily list subscriber churn growth going up and to the right. We received a huge influx of subscribers during the HVLC phase most likely due to being featured in an Entrepreneur article and implementing an exit popup module, and this more than canceled out for the churn.

Section Takeaway

Instant seems to be the only list affected by the experiment, and it’s clear that publishing more posts is associated with more unsubscribes.

Part 1 Conclusion

In short: LVHC isn’t a viable strategy for us. The traffic and leads losses are too high, and the dip in subscriber churn isn’t enough to make up for those losses.

And when comparing HVLC and Benchmark phases, they tend to generate comparable results. HVLC gets slightly more traffic, but not by much. It also generates more leads, but considering new blog posts only account for 8% of our month’s lead generation capabilities, the difference in leads between these two phases is negligible. Plus, the Benchmark Phase lost fewer subscribers overall.

The only significant difference between the two phases? The amount of effort our team used to get a week’s posts out the door.

With us seeing diminishing returns with a HVLC strategy, it makes sense for us to continue at the Benchmark level, while continuing to tweak our editorial distribution to focus on “high return” posts (those that’d return more traffic and leads than the average post).

But which types of posts are “high return”? While Moz finished running the previous test, we decided to dive into our analytics to find that question's answer.

Part 2: What Types of Posts Get the Highest Return?

When we think about hitting our blog’s goals, we typically measure their immediate return. How did they help us hit our blog’s monthly goals?

But blogging is a long-term play. New posts that you publish have an opportunity to generate results long after they’re published.

So when we set out to find those “high return” posts, we needed to think both short- and long-term. Which posts would do well in their first month, and which posts would do well over time?

Methodology

In this part of the experiment, we looked at the long-term traffic- and lead-generating capabilities of different post types.

I exported a list of new blog posts published between January 1, 2014 and April 16, 2015 on the Marketing blog -- a total of 1,956 posts. Then, we manually categorized blog posts into the predefined categories. Here’s how many posts we had in each category^†:

• Deep Tactical: 145
• Tactical: 1,215
• Editorial: 105
• Infographic/SlideShare: 178
• Syndication: 247
• Promo: 35*
• TOFU: 31*

*These posts have smaller sample sizes, so their results below may appear more inflated than they normally would.

^†Misc has been removed because its posts didn't contain enough unifying characteristics. 

Then, my teammate Zack Wolfson built a script that used our API to pull each post’s views (and their sources) over its first 6 months (this includes paid data, unlike the first part of the experiment). 

Since the API can’t pull the same information for leads, I categorized each of the posts from the first part of the experiment and calculated their conversion rates for the first week they were live. Assuming conversion rates would fluctuate in proportion to traffic, I used those conversion rates to estimate the lead generation capability of each post type. (Note: Because these are estimates, we did not look at leads in Sections 3 and 4 below.)

Here’s what we found.

Our Results

1) TOFU, Deep Tactical, and Infographic/SlideShare posts generate the most traffic, and Promo and Tactical posts generate the most leads.

Traffic

We looked at the average number of views each post type garners over its first six months.

If you take a quick look at the graphs below, you’ll see that all post types generate the most traffic in Month 1, but fall off in Months 2 - 6. The only differences between the posts are 1) how high their initial traffic spike is, and 2) how quickly traffic to that post type falls off in Months 2 - 6.

When you look at Month 1, you’ll notice:

• TOFU, Deep Tactical, and Infographic/SlideShare post types generate the highest number of average views.
• The rest of the posts (Tactical, Editorial, Syndication, and Promo) generate similar numbers of average views in Month 1.

In Months 2 - 6, there’s a pretty big drop-off in average views for all post types, but there are five larger trends:

• TOFU continues to perform the best, followed by Deep Tactical.
• The rest of the post types (Infographic/SlideShare, Tactical, Editorial, Syndication, and Promo) receive similar numbers of average views.
• Syndications perform the worst.

When we add up all the traffic each post type receives on average over the 6 month period, we found that TOFU posts perform by far the best, followed by Deep Tactical and Infographic/SlideShare. The worst traffic generators? Syndications.

Estimated Leads

Using each post type’s estimated visit-to-lead conversion rates, I calculated the post's effects on lead generation. Here's what we found:

• As you can see in the chart above, Promo posts are a clear winner. They generate the largest number of leads on average in every single month after they’re published.
• After Promo posts, Tactical posts are the next highest lead generators.
• Deep Tactical and Infographic/SlideShare posts receive roughly the same amount of leads over time.
• Syndication posts hardly generate any leads (but since they usually don’t have lead gen CTAs on them, this is self-fulfilling prophecy).

When we add up all the leads generated on average for each post type, the takeaway becomes even clearer: Promo posts generate the most leads on average of any post type, and most others don’t contribute many leads in general.

Section Takeaways

For both traffic and leads, if the post type performs well in Month 1, it performs well over time.

There was no “bait and switch” -- a post’s traffic and lead gen success in Month 1 is predicative of its long-term success. TOFU, Deep Tactical, and Infographic/SlideShare posts generate the most traffic, and Promo and Tactical generate the most leads.

2) The “High Returns” Posts: None of the post types generate above-average traffic and leads.

Knowing the average number of traffic and leads each of these post types generates in six months is great -- but it’s lacking context. Instead of just calculating what each post average is, we need to compare it to the average views and leads our typical post gets.

Compared to the typical post, three posts are “high returns” (i.e. generate above-average views over 6 months):

• TOFU: +178%
• Deep Tactical: +75%
• Infographic/SlideShare: +30%

And four generate average or low returns for views over six months:

• Promo: -0.29%
• Tactical: -9%
• Editorial: -18%
• Syndication: -30%

I also compared the average number of leads each post type generates over 6 months. I found that only Promo and Tactical generated above average leads (+533% and +24%, respectively). The rest of the categories generated below-average leads.

Section Takeaways

In the long-term, we get way more bang for our traffic buck by focusing on TOFU, Deep Tactical, and Infographic/SlideShare posts than the others. But these post types aren’t the ones that are "high returns" for leads -- those are Promo and Tactical posts.

In short: There are no posts that are a grand slam for both traffic and leads.

3) Our post types all have very similar sources of traffic.

So if I wanted to improve an editorial post’s performance, for example, which traffic source should I focus on? To find the answer to that question, I needed to dive into the average traffic source breakdown of each post type.

When I did that, I found their traffic sources over time look almost identical. Email, direct, and social accounts for the most of the initial traffic, and then peters off. Organic stays pretty steady over time. Here roughly what most posts’ traffic sources look like:

Section Takeaways

All post types receive pretty similar traffic sources over time. The reason their overall average traffic differs greatly is because the volume of what they receive on each channel can be very different. So if I want to make improvements to different post types, I need to dive into each channel and compare how each post type performs.

4) Certain post types perform better than others on certain platforms.

Email

As you can see in the graph below, this channel has a huge spike in Month 1, then drops off almost completely. Here’s how the post formats rank on this channel:

1. TOFU
2. Deep Tactical
3. Infographic
4. The Rest (Editorial, Promo, Syndication, Tactical)

Even though this channel is circuitous by nature (our Daily and Weekly emails are a deliberate curation of our top posts of the day or week), all the top post types are the ones that are doing best on this channel. While it’s possible this means we’re influencing the above results, I’d suspect we’re just influencing the steepness of the graph -- these post types tend to do well anyway.

It’s also interesting (but not surprising) to note that almost all email traffic goes away after Month 1 -- for us, email is an ephemeral source of traffic.

Organic

Unlike all of the other traffic sources, success with organic traffic in Month 1 doesn’t predict success in Months 2 - 6.

As you can see below, the TOFU category of posts doesn’t follow this pattern. It’s much more successful in Months 2 - 6 than Month 1, and receives the most organic traffic of all the post types. (Note: This category has the smallest number of posts in it, so its line below is probably higher than it is in reality.)

But besides TOFU, the rest of the post types receives similar levels of organic traffic over time. Deep Tactical and Tactical are the next most popular -- they’re the most immune to traffic decay over time.

Social

Like Email, traffic from Social tends to spike in Month 1, then peters off during Months 2 - 6. In Month 1, the posts reaping the most average traffic from this channel are:

1. TOFU
2. Promo
3. Infographic
4. Deep Tactical

But after Month 1, most of the social traffic drops off completely (and Deep Tactical is most immune to the traffic decay effects).

Direct

Direct also drives most traffic to each of the post types in Month 1. Deep Tactical and TOFU posts receive the most direct traffic in all of the six months.

Referral

Of all the channels, Referral drives relatively little overall traffic. Like the other sources, it drives the bulk of traffic during Month 1, primarily for Infographic and Deep Tactical. What’s most interesting in this channel is that it’s the only other time Infographic gets more traffic than Deep Tactical in Month 1 (the other source is Social).

Section Takeaways

Most of the traffic sources above are important to driving traffic in month. The only one that drives long-term, sustainable traffic is organic. This isn’t super surprising -- it just confirms what we’ve been preaching all these years.

When we dive into each of the traffic sources, TOFU, Deep Tactical, and Infographic/SlideShare dominate -- but the channels in which they shine are slightly different:

• TOFU: Almost all, especially Organic. This is the only post type that receives more traffic over time than it does in Month 1, most likely due to the fact that this type of post is more relevant to a wider audience of people than our typical readers. For example, one of our top TOFU posts is around funny out of office replies -- something that marketers relate to, but isn’t applicable to just marketers.
• Deep Tactical: Mainly Email, Organic, Direct, and Referral. The main difference between Deep Tactical and TOFU sources is Referral -- Deep Tactical does way better on that channel. I suspect this is because these posts are more helpful, so people are more likely to recommend them as a resource/source for their articles.
• Infographic/SlideShare: Mainly Social and Referral. Since these posts are heavily based on popular external content, it’s not surprising that they take off on two visible, sharing-oriented channels.

Part 2 Conclusion

Though most of our posts tend to follow similar traffic patterns over time, there are some posts that perform better than others. “High return” post types for traffic are TOFU, Deep Tactical, and Infographic/SlideShare posts, while the “high return” post types for leads are Promo and Tactical.

Since there’s no one silver bullet for traffic and leads, we’ll need to have a mixture of both on our editorial calendar to reach (and exceed) our goals.  

What We’re Going to Do About This

Takeaways & Recommendations

For us to more efficiently grow the blog, here’s what I recommend we do.

1) Keep our Benchmark frequency constant.

This phase gives us the highest number of views for the effort we’re putting into each post. The HVLC phase only generated a few thousand extra views -- we can make up that discrepancy in other, non-frequency-related ways.

This phase also still returned a decent number of new leads. Though the HVLC phase returned the most leads of the bunch, in the overall scope of our blog’s lead generation capabilities, choosing HVLC over Benchmark will only increase our total lead output by 1%. Since it takes way more effort to run a HVLC editorial strategy than the Benchmark strategy, I’d rather get smart about making up for that 1% of leads with a different editorial distribution.

2) Increase the number of Deep Tactical posts we publish.

They drive more traffic on average than most other post types, and they continue to bring in relevant traffic over time. (Note: Posts that we’re updating and republishing will count toward this weekly number.)

3) Slightly reduce the number of Tactical posts we publish, but lean more heavily on keyword research to pick their topics.

This format doesn't do especially well -- except on organic search. (And it could be doing even better.) Doing more up-front research when picking topics will help us increase our organic traffic over time (and provide fodder for future post updates). 

4) In time that we would have spent writing or editing Tactical posts, produce more TOFU and Infographic/SlideShare posts.

Both of these post types generate lots of short- and long-term traffic for very little effort.

But I know what you're thinking -- since TOFU posts are the best-performing traffic posts, why wouldn't we overload on them even more? It all comes back to the fact that we are a business -- and this post type doesn't generate much bottom-line results. While TOFU posts help increase our audience size and diversity, Deep Tactical posts do a better job of addressing both the traffic and lead gen goals.

5) Continue to use Syndications to boost weekend traffic.

They take very little effort to put together, but give us a decent short-term boost. Since we’re currently using them to drive short-term traffic on the weekend, and the increased Deep Tactical, TOFU, and Infographic slots will increase traffic in the long-term, I’m not worried about them taking away from our growth.

6) Plan for Editorial posts roughly 1X a week -- but do more upfront research to determine topics.

These post types help us reach a more sophisticated audience, so we shouldn’t eliminate them altogether. Since they’re currently more of an average short-term play, we should do more upfront research on search and social trends to guide topic selection -- and re-measure this format’s effectiveness in another 6 months.

7) Build in two slots a week for Promo posts.

If we reduce Tactical in our editorial repertoire, we’ll be reducing the number of leads on average new posts are generating. To make up for that change, we should dedicate two Promo slots a week to new and old offers.

Our New Editorial Distribution

This is what our old weekly editorial distribution looked like:

And here's what we're going to switch to:

If we follow this new model and each post type generates the average number of views it’s supposed to get, we could be generating 18.5% more traffic each month -- and because of the compounding effects of blogging, the new strategy will generate an increasing number of views and leads over time.

Of course, there is always a trade-off. With this new editorial strategy, we will lose 3.4% of leads from new posts each month. But because new posts don’t contribute to much of our monthly lead gen, this change will not make a difference in our overall lead gen -- the old posts we’re optimizing, updating, and republishing will continue to generate the lion’s share of our leads.

Over to You: Tips for Running This Test on Your Own Blog

Just because we've been around for a while, doesn't mean the Marketing blog is done growing. By running these tests, we've been able to figure out how to grow faster and have a bigger impact on the business as a whole. 

But I'm not going to lie: This experiment took lots of time, manual labor, and developer help -- which isn't a reality for many companies. If you decide you'd like to run a similar test, here are some tips I'd recommend.

• Start assigning your post types now. It's much easier to categorize your post types ahead of time. If your marketing software lets you, you can add tags to your post types to categorize them. Otherwise, I'd suggest using a Google Doc or Excel spreadsheet to keep track of them all. 
• If you haven't done the previous bullet and you have a lot of posts to categorize, get your team on board to help you do it. Buy them pizza. Or beer. Or both. They'll be crucial to helping you get this done at a decent pace.
• If you can, get some dev help -- any dev help -- to help you pull traffic and lead data. Otherwise, you will probably have to do it by hand. Not only is it tedious, but it also opens you up to more possibilities for errors. It's worth spending the resources or money on a developer to help you get the information you need.
• Brush up on Excel ahead of time. When I first started this project, I stumbled through formula after formula. Save yourself the pain and just learn some Excel basics. (Here's my favorite post that I bookmarked and used to help me analyze this data.)
• Get okay with "good enough." As much as you want to control all the variables and remove extraneous data before you get analyzing, sometimes, you just need to work with what you've got. 

With the right experiment in place, you can get beyond the quality vs. quantity debate and answer the questions that matter: What do your readers like reading, and how often do they like reading it? 

Have you ever run an experiment to find your optimal post frequency and type? I'd love to hear from you. Leave a comment below or tweet me @gsosk.

Read more...
via IFTTT