Monday, January 19, 2015

The Murder of The Mystery Genre (And Its Rebirth)

I keep reading articles about how the Mystery genre is in trouble because it's readership is aging.  Young people aren't going for this genre like previous generations!  We're doomed!  It's the DEATH of Mystery!

These prognostications puzzle me, because this is old news. It happened back in the 1990's.  And it wasn't just a death.  It was murder.

And what we're seeing right now is actually a recovery of a sorts.

Maybe people don't realize that because few of the prognosticators are of an age to remember it all that well.  (And honestly, some who are older didn't realize what was happening at the time.)

So I thought I might explain a little about how we got here, and then maybe next time I'll propose some solutions, though I'm not sure we need any.

A History Lesson -- How Things Used To Be

This might be news to a lot of people in publishing right now, but for three quarters of the last century, the whole publishing industry and bookselling in general, worked a lot like Amazon does now. 

The point was to find and deliver the books that the audience wanted to read.  Every bookseller was thrilled to match a hard to please reader with an esoteric book.  Publishers considered their backlists to be their bread and butter -- and were perfectly happy to let an author, a series, or even an individual book take time to find an audience.

There were two things that made this possible.

One was large print runs.  Back in the day, publishers would routinely print huge numbers of a book -- even ordinary books -- and keep large quantities in a warehouse for later.  It saved a lot of money on printing to do these huge print runs.

And just as with ebooks today, it made complete sense to have backlist available every time an author published a new book -- because there would always be some new readers who haven't read the old book, and they'd buy both.  The more books a publisher had in stock from the backlist, the more they could upsell to customers.

This is, of course, how Amazon works; when a customer finds what he or she wants, be sure to show them as many related items as possible.

The other thing that helped this system work was that the distribution system was broad and varied.  An awful lot of those books, especially paperbacks, were distributed by small jobbers -- people who would maintain racks of books in all kinds of stores.  (Which was how most people bought books -- at corner stores and grocery stores and hardware stores, etc.)

These jobbers knew their audience, and it was worth their while to serve small communities and little stores.  They didn't have to worry about whether a book, or genre, or series was selling enough to pull it's weight with an international chain. They only had to worry about their own customers.

And because publishers kept all those books in print, and kept warehouses full of backlists, etc., the jobbers really had the flexibility to serve their audience the way that audience wanted to be served.

Add to this to the fact that libraries were healthy and well used, and you have the perfect environment for the mystery genre to thrive.

The Mystery Audience

Here's the thing about the mystery audience -- it was huge.  It was the biggest genre of all, and held that crown throughout most of the century. 

Mystery, for most of the century, wasn't really a genre in the sense of being only one kind of story.  It was just a set of plot types that crossed over with every other genre.  Everything had a mystery in it.  And mystery had everything else in it.

So you could say that mystery thrived not so much because the genre itself was popular, but because whatever you liked, you could find a mystery of that sort.  Serious, scary, funny, romantic, gritty, realistic, historical, comforting, disturbing.

And the mystery audience developed another peculiar habit: We loved long series.  Long long long long series.  We wanted our series to be like a genre in itself.  And though we loved it when our authors wrote four or more books  a year, we also remained loyal to series that only appeared every decade. (Tuppence and Tommy, anyone?)

As a result, mystery readers of previous generations had a peculiar habit: We often didn't start a series until there were a BUNCH of books in it. 

Because we treat our series as kind of like a genre in itself, we're reluctant to try new authors and books.  We sniff and circle, and squint, and then, as often as not, if it's a first or second book in a series, we decide to wait and see if more books will appear in the series before giving it a try.

After all, it's somewhere around five books that a mature series really hits its stride.

And in the old days, publishers and booksellers didn't mind this because they knew that, once we actually decided to buy, we would consume it madly.  We wanted it all, we wanted it now, and we wanted more.  They could count on us forever and ever.

So it was worth taking time to let a series develop.

Mystery readers still hold this attitude.  We have definitely evolved, and have normal variations in demographic -- but go to a mystery oriented group and ask them "how many books do you like to see in a series before you try it?" and they will answer in the range of 5-7.

Keep this in mind as I tell you about what happened next.

The Crime's Backstory - Thor Power Tools vs. the IRS

So... remember how I said that publishers would print these HUGE print runs and just hold them in a warehouse to sell over many years?

Well, one of the things that made this possible was that the publishers didn't have to pay taxes on the money they spent on printing.  It was an expense, not a profit.  You deduct expenses.

But in the late 1970s, the IRS went after a little tool company called Thor Power Tool Co.  They felt that this practice of buying up of supplies ahead of time was a tax dodge.  It went all the way to the Supreme Court, and the Court found for the IRS.

And that knocked a leg out from under the publishing industry.

Suddenly, it was not so profitable to print such large print runs.  And there was no longer much incentive to keep books around in the warehouses. As a matter of fact, it was the reverse: if you shred the books, you can claim them as a loss and don't have to pay taxes on them.

And because of that, there was less incentive to continue any series or author.  In terms of cost and risk, every book was now kinda like a first book.

And yet... the publishing culture was still the same.  They still saw the mystery and series fiction and backlists as their bread and butter.

Because, after all, book people are not really very good business people, and the editorial side didn't really see the ramifications.  So when new business practices hit, they fell back on their instincts and just kept doing what they always did....

The "Overbought Mystery" Crisis of the Late 1980s

It was harder to make a buck at the old publishing model now.  Furthermore, publishers had new big corporate overlords who demanded more out of their bottom line. 

So editorial staffs did what had always worked before: they started buying up new mystery series (and science fiction trilogies, I hear).  Those were their bread and butter!  If they wanted more bread and butter, they needed more series!


Except that these were new series.  And as I said above, mystery readers are slow to take up new series.  They want new series, but they want them to ripen first.

And though publishing people wanted to give it time... but their corporate overlords were not happy.  And there were a great many firings of editorial staff who bought those new mystery series.

Which meant that those who were left were not mystery-oriented, and were also afraid to commit to anything, least of all backlist books and mysteries.

And then the deathblow came.

The Weapon: Automated Buying Systems of the 1990s

The thing is, readers still loved mystery -- and series and genres and backlists.  It wasn't like those books failed in the 1980s because readers changed.  The readers still wanted what they wanted.

So publishers struggled along to provide these things. Even if it wasn't as profitable, there wasn't an alternative.

Then something terrible happened to distribution: Two things, actually, both related to big box chain stores.

First the small "jobber" distribution chain collapsed, because the big retailing chains decided that they only wanted to deal with one or two distributors.  Overnight, the small distributors -- those who supplied the jobbers for small stores, and also for independent booksellers-- disappeared.

The second one was less obvious, but in my opinion, more insidious: Borders invented an automated book distribution system that seemed at first like it would do what the small jobbers used to do.  It could track those smaller niche markets and provide those smaller retailers and customers with the books they most wanted.

Except that's not how it ended up working.  Remember the whole Corporate Overlords and the Bottom Line thing? That's still in play. That's even more in play.

So by the time Barnes and Noble started doing the same thing, it was no longer about providing niche customers with niche books.  It was about the Pareto Principle instead.

The Pareto Principle is a realistic observation, that turned into an extremely destructive practice.  The observation is this: if you look at your products or your customers, you will find that eighty percent of your profits come from twenty percent of your products or customers.

Of course, it turns out that the less profitable eighty percent is still needed for that profitable minority to work.  So you can't just cut them out.

And though the corporate overlords kinda, sorta knew that, they still put their main drive into cutting every bit of cost you could on that less profitable eighty percent.

For the big distributors, it was all about raw sales numbers.  Forget the niche thing or customer satisfaction thing.   Only raw, total sales numbers mattered.

And this resulted in a strategy called "Churn."  It worked like this:

Assume you need the eighty percent of books which are less profitable to find the twenty percent that is more profitable.  Fine, continue to offer lots of titles -- but only keep the titles that make it to the top tier.  Toss the rest and bring in new.

And because it's a numbers game, don't waste time with analyzing it.  Providing the titles -- the fodder -- is the headache of the publishers, so let them do that.  The distributors can just let the system crank and reap the profits, no matter how destructive it was to the industry.

Suddenly, it didn't matter why a book might fail.  The distributor didn't care if there was a catastrophic misprint (like the first chapter was left out), or if the book had it's launch on September 12, 2001.  Sorry, that's too bad.  The numbers do not lie, and the numbers say that the book failed.  So it's a failure and should not be reprinted.  No matter how many pre-orders or requests.

To the distributor -- who doesn't invest in the book up front -- it's more profitable to just force the customer to buy something else.  It's not like the customer had a choice in those days. If the book you wanted wasn't distributed, you were out of luck.

But it was worse than that.  Those bad sales numbers were now a permanent part of an author's record.

When the automated system decides what to buy next, it looks at how many copies the last book sold.  Not how many they ordered last time, but how many actually found their way into the customers' hands.

And this happens no matter what the buzz.  No matter the requests or how eager the audience is.  No matter what amazing extenuating circumstance has turned the author into a star in the meantime.   If the last book sold 3538 copies, then that is all that will be ordered.

Now, to sell 3538 copies, you might need 5000 copies out there in the book stores.  No matter how few you distribute, not all copies will sell.  Because even if there are 5000 customers who want the book -- or even more -- the copies are scattered around and those customers won't see them.

If you order 3500 copies, you'll probably only sell 2000, no matter how many customers want them.

So it doesn't matter if the audience is growing; that author's track record says he's on the way down hill.

The third book, if there is one, will only have 2000 copies distributed, and will only sell 1200.

And if that falls below what the distributor considers viable, the author is then blacklisted by the system.

Even when authors did manage to stay above the cut off line, it was almost impossible to have a break out -- because the system would never order more of a later book.

This is why, back in the 1990s, authors started changing their pen names every three books or so.  It was actually the publishers who started this as a desperate attempt to beat the system.

What Happened To Mysteries?

Mysteries were really vulnerable to this new system.  With the new cut off for a series or career being about three books, it was really super hard to reach that magic five to seven number that the mystery audience was looking for.  Suddenly the whole genre changed.

Or more specifically, it narrowed and split into three: 

1.) There were the well-established authors who had already gained a good audience before the crash, and they survived the distribution problems.

2.) And there were the high-powered thrillers that appealed to the general best-seller audience.

3.) And there were the "hook" books.  Series about cats, or catering, or wedding planners. (Or Vampires or Witches.)

Now, these books existed before the crash.  They weren't a change so much as they were what survived the crash. And they survived because they had a kind of immunity to the numbers games.

Lemme explain:
Before the crash, most mysteries didn't have a hook. This is why the readers circled them slowly before buying. The reader took on a new series as if it were a new member of the family.  You're really looking for a new favorite.  You were looking for a real commitment.

On the other hand, if you really love Siamese cats, you might buy a mystery about a Siamese cat even if you aren't sure you'll love it, because... Meezers!

Such a book still has to be good enough to get you to buy the next one, but because it had a lower "barrier to entry" so to speak, the first book would sell in relatively higher numbers.  Which means the second book would get a larger print run -- and because it was also about cats, it had a better chance of catching the eye of someone who didn't see the first.

This is not necessarily true of a twisty domestic crime story with a detective who has no outstanding "hook" quality, but who touches your soul.  That might be the kind of book you're looking for, but when you pick up such a book, you can't tell at a glance if you've found it.  So you're more likely to wait until there are more in the series.

So before the crash... some of the strongest series were those that couldn't easily be described, and often the very best thing about those books was that they featured a twist that you didn't want to know about.  It would spoil the book!  With those sorts of books, the audience couldn't tell what to expect.  They were taking a risk, at least until the author was famous enough so you knew what to expect.

But after the crash, the strongest series were those which could show the audience what to expect from the start, so they wouldn't hesitate in buying.

And at first that had a bad effect on the genre.  Because once the desperate publishers, who had invested in all these books Barnes and Nobel was killing off with abandon, learned that gimmicks worked, and they gamed the system like mad.

And that did two things: It gave mysteries a bad name to many new readers.  (This is why those Nielson numbers saying that the mystery audience is largely over fifty-five -- that the age that remembers what mysteries were like before the crash.)

But it also drove away a lot of the existing audience. Many of us stopped buying new mysteries. We stopped even looking at the new mystery shelves.  We went, instead for used books.  Or fled to other genres (because, remember, other genres often have mystery and suspense plots).

And, of course, this shrinking audience made mystery less important to Barnes and Noble's all important algorithms, which meant that even the best books of these narrower categories got canceled, or were forced so far to the margins, that the author gave up for lack of a decent income.

The genre almost died in the 1990s.  So many promising series (both of the older type, and of the hook type) simply disappeared just as they got interesting.

But it didn't quite die, because....


The White Knight

Yes, Amazon is our White Knight who actually saved publishing from those automated ordering systems.  Publishers are crazy not to realize this.  (Go ahead, fear the power of the giant, but do not forget what it did FOR us.)

First, Amazon made it possible to get a hold of those books that didn't get wide distribution. So if you were one of the 5000 customers who wanted that book that never made it to your local store, you still had a shot at finding it.

(You young whippersnappers have NO idea what it was like before Amazon.)

In some cases, that meant that publishers could keep a marginal book in print.

Second, Amazon started selling used books.

At first publishers and authors were horrified, but soon it became clear that for those authors who were being dropped, that their fans could still find their books.  And for those authors who had written, say, a trilogy, and the distributors refused to make book one and two available, customers could still find those first books used.  Which was critical, because customers won't buy book three if they can't find one and two.

That was life support to many genres, but especially to Mystery.  It kept those of us who had left the genre going, until sanity finally started returning to the publishing industry.

The Return of the Back List

The third thing that Amazon did to save the mystery genre was the Kindle, which caused ebooks to go mainstream.  eBooks removed the cost issues that came up with Thor Power Tools.  They are the equivalent of every book having an infinite print run.

But even before that, I noticed our genre slowly broadening out from those horrible years of "gimmick or die."  And we're seeing more and more series lasting more than three books.

We were in BAD shape for a while there.

Heck yeah our audience is old.  The top of our genre is mostly made up of classics, and authors who established themselves before the shake out.  The current genre is still narrow, except where it had blended with other genres.  Our newer authors are like orphans who pulled themselves up by their bootstrapes: they have grandparents -- whose experience is from another world -- but no parents to help along with anything recent.

And in spite of the fact that so many of us in the audience left the genre back in the 1990s, it's still a big genre.

A big genre with a tremendous amount of room to grow.  There's so much variety to mystery that really isn't being explored right now.  And a whole generation that isn't familiar with it.

Yes, the audience that identifies as formal mystery readers is aging.  But, you know, so is the population.  And the next generation?  They're interested in crime, and solving problems and puzzles, and in suspense and in twists and in good and evil.

Everybody is.

So it is my opinion that we aren't seeing the fading of a genre, but rather are in the first stages of a rebirth.  And no, we don't need to do anything, because what the next generation wants, it will get.

However, I DO have some ideas about what we, as authors and publishing folks, can do to nurture our own revival.

But I'll get to that next time.

See you in the funny papers.


Elizabeth Spann Craig said...

A very interesting history on mystery publishing and reader habits, Camille! I enjoyed it.

I do believe there will definitely be a rebirth of the genre, beyond thrillers. I know that my *own* readers are older just through the ones who contact me on Facebook and I'm trying to hook in younger readers, too, which seems to be working. I was, most definitely, an odd kid, but I loved "Murder, She Wrote" as a teen and Agatha Christie.

I think you're right about the hook mysteries and their easy appeal/low risk. I remember being shocked when my Myrtle Clover series canceled--but it didn't have a hook, which I now think is a big reason it was axed. Worked out well for me in the long run, though...through Amazon.

Peter Spenser said...

This is a great post!

It is, without a doubt, the second-best analysis of the publishing industry that I have ever read. The first (only because it is longer—in fact it might be something that you would want to look at yourself) is a book by Kenneth C. Davis called TWO-BIT CULTURE: The Paperbacking of America, published in 1984 by Houghton Mifflin. It is specifically about the paperback book industry, but it’s such a good source of all sorts of information about the history of publishing that it was plagiarized in 2012 by some Internet columnist and then re-published on-line again (and again uncredited) in 2014.

I have signed up for your newsletter and am looking forward to your next post, where you will talk about “what we, as authors and publishing folks, can do to nurture our own revival.”

Best wishes.

Casper Bogart said...

About as good an explanation of the history of our genre as I've ever read. Thank you for taking the time to write this.

Unknown said...

That was great. Thank you for the history lesson.

The Daring Novelist said...

Wow, I go out to run some errands, and come back to a burst of visits and comments!

Elizabeth: you're ahead of the curve on this, and I think it's because you've just persisted along with what you do. You listen to your audience, and follow up on ideas that make sense to you, and don't get caught up in the kerfuffle of various publishing wars.

Peter: I probably read Two-Bit Culture back when it was published, but I don't remember it. I'm sure it influenced my understanding of what was going on around me.

Casper and Rob: thanks! I've been meaning to write about some of these issues, because so many people clearly don't know anything about them.

Unknown said...

Great job 'splaining things ;)

Markets are funny things, aren't they? While I'm grateful for Amazon's current model, it's important to remember that Amazon acts in its own self-interest, always. When the tide shifts again, the market (and Amazon) will adapt. Our job as writers and publishers, is to make sure we're prepared for change.

Adam Netherlund said...

Awesome post, Camille! Lookin forward to the next post.

Elise M. Stone said...

An awesome post, Camille! I've read most of this history before between Kristine Kathryn Rusch and Holly Lisle, but you added new insight for mysteries, which are my favorite genre.

I quickly tired of the "hook" series and wondered why I couldn't find more new series that appealed to me. Thanks for explaining the five books factor. I have hope now that publishing is no longer totally in control of the mega-corporations.

Looking forward to the next installment.

The Daring Novelist said...

T.D.: thanks. You know, the thing to think about Amazon is not whether they are a benign friend or a ruthless adversary, but rather that they are a Force of Nature. If we understand them -- what they are really doing and why -- they become a tool we can use.

Adam: thanks. I hope the next post doesn't disappoint. (I can give you context, but it's harder to recommend actions.)

Elise: Hooks will always be valuable, but I think in this internet age, the Power of Personality (i.e. the personality of your sleuth, of your books, and even of you) is the biggest power.

Judy Penz Sheluk said...

Super post. A fascinating look at the publishing industry.

susanalbert said...

Camille, excellent post. As a mystery author (my China Bayles series began in 1992 and is now in the 24th book) I've been watching this history develop. You've done a very good job describing the process. Gotta say that the field looks much more open these days, now that writers have more publishing choices.

The Daring Novelist said...

Thanks, Judy!

Susan: I always thought that your China Bayles was a book that threaded the needle - it's character-based ("The China Bayles series") but it also has just enough of a hook (herbal mysteries) to help it along in the new environment.

Kat Sheridan said...

Camille, excellent article, and one I'm passing along to friends. I don't write mystery (well, sort of: old school gothic romance, which is even more niche and weird), but I love reading them. I wondered why I was suddenly left with books that weren't really doing it for me. Great analysis!

The Daring Novelist said...

I would LOVE to see a return of the good, old-fashioned romantic suspense -- especially the gothics.

When I was a kid, those were mostly shelved with the mysteries. Because... they WERE mysteries.

Gloria Alden said...

What a great blog, and one that makes me so glad I decided to Indie Publish, both because I can write what I want, but also because I don't have to be worried about the sales of my books. I'm working on my firth right now and quite content with my choice.

Lesley Diehl said...

An insightful history. I feel better about my struggling mysteries now, and you gave me direction for where I go from here. Thanks.

Michael Ryder said...

Fascinating and insightful -- thank you for posting.

As a long-time reader of mysteries and thrillers, I appreciate learning how a tax ruling, an ordering system, and my own reading preferences (as so accurately described by you) shaped a genre I've loved since I was a kid.

As a writer, I'm encouraged by your optimism for the future.

I await your next post with bated breath!

The Daring Novelist said...

Thanks, Gloria, Lesley and Michael!

Yes, enabling indie publishing is yet one more thing that Amazon has done to help the mystery come back. It allows us to start as slow as we need to, for one thing.

Melissa Yi/Yuan-Innes said...

Camille, I call a +1 to buy a book. If they've never heard of me, but they want to read a medical book, +1. If they want to buy something for their kids, +1 for my kids' stuff. Etc. Since I write across genres, I can see the +1's adding up, or not.

That's what I was thinking about for your lower barrier to entry with Siamese cats and vampires. Cool. As always, thanks for sharing your brain.


Leigh Lundin said...

Melissa (above) drew my attention to this interesting thesis about the Goliath– er, Amazon– we love to hate.

I recognize much of this as true and realize from the article how early 20th century marketers set a pattern. For example, the Stratemeyer Syndicate didn’t wait for a series to accumulate volumes… they often released the first four or so at once, offering a ready-made reading list. At the end of each novel would be a sentence, something like… “As exciting as this case was, the Hardy Boys faced greater danger in The Lost City of Limpopo.”

One of my friends took an evening job at Borders. None of us knew why at the time, but she was intensely frustrated that staff would order books that would be ignored at the corporate level. Local stores were shipped what corporate demanded they sell, not what customers wanted.

The article’s also right about former buying venues. I used to seek out kids’ series in a store something like Woolworths, more than a dime store but not a big city department store. And perhaps it was a Woolworths. It had a nook with inexpensive hardbacks for children and youths. But now in towns across America, there are no places to buy books. A buyer either has to order on-line or visit a city because sources to buy have either consolidated or dried up.

And if one buys on-line… who comes to mind first?

Bill Peschel said...

Long post, but well worth it. It combines all the threads I've been encountering since the '80s, with the industry's response. Great job.

The Daring Novelist said...

Thanks, Melissa, Leigh, Bill!

Leigh: a lot of indies are using many of the techniques of the Stratemeyer syndicate. I also expect that some traditional publishers will start moving toward a packager model -- hiring writers to produce the company's intellectual property -- but that's a whole different issue.

Michele Drier said...

A great look at publishing history and analysis of the current trends. Thanks Camille!

Lisabet Sarai said...

Thanks for putting things in historical perspective, Camille.

Now where does self-publishing fit into your analysis?

Dragonquillca said...

Fascinating post, and educational as well. I understand now why I bought the way I did, and you're right, I sniffed, circled and squinted while trying to decide if I wanted to make that reading commitment. I did it to fantasy as well as mystery. You've explained wonderfully why my beloved bookstore had less and less of what I wanted. I get it now. (They were still wrong) Now, more than ever, I want to open a bookstore. One that listens to readers and authors and makes a personal connection and numbers be damned.

As an author, I have hope now, thank you. I will continue to write what excites me and never mind writing to the numbers (which I never really did). If we write to the numbers then we write for the corporate overlords, not for our fans. Or ourselves.
I look forward to your next post. Thank you.

The Daring Novelist said...

Thanks, Michele, Carolyn.

Lisabet: Self-publishing is a part of the ebook revolution. It means even the most marginal of books can get into print and stay there. I didn't talk about it specifically in this article, simply because these forces affect the whole industry. Traditional publishing now has tools to address the problems, too -- however, they take longer to change course.

Also, because mystery readers are so careful about trying new things, indie publishing is one of the things that are slower to start in our genre than others.

But in both cases we are rebuilding.

(NOTE TO ALL: I hope to get the follow up post in today.)

chacha1 said...

This was great. I'm going to share it with my mum, who has bemoaned the State of the Mystery of late (she is a little behind the curve).

Will look forward to the follow-up. As to Amazon, well ... I wouldn't be published at all if it weren't for their platform.

Cindy Sample said...

Camille, thanks for the insightful post. I learned so much publishing history. I've been an avid mystery reader since I discovered Nancy Drew, which was when I determined that would be my career path someday. That particular path was delayed a half century but I was fortunate to publish during these exciting times.

Neil A. Waring said...

Great post, not sure it will die. I love westerns also and the old style has died but the newer modern day setting western mysteries seem to do well. Think the same about mysteries, old style may be gone but today's mysteries(with characters like Jack Reacher, Sheriff Walt Longmier or Joe Pickett seem to be doing well.

Rachel Heffington said...

As an author just edging into the mystery genre, this post was so helpful to me. I identify with that readership that doesn't want a hook and takes time before deciding to adore this series for ever and always. THANK YOU for explaining this! It is massively helpful to hear it all laid out like this. And three cheers for Amazon. What a ducky old fellow he is.

The Daring Novelist said...

Sorry so late in responding to comments on this post.

I want to thank people for commenting. I'm glad so many people find it helpful or enlightening.

This has become one of the most popular posts on my blog. I suppose it shows how much people crave context in today's world of instantaneous everything.