When you start thinking about this question in depth, it can get pretty confusing. I personally spent many years mulling it over and experimenting with both approaches. But when you get it all untangled, the answer is pretty simple.
You should seriously consider self-publishing if you have an entrepreneurial personality.
What do I mean by that? By having an entrepreneurial personality, I mean you’re the type of person that has very strong feelings about anything you create, not only about how it is made, but how it is packaged, marketed and sold to the customer. You like to make all those decisions yourself. You like 100% control over everything you do. You are naturally good at presenting yourself, and anything you create, to other people. You are probably not a very good “team player.” You believe you are right most of the time, even when you’re not. And you probably (if you’re like me) loathe bureaucracy. There’s nothing that gets on your nerves more than long, boring meetings, and you especially hate decision-making by committee.
What does all this mean? It means that if you have an entrepreneurial personality, you’re not likely to get along with big publishing houses whether they want to publish your book or not. Big companies are bureaucracies, and they are going to do things their way.
If you are able to land a good literary agent (no small feat), and that literary agent is able to get a big publisher interested in your book (also no small feat), Big Publisher is going to want control over virtually every aspect of your baby. They will demand control over the basic product itself (editing and fine-tuning the manuscript), they will demand control over the packaging (the title, cover illustration, and jacket promotional copy), and they will demand control over the promotion and distribution of the product (advertisements, press releases, etc.)
And why shouldn’t they? It’s their money that’s being invested! Would it be any different if your were an inventor and took your brilliant new gadget to a big company and wanted them to license it, manufacture it, package it, and market and sell it through their existing distribution channels and pay you a royalty for each unit sold? No. It’s exactly the same situation. Big Company thinks they know best, and they are risking their money, so naturally Big Company not only expects but demands to have control over every aspect of the product that affects their return on investment.
Now, this doesn’t bother some people at all. Many writers say “Fine, I understand that, it makes sense—I would do the same if I were in their shoes.” And these writers are able to comfortably work with the big publishers and often achieve fantastic results.
If you are one of those people (I’m not), I envy you. I would strongly recommend that you go the traditional publishing route and persist, persist, persist until you break through and land that good agent, and get that sweet offer (with all the strings attached) from Big Publisher. (I have lots of helpful articles about that process here).
However, if you are not one of those people, you should consider self-publishing. I’m quite sure if I were ever present in a committee meeting at Random House where they were discussing the packaging of my book, I would end up stabbing some button-down executive to death with a letter opener.
There is one exception I would add, though, and it’s an important one. If you are the type of person who can get along well with Big Publisher (and you intuitively know this even at this moment, even if you’ve never had any contact with one), there is a case when you may still want to self-publish. That would when you’ve made a truly valiant effort to go the traditional route, but you continue to be rejected, not due to the quality of your writing, but because the publishers don’t believe there is a large enough market for your work (large enough for them to make their money back plus a profit).
In this situation, you can self-publish your book for the sole purpose of proving the market for it. If you can sell an impressive number of copies of your book on your own, agents and publishers will start getting interested. I’m not talking Amanda Hocking’s or John Locke’s numbers—10,000 copies are enough. Then, as Amanda Hocking has done, you can switch over the traditional route and let other people handle most of the editing, packaging and promotional details of your book.
Is 10,000 copies a lot of books? Yes, it is. And that’s where the downside comes in—if you don’t have an entrepreneurial personality, you may find that selling even 1,000 copies is beyond your capacity. So, in that case, it’s best to stick to the traditional route and persist, persist, persist.
I hope this helps. As always, comments are welcome.