I went to Westercon a few weeks back and attended quite a few panels, including one on the evolution of cover art. Despite the fact that the presentation could have benefited greatly from visual aides (I find it hard to believe that they couldn't find anyone at a geek con who could create and play a slideshow), it spawned some discussion of how the great book covers manage to straddle the line between placing a book firmly within its genre, while standing out among other similar titles.
And while no one on the panel claimed to know the secret of finding that cover art sweet spot, one of the recommendations was to look at a lot of covers in your category/genre and get a sense of which ones make you pick up the book. I think this is particularly crucial with self-published books because you're already up against a perception that the writing and editing might not be up to par.
More and more readers are willing to give self-published books a chance, but the cover is seen as a reflection of how much time and effort the writer put into the book as a whole. Amateurish, poorly-designed covers are often a sign that the author wasn't willing to spring for a professional editor or book designer either. If you don't have a background in graphic design, there are so many people out there working on a freelance basis that there's really no excuse for grabbing your nephew's drawing off your sister's refrigerator and slapping it on your book.
But before this post gets too long, back to that advice to look at a bunch of covers--this has become a lot harder now that there are fewer book stores with sections to browse. You can certainly do a search for your genre on Amazon or Barnes & Noble and check out covers that way, or wait for Kate Hart to do more YA Book Cover Infographics, but I wanted to point out a few more helpful sites.
The Book Designer does a monthly roundup of e-book Cover Design Awards and though they are subjective critiques, what I find particularly helpful is that they include information on the designer. So if you like a particular cover, it's easy to do a search and get in touch with that artist/designer about what you'd like to see on your book.
And Design Observer spotlights 50 outstanding book covers; you can see winners and nominations for 2011 and 2012 on their site. The winning book covers going back even further are in the archives of AIGA. The individual artists and designers also comment on what the assignment was and why they went with that particular artwork, which can be a mini-seminar on the evolution of book covers in itself.
If you know of anymore sources for book cover design ideas, feel free to leave them in the comments. Or maybe give a shoutout for your favorite cover! This is one of my favorites: