The Lemons Meme in Software
A few weeks ago Bruce Schneier discovered a classic economics paper, “The Market for Lemons”. The paper describes the behavior of markets where sellers have detailed information about the products, particularly the quality of the products, that buyers do not have. It uses the example of used cars.
In these markets, the price buyers are willing to pay is defined by average quality of good. Buyers lack information, so can only assume they are going to get a product of average value. Unfortunately, this lower price drives the best products out of the market, because sellers (who know they have the best goods) won’t accept that price. When the best goods are removed from the market, the average quality drops, the price drops, and the next best goods are removed from the market. The conclusion is that in these markets quality falls until it matches the amount of information that buyers have.
Bruce applies this principle of economics to explain why there are so many bad security products. But others in the blogosphere have picked it up as a way to describe other parts of software. I first saw it at David Anderson’s blog, where he talks about the lack of information when hiring software engineers. The most stark application to the job market comes in a Reddit comment through:
I just realized that it applies to the IT job market. Here the seller (the applicant) has all the info about himself, while the employer knows nothing. So what happens is that companies expect the average, and pay accordingly. That’s why people who are smarter than average shouldn’t go on job interviews, because they’ll likely to get below what they are worth.
In the IT market it also helps to explain the pervasive use of certifications. Even if they don’t indicate that a candidate is good, they do cut out the bottom of the distribution of candidates (the truly terrible sysadmins). Since top people will have already taken themselves out of the market for these jobs, it’s ok to alienate them. Removing the bottom people pulls up the average, so prices (salaries) presumably rise.
As much as I like thinking about the talent market, my favorite application of this meme is Reg Braithwaite’s The Not So Big Software Design where he applies it to tract housing (a pet issue of mine) and by metaphor to custom software development. The customers for both new houses and custom software are definitely ignorant, and they tend to buy based on what we in the software industry call buzzwords. For new home buyers, these are things like granite countertops, en-suite master bathrooms, the number of bedrooms, etc. Builders optimize for these easy-to-observe items, at the expense of things that can be critical to livability or maintainability of homes.
It’s a good discussion of the realities of custom software. The metaphor does tragically fall down though. At least buyers of used cars and tract housing can resell them to the next ignorant buyer. Companies are generally stuck with their custom software.