Latest Publications

Lessons Learned from Startup Weekend

A year ago this month I attended startup weekend in Boston. The company that I helped to launch there, http://sellercrowd.com, is now a successful ongoing concern, with funding and a handful of employees. So I think that counts as a success story for startup weekend, and my conclusion is that it’s a great program and can work well if you make the right choices going into it. To that end, I’d like to share some of my lessons learned from my first startup weekend, and solicit anyone else to add them in the comments.

First, always come with an idea. Create a good pitch. This is the way the other participants will know you, and it’s worth giving them a chance to get a sense of you in this context. You don’t have to bring your best idea to startup weekend, but it helps if you can bring your best startup weekend idea.

Second, don’t be too attached to your own idea. In fact, you might get more out of this if you can join another team that has an idea that is even better suited to startup weekend than your own. Very few good ideas can be implemented and validated in 48 hours. Plan to pick the best one, and don’t worry if it isn’t your own. You can always take what you learn and apply it to your own idea on a more practical timetable.

The best ideas have a few unique characteristics that make them fit into the timeline, audience, and competitive aspect of Startup Weekend. A few principles of idea selection:

  • Select an idea that fits into a weekend. Ideally you will want to validate your idea with customers, build a prototype, validate more, prepare a demo, and make a presentation. That means you can’t invent any substantial new technology, you are looking for something that is class-project sized, not 20 engineer months. If you have a big idea, cut it down until it feels like it might fit, then cut it down by another two-thirds.
  • Appeal to the audience. Present an idea that people in the room will want to use. That makes it easy for them to understand and get excited about, and will give you your first customer validation. The people in the room will nearly all be younger professionals in the tech sector. Ideas related to socialization, recruiting, and personal services are easy to get excited about. Ideas related to financial risk management or construction not so much. Surprisingly, medically oriented ideas got people pretty excited, I think because “doing well by doing good” is hot. Feel free to trend-follow here.
  • Appeal to the judges. It’s a competition and people like to win. Choose an idea that can be validated in a weekend. You probably won’t get any CFOs on the phone on a Saturday, but you can go outside and find consumers. Don’t choose an idea with huge market risk, or where your team will be clearly unqualified. Choose an idea that resonates with current investing trends, with an additional hook.
  • Explain why you are the leader. Ideally come with some kind of pre-validation, and include that in your pitch. “I already have a list of 20 prospective customers who are willing to do interviews this weekend, and I need people to do the interviewing and build a prototype based on my vision and feedback” makes your idea feel much more achievable and worth joining.

When you are selecting an idea (including if you select your own), you are actually selecting a team. So if you can find someone with an appropriate idea, a great idea, who also has the personality or presence to pull together a team of people in the chaos that is startup weekend, all the better. The number one thing you are looking for in a team is a good breadth of skills and a good number of people, enough people and skills to get something real done and have someone paying attention to all the parts.

Beware of teams with only 1-2 engineers unless the idea is super simple, and avoid teams with more than 3-4 business people. If your idea needs a designer, your team needs a designer. In my experience, the designer was the most valuable member of the team (even though he hadn’t ever done startup web design before) and the rarest find. The Startup Weekend organizers try to keep the skills balanced, but it’s never perfect. Too few engineers and your engineers (maybe you) will be death-marching for the weekend. Too many business people and they will be off on distracting tangents.

Usually you are going to be implementing some kind of application, web or mobile, whether it is a prototype or not. Choose a team where the engineers have good skills overlap and can quickly converge on a technology (e.g. Ruby on Rails, Python Django, node.js and seven other libraries). Avoid science projects. Avoid any devops, host on Heroku or someone’s personal machine or whatever someone already knows. We lost 3 engineer hours to fighting with Amazon hosting, which was completely uncessary. Choose technologies you know and that you know to work, and focus the risk on the application you are building and the ridiculous timeline.

Speaking of building things on a ridiculous timeline, the best advice is to begin with the end in mind. Know what you are going to be doing at the end, which is presenting for 5 minutes to a group of VC/Angel/Entrepreneur judges who will want to see that (a) you build something impressive, (b) your idea is coherent, (c) there could be a business here, and (d) that business would actually do something important. To that end, start building a demo script and a PowerPoint as soon as you have converged on what your idea is. Make sure that whatever you build as a product supports a well-scripted demo, and that whatever work you do around customer validation fits into the pitch.

Finally, don’t sweat the ownership until you have something real. This is the trickiest aspect of startup weekend. Failure is an orphan but success has many fathers, as they say. In our experience, having created something successful that was on track to get funded, we were able to more than satisfy those people who helped at startup weekend with a token amount of stock (a few hundred shares) once we made clear the additional work that was going to be required. Of course, it is always possible for someone to make a stink, and this is why you should be careful about what ideas you choose for startup weekend.

I hope these tips are helpful to anyone doing SW, and if you have more please add them in the comments. I encourage everyone with an interest in startups to try out Startup Weekend. The environment is very high energy, and you get nearly immediate feedback on what you are doing. Concentrated learning and coworking in such a short period of time is great fun. And if you are lucky, you may come away with a startup you are really excited about.

Recommending the Franklyn D Resort (FDR) in Jamaica

I just got back from a week in Jamaica, and I’ve been talking off everyone’s ear about the experience. That’s because the Franklyn D. Resort has one major thing going for it: every family gets a full time nanny for the duration of your vacation.  Larger families, particularly with multiple age groups of kids, often had two nannies. The resort seemed especially friendly to large families, with big multi-bedroom suites and reasonable rates for additional children. For our trip they were also running a “grandparents stay free” promotion, which I think they do every January. If you are planning a family-oriented vacation, particularly with young kids, you have to consider the Franklyn D. Resort.

So while I’ve been recommending the resort to people verbally, I wanted to take a moment to write down what I learned vacationing there, and what people might want to know before they take my advice and book their own trip:

  • Having a nanny while you are on vacation, as I mentioned on Twitter, is amazing. Compared to my week in Puerto Rico last year with the baby, this was total relaxation. I had exactly as much toddler time as I wanted. And I was lying in the hammock, and Patrick wanted to play basketball, Carlene our nanny was happy to take him, and I could go with them or catch up later. I was able to do my own things, like trying out scuba diving, without having to negotiate childcare with my wife.
  • The resort is wonderful, for what it is. It is locally owned, and definitely not maintained to Disney standards. They stay on top of maintenance, but there is still chipped paint to be found, overgrown plants in places, and parts of the pool deck collect water. But the price is good, the staff members are wonderful, and I didn’t have any facility problems that actually impacted my vacation.
  • If you want to spend time with your kids, you have to be really clear with the nannies. Their default mode is to whisk the kids off into kid-oriented parts of the resort, where there are lots of activities, lots of other kids, and other nannies to socialize and share duties with. Our son was a bit sick, and tended to be clingy. But whenever he wanted to head off to the playground, our nanny Carlene was happy to take him. By day three we realized we were missing him. But it took a couple of conversations to get to the right balance of Patrick-directed, Carlene-directed, and parent-directed activities.
  • “Gratuity included” is only technically true. While the prices and the organizations is gratuity included, many of your fellow resort-goers, particularly the repeat customers, will be ignoring this rule. Tips are not large, mostly one dollar US or one hundred Jamaican (which is about $1.20), but they help to get you priority service at the bar and at breakfast. If the resort had been more crowded, they might have been more necessary.
  • Nannies are happy to stay late, but you won’t get a lot of guidance from the hotel management. Based on conversations with our nanny and others at the resort, a large fraction of their income comes from overtime when you hire them after 5pm for additional childcare at $6/hour. We converged on having Carlene take a break and come back at 6:30 for a few hours while we went to dinner. Tips are also big for them, it seemed standard to tip the nanny $100 for the week if you were happy.

The resort wasn’t without a few opportunities for improvement:

  • The food isn’t very good. This may be par for mid-range all-inclusive resorts. The menu gets a bit repetitive, and in particular the deserts were downright poorly executed. Also, presumably because it was a light week, only 2 of the 3 restaurants were open on any given night. On the bright side, this probably helped me avoid gaining weight, and the breakfasts were reasonably good.
  • Some parts of the facility aren’t as child-safe as I would like. Part of this is Jamaica’s more relaxed building codes, and part of it is that kids are pretty much always being watched, so they don’t have much opportunity to get into trouble. But, for example, the railing between the pier and the ocean had big gapes, definitely not up to US code. And the cribs they provided were not SIDS safe. If you are a parent who is especially concerned about these things, it could be quite frustrating.
  • It could be crowded. We were there the slowest week of the year. If all the rooms were full, I expect the deck chairs around the pool and other facilities could be more taxed. It’s something I’ll keep an eye on when booking our next trip.
  • They are not helpful with arranging off-property trips unless the trip is run by one of their partners. This means that if you are interested in walking ten minutes down the road to the town center in Runaway Bay, and want advice for a place to eat, you won’t get any answers from the staff. Of course, Jamaica is a high crime island, so maybe you are best off staying on the resort. But if you want to catch a cab or their shuttle bus to do some shopping at a tourist-friendly Jamaican mall, expect to be taken advantage of unless you haggle the price and set things up in advance. Just because the resort booked your trip doesn’t mean you will get a good price or good service.

Of course I’m still very positive about the resort, and if you have young kids and are looking for a relaxing vacation I highly recommend it.

I hope you find what I wrote here useful. Especially if you are booking a trip to the FDR, leave a comment to let me know. Thanks.

Toddler Science and Big Data

"Stand Back, I'm Going To Try SCIENCE"I’ve been spending a lot of time following my son Patrick around watching him explore the world. I’ve shared a few of his important discoveries with Twitter and with friends, under the tag “Toddler Science”. Key discoveries include that tissue boxes contain a finite supply of tissue and that cat magnets do not stick to cats. I spent the New Years weekend with friends, and they too had an opportunity to watch Patrick learning.

The metaphor that keeps coming to mind, particularly when watching the destructive testing of my belongings, is that a toddler is a video gamer playing with a new engine. They get their bearings by testing out the physics model: how far can I throw a grenade? Do they bounce? If I shoot up the wall and leave the room, does the damage persist or does the state reset?

Patrick is learning the physics engine of his world. To do it, he is doing a large number of observations. And try to make sense of these observations, which is quite difficult.

“He needs to figure out if he is a universe governed by Newtonian mechanics, or just the Quake engine.”

“Aristotle never got past the Quake engine.”

Of course, humans don’t make use of nearly all the data that is available to them. If they did, they would learn a whole lot more a whole lot more quickly, as we can learn from Eliezer Yudkowsky’s fable That Alien Message. But we do receive a whole lot of data, and somehow filter through it to learn to interact with a complex universe, to empathize and communicate with other humans, to learn and to create new ideas. The amount of data consumed by a child before he learns to speak or throw a ball is staggering.

Computer Science is just starting to learn how to manage and analyze data sets at this scale. Today’s Big Data systems come in many flavors, from Google and Facebook to Walmart and eBay. There is some debate about what big data means, with Curt Monash and Dan Abadi having recent posts on the topic.

Regardless of how you define it, there is lots of data available that computers are still ignoring. What passes for big data in artificial intelligence is only starting to approach what passes for big data in biological systems. As big data gets truly big, interesting things will happen. It may be that previous generations of AI weren’t bad ideas. They were just data starved.

Book Review: Early Exits: Exit Strategies for Entrepreneurs and Angel Investors

“>As someone several years into a successful venture backed enterprise software startup, I find myself spending a lot of time looking at the green grass on the other side of the startup fence: angel funding and quick flips. People have mixed opinions on flipping, both the practicalities and the ethics of quickly flipping a company. Ethically, I tend to agree that “Flipping is Good“. But practically, there are a lot of moving parts in a company and lining them up for a quick windfall seems to require more than a bit of luck. It was in hopes of better understanding these practicalities that I read Basil Peters book Early Exits: Exit Strategies for Entrepreneurs and Angel Investors (But Maybe Not Venture Capitalists).

Many business books have only one good idea. This book has three, which is nice:

  1. The business environment is in great shape for early exits – If you want to sell a company for $5-30 million, there are a lot of buyers, they have gotten easier to find, and the deals have gotten simpler. For many more businesses than in the past, an early exit is feasible.
  2. Align your exit timeline with your investors to make an early exit possible – If you take traditional venture money, that is a commitment to swing for the fences. It can be very difficult to accept a $5-30 million exit after you have raised even seed money from a VC, because they are depending on your company to consume growth capital and deliver the kind of returns that their funds require.
  3. Employ an outside party to manage the exit – The book really drove this point home, presumably because the author is making a career out of this role. It was an interesting perspective to me, not something I hear much about in the blogosphere, and something I’ll have to investigate further.

Unfortunately there were a few problems with the execution of the book:

  • Lack of rigor – Peters’s financial models are quite simple, and generally not backed up by broadly sourced data. The traditional-versus-early-exit model that makes the cover of the book and drives home the point about avoiding VC is purely hypothetical numbers, for example.
  • Canadian heavy - Writing about what you know is great, but more information about key geographies like the United States would have helped make the book more credible. There is plenty of research and statistics available, and the author only scratched the surface.
  • Simplistic Angels versus VCs viewpoint – There are a lot of opinions flying around about what makes someone a super-angel, or what makes a seed-stage VC firm. Suffice to say the book’s black and white distinction between angels and VCs is simplistic. I would have expected more reasoned discussion of investor types, given that investor alignment is a key point.
  • Insufficient detail – The later chapters in the book, notably around pricing, are insultingly light on detail. I realized that pricing is complicated and it’s best to work with a professional, but that’s no excuse for introducing net present value of cashflow and then leaving everything else as an exercise for a consult.
  • Too much of a commercial – I understand that the point of this book, and many others, is to promote the author’s services. However, there were too many pages dedicated to the need for such services, and too few to the nuts and bolts of what would be done or how value would be realized. That, in the end, is the real problem with this book. I was hoping the author would tell me more of what he knows, since I’ll probably never have the opportunity to hire him directly.

The author has a blog (angelblog.net), and it remains somewhat unclear to me why he put the effort into a book rather than sticking with the blog format. Much of the information in this book is available on his blog and around the blogosphere. I had been hoping for more new work, or more depth. Unfortunately I did not find it.

If you’re considering starting a company with an eye to early exits, it’s easier to read this book than to track down 100 good blog posts on the topic. Unfortunately, you’re going to need the blog posts either way to fill in the gaps.

2010 The Year in Bookmarks

As you may have heard, Delicious is going away, or at least being sold off, or otherwise being destroyed by Yahoo. In the last 72 hours, I’ve tried out the two leading alternatives, Diigo and pinboard. Along the way, I got to wondering why I keep all these bookmarks around. I do often search back for things. And in paging through my 1173 bookmarks from 2010, they formed an interesting lens through which to look back at the year. And so I present 2010, the year in bookmarks.

Startups and the Death of VC

A lot of the links in my bookmarks are about startups and venture capital. A popular topic in 2010 is the death of VC, or debates about angels versus VCs. Also, discussions about whether Boston or New York are the number two startup hub to Silicon Valley.

Startup Visa

The notion of a startup visa also caught my attention. Here are just a few links.

Startups Without Software

At the same time, everyone is excited about the prospects of tech startups that don’t have any technology risk, because they don’t actually require new technology. Unfortunately, the best examples of these startups are fairly small businesses

Old Ideas are New Again

Another theme was all the old ideas that are new again, demonstrating that timing and execution can both make all the difference.

The Maker Movement

I’m not sure if it is old or new, but the “Maker Movement” is getting a lot more press attention, and a lot more bookmarks from me.

iPad

The iPad was the biggest new consumer technology of the year

  • Best New (to me) Blogs
  • Memes

    Far be it from me to catalog all the memes, but here are two

    Next Generation Marketing:

    Though speaking of memes, I also have a bunch of bookmarks around next generation marketing.

    Data and Journalism

    The use of data to drive new journalism was a big trend.

    Visualization

    If you are going to do journalism with data, visualization matters a great deal:

    NoSQL

    Software

    Who Programs and Who Doesn’t

    Video Games

    Prohibition

    Birth Control

    It was the 50th anniversary of the birth control pill, inspiring a range of articles, including two contrarian pieces:

    The Beatles

    It was also the 30th anniversary of John Lennon’s death

    Keeping Your Eyes Open

    Finally, two links about keeping your eyes open for the surreal and the wonderful things around you every day:

    There Can Be Too Many Suckers at the Table

    Everyone knows the old saw about poker: If you can’t spot the sucker, it’s probably you. It’s nice to think you could sit down at a table with incompetent players and take their money. It probably works in poker. Mark Suster applies the aphorism to angel investing in his post Dealflow – Are You Sitting at The Right Poker Table? It’s a good post, dealflow is really important. But the poker metaphor falls short. Too many suckers actually ruin a startup market.

    Consider the music business. In nearly every segment, from performers to promotion to distribution, there is far more talent and capital being applied than will ever be able to be profitably returned. People get into the music business as a life choice. Nashville is full of artists just barely getting by, but thankful to be in the business. On the web there are hundreds of startups trying out dozens of business models for selling/giving/pushing/pulling music. Hardly any of them are making profits, and none of them are seeing big exits. But money and talent keeps flowing in, whether it is hedge fund veterans backing LimeWire or  Google trying several strategies.

    When there is too much capital, the bad can start to crowd out the good. There are only so many early adopters, and they get tired of hearing about all the new things. Your competitors are losing money on every sale, making it up in volume, and setting customer expectations all wrong. Lavish launch parties, expensive marketing campaigns, and picky consumers drain your resources. Incumbents with natural or historic monopolies become demanding and exploitive of new partners. A new entrant who tries to be above the fray doesn’t find anyone doing business at that level.

    Music isn’t the only market with this problem. Many other media segments have it too, including video games, Broadway theater, and film. For most of its history the airline business has been losing investor’s money. Nearly any tech-VC fad, particularly one with light capital or intellectual property requirements, can fill up with suckers. The restaurant business may be subject to the same problem, and the nightclub business probably is.

    If you know you are in a market full of suckers, it might still be possible to build a successful business. You can find an angle to exploit people who want to be in the market, either in the style of The Producers (generally illegal) or in the style of Ticketmaster (potentially illegal). You can be consistently and significantly better than everyone else, like many Hollywood directors or Apple’s iTunes. Or use the market to prove out your idea and build your brand, but make your money in a more diverse market, like Moontoast (whose presentation at WebInnoinspired this post).

    Two Seven Off Suit

    At the poker table, it’s not very often that a sucker turns his two-seven offsuit into a full house. When it comes to startups, there are a lot more ways for people with capital and skills to spoil the game for everyone. If you look around the table and everyone else looks like a sucker, you might be one too.

    Yes Virginia, You Can Work on Great Technology at Startups

    You can work on great technology at startups. You wouldn’t think that would be a controversial statement. But it is if you believe Ted Tso’s defense of Google, “Google has a problem retaining great engineers? Bullcrap.” Ted dismisses the engineering that goes on in a startup, saying:

    Similarly, you don’t work on great technology at a startup.  Startups, by and large, aren’t about technology — at least, not the Web 2.0 startups like Facebook, Foursquare, Twitter, Groupon, etc.   They are about business model discovery.  So if you are fundamentally a technologist at heart, whose heart sings when you’re making a better file system, or fixing a kernel bug, you’re not going to be happy at a startup.   At least, not if the startup is run competently.

    Ted might have a point about Web 2.0 startups, but there are still  technology startups in software. These startups generally need to prove out their product and market rather than their business model. Business model innovation is sometimes part of the exercise. But more often the company is executing on a standard business model, with some need to validate the market, a greater need to validate/implement the technology, and most importantly a need to link the innovative technology to an addressable market. Much has been written about this, because it is the traditional structure of startups.

    Web 2.0 startups are trendy right now because they are disturbingly capital efficient. Companies like Diapers.com and Groupon have negligible technology risk. Proving out the business model costs very little money in the age of Everything as a Service (EaaS). They generate good stories about selling virtual goods before they exist, of zero-inventory supply chains and zero-employee companies. Investors like the idea of low risk high reward returns, even if they are still uncomfortable with the decreased emphasis on capital.

    But while those companies are grabbing headlines and mindshare there is plenty of deep technology innovation going on in startups. There are more innovative database startups at various stages in their life than I can remember right now (e.g. Vertica, Clustrix, Tokutek), not to mention the NoSQL startups (Cloudera, Basho), messaging companies (Solace, Kaplan, 29west), visualization companies (Panopticon, Spotfire), and hundreds of other software startups with a sizable technical product innovation challenge ahead of them. And there are plenty of recent success stories that wouldn’t have been able to build their company without great technology (VMware, Google, Amazon).

    It’s great that business model innovation is well enough understood that it is top of mind for developers. Understanding what key innovations are required, be they business or technical, and what are the most efficient ways to validate them, is key to success in any startup. It’s too bad that some engineers think that there is no longer a place for great engineering at startups. Not all startups require great engineering, but many still do.

    Ted’s trying to defend Google against claims that Facebook is poaching all the engineers. From where I stand, he’s right. Plenty of great engineers are going to work at Google, more than are leaving. And Google is able to run projects like ChromeOS, LLVM, and AppEngine. Projects that wouldn’t be the same in a startup.

    But if you were going to find fault with Google in this, consider: Googlers now believe they are doing engineering that can’t be done anywhere else. If that was true, it would mean they don’t have anything to fear from startups. Believing that is a step towards the hubris and ossification that Google is working so hard to avoid.

    Three Months Without Cable

    As was widely reported in the media, the second and third quarter of 2010 show a steady decline in cable subscriptions. This is earth-shaking for the cable companies, who have seen growth in US subscribers over their entire history. It’s a key indicator of not only consumers being more careful with their spending, but the rise of Internet-delivered media as a compelling alternative.

    In August of this year I joined the ranks of people “cutting the cord”. I was moving, and when we set up Verizon FiOS at the new house we left off video. Three months later, I’d like to fill you in on how it has gone and what I see in the future of consumer video delivery.

    Leer más »

    Captchas: The Bear Proof Trash Can Problem

    Not actually my captcha fail, but same idea, a captcha in greek from linked inLately I’ve been selling a lot of things on Craigslist. Along with adventures in capitalism, every post to Craigslist requires filling out a CAPTCHA, specifically a reCAPTCHA. I’ve noticed that they have gotten quite difficult. In fact, at least one of the captchas I got recently was in Greek.

    Captchas area a really clever idea, but they represent a special kind of arms race. The spammers are always improving their automatic and semi-automatic captcha solvers. At the same time, the average web user is not getting any better at solving captchas. The goal of the captcha company is to hit the window between what motivated spammers can do automatically and what web users can do manually.

    I call this the Bear-Proof Trashcan Problem. If you have ever walked up to a trash can in a bearful park, you know the experience. The instructions on the trash cans keep getting longer, the mechanical bits more complicated and more hidden. The result is tourists leaving trash outside the cans, which is as bad as not bear-proofing the cans at all. But if the cans are simpler, or require less manual dexterity, bears figure them out. The bears are willing to put a lot of time into it. And as one park ranger put it, “The smartest bears are smarter than the dumbest tourists.”

    When you are building software license enforcement, or writing tax law, or creating frequent flyer programs, you face the same problem: the desirable majority is willing to spend much less time dealing with whatever you create than the undesirable minority is going to spend breaking it. Very often people forget this rule, and build systems which focus on preventing the undesirable behavior, driving away the desirable but uncommitted majority. It’s easy to build a bear proof trash can. It’s hard to build one that a tourist can use.

    Proactive Assumption Violation: Avoiding Bugs By Behaving Badly

    Bugs are a fact of life in software, and probably always will be. Some bugs are probably unavoidable, but a lot of bugs can be avoided through good architecture, defensive programming, immutability, and other techniques. One major source of bugs, especially frustrating bugs, is non-deterministic behavior. Every programmer has experienced bugs which don’t reproduce, which require a special environment, or special timing, or even just luck to make happen. To avoid these bugs, programmers learn to favor determinism, making sure their software behaves the same way every time.

    But sometimes a little extra non-determinism can help to avoid bugs. When designing a library the specification may contain caveats which the implementation does not exercise. If a program only ever uses one implementation, or doesn’t exercise the full range of behaviors while testing, then the program may depend on behaviors which are an artifact of the implementation. When an alternative implementation is used, or circumstances exercise other behaviors, then the program will exhibit bugs. What I would like to suggest is that instead the implementation proactively violate any assumptions the client might have, by deliberately and non-deterministically taking advantage of all the caveats in the interface specification. This will force clients to code defensively, and help to eliminate this class of bugs.

    For example, an interface for Set might include iteration, but say that iteration order is unspecified. Most Set implementations, like Java’s HashSet, will iterate in a stable order if the set doesn’t change. The order might even be consistent from one run of the program to the next. But if programs depend on iteration stability, substituting a different implementation, such as one based on a splay tree, will introduce bugs. If instead the implementation of Set iteration deliberately returned a different iteration order each time, then programs would be unable to depend upon it.

    For a real-world example, consider the standard C function memcpy. According to the specification, if the source and destination buffers overlap, behavior is undefined. But what does undefined really mean? Recently, Linux switched to a new memcpy implementation, one which copies backwards (high bytes first, low bytes last), because it is faster on modern hardware. The result is a dramatic change in overlapping buffer behavior, leading to difficult to isolate bugs like Red Hat Bug 639477: Strange sound on mp3 flash website. The bug was eventually tracked down to memcpy using valgrind. But if the original memcpy had been more deliberately harmful in the case of overlapping buffers, than bugs like this would not be created in the first place.

    Another place where this comes up is thread safety. Many APIs are not thread safe, but the results of using them in a thread-unsafe way are benign, or unlikely. Take the Java DOM API for XML documents. This is not a thread safe API, which is not surprising for a complex mutable data structure. What is a bit surprising is that even just reading the Java DOM from multiple threads can have unintended consequences. This is because of a cache for reuse of Node objects, and the failure mode is that very occasionally accessor functions return null when they are called from multiple threads simultaneously. Debugging a program that was suffering from this behavior took several hours, because the undesirable behavior is very infrequent. Is there a way we can apply the principle of proactive assumption violation to make this sort of bug less common?

    System that cope with infrastructure faults by degrading their behavior are another case where proactive assumption violation can reduce bugs later. NoSQL databases are well known for taking approaches like eventual consistency in order to offer better performance and availability. But that means that when the system is under heavy load or suffering from partial outages, consistency may take a long time to resolve. I ran into this as a user of Netflix the other night. My television and my laptop had two different ideas of what my current Netflix queue contained; my television couldn’t see the recent updates. It turns out there is even a slide deck from Netflix describing the architecture choices that led to my undesirable user experience. Most of the time things are consistent enough that I wouldn’t see this asynchrony. But when I do see it, as a user there is no obvious way to get around it or even to know that it is happening. Would a NoSQL infrastructure that let consistency drift more often end up with better average user experience?

    Clients of interfaces often make assumptions about how those interfaces work, assumptions that are explicitly or implicitly not part of the spec. But if implementations of the interface don’t violate those assumptions, then programs can be developed which require them to be true. This leads to unexpected and expensive bugs if and when those assumptions are violated. One solutions is for implementations to deliberately violate these assumptions, for no other reason than to force clients of their interface to future-proof. The result is more work up front for programmers, but fewer bugs in the long run.

    What do you think about proactive assumption violation? Is it a technique you have ever used? Have you experienced bugs which would be avoided if others had employed proactive assumption violation?