The metaphor that keeps coming to mind, particularly when watching the destructive testing of my belongings, is that a toddler is a video gamer playing with a new engine. They get their bearings by testing out the physics model: how far can I throw a grenade? Do they bounce? If I shoot up the wall and leave the room, does the damage persist or does the state reset?

Patrick is learning the physics engine of his world. To do it, he is doing a large number of observations. And try to make sense of these observations, which is quite difficult.

“He needs to figure out if he is a universe governed by Newtonian mechanics, or just the Quake engine.”

“Aristotle never got past the Quake engine.”

Of course, humans don’t make use of nearly all the data that is available to them. If they did, they would learn a whole lot more a whole lot more quickly, as we can learn from Eliezer Yudkowsky’s fable That Alien Message. But we do receive a whole lot of data, and somehow filter through it to learn to interact with a complex universe, to empathize and communicate with other humans, to learn and to create new ideas. The amount of data consumed by a child before he learns to speak or throw a ball is staggering.

Computer Science is just starting to learn how to manage and analyze data sets at this scale. Today’s Big Data systems come in many flavors, from Google and Facebook to Walmart and eBay. There is some debate about what big data means, with Curt Monash and Dan Abadi having recent posts on the topic.

Regardless of how you define it, there is lots of data available that computers are still ignoring. What passes for big data in artificial intelligence is only starting to approach what passes for big data in biological systems. As big data gets truly big, interesting things will happen. It may be that previous generations of AI weren’t bad ideas. They were just data starved.

Startups and the Death of VC

A lot of the links in my bookmarks are about startups and venture capital. A popular topic in 2010 is the death of VC, or debates about angels versus VCs. Also, discussions about whether Boston or New York are the number two startup hub to Silicon Valley.

• Twilight of Venture Capital: Back to the Future? How about Forward to the Past! and It IS Broke!
• Every Time an Engineer Joins Google, a Startup Dies
• Warren Buffet on the hidden risks of investing in rapidly growing industries
• It’s not East Coast versus West Coast, it’s about making more places like the valley
• IPO Anxiety East Coast Version
• Canada Now Somewhat Less Anti-Startup

Startup Visa

The notion of a startup visa also caught my attention. Here are just a few links.

• Maine Business is Shut Without a Renewed Visa
• Startup Visa and Startup Visa
• Tech Entrepreneurs and US Immigration: A Documentary

Startups Without Software

At the same time, everyone is excited about the prospects of tech startups that don’t have any technology risk, because they don’t actually require new technology. Unfortunately, the best examples of these startups are fairly small businesses

• How I set up an sold a product using Unbounce, Wufoo, and Chargify
• How I Made $6k With My eBook

Old Ideas are New Again

Another theme was all the old ideas that are new again, demonstrating that timing and execution can both make all the difference.

• Party Like It’s 1999: 10 Old Tech Ideas That Are New Again
• The 10 Most Tempting Software Startup Categories

The Maker Movement

I’m not sure if it is old or new, but the “Maker Movement” is getting a lot more press attention, and a lot more bookmarks from me.

• Adafruit’s successful sponsorship of Kinect driver hacking, which got mainstream attention.
• Not just tech, there is a lot of food blogging with a maker feel, from Unwholesome Food’s Turdunkin to My Jello Americans artistic jello shots.
• Sometimes tech and food go together: Microwave Vacuum Dehydration: failed experiment “What I hadn’t thought of was that under partial vacuum, microwave ovens really, really want to make balls of plasma.”

iPad

The iPad was the biggest new consumer technology of the year

• What iPads did to My Family
• The iPad: a Near-Miracle for My Son With Autism
• Seth Godin asked for the iPad App of his Dreams
  • Set up a wiki about it
  • And someone built it

• Hyperbole and a Half
  • God of Cake
  • My Boyfriend Doesn’t Have Ebola Probably
• In The Pipeline
  • How Not To Do It
  • Things I Won’t Work With

Memes

Far be it from me to catalog all the memes, but here are two

• Some of the strangest stuff to come out of America’s two wars: Navy I’m on a Boat also Afganistan Army Telephone.
• Old, but apparently I didn’t see it until 2010: Hitler Orders a DMCA Takedown

Next Generation Marketing:

Though speaking of memes, I also have a bunch of bookmarks around next generation marketing.

• The Insane Clown Posse have possibly the most clever marketing ever around their Miracles video and How Magnets Work.“I know this sounds crazy, but I’m being as honest as I can: We planned all this out.” Or maybe he’s lying.
• For a different technique: Groupon CEO Andrew Mason: “I Decided To Set The Bar Very Low And Make Up Lies About Myself That Make Me Sound Lame”
  
• Or, more historically: How Viagra and related drugs were introduced to the scientific community through edgy marketing: How (not) to communicate new scientific information: a memoir of the famous Brindley lecture
• A big change in 2010 was increased optimization of headlines for social media. I saw evidence of this constantly. Here is a how-two: Blogging by Numbers: How to Create Headlines That Get Retweeted
• Of course, those tweetable headlines make for questionable journalism. As the Columbia Journalism Review found: Business Insider and Financial Press Sensationalism

Data and Journalism

The use of data to drive new journalism was a big trend.

• Databases will soon be eligible for the top prize in journalism: Pulitzer board announces changes for 2011 competition
• NYTimes Netflix Culture Map
• Mapping America
• Guardian publishes their data for public access
• Chicagocrime.org is now EveryBlock
• Analytic X has set up a prize for social predictions
• Very Small Array: The Streets of Park Slope, Brooklyn has a clever take on mapping culture
• The OkCupid blog does a lot with data about online dating: Don’t Be Ugly By Accident and How Races and Religions Match in Online Dating and The Four Big Myths of Profile Pictures

Visualization

If you are going to do journalism with data, visualization matters a great deal:

• A Tour Through the Visualization Zoo
• Unfortunately visualization oriented startup Verifiable had to shutdown because customers were not interested in paying for the product.
• But you can get Sparklines in Microsoft Excel
• And IBM Research’s Many Eyes
• Quantitative Brokers has a great visualization of market microstructure on the CME

NoSQL

• MapReduce and Parallel DBMSs: Friends or Foes?
• NoSQL v. SQL is the worst holy war ever
• MongoDB is Web Scale
  
• Visual Guide to NoSQL Systems
• Using MySQL as a NoSQL: a story for exceeding 750,000 qps on a commodity server

Software

• PyPy JIT for Regular Expressions
• God’s Number is 20: http://www.cube20.org/
• Putting It All Together: Using Socio-Technical Networks to Predict Failures

Who Programs and Who Doesn’t

• If You Were Hacking Since Age 8 It Means You Were Privileged
• Why Johnny Can’t Program
• Separating Programming Sheep from Non-Programming Goats
• How does biology explain the low numbers of women in computer science? Hint: It doesn’t.
• Why Diversity Matters
• Female Teachers Transmit Math Anxiety to Female Students

Video Games

• Video Games: The Addiction
• Minecraft
  • Penny Arcade on Minecraft Part 1 and Part 2
  • How to Survive your First Night
  • 16-bit ALU in Minecraft
• Nethack: The Greatest Game You Will Ever Play

Prohibition

• Wayne B Wheeler: The Man Who Turns Off the Taps A history of prohibition, women’s suffrage, and the origins of single-issue politics
• Who’s Got the Acid? These days, almost nobody. and This Is Your Country On Drugs by Ryan Grim

Birth Control

It was the 50th anniversary of the birth control pill, inspiring a range of articles, including two contrarian pieces:

• Why I Hate The Pill
• Waking Up From The Pill

The Beatles

It was also the 30th anniversary of John Lennon’s death

• I really enjoyed reading Scenes From An Alternate Universe Where The Beatles Accepted Lorne Michaels’ Generous Offer
• The Future History of the Beatles is very clever
• I had never read 1980 Playboy Interview with John Lennon and Yoko Ono and it clarified a lot of things about Lennon for me.

Keeping Your Eyes Open

Finally, two links about keeping your eyes open for the surreal and the wonderful things around you every day:

• Entrances to Hell In and Around the UK
• There is a Horse in the Apple Store

Similarly, you don’t work on great technology at a startup.  Startups, by and large, aren’t about technology — at least, not the Web 2.0 startups like Facebook, Foursquare, Twitter, Groupon, etc.   They are about business model discovery.  So if you are fundamentally a technologist at heart, whose heart sings when you’re making a better file system, or fixing a kernel bug, you’re not going to be happy at a startup.   At least, not if the startup is run competently.

Ted might have a point about Web 2.0 startups, but there are still  technology startups in software. These startups generally need to prove out their product and market rather than their business model. Business model innovation is sometimes part of the exercise. But more often the company is executing on a standard business model, with some need to validate the market, a greater need to validate/implement the technology, and most importantly a need to link the innovative technology to an addressable market. Much has been written about this, because it is the traditional structure of startups.

Web 2.0 startups are trendy right now because they are disturbingly capital efficient. Companies like Diapers.com and Groupon have negligible technology risk. Proving out the business model costs very little money in the age of Everything as a Service (EaaS). They generate good stories about selling virtual goods before they exist, of zero-inventory supply chains and zero-employee companies. Investors like the idea of low risk high reward returns, even if they are still uncomfortable with the decreased emphasis on capital.

But while those companies are grabbing headlines and mindshare there is plenty of deep technology innovation going on in startups. There are more innovative database startups at various stages in their life than I can remember right now (e.g. Vertica, Clustrix, Tokutek), not to mention the NoSQL startups (Cloudera, Basho), messaging companies (Solace, Kaplan, 29west), visualization companies (Panopticon, Spotfire), and hundreds of other software startups with a sizable technical product innovation challenge ahead of them. And there are plenty of recent success stories that wouldn’t have been able to build their company without great technology (VMware, Google, Amazon).

It’s great that business model innovation is well enough understood that it is top of mind for developers. Understanding what key innovations are required, be they business or technical, and what are the most efficient ways to validate them, is key to success in any startup. It’s too bad that some engineers think that there is no longer a place for great engineering at startups. Not all startups require great engineering, but many still do.

Ted’s trying to defend Google against claims that Facebook is poaching all the engineers. From where I stand, he’s right. Plenty of great engineers are going to work at Google, more than are leaving. And Google is able to run projects like ChromeOS, LLVM, and AppEngine. Projects that wouldn’t be the same in a startup.

But if you were going to find fault with Google in this, consider: Googlers now believe they are doing engineering that can’t be done anywhere else. If that was true, it would mean they don’t have anything to fear from startups. Believing that is a step towards the hubris and ossification that Google is working so hard to avoid.

In August of this year I joined the ranks of people “cutting the cord”. I was moving, and when we set up Verizon FiOS at the new house we left off video. Three months later, I’d like to fill you in on how it has gone and what I see in the future of consumer video delivery.

I dropped FiOS television for two reasons: price and customer service. As for the price, paying more than $50 per month for cable television has always seemed excessive to me. Verizon kept edging up the costs, to something like $65 for fairly-basic service. “Fairly basic” is where customer service comes in, it was always difficult to navigate the different offerings, promotional rates, and timeouts.

In comparison, Verizon FiOS internet service is easy to buy, transparently priced, and has had fabulous technical support the few times I have had to use them (they have never asked me to reboot anything). I highly recommend FiOS internet service if it is available to you.

But back to television. Having decided to drop cable but not to suspend all consumption of video media, we replaced it with a few a la carte offerings:

1. Netflix Streaming ($16.99 $9.99) – This works great. We’ve been long time Netflix customers. The number of movies and especially television available streaming keeps going up. We’ve hardly used the shipped-DVDs part of our Netflix subscription in months. They just changed around the plans, so we dropped to $10/month which only allows one DVD at a time, but unlimited streaming.
2. Hulu Plus ($7.99/month) – We’ve been using Hulu for a long time. The “plus” mostly means you have access to current shows (e.g. Glee) without them aging out. Otherwise you can only play the most recent handful of episodes. Hulu Plus also includes lots of rerun-grade television from the 80s and 90s.
3. Amazon Unbox ($2/episode) Some premium shows aren’t on Netflix or Hulu, like Mad Men, but they are available pay-per-episode on Amazon.
4. Apple iTunes ($3/episode) Other premium shows (True Blood) are only on iTunes.

All this runs into a Mac Mini we bought a few years ago. I like the Mac because it is an open enough platform to run all the various media services I might want, while not having the maintenance headaches a Windows Media Center or Linux machine would have (I used to run MythTV before I switch to the Mac). There are many other ways to do it, like single purpose set top boxes (Netflix, Apple TV, Google TV, Boxee, etc) or game consoles or TVs with built in streaming media functionality. But I like the lack of lock-in that comes from having a general purpose computer attached to the television.

The total cost of this kind of setup is thus $600 for the hardware (not counting the actual television), under $20 of fixed costs per month, and another $15 of premium content purchases per month. That’s about $30 per month cheaper than the FiOS television I was getting.

What’s the downside? I have missed out on some content. The two things I wanted to watch but could not find online were the Emmy Awards (I could get red carpet and backstage feeds, but not the actual event) and the Macy’s Thanksgiving Day Parade (nice website but no streaming). I also no longer have access to Red Sox games, or other Boston sports. Given the tough season the Red Sox were having, I didn’t really miss it, but that might be a problem next year. MLB.com will sell you video feeds of games, but only outside the home market for the team. I could get around that restriction, but paying $200 for content only to have to violate the license terms seems abusive.

The Future of Television

I’m liking my new media lifestyle, it’s hard to imagine going back. If anything, it seems like more content is coming to me. The question for the content-producers is how are they going to get it to me. There are fortunes to be made in correctly predicting the new media distribution landscape.

One big opening I see is around children’s television. There isn’t a clear leader in this space. Netflix has a lot of children’s content, but doesn’t really offer the Saturday Morning Cartoons experience. Of course, maybe children will be looking for a more interactive media experience on their Saturday mornings. Media companies are still producing a lot of children’s television, and kids consume it. I predict a $10/month service focused on content for children, with optimized catalog and playback.

Another obvious opportunity is in sports. This is likely to be dominated by incumbents. I would happily pay per-game, or buy a reasonably priced subscription. $200 for the season from MLB is kind of high since I don’t come close to watching every game.

Finally, the big question is whether consumers in general will want this kind of a la carte experience, buying a $10 subscription here, a $2 video there, or if they would really rather have the bundled experience that cable offers. Even if someone does put together a bundled service, it’s hard for me to imagine people paying $60-100/month for internet-delivered television. Not with Netflix and Hulu setting price expectations so low.

Captchas area a really clever idea, but they represent a special kind of arms race. The spammers are always improving their automatic and semi-automatic captcha solvers. At the same time, the average web user is not getting any better at solving captchas. The goal of the captcha company is to hit the window between what motivated spammers can do automatically and what web users can do manually.

I call this the Bear-Proof Trashcan Problem. If you have ever walked up to a trash can in a bearful park, you know the experience. The instructions on the trash cans keep getting longer, the mechanical bits more complicated and more hidden. The result is tourists leaving trash outside the cans, which is as bad as not bear-proofing the cans at all. But if the cans are simpler, or require less manual dexterity, bears figure them out. The bears are willing to put a lot of time into it. And as one park ranger put it, “The smartest bears are smarter than the dumbest tourists.”

When you are building software license enforcement, or writing tax law, or creating frequent flyer programs, you face the same problem: the desirable majority is willing to spend much less time dealing with whatever you create than the undesirable minority is going to spend breaking it. Very often people forget this rule, and build systems which focus on preventing the undesirable behavior, driving away the desirable but uncommitted majority. It’s easy to build a bear proof trash can. It’s hard to build one that a tourist can use.

But sometimes a little extra non-determinism can help to avoid bugs. When designing a library the specification may contain caveats which the implementation does not exercise. If a program only ever uses one implementation, or doesn’t exercise the full range of behaviors while testing, then the program may depend on behaviors which are an artifact of the implementation. When an alternative implementation is used, or circumstances exercise other behaviors, then the program will exhibit bugs. What I would like to suggest is that instead the implementation proactively violate any assumptions the client might have, by deliberately and non-deterministically taking advantage of all the caveats in the interface specification. This will force clients to code defensively, and help to eliminate this class of bugs.

For example, an interface for Set might include iteration, but say that iteration order is unspecified. Most Set implementations, like Java’s HashSet, will iterate in a stable order if the set doesn’t change. The order might even be consistent from one run of the program to the next. But if programs depend on iteration stability, substituting a different implementation, such as one based on a splay tree, will introduce bugs. If instead the implementation of Set iteration deliberately returned a different iteration order each time, then programs would be unable to depend upon it.

For a real-world example, consider the standard C function memcpy. According to the specification, if the source and destination buffers overlap, behavior is undefined. But what does undefined really mean? Recently, Linux switched to a new memcpy implementation, one which copies backwards (high bytes first, low bytes last), because it is faster on modern hardware. The result is a dramatic change in overlapping buffer behavior, leading to difficult to isolate bugs like Red Hat Bug 639477: Strange sound on mp3 flash website. The bug was eventually tracked down to memcpy using valgrind. But if the original memcpy had been more deliberately harmful in the case of overlapping buffers, than bugs like this would not be created in the first place.

Another place where this comes up is thread safety. Many APIs are not thread safe, but the results of using them in a thread-unsafe way are benign, or unlikely. Take the Java DOM API for XML documents. This is not a thread safe API, which is not surprising for a complex mutable data structure. What is a bit surprising is that even just reading the Java DOM from multiple threads can have unintended consequences. This is because of a cache for reuse of Node objects, and the failure mode is that very occasionally accessor functions return null when they are called from multiple threads simultaneously. Debugging a program that was suffering from this behavior took several hours, because the undesirable behavior is very infrequent. Is there a way we can apply the principle of proactive assumption violation to make this sort of bug less common?

System that cope with infrastructure faults by degrading their behavior are another case where proactive assumption violation can reduce bugs later. NoSQL databases are well known for taking approaches like eventual consistency in order to offer better performance and availability. But that means that when the system is under heavy load or suffering from partial outages, consistency may take a long time to resolve. I ran into this as a user of Netflix the other night. My television and my laptop had two different ideas of what my current Netflix queue contained; my television couldn’t see the recent updates. It turns out there is even a slide deck from Netflix describing the architecture choices that led to my undesirable user experience. Most of the time things are consistent enough that I wouldn’t see this asynchrony. But when I do see it, as a user there is no obvious way to get around it or even to know that it is happening. Would a NoSQL infrastructure that let consistency drift more often end up with better average user experience?

Clients of interfaces often make assumptions about how those interfaces work, assumptions that are explicitly or implicitly not part of the spec. But if implementations of the interface don’t violate those assumptions, then programs can be developed which require them to be true. This leads to unexpected and expensive bugs if and when those assumptions are violated. One solutions is for implementations to deliberately violate these assumptions, for no other reason than to force clients of their interface to future-proof. The result is more work up front for programmers, but fewer bugs in the long run.

What do you think about proactive assumption violation? Is it a technique you have ever used? Have you experienced bugs which would be avoided if others had employed proactive assumption violation?

In laptops and desktops, Apple is unassailed as the luxury brand. Whenever I talk to non-engineers about buying laptops or desktops, this is clear. If I suggest they get a Mac, the response is never “I don’t know if it will run my software” or “I prefer Windows”, but rather “I can’t afford one” or “it feels unnecessary.” Rather like if someone asked me what car to buy and I suggested a BMW. As far as I can tell, non-geeks would all buy Macs if money were no object. And there is a strong correlation between people who display what computer they use socially (geeks, coffee-shop denizens) and Mac users (gamers have their own tastes and displays). Thanks to the fact that the OS no longer matters, consumers are free to select either a utilitarian lowest-bidder machine, or a Mac.

Apple doesn’t really have any competition in this market. Sony has tried several times, and makes some really nice (and really expensive) machines. But because the Sony brand still doesn’t mean luxury to the man on the street, it doesn’t give people the opportunity to show off that they require from their luxury goods. And so Apple has a near-monopoly on expensive computers. In June 2009, Apple had 90% market share for non-business computers costing more than $1000. Their consumers are not price sensitive, and so Apple gets correspondingly high margins, creating a lucrative and stable business.

The iPod is also a luxury good, albeit a luxury that nearly everyone can afford. I think it is a fluke that Apple dominates the portable music player market. I think the iPod is the Coach purse of the Apple lineup: It’s a luxury good, but one that nearly anyone can afford and is easy to justify. And with those ubiquitous white earbuds, you can show off your good taste even when the player is in your pocket.

Speaking of the iPod, some people think that Apple is becoming a media company, leveraging their control of the player into domination of the music business. Far from it, I would say. Apple is happily participating in the demise of the music industry, carrying the record labels in a hand basket towards the free or nearly-free distribution of recorded music that is the obvious conclusion of technological improvements. If you told Steve Jobs that all music is going to be free tomorrow, the logical response would be “great, people are going to need new iPods with more storage.”

The iPad is clearly a luxury good. Early adopters proudly show off their iPad. It’s very expensive and has little competition. There is a big question as to how the market for tablets will develop. It may go the way of MP3 players, an expensive but pleasant toy where everyone buys the nice one from Apple. Or it may look more like the modern PC world, where anyone can get a decent table from Acer/Dell/HP/etc for $200. A lot depends on how broad the demand is for tablets.

The iPhone initially headed in the direction of the iPod, looking like mainstream consumers were choosing between an iPhone and no smartphone at all. But Android has created a credible option, in fact a wealth of credible option, that more practical consumers see as the better option when it comes to price, service availability, etc. But Apple still sells plenty of phones to people who want the new iPhone, even if the antenna is broken, the service is terrible, and their preferred carrier doesn’t have it.

Expect Apple to maintain a high price point and air of exclusivity around the iPad and iPhone. In the face of dozens of perfectly adequate Android competitors, Apple may well cede the low end of the market. Their branding, integration, and user experience will allow them to capture a premium price at the high end. Their product line will stay simple; customer’s aren’t interested in the optimal price/performance or choosing features. Customers just want the new Apple device, and will not be especially conscious about the price or comparisons to third party products.

Much has been written about developers fleeing iOS for Android. It’s true that Apple has been difficult to do business with. I expect mobile app developers to realize that Apple has the customers they want. Years ago, Apple was able to keep developers on the Mac platform when their market share was in the low single digits, because the average Apple user bought a lot more third party software than the average Windows user. Similarly, by hanging on to the high-end, $4-latte-drinking customer, Apple will be the place to go for developers selling $4 apps. Expect comparisons of per-user app spend to be forthcoming, and the numbers to be in Apple’s favor.

Apple has figured out how to be the only mass market luxury vendor in desktops, laptops, and MP3 players. By applying the same techniques to tablet computers and mobile phones, they might not maintain raw market share, but they can hang on to the most profitable customers, which is more important. They will do it not by offering the best products on some absolute scale understood only by geeks, but by offering a user experience that starts in the store, a brand which is increasingly well recognized, and a set of stories that tell people they are buying something more than just luxury.

As many of my readers probably already know, a core dump is an image of the contents of a programs memory. Generally core dumps are created when a program fails in a particularly catastrophic way, such as a segmentation fault. Core dumps help programmers find out what lead to the failure. Usually it’s easy to get a core dump, you just do a kill -11 of the process ID, faking a segfault. This takes down the program and writes a core dump.

Unfortunately since core dumps aren’t useful to non programmers, the environment on OSX by default does not make them, even when there are segmentation faults. One can change this for a given shell or process or login session using the ulimit command or the associated syscalls, but Murphy was with me today and so ecto was not running with such a setting. It is possible to change the setting of a running process by connecting with a debugger like gdb and making the right syscall, but that felt a little risky, since if I messed it up the process would be gone, whether I got a core dump or not.

Instead, let’s figure out how to not kill the process at all, doing the work of taking the memory snapshot ourselves. This should be possible with modern process inspection APIs. The book you want for this is Mac OSX Internals. I have a copy, and was all set to begin some deep yak shaving figuring all this out. However, the book saw me coming, and already laid out an example in detail, called Process Photography.

So, if you want to know a lot more about making your own core dump utility, you can read that post. Or if you are still with me because you just want to know how to recover a blog post, then go there and download gcore-1.3.tar.gz at the end of the post. Untar it and compile your gcore utility. Now you can create a core dump by running gcore -c ecto.core PID. If your experience is like mine, this will generate a 1.3 gigabyte core file, because modern programs are not shy about using memory, virtual and otherwise.

Now, this 1.3 gigabyte core file contains everything from program text and mmaped files to active memory to freed memory that hasn’t been reused yet. It’s a vast expanse of stuff you don’t need, must of it binary. Luckily, most programs will just store textual content as ASCII or UTF8. Assuming you were writing English, then the strings utility will be sufficient to find your text. So you can run strings ecto.core > ecto.strings. This will generate another large file (64 megs this time, not 1.3 gigs) with just the ascii string data from your programs memory. Still a lot to wade through, so I use grep -i to look for uncommon words in my post, and less to be able to page around the file quickly.

I wish that the story had a happy ending, but after all that I found that the ecto memory space contained a dozen copies of my post, but all of them were the truncated version that it had posted on my blog, rather than the actual text I wrote. So you will have to wait until next week (or at least tomorrow) to learn what I had to say about house renovation.

Our civilization has had tabbed interfaces since 1988. Mainstream browsers (well, Mozilla) have supported tabbed browsing since 2001. As tabbing has become more mainstream, as the web has gotten more complex, and as computers have gotten fast enough to handle dozens of open web pages, people have opened more and more tabs. The result is that nearly everyone knows the experience of “just having to close some tabs” before you reboot, or so you can get on with more important work. It’s easy to overwhelm yourself with the amount of content you can have open in tabs, and clearing it out is often an archeological experience spanning the last week or month of your web activities.

Browser tab management represents the greatest software usability challenge of our time. We are all facing information overload of one form or another, and this is an opportunity to improve the way people find, consume, retain, and manage information. Lately we are seeing a lot of attempts at innovation here, from Chrome’s tear-away tabs and performance optimization, to Firefox extensions like Ctrl-Tab. Most recently today we saw Tab Candy, a preview of new functionality in Firefox 4.

I’ve often said that I’ll switch to the first browser that doesn’t make me feel guilty for having 100+ open tabs. Looking at Tab Candy and other innovations, I see that we are moving in the right direction. But there are still many aspect of the problem that aren’t being sufficiently addressed.

• Application-oriented tab organization – Many of the sites I use today are really applications, whether it is Google Docs, Facebook, or Amazon. Taking application behavior into account, and helping me to avoid opening the same heavy application (e.g. gmail) multiple times is part of getting tab management correct.
• Automatic tab organization – Systems like Tab Candy require that users manage tabs. But wouldn’t it be better if tabs were managed and grouped automatically?
• Managing the tab/bookmark continuum – Leaving something open in a tab, or opening it in a tab, is often a way of deciding to come back to it later. Bookmarks accomplish the same thing, but most people’s bookmarks are themselves a usability nightmare. Automatically migrating tabs into bookmarks and bookmarks back into tabs might be the solution to a lot of these tab problems.
• Excursion and history management – Web browsing isn’t a linear process, the way browser history would have you think. It is at least a branching tree, which tabs support. But often the process of web browsing is a product itself, as when researching a new subject or deciding on a purchase. Being able to not only manage the process as it is happening, but also to archive it for later resumption or reference, would improve the efficiency of many browser tasks.
• CPU efficiency – An implementation detail to be sure, but one of the obstacles to running with 100+ tabs is the CPU load of all the little Javascripts. Some way to manage this, pausing or closing pages which are not visible, is likely to be required.

As you can see, there is still plenty of room for improvement in the browser interface, particularly around large numbers of tabs. What other cutting edge stuff have you seen? What would you like to see implemented?

• David Jeffery’s keynote on Betfair’s event driven architecture, past present and future. David presented not only on the performance challenges of running the core betting exchange, but also on the soft benefits. Thanks to their event driven architecture, Betfair is able to do more experimentation with less disruption, and be more agile as an organization.
• Dan O’Keeffe presented Reliable Complex Event Detection for Pervasive Computing, a system for compensating for missing data. Most importantly, it lays out a selection of strategies that developers can select and the system can use to automatically choose the correct approach, based on wether the system should be optimistic, pessimistic, or something in between.
• Quilt: A Patchwork of Multicast Regions was presented by one of the local students on short notice, because the author was not able to attend due to visa problems. This is especially frustrating when the paper is so interesting and the analysis quite strong. Quilt is a system for combining multiple delivery mechanisms to achieve efficient wide area distribution. Combining overlay-network based protocols with “patches” of true multicast where available, Quilt optimizes message routing for efficiency, reliability, and latency.
• An Approach for Iterative Event Pattern Recommendation which was also presented by a colleague rather than one of the authors due to visa issues. The paper describes a system for recommending event patterns to domain experts based on their initial attempts to define the pattern. In a controlled user study, the recommendation system substantially reduced the time taken by users to define novel patterns. It was good to see a real user study of a programming efficiency system, but there is lots of room for further measurement.
• Experiences with Codifying Event Processing Function Patterns, presented by the author Anand Ranganathan, dealt with a different kind of pattern. In systems using Ranganathan’s system, developers build template network flows, annotated with tags to define what components can be used in the flow. Any component that matches the inputs, outputs, and tags can be inserted into the template, and all possible template instantiations represent a domain of valid applications. End users are able to search this combinatorially large domain with a flexible interface, to find and instantiate applications that meet their needs.

There are many great papers and talks not listed here. Obviously my own bias shows, as does the fact that I didn’t attend every talk or read every paper.

Unfortunately most of the links go to the ACM portal, which doesn’t offer public access. My frustration with and thoughts on the future of academic publishing will have to wait for a future post. I recommend googling the titles and finding them on author pages if you can. Or ping me and I can send you papers.

Next year the conference will be hosted in New York by IBM research. Whether you attended or not, if you have any feedback on how to make the conference more appealing, I’m happy to hear it.