Moments ago I was writing a different blog post, about home renovation. Unfortunately just as I posted it a bug in ecto, the aging client I still use to edit blog posts, caused the complete loss of the text with no backup copies. Because messing about with system tools is more fun than rewriting that blog post, I present instead a brief howto for creating a core dump on OSX without killing a process, and inspecting that core dump to attempt to recover your data.

As many of my readers probably already know, a core dump is an image of the contents of a programs memory. Generally core dumps are created when a program fails in a particularly catastrophic way, such as a segmentation fault. Core dumps help programmers find out what lead to the failure. Usually it’s easy to get a core dump, you just do a kill -11 of the process ID, faking a segfault. This takes down the program and writes a core dump.

Unfortunately since core dumps aren’t useful to non programmers, the environment on OSX by default does not make them, even when there are segmentation faults. One can change this for a given shell or process or login session using the ulimit command or the associated syscalls, but Murphy was with me today and so ecto was not running with such a setting. It is possible to change the setting of a running process by connecting with a debugger like gdb and making the right syscall, but that felt a little risky, since if I messed it up the process would be gone, whether I got a core dump or not.

Instead, let’s figure out how to not kill the process at all, doing the work of taking the memory snapshot ourselves. This should be possible with modern process inspection APIs. The book you want for this is Mac OSX Internals. I have a copy, and was all set to begin some deep yak shaving figuring all this out. However, the book saw me coming, and already laid out an example in detail, called Process Photography.

So, if you want to know a lot more about making your own core dump utility, you can read that post. Or if you are still with me because you just want to know how to recover a blog post, then go there and download gcore-1.3.tar.gz at the end of the post. Untar it and compile your gcore utility. Now you can create a core dump by running gcore -c ecto.core PID. If your experience is like mine, this will generate a 1.3 gigabyte core file, because modern programs are not shy about using memory, virtual and otherwise.

Now, this 1.3 gigabyte core file contains everything from program text and mmaped files to active memory to freed memory that hasn’t been reused yet. It’s a vast expanse of stuff you don’t need, must of it binary. Luckily, most programs will just store textual content as ASCII or UTF8. Assuming you were writing English, then the strings utility will be sufficient to find your text. So you can run strings ecto.core > ecto.strings. This will generate another large file (64 megs this time, not 1.3 gigs) with just the ascii string data from your programs memory. Still a lot to wade through, so I use grep -i to look for uncommon words in my post, and less to be able to page around the file quickly.

I wish that the story had a happy ending, but after all that I found that the ecto memory space contained a dozen copies of my post, but all of them were the truncated version that it had posted on my blog, rather than the actual text I wrote. So you will have to wait until next week (or at least tomorrow) to learn what I had to say about house renovation.

Firefox just crashed, and when I restarted it I was informed that it had some trouble reopening my 115 tabs. Understandable, but I went ahead and clicked the button that encouraged it to try harder. The result, after consuming what would have been several thousand dollars of computer time back when they charged for it, is that I’m back in the mess I made myself, with an unmanageable collection of pages open, covering topics like faucets, washing machines, coffee brewing, JVMs, whatever else came out of Google Reader, and the research I was doing for this post.

Our civilization has had tabbed interfaces since 1988. Mainstream browsers (well, Mozilla) have supported tabbed browsing since 2001. As tabbing has become more mainstream, as the web has gotten more complex, and as computers have gotten fast enough to handle dozens of open web pages, people have opened more and more tabs. The result is that nearly everyone knows the experience of “just having to close some tabs” before you reboot, or so you can get on with more important work. It’s easy to overwhelm yourself with the amount of content you can have open in tabs, and clearing it out is often an archeological experience spanning the last week or month of your web activities.

Browser tab management represents the greatest software usability challenge of our time. We are all facing information overload of one form or another, and this is an opportunity to improve the way people find, consume, retain, and manage information. Lately we are seeing a lot of attempts at innovation here, from Chrome’s tear-away tabs and performance optimization, to Firefox extensions like Ctrl-Tab. Most recently today we saw Tab Candy, a preview of new functionality in Firefox 4.

I’ve often said that I’ll switch to the first browser that doesn’t make me feel guilty for having 100+ open tabs. Looking at Tab Candy and other innovations, I see that we are moving in the right direction. But there are still many aspect of the problem that aren’t being sufficiently addressed.

• Application-oriented tab organization – Many of the sites I use today are really applications, whether it is Google Docs, Facebook, or Amazon. Taking application behavior into account, and helping me to avoid opening the same heavy application (e.g. gmail) multiple times is part of getting tab management correct.
• Automatic tab organization – Systems like Tab Candy require that users manage tabs. But wouldn’t it be better if tabs were managed and grouped automatically?
• Managing the tab/bookmark continuum – Leaving something open in a tab, or opening it in a tab, is often a way of deciding to come back to it later. Bookmarks accomplish the same thing, but most people’s bookmarks are themselves a usability nightmare. Automatically migrating tabs into bookmarks and bookmarks back into tabs might be the solution to a lot of these tab problems.
• Excursion and history management – Web browsing isn’t a linear process, the way browser history would have you think. It is at least a branching tree, which tabs support. But often the process of web browsing is a product itself, as when researching a new subject or deciding on a purchase. Being able to not only manage the process as it is happening, but also to archive it for later resumption or reference, would improve the efficiency of many browser tasks.
• CPU efficiency – An implementation detail to be sure, but one of the obstacles to running with 100+ tabs is the CPU load of all the little Javascripts. Some way to manage this, pausing or closing pages which are not visible, is likely to be required.

As you can see, there is still plenty of room for improvement in the browser interface, particularly around large numbers of tabs. What other cutting edge stuff have you seen? What would you like to see implemented?

I spent the first half of last week at DEBS 2010 at King’s College in Cambridge, UK. It was a great conference, many good papers and interesting attendees. As usual some of the best ideas came from the hallway sessions. But I’d like to provide some pointers to my favorite papers of the conference:

• David Jeffery’s keynote on Betfair’s event driven architecture, past present and future. David presented not only on the performance challenges of running the core betting exchange, but also on the soft benefits. Thanks to their event driven architecture, Betfair is able to do more experimentation with less disruption, and be more agile as an organization.
• Dan O’Keeffe presented Reliable Complex Event Detection for Pervasive Computing, a system for compensating for missing data. Most importantly, it lays out a selection of strategies that developers can select and the system can use to automatically choose the correct approach, based on wether the system should be optimistic, pessimistic, or something in between.
• Quilt: A Patchwork of Multicast Regions was presented by one of the local students on short notice, because the author was not able to attend due to visa problems. This is especially frustrating when the paper is so interesting and the analysis quite strong. Quilt is a system for combining multiple delivery mechanisms to achieve efficient wide area distribution. Combining overlay-network based protocols with “patches” of true multicast where available, Quilt optimizes message routing for efficiency, reliability, and latency.
• An Approach for Iterative Event Pattern Recommendation which was also presented by a colleague rather than one of the authors due to visa issues. The paper describes a system for recommending event patterns to domain experts based on their initial attempts to define the pattern. In a controlled user study, the recommendation system substantially reduced the time taken by users to define novel patterns. It was good to see a real user study of a programming efficiency system, but there is lots of room for further measurement.
• Experiences with Codifying Event Processing Function Patterns, presented by the author Anand Ranganathan, dealt with a different kind of pattern. In systems using Ranganathan’s system, developers build template network flows, annotated with tags to define what components can be used in the flow. Any component that matches the inputs, outputs, and tags can be inserted into the template, and all possible template instantiations represent a domain of valid applications. End users are able to search this combinatorially large domain with a flexible interface, to find and instantiate applications that meet their needs.

There are many great papers and talks not listed here. Obviously my own bias shows, as does the fact that I didn’t attend every talk or read every paper.

Unfortunately most of the links go to the ACM portal, which doesn’t offer public access. My frustration with and thoughts on the future of academic publishing will have to wait for a future post. I recommend googling the titles and finding them on author pages if you can. Or ping me and I can send you papers.

Next year the conference will be hosted in New York by IBM research. Whether you attended or not, if you have any feedback on how to make the conference more appealing, I’m happy to hear it.

Talking to some friends recently, I’ve realized that many people don’t think of themselves as networkers. They are reluctant to get started playing that “game”. Even when they are highly capable engineers looking for a new position at a startup, they don’t want to “ask their friends to find them a job.”

Everyone knows that networking is the best way to get a job. Not everyone is prepared to attend “networking events”, hand out business cards to people they meet, or spend all their time maintaining professional relationships. If you are one of these people, I have good news. You can still benefit from networking to find jobs and other opportunities. Merely by being non-hostile, open to the possibility of networking, you can benefit from the networks of people you already know, your friends and coworkers.

The reason is simple: networkers need you. People in the world who network, who spend lots of time maintaining relationships, are participating in a gift economy. They trade favors, introductions, contacts, and other information. Most networkers are looking for win-win situations, connections they can make which benefit both parties. When they get you a job, they are also helping someone fill a position.

These networkers need raw material, which comes from people like you who are not otherwise plugged into the network. By being open to networking, you let them help you. Here are three simple principles to be open to networking:

• Be open. When you meet someone socially, and they are interested in you, tell them about yourself. If you are looking for a new job, or new opportunities, or about to finish a program at school, or an expert in some part of your field, feel free to volunteer that fact. This isn’t begging, it is giving people the opportunity to help you out.
• Be specific. The more specific you are when telling people about your interests, the easier it is for them to help. No one wants to flood their network with a request for “a job”, but if you are “an experienced robotics engineer looking for a clean tech startup,” that message is valuable and easy to route to the right place.
• Be appreciative. People who network do it because they like helping people. But do not immediately respond with a gift or other token. Networking is long term game, and there will likely be opportunities to reciprocate in the future. Or you can pay it forward, by doing your own part to help others in a similar way.

Networking doesn’t have to mean pushy conversations with strangers. By maintaining your existing social relationships, and being open, specific, and appreciative, you can let other people do the networking for you. Good luck.

Since writing about stock market crashes and normal accidents, I spent even more time talking about the events of May 6th. Good analysis is starting to come out. The best I have seen so far is Nanex’s Flash Crash Analysis. Their conclusion is that the crash was precipitated primarily by a queuing and timestamping bug at NYSE, which lead to understandable but unfortunate flash mobbing by high frequency trading firms attempting to execute against the delayed prices being quoted out. I recommend their analysis and the supporting charts, which are quite compelling.

Nanex makes a few concrete suggestions: NYSE should fix their timestamping logic, HFT firms should be discouraged from what they call “quote stuffing”, and quotes should have a minimum time to live of 50 milliseconds. The last suggestion is the most interesting, and would have a significant impact on market dynamics, but possibly not a significant impact on prices or the availability of liquidity. Many other markets (Brazilian equities, US futures) have some kind of charge for canceling or modifying orders. This kind of restriction or discrimination doesn’t prevent high frequency traders from operating, but it does change their strategies, and it might help to discourage runaway markets.

What I have found myself recommending, rather than these changes to address the immediate problem, is a need for cultural change, to identify potential future system accidents and resolve them before they make headlines. In part 2 of the Nanex report, they identify two previous days on which a similar delay in NYSE quotes occurred without triggering a run on the market. With 20-20 hindsight, one might guess that investigations of these previous events would have been sufficient to prevent or limit the damage on May 6th.

Unfortunately, the culture of American capital markets is hyper-competitive, and that’s something our governments has encouraged through deregulation. Market operators are in competition with one another, brokers are in competition, trading firms (HFT and otherwise) are in competition. A culture of secrecy, and of exploiting weaknesses, makes investigating anomalous market events and other bugs very difficult. Technological transparency is important for the health of the whole financial system, but firms aren’t yet ready for it.

Short of distasteful regulatory enforcement, it’s hard to see how we would get people to participate in any kind of information sharing. But if we want to avoid future instances of market, we’ll need to find a way. There are always going to be bugs in the software we use to operate our financial system. The question is whether we find them among friends, or if we wait for the bugs to make headlines.

In the weeks since the precipitous and brief stock market crash on May 6th, I have found myself answering questions about it from people outside the capital markets and discussing it with insiders on many occasions. While I have some thoughts about what went on, I’m often unable to satisfy people’s desire to blame a single precipitating cause. I think what is going on is that too few people understand the nature of complex systems and what is called a “normal accident.” Given the sophistication of the markets, the number of safety checks and balances, as well as the complexity of the implementations, it is not surprising that events such as May 6th happened, nor should people think it is possible to entirely eliminate them.

Normal Accidents (or as wikipedia calls them “System Accidents“) are major failures caused by unintended and unexpected interactions of many small failures. The term was coined by Yale professor Charles Perrow in his book Normal Accidents: Living with High-Risk Technologies. Complex systems fail for complex reasons. In systems engineered for safety and redundancy the failures that do happen require many contributing factors. Perrow’s focus was on large industrial systems, such as power plants, chemical plants, aircraft, shipping, and military operations. TIme and again we see complex failures in places like Three Mile Island, the Challenger shuttle, or BP’s current oil spill.

In a normal accident, the contributing factors come from many areas and often many organizations. Errors result from poor regulation, lack of training, operator error, specification errors, mechanical failures, lax maintenance, poor morale, organizational structures, economic incentives, and many other areas. Because systems are tightly coupled, many of these factors are able to mutually reinforce one another to lead to systemic failure. The resulting cascade of failures can look like a Rube-Goldberg machine in it’s complexity.

In these tightly coupled systems, potential-normal-accidents are happening all the time. Systems are too complex to be entirely without failures. However, in the common case these partial failures are caught and resolved quietly. In fact, these near misses are an opportunity to understand the unintended failure modes of the system. Rather than build once and deploy, safety must be a continuous process of improvement and understanding. Systems aren’t stable and they are not deployed in a vacuum. As they evolve, failures and near misses must be examined and used to drive improvements.

Software, especially modern networked software, dramatically increases the incidence of normal accidents. As anyone who has ever created, deployed, and debugged software knows, it is common for individual software bugs to have all the characteristics of a normal accident all by themselves. Add together software written by multiple different organizations connecting over a network and it’s a wonder anything works at all.

Getting back to the events of May 6th, the “Flash Crash“, they are best explained as a system accident. People have tried to blame one cause or another, from a fat fingered trader or a faulty brokerage system, to investor agitation over Greek debt and high frequency trading firms going wild, to the NYSE hybrid market system, bugs in other members of the national market system, and outdated circuit breaker regulations. Without going into detail about all these potential causes, I’d like to suggest that the most likely explanation is that all of these causes, together, are what created the exceptional failure of market prices, broken trades, and finger pointing. No one cause is really more precipitating than any other, and apportioning blame is much less important than understanding in detail what happened.

It is impossible to eliminate normal accidents as we increase the complexity of our systems. The best we can do is to learn from accidents, and from near misses, to introduce the kind of slack in our systems that will protect us from the worst accidents. Learning requires transparency. But in systems which cross organizational and regulatory boundaries, with billions of dollars and reputations at stake, transparency is going to be a challenge.

PS: If you’re interest in learning more, I suggest this NASA powerpoint on normal accidents.

Lately I’ve been looking at a lot of houses. I’ve also been reading A Timeless Way of Building (ATWB). The net result has been a deep dissatisfaction with the available housing stock in Arlington, and probably in the entire United States. So while I would like to recommend the book, it comes with the disclaimer of being hostile to casual house hunting. Instead it will help you develop opinions about everything.

I started out reading ATWB because of the Computer Science implications. Design patterns, a popular notion in software development, are based on the notions of pattern language developed in ATWB by Christopher Alexander. Alexander’s book is about the architecture of buildings and towns, rather than of computer programs. But I wanted to get back to the source to understand where design patterns came from. More on that later.

As a book about architecture, being read by a layman (me), A Timeless Way of Building is fabulous. It lays out the general notion of patterns, and helps you begin to understand why some buildings work while others do not. Beyond design principles for good buildings, ATWB lays out the societal drivers for bad buildings in our culture. Good buildings are defined by patterns, patterns that work together to form a pattern language. Bad buildings generally result from failing to understand the pattern it is trying to follow, or from not having a pattern at all.

A pattern is a way of build something, a functional unit of building. For example, a parlor at the front of the house, or a front porch, or a farmhouse kitchen. Some patterns may nest inside other patterns, for example an eating alcove might be part of a farmhouse kitchen. A pattern language combines a set of interdependent and self-sustaining patterns to form an ecosystem of buildings that work well together. For example, a pattern language might describe everything from the town square to the livestock pens of a rural French farming village.

Alexander’s patterns are based around human behavior. The pattern only comes about because of how the users interact with the building, and often how the culture of the users constructs that behavior. A front porch isn’t really a front porch until you sit on it in the evening, and neighbors out for a stroll stop and say hello. A farmhouse kitchen isn’t just a big room with lots of work surface, plumbing, and appliances; the work that goes on there defines the pattern of the space, and makes it fit into the pattern language around it.

In Alexander’s mind, modern construction and architecture is blind to pattern languages because we have separated the concerns of the users from the builders.

Traditionally when farmers built a cow barn (or when the Amish build a cow barn today) the people building the barn were experts in its use. They were cow farmers themselves, from the local community. The barn was built, with only small variations, in the same way all the other cow barns were built, because that worked. And if some variation in the construction process interfered with cow farming, the builders would be capable of identifying it and correcting it.

Today, when computer scientists decide to build a research center, the job is put out to bid by university committees composed of administrators and researchers. An architect is selected, a building is designed, and building firms are contracted. Through the process new ideas are developed and handed down the chain to be implemented. But at the end of the day, the workers pouring the concrete know nothing about the work that will be done in the new building. And neither do the foremen, the draftsmen, or anyone else. The architect might know a little, but is unlikely to have ever gotten hands on with the work. And the computer scientists, who understand their work, don’t feel they have standing to participate in the building process.

The result can be a woefully inadequate building. The two cases I’m familiar with, are the MIT Media Lab and the Stata Center. Both are cutting edge buildings, and very nice in some ways. But both also have many problems which have been chronicled by their residents as well as independent authorities. The Media Lab features prominently as a failure in How Buildings Learn. The Stata Center is the cover story of Architecture of the Absurd. The latter in particular blames architects and the procurement process. But from the perspective of pattern languages the problems are more systemic.

Getting back to single family houses, they do not suffer from the university procurement problem. But they do suffer from a lack of understanding by builders of how the buildings will be used.

One might expect that any individual would know how a single family home is to be used. This might be true, but what we all lack is a shared pattern language that helps our homes and our neighborhoods work together with our lifestyles and our culture. And that’s a tall order. Our lifestyles and culture are changing dramatically from one generation to the next, faster than we replace our housing stock. It’s not surprising that a house built for young families in 1950 doesn’t match a young family in 2010. Or a three-decker built for middle class professionals in 1910 doesn’t precisely fit the needs of nine grad students in 2010.

But even if we look at new construction it is hard to identify clear patterns. There are some features people like, such as granite counters, big closets, multi-car garages, and open floorplans. But these don’t come together to form a pattern language. They don’t say how large the family will be, how it will take meals, or how social entertaining will be organized. New houses built on spec are designed to sell, with curb appeal and attractive luxury features prioritized over usability. People buy what they’ve seen on TV, even though most of the houses they see on TV only have three walls.

Reading a book on capes, I learned that the cape house was designed to be built over time as your family grew. You’d start with a fireplace and two rooms, the door on one side. When you had children, you would build the other side of the house. They were easy to extend, adding breezeways, outbuildings, workshops. Or add dormers upstairs and sleep on the second floor. The evolution of the house mirrored the lifestages of the family.

In 2010, few families if any build their house in stages this way. They buy complete houses, and either move or have custom renovations done by professionals when the house no longer meets their needs. And families don’t follow a single path through the stages of life. Some families live multigenerationally, others do not. Some families entertain, or have formal meals, or cook, or don’t cook. Families have 0, 1, 2, or more children. All these choices and more mean that your neighbors probably do not live like you.

And that, more than the discipline of modern architecture, is why we have lost our pattern language for single family homes. Pattern languages evolve slowly, as new structures are built. But houses last 60 years or more. 60 years ago there was no birth control, no pizza delivery, no internet, no supermarkets, and no thermal glass. Most families had one car and milk was delivered daily.

Our culture is changing too fast for any evolved pattern language to keep up. We’re stuck with buildings created through intelligent design. Unfortunately, intelligent design isn’t very good.

A number of months ago I posted my disappointment in the version 1 Kindle. I’ve also tried out the version 2, and continue to be convinced that the Sony Reader is a better piece of hardware for dedicated book reading.

But (if there wasn’t a but there wouldn’t be much of a post here) the Sony eBook store is painfully terrible. Titles are expensive, hard to search for, and often not available. The result is that my book buying on the Sony gradually trailed off. On my last two business trips I haven’t bothered to bring the Reader, and it sits on a shelf right now, probably running out of battery.

Last week Aletta wanted a book which was available from Amazon, and didn’t want to wait for it. She downloaded the iPhone Kindle app, bought the book, and was pleasantly surprised. Reading on a small screen is more pleasant than either of us expected, and the Kindle app is quite well designed.

I carry an iPod Touch with me basically everywhere. Switching to Kindle means I can have my books with me even when I don’t want to carry a dedicated eink-reader. I have the option of buying a dedicated reader if I want one.

The only downside is that book ownership is still restricted to a single account, from what I can tell. Aletta and I solve this by keeping all our ebooks on a single account, and that’s no worse than Sony. But there is definitely room for improvement in managing household book collections and book sharing. Hopefully between Sony, Amazon, Apple, and Google we have enough competition to find a good set of structures.

In summary, on Kindle:

• Books are easy to buy
• Books availability is superior
• Books are available across multiple devices
• Books are available on devices I already own
• Book access feels a little more future proof against DRM fail. Or at least if there is fail, there will be a critical mass of people building cracking tools.

I hope Amazon gets an Android port out soon, and starts encouraging other companies to make eink-readers that support Kindle. It could be a great ecosystem.

Sony, it’s time to realize that user experience is about a lot more than just the industrial design of the physical product.

In computer science, the halting problem is very well known. The problem states that it is impossible to build a software program that can analyze other software programs to determine if they will eventually terminate, or halt. This is a useful problem to understand, because many software problems that look possible at first can be reduced to the halting problem and thus demonstrated to be impossible. It’s common to hear someone say “actually, that seems like a halting problem” when discussing compiler optimization, program analysis, and related problems in computer science. This is much like a physicist might say “but that’s perpetual motion.”

In the sphere of financial derivatives, our civilization has recently come to understand that there are a whole class of financial products which look attractive, and perform reasonably well some of the time, but which eventually fail. The most obvious of these are the credit default swaps of the latest crisis. But other examples include portfolio insurance, made famous in the 1987 market crash. The problem with these products is that they are designed to protect the buyer against losses in all circumstances, even when the market is behaving badly. But when the market is behaving badly, it can behave very badly. These products reduce to end of the world insurance. When the world is ending, who is left to pay out the insurance?

I think there is a useful parallel to the halting problem. If your new financial product can be used as end of the world insurance, it probably will be. And since end of the world insurance is fundamentally flawed, it should raise questions about what your product is really accomplishing.

My last post ended with me losing the house due to being outbid. In a strange turn of events we may have won the bidding war without ever submitting the highest offer. I’ll try to provide more details at a future date, if it turns out they are interesting. At the moment we are still in negotiation with the seller.