Project Euler is a neat website that presents a series of computer science problems suitable for students and non-students to practice the skill of writing algorithms to solve numerical puzzles. It’s a great way to learn a new programming language, or just to kill time in a mentally engaging way.

Two of the peculiarities of the MIT Mystery Hunt are that the puzzles are drawn from a wide range of domains, and that there are no restrictions on what resources can be brought to bear on a puzzle. The result is both puzzles that are explicitly about software (e.g. perl program cryptograms) as well as puzzles that are best solved by writing some kind of program. Many of my favorite puzzles over the years have come from this category. Some hunts have few, some hunts have many, but every year there is a call to write some kind of software solution to a puzzle.

The programming skills required to write puzzle solving programs in high pressure situations are different from those normally required to write software, and so it can be useful to practice them. I have collected and categorized here all of the software puzzles since the 2000 hunt. In most cases, I recommend attempting to solve the puzzle, knowing that software is likely the best way. But because the solutions are available, if you are stumped it can still be interesting to read the solution and attempt to duplicate it in software. In most cases the solution is linked from the puzzle page; where it is not I have provided a link. The categories here are presented from least elegant to most elegant, in my opinion. Feel free to jump around and do puzzles which you find attractive.

Just Decompile It

A useful trick is to decompile any java applet or flash thing that you are given in a puzzle. The answer might just be in a string constant, though more likely it will just give you a different puzzle to solve.

• Maze (2005)
• String Theory (2003)
• Gnireenigne Lab (2003) This reverse engineering puzzle is pretty challenging. Only one team solved it during the longest hunt on record.
• Cubology (2002) Unfortunately decompiling this one doesn’t really help, but what the hey. (answer)
• Twisty Little Passages (2008) Not quite decompilation, but pulling the whole maze down to your local machine is the best way to handle this puzzle.

MIT Computing

• Some Trolleys Named Lust (2006) The title clues Moira, the computer system that does configuration management for MIT Athena, which has command line utilities named after characters from A Streetcar Named Desire. This puzzle contains components which are no longer online, so you are limited to reading through the solution.

Mathematical Manipulation and Brute Force Enumeration

Many search problems can be solves on modern hardware with brute force enumeration, or maybe slightly smart enumeration.

• Bars of Soap (2005) Unfortunately the puzzle isn’t available, only the solution, though it still provides a useful programming exercise.
• Ginormous (2005) Prime factorization is always a fun thing to do with numbers.
• The Road Signs Of Unspeakable Chaos (2003)
• Mathophobia (2001) (answer)

Text Manipulation

Sometimes a program is the easiest way to do a bulk text manipulation. Particularly when you need to experiment with various transforms, or might need to do them repeatedly. This includes basically all cryptograms, which I am not including here, though software often comes in handy solving them.

• Blather (2007)
• Decode This (2006)
• Long Division (2006) Not so much text manipulation as diagram manipulation, but you can get the diagram as postscript, which makes it easy to extract the data. Doesn’t help you figuring out the algorithm, but it does make trying out variations easier.
• Shotgun Wedding (2005) Gene sequencing also comes up occasionally. There are some pretty simple text matching algorithms that turn out to be critical parts of that work.
• Lost (2004)
• Reminders World (2004)
• Whoa — I have a Migraine! (2003)
• Famous First Words (2002) (answer)
• Lions and Tigers and Bears (2000) My first attempt to solve a puzzle with software, this one wasn’t very successful, partly because of the challenge of getting the grid and the source data accurately input, and partly because of a slow algorithm. The fact that computers were notably slower in 2000 might have also been relevant. I think the most powerful machine available to me was an Ultra 10. In the end people doing it by hand finished first. (answer)

File Identification and Manipulation

Many puzzles require you to identify files, or to do interesting manipulations to entire files. Often both at the same time. Sometimes the files are programs written in other languages.

• Hyperextensions (2009)
• Surgical Files (2009) This puzzle triggered a programming language race, between Perl and Haskell. The race ended in a tie, but I learned some cool Haskell tricks from a teammate in the process.
• White Noise (2006) Being able to manipulate audio files with programs or tools is also important.
• Noise in the Air (2004) Knowing your off the shelf cryptography tools (ie, openssl) is sometimes helpful
• Two-Timer (2004)
• A Problem With Printing (2003) This is probably my favorite mystery hunt software puzzle. It’s written in the postscript language, which is a great language for puzzlers to know, and makes use of the peculiarities of that language.

Programming Language Identification and Cryptograms

Many puzzles come down to identifying obscure or not very obscure programming languages. One thinng to be on the lookout for is knitting notation, which can often look like a programming language. Of course, writing programs is often still more efficient than knitting an actual object. Adding complexity, often the programs you are given are cryptograms, or otherwise obfuscated.

• Tragedy (2008)
• Badness 10000 (2006) Unfortunately no longer online, but I like the structure. You can read the solution.
• Square Mess (2005) presents you with a machine language and a set of constraints on the notation used for a program in it.
• Who’s There (2004)
• Sixty Degrees of Separation (2004)
• What Do You Do With A Genteel Sailor (2004)
• Whoa — I Know Knitting! (2003)
• A Nugget of Wisdom (2002) (answer)
• Use of Gothicism Considered Harmful (2002) (answer)

Honorable Mention

Blue Steel (2006) You can try to solve, but you probably just want to read the solution. It will remind you not to think so hard sometimes.

[Edited 2010-01-11 to add Twisty Little Passages (2008). Feel free to add other missing puzzles in the comments.]

Well, it’s been going around my corner of the blogosphere that kdb+ is now free for personal use. I first heard about it from Marc Adler. You can go download it from the kx download page. This represents a big step towards openness, which I think will be good for everyone.

The q environment is impressive, you have to give them that. There is an emphasis on brevity; the OSX binary of kdb+ is only 227K. That’s smaller than the ncurses library it ships with. And brevity doesn’t stop there. Utterances in the language are well known for their complexity and impenetrable internal logic. A lot of q code makes obfuscated perl look clear and verbose. It doesn’t help that the culture of kx programmers discourages commenting. Error handling is tricky at best, and modularity and maintainability are in short supply. For confusion, q adds a bunch of SQL keywords on top of the previous language, k, in an almost but not quite fully compatible way.

But for all the faults you can kind find some really interesting features in q. And if nothing else, it is an example of a novel programming language, tightly integrated with a data management system, finding commercial success, which is always nice to see. And if you learn it, you might find a sweet job on wall street as a result.

• There was of course the workshop on associations as a first class language construct, which I already blogged about here.
• As part of that, Jiri Soukup talked about his project IN_CODE and in particular his UML Layout tool, which attempts to provide sufficiently good graphical layout of UML that the machine-generated layout can be the only layout. The theme of graphical languages with a single canonical display was recurring. I picked it up again from Jonathan Edwards (author of Subtext), who has been working on a canonical visual representation for conditional constructs, and presented it as No Ifs, Ands, or Buts. While I missed the talk, I like the ideas.
• I saw a demo of AmbientTalk/2, which is a cute language that makes event-driven programming the norm. The big idea is that all objects belong to a specific thread of control (an actor), and their methods can only be executed in that thread. At method call sites the caller explicitly calls either synchronously (with a ‘.’ operator) or asynchronously (with a ‘<-’ operator). These ideas may have originated in earlier languages with which I was not familiar (E was mentioned). This definitely buys a chunk of attractive simplicity. However, the demo failed to present any complicated synchronization patterns. I’m forced to wonder if we need better benchmarks for the expressivity of parallel languages. Building an IM system just isn’t very hard.
• Erik Meijer’s Confessions of a Used Programming Languages Salesman was pretty clever, mostly talking about his experience trying to bring fire down off the mount to Visual Basic programmers. Computer Scientists need regular injections of reality when it comes to things like industrial adoption (usually non-adoption) of their ideas. It was also one of several talks at OOPSLA about LINQ and how awesome C# 3.0 is. I’m definitely envious of some of the features of C#, and I hope Java catches up soon (1.7 is looking pretty good in some ways, but certainly not all). Erik talked a lot about the extension points added to C#, but I’m at the moment unable to find anything on the web that tells me how to add LINQ-style keywords to the language for arbitrary languages, like say an extended version of SQL.
• The Domain Specific Languages panel was entertaining, but neither conclusive nor particularly inspired. It was the first place the phrase “Silver Bullet” came up, but far from the last. Unfortunately (and ironically) everyone seems to have a different idea of what a Domain Specific Language (DSL) is, leading to wildly different concerns. More on this in a future post.
• Sean McDirmid’s talk and live coding demo of SuperGlue was the most impressive new thing I saw at the show. SuperGlue is an event driven language and a live programming environment. As you edit programs (in Eclipse, of course) the program is continuously executed and re-executed. Sean built up a game of Pac-man, Julia-Child-style. As each new block of code was added to the program, the game gained new behavior. Even, thanks in part to eclipse, responding to typing successive characters of a numeric constant, like “330″. I really hope that he makes his code, or at least videos, available. Unfortunately, I’m afraid that Sean’s demonstration and discussion of the benefits of live programming overshadowed his more theoretical contribution of object-oriented signalling. His paper from ECOOP 2006, SuperGlue: Component Programming with Object-oriented Signals may discuss that in more detail. Sean is unlikely to keep working on SuperGlue, which is implemented in Scala on Eclipse, as he is going to work for Microsoft.
• Sharing the stage with Sean, linked in the minds of the organizers because both talks had “live” in their title, was Living in the Comfort Zone, which proposes restricting program inputs using a “rectifier” in order to avoid exposing programs to inputs that, while legal, might be unexpected and trigger bugs. Interesting idea.
• The Tuesday night performance of 50 in 50 by Guy Steele and Dick Gabriel was indescribably awesome. I expect a video, and will post a link when it becomes available. Various people compared it to Steele’s 1998 performance-essay Growing a Language, which I had not previously seen and which I can now recommend (you may not understand what he is getting at until page 4).
• Tony Kay presented a way to punctuate transactional-memory transactions to make them nest, which just further convinced me that transactional memory is far from the solution to the difficulty of threaded programming. Their model might be better, but it is at best a patch which solves only some of the problems caused by an insufficient underlying abstraction.
• StreamFlex was definitely interesting. Inspired by StreamIT and other stream processing work, it provides an events-and-operators model not unlike Aurora (the prototype for StreamBase), where the operators are kernels of Java code. The main contribution, to me, was the approach to real-time semantics. They instrumented their virtual machine to provide some simple transactional memory features, and then use them to assure real-time performance of the real-time thread. All method calls into objects shared with the real-time thread are logged using transactional memory. In any conflict between the real-time thread and another thread, the transaction is aborted for the non-real-time thread. However, to make this work they introduce a somewhat complicated object-ownership system, enforced by compile-time typechecking. Combining the real-time approach with the object ownership of AmbientTalk I mentioned above might be interesting.
• Thomas Kuehne tried to answer the question Can Programming be Liberated from the Two Level Style?, but I remained unconvinced. The two level style means classes and objects. In many programs, the “item-description pattern” is used to allow a richer set of “is-a” relationships than the class hierarchy generally affords. While I support the idea of turning design patterns into language features, the mechanics of DeepJava failed to inspire me with a desire to teach them to the average 6.170 student. Inheritance is tricky enough without meta-inheritance.
• No Silver Bullet Reloaded was the Panel Of The Stars at OOPSLA, with Fred Brooks, David Parnas, and other panelists who would have seemed famous in other company. Martin Fowler, in particularly, put on a fabulous performance as The Werewolf (who spent much of his time gloating). The forgone conclusion was that no, there had been no successful silver bullet in the 10 years since Mythical Man Month was updated. As for discussion of what we should do about it, the general idea was that we should do all those good things software engineering pundits have been telling us for years, and that we should make sure to employ smart people. Interestingly, panelists from industry were the ones pushing for employing only smart people, which seems like an ineffective strategy when you are trying to develop software in the large at a company like Cisco. The attitude of cynical self-satisfaction (“industry just needs to do what we already told them to do, which they will never do”) seemed distressingly pervasive at OOPSLA this year.
• I caught the end of a talk by John McCarthy about Elephant 2000, a new language which he has been developing for over a decade. I didn’t really follow his description of Elephant, and after talking to others about it I think it is ill conceived. The second question was, roughly, “Could you talk about your experience inventing LISP instead?”. McCarthy seemed happy to do that, and what followed was 30 minutes of entertaining anecdotes about early computing.
• There were a number of interesting projects presented only on posters at the conference. Chris Pickett of McGill presented Software Speculative Multithreading for Java, which not only contained several interesting ideas, but was the most interactive poster I’ve seen (photo here), being made of movable felt pieces that could be used to talk through demonstrations. The general idea was that at each method call site, come up with a result predictor, and spawn a speculation thread that will continue the computation as if that result had been returned. Speculation can be running several levels deep at a time. This required significant VM modification, which he is now turning into a library so that it can be added to the IBM JVM. There is also a lot of work to be done on heuristics for choosing call sites for speculation, and choosing predictors.
• Talking to people about startups versus large companies, I got an interesting anecdote from someone involved in VoIP at Nortel: In their estimation, it takes them 2 years from project start to get a product into the marketplace, while it takes their startup competitors only 1 year. However, after 2 years they have a product which meets the needs of their customer base, while it takes the startup 3.5 years. Makes me wonder what these constants look like in other product areas.
• I attended the Seaside BoF. I find a lot of the Seaside technology very interesting, but as befits their roots in the Smalltalk community, they have a lot of difficulty marketing themselves. I ended up serving as a resident heretic, which I hope was helpful, as their main topic of discussion was how to make Seaside adoption as easy as Ruby on Rails adoption. My heresies included “What do you mean I have to use a special smalltalk editor instead of emacs?”, “Are continuations really scalable? Why can’t I get a straight answer?” and “How am I supposed to choose a runtime/database/editor/etc when I don’t know anything about Smalltalk yet?” This turned into some extensive discussions about the adoption curve of destabilizing development tools, which I will try to write up later.
• I had a slow morning on Thursday and missed Context, Perspective and Programs, which for many people was something of a religious (or drug-induced) experience. That’s what I get for not reading speaker bios in addition to abstracts. Hopefully there will be a video.
• I sat in on a talk titled Neovictorian Computing, which had a lot of good quotes, but didn’t say a whole lot. The best quote, which reflected a lot of what went on at OOPSLA, was “The pedagogy of software engineering is almost entirely about guilt.” The speaker is the creator of Tinderbox, a tool for keeping personal notes. The thrust of the talk was that we need to empower our users to be craftspeople, and to produce small software systems that meet their own unique needs. Tinderbox, as described in the talk, seemed potentially the right system for my contact management needs (I’ve been unsatisfied with all off the shelf contact managers). Unfortunately, I found it difficult to use and not nearly as interesting as the talk had made it seem. Maybe it is the best we can do, when targeting end user programmers (the tool is used mostly by authors to manage book notes), but I would hope we could do better.
• I had a couple of talks with Canadians about how their country could benefit from more startups. Unfortunately, most encouragement of startups seems to be government funding, which I think is fundamentally flawed. More on this later.
• The final talk of OOPSLA was Brian Foote, revisiting his paper Big Ball of Mud, about the commonness of evolved software systems. He is a very good speaker, and it was an entertaining talk. His focus was on finding the right metaphors to communicate how something is working. My favorite was “sustainable development”, suggesting that many deadline-driven software projects take unsustainable shortcuts, much like some farmers in the Amazon, prematurely depleting a programs maintainability. But he was also quick to point out that some techniques that software engineering pundits disdain, like cut and paste, are a fundamental part of nearly all software development, and that when you let reuse become the overriding concern, you can end up with something highly inefficient, like the Space Shuttle (the reuse of the machine, not the software, in that case). He also described at length about how we are losing the “War on Bugs”, how attempts to crack down on domestic production have merely driven bug production overseas. Programmers start with “gateway bugs” like unchecked error codes and undocumented exceptions, and move on to harder bugs like cross cutting dependencies and threading. Foote went on like that for a while. I may just pull the quotes out of my notes and post them separately.

That’s it for my train of consciousness notes from OOPSLA 2007. It inspired a lot of thoughts, so you may be hearing more from me on some of these topics.

The title implied it was the workshop about graphical programming languages and tools. As an author of a graphical programming language, StreamBase StreamSQL EventFlow, it seemed like the place to be. My plan had been to pick up some contacts in the graphical languages space, and learn about what has come before. However, the participants (with one vocal exception) had already accepted as a given that UML would eventually decline in popularity, to be replaced by a and they were focused on what would come next.

In fact, the general agreement about what comes next is first class associations. An association is a relationship between two classes that relates instances of the first with instances of the second. These are the lines in UML diagrams. The focus of the workshop was on creating language constructs to allow realization of these lines (associations) in code and the creation of libraries of types of associations.

In UML, most boxes are realized concretely as a Class. This makes translation to and from UML diagrams and other models straightforward for classes. But most lines are realized in an implementation dependent and error prone fashion. For example, there might be a pointer from objects of one class to the other. Or there might be pointers in both directions. Or a set of pointers, or an externally stored map.

Any of these implementations requires maintenance logic, generally embedded in one or both class definitions. This logic is error prone and can get out of sync. If the association is subject to additional constraints, such as a graph being acyclic or conforming to some security model, the maintenance logic can get very complex. At some point a separate class might be created just to track instances of the relationship, but this is a discontinuous change to the program, and impacts all clients of the relation. Much better if there were a concrete realization of associations somewhere in the program, where changes to the association could be made.

Three concrete implementations were presented, for C++, Java, and C#. The C++ implementation, by Jiri Soukup and called incode, was implemented via code generation. The Java and C# systems, presented by James Nobel and Kasper Østerbye respectively, utilized Aspect Oriented Programming (AOP) tools to inject required logic into classes at runtime. Kasper’s work is described in a paper, Associations as a Language Construct, James’s in Relationship Aspect Patterns. Stephen Nelson also described but did not present his work, and Mike Heffernan presented an in-house system he has developed at OpusEdge.

The OpusEdge system, in particular, focused on using associations and the objects they related to control database schema generation and database access. In this case the associations and objects form a Model in the context of Model, View, Controller. His system, implemented in .NET, provided transparent network access to the model from clients, as well as server side access.

While this was not the workshop I had intended to attend, it was all very interesting to me. The interplay between databases and programming languages, and between the relational model and object oriented models, is a subject which holds long-term interest to me. The question of whether associations should be binary or N-ary is similar to the question of RDF versus fully-relational in databases.

If we expand language functionality to enable associations, or just let the AOP beast out of its cage all together, it would greatly improve the rapid-development experience. Today, many MVC frameworks create degenerate views given a model, and some even take a programming by exception approach to generating more interesting views. WIth associations as first class manipulable constructs, the model can be enriched with more information (by using a library of association subtypes). This can in turn drive better automatic behavior in areas such as view generation, model validation, and persistence.

We are going to continue conversations about associations, electronically, in a forum moderated by Stephen Nelson. I will be interested to see what progress is made. I have two burning questions: How can we leverage associations to improve database interactions? And, once we have associations, what traditional programming language constructs can we drop?

The focus of the talk was on the subtle issues that come from this. It dismissed a few of the more positive myths about transactional memory:

• Programs using locks cannot be trivially converted to use transactions. In fact, a correctly behaving program using locks to manage things like inter-thread communication can easily deadlock or livelock when using transactions.
• Transactions are not perfectly composable or nestable, as is often claimed. This means heirarchical or nested transactions, where there is a program-level transaction wrapping several library-level transactions, can lead to deadlock or livelock.
• Whether a transactional system is weak (non-transactional code executes concurrently with transactional code, with side effects visible between) or strong transactional (a transaction is atomic not just from the perspective of other transactions, but also to other non-transactional code) can have a significant effect on program behavior. Not only can a program designed for strong transactions have problems under weak transactions, but a program that behaves correctly in a weak transactional system may deadlock in a strong transactional system.

This leads me to two primary conclusions, with which I think the speaker would agree: First, transactional memory can have significant confusing side effects, just like locks, and so it is not a solution to the difficulties of multithreaded programming. Second, if processor vendors implement, and many programmers use, weak transactions then we may never get strong transactions.

I’d like to take this one step further. I think language designers and computer architects who are excited about transactional memory are missing the point. They are trying to create a single concurrency control mechanism that can solve everyone’s problems. In fact, I think there are two separate classes of problem that are best addressed separately: Concurrency control for systems programming, and concurrency control for application programming.

Systems programmers are close to the hardware, inside or right on top of the operating system. They are the people who are most comfortable with the concurrency control mechanisms of today (pthreads and java.util.concurrent). Transactional Memory should be targetting these users. The goal is not to give them an easy way to do concurrency, but to make optimistic concurrency control possible and efficient. Continuing to discuss the merits of strong and weak transactions is reasonable, but it should be in the context of what most efficiently represents the capabilities of the hardware.

Application programmers generally work on top of a platform, and don’t have to think about concurrency in the same way. if they don’t work on a platform then they should. For example, application programmers are often writing business applications as web services on top of J2EE. By using platforms, they can work with much higher level management and concurrency control. The most common idiom for concurrency control is the database transaction (or distributed transaction). It is able to handle most concurrency problems in traditional RPC-and-datastore systems.

To help application programmers, we need to develop additional idioms. Not all programs can utilize the J2EE style of platform. I expect to see a significant advance in scientific computing tools like MATLAB to take advantage of multiprocessor systems (in my ideal world it would be by vectorizing based on static and dynamic analysis, but I digress). Furthermore, there are a large set of event processing (stream processing, complex event processing) which are also not well served by J2EE-type platforms and database-transactions. Which is why I work on StreamBase (and on StreamSQL), which is creating next generation programming models for those kinds of applications, to enable parallelization, managability, and scalability.</shameless-plug>