If you read the title of the post, you can already guess that my answer was neither.

Traditional row stores are optimized for transaction processing, in which records are updated. They are also sufficient for small data sets and ad hoc queries, as you find in a normal web app. Column stores are optimized for ad hoc queries, but this time against very large sets of data.

In either case, the data store is designed as a shared resource to cope with IO bound operations. But for many applications, the database is neither shared nor IO bound. In this application, the large data store is used by a small set of applications (the data loader and the data reducer), which are long running with predicable access patterns. This is in sharp contrast to the kind of load databases are designed to support.

The answer is to look at much simpler tools like BigTable, which are optimized for precisely this kind of access pattern. BigTable stores data in columns, but spends considerably less effort optimizing the IO of the data. Instead, it assumes the application will access the data in a reasonably efficient way. It’s great if you have a single application or single kind of application working over your data. In fact, BigTable even interoperates well with Map/Reduce, which is a reasonable basis for the compute intensive part of this application. The Hadoop implementation of these technologies looks pretty good.

There is an ad hoc query component to the application, but it is ad hoc queries of the results of the data reduction. The results are effectively a pre-aggregation of the large data set. There is a two-tiered storage model already built in to the design.

Now, an idealized database management system, the kind that you get along with your flying car, might make this two tiered design unnecessary, or at least automatic. It ought to be able to index the huge data set and support queries against it, notice patterns in those queries and store pre-aggregated information in order to service common queries quickly. But no readily available system comes even close to that level of self management. In fact, you would have a hard time finding a DBMS that gives you really rich and pleasant to use semantics for maintaining materialized views, never mind automatically creating them.

So, for now we build knowledge of these limitations into our system architecture, take a two tiered approach, and can benefit from the good work done by Google at building a very large, very dumb data store.

Google is quietly at it again, this time with the Google AppEngine.

This time, they are taking a shot at transaction processing. This is particularly amusing, because Stonebraker himself has been predicting an upset in transaction processing for a few years now. But rather than demonstrate tight integration between a app language and the database optimizer, or use traditional stored procedures, Google is again causing problems by building the simplest thing that can possibly work. And simple developers with simple applications are going to like it.

The Google App Engine data store is a non-relational database, with support for an SQL-like but limited language called (unsurprisingly) GQL. There are no joins. The lock management is simplistic. It is basically an entity-relationship database, with fixed and variable attributes associated with each entity. Every query is always a SELECT * (“The SQL syntax is retained for familiarity”).

The whole thing is wrapped in a Python object-mapper, which means it looks quite a bit like an object database with a simplified model. This turns out to be what most people who are building web applications, whether in PHP, Ruby on Rails, or Java (Hibernate). Delivering a database which is limited to this functionality puts google in a powerful position to optimize specific use cases for simplicity and performance.

We are at the start of (or possibly in the middle of) a cycle that we haven’t seen in over 10 years: Existing assumptions about databases are being questioned, and a thousand flowers are blooming. Stonebraker himself pointed this out in his One Size Doesn’t Fit All papers, talks, and columns. However, it is not just the Oracle of the world who need to change or get out of the way. It is also the leaders of the database research community and their dominant paradigms. Things are going to be messy for a while yet. But in the end we will have technology that better fits modern infrastructure and modern usage.

The title implied it was the workshop about graphical programming languages and tools. As an author of a graphical programming language, StreamBase StreamSQL EventFlow, it seemed like the place to be. My plan had been to pick up some contacts in the graphical languages space, and learn about what has come before. However, the participants (with one vocal exception) had already accepted as a given that UML would eventually decline in popularity, to be replaced by a and they were focused on what would come next.

In fact, the general agreement about what comes next is first class associations. An association is a relationship between two classes that relates instances of the first with instances of the second. These are the lines in UML diagrams. The focus of the workshop was on creating language constructs to allow realization of these lines (associations) in code and the creation of libraries of types of associations.

In UML, most boxes are realized concretely as a Class. This makes translation to and from UML diagrams and other models straightforward for classes. But most lines are realized in an implementation dependent and error prone fashion. For example, there might be a pointer from objects of one class to the other. Or there might be pointers in both directions. Or a set of pointers, or an externally stored map.

Any of these implementations requires maintenance logic, generally embedded in one or both class definitions. This logic is error prone and can get out of sync. If the association is subject to additional constraints, such as a graph being acyclic or conforming to some security model, the maintenance logic can get very complex. At some point a separate class might be created just to track instances of the relationship, but this is a discontinuous change to the program, and impacts all clients of the relation. Much better if there were a concrete realization of associations somewhere in the program, where changes to the association could be made.

Three concrete implementations were presented, for C++, Java, and C#. The C++ implementation, by Jiri Soukup and called incode, was implemented via code generation. The Java and C# systems, presented by James Nobel and Kasper Østerbye respectively, utilized Aspect Oriented Programming (AOP) tools to inject required logic into classes at runtime. Kasper’s work is described in a paper, Associations as a Language Construct, James’s in Relationship Aspect Patterns. Stephen Nelson also described but did not present his work, and Mike Heffernan presented an in-house system he has developed at OpusEdge.

The OpusEdge system, in particular, focused on using associations and the objects they related to control database schema generation and database access. In this case the associations and objects form a Model in the context of Model, View, Controller. His system, implemented in .NET, provided transparent network access to the model from clients, as well as server side access.

While this was not the workshop I had intended to attend, it was all very interesting to me. The interplay between databases and programming languages, and between the relational model and object oriented models, is a subject which holds long-term interest to me. The question of whether associations should be binary or N-ary is similar to the question of RDF versus fully-relational in databases.

If we expand language functionality to enable associations, or just let the AOP beast out of its cage all together, it would greatly improve the rapid-development experience. Today, many MVC frameworks create degenerate views given a model, and some even take a programming by exception approach to generating more interesting views. WIth associations as first class manipulable constructs, the model can be enriched with more information (by using a library of association subtypes). This can in turn drive better automatic behavior in areas such as view generation, model validation, and persistence.

We are going to continue conversations about associations, electronically, in a forum moderated by Stephen Nelson. I will be interested to see what progress is made. I have two burning questions: How can we leverage associations to improve database interactions? And, once we have associations, what traditional programming language constructs can we drop?

Vogels presented lots of information. A big part was about the operational realities of a site as large as Amazon. This has been said before, but saying it to a room full of database researchers was a good thing:

• Incremental scalability, up or down, one node at a time, is a key requirement
• Adding resources for redundancy must not hurt performance (best if it helps).
• Failures are not uncorrelated, they are generally quite correlated.
• Systems that fail do not fail stop, they generally fail in some more complex way (eg, periodically coming back up and failing again, or outputting garbage).

Amazon’s query workload shares a lot of characteristics with other web sites with which I am familiar, such as eBay, Google, Facebook, and LiveJournal. Amazon uses lots of read mostly workloads and lots of pkey-lookup-only workload (65% of all queries are primary key). Only 5% of their queries need full RDBMS query functionality. Most of their writes are also by primary key, and many storage systems support only that kind of write.

When it comes to those writes.

• Only 10% of queries need strong consistency.
• “We consider strong consistency to be evil,” because it is impossible to implement without the potential for downtime or failure.
• Most systems at Amazon are designed for eventual consistency.
• One interesting case is the need for always-writable data storage. One use case is ordering. “If the customer decides to give you his money, you must always take it, that’s a business principle.” Always writable storage obviously implies a conflict resolution scheme.

All this points to a traditional RDBMS being the wrong system. Interestingly, if you assume you need a single general purpose system, then RDBMS is the best available. But the real answer is to use a set of customized data management tools. This all echos Stonebraker’s claim that one size no longer fits all uses.

Where do we go from here? The real problem for the research community is not to help Amazon and Google build a 5% better system. The real problem is to package up (aka abstract) these tools in such a way that, like databases, average programmers can work with them. Because problems that PhDs at Amazon and Google solve today will soon be problems everyone needs to solve.

The focus of the talk was on the subtle issues that come from this. It dismissed a few of the more positive myths about transactional memory:

• Programs using locks cannot be trivially converted to use transactions. In fact, a correctly behaving program using locks to manage things like inter-thread communication can easily deadlock or livelock when using transactions.
• Transactions are not perfectly composable or nestable, as is often claimed. This means heirarchical or nested transactions, where there is a program-level transaction wrapping several library-level transactions, can lead to deadlock or livelock.
• Whether a transactional system is weak (non-transactional code executes concurrently with transactional code, with side effects visible between) or strong transactional (a transaction is atomic not just from the perspective of other transactions, but also to other non-transactional code) can have a significant effect on program behavior. Not only can a program designed for strong transactions have problems under weak transactions, but a program that behaves correctly in a weak transactional system may deadlock in a strong transactional system.

This leads me to two primary conclusions, with which I think the speaker would agree: First, transactional memory can have significant confusing side effects, just like locks, and so it is not a solution to the difficulties of multithreaded programming. Second, if processor vendors implement, and many programmers use, weak transactions then we may never get strong transactions.

I’d like to take this one step further. I think language designers and computer architects who are excited about transactional memory are missing the point. They are trying to create a single concurrency control mechanism that can solve everyone’s problems. In fact, I think there are two separate classes of problem that are best addressed separately: Concurrency control for systems programming, and concurrency control for application programming.

Systems programmers are close to the hardware, inside or right on top of the operating system. They are the people who are most comfortable with the concurrency control mechanisms of today (pthreads and java.util.concurrent). Transactional Memory should be targetting these users. The goal is not to give them an easy way to do concurrency, but to make optimistic concurrency control possible and efficient. Continuing to discuss the merits of strong and weak transactions is reasonable, but it should be in the context of what most efficiently represents the capabilities of the hardware.

Application programmers generally work on top of a platform, and don’t have to think about concurrency in the same way. if they don’t work on a platform then they should. For example, application programmers are often writing business applications as web services on top of J2EE. By using platforms, they can work with much higher level management and concurrency control. The most common idiom for concurrency control is the database transaction (or distributed transaction). It is able to handle most concurrency problems in traditional RPC-and-datastore systems.

To help application programmers, we need to develop additional idioms. Not all programs can utilize the J2EE style of platform. I expect to see a significant advance in scientific computing tools like MATLAB to take advantage of multiprocessor systems (in my ideal world it would be by vectorizing based on static and dynamic analysis, but I digress). Furthermore, there are a large set of event processing (stream processing, complex event processing) which are also not well served by J2EE-type platforms and database-transactions. Which is why I work on StreamBase (and on StreamSQL), which is creating next generation programming models for those kinds of applications, to enable parallelization, managability, and scalability.</shameless-plug>