Is anyone dumb enough to think yet another parallel language will solve our problems? I MIGHT be!

I have been pleased by the attention my blog on choice overload has received. I must admit, I overstated things a bit just to get people talking … and on that count, I clearly succeeded.

In my present blog, I’d like to respond to comments on my choice overload blog .. and then discuss how dumb I might be.

To start with, its important to note that this parallel programming problem is not an Intel problem. Every major vendor of general purpose CPUs has fully committed to the multi-core path. And hence, all of us are in the same boat … if our processors are to be of any value as the core counts climb, we need parallel workloads. This isn’t whining; it’s reality, and the entire industry must adapt.

Also, I can’t stress enough that my “choice overload” blog was not merely theoretical musings. It is based it on experience. Parallel computing is not new. I’ve been involved with parallel computing since the mid 80’s. I have been directly involved with serious attempts to establish 4 new parallel languages and peripherally involved with several others. Some (such as OpenMP) have met with some level of success. Most have failed. I’ve worked with software engineers to help them transition to these new languages. I’ve shown them the clear superiority and ease of use of these new parallel languages; and then watched these new languages go down in flames. I’m telling you … based on my experience living through the early decades of the parallel computing era, choice overload is real and we as an industry ignore it at our peril.

At the same time, I need to be clear on who exactly is doing the choosing. I spoke of the “masses of programmers” without being clear on who these “masses” were. My focus is on the countless hordes of professional software engineers who create the software we use on our computers. These are the people motivated by hitting deadlines, shipping new features, and earning their living with their software.

I am much less concerned with computer scientists doing research in languages or parallel computing. They are interested in research agendas, exploring new territory, and fundamentally changing the direction of computing. For them, new languages are fine. It’s what they like to do for fun, and we need them to keep doing this research.

I also am less concerned with “bleeding edge” HPC programmers. They are very important since they are the pioneers in parallel computing. They have an important role to play in helping to educate the new generation of parallel programmers. But they also like to whine about parallel programming languages when MPI gives them just about everything they need. But more importantly, they usually work from source code and have a host of options real software engineers don’t have.

With this context in mind, think of my central point about choice overload. Software engineers selecting a language are faced with choices. Numerous studies have shown that choice overload is a general feature of being a human being. Hence, until proven otherwise, I assume software engineers are human … and hence are subject to choice overload.

I really appreciate the comments several people made building on the nature of choice and taking things to the next level of detail. Reading between the lines and pulling together several conversations I’ve had since my blog came out, the crucial point is that the number of options should be small when a choice is being made. If the initial set of choices is huge, but the consumer is guided through well organized sets of options so the final choice is between a few items, then choice overload can be managed. As someone pointed out, automobile manufactures have become masters at this technique. There is no reason we can’t do something similar for parallel programming languages.

I want to come back to the question of new languages and the role they can play in addressing the parallel programming problem. I have been accused of being against new languages. Just to be absolutely clear, research on new languages is fine. In fact, we need it. But when you move from research to deployment and reach out to the huddled masses of overworked software engineers, that’s when you really need to be careful. Because choice overload is real.

Language researchers need to appreciate that much of what passes itself as new is really a “been there, done that” situation. Data parallel languages, guarded horn clauses, functional languages … come on, can anyone be so ignorant to miss the fact that all of these have been tried in the past. And they largely failed (as measured by adoption by software engineers producing commercial products). I’m not saying new languages can’t work, but before wasting anyone’s time with these old solutions, you had better figure out what has changed that will make them work this time around.

One person mentioned the importance of research on runtimes. I was pleased to read that comment. This has been a constant mantra in my work at Intel. We have several parallel languages each with their own runtimes. And these runtimes, of course, are ignorant of each other. The result is that it can be difficult if not impossible to build programs from libraries written with multiple languages or even multiple instance of the same language. And if you look at how modern applications are created, they are build from many disparate modules usually not available as source code (hence recompilation is not an options).

We need a massive research effort in the parallel programming community to resolve this issue and define a common runtime infrastructure that can be used across parallel languages. This is a far more important problem that new parallel programming languages or transactional memory or most parallel programming research I see going on around me.

Much of my frustration with new languages is because the parallel programming community is largely ignoring these huge runtime issues. I understand why … runtimes aren’t “sexy.” It just sounds better when bragging about our work to friends and family to describe the new programming languages we’re creating rather than a runtime environment. But in terms of impact, the runtime is where we need to be focused right now.

But I have gone on way too long … and long blogs are boring. I want to return to my opening question:

“Is anyone dumb enough to think we can solve the parallel programming problem with a new languages”.

Well I might be that dumb. In my warning about choice overload, I was urging us to deploy fewer languages. I was asking that we spend more time figuring out how to make current languages actually work instead of creating new ones. But over time, I truly hope new languages will emerge. But history has shown us how they will emerge … and intelligent researchers working on new languages had better pay attention to these lessons.

1. Successful new languages build on existing languages and where possible, support legacy software. C++ grew our of C. java grew out of C++. To the programmer, they are all one continuous family of C languages. So I urge language designers to develop their new abstractions and then deploy them as extensions of existing languages. This is why I follow IBMs X10 project so closely. By building on Java, they’ve decreased the “new language gradient”. TBB is another example of “getting this right” by building off common practice in generic programming with C++.

2. Successful new languages emerge from a “pull model” with the pull coming from applications communities; not by a push model driven by computer scientists. Java rose to dominance because internet programmers found it solved many of their problems. “Ruby on Rails” pushed Ruby onto center stage because programmers of web applications found it so useful. Erlang might just take off since it was created by telecommunications users for telecommunications applications. Throughout history, the most successful programming technologies succeeded due to pull from an applications community. Exceptions to this rule exist, but they are rare.

3. Building on point 2, this suggests that the place to look for new languages are in specific application domains. Domain specific languages may be our best hope. And I mean “domain” in the classic sense of an “application domain”, not an algorithm domain (i.e. data parallel languages are NOT domain specific languages).

So yes … I might just be dumb enough to think a new language can help us solve the parallel programming problem. But only if we do it right … and that means we avoid scaring off the software developers.

12 Responses to Is anyone dumb enough to think yet another parallel language will solve our problems? I MIGHT be!

  1. Mark Hoemmen says:

    3 is a good point and something we’ve both seen work to great success at the national labs with parallel libraries (PETSc and the like). Coders write something that looks like a sequential program but has hidden parallelism.
    With #2: the “pull model” generally justifies spending effort to tune a runtime — which is tricky work, as I know from my colleagues who work on the GASNet backend.
    With #1 and the TBB: when are people going to realize that C++ doesn’t have closures as first-class objects? ;-P C++ syntax for making function objects and the like (which are essential for TBB, from what I understand by reading the TBB docs) is annoying and ugly. I don’t see the “huddled masses” embracing that any more than they have embraced APL.
    The “masses” might favor Cilk’s clean syntax, as it is easy to understand for typical C code monkeys, but is Cilk to parallel performance like Java or Ruby on Rails were to web pages? It doesn’t come with any “killer libraries” like Ruby on Rails; does it need to?

  2. Lisp has traditionally been a good language for experimenting with language design. Gambit Scheme is a Lisp dialect with thread performance close to Erlang’s and mailbox-per-thread message sending much like Erlang’s as well. (Although the message sending is not yet as tuned as Erlang’s.)
    I would imagine writing translators from some language like Python or Ruby to Gambit Scheme would provide those languages with (1) native code with relatively little effort; and (2) Erlang-like concurrency with relatively little effort. Either would be a plus, but in this case you’d get both.
    You could get both running in the same OS process as well, if you don’t assume there must be a single “top level” environment. So Ruby and Python could send messages to each other in a shared-nothing runtime or to other nodes, just as with Erlang processes.

  3. Trae Fuller says:

    Kind of like the mind-body disconnect, the gap between software and hardware makes exploiting parallelism a toughie. Different types of people develop hardware from those who develop software. Programming languages, operating systems, and low-level drivers bridge the gap between hardware and software, but again, different types of people are the driving force between their development as well. A new programming language can help in this effort, but I think the problem is bigger and probably as tough as ‘belling the cat’ – where parallel systems are developed out of an integrated effort of both hardware and software where parallelism is an integral part. Is this realistic tho? Since the real-world has to be considered?

  4. Cory says:

    I have only dabbled in parallel programming environments, but the most interesting system I’ve seen so far is Codeplay’s Sieve.
    http://www.codeplay.com/customcompilers/multi-core.html
    On the surface it is very similar to OpenMP with one important difference: It can prove that a program will get the same results when run multi-threaded as it would when run single-threaded and if it won’t, it can tell you why not so you can fix it.

  5. Igor says:

    How about instead of developing yet another useless programming language developers learn how to program using the existing ones correctly for a change?!?

  6. I echo Tim’s experiences with new, objectively “better” parallel languages failing, somewhat inexplicably. I think of choice overload in the context of a selectionist system; i.e., where various approaches (species) compete for resources, some succeeding and some dying out. It seems that so many options persist because a) the supporters of a fringe approach can keep it alive with little selective pressure (by working on it in their personal time) and b) the established approaches aren’t meeting all the needs of programmers wanting to program in parallel. If we accept b), then it’s a Good Thing that there are choices. As the number of cores per die continues its explosive growth, the demand for effective parallel programming tools will come from much larger groups of programmers, and (I believe) some approaches will emerge as dominant and others will effectively recede into the background. (The converse is that we don’t really figure out how to program these multicore chips and all the hardware vendors grind to a halt because more cores don’t provide added economic value – seems unlikely.)
    At Interactive Supercomputing, we’re focused on a (mostly) different group of programmers than the professional software engineers you’re targeting, namely the ones who today program in very high-level languages (think Python, MATLAB, or R) on the desktop and need to solve problems that don’t fit on the desktop or want to solve problems faster than they can on a single core. Success appears to be defined by performance, ease of use (mainly defined by degree of change from the base language), and whether the approach is viewed as long-lived enough to warrant programmers investing time into using it in their programs.
    For those users, it appears that several of the well-known parallel approaches, like data parallelism, meet the success criteria pretty well. As you say, we can’t be blind to the issues that these approaches encountered in the past, but these users also weight the success criteria differently than the bleeding-edge HPC users who have stoked much of the recent parallel language work.
    For any of these approaches, a huge effect is how well they really work in practice, and the run-time is a first-order effect on that, so I strongly agree with your focus on the run-time and interoperability of languages.

  7. What’s in parallel computing?
    Seems that my propounded law(theory of semiconductor realusion) is working quiet effecient and can bring the pentium 5 technology to a reality..

  8. Shang Lei says:

    Specific Programming Language in specific Domain, I think, may be a good direction. Since differenct apps have different characteristics, it is hard to write all apps in one parallel language as we did before in the sequetial world.
    Complexity in Parallel Computing is driving us use different languages. There can’t be a monopoly in the future.
    Not only Prgoramming Models, but also the systems include processors and networks should specify. Niagara T1 is designed for Commercial Server only, even it has only 1 FPU with 8 cores. Computer for Life-science research is being built in my institute, and relative support systems are in schedule.

  9. Louis Savain says:

    We don’t need new languages for the current thread-based model. We need a new software model. The thread-based approach to parallelism is a joke, for the simple reason that it’s not working. For one, you can forget about fine-grain parallelism when using threads. Second, threads are inherently non-deterministic.
    What amazes me is that we have had the correct model in our faces for decades but all the brilliant savants at Intel and academia are blind to it. We’ve been using it to emulate parallelism in such applications as neural networks, cellular automata, VHDL, Verilog, simulations, video games for decades. No threads are involved (or even necessary) and the temporal determinism that is inherent in this model eliminates all the problems of thread-based concurrency (e.g., deadlocks). I am inclined to ask, what is the matter with you people?
    This model of parallel computing is what I call the non-algorithmic, synchronous, reactive software model. The problem is, a program that uses this model would be too slow because current CPUs are designed for algorithmic computing. All we need to do is to design a new multicore CPU architecture for the new model and voila! You get ease of programming and fine-grained parallelism in an MIMD environment. Not to mention the rock-solid reliability that comes with temporal determinism.

  10. Sam Martin says:

    I can see where you are coming from, but I have to disagree.
    From my standpoint, I think you underestimate both the ability of developers working under pressure to make big changes, and their ability to make informed choices.
    As (games) developer where a significant part of my job is to develop more efficient code than the next guy, I’m not going to randomly sample some multicore jams and move on. I’m going to become a jam connoisseur or cook up my own if I have to.
    I’ve looked at the current jams on offer, and frankly they look pretty rough. They are all rather C-focused, often rather similar, and don’t make the prospect of multicore C++ any less terrifying.
    The best jams I can currently find are actually in Haskell – about as far from C you can really get. But Haskell is getting very little investment and its potential won’t be realised unless this turns around.

  11. Kenshin Himura says:

    To Mark Hoemmen.
    Sorry, your comment about C++ function object syntax & closures proves you still have not got it!
    1. C++0x lambdas make this creation much easier.
    2. To countless C++ programmers, the syntax is well known & established.
    Mark, the point is, if C++ does not suit you,use the tool you are most acquainted & fluent with. As a C++ programmer, I found TBB intuitive cause of my background in the use of the STL.

  12. Bill Long says:

    After reading the interview with Tim in HPCWire about parallel language overload, I found this blog on the same topic. I agree with two of the main points of the interview and the original blog here: That standardization (vendor independent) and extension of existing languages are very good qualities. It is odd, then, that the obvious candidate was never mentioned (though there was a comment that HPC was not a focus here). Fortran is an explicitly parallel language now, building on the previous versions of Fortran. This makes adding in the parallel constructs to an existing code much easier than starting from scratch in a new language. Fortran is also a full ISO standard language, like C and C++, arguably even more “standard” than things like MPI and OpenMP that are standardized by organizations outside the formal standardization process. The Fortran standard has multiple vendor input as well as input from academia and national labs, and is ultimately voted on by countries through their national body representation in ISO. Fortran seems to be an ideal example of the goals promoted here, and is already a ‘done deal’. The Intel compiler supports the new parallel features of Fortran. It’s worth taking a look if you are involved in scientific computing.