Parallel programming environments: less is more

The single most important paper for programming language designers to read came out in 2000. It wasn’t written by a computer scientist, mathematician, or physical scientist. It was written by a couple professors studying social psychology:

“When Choice is Demotivating: Can One Desire too Much of a Good Thing?” Iyengar, S. S., & Lepper, M. Journal of Personality and Social Psychology, 79, 995-1006. (2000).

This paper explored the phenomena of “choice overload.” Here is what they did.

They created two displays of gourmet jams. One display had 24 jars. The other had 6. Each display invited people to try the jams and offered them a discount coupon to buy the jam. They alternated these displays in a grocery store and tracked how many people passed the displays, how many people stopped and sampled the jams, and how many subsequently used the offered coupon to buy the jam.

The results were surprising.

  • 24 jar display: 60% of the people passing the display sampled the jam, 3% purchased jam.
  • 6 jar display: 40% of the people passing the display sampled the jam, 30% purchased jam.

The larger display was better at getting people’s attention. But the number of choices overwhelmed them and they just walked away with out deciding to purchase a jam. In other words, if the goal is to attract consumers, less is more. Too much choice is demotivating.

Admittedly, selecting a gourmet jam is insignificant. Maybe for more important issues, “choice overload” is not relevant? The authors of this paper, however, went on to consider more important choices such as 401K plans, and once again, a clear choice overload effect was found. Choice overload is real. When people are faced with too many choices, the natural tendency is to “not make a choice” and just walk away (probably in frustration).

Why is this relevant to parallel programming?

Think about it. We (that is, computer companies) want to sell hardware. To do that, we need software. We display our platforms and hope software developers will spend their valuable development dollars porting to our platform.

So what is the situation today with multi-core processors? A software vendor walks up to “our display.” We show them our nice hardware with its many cores and we tell them they will need to convert their software so that it will scale. And then we show them the parallel programming environments they can work with: MPI, OpenMP, Ct, HPF, TBB, Erlang, Shmemm, Portals, ZPL, BSP, CHARM++, Cilk, Co-array Fortran, PVM, Pthreads, windows threads, Tstreams, GA, Java, UPC, Titanium, Parlog, NESL,Split-C … and the list goes on and on. If we aren’t careful, the result could very well be a “choice overload” experience with softwre vendors running away in frustration.

Think about the impression this glut of choices creates. If we “experts” can’t agree on how to write a parallel program, what makes us believe parallel programming is ready for the masses? In our quest to find that perfect language to make parallel programming easy, we actually harm our agenda and scare away the software developers we need.

We need to spend less time creating new languages and more time making the languages we have work. This is why anytime I hear someone talk about their great new language, I pretty much ignore them. Tell me how to make OpenMP work. Tell me how to fix MPI so it runs with equal efficiency on shared memory and distributed memory systems. Help me figure out how to get pthreads and OpenMP components to work together. Help me understand solution frameworks so high level programmers can create the software they need without becoming parallel algorithm experts. But don’t waste my time with new languages. With hundreds of languages and API’s out there, is anyone really dumb enough to think “yet another one” will fix our parallel programming problems?

[Ed. Note: Tim answers this question, responds to comments below, and says that he might even be “dumb enough,” in his follow up blog]

29 Responses to Parallel programming environments: less is more

  1. Rick Hanson says:

    Software developers are not akin to the guy trying and buying jam. For instance, I don’t really know or have to care too much about how jam is made, but software developers *better* know about the layers of abstraction above and below where they work.
    Also, you were unclear if developers are in “the masses” to which you were referring. Nevertheless, I disagree with the false dichotomy you appeared to set up between refactoring old frameworks/languages for parallelism and developing new frameworks/languages for it. Should we stop language development for parallelism (or for any other good reason) because someone feels that the spectrum of diversity is too wide? I don’t think so. I think choice is just fine for software developers (maybe not for jam consumers though).

  2. B. Swaile says:

    This makes one consider drawing a parallel with the lack of choice regarding selection of M$ Windows VS the 230 Linux distributions out there.

  3. Tony says:

    IIRC there was another part of this study that involved shampoo. In response one shampoo company reduced their number of choices by around 70% and actually saw a significant increase in total sales.

  4. Andrew says:

    This is not just a problem in the parallel computing world. Just imagine that you are a website developer (say a new one) and you want to build the next greatest 2 point Oh! program, which language will you use?
    PHP, PERL, RUBY, JAVA, Javascript, AJAX, XML (with schemas or DTDs?), HTML, XMTHL…
    Okay so some of these are not *languages* per se, and there are more available than I can think of at the moment, but it is the same problem.
    When Ruby hit the big time-it was like mana from somewhere. It would solve all your web problems. It will make you work faster, build time will be cut and if you use our framework you won’t even have to think… But it didn’t do any of that.
    Languages are expressions of ideas, not ideas themselves. No language in the world will make your website/program user friendly, efficient, or better.

  5. Matt says:

    Andrew, you do make a good point about the number of programming languages. The difference between programming languages and “jam” though, is that different programming languages allow us to think about the problem space in a different light. It’s more akin to the natural languages that we read and write.
    For example, Ruby was never meant for making websites without thinking. It was meant to be a more expressive language that had many higher-order abilities that Lisp has. In this sense it has succeeded. It’s even forced Java to implement new features. There is no perfect programming language (read “No Silver Bullet” by Fred Brooks), but there are great new ideas in the different languages that need to be investigated.
    As for parallel programming frameworks, I think the reason we have so many is because parallel programming is a very very complex idea. Especially when you try to write them in languages like C and Java. Functional programming (eg Lisp, Haskell) is far better suited for parallel programming than imperative programming, but the market demands it for C and Java.
    My point is that we do have too many parallel programming, but the solution isn’t to just “get rid of some of them”. It’s to analyze “why” we have too many.

  6. Mark Hoemmen says:

    I would argue that making a good parallel runtime is a much more interesting problem than designing a new parallel language. Once you have a good parallel runtime for a target collection of architectures, then it really doesn’t matter much what language you use — if you don’t like an existing one, you just write your own and use the parallel runtime as its back end.
    If jam machines were easy to use and fresh fruit were cheap, then the problem of which jam to buy would disappear (just make your own at home). (Nobody complains that there are too many varieties of fruit in the grocery store, do they?)
    Similarly, if everybody had a good toolkit for implementing the parallel language that suits their needs, then the whole problem of “TOO MANY CHOICES!” would disappear.
    1. Some of my colleagues work on GASNet (, which is a communications runtime intended as a compilation target for shared-address-space SPMD parallel languages.
    2. I’m a Lisp hacker and so I favor users writing their own domain-specific languages (because this task is easy in Lisp) rather than vendors attempting to write general-purpose languages.
    3. Although I’m a TA this semester for a parallel programming class for undergrads, I am commenting here unofficially and as a private person rather than in my TA role.

  7. Paddy3118 says:

    In the Python world there has long been a large choice of web frameworks to choose from, and a lively debate as to whether the Python standard library should favour one by adopting it.
    For many years Guido the Python BDFL, had resisted any favouritism but he then came out with a statement in which he has endorsed one web framework.
    Maybe its natural when considering any problem such as parallelism that there emerge a wealth of solutions that then converge on a good enough solution for many tasks? Lets hope its not the least common denominator.

  8. Dr Chris Blair says:

    The Psychology of Persuasion: How to Persuade Others to Your Way of Thinking (Hardcover)
    by Kevin Hogan (Author)is helpful in understanding how people come to make decisions.And how important context is when making those decisions.
    Less can be more when making choices – the traditional view is that increasing the number to 3 options – basic, value and premium – is a good way to optimize total sales.
    Remember we are primitives hard wired to make certain decisions, regardless of logic.
    Flight or fight decision making is often involved even if we are not aware of it.

  9. J WILEY says:

    As far as programming goes, nothing new about those problems. Even with simgle processing languages, look at the “flood” of choices-
    C? alright, who’s version of C, Basic (not dead yet) which one? There are multiple versions, some radically different, Forth?-
    practically anyone using it has thier own version and the list gets longer and longer and longer. Most PC users do not care about “which language” you use. The important thing is to end up with a product that works and correctly the first time the customer tries to use it and keeps working without problems.

  10. Sam Provencher says:

    I refer to this as the “Schooling Fish” theory. Fish create schools because when a predator attacks, the vast number of target choices makes it hard to focus on any one target causing him to fail and he ends up having to try again. This demonstrates that the behavioral pattern can even work against a determined individual given certain circumstances.
    In this case, the number of choices appears to be sufficiently large to cause an elevated level of confusion in the potential customer. It would be interesting to try to determine what the factors are that affect this behavior, it might not be solely determined by the number of choices; size, color and pattern come to mind.

  11. Matt says:

    If I were a member of a large group of programmers tasked with improving the performance of an existing product and required to use some language like Java or C, I would agree with what you’re saying. From that point of view, I don’t want to know about alternative programming languages, I want to know about the approaches to concurrency within the technological bounds set for me from above. For example, I might pick up a framework like DataRush to give me a higher-level interface to concurrent programming in Java.
    However, if my goal were to create a new product that takes full advantage of multicore hardware, I would be quite interested in comparing the various concurrent programming languages. Why would I choose a primitive language like Java with low-level, error-prone constructs for parallelism when I could use something like Erlang designed from the ground up to write parallel code?
    Interestingly, Haskell was motivated by an observation quite similar to yours. At the time, the plethora of functional programming languages was somewhat overwhelming and kept academics from speaking a common language. But they reached a totally different conclusion: they looked for the best characteristics of each of these alternatives and arrived at a new synthesis in Haskell. It seems only natural that PL design go through cycles of expansion and contraction: new ideas are thrown out, rapidly increasing the number of languages, then they are consolidated into better designs. And thus languages progress.

  12. Ashwin says:

    Judgement vs Objective Elimination?
    Suppose you dont know what you are looking for in *anything* (be it Programming language or Jam), you have to make a judgement call based on your experiences in life. Its also hard to explain why you decided to commit to one of them over the other. Bascially, You are not sure what you want! That is why in most places Judgement calls are made by some other people and people who implement somethings are another group.
    I can try to explain this better. Suppose you go to decide to buy a car. If you are the only person driving it and if you have a lot of money, the number of choices that are available to you are plenty. So making a choice trying to get optimal result out of many parameters that you have is very hard (Good looking, nice performing etc). So you walk away!
    But suppose you have a family of 4 with a wife who doesnt know how to drive a stick, you automatically rule out (all 2 seaters, all stick cars) etc. So the more parameters you have, u are happy to eliminate. But if there are many cars that satisfy your requirements it becomes hard to make a judgement call!

  13. Louis Savain says:

    Algorithmic Cocaine
    It seems that all Intel does lately is bitch about how hard parallel programming is and how programmers are not using enough threads. The latest tantrum is about how there are too many parallel languages to choose from. Does anybody else sense a wee bit of panic in Intel’s camp?
    I got news for you, Intel. Stop blaming others for your own mistakes. You are the primary cause of the problem. You, more than any other company in this industry, got us into this sorry mess. You made so much money, over the years, milking algorithmic cocaine from that fat cow of yours that it never occurred to you that the cow might run dry some day. Now that you’ve got everybody stoned and addicted, they keep coming back for more. But there is no more. Moore’s law is no longer the undisputed law of the land. “Mix threads with your dope!”, you scream at them with despair in your voice, but they’re not listening. And they keep coming. Worse, you got so stoned consuming your own dope, you cannot see a way out of your self-made predicament.
    Read the rest of my rant if you’re interested.

  14. “Give me land, lots of land, under starry skies above, don’t fence me in…”
    The more the merrier. At the same time, there is great work going on in some standards bodies, like the ISO C++ 0x draft that is focusing some of its work on parallel programming. Parallel extensions to Fortran have been around for years.
    New languages like Erlang and functional languages like Haskell are showing the way to the future.
    As more parallel / multi-core machines get into every developer’s hands and run on all servers, desktops, and even in our hand held devices, we will need more solutions for each one.

  15. Tom says:

    I have to agree with him. I dont think we should be compared to people testing and buying Jam. On the other hand 1 more language is not going to help the situation.

  16. sharlee says:

    I agree… when i feel like there are to many decisions, i get frustrated and decide that the choices are too difficult… we just need to keep things simple.

  17. Baba says:

    I do agree… for me, too many choices means most of the thing in question is not so good and you should spend your time and money to identify the good ones and then find out the right choice from those good items… thatz stupid…

  18. Igor says:

    “different programming languages allow us to think about the problem space in a different light.”
    No they don’t. Problem first gets translated into an algorithm, not into the specific language.
    For example, you can implement radix sort in BASIC and in C. Granted, their efficiency will differ but the algorithm will still be the same.
    In other words, nothing prevents you from using radix sort in BASIC just like nothing enables you to automatically use it in C.
    It is only your level of knowledge which is limiting. Since less knowledge is required to use BASIC you are less likely to be familiar with radix sort, thus you are incorrectly deducing that C is creating new possibilities.

  19. Darrell Reich says:

    I agree. While competition is good. The overload of projects with cryptic names isn’t helping. I remember Dr Don Norman saying 7 +/- 2 in my cognitive science class. He was referring to short term memory at the time but it seems to apply to jams too. We need something inbetween a standard like everyone must program in Fortran and open competition where everyone goes off and recreates their own saying it’s better. The code has to keep up with the technology but we can’t keep rewriting things every year with the latest “sliver bullet”. Maybe someday the compiler will be smarter than us and just parallelize it!

  20. barney says:

    “different programming languages allow us to think about the problem space in a different light.”
    Igor: “No they don’t”
    Of course they do! They *allow* us to. You don’t have to if you don’t want to. 🙂

  21. James says:

    One size won’t fit all, especially when you need performance.
    People have been talking about MPI and OpenMP for ages, but emerging HPC apps, such as high performance analytics applications require different approaches than MPI or OpenMP.
    Look at Symphony from Platform (now avaialbe for free), Didiede Network, as well as Microsoft HPC Server 2008, a new paradigm called HPC SOA that demonstrate many advantages over MPI/OpenMP in a scale-out, dynamic cluster.

  22. Reza Bakhshi says:

    Two points with different tastes!
    first, each programming language as some of other commenters have pointed out is developed with a concept in its developers’ minds. They help thinking and considering the problem from different views which helps finding better solutions.
    Nevertheless, second, having a wide range of selections between languages to solve a problem is just another hell and problem to be solved.
    Considering these two, I personally believe that its better to after testing new features for a language, add them eventually to existing ones.

  23. george naing says:

    If distributed computing is a subset of paralell computing, can’t we exploit concurrency inherent in our problem just by using process-based parallelsim?
    Then, we may even use a hetrogeneous network?

  24. Thomas says:

    I concur with Mark Hoemmen regarding a runtime system approach. I designed/implemented an Ada Parallel runtime system from the sequential runtime code with enhancements 20 years ago in a large shared memory multiprocessor w/20 processors (2-system, 18-user). It tested at near linear speedup as processors were added to the execution mix with less than a 1% collision locking rate on real Military donated code. This is an example of a medium grain approach, as opposed to a fine-grain approach with optimizing compilers and a course-grain approach at the process level with OS multiprogramming to achieve parallel speedup (with load balancing, etc.).
    In my view, the runtime represents the best trade-off for existing parallel languages that run in a uniprocessor environment. The programmers do not need to be retrained in some new language, and you let the system finesse the multi-core environment of the hardware below.

  25. skyhover says:

    I disagree the conclusion at the end of the essay, although I agree the “choice overload” phenomenon. IMO, so many languages/frameworks are due to the fact that we don’t know what is the best (for different scenarios) yet; researching new languages/frameworks should be viewed as an effort for replacing old ones, reducing the total number of choices, and thus making vendors’ and developers’ life easier and increasing your revenue. Reducing the effort for research on better languages/frameworks is just as “dumb” as throwing the future away, unless that goal has been shown impossible.