Parallel Computing: making sequential software rare

In my job at Intel, I get to travel far and wide to meet with research groups working on parallel computing. And as travel, I am constantly struck by the differences between the state of HW and SW in parallel computing.

On the HW front, we are sweating the details. We know there will be many cores. We know the cores in a single processor may be different. We know there will be a scalable on-die interconnect and that the caches will need to adapt to the workloads to maximize locality. I’ve spoken to countless HW architects working on many core systems and frankly, the “big picture” view seems pretty well locked up.

But on the software front, it’s chaos.

• Will we need new languages? Or will it be enough to extend existing languages?

• Will data parallel programming languages allow general purpose programmers to write parallel code? Or are they the nightmare of HPF (High Performance Fortran) all over again and hence an utter waste of time?

• All highly scalable codes use message passing … or they use a shared memory system with a message-passing-like approach. Can we make message passing easy enough for the general programmer to use? Or is it too hard for programmers and will never be adopted outside of HPC?

• Frameworks such as Visual Basic revolutionized PC-applications. Can we design a framework-based approach for parallel programming?

• How can we teach programmers to “think parallel”?

I’m told by people I greatly respect that I shouldn’t worry. With many core chips going mainstream, there will be countless software engineers motivated to work on parallel programming. Someone will solve the problems for us.

But I do worry. The high performance computing community has been working on the parallel programming problem for over 25 years. And frankly, we don’t have much to show for it. I hope the optimists are right and this problem will take care of itself, but this seems like faith-based reasoning to me. This problem is too big and the consequences of failure too dire to leave this to faith. We need to ask the hard questions and find a deliberate and systematic solution.

And laying out a path to that solution will be the goal of my blog. I’ve spend over 20 years working in this field. I’ve been beaten down and battered in the battles over parallel-programming languages. As an application programmer, I’ve used all the major classes of parallel programming notations “out there”. I have a unique perspective on this problem and I am just arrogant enough to think I know how to “get it right” this time around.

And I hope you, the readers of this blog, will join me in this journey. With your patience and your comments, I’m hoping we can hold a productive dialog and working together, blaze a trail to a future where “sequential programs are rare”.

[Editors note: please see Tim's next blog for responses to comments below]

8 Responses to Parallel Computing: making sequential software rare

  1. Jan Liband says:

    You get right to the heart of the software issue in noting that 25 years of parallel programming research hasn’t resulted in any “holy grail” solutions to the concurrent programming issue. It’s a tough challenge.
    Many of the experts are pointing out the inherent advantages of higher-level, declarative languages for multi-core programming. Languages like SQL let developers declare what they want and put the implementation (i.e. parallelization) burden on some of the underlying tools. At Encirq, our DeviceSQL technology automatically decomposes functions like complex queries into atomic elements that can be processed concurrently for optimum performance on multi-core architectures. There’s more work to do, but these kinds of approaches hold out promise of simplifying parallel programming without having to rely on completely new and esoteric languages.
    Also worth noting are the efforts of the Multicore Association (of which Intel is a member). The Multicore Association is helping to resolve multicore programming issues and drive standards that will benefit all.

  2. Louis Savain says:

    Nice article. Huge problem. You ask,” Will we need new languages? Or will it be enough to extend existing languages?”
    It seems to me that languages are the worst way to program parallel systems given that they are inherently sequential. Languages force us to think algorithmically and therein lies a major part of the problem. What is needed is a non-algorithmic, synchronous software model and a graphical environment for development. Once the software model is in place and all the kinks are ironed out, then you can go ahead and base your hardware architecture on the new model. Not the other way around, as it is now being done (the Cell is a good example of how not to do it).
    It is critical that the software model uses synchronously operating parallel processes because that will solve another equally vexing problem in computing, unreliability. This will sound arrogant but asynchronous threads are the work of the devil. Deterministic timing is crucial to reliability. Check out Project COSA when you get a chance.

  3. John Dallman says:

    Something to remember, which is easy to forget. Serial code is fast enough for many things. There’s no point, for example, in trying to use multiple threads in the editing loop of a text editor, because that’s already fast enough. Another thread for searching help, sure. Another thread for background pagination, or printing, fine. But there is no need to have one thread for inserting/deleting text in file, and another thread for updating the display. They’ll need to lock against each other just too frequently. There is a point at which the complexity of threads is not worth the effort.
    Secondly, deterministic execution may sound very appealing, but I’m dubious as to its practicality. It means you need separate processing power for all input and output, because network packets arrive at unpredictible times, as do keystrokes, mouse input and everything else. If you take any interrupts on your deterministic processor to handle i/o, it isn’t deterministic any more. And even if you have separate deterministic and non-deterministic address spaces and processor allocation, the presence, or otherwise, of input makes a difference.

  4. Lord Volton says:

    It’s time like these I wish I were a programmer.
    Why not simply have a few cores dedicated to sequential operations that don’t benefit from parallel programming (word processor, spreadsheet, etc.)
    And then a few that tackle programs that are written with parallel processing in mind (video games, Maya, Final Cut, etc.)
    That way I can be running several programs at once without having to close down windows and each program will run according to its intended purpose (single or multiple core).
    This is probably so obvious that you’re already well down this path.
    P.S. I really am looking forward to the day when I have can 10 programs running simultaneously without any noticeable lag. It’s like taking all of your toys out of the toy box and never having to put them away. Which reminds me, Transformers just opened.

  5. gw says:

    IMHO we need a new parallel programming language with an efficient and easy to use IDE. Programming in it should be realized from ground up graphically (2D or maybe 3D), because writing and reading text/code forces us to think sequentially. In theory the language should be capable of representing an acyclic directed dependency graph of atomical operations, but unfortunately no one yet managed to do this efficiently. Maybe the problem lies in user input devices that are clumsy in a way (keyboard, mouse)…
    Who knows, but I truly hope that someone will soon create a good language with an efficient IDE (that will run everywhere) and make it popular.

  6. stl-fan says:

    “Will we need new languages? Or will it be enough to extend existing languages?”
    Most C++ programmers just want extensions – libraries or languages : TBB, STAPL, rpa (Range Partition Adaptors), OpenMP etc… it allows them to experiment multi-threading, in real-life applications – without taking too much risks, and retaining ‘good old’ sequential high-level performance techniques. There is no one-fit-for-all solution in this field, developpers have to test/try more, and need simple and flexible tools for that.

  7. Leslie Smith says:

    I finished a PhD on this area in 1981 (!). Then moved off into Neural Networks & Bio-inspired computing. My view is that parallel programming based on C(++)(#) like languages is as OK as assembler is for current PCs. But the important thing is the new capabilities that we can use. Forget trying to be economic with CPU time. For signal processing (and that goes all the way from image anaysis/processing to audio input/output) forget sequential operations (unless they’re totally pipelined like they would be on a DSP). Instead compute lots of things that you might or might not need (in a greedy way), and select what’s required for the current task. Of course, this only applies to real-time work (like processors used where thay have to be continuously reacting to their environment) – but I think that’s where we’re going: Move forward from the “desktop” metaphor to the embedded system/helpful colleague!