Category Archives: Distributed Computing

Parallel and distributed computing

Review of Intel Parallel Studio 2017: Advisor

Recently, I got access to the latest release of Parallel Studio with an update version of Advisor. 6 years after my last review, let’s dive into it again!
Continue reading Review of Intel Parallel Studio 2017: Advisor

Book review: Weapons Of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

Big data is the current hype, the thing you need to do to find the best job in the world. I’ve started using machine learning tools a decade ago, and when I saw this book, it felt like it was answering some concerns I had. Let’s see what’s inside.

Continue reading Book review: Weapons Of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

On efficient reading and writing for large scale simulations

Last year, my colleagues and I presented a paper on giga model simulations in an SPE conference: Giga-Model Simulations In A Commercial Simulator – Challenges & Solutions. During this talk, we talked about the complexity of I/O for such simulations. We had ordered data as input that we needed to split in chunks to send them on the relevant MPI ranks, and then the same process was required for writing the results, gathering the chunks and then writing them down to the disk.

The central point is that some clusters have parallel file systems, and these works well when you try to access big blobs of aligned data. In fact, as they are the bottleneck of the whole system, you need to limit the number of accesses to what you actually require. For instance in HDF5, you can specify the alignment of datasets, so you can say that all HDF5 datasets will be aligned on the filesystem specifications (so for instance 1MB if your Lustre/GPFS has a chunk size of 1MB) and read or write chunks that are multiple of these values.

Continue reading On efficient reading and writing for large scale simulations

Convert HPCToolkit files to callgrind format

After my post on HPCToolkit, I felt that I prefered QCacheGrind as a GUI to explore profiling results. So here is a gist with a Python script to convert XML HPCToolkit experiments to callgrind format:

For instance, this is a display of an Audio Toolkit test of L2 cache misses:

ATK L2 cache misses profile
ATK L2 cache misses profile


Profiling with HPC Toolkit

I’ve started working with the HPC Toolkit, especially the part where it can use PAPI to read hardware counters and display the result in a nice GUI.

Continue reading Profiling with HPC Toolkit

On the importance of not reinventing the wheel in distributed applications

Sometimes, it’s so easy to rewrite some existing code because it doesn’t fit exactly your bill.

I just so the example with an All To All communication that was written by hand. The goal was to share how many elements would be sent from one MPI process to another, and these elements were stored on one process in different structure instances, one for each MPI process. So in the end, you had n structures on each of the n MPI processes.

The MPI_Alltoall cannot map directly to this scattered structure, so it sounds fair to assume that using MPI_Isend and MPI_Irecv would be simpler to implement. The issue is that this pattern uses buffers on each process for each other process it will send values to or receive values from. A lot of MPI library allocate their buffer when needed, but will never let go of the memory until the end. So you end up with a memory consumption that doesn’t scale. In my case, when using more than 1000 cores, the MPI library uses more than 1GB per MPI process when it hits these calls, just for these additional hidden buffers. This is just no manageable.

Now, if you use MPI_Alltoall, two things happen:

  • there are no additional buffer allocated, so this scales nicely when you increase the number of cores
  • it is actually faster than your custom implementation

Now with MPI 3 standard having non-blocking collective operations, there is absolutely no reason to try to outsmart the library when you need a collective operation. It has heuristics when it knows that it is doing a collective call, so let them work. You won’t be smarter if you try, but you will if you use them.

In my case, the code to retrieve all values and store them in an intermediate buffer was smaller that the one with the Isend/Irecv.

Book review: CUDA By Example

It has been a while since my last post here, but I’m back! I had access to the French version of this book, thanks to the publisher.

CUDA is now in the trend, and there are several books, one of them I’ve also reviewed.
Continue reading Book review: CUDA By Example

Book review: C++ Concurrency in Action: Practical Multithreading

In the new C++ standard, multithread finally appears, with the old standard supported with TR2. This new addition has numerous implications on how programs are coded, and there are of course almost no book on this matter. This one is an exception.

Note: this review is not based on the final version that is now available (June the 28th), but on the MEAP one. There may be some differences between the final draft and the one I based my review on, although I don’t expect many, and certainly not any huge change.
Continue reading Book review: C++ Concurrency in Action: Practical Multithreading

Book review: Introduction to High Performance Computing for Scientists and Engineers

We know now that we won’t have the same serial computing increase we had in the last decades. We have to cope with optimizing serial codes, and programming parallel and concurrent ones, and this means that all coders have to cope with this paradigm shift. If computer scientists are aware of the tools to use, it is not the same for the “average” scientist or engineer. And this is the purpose of this book: educate the average coder.
Continue reading Book review: Introduction to High Performance Computing for Scientists and Engineers