In Julia, where the paralleization options are explicit (SIMD, AVX, threads or multiprocessing), it always depends on the load, for small operation (around 10000 elements) a single thread is faster only for the thread spawning time (around 1 microsecond). And there is the issue of the independent Blas threaded model, where the Blas threads sometimes interfere with Julia threads...
In a nutshell, parallelization is not a magical bullet, but is a good bullet to have at your disposal anyway
> And there is the issue of the independent Blas threaded model, where the Blas threads sometimes interfere with Julia threads
Julia has composible multithreading, and using that model fixed composing FFTW threads with Julia's. This can be done to OpenBLAS as well, and IIRC there is a PR open for it.
My laptop's 9980HK will boost to ~4.5 GHz when only loaded to a single core.
However, when I load up all 8 cores, it might only sustain ~3.5 GHz.
Therefore the 8 cores might not actually result in the work being completed 8 times as fast, only 6.2x (8*[3.5/4.5]) real-time due to the lowered clock rate of each individual core.
This will show up as additional user time, since each individual core is able to do less work for each unit of time (seconds) compared to the single-core case.
Of course in the general case Amdahl's law is inescapable, but some tasks on modern systems can show > ×N speedup over single-threaded performance if, for example, a single thread can only exploit at maximum some fraction of the total memory bandwidth or some level of the cache hierarchy.
well before that kicks in, if your code requires any coordination at all (not Monte Carlo) then those overheads can scale with the number of processes. so rather than hitting an asymptote as you'd get with just Ahmdal, you actually start to go down in absolute terms as the number of processes increases.
None of this is surprising, right? Unless your system has fewer threads than cores (which it probably doesn't even without your program) there will always be some context-switching overhead. It's worth keeping in mind I guess - especially the fact that numpy parallelizes transparently - but generally these results are to be expected.
The title is also misleading; it suggests that the wall clock time might be longer for parallel code in certain cases. While not impossible, that isn't what the article covers.
> the wall clock time might be longer for parallel code
That is exactly the case, if CPU is the bottleneck in your already-parallel application. It's a case where we really shouldn't be layering different parallel bits together in one codebase, but might be doing it naively.
The article uses the term "parallelism" when it is talking, instead, about concurrency.
Parallelism is specifically the stuff that actually does happen completely independently on all processing units, that actually goes Nx as fast on N units (clock depression aside). Concurrency refers to the overhead of coordinating activity of those units, that keeps you from getting your Nx. It is overhead on top of any actually serial parts of the computation, which Amdahl's law addresses.
In other words: Parallelism giveth, and concurrency taketh away.
The distinction gets more useful the more you think about the subject.
longemen3000|6 years ago
ChrisRackauckas|6 years ago
Julia has composible multithreading, and using that model fixed composing FFTW threads with Julia's. This can be done to OpenBLAS as well, and IIRC there is a PR open for it.
The_rationalist|6 years ago
AzN1337c0d3r|6 years ago
My laptop's 9980HK will boost to ~4.5 GHz when only loaded to a single core.
However, when I load up all 8 cores, it might only sustain ~3.5 GHz.
Therefore the 8 cores might not actually result in the work being completed 8 times as fast, only 6.2x (8*[3.5/4.5]) real-time due to the lowered clock rate of each individual core.
This will show up as additional user time, since each individual core is able to do less work for each unit of time (seconds) compared to the single-core case.
maweki|6 years ago
Basically impossible by Ahmdal's law.
twoodfin|6 years ago
convolvatron|6 years ago
_bxg1|6 years ago
The title is also misleading; it suggests that the wall clock time might be longer for parallel code in certain cases. While not impossible, that isn't what the article covers.
winkeltripel|6 years ago
That is exactly the case, if CPU is the bottleneck in your already-parallel application. It's a case where we really shouldn't be layering different parallel bits together in one codebase, but might be doing it naively.
ncmncm|6 years ago
Parallelism is specifically the stuff that actually does happen completely independently on all processing units, that actually goes Nx as fast on N units (clock depression aside). Concurrency refers to the overhead of coordinating activity of those units, that keeps you from getting your Nx. It is overhead on top of any actually serial parts of the computation, which Amdahl's law addresses.
In other words: Parallelism giveth, and concurrency taketh away.
The distinction gets more useful the more you think about the subject.
yaroslavvb|6 years ago
wtracy|6 years ago
I'm on my phone, so rather than trying to type out an explanation, I'm going to link to Wikipedia: https://en.wikipedia.org/wiki/Hyper-threading
itamarst|6 years ago