We have a workstation with 16 physical cores (32 threads with hyper threading) and we want to run DESeq2 faster using parallelisation. However, setting parallel=TRUE often runs slower than without parallelisation, if it doesn't fail at some stage, such as fitting models and testing. Failing does not throw an error, but the process is stuck with no apparent CPU utilisation. Moreover, we can't control the number of cores used in parallelisation (even with BPPARAMS=MulticoreParam(4)), it always uses all 32 cores. We tried it on RStudio and directly on R interpreter, still the same.

Is it a known issue with DESeq2, or do we have to fix something on our system?

The machine is running CentOS 7. R 3.6.3 was compiled from source with GCC 8.2.0 and we use OpenBLAS 0.3.5 for BLAS/LAPACK.

Any advice will be appreciated. Thanks a lot.

My guess is that this is not a DESeq2 issue. We observed similar problems on our system (Also CentOS 7) when not using the default R BLAS (we were exploring OpenBLAS 0.3.5 built with the SkylakeX architecture). Notably, we also got wildly wrong answers sometimes - for example, doing an SVD with the irlba package (e.g. using the small example here: https://github.com/satijalab/seurat/issues/1188#issuecomment-469390036). There are some issues in the OpenBLAS repository that might be helpful:

• https://github.com/xianyi/OpenBLAS/issues/2029
• https://github.com/xianyi/OpenBLAS/issues/1955

We have also observed the issue of not being able to control the number of cores used - adding

export OPENBLAS_NUM_THREADS=1

in the .bashrc might help (https://github.com/xianyi/OpenBLAS/wiki/Faq#multi-threaded).

Two notes:

1) Parallelization with BiocParallel works well when it's correctly setup. I have some posts here which demonstrate effective speedup with 4 or 8 workers for example.

2) I’d recommend debugging your setup with bplapply() which is used within DESeq2.