BiocParallel example does not work any more (sbatch produced exit code 1)
2
0
Entering edit mode
mertes • 0
@mertes-9153
Last seen 5.7 years ago
Germany

Hey,

since we upgraded our slurm to the version 15.08.3 the Biocparallel package does not work any more.

When running the example code `xx <- bplapply(1:100, FUN)` from the vignette it fails with the following error:

Error in (function (reg, ids, resources = list(), wait, max.retries = 10L,  :
  Fatal error occured: 101. sbatch produced exit code 1; output sbatch: error: Invalid numeric value "" for number of tasks.

This is the param object

> param
class: BatchJobsParam
  bpjobname:BPJOB; bpworkers:1; bpisup:TRUE
  bpstopOnError:FALSE; bpprogressbar:TRUE
  cleanup:TRUE
> str(param)
Reference class 'BatchJobsParam' [package "BiocParallel"] with 10 fields
 $ workers      : int 1
 $ tasks        : int 0
 $ jobname      : chr "BPJOB"
 $ catch.errors : logi TRUE
 $ stop.on.error: logi FALSE
 $ progressbar  : logi TRUE
 $ reg.pars     :List of 1
  ..$ work.dir: chr "<somepath>/testslurmout"
 $ submit.pars  :List of 1
  ..$ resources:List of 1
  .. ..$ ncpus: num 1
 $ conf.pars    :List of 12
  ..$ staged.queries     : logi TRUE
  ..$ max.concurrent.jobs: num Inf
  ..$ default.resources  : list()
  ..$ cluster.functions  :List of 5
  .. ..$ name             : chr "SLURM"
  .. ..$ submitJob        :function (conf, reg, job.name, rscript, log.file, job.dir, resources, arrayjobs)  
  .. ..$ killJob          :function (conf, reg, batch.job.id)  
  .. ..$ listJobs         :function (conf, reg)  
  .. ..$ getArrayEnvirName:function ()  
  .. ..- attr(*, "class")= chr "ClusterFunctions"
  ..$ raise.warnings     : logi FALSE
  ..$ db.options         : list()
  ..$ mail.start         : chr "none"
  ..$ fs.timeout         : num NA
  ..$ db.driver          : chr "SQLite"
  ..$ mail.error         : chr "none"
  ..$ mail.done          : chr "none"
  ..$ debug              : logi FALSE
 $ cleanup      : logi TRUE
 and 18 methods, of which 4 are  possibly relevant:
   initialize, initialize#BiocParallelParam, show#BiocParallelParam,
   show#envRefClass

This is my SessionInfo output:

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Scientific Linux release 6.7 (Carbon)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] checkmate_1.6.3    BatchJobs_1.6      BBmisc_1.9         BiocParallel_1.4.0

loaded via a namespace (and not attached):
 [1] digest_0.6.8         futile.options_1.0.0 DBI_0.3.1           
 [4] magrittr_1.5         RSQLite_1.0.0        stringi_1.0-1       
 [7] futile.logger_1.4.1  brew_1.0-6           lambda.r_1.1.7      
[10] tools_3.2.2          stringr_1.0.0        parallel_3.2.2      
[13] sendmailR_1.2-1      base64enc_0.1-3      fail_1.3            

 

My slurm.tmpl looks like the example one from the vignette.

I hope, that those information can help debugging the problem. If some more details are needed please let me know. I try to provide all information.

Best

Christian Mertes

biocparallel slurm • 2.6k views
ADD COMMENT
2
Entering edit mode
@valerie-obenchain-4275
Last seen 21 months ago
United States

Hi,

The number of tasks is defined by the 'resources' passed to BatchJobsParam(). You can see how this is defined in the slurm template in the BiocParallel vignette:

#SBATCH --ntasks=<%= resources$ntasks %>

In the code above it looks like you're passing an empty 'resources' list when instead it should contain the names of machines/nodes. More details about how to specify the 'resources' list is here, specifically see the 'Resource Allocation' section.

Valerie

ADD COMMENT
0
Entering edit mode

Hello Valerie,

thx for the reply. Since I just copied and pasted the vignette from BiocParallel I didnt thought about an error in there.

The vignette shows the ntasks in the slurm.tmpl file, but does not specify it in the R command. this is in the r-centric section of biocparallel (4.3.2, Introduction to BiocParallel Edited: May 6, 2015; Compiled: October 14, 2015).

To whom should I report this error in the Vignette? Is it here enough?

Best

Christian Mertes

 

ADD REPLY
0
Entering edit mode
@valerie-obenchain-4275
Last seen 21 months ago
United States

The vignette does show how to specify it. See the code just after "The R script run interactively or from the command line ..." which shows how to create a BatchJobsParam with resources:

## register SLURM cluster instructions from the template file
funs <- makeClusterFunctionsSLURM("slurm.tmpl")
param <- BatchJobsParam(4, resources=list(ncpus=1), cluster.functions=funs)

Valerie

ADD COMMENT

Login before adding your answer.

Traffic: 471 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6