Search
Question: BiocParallel example does not work any more (sbatch produced exit code 1)
0
gravatar for mertes
21 months ago by
mertes0
Germany
mertes0 wrote:

Hey,

since we upgraded our slurm to the version 15.08.3 the Biocparallel package does not work any more.

When running the example code `xx <- bplapply(1:100, FUN)` from the vignette it fails with the following error:

Error in (function (reg, ids, resources = list(), wait, max.retries = 10L,  :
  Fatal error occured: 101. sbatch produced exit code 1; output sbatch: error: Invalid numeric value "" for number of tasks.

This is the param object

> param
class: BatchJobsParam
  bpjobname:BPJOB; bpworkers:1; bpisup:TRUE
  bpstopOnError:FALSE; bpprogressbar:TRUE
  cleanup:TRUE
> str(param)
Reference class 'BatchJobsParam' [package "BiocParallel"] with 10 fields
 $ workers      : int 1
 $ tasks        : int 0
 $ jobname      : chr "BPJOB"
 $ catch.errors : logi TRUE
 $ stop.on.error: logi FALSE
 $ progressbar  : logi TRUE
 $ reg.pars     :List of 1
  ..$ work.dir: chr "<somepath>/testslurmout"
 $ submit.pars  :List of 1
  ..$ resources:List of 1
  .. ..$ ncpus: num 1
 $ conf.pars    :List of 12
  ..$ staged.queries     : logi TRUE
  ..$ max.concurrent.jobs: num Inf
  ..$ default.resources  : list()
  ..$ cluster.functions  :List of 5
  .. ..$ name             : chr "SLURM"
  .. ..$ submitJob        :function (conf, reg, job.name, rscript, log.file, job.dir, resources, arrayjobs)  
  .. ..$ killJob          :function (conf, reg, batch.job.id)  
  .. ..$ listJobs         :function (conf, reg)  
  .. ..$ getArrayEnvirName:function ()  
  .. ..- attr(*, "class")= chr "ClusterFunctions"
  ..$ raise.warnings     : logi FALSE
  ..$ db.options         : list()
  ..$ mail.start         : chr "none"
  ..$ fs.timeout         : num NA
  ..$ db.driver          : chr "SQLite"
  ..$ mail.error         : chr "none"
  ..$ mail.done          : chr "none"
  ..$ debug              : logi FALSE
 $ cleanup      : logi TRUE
 and 18 methods, of which 4 are  possibly relevant:
   initialize, initialize#BiocParallelParam, show#BiocParallelParam,
   show#envRefClass

This is my SessionInfo output:

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Scientific Linux release 6.7 (Carbon)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] checkmate_1.6.3    BatchJobs_1.6      BBmisc_1.9         BiocParallel_1.4.0

loaded via a namespace (and not attached):
 [1] digest_0.6.8         futile.options_1.0.0 DBI_0.3.1           
 [4] magrittr_1.5         RSQLite_1.0.0        stringi_1.0-1       
 [7] futile.logger_1.4.1  brew_1.0-6           lambda.r_1.1.7      
[10] tools_3.2.2          stringr_1.0.0        parallel_3.2.2      
[13] sendmailR_1.2-1      base64enc_0.1-3      fail_1.3            

 

My slurm.tmpl looks like the example one from the vignette.

I hope, that those information can help debugging the problem. If some more details are needed please let me know. I try to provide all information.

Best

Christian Mertes

ADD COMMENTlink modified 21 months ago by Valerie Obenchain ♦♦ 6.2k • written 21 months ago by mertes0
2
gravatar for Valerie Obenchain
21 months ago by
Valerie Obenchain ♦♦ 6.2k
United States
Valerie Obenchain ♦♦ 6.2k wrote:

Hi,

The number of tasks is defined by the 'resources' passed to BatchJobsParam(). You can see how this is defined in the slurm template in the BiocParallel vignette:

#SBATCH --ntasks=<%= resources$ntasks %>

In the code above it looks like you're passing an empty 'resources' list when instead it should contain the names of machines/nodes. More details about how to specify the 'resources' list is here, specifically see the 'Resource Allocation' section.

Valerie

ADD COMMENTlink modified 21 months ago • written 21 months ago by Valerie Obenchain ♦♦ 6.2k

Hello Valerie,

thx for the reply. Since I just copied and pasted the vignette from BiocParallel I didnt thought about an error in there.

The vignette shows the ntasks in the slurm.tmpl file, but does not specify it in the R command. this is in the r-centric section of biocparallel (4.3.2, Introduction to BiocParallel Edited: May 6, 2015; Compiled: October 14, 2015).

To whom should I report this error in the Vignette? Is it here enough?

Best

Christian Mertes

 

ADD REPLYlink written 21 months ago by mertes0
0
gravatar for Valerie Obenchain
21 months ago by
Valerie Obenchain ♦♦ 6.2k
United States
Valerie Obenchain ♦♦ 6.2k wrote:

The vignette does show how to specify it. See the code just after "The R script run interactively or from the command line ..." which shows how to create a BatchJobsParam with resources:

## register SLURM cluster instructions from the template file
funs <- makeClusterFunctionsSLURM("slurm.tmpl")
param <- BatchJobsParam(4, resources=list(ncpus=1), cluster.functions=funs)

Valerie

ADD COMMENTlink modified 21 months ago • written 21 months ago by Valerie Obenchain ♦♦ 6.2k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.2.0
Traffic: 186 users visited in the last hour