Question: BiocParallel example does not work any more (sbatch produced exit code 1)
0
gravatar for mertes
3.8 years ago by
mertes0
Germany
mertes0 wrote:

Hey,

since we upgraded our slurm to the version 15.08.3 the Biocparallel package does not work any more.

When running the example code `xx <- bplapply(1:100, FUN)` from the vignette it fails with the following error:

Error in (function (reg, ids, resources = list(), wait, max.retries = 10L,  :
  Fatal error occured: 101. sbatch produced exit code 1; output sbatch: error: Invalid numeric value "" for number of tasks.

This is the param object

> param
class: BatchJobsParam
  bpjobname:BPJOB; bpworkers:1; bpisup:TRUE
  bpstopOnError:FALSE; bpprogressbar:TRUE
  cleanup:TRUE
> str(param)
Reference class 'BatchJobsParam' [package "BiocParallel"] with 10 fields
 $ workers      : int 1
 $ tasks        : int 0
 $ jobname      : chr "BPJOB"
 $ catch.errors : logi TRUE
 $ stop.on.error: logi FALSE
 $ progressbar  : logi TRUE
 $ reg.pars     :List of 1
  ..$ work.dir: chr "<somepath>/testslurmout"
 $ submit.pars  :List of 1
  ..$ resources:List of 1
  .. ..$ ncpus: num 1
 $ conf.pars    :List of 12
  ..$ staged.queries     : logi TRUE
  ..$ max.concurrent.jobs: num Inf
  ..$ default.resources  : list()
  ..$ cluster.functions  :List of 5
  .. ..$ name             : chr "SLURM"
  .. ..$ submitJob        :function (conf, reg, job.name, rscript, log.file, job.dir, resources, arrayjobs)  
  .. ..$ killJob          :function (conf, reg, batch.job.id)  
  .. ..$ listJobs         :function (conf, reg)  
  .. ..$ getArrayEnvirName:function ()  
  .. ..- attr(*, "class")= chr "ClusterFunctions"
  ..$ raise.warnings     : logi FALSE
  ..$ db.options         : list()
  ..$ mail.start         : chr "none"
  ..$ fs.timeout         : num NA
  ..$ db.driver          : chr "SQLite"
  ..$ mail.error         : chr "none"
  ..$ mail.done          : chr "none"
  ..$ debug              : logi FALSE
 $ cleanup      : logi TRUE
 and 18 methods, of which 4 are  possibly relevant:
   initialize, initialize#BiocParallelParam, show#BiocParallelParam,
   show#envRefClass

This is my SessionInfo output:

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Scientific Linux release 6.7 (Carbon)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] checkmate_1.6.3    BatchJobs_1.6      BBmisc_1.9         BiocParallel_1.4.0

loaded via a namespace (and not attached):
 [1] digest_0.6.8         futile.options_1.0.0 DBI_0.3.1           
 [4] magrittr_1.5         RSQLite_1.0.0        stringi_1.0-1       
 [7] futile.logger_1.4.1  brew_1.0-6           lambda.r_1.1.7      
[10] tools_3.2.2          stringr_1.0.0        parallel_3.2.2      
[13] sendmailR_1.2-1      base64enc_0.1-3      fail_1.3            

 

My slurm.tmpl looks like the example one from the vignette.

I hope, that those information can help debugging the problem. If some more details are needed please let me know. I try to provide all information.

Best

Christian Mertes

biocparallel slurm • 1.2k views
ADD COMMENTlink modified 3.8 years ago by Valerie Obenchain6.7k • written 3.8 years ago by mertes0
Answer: BiocParallel example does not work any more (sbatch produced exit code 1)
2
gravatar for Valerie Obenchain
3.8 years ago by
United States
Valerie Obenchain6.7k wrote:

Hi,

The number of tasks is defined by the 'resources' passed to BatchJobsParam(). You can see how this is defined in the slurm template in the BiocParallel vignette:

#SBATCH --ntasks=<%= resources$ntasks %>

In the code above it looks like you're passing an empty 'resources' list when instead it should contain the names of machines/nodes. More details about how to specify the 'resources' list is here, specifically see the 'Resource Allocation' section.

Valerie

ADD COMMENTlink modified 3.8 years ago • written 3.8 years ago by Valerie Obenchain6.7k

Hello Valerie,

thx for the reply. Since I just copied and pasted the vignette from BiocParallel I didnt thought about an error in there.

The vignette shows the ntasks in the slurm.tmpl file, but does not specify it in the R command. this is in the r-centric section of biocparallel (4.3.2, Introduction to BiocParallel Edited: May 6, 2015; Compiled: October 14, 2015).

To whom should I report this error in the Vignette? Is it here enough?

Best

Christian Mertes

 

ADD REPLYlink written 3.8 years ago by mertes0
Answer: BiocParallel example does not work any more (sbatch produced exit code 1)
0
gravatar for Valerie Obenchain
3.8 years ago by
United States
Valerie Obenchain6.7k wrote:

The vignette does show how to specify it. See the code just after "The R script run interactively or from the command line ..." which shows how to create a BatchJobsParam with resources:

## register SLURM cluster instructions from the template file
funs <- makeClusterFunctionsSLURM("slurm.tmpl")
param <- BatchJobsParam(4, resources=list(ncpus=1), cluster.functions=funs)

Valerie

ADD COMMENTlink modified 3.8 years ago • written 3.8 years ago by Valerie Obenchain6.7k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 89 users visited in the last hour