Entering edit mode
Hi,
I'm trying to tally variants using VariantTools using the code below, but I keep getting varying warnings of jobs not delivering and the same error: "Error in env[[as.character(i)]] <- value". I've tried looking around, but I couldn't figure out why it's not working.
`bpp <- BiocParallel::MulticoreParam(16L)
tally.Param <- TallyVariantsParam(refgen, high_base_quality = 23L,
indels = TRUE,
exon_iit = TxDb.Hsapiens.UCSC.hg19.knownGene,
IIT_BPPARAM = bpp)`
Finished reading FASTA file -- total entries: 23459
Maximum coordinate: 249213345
Total label length: 124996 + 23459 separators
Total annotation length: 5380546 + 23459 separators
Saw 40 distinct divisions/chromosomes
Saw 1 distinct tags/types
Writing IIT file header information...done
Processing null division/chromosome...sorting...writing...done (0 intervals)
Processing division/chromosome chr19...sorting...writing...done (1610 intervals)
Processing division/chromosome chr8...sorting...writing...done (824 intervals)
Processing division/chromosome chr20...sorting...writing...done (647 intervals)
Processing division/chromosome chr18...sorting...writing...done (341 intervals)
Processing division/chromosome chr1...sorting...writing...done (2344 intervals)
Processing division/chromosome chrX...sorting...writing...done (971 intervals)
Processing division/chromosome chr3...sorting...writing...done (1274 intervals)
Processing division/chromosome chr14...sorting...writing...done (780 intervals)
Processing division/chromosome chr15...sorting...writing...done (799 intervals)
Processing division/chromosome chr11...sorting...writing...done (1441 intervals)
Processing division/chromosome chr9...sorting...writing...done (950 intervals)
Processing division/chromosome chr22...sorting...writing...done (537 intervals)
Processing division/chromosome chr10...sorting...writing...done (908 intervals)
Processing division/chromosome chr5...sorting...writing...done (1022 intervals)
Processing division/chromosome chr4...sorting...writing...done (881 intervals)
Processing division/chromosome chr7...sorting...writing...done (1117 intervals)
Processing division/chromosome chr16...sorting...writing...done (944 intervals)
Processing division/chromosome chr2...sorting...writing...done (1479 intervals)
Processing division/chromosome chrY...sorting...writing...done (79 intervals)
Processing division/chromosome chr13...sorting...writing...done (449 intervals)
Processing division/chromosome chr6...sorting...writing...done (1207 intervals)
Processing division/chromosome chr17...sorting...writing...done (1358 intervals)
Processing division/chromosome chr12...sorting...writing...done (1176 intervals)
Processing division/chromosome chr21...sorting...writing...done (296 intervals)
Processing division/chromosome chr19_gl000209_random...sorting...writing...done (6 intervals)
Processing division/chromosome chrUn_gl000218...sorting...writing...done (2 intervals)
Processing division/chromosome chrUn_gl000228...sorting...writing...done (3 intervals)
Processing division/chromosome chrUn_gl000213...sorting...writing...done (1 intervals)
Processing division/chromosome chrUn_gl000220...sorting...writing...done (1 intervals)
Processing division/chromosome chrUn_gl000212...sorting...writing...done (1 intervals)
Processing division/chromosome chrUn_gl000219...sorting...writing...done (1 intervals)
Processing division/chromosome chr6_cox_hap2...sorting...writing...done (1 intervals)
Processing division/chromosome chr6_mcf_hap5...sorting...writing...done (2 intervals)
Processing division/chromosome chr7_gl000195_random...sorting...writing...done (1 intervals)
Processing division/chromosome chr17_gl000205_random...sorting...writing...done (1 intervals)
Processing division/chromosome chr4_gl000193_random...sorting...writing...done (1 intervals)
Processing division/chromosome chr6_mann_hap4...sorting...writing...done (1 intervals)
Processing division/chromosome chrUn_gl000211...sorting...writing...done (1 intervals)
Processing division/chromosome chr6_ssto_hap7...sorting...writing...done (1 intervals)
Processing division/chromosome chr4_gl000194_random...sorting...writing...done (1 intervals)
Writing IIT file footer information...done
`tallies <- tallyVariants(bamfl, tally.Param, BPPARAM = bpp)`
Error in env[[as.character(i)]] <- value :
wrong args for environment subassignment
In addition: Warning message:
In parallel::mccollect(wait = FALSE, timeout = 1) :
1 parallel job did not deliver a result
`traceback( )`
12: add_inorder(i, value)
11: reducer$add(njob, value)
10: bploop.iterate(manager = manager, ITER = ITER, FUN = FUN, ARGS = ARGS,
BPPARAM = BPPARAM, reduce.in.order = TRUE, progress.length = length(X))
9: bploop(manager = manager, ITER = ITER, FUN = FUN, ARGS = ARGS,
BPPARAM = BPPARAM, reduce.in.order = TRUE, progress.length = length(X))
8: bploop.lapply(manager, FUN = FUN, BPPARAM = BPPARAM, ...)
7: bploop(manager, FUN = FUN, BPPARAM = BPPARAM, ...)
6: .bpinit(manager = manager, X = X, FUN = FUN, ARGS = ARGS, BPPARAM = BPPARAM,
BPREDO = BPREDO)
5: bplapply(which, tally_region_job, x = x, param = param, BPPARAM = BPPARAM)
4: bplapply(which, tally_region_job, x = x, param = param, BPPARAM = BPPARAM)
3: .local(x, ...)
2: tallyVariants(bamfl, tally.Param, BPPARAM = bpp)
1: tallyVariants(bamfl, tally.Param, BPPARAM = bpp)
sessionInfo( )
R version 4.1.3 (2022-03-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.6 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] tools stats4 tcltk stats graphics grDevices utils datasets
[9] methods base
other attached packages:
[1] TxDb.Hsapiens.UCSC.hg19.knownGene_3.2.2 GenomicFeatures_1.46.5
[3] AnnotationDbi_1.56.2 vcfR_1.13.0
[5] saasCNV_0.3.4 DNAcopy_1.68.0
[7] RANN_2.6.1 BiocParallel_1.28.3
[9] VariantTools_1.36.0 VariantAnnotation_1.40.0
[11] R.utils_2.11.0 R.oo_1.24.0
[13] R.methodsS3_1.8.1 gmapR_1.36.0
[15] GenomicAlignments_1.30.0 Rsamtools_2.10.0
[17] SummarizedExperiment_1.24.0 Biobase_2.54.0
[19] MatrixGenerics_1.6.0 matrixStats_0.61.0
[21] BSgenome.Hsapiens.UCSC.hg19_1.4.3 BSgenome_1.62.0
[23] rtracklayer_1.54.0 Biostrings_2.62.0
[25] XVector_0.34.0 GenomicRanges_1.46.1
[27] GenomeInfoDb_1.30.1 IRanges_2.28.0
[29] S4Vectors_0.32.4 BiocGenerics_0.40.0
[31] Rfastp_1.4.0
loaded via a namespace (and not attached):
[1] nlme_3.1-155 bitops_1.0-7 bit64_4.0.5
[4] filelock_1.0.2 progress_1.2.2 httr_1.4.2
[7] vegan_2.5-7 utf8_1.2.2 R6_2.5.1
[10] mgcv_1.8-39 DBI_1.1.2 colorspace_2.0-3
[13] permute_0.9-7 tidyselect_1.1.2 prettyunits_1.1.1
[16] bit_4.0.4 curl_4.3.2 compiler_4.1.3
[19] cli_3.2.0 xml2_1.3.3 DelayedArray_0.20.0
[22] scales_1.1.1 rappdirs_0.3.3 stringr_1.4.0
[25] digest_0.6.29 pkgconfig_2.0.3 dbplyr_2.1.1
[28] fastmap_1.1.0 rlang_1.0.2 rstudioapi_0.13
[31] RSQLite_2.2.11 BiocIO_1.4.0 generics_0.1.2
[34] dplyr_1.0.8 RCurl_1.98-1.6 magrittr_2.0.3
[37] GenomeInfoDbData_1.2.7 Matrix_1.4-0 Rcpp_1.0.8.3
[40] munsell_0.5.0 fansi_1.0.3 ape_5.6-2
[43] lifecycle_1.0.1 stringi_1.7.6 yaml_2.3.5
[46] MASS_7.3-56 zlibbioc_1.40.0 plyr_1.8.7
[49] BiocFileCache_2.2.1 pinfsc50_1.2.0 grid_4.1.3
[52] blob_1.2.2 parallel_4.1.3 crayon_1.5.1
[55] lattice_0.20-45 splines_4.1.3 hms_1.1.1
[58] KEGGREST_1.34.0 pillar_1.7.0 rjson_0.2.21
[61] reshape2_1.4.4 biomaRt_2.50.3 XML_3.99-0.9
[64] glue_1.6.2 BiocManager_1.30.16 png_0.1-7
[67] vctrs_0.4.0 gtable_0.3.0 purrr_0.3.4
[70] assertthat_0.2.1 cachem_1.0.6 ggplot2_3.3.5
[73] restfulr_0.0.13 viridisLite_0.4.0 tibble_3.1.6
[76] memoise_2.0.1 cluster_2.1.2 ellipsis_0.3.2
Thanks for the clear post. I am working on installing VariantTools, which depends on gmapR ... so it is taking some time. Current code base is tested against R 4.2 and you are using 4.1. This could be an obstacle to thorough reconciliation. To debug problems that may be related to parallelization, it often helps to 1) cut down the problem (limited regions on a small number of chromosomes), 2) verify that the limited tasks succeed with serial execution (e.g., register(SerialParam()) before testing).
I couldn't reproduce any aspect of your problem with the example in tallyVariants. Noting that you have
1 parallel job did not deliver a result, perhaps you just need to turn down the number of cores in use -- maybe your jobs contended and one was knocked out. Otherwise I think you'll have to give us access to a reproducible example.