Question: rhdf5 segfault
0
5.6 years ago by
Dear Bernd and List Members: I'm getting a segmentation fault reading in a character vector (attached) written by rhdf5. If anyone can make a suggestion how to get rhdf5 to read this without throwing the error I would appreciate it. The file appears to be written correctly since h5dump can read it. I got the same error with the same vector on both my mac laptop and our linux server, in both cases using the latest version of rhdf5 (2.7.4). Brad > library(rhdf5) > load("charvec.rda") ## provides "charvec" > str(charvec) chr [1:16630] "lincRNA:chr8:119735533-119736887 forward strand" ... > hdf5.file <- tempfile() > h5createFile(hdf5.file) [1] TRUE > h5write(charvec, hdf5.file, "foo") > got <- h5read(hdf5.file, "foo") Error: segfault from C stack overflow > sessionInfo() R version 3.0.1 (2013-05-16) Platform: x86_64-apple-darwin10.8.0 (64-bit) locale: [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] rhdf5_2.7.4 loaded via a namespace (and not attached): [1] zlibbioc_1.6.0
rhdf5 • 1.0k views
modified 4.5 years ago by imadcat0 • written 5.6 years ago by Brad Friedman40
1
5.6 years ago by
You might try changing your stack size limits in your shell, before launching R. e.g. ulimit -s unlimited I know I encountered a similar issue with rhdf5 not to long ago reading from a file with a fairly large number of strings, and that was a suitable work around. Note within R Cstack_info() will also tell you stack size. At the time I looked into it, at least in my case I tracked it down to a segfault occuring in the H5Dread_helper_STRING function of file H5D.C at line 244, but it looked like the issue might be with the allocation: char bufSTR[n][size] in the previous line and I can only assume that n*size is too large for the C stack (in my instance, and perhaps yours) at the default settings in your shell. Best, Ben On Dec 23, 2013, at 4:11 PM, Brad Friedman <friedman.brad@gene.com> wrote: > Dear Bernd and List Members: > > I'm getting a segmentation fault reading in a character vector (attached) > written by rhdf5. If anyone can make a suggestion how to get rhdf5 to read > this without throwing the error I would appreciate it. The file appears to > be written correctly since h5dump can read it. > > I got the same error with the same vector on both my mac laptop and our > linux server, in both cases using the latest version of rhdf5 (2.7.4). > > Brad > >> library(rhdf5) >> load("charvec.rda") ## provides "charvec" > > >> str(charvec) > chr [1:16630] "lincRNA:chr8:119735533-119736887 forward strand" ... >> hdf5.file <- tempfile() >> h5createFile(hdf5.file) > [1] TRUE >> h5write(charvec, hdf5.file, "foo") >> got <- h5read(hdf5.file, "foo") > Error: segfault from C stack overflow >> sessionInfo() > R version 3.0.1 (2013-05-16) > Platform: x86_64-apple-darwin10.8.0 (64-bit) > > locale: > [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] rhdf5_2.7.4 > > loaded via a namespace (and not attached): > [1] zlibbioc_1.6.0 > _______________________________________________ > Bioconductor mailing list > Bioconductor@r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor [[alternative HTML version deleted]]
On Dec 23, 2013, at 10:56 PM, Ben Bolstad <bmb at="" bmbolstad.com=""> wrote: > You might try changing your stack size limits in your shell, before launching R. e.g. > > ulimit -s unlimited I don?t think that is possible, at least as you put it. The soft limit tends to be smaller than the hard limit and is the only thing that can be increased. You may not get unlimited, but this should work and might help: limit -Ss ulimit -Hs > I know I encountered a similar issue with rhdf5 not to long ago reading from a file with a fairly large number of strings, and that was a suitable work around. Note within R Cstack_info() will also tell you stack size. > > At the time I looked into it, at least in my case I tracked it down to a segfault occuring in the H5Dread_helper_STRING function of file H5D.C at line 244, but it looked like the issue might be with the allocation: > > char bufSTR[n][size] > > in the previous line and I can only assume that n*size is too large for the C stack (in my instance, and perhaps yours) at the default settings in your shell. > > > Best, > > Ben > > > > On Dec 23, 2013, at 4:11 PM, Brad Friedman <friedman.brad at="" gene.com=""> wrote: > >> Dear Bernd and List Members: >> >> I'm getting a segmentation fault reading in a character vector (attached) >> written by rhdf5. If anyone can make a suggestion how to get rhdf5 to read >> this without throwing the error I would appreciate it. The file appears to >> be written correctly since h5dump can read it. >> >> I got the same error with the same vector on both my mac laptop and our >> linux server, in both cases using the latest version of rhdf5 (2.7.4). >> >> Brad >> >>> library(rhdf5) >>> load("charvec.rda") ## provides "charvec" >> >> >>> str(charvec) >> chr [1:16630] "lincRNA:chr8:119735533-119736887 forward strand" ... >>> hdf5.file <- tempfile() >>> h5createFile(hdf5.file) >> [1] TRUE >>> h5write(charvec, hdf5.file, "foo") >>> got <- h5read(hdf5.file, "foo") >> Error: segfault from C stack overflow >>> sessionInfo() >> R version 3.0.1 (2013-05-16) >> Platform: x86_64-apple-darwin10.8.0 (64-bit) >> >> locale: >> [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8 >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> other attached packages: >> [1] rhdf5_2.7.4 >> >> loaded via a namespace (and not attached): >> [1] zlibbioc_1.6.0 >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at r-project.org >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > > [[alternative HTML version deleted]] > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor
0
5.6 years ago by
EMBL European Molecular Biology Laboratory
Wolfgang Huber13k wrote:
Dear Brad thank you for the feedback, much appreciated. I could not run your code since the file ?charvec.rda? is presumably only on your harddrive, but a similar piece of code (below) worked fine for me. This is on Mac OS 10.9.1, I built R and rhdf5 from source using system libraries from Mac Ports. Since this seems to be a problem specific for certain installations, we?ll need a reproducible example, exact system description, and more detailed debugging information at the C level (afaIu there are people locally at your workplace who can help with that). Kind regards Wolfgang -------------------------- library("rhdf5") ## load("charvec.rda") ## provides "charvec" charvec = paste(1:16630) str(charvec) hdf5.file <- tempfile() h5createFile(hdf5.file) h5write(charvec, hdf5.file, "foo") got <- h5read(hdf5.file, "foo") stopifnot(all(got==charvec), length(got)==length(charvec)) sessionInfo() ---------------------------------------------- > sessionInfo() R Under development (unstable) (2013-12-25 r64540) Platform: x86_64-apple-darwin13.0.0/x86_64 (64-bit) locale: [1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] rhdf5_2.7.4 fortunes_1.5-2 loaded via a namespace (and not attached): [1] zlibbioc_1.8.0 On 24 Dec 2013, at 01:11, Brad Friedman <friedman.brad at="" gene.com=""> wrote: > Dear Bernd and List Members: > > I'm getting a segmentation fault reading in a character vector (attached) > written by rhdf5. If anyone can make a suggestion how to get rhdf5 to read > this without throwing the error I would appreciate it. The file appears to > be written correctly since h5dump can read it. > > I got the same error with the same vector on both my mac laptop and our > linux server, in both cases using the latest version of rhdf5 (2.7.4). > > Brad > >> library(rhdf5) >> load("charvec.rda") ## provides "charvec" > > >> str(charvec) > chr [1:16630] "lincRNA:chr8:119735533-119736887 forward strand" ... >> hdf5.file <- tempfile() >> h5createFile(hdf5.file) > [1] TRUE >> h5write(charvec, hdf5.file, "foo") >> got <- h5read(hdf5.file, "foo") > Error: segfault from C stack overflow >> sessionInfo() > R version 3.0.1 (2013-05-16) > Platform: x86_64-apple-darwin10.8.0 (64-bit) > > locale: > [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] rhdf5_2.7.4 > > loaded via a namespace (and not attached): > [1] zlibbioc_1.6.0 > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor
Hi, Wolfgang Thanks for your response. I attached a file charvec.rda to the original posting....maybe I did this wrong? I'm attaching it here. Do you mind trying one more time if you can reproduce my error? Thanks, Brad On Wed, Dec 25, 2013 at 11:06 AM, Wolfgang Huber <whuber at="" embl.de=""> wrote: > Dear Brad > > thank you for the feedback, much appreciated. I could not run your code > since the file ?charvec.rda? is presumably only on your harddrive, but a > similar piece of code (below) worked fine for me. This is on Mac OS > 10.9.1, I built R and rhdf5 from source using system libraries from Mac > Ports. > > Since this seems to be a problem specific for certain installations, we?ll > need a reproducible example, exact system description, and more detailed > debugging information at the C level (afaIu there are people locally at > your workplace who can help with that). > > Kind regards > Wolfgang > > -------------------------- > library("rhdf5") > > ## load("charvec.rda") ## provides "charvec" > charvec = paste(1:16630) > > str(charvec) > > hdf5.file <- tempfile() > h5createFile(hdf5.file) > > h5write(charvec, hdf5.file, "foo") > got <- h5read(hdf5.file, "foo") > > stopifnot(all(got==charvec), length(got)==length(charvec)) > > sessionInfo() > > ---------------------------------------------- > > sessionInfo() > R Under development (unstable) (2013-12-25 r64540) > Platform: x86_64-apple-darwin13.0.0/x86_64 (64-bit) > > locale: > [1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] rhdf5_2.7.4 fortunes_1.5-2 > > loaded via a namespace (and not attached): > [1] zlibbioc_1.8.0 > > > > > On 24 Dec 2013, at 01:11, Brad Friedman <friedman.brad at="" gene.com=""> wrote: > > > Dear Bernd and List Members: > > > > I'm getting a segmentation fault reading in a character vector (attached) > > written by rhdf5. If anyone can make a suggestion how to get rhdf5 to > read > > this without throwing the error I would appreciate it. The file appears > to > > be written correctly since h5dump can read it. > > > > I got the same error with the same vector on both my mac laptop and our > > linux server, in both cases using the latest version of rhdf5 (2.7.4). > > > > Brad > > > >> library(rhdf5) > >> load("charvec.rda") ## provides "charvec" > > > > > >> str(charvec) > > chr [1:16630] "lincRNA:chr8:119735533-119736887 forward strand" ... > >> hdf5.file <- tempfile() > >> h5createFile(hdf5.file) > > [1] TRUE > >> h5write(charvec, hdf5.file, "foo") > >> got <- h5read(hdf5.file, "foo") > > Error: segfault from C stack overflow > >> sessionInfo() > > R version 3.0.1 (2013-05-16) > > Platform: x86_64-apple-darwin10.8.0 (64-bit) > > > > locale: > > [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8 > > > > attached base packages: > > [1] stats graphics grDevices utils datasets methods base > > > > other attached packages: > > [1] rhdf5_2.7.4 > > > > loaded via a namespace (and not attached): > > [1] zlibbioc_1.6.0 > > _______________________________________________ > > Bioconductor mailing list > > Bioconductor at r-project.org > > https://stat.ethz.ch/mailman/listinfo/bioconductor > > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor > >
0
4.5 years ago by
Shanghai, China

The solution (ulimit -s unlimited) given by Ben Bolstad works on Linux machine, but not on Mac OS. ulimit -s won's accept unlimited as a valid parameter, instead you have to specify the size in number such as  ulimit -s 20000.

It appears that in Mac OS 10.10 ulimit no longer works, you should instead use launchctl limit stack to set the stack size, but the size can not exceed the system stack hard limit.

So first run a

launchctl limit stack

to check the 2nd number (in my case 67104768).  Then

sudo launchctl limit stack 67104768

to set the current stack limit to the hard limit.

Restart your terminal or rstudio, now h5read should work without segment fault.