Search
Question: Rsubread crashes in 32bit linux
0
gravatar for Robert Castelo
5.5 years ago by
Robert Castelo2.0k
Spain/Barcelona/Universitat Pompeu Fabra
Robert Castelo2.0k wrote:
hi, the computer room at my university where we do practicals on R & Bioconductor runs a 32bit linux distribution and when i tried to run the latest version of the Rsubread package (1.6.3) it crashes when calling the buildindex() function on a multifasta file with the yeast genome. this does *not* happen under a 64bit linux distribution. i have verified that installing the version before (1.4.4) on the current R 2.15 it also crashes (on the 32bit), but two versions before, the 1.1.1, it does *not* and it works smoothly on this 32bit linux distribution. i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 where allChr.fa is the multifasta file with the yeast genome. so i can manage by now the problem by using the 1.1.1 version on R 2.15 for my teaching but i wonder whether there would be some easy solution for this, or even if it could be a symptom of something else that the Rsubread developers should worry about. i know that using a 32bit system nowadays is quite obsolete but this is what i got for teaching :( and i would be happy to let my students play with the latest version of Rsubread in the future. thanks!!! robert. ======================Rsubread 1.6.3 on R 2.15======================= > library(Rsubread) > sessionInfo() R version 2.15.0 (2012-03-30) Platform: i686-pc-linux-gnu (32-bit) locale: [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 [7] LC_PAPER=C LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] Rsubread_1.6.3 > buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) Building a base-space index. Size of memory used=2500 MB Base name of the built index = subreadindex *** caught segfault *** address 0xdf670cc0, cause 'memory not mapped' Traceback: 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = as.character(cmd), PACKAGE = "Rsubread") 2: buildindex(basename = "subreadindex", reference = "allChr.fa", memory = 2500) Possible actions: 1: abort (with core dump, if enabled) 2: normal R exit 3: exit R without saving workspace 4: exit R saving workspace Selection: ======================Rsubread 1.1.1 on R 2.15======================= > library(Rsubread) > buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) Building the index in the base space. Size of memory requested=2500 MB Index base name = subreadindex INDEX ITEMS PER PARTITION = 275940352 completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps completed=81.76%; time used=2.4s; rate=4111.8k bps/s; total=12m bps All the chromosome files are processed. | Dumping index [===========================================================>] Index subreadindex is successfully built. > sessionInfo() R version 2.15.0 (2012-03-30) Platform: i686-pc-linux-gnu (32-bit) locale: [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 [7] LC_PAPER=C LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] Rsubread_1.1.1
ADD COMMENTlink modified 5.5 years ago by Wei Shi2.7k • written 5.5 years ago by Robert Castelo2.0k
0
gravatar for Dan Du
5.5 years ago by
Dan Du210
Germany
Dan Du210 wrote:
Hi Robert, I have been experiencing something else, possibly related to yours, on a 64bit ubuntu laptop with 6g of ram. As I recall, when bumping to Bioc 2.10, the Rsubread installation kind of ate all the memory, basically froze the system so I had to call it off, yet building it on the server side turned out fine. So I think I just accepted that the new version may be 'computationally heavy' thus not suitable for a normal pc, though I did not find any mentioning of this increased memory requirement in the NEWS file. So currently Rsubread stays at 1.4.4 on that pc, all subsequent versions of Rsubread drain the memory in the same way when compiling Rsubread.so. Now I think I can confirm this on a 32-bit opensuse box, it did successfully built, but when running the example code in the manual, same segfault happens. > library(Rsubread) > ref <- system.file("extdata","reference.fa",package="Rsubread") > path <- system.file("extdata",package="Rsubread") > buildindex(basename=file.path(path,"reference_index"),reference=ref) Building a base-space index. Size of memory used=3700 MB Base name of the built index = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index *** caught segfault *** address 0xdf03ee80, cause 'memory not mapped' Traceback: 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = as.character(cmd), PACKAGE = "Rsubread") 2: buildindex(basename = file.path(path, "reference_index"), reference = ref) > sessionInfo() R version 2.15.0 Patched (2012-06-04 r59517) Platform: i686-pc-linux-gnu (32-bit) locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=C LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] Rsubread_1.6.3 Regards, Dan On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: > hi, > > the computer room at my university where we do practicals on R & Bioconductor runs a 32bit linux distribution and when i tried to run the latest version of the Rsubread package (1.6.3) it crashes when calling the buildindex() function on a multifasta file with the yeast genome. this does *not* happen under a 64bit linux distribution. > > i have verified that installing the version before (1.4.4) on the current R 2.15 it also crashes (on the 32bit), but two versions before, the 1.1.1, it does *not* and it works smoothly on this 32bit linux distribution. > > i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 where allChr.fa is the multifasta file with the yeast genome. > > so i can manage by now the problem by using the 1.1.1 version on R 2.15 for my teaching but i wonder whether there would be some easy solution for this, or even if it could be a symptom of something else that the Rsubread developers should worry about. i know that using a 32bit system nowadays is quite obsolete but this is what i got for teaching :( and i would be happy to let my students play with the latest version of Rsubread in the future. > > > thanks!!! > robert. > > ======================Rsubread 1.6.3 on R 2.15======================= > > > library(Rsubread) > > sessionInfo() > R version 2.15.0 (2012-03-30) > Platform: i686-pc-linux-gnu (32-bit) > > locale: > [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > [7] LC_PAPER=C LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] Rsubread_1.6.3 > > > buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) > > Building a base-space index. > Size of memory used=2500 MB > Base name of the built index = subreadindex > > *** caught segfault *** > address 0xdf670cc0, cause 'memory not mapped' > > Traceback: > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = as.character(cmd), PACKAGE = "Rsubread") > 2: buildindex(basename = "subreadindex", reference = "allChr.fa", memory = 2500) > > Possible actions: > 1: abort (with core dump, if enabled) > 2: normal R exit > 3: exit R without saving workspace > 4: exit R saving workspace > Selection: > > > ======================Rsubread 1.1.1 on R 2.15======================= > > > library(Rsubread) > > buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) > > Building the index in the base space. > Size of memory requested=2500 MB > Index base name = subreadindex > INDEX ITEMS PER PARTITION = 275940352 > > completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps completed=81.76%; time used=2.4s; rate=4111.8k bps/s; total=12m bps > All the chromosome files are processed. > | Dumping index [===========================================================>] > Index subreadindex is successfully built. > > sessionInfo() > R version 2.15.0 (2012-03-30) > Platform: i686-pc-linux-gnu (32-bit) > > locale: > [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > [7] LC_PAPER=C LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] Rsubread_1.1.1 > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor
ADD COMMENTlink written 5.5 years ago by Dan Du210
Dear Dan, It is probably because including genome sequences into the index slowed down your laptop. But I believe it should be alleviated if you give smaller values to the 'memory ' parameter of the buildindex() function. Also, the index building is an one-off operation, you do not need to redo it even when new releases come. For your 32-bit opensuse box, I guess the problem will be solved if you change the amount of memory requested to be 1000MB. Cheers, Wei On Jun 5, 2012, at 11:43 PM, Dan Du wrote: > Hi Robert, > > I have been experiencing something else, possibly related to yours, > on a 64bit ubuntu laptop with 6g of ram. > > As I recall, when bumping to Bioc 2.10, the Rsubread installation kind > of ate all the memory, basically froze the system so I had to call it > off, yet building it on the server side turned out fine. So I think I > just accepted that the new version may be 'computationally heavy' thus > not suitable for a normal pc, though I did not find any mentioning of > this increased memory requirement in the NEWS file. > > So currently Rsubread stays at 1.4.4 on that pc, all subsequent versions > of Rsubread drain the memory in the same way when compiling Rsubread.so. > > Now I think I can confirm this on a 32-bit opensuse box, it did > successfully built, but when running the example code in the manual, > same segfault happens. > > >> library(Rsubread) >> ref <- system.file("extdata","reference.fa",package="Rsubread") >> path <- system.file("extdata",package="Rsubread") >> buildindex(basename=file.path(path,"reference_index"),reference=ref) > > Building a base-space index. > Size of memory used=3700 MB > Base name of the built index > = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index > > *** caught segfault *** > address 0xdf03ee80, cause 'memory not mapped' > > Traceback: > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = > as.character(cmd), PACKAGE = "Rsubread") > 2: buildindex(basename = file.path(path, "reference_index"), reference > = ref) > >> sessionInfo() > R version 2.15.0 Patched (2012-06-04 r59517) > Platform: i686-pc-linux-gnu (32-bit) > > locale: > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > [7] LC_PAPER=C LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods > base > > other attached packages: > [1] Rsubread_1.6.3 > > > Regards, > Dan > > On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: >> hi, >> >> the computer room at my university where we do practicals on R & Bioconductor runs a 32bit linux distribution and when i tried to run the latest version of the Rsubread package (1.6.3) it crashes when calling the buildindex() function on a multifasta file with the yeast genome. this does *not* happen under a 64bit linux distribution. >> >> i have verified that installing the version before (1.4.4) on the current R 2.15 it also crashes (on the 32bit), but two versions before, the 1.1.1, it does *not* and it works smoothly on this 32bit linux distribution. >> >> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 where allChr.fa is the multifasta file with the yeast genome. >> >> so i can manage by now the problem by using the 1.1.1 version on R 2.15 for my teaching but i wonder whether there would be some easy solution for this, or even if it could be a symptom of something else that the Rsubread developers should worry about. i know that using a 32bit system nowadays is quite obsolete but this is what i got for teaching :( and i would be happy to let my students play with the latest version of Rsubread in the future. >> >> >> thanks!!! >> robert. >> >> ======================Rsubread 1.6.3 on R 2.15======================= >> >>> library(Rsubread) >>> sessionInfo() >> R version 2.15.0 (2012-03-30) >> Platform: i686-pc-linux-gnu (32-bit) >> >> locale: >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >> [7] LC_PAPER=C LC_NAME=C >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> other attached packages: >> [1] Rsubread_1.6.3 >> >>> buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) >> >> Building a base-space index. >> Size of memory used=2500 MB >> Base name of the built index = subreadindex >> >> *** caught segfault *** >> address 0xdf670cc0, cause 'memory not mapped' >> >> Traceback: >> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = as.character(cmd), PACKAGE = "Rsubread") >> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", memory = 2500) >> >> Possible actions: >> 1: abort (with core dump, if enabled) >> 2: normal R exit >> 3: exit R without saving workspace >> 4: exit R saving workspace >> Selection: >> >> >> ======================Rsubread 1.1.1 on R 2.15======================= >> >>> library(Rsubread) >>> buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) >> >> Building the index in the base space. >> Size of memory requested=2500 MB >> Index base name = subreadindex >> INDEX ITEMS PER PARTITION = 275940352 >> >> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps completed=81.76%; time used=2.4s; rate=4111.8k bps/s; total=12m bps >> All the chromosome files are processed. >> | Dumping index [===========================================================>] >> Index subreadindex is successfully built. >>> sessionInfo() >> R version 2.15.0 (2012-03-30) >> Platform: i686-pc-linux-gnu (32-bit) >> >> locale: >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >> [7] LC_PAPER=C LC_NAME=C >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> other attached packages: >> [1] Rsubread_1.1.1 >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at r-project.org >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:6}}
ADD REPLYlink written 5.5 years ago by Wei Shi2.7k
Dear Wei, Unfortunately reducing the memory parameter to 1000, still causes the segfault. I guess with 3g ram limit on a 32bit system, there is still a fat chance that you can not request a continuous 1g block. For that 64bit laptop, it is still strange about the 6g memory draining. It is happing during the installation when compiling the shared library Rsubread.so, not running the buildindex function. Btw, the gcc version is 4.4.3. Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. Regards, Dan On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: > Dear Dan, > > It is probably because including genome sequences into the index slowed down your laptop. But I believe it should be alleviated if you give smaller values to the 'memory ' parameter of the buildindex() function. Also, the index building is an one-off operation, you do not need to redo it even when new releases come. > > For your 32-bit opensuse box, I guess the problem will be solved if you change the amount of memory requested to be 1000MB. > > Cheers, > Wei > > On Jun 5, 2012, at 11:43 PM, Dan Du wrote: > > > Hi Robert, > > > > I have been experiencing something else, possibly related to yours, > > on a 64bit ubuntu laptop with 6g of ram. > > > > As I recall, when bumping to Bioc 2.10, the Rsubread installation kind > > of ate all the memory, basically froze the system so I had to call it > > off, yet building it on the server side turned out fine. So I think I > > just accepted that the new version may be 'computationally heavy' thus > > not suitable for a normal pc, though I did not find any mentioning of > > this increased memory requirement in the NEWS file. > > > > So currently Rsubread stays at 1.4.4 on that pc, all subsequent versions > > of Rsubread drain the memory in the same way when compiling Rsubread.so. > > > > Now I think I can confirm this on a 32-bit opensuse box, it did > > successfully built, but when running the example code in the manual, > > same segfault happens. > > > > > >> library(Rsubread) > >> ref <- system.file("extdata","reference.fa",package="Rsubread") > >> path <- system.file("extdata",package="Rsubread") > >> buildindex(basename=file.path(path,"reference_index"),reference=ref) > > > > Building a base-space index. > > Size of memory used=3700 MB > > Base name of the built index > > = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index > > > > *** caught segfault *** > > address 0xdf03ee80, cause 'memory not mapped' > > > > Traceback: > > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = > > as.character(cmd), PACKAGE = "Rsubread") > > 2: buildindex(basename = file.path(path, "reference_index"), reference > > = ref) > > > >> sessionInfo() > > R version 2.15.0 Patched (2012-06-04 r59517) > > Platform: i686-pc-linux-gnu (32-bit) > > > > locale: > > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > > [7] LC_PAPER=C LC_NAME=C > > [9] LC_ADDRESS=C LC_TELEPHONE=C > > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > > > > attached base packages: > > [1] stats graphics grDevices utils datasets methods > > base > > > > other attached packages: > > [1] Rsubread_1.6.3 > > > > > > Regards, > > Dan > > > > On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: > >> hi, > >> > >> the computer room at my university where we do practicals on R & Bioconductor runs a 32bit linux distribution and when i tried to run the latest version of the Rsubread package (1.6.3) it crashes when calling the buildindex() function on a multifasta file with the yeast genome. this does *not* happen under a 64bit linux distribution. > >> > >> i have verified that installing the version before (1.4.4) on the current R 2.15 it also crashes (on the 32bit), but two versions before, the 1.1.1, it does *not* and it works smoothly on this 32bit linux distribution. > >> > >> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 where allChr.fa is the multifasta file with the yeast genome. > >> > >> so i can manage by now the problem by using the 1.1.1 version on R 2.15 for my teaching but i wonder whether there would be some easy solution for this, or even if it could be a symptom of something else that the Rsubread developers should worry about. i know that using a 32bit system nowadays is quite obsolete but this is what i got for teaching :( and i would be happy to let my students play with the latest version of Rsubread in the future. > >> > >> > >> thanks!!! > >> robert. > >> > >> ======================Rsubread 1.6.3 on R 2.15======================= > >> > >>> library(Rsubread) > >>> sessionInfo() > >> R version 2.15.0 (2012-03-30) > >> Platform: i686-pc-linux-gnu (32-bit) > >> > >> locale: > >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > >> [7] LC_PAPER=C LC_NAME=C > >> [9] LC_ADDRESS=C LC_TELEPHONE=C > >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > >> > >> attached base packages: > >> [1] stats graphics grDevices utils datasets methods base > >> > >> other attached packages: > >> [1] Rsubread_1.6.3 > >> > >>> buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) > >> > >> Building a base-space index. > >> Size of memory used=2500 MB > >> Base name of the built index = subreadindex > >> > >> *** caught segfault *** > >> address 0xdf670cc0, cause 'memory not mapped' > >> > >> Traceback: > >> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = as.character(cmd), PACKAGE = "Rsubread") > >> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", memory = 2500) > >> > >> Possible actions: > >> 1: abort (with core dump, if enabled) > >> 2: normal R exit > >> 3: exit R without saving workspace > >> 4: exit R saving workspace > >> Selection: > >> > >> > >> ======================Rsubread 1.1.1 on R 2.15======================= > >> > >>> library(Rsubread) > >>> buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) > >> > >> Building the index in the base space. > >> Size of memory requested=2500 MB > >> Index base name = subreadindex > >> INDEX ITEMS PER PARTITION = 275940352 > >> > >> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps completed=81.76%; time used=2.4s; rate=4111.8k bps/s; total=12m bps > >> All the chromosome files are processed. > >> | Dumping index [===========================================================>] > >> Index subreadindex is successfully built. > >>> sessionInfo() > >> R version 2.15.0 (2012-03-30) > >> Platform: i686-pc-linux-gnu (32-bit) > >> > >> locale: > >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > >> [7] LC_PAPER=C LC_NAME=C > >> [9] LC_ADDRESS=C LC_TELEPHONE=C > >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > >> > >> attached base packages: > >> [1] stats graphics grDevices utils datasets methods base > >> > >> other attached packages: > >> [1] Rsubread_1.1.1 > >> > >> _______________________________________________ > >> Bioconductor mailing list > >> Bioconductor at r-project.org > >> https://stat.ethz.ch/mailman/listinfo/bioconductor > >> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > > > _______________________________________________ > > Bioconductor mailing list > > Bioconductor at r-project.org > > https://stat.ethz.ch/mailman/listinfo/bioconductor > > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > > ______________________________________________________________________ > The information in this email is confidential and inte...{{dropped:6}}
ADD REPLYlink written 5.5 years ago by Dan Du210
Dear Wei, I also encounter a (perhaps related) problem using Rsubread (see below). Kind regards Peter > library(Rsubread) Error in dyn.load(file, DLLpath = DLLpath, ...) : unable to load shared object '/opt/ghpc/R-2.15.0/lib64/R/library/Rsubread/libs/Rsubread.so': /opt/ghpc/R-2.15.0/lib64/R/library/Rsubread/libs/Rsubread.so: cannot map zero-fill pages: Cannot allocate memory Error: package/namespace load failed for 'Rsubread' > ref <- system.file("extdata","reference.fa",package="Rsubread") > path <- system.file("extdata",package="Rsubread") > buildindex(basename=file.path(path,"reference_index"),reference=ref) Error: could not find function "buildindex" > sessionInfo() R version 2.15.0 (2012-03-30) Platform: x86_64-unknown-linux-gnu (64-bit) locale: [1] LC_CTYPE=en_US LC_NUMERIC=C LC_TIME=en_US [4] LC_COLLATE=en_US LC_MONETARY=en_US LC_MESSAGES=en_US [7] LC_PAPER=C LC_NAME=C LC_ADDRESS=C [10] LC_TELEPHONE=C LC_MEASUREMENT=en_US LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base > ________________________________________ From: bioconductor-bounces@r-project.org [bioconductor- bounces@r-project.org] On Behalf Of Dan Du [tooyoung@gmail.com] Sent: Wednesday, June 06, 2012 8:50 AM To: Wei Shi Cc: Yang Liao; bioconductor at r-project.org mailman Subject: Re: [BioC] Rsubread crashes in 32bit linux Dear Wei, Unfortunately reducing the memory parameter to 1000, still causes the segfault. I guess with 3g ram limit on a 32bit system, there is still a fat chance that you can not request a continuous 1g block. For that 64bit laptop, it is still strange about the 6g memory draining. It is happing during the installation when compiling the shared library Rsubread.so, not running the buildindex function. Btw, the gcc version is 4.4.3. Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. Regards, Dan On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: > Dear Dan, > > It is probably because including genome sequences into the index slowed down your laptop. But I believe it should be alleviated if you give smaller values to the 'memory ' parameter of the buildindex() function. Also, the index building is an one-off operation, you do not need to redo it even when new releases come. > > For your 32-bit opensuse box, I guess the problem will be solved if you change the amount of memory requested to be 1000MB. > > Cheers, > Wei > > On Jun 5, 2012, at 11:43 PM, Dan Du wrote: > > > Hi Robert, > > > > I have been experiencing something else, possibly related to yours, > > on a 64bit ubuntu laptop with 6g of ram. > > > > As I recall, when bumping to Bioc 2.10, the Rsubread installation kind > > of ate all the memory, basically froze the system so I had to call it > > off, yet building it on the server side turned out fine. So I think I > > just accepted that the new version may be 'computationally heavy' thus > > not suitable for a normal pc, though I did not find any mentioning of > > this increased memory requirement in the NEWS file. > > > > So currently Rsubread stays at 1.4.4 on that pc, all subsequent versions > > of Rsubread drain the memory in the same way when compiling Rsubread.so. > > > > Now I think I can confirm this on a 32-bit opensuse box, it did > > successfully built, but when running the example code in the manual, > > same segfault happens. > > > > > >> library(Rsubread) > >> ref <- system.file("extdata","reference.fa",package="Rsubread") > >> path <- system.file("extdata",package="Rsubread") > >> buildindex(basename=file.path(path,"reference_index"),reference=ref) > > > > Building a base-space index. > > Size of memory used=3700 MB > > Base name of the built index > > = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index > > > > *** caught segfault *** > > address 0xdf03ee80, cause 'memory not mapped' > > > > Traceback: > > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = > > as.character(cmd), PACKAGE = "Rsubread") > > 2: buildindex(basename = file.path(path, "reference_index"), reference > > = ref) > > > >> sessionInfo() > > R version 2.15.0 Patched (2012-06-04 r59517) > > Platform: i686-pc-linux-gnu (32-bit) > > > > locale: > > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > > [7] LC_PAPER=C LC_NAME=C > > [9] LC_ADDRESS=C LC_TELEPHONE=C > > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > > > > attached base packages: > > [1] stats graphics grDevices utils datasets methods > > base > > > > other attached packages: > > [1] Rsubread_1.6.3 > > > > > > Regards, > > Dan > > > > On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: > >> hi, > >> > >> the computer room at my university where we do practicals on R & Bioconductor runs a 32bit linux distribution and when i tried to run the latest version of the Rsubread package (1.6.3) it crashes when calling the buildindex() function on a multifasta file with the yeast genome. this does *not* happen under a 64bit linux distribution. > >> > >> i have verified that installing the version before (1.4.4) on the current R 2.15 it also crashes (on the 32bit), but two versions before, the 1.1.1, it does *not* and it works smoothly on this 32bit linux distribution. > >> > >> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 where allChr.fa is the multifasta file with the yeast genome. > >> > >> so i can manage by now the problem by using the 1.1.1 version on R 2.15 for my teaching but i wonder whether there would be some easy solution for this, or even if it could be a symptom of something else that the Rsubread developers should worry about. i know that using a 32bit system nowadays is quite obsolete but this is what i got for teaching :( and i would be happy to let my students play with the latest version of Rsubread in the future. > >> > >> > >> thanks!!! > >> robert. > >> > >> ======================Rsubread 1.6.3 on R 2.15======================= > >> > >>> library(Rsubread) > >>> sessionInfo() > >> R version 2.15.0 (2012-03-30) > >> Platform: i686-pc-linux-gnu (32-bit) > >> > >> locale: > >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > >> [7] LC_PAPER=C LC_NAME=C > >> [9] LC_ADDRESS=C LC_TELEPHONE=C > >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > >> > >> attached base packages: > >> [1] stats graphics grDevices utils datasets methods base > >> > >> other attached packages: > >> [1] Rsubread_1.6.3 > >> > >>> buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) > >> > >> Building a base-space index. > >> Size of memory used=2500 MB > >> Base name of the built index = subreadindex > >> > >> *** caught segfault *** > >> address 0xdf670cc0, cause 'memory not mapped' > >> > >> Traceback: > >> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = as.character(cmd), PACKAGE = "Rsubread") > >> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", memory = 2500) > >> > >> Possible actions: > >> 1: abort (with core dump, if enabled) > >> 2: normal R exit > >> 3: exit R without saving workspace > >> 4: exit R saving workspace > >> Selection: > >> > >> > >> ======================Rsubread 1.1.1 on R 2.15======================= > >> > >>> library(Rsubread) > >>> buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) > >> > >> Building the index in the base space. > >> Size of memory requested=2500 MB > >> Index base name = subreadindex > >> INDEX ITEMS PER PARTITION = 275940352 > >> > >> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps completed=81.76%; time used=2.4s; rate=4111.8k bps/s; total=12m bps > >> All the chromosome files are processed. > >> | Dumping index [===========================================================>] > >> Index subreadindex is successfully built. > >>> sessionInfo() > >> R version 2.15.0 (2012-03-30) > >> Platform: i686-pc-linux-gnu (32-bit) > >> > >> locale: > >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > >> [7] LC_PAPER=C LC_NAME=C > >> [9] LC_ADDRESS=C LC_TELEPHONE=C > >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > >> > >> attached base packages: > >> [1] stats graphics grDevices utils datasets methods base > >> > >> other attached packages: > >> [1] Rsubread_1.1.1 > >> > >> _______________________________________________ > >> Bioconductor mailing list > >> Bioconductor at r-project.org > >> https://stat.ethz.ch/mailman/listinfo/bioconductor > >> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > > > _______________________________________________ > > Bioconductor mailing list > > Bioconductor at r-project.org > > https://stat.ethz.ch/mailman/listinfo/bioconductor > > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > > ______________________________________________________________________ > The information in this email is confidential and inte...{{dropped:9}}
ADD REPLYlink written 5.5 years ago by Peter Sørensen (HAG)60
Dear Dan, It didn't seem to be problem of requesting a continuous 1GB block in our investigation. We tracked the memory usage of buildindex() function when running it on yeast genome using a 32-bit VM, and found that the segfault happened right after a request of a few KB of memory was sent to the system when the memory parameter was set to 2500. However, the problem was gone when the memory parameter was changed to 1000. Removing highly repetitive 16 mers required a continuous 1GB block of memory, but this step was always executed successfully. This step also included in the old version of Rsubread (1.1.1), and it did not have problem there either. Could you please provide us your complete code for running your test and also session info? This will help us to diagnose what the problem could be because we couldn't reproduce what you saw from our end. For the compilation issue on your 64bit laptop, could you provide us more details as well, including the message output from gcc? Thanks, Wei > Dear Wei, > > Unfortunately reducing the memory parameter to 1000, still causes the > segfault. I guess with 3g ram limit on a 32bit system, there is still a > fat chance that you can not request a continuous 1g block. > > For that 64bit laptop, it is still strange about the 6g memory draining. > It is happing during the installation when compiling the shared library > Rsubread.so, not running the buildindex function. Btw, the gcc version > is 4.4.3. > > Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. > > Regards, > Dan > > On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: >> Dear Dan, >> >> It is probably because including genome sequences into the index slowed >> down your laptop. But I believe it should be alleviated if you give >> smaller values to the 'memory ' parameter of the buildindex() function. >> Also, the index building is an one-off operation, you do not need to >> redo it even when new releases come. >> >> For your 32-bit opensuse box, I guess the problem will be solved if you >> change the amount of memory requested to be 1000MB. >> >> Cheers, >> Wei >> >> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: >> >> > Hi Robert, >> > >> > I have been experiencing something else, possibly related to yours, >> > on a 64bit ubuntu laptop with 6g of ram. >> > >> > As I recall, when bumping to Bioc 2.10, the Rsubread installation kind >> > of ate all the memory, basically froze the system so I had to call it >> > off, yet building it on the server side turned out fine. So I think I >> > just accepted that the new version may be 'computationally heavy' thus >> > not suitable for a normal pc, though I did not find any mentioning of >> > this increased memory requirement in the NEWS file. >> > >> > So currently Rsubread stays at 1.4.4 on that pc, all subsequent >> versions >> > of Rsubread drain the memory in the same way when compiling >> Rsubread.so. >> > >> > Now I think I can confirm this on a 32-bit opensuse box, it did >> > successfully built, but when running the example code in the manual, >> > same segfault happens. >> > >> > >> >> library(Rsubread) >> >> ref <- system.file("extdata","reference.fa",package="Rsubread") >> >> path <- system.file("extdata",package="Rsubread") >> >> buildindex(basename=file.path(path,"reference_index"),reference=ref) >> > >> > Building a base-space index. >> > Size of memory used=3700 MB >> > Base name of the built index >> > = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index >> > >> > *** caught segfault *** >> > address 0xdf03ee80, cause 'memory not mapped' >> > >> > Traceback: >> > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >> > as.character(cmd), PACKAGE = "Rsubread") >> > 2: buildindex(basename = file.path(path, "reference_index"), reference >> > = ref) >> > >> >> sessionInfo() >> > R version 2.15.0 Patched (2012-06-04 r59517) >> > Platform: i686-pc-linux-gnu (32-bit) >> > >> > locale: >> > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >> > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >> > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >> > [7] LC_PAPER=C LC_NAME=C >> > [9] LC_ADDRESS=C LC_TELEPHONE=C >> > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >> > >> > attached base packages: >> > [1] stats graphics grDevices utils datasets methods >> > base >> > >> > other attached packages: >> > [1] Rsubread_1.6.3 >> > >> > >> > Regards, >> > Dan >> > >> > On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: >> >> hi, >> >> >> >> the computer room at my university where we do practicals on R & >> Bioconductor runs a 32bit linux distribution and when i tried to run >> the latest version of the Rsubread package (1.6.3) it crashes when >> calling the buildindex() function on a multifasta file with the yeast >> genome. this does *not* happen under a 64bit linux distribution. >> >> >> >> i have verified that installing the version before (1.4.4) on the >> current R 2.15 it also crashes (on the 32bit), but two versions >> before, the 1.1.1, it does *not* and it works smoothly on this 32bit >> linux distribution. >> >> >> >> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 >> where allChr.fa is the multifasta file with the yeast genome. >> >> >> >> so i can manage by now the problem by using the 1.1.1 version on R >> 2.15 for my teaching but i wonder whether there would be some easy >> solution for this, or even if it could be a symptom of something else >> that the Rsubread developers should worry about. i know that using a >> 32bit system nowadays is quite obsolete but this is what i got for >> teaching :( and i would be happy to let my students play with the >> latest version of Rsubread in the future. >> >> >> >> >> >> thanks!!! >> >> robert. >> >> >> >> ======================Rsubread 1.6.3 on R 2.15======================= >> >> >> >>> library(Rsubread) >> >>> sessionInfo() >> >> R version 2.15.0 (2012-03-30) >> >> Platform: i686-pc-linux-gnu (32-bit) >> >> >> >> locale: >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >> >> [7] LC_PAPER=C LC_NAME=C >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >> >> >> >> attached base packages: >> >> [1] stats graphics grDevices utils datasets methods base >> >> >> >> other attached packages: >> >> [1] Rsubread_1.6.3 >> >> >> >>> buildindex(basename="subreadindex", reference="allChr.fa", >> memory=2500) >> >> >> >> Building a base-space index. >> >> Size of memory used=2500 MB >> >> Base name of the built index = subreadindex >> >> >> >> *** caught segfault *** >> >> address 0xdf670cc0, cause 'memory not mapped' >> >> >> >> Traceback: >> >> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >> as.character(cmd), PACKAGE = "Rsubread") >> >> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", >> memory = 2500) >> >> >> >> Possible actions: >> >> 1: abort (with core dump, if enabled) >> >> 2: normal R exit >> >> 3: exit R without saving workspace >> >> 4: exit R saving workspace >> >> Selection: >> >> >> >> >> >> ======================Rsubread 1.1.1 on R 2.15======================= >> >> >> >>> library(Rsubread) >> >>> buildindex(basename="subreadindex", reference="allChr.fa", >> memory=2500) >> >> >> >> Building the index in the base space. >> >> Size of memory requested=2500 MB >> >> Index base name = subreadindex >> >> INDEX ITEMS PER PARTITION = 275940352 >> >> >> >> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps >> completed=81.76%; time used=2.4s; rate=4111.8k >> bps/s; total=12m bps >> >> All the chromosome files are processed. >> >> | Dumping index >> [===========================================================>] >> >> Index subreadindex is successfully built. >> >>> sessionInfo() >> >> R version 2.15.0 (2012-03-30) >> >> Platform: i686-pc-linux-gnu (32-bit) >> >> >> >> locale: >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >> >> [7] LC_PAPER=C LC_NAME=C >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >> >> >> >> attached base packages: >> >> [1] stats graphics grDevices utils datasets methods base >> >> >> >> other attached packages: >> >> [1] Rsubread_1.1.1 >> >> >> >> _______________________________________________ >> >> Bioconductor mailing list >> >> Bioconductor at r-project.org >> >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> >> Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor >> > >> > _______________________________________________ >> > Bioconductor mailing list >> > Bioconductor at r-project.org >> > https://stat.ethz.ch/mailman/listinfo/bioconductor >> > Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor >> >> >> ______________________________________________________________________ >> The information in this email is confidential and intended solely for >> the addressee. >> You must not disclose, forward, print or use it without the permission >> of the sender. >> ______________________________________________________________________ > > > > ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:4}}
ADD REPLYlink written 5.5 years ago by Wei Shi2.7k
Dear Wei, Here is a standard bioclite update, I think it is at the last step when compiling Rsubread.so, the memory usage exceeds 5.5g, then system freeze and I have to call it off. Same result when runing 'R CMD INSTALL Rsubread_1.6.3.tar.gz' from shell, or manually compile all .c file and run the last gcc statement. I guess there might just be a minimum ram requirement somewhere higher than 6g... I will do some more poking when I have time. 'gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o gene-value-index.o hashtable.o index-builder.o input-files.o processExons.o propmapped.o qualityScores.o readSummary.o removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR' Also down there are the sessionInfo and full gcc version, please let me know if you need more information. Regards, Dan -------------------------------------------------------------------- > source('http://www.bioconductor.org/biocLite.R') > biocLite('') BioC_mirror: http://bioconductor.org Using R version 2.15, BiocInstaller version 1.4.6. Installing package(s) '' Old packages: 'Rsubread' Update all/some/none? [a/s/n]: a trying URL 'http://www.bioconductor.org/packages/2.10/bioc/src/contrib/Rsubread_1 .6.3.tar.gz' Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb) opened URL ================================================== downloaded 20.9 Mb WARNING: ignoring environment value of R_HOME * installing *source* package ?Rsubread? ... ** libs gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c R_wrapper.c -o R_wrapper.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c SNP_calling.c -o SNP_calling.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c aligner.c -o aligner.o gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c atgcContent.c -o atgcContent.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c detectionCall.c -o detectionCall.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c detectionCallAnnotation.c -o detectionCallAnnotation.o detectionCallAnnotation.c: In function ?calculateExonGCContent?: detectionCallAnnotation.c:175: warning: ignoring return value of ?fgets?, declared with attribute warn_unused_result detectionCallAnnotation.c: In function ?calculateIRGCContent?: detectionCallAnnotation.c:262: warning: ignoring return value of ?fgets?, declared with attribute warn_unused_result gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c exon-algorithms.c -o exon-algorithms.o gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c exon-align.c -o exon-align.o gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c fullscan.c -o fullscan.o gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c gene-algorithms.c -o gene-algorithms.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c gene-value-index.c -o gene-value-index.o gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c hashtable.c -o hashtable.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c index-builder.c -o index-builder.o gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c input-files.c -o input-files.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c processExons.c -o processExons.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c propmapped.c -o propmapped.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c qualityScores.c -o qualityScores.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c readSummary.c -o readSummary.o readSummary.c: In function ?readSummary?: readSummary.c:122: warning: format ?%d? expects type ?int?, but argument 5 has type ?long int? readSummary.c:122: warning: format ?%d? expects type ?int?, but argument 6 has type ?long int? readSummary.c:39: warning: ignoring return value of ?getline?, declared with attribute warn_unused_result readSummary.c:52: warning: ignoring return value of ?getline?, declared with attribute warn_unused_result readSummary.c:55: warning: ignoring return value of ?getline?, declared with attribute warn_unused_result gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c removeDuplicatedReads.c -o removeDuplicatedReads.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c sam2bed.c -o sam2bed.o gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic -O3 -pipe -g -c sorted-hashtable.c -o sorted-hashtable.o gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? declared but never defined gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared but never defined gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o gene-value-index.o hashtable.o index-builder.o input-files.o processExons.o propmapped.o qualityScores.o readSummary.o removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR ^Cmake: *** Deleting file `Rsubread.so' make: *** [Rsubread.so] Interrupt ** R ** inst ** preparing package for lazy loading ** help *** installing help indices ** building package indices ** installing vignettes ?Rsubread.Rnw? ** testing if installed package can be loaded Error in library.dynam(lib, package, package.lib) : shared object ?Rsubread.so? not found Error: loading failed Execution halted -------------------------------------------------------------------- > sessionInfo() R version 2.15.0 (2012-03-30) Platform: x86_64-pc-linux-gnu (64-bit) locale: [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C [3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8 [5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8 [7] LC_PAPER=C LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base -------------------------------------------------------------------- $ gcc -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.3-4ubuntu5.1' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-plugin --enable-objc-gc --disable-werror --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1) -------------------------------------------------------------------- On Wed, 2012-06-06 at 20:10 +1000, Wei Shi wrote: > Dear Dan, > > It didn't seem to be problem of requesting a continuous 1GB block in our > investigation. We tracked the memory usage of buildindex() function when > running it on yeast genome using a 32-bit VM, and found that the segfault > happened right after a request of a few KB of memory was sent to the > system when the memory parameter was set to 2500. However, the problem was > gone when the memory parameter was changed to 1000. > > Removing highly repetitive 16 mers required a continuous 1GB block of > memory, but this step was always executed successfully. This step also > included in the old version of Rsubread (1.1.1), and it did not have > problem there either. > > Could you please provide us your complete code for running your test and > also session info? This will help us to diagnose what the problem could be > because we couldn't reproduce what you saw from our end. > > For the compilation issue on your 64bit laptop, could you provide us more > details as well, including the message output from gcc? > > Thanks, > Wei > > > Dear Wei, > > > > Unfortunately reducing the memory parameter to 1000, still causes the > > segfault. I guess with 3g ram limit on a 32bit system, there is still a > > fat chance that you can not request a continuous 1g block. > > > > For that 64bit laptop, it is still strange about the 6g memory draining. > > It is happing during the installation when compiling the shared library > > Rsubread.so, not running the buildindex function. Btw, the gcc version > > is 4.4.3. > > > > Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. > > > > Regards, > > Dan > > > > On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: > >> Dear Dan, > >> > >> It is probably because including genome sequences into the index slowed > >> down your laptop. But I believe it should be alleviated if you give > >> smaller values to the 'memory ' parameter of the buildindex() function. > >> Also, the index building is an one-off operation, you do not need to > >> redo it even when new releases come. > >> > >> For your 32-bit opensuse box, I guess the problem will be solved if you > >> change the amount of memory requested to be 1000MB. > >> > >> Cheers, > >> Wei > >> > >> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: > >> > >> > Hi Robert, > >> > > >> > I have been experiencing something else, possibly related to yours, > >> > on a 64bit ubuntu laptop with 6g of ram. > >> > > >> > As I recall, when bumping to Bioc 2.10, the Rsubread installation kind > >> > of ate all the memory, basically froze the system so I had to call it > >> > off, yet building it on the server side turned out fine. So I think I > >> > just accepted that the new version may be 'computationally heavy' thus > >> > not suitable for a normal pc, though I did not find any mentioning of > >> > this increased memory requirement in the NEWS file. > >> > > >> > So currently Rsubread stays at 1.4.4 on that pc, all subsequent > >> versions > >> > of Rsubread drain the memory in the same way when compiling > >> Rsubread.so. > >> > > >> > Now I think I can confirm this on a 32-bit opensuse box, it did > >> > successfully built, but when running the example code in the manual, > >> > same segfault happens. > >> > > >> > > >> >> library(Rsubread) > >> >> ref <- system.file("extdata","reference.fa",package="Rsubread") > >> >> path <- system.file("extdata",package="Rsubread") > >> >> buildindex(basename=file.path(path,"reference_index"),reference=ref) > >> > > >> > Building a base-space index. > >> > Size of memory used=3700 MB > >> > Base name of the built index > >> > = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index > >> > > >> > *** caught segfault *** > >> > address 0xdf03ee80, cause 'memory not mapped' > >> > > >> > Traceback: > >> > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = > >> > as.character(cmd), PACKAGE = "Rsubread") > >> > 2: buildindex(basename = file.path(path, "reference_index"), reference > >> > = ref) > >> > > >> >> sessionInfo() > >> > R version 2.15.0 Patched (2012-06-04 r59517) > >> > Platform: i686-pc-linux-gnu (32-bit) > >> > > >> > locale: > >> > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > >> > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > >> > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > >> > [7] LC_PAPER=C LC_NAME=C > >> > [9] LC_ADDRESS=C LC_TELEPHONE=C > >> > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > >> > > >> > attached base packages: > >> > [1] stats graphics grDevices utils datasets methods > >> > base > >> > > >> > other attached packages: > >> > [1] Rsubread_1.6.3 > >> > > >> > > >> > Regards, > >> > Dan > >> > > >> > On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: > >> >> hi, > >> >> > >> >> the computer room at my university where we do practicals on R & > >> Bioconductor runs a 32bit linux distribution and when i tried to run > >> the latest version of the Rsubread package (1.6.3) it crashes when > >> calling the buildindex() function on a multifasta file with the yeast > >> genome. this does *not* happen under a 64bit linux distribution. > >> >> > >> >> i have verified that installing the version before (1.4.4) on the > >> current R 2.15 it also crashes (on the 32bit), but two versions > >> before, the 1.1.1, it does *not* and it works smoothly on this 32bit > >> linux distribution. > >> >> > >> >> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 > >> where allChr.fa is the multifasta file with the yeast genome. > >> >> > >> >> so i can manage by now the problem by using the 1.1.1 version on R > >> 2.15 for my teaching but i wonder whether there would be some easy > >> solution for this, or even if it could be a symptom of something else > >> that the Rsubread developers should worry about. i know that using a > >> 32bit system nowadays is quite obsolete but this is what i got for > >> teaching :( and i would be happy to let my students play with the > >> latest version of Rsubread in the future. > >> >> > >> >> > >> >> thanks!!! > >> >> robert. > >> >> > >> >> ======================Rsubread 1.6.3 on R 2.15======================= > >> >> > >> >>> library(Rsubread) > >> >>> sessionInfo() > >> >> R version 2.15.0 (2012-03-30) > >> >> Platform: i686-pc-linux-gnu (32-bit) > >> >> > >> >> locale: > >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > >> >> [7] LC_PAPER=C LC_NAME=C > >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C > >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > >> >> > >> >> attached base packages: > >> >> [1] stats graphics grDevices utils datasets methods base > >> >> > >> >> other attached packages: > >> >> [1] Rsubread_1.6.3 > >> >> > >> >>> buildindex(basename="subreadindex", reference="allChr.fa", > >> memory=2500) > >> >> > >> >> Building a base-space index. > >> >> Size of memory used=2500 MB > >> >> Base name of the built index = subreadindex > >> >> > >> >> *** caught segfault *** > >> >> address 0xdf670cc0, cause 'memory not mapped' > >> >> > >> >> Traceback: > >> >> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = > >> as.character(cmd), PACKAGE = "Rsubread") > >> >> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", > >> memory = 2500) > >> >> > >> >> Possible actions: > >> >> 1: abort (with core dump, if enabled) > >> >> 2: normal R exit > >> >> 3: exit R without saving workspace > >> >> 4: exit R saving workspace > >> >> Selection: > >> >> > >> >> > >> >> ======================Rsubread 1.1.1 on R 2.15======================= > >> >> > >> >>> library(Rsubread) > >> >>> buildindex(basename="subreadindex", reference="allChr.fa", > >> memory=2500) > >> >> > >> >> Building the index in the base space. > >> >> Size of memory requested=2500 MB > >> >> Index base name = subreadindex > >> >> INDEX ITEMS PER PARTITION = 275940352 > >> >> > >> >> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps > >> completed=81.76%; time used=2.4s; rate=4111.8k > >> bps/s; total=12m bps > >> >> All the chromosome files are processed. > >> >> | Dumping index > >> [===========================================================>] > >> >> Index subreadindex is successfully built. > >> >>> sessionInfo() > >> >> R version 2.15.0 (2012-03-30) > >> >> Platform: i686-pc-linux-gnu (32-bit) > >> >> > >> >> locale: > >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > >> >> [7] LC_PAPER=C LC_NAME=C > >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C > >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > >> >> > >> >> attached base packages: > >> >> [1] stats graphics grDevices utils datasets methods base > >> >> > >> >> other attached packages: > >> >> [1] Rsubread_1.1.1 > >> >> > >> >> _______________________________________________ > >> >> Bioconductor mailing list > >> >> Bioconductor at r-project.org > >> >> https://stat.ethz.ch/mailman/listinfo/bioconductor > >> >> Search the archives: > >> http://news.gmane.org/gmane.science.biology.informatics.conductor > >> > > >> > _______________________________________________ > >> > Bioconductor mailing list > >> > Bioconductor at r-project.org > >> > https://stat.ethz.ch/mailman/listinfo/bioconductor > >> > Search the archives: > >> http://news.gmane.org/gmane.science.biology.informatics.conductor > >> > >> > >> ______________________________________________________________________ > >> The information in this email is confidential and intended solely for > >> the addressee. > >> You must not disclose, forward, print or use it without the permission > >> of the sender. > >> ______________________________________________________________________ > > > > > > > > > > > > ______________________________________________________________________ > The information in this email is confidential and inte...{{dropped:5}}
ADD REPLYlink written 5.5 years ago by Dan Du210
Dear Dan, Thanks a lot for the information. The final step of the gcc compilation, which will link all the object files and creates the dynamic library Rsubread.so, should not have a large consumption of the memory and should be done fairly quick. So there seems to a problem with this step. Which Linux distribution are you running on your laptop? Would you please install a C version of the subread aligner onto your laptop? This will be able to figure out it is C or R which caused the problem. The C version can be downloaded from http://subread.sourceforge.net Thanks, Wei On Jun 6, 2012, at 9:08 PM, Dan Du wrote: > Dear Wei, > > Here is a standard bioclite update, I think it is at the last step when > compiling Rsubread.so, the memory usage exceeds 5.5g, then system freeze > and I have to call it off. Same result when runing 'R CMD INSTALL > Rsubread_1.6.3.tar.gz' from shell, or manually compile all .c file and > run the last gcc statement. I guess there might just be a minimum ram > requirement somewhere higher than 6g... I will do some more poking when > I have time. > > 'gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o > aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o > exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o > gene-value-index.o hashtable.o index-builder.o input-files.o > processExons.o propmapped.o qualityScores.o readSummary.o > removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread > -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR' > > Also down there are the sessionInfo and full gcc version, please let me > know if you need more information. > > Regards, > Dan > -------------------------------------------------------------------- >> source('http://www.bioconductor.org/biocLite.R') >> biocLite('') > BioC_mirror: http://bioconductor.org > Using R version 2.15, BiocInstaller version 1.4.6. > Installing package(s) '' > Old packages: 'Rsubread' > Update all/some/none? [a/s/n]: a > trying URL > 'http://www.bioconductor.org/packages/2.10/bioc/src/contrib/Rsubread _1.6.3.tar.gz' > Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb) > opened URL > ================================================== > downloaded 20.9 Mb > > WARNING: ignoring environment value of R_HOME > * installing *source* package ?Rsubread? ... > ** libs > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c R_wrapper.c -o R_wrapper.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c SNP_calling.c -o SNP_calling.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c aligner.c -o aligner.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c atgcContent.c -o atgcContent.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c detectionCall.c -o detectionCall.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c detectionCallAnnotation.c -o detectionCallAnnotation.o > detectionCallAnnotation.c: In function ?calculateExonGCContent?: > detectionCallAnnotation.c:175: warning: ignoring return value of > ?fgets?, declared with attribute warn_unused_result > detectionCallAnnotation.c: In function ?calculateIRGCContent?: > detectionCallAnnotation.c:262: warning: ignoring return value of > ?fgets?, declared with attribute warn_unused_result > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c exon-algorithms.c -o exon-algorithms.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c exon-align.c -o exon-align.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c fullscan.c -o fullscan.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c gene-algorithms.c -o gene-algorithms.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c gene-value-index.c -o gene-value-index.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c hashtable.c -o hashtable.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c index-builder.c -o index-builder.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c input-files.c -o input-files.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c processExons.c -o processExons.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c propmapped.c -o propmapped.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c qualityScores.c -o qualityScores.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c readSummary.c -o readSummary.o > readSummary.c: In function ?readSummary?: > readSummary.c:122: warning: format ?%d? expects type ?int?, but argument > 5 has type ?long int? > readSummary.c:122: warning: format ?%d? expects type ?int?, but argument > 6 has type ?long int? > readSummary.c:39: warning: ignoring return value of ?getline?, declared > with attribute warn_unused_result > readSummary.c:52: warning: ignoring return value of ?getline?, declared > with attribute warn_unused_result > readSummary.c:55: warning: ignoring return value of ?getline?, declared > with attribute warn_unused_result > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c removeDuplicatedReads.c -o removeDuplicatedReads.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c sam2bed.c -o sam2bed.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c sorted-hashtable.c -o sorted-hashtable.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o > aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o > exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o > gene-value-index.o hashtable.o index-builder.o input-files.o > processExons.o propmapped.o qualityScores.o readSummary.o > removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread > -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR > ^Cmake: *** Deleting file `Rsubread.so' > make: *** [Rsubread.so] Interrupt > ** R > ** inst > ** preparing package for lazy loading > ** help > *** installing help indices > ** building package indices > ** installing vignettes > ?Rsubread.Rnw? > ** testing if installed package can be loaded > Error in library.dynam(lib, package, package.lib) : > shared object ?Rsubread.so? not found > Error: loading failed > Execution halted > -------------------------------------------------------------------- >> sessionInfo() > R version 2.15.0 (2012-03-30) > Platform: x86_64-pc-linux-gnu (64-bit) > > locale: > [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C > [3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8 > [5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8 > [7] LC_PAPER=C LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > -------------------------------------------------------------------- > $ gcc -v > Using built-in specs. > Target: x86_64-linux-gnu > Configured with: ../src/configure -v --with-pkgversion='Ubuntu > 4.4.3-4ubuntu5.1' > --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs > --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr > --enable-shared --enable-multiarch --enable-linker-build-id > --with-system-zlib --libexecdir=/usr/lib --without-included-gettext > --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 > --program-suffix=-4.4 --enable-nls --enable-clocale=gnu > --enable-libstdcxx-debug --enable-plugin --enable-objc-gc > --disable-werror --with-arch-32=i486 --with-tune=generic > --enable-checking=release --build=x86_64-linux-gnu > --host=x86_64-linux-gnu --target=x86_64-linux-gnu > Thread model: posix > gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1) > -------------------------------------------------------------------- > > On Wed, 2012-06-06 at 20:10 +1000, Wei Shi wrote: >> Dear Dan, >> >> It didn't seem to be problem of requesting a continuous 1GB block in our >> investigation. We tracked the memory usage of buildindex() function when >> running it on yeast genome using a 32-bit VM, and found that the segfault >> happened right after a request of a few KB of memory was sent to the >> system when the memory parameter was set to 2500. However, the problem was >> gone when the memory parameter was changed to 1000. >> >> Removing highly repetitive 16 mers required a continuous 1GB block of >> memory, but this step was always executed successfully. This step also >> included in the old version of Rsubread (1.1.1), and it did not have >> problem there either. >> >> Could you please provide us your complete code for running your test and >> also session info? This will help us to diagnose what the problem could be >> because we couldn't reproduce what you saw from our end. >> >> For the compilation issue on your 64bit laptop, could you provide us more >> details as well, including the message output from gcc? >> >> Thanks, >> Wei >> >>> Dear Wei, >>> >>> Unfortunately reducing the memory parameter to 1000, still causes the >>> segfault. I guess with 3g ram limit on a 32bit system, there is still a >>> fat chance that you can not request a continuous 1g block. >>> >>> For that 64bit laptop, it is still strange about the 6g memory draining. >>> It is happing during the installation when compiling the shared library >>> Rsubread.so, not running the buildindex function. Btw, the gcc version >>> is 4.4.3. >>> >>> Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. >>> >>> Regards, >>> Dan >>> >>> On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: >>>> Dear Dan, >>>> >>>> It is probably because including genome sequences into the index slowed >>>> down your laptop. But I believe it should be alleviated if you give >>>> smaller values to the 'memory ' parameter of the buildindex() function. >>>> Also, the index building is an one-off operation, you do not need to >>>> redo it even when new releases come. >>>> >>>> For your 32-bit opensuse box, I guess the problem will be solved if you >>>> change the amount of memory requested to be 1000MB. >>>> >>>> Cheers, >>>> Wei >>>> >>>> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: >>>> >>>>> Hi Robert, >>>>> >>>>> I have been experiencing something else, possibly related to yours, >>>>> on a 64bit ubuntu laptop with 6g of ram. >>>>> >>>>> As I recall, when bumping to Bioc 2.10, the Rsubread installation kind >>>>> of ate all the memory, basically froze the system so I had to call it >>>>> off, yet building it on the server side turned out fine. So I think I >>>>> just accepted that the new version may be 'computationally heavy' thus >>>>> not suitable for a normal pc, though I did not find any mentioning of >>>>> this increased memory requirement in the NEWS file. >>>>> >>>>> So currently Rsubread stays at 1.4.4 on that pc, all subsequent >>>> versions >>>>> of Rsubread drain the memory in the same way when compiling >>>> Rsubread.so. >>>>> >>>>> Now I think I can confirm this on a 32-bit opensuse box, it did >>>>> successfully built, but when running the example code in the manual, >>>>> same segfault happens. >>>>> >>>>> >>>>>> library(Rsubread) >>>>>> ref <- system.file("extdata","reference.fa",package="Rsubread") >>>>>> path <- system.file("extdata",package="Rsubread") >>>>>> buildindex(basename=file.path(path,"reference_index"),reference=ref) >>>>> >>>>> Building a base-space index. >>>>> Size of memory used=3700 MB >>>>> Base name of the built index >>>>> = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index >>>>> >>>>> *** caught segfault *** >>>>> address 0xdf03ee80, cause 'memory not mapped' >>>>> >>>>> Traceback: >>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >>>>> as.character(cmd), PACKAGE = "Rsubread") >>>>> 2: buildindex(basename = file.path(path, "reference_index"), reference >>>>> = ref) >>>>> >>>>>> sessionInfo() >>>>> R version 2.15.0 Patched (2012-06-04 r59517) >>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>> >>>>> locale: >>>>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >>>>> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >>>>> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >>>>> [7] LC_PAPER=C LC_NAME=C >>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >>>>> >>>>> attached base packages: >>>>> [1] stats graphics grDevices utils datasets methods >>>>> base >>>>> >>>>> other attached packages: >>>>> [1] Rsubread_1.6.3 >>>>> >>>>> >>>>> Regards, >>>>> Dan >>>>> >>>>> On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: >>>>>> hi, >>>>>> >>>>>> the computer room at my university where we do practicals on R & >>>> Bioconductor runs a 32bit linux distribution and when i tried to run >>>> the latest version of the Rsubread package (1.6.3) it crashes when >>>> calling the buildindex() function on a multifasta file with the yeast >>>> genome. this does *not* happen under a 64bit linux distribution. >>>>>> >>>>>> i have verified that installing the version before (1.4.4) on the >>>> current R 2.15 it also crashes (on the 32bit), but two versions >>>> before, the 1.1.1, it does *not* and it works smoothly on this 32bit >>>> linux distribution. >>>>>> >>>>>> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 >>>> where allChr.fa is the multifasta file with the yeast genome. >>>>>> >>>>>> so i can manage by now the problem by using the 1.1.1 version on R >>>> 2.15 for my teaching but i wonder whether there would be some easy >>>> solution for this, or even if it could be a symptom of something else >>>> that the Rsubread developers should worry about. i know that using a >>>> 32bit system nowadays is quite obsolete but this is what i got for >>>> teaching :( and i would be happy to let my students play with the >>>> latest version of Rsubread in the future. >>>>>> >>>>>> >>>>>> thanks!!! >>>>>> robert. >>>>>> >>>>>> ======================Rsubread 1.6.3 on R 2.15======================= >>>>>> >>>>>>> library(Rsubread) >>>>>>> sessionInfo() >>>>>> R version 2.15.0 (2012-03-30) >>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>> >>>>>> locale: >>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >>>>>> >>>>>> attached base packages: >>>>>> [1] stats graphics grDevices utils datasets methods base >>>>>> >>>>>> other attached packages: >>>>>> [1] Rsubread_1.6.3 >>>>>> >>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", >>>> memory=2500) >>>>>> >>>>>> Building a base-space index. >>>>>> Size of memory used=2500 MB >>>>>> Base name of the built index = subreadindex >>>>>> >>>>>> *** caught segfault *** >>>>>> address 0xdf670cc0, cause 'memory not mapped' >>>>>> >>>>>> Traceback: >>>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >>>> as.character(cmd), PACKAGE = "Rsubread") >>>>>> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", >>>> memory = 2500) >>>>>> >>>>>> Possible actions: >>>>>> 1: abort (with core dump, if enabled) >>>>>> 2: normal R exit >>>>>> 3: exit R without saving workspace >>>>>> 4: exit R saving workspace >>>>>> Selection: >>>>>> >>>>>> >>>>>> ======================Rsubread 1.1.1 on R 2.15======================= >>>>>> >>>>>>> library(Rsubread) >>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", >>>> memory=2500) >>>>>> >>>>>> Building the index in the base space. >>>>>> Size of memory requested=2500 MB >>>>>> Index base name = subreadindex >>>>>> INDEX ITEMS PER PARTITION = 275940352 >>>>>> >>>>>> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps >>>> completed=81.76%; time used=2.4s; rate=4111.8k >>>> bps/s; total=12m bps >>>>>> All the chromosome files are processed. >>>>>> | Dumping index >>>> [===========================================================>] >>>>>> Index subreadindex is successfully built. >>>>>>> sessionInfo() >>>>>> R version 2.15.0 (2012-03-30) >>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>> >>>>>> locale: >>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >>>>>> >>>>>> attached base packages: >>>>>> [1] stats graphics grDevices utils datasets methods base >>>>>> >>>>>> other attached packages: >>>>>> [1] Rsubread_1.1.1 >>>>>> >>>>>> _______________________________________________ >>>>>> Bioconductor mailing list >>>>>> Bioconductor at r-project.org >>>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>>> Search the archives: >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>>> >>>>> _______________________________________________ >>>>> Bioconductor mailing list >>>>> Bioconductor at r-project.org >>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>> Search the archives: >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>> >>>> >>>> ______________________________________________________________________ >>>> The information in this email is confidential and intended solely for >>>> the addressee. >>>> You must not disclose, forward, print or use it without the permission >>>> of the sender. >>>> ______________________________________________________________________ >>> >>> >>> >>> >> >> >> >> ______________________________________________________________________ >> The information in this email is confidential and intended solely for the addressee. >> You must not disclose, forward, print or use it without the permission of the sender. >> ______________________________________________________________________ > > ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:6}}
ADD REPLYlink written 5.5 years ago by Wei Shi2.7k
Dear Robert, Dan and Peter, We have made changes to a number of functions in the package to reduce the memory allocated to Rsubread by the operating system when it was loaded. The new version has been committed to both bioc release (Rsubread 1.6.4) and bioc devel (Rsubread 1.7.4). They should be available to you in a day or two. Also, the buildindex() function no longer needs the allocation of 1GB continuous memory region. But it will still consume at least 1GB of memory when it is running, no matter what the given value of the 'memory' parameter is. We have tested the new version on our 32-bit VM machine (it has 3GB of memory and the value of 'memory' parameter used by buildindex was 2500) and it solves all the reported problems, so we are pretty happy with it. I hope the new version works in your computers/laptops, but please do let us know if it doesn't. Sorry about the problems you have encountered. It's always a challenge to develop a R package with so much C code in it! Cheers, Wei On Jun 6, 2012, at 9:08 PM, Dan Du wrote: > Dear Wei, > > Here is a standard bioclite update, I think it is at the last step when > compiling Rsubread.so, the memory usage exceeds 5.5g, then system freeze > and I have to call it off. Same result when runing 'R CMD INSTALL > Rsubread_1.6.3.tar.gz' from shell, or manually compile all .c file and > run the last gcc statement. I guess there might just be a minimum ram > requirement somewhere higher than 6g... I will do some more poking when > I have time. > > 'gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o > aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o > exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o > gene-value-index.o hashtable.o index-builder.o input-files.o > processExons.o propmapped.o qualityScores.o readSummary.o > removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread > -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR' > > Also down there are the sessionInfo and full gcc version, please let me > know if you need more information. > > Regards, > Dan > -------------------------------------------------------------------- >> source('http://www.bioconductor.org/biocLite.R') >> biocLite('') > BioC_mirror: http://bioconductor.org > Using R version 2.15, BiocInstaller version 1.4.6. > Installing package(s) '' > Old packages: 'Rsubread' > Update all/some/none? [a/s/n]: a > trying URL > 'http://www.bioconductor.org/packages/2.10/bioc/src/contrib/Rsubread _1.6.3.tar.gz' > Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb) > opened URL > ================================================== > downloaded 20.9 Mb > > WARNING: ignoring environment value of R_HOME > * installing *source* package ?Rsubread? ... > ** libs > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c R_wrapper.c -o R_wrapper.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c SNP_calling.c -o SNP_calling.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c aligner.c -o aligner.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c atgcContent.c -o atgcContent.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c detectionCall.c -o detectionCall.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c detectionCallAnnotation.c -o detectionCallAnnotation.o > detectionCallAnnotation.c: In function ?calculateExonGCContent?: > detectionCallAnnotation.c:175: warning: ignoring return value of > ?fgets?, declared with attribute warn_unused_result > detectionCallAnnotation.c: In function ?calculateIRGCContent?: > detectionCallAnnotation.c:262: warning: ignoring return value of > ?fgets?, declared with attribute warn_unused_result > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c exon-algorithms.c -o exon-algorithms.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c exon-align.c -o exon-align.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c fullscan.c -o fullscan.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c gene-algorithms.c -o gene-algorithms.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c gene-value-index.c -o gene-value-index.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c hashtable.c -o hashtable.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c index-builder.c -o index-builder.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c input-files.c -o input-files.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c processExons.c -o processExons.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c propmapped.c -o propmapped.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c qualityScores.c -o qualityScores.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c readSummary.c -o readSummary.o > readSummary.c: In function ?readSummary?: > readSummary.c:122: warning: format ?%d? expects type ?int?, but argument > 5 has type ?long int? > readSummary.c:122: warning: format ?%d? expects type ?int?, but argument > 6 has type ?long int? > readSummary.c:39: warning: ignoring return value of ?getline?, declared > with attribute warn_unused_result > readSummary.c:52: warning: ignoring return value of ?getline?, declared > with attribute warn_unused_result > readSummary.c:55: warning: ignoring return value of ?getline?, declared > with attribute warn_unused_result > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c removeDuplicatedReads.c -o removeDuplicatedReads.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c sam2bed.c -o sam2bed.o > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > -O3 -pipe -g -c sorted-hashtable.c -o sorted-hashtable.o > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > declared but never defined > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > but never defined > gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o > aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o > exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o > gene-value-index.o hashtable.o index-builder.o input-files.o > processExons.o propmapped.o qualityScores.o readSummary.o > removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread > -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR > ^Cmake: *** Deleting file `Rsubread.so' > make: *** [Rsubread.so] Interrupt > ** R > ** inst > ** preparing package for lazy loading > ** help > *** installing help indices > ** building package indices > ** installing vignettes > ?Rsubread.Rnw? > ** testing if installed package can be loaded > Error in library.dynam(lib, package, package.lib) : > shared object ?Rsubread.so? not found > Error: loading failed > Execution halted > -------------------------------------------------------------------- >> sessionInfo() > R version 2.15.0 (2012-03-30) > Platform: x86_64-pc-linux-gnu (64-bit) > > locale: > [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C > [3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8 > [5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8 > [7] LC_PAPER=C LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > -------------------------------------------------------------------- > $ gcc -v > Using built-in specs. > Target: x86_64-linux-gnu > Configured with: ../src/configure -v --with-pkgversion='Ubuntu > 4.4.3-4ubuntu5.1' > --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs > --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr > --enable-shared --enable-multiarch --enable-linker-build-id > --with-system-zlib --libexecdir=/usr/lib --without-included-gettext > --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 > --program-suffix=-4.4 --enable-nls --enable-clocale=gnu > --enable-libstdcxx-debug --enable-plugin --enable-objc-gc > --disable-werror --with-arch-32=i486 --with-tune=generic > --enable-checking=release --build=x86_64-linux-gnu > --host=x86_64-linux-gnu --target=x86_64-linux-gnu > Thread model: posix > gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1) > -------------------------------------------------------------------- > > On Wed, 2012-06-06 at 20:10 +1000, Wei Shi wrote: >> Dear Dan, >> >> It didn't seem to be problem of requesting a continuous 1GB block in our >> investigation. We tracked the memory usage of buildindex() function when >> running it on yeast genome using a 32-bit VM, and found that the segfault >> happened right after a request of a few KB of memory was sent to the >> system when the memory parameter was set to 2500. However, the problem was >> gone when the memory parameter was changed to 1000. >> >> Removing highly repetitive 16 mers required a continuous 1GB block of >> memory, but this step was always executed successfully. This step also >> included in the old version of Rsubread (1.1.1), and it did not have >> problem there either. >> >> Could you please provide us your complete code for running your test and >> also session info? This will help us to diagnose what the problem could be >> because we couldn't reproduce what you saw from our end. >> >> For the compilation issue on your 64bit laptop, could you provide us more >> details as well, including the message output from gcc? >> >> Thanks, >> Wei >> >>> Dear Wei, >>> >>> Unfortunately reducing the memory parameter to 1000, still causes the >>> segfault. I guess with 3g ram limit on a 32bit system, there is still a >>> fat chance that you can not request a continuous 1g block. >>> >>> For that 64bit laptop, it is still strange about the 6g memory draining. >>> It is happing during the installation when compiling the shared library >>> Rsubread.so, not running the buildindex function. Btw, the gcc version >>> is 4.4.3. >>> >>> Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. >>> >>> Regards, >>> Dan >>> >>> On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: >>>> Dear Dan, >>>> >>>> It is probably because including genome sequences into the index slowed >>>> down your laptop. But I believe it should be alleviated if you give >>>> smaller values to the 'memory ' parameter of the buildindex() function. >>>> Also, the index building is an one-off operation, you do not need to >>>> redo it even when new releases come. >>>> >>>> For your 32-bit opensuse box, I guess the problem will be solved if you >>>> change the amount of memory requested to be 1000MB. >>>> >>>> Cheers, >>>> Wei >>>> >>>> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: >>>> >>>>> Hi Robert, >>>>> >>>>> I have been experiencing something else, possibly related to yours, >>>>> on a 64bit ubuntu laptop with 6g of ram. >>>>> >>>>> As I recall, when bumping to Bioc 2.10, the Rsubread installation kind >>>>> of ate all the memory, basically froze the system so I had to call it >>>>> off, yet building it on the server side turned out fine. So I think I >>>>> just accepted that the new version may be 'computationally heavy' thus >>>>> not suitable for a normal pc, though I did not find any mentioning of >>>>> this increased memory requirement in the NEWS file. >>>>> >>>>> So currently Rsubread stays at 1.4.4 on that pc, all subsequent >>>> versions >>>>> of Rsubread drain the memory in the same way when compiling >>>> Rsubread.so. >>>>> >>>>> Now I think I can confirm this on a 32-bit opensuse box, it did >>>>> successfully built, but when running the example code in the manual, >>>>> same segfault happens. >>>>> >>>>> >>>>>> library(Rsubread) >>>>>> ref <- system.file("extdata","reference.fa",package="Rsubread") >>>>>> path <- system.file("extdata",package="Rsubread") >>>>>> buildindex(basename=file.path(path,"reference_index"),reference=ref) >>>>> >>>>> Building a base-space index. >>>>> Size of memory used=3700 MB >>>>> Base name of the built index >>>>> = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index >>>>> >>>>> *** caught segfault *** >>>>> address 0xdf03ee80, cause 'memory not mapped' >>>>> >>>>> Traceback: >>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >>>>> as.character(cmd), PACKAGE = "Rsubread") >>>>> 2: buildindex(basename = file.path(path, "reference_index"), reference >>>>> = ref) >>>>> >>>>>> sessionInfo() >>>>> R version 2.15.0 Patched (2012-06-04 r59517) >>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>> >>>>> locale: >>>>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >>>>> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >>>>> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >>>>> [7] LC_PAPER=C LC_NAME=C >>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >>>>> >>>>> attached base packages: >>>>> [1] stats graphics grDevices utils datasets methods >>>>> base >>>>> >>>>> other attached packages: >>>>> [1] Rsubread_1.6.3 >>>>> >>>>> >>>>> Regards, >>>>> Dan >>>>> >>>>> On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: >>>>>> hi, >>>>>> >>>>>> the computer room at my university where we do practicals on R & >>>> Bioconductor runs a 32bit linux distribution and when i tried to run >>>> the latest version of the Rsubread package (1.6.3) it crashes when >>>> calling the buildindex() function on a multifasta file with the yeast >>>> genome. this does *not* happen under a 64bit linux distribution. >>>>>> >>>>>> i have verified that installing the version before (1.4.4) on the >>>> current R 2.15 it also crashes (on the 32bit), but two versions >>>> before, the 1.1.1, it does *not* and it works smoothly on this 32bit >>>> linux distribution. >>>>>> >>>>>> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 >>>> where allChr.fa is the multifasta file with the yeast genome. >>>>>> >>>>>> so i can manage by now the problem by using the 1.1.1 version on R >>>> 2.15 for my teaching but i wonder whether there would be some easy >>>> solution for this, or even if it could be a symptom of something else >>>> that the Rsubread developers should worry about. i know that using a >>>> 32bit system nowadays is quite obsolete but this is what i got for >>>> teaching :( and i would be happy to let my students play with the >>>> latest version of Rsubread in the future. >>>>>> >>>>>> >>>>>> thanks!!! >>>>>> robert. >>>>>> >>>>>> ======================Rsubread 1.6.3 on R 2.15======================= >>>>>> >>>>>>> library(Rsubread) >>>>>>> sessionInfo() >>>>>> R version 2.15.0 (2012-03-30) >>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>> >>>>>> locale: >>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >>>>>> >>>>>> attached base packages: >>>>>> [1] stats graphics grDevices utils datasets methods base >>>>>> >>>>>> other attached packages: >>>>>> [1] Rsubread_1.6.3 >>>>>> >>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", >>>> memory=2500) >>>>>> >>>>>> Building a base-space index. >>>>>> Size of memory used=2500 MB >>>>>> Base name of the built index = subreadindex >>>>>> >>>>>> *** caught segfault *** >>>>>> address 0xdf670cc0, cause 'memory not mapped' >>>>>> >>>>>> Traceback: >>>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >>>> as.character(cmd), PACKAGE = "Rsubread") >>>>>> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", >>>> memory = 2500) >>>>>> >>>>>> Possible actions: >>>>>> 1: abort (with core dump, if enabled) >>>>>> 2: normal R exit >>>>>> 3: exit R without saving workspace >>>>>> 4: exit R saving workspace >>>>>> Selection: >>>>>> >>>>>> >>>>>> ======================Rsubread 1.1.1 on R 2.15======================= >>>>>> >>>>>>> library(Rsubread) >>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", >>>> memory=2500) >>>>>> >>>>>> Building the index in the base space. >>>>>> Size of memory requested=2500 MB >>>>>> Index base name = subreadindex >>>>>> INDEX ITEMS PER PARTITION = 275940352 >>>>>> >>>>>> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps >>>> completed=81.76%; time used=2.4s; rate=4111.8k >>>> bps/s; total=12m bps >>>>>> All the chromosome files are processed. >>>>>> | Dumping index >>>> [===========================================================>] >>>>>> Index subreadindex is successfully built. >>>>>>> sessionInfo() >>>>>> R version 2.15.0 (2012-03-30) >>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>> >>>>>> locale: >>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >>>>>> >>>>>> attached base packages: >>>>>> [1] stats graphics grDevices utils datasets methods base >>>>>> >>>>>> other attached packages: >>>>>> [1] Rsubread_1.1.1 >>>>>> >>>>>> _______________________________________________ >>>>>> Bioconductor mailing list >>>>>> Bioconductor at r-project.org >>>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>>> Search the archives: >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>>> >>>>> _______________________________________________ >>>>> Bioconductor mailing list >>>>> Bioconductor at r-project.org >>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>> Search the archives: >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>> >>>> >>>> ______________________________________________________________________ >>>> The information in this email is confidential and intended solely for >>>> the addressee. >>>> You must not disclose, forward, print or use it without the permission >>>> of the sender. >>>> ______________________________________________________________________ >>> >>> >>> >>> >> >> >> >> ______________________________________________________________________ >> The information in this email is confidential and intended solely for the addressee. >> You must not disclose, forward, print or use it without the permission of the sender. >> ______________________________________________________________________ > > ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:6}}
ADD REPLYlink written 5.4 years ago by Wei Shi2.7k
Dear Wei, Good work, memory freed. Just checkout the devel version, package built and installed successfully with no hiccup, test codes runs fine. Regards, Dan On Fri, 2012-06-08 at 12:10 +1000, Wei Shi wrote: > Dear Robert, Dan and Peter, > > We have made changes to a number of functions in the package to reduce the memory allocated to Rsubread by the operating system when it was loaded. The new version has been committed to both bioc release (Rsubread 1.6.4) and bioc devel (Rsubread 1.7.4). They should be available to you in a day or two. > > Also, the buildindex() function no longer needs the allocation of 1GB continuous memory region. But it will still consume at least 1GB of memory when it is running, no matter what the given value of the 'memory' parameter is. > > We have tested the new version on our 32-bit VM machine (it has 3GB of memory and the value of 'memory' parameter used by buildindex was 2500) and it solves all the reported problems, so we are pretty happy with it. I hope the new version works in your computers/laptops, but please do let us know if it doesn't. > > Sorry about the problems you have encountered. It's always a challenge to develop a R package with so much C code in it! > > Cheers, > Wei > > > > On Jun 6, 2012, at 9:08 PM, Dan Du wrote: > > > Dear Wei, > > > > Here is a standard bioclite update, I think it is at the last step when > > compiling Rsubread.so, the memory usage exceeds 5.5g, then system freeze > > and I have to call it off. Same result when runing 'R CMD INSTALL > > Rsubread_1.6.3.tar.gz' from shell, or manually compile all .c file and > > run the last gcc statement. I guess there might just be a minimum ram > > requirement somewhere higher than 6g... I will do some more poking when > > I have time. > > > > 'gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o > > aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o > > exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o > > gene-value-index.o hashtable.o index-builder.o input-files.o > > processExons.o propmapped.o qualityScores.o readSummary.o > > removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread > > -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR' > > > > Also down there are the sessionInfo and full gcc version, please let me > > know if you need more information. > > > > Regards, > > Dan > > -------------------------------------------------------------------- > >> source('http://www.bioconductor.org/biocLite.R') > >> biocLite('') > > BioC_mirror: http://bioconductor.org > > Using R version 2.15, BiocInstaller version 1.4.6. > > Installing package(s) '' > > Old packages: 'Rsubread' > > Update all/some/none? [a/s/n]: a > > trying URL > > 'http://www.bioconductor.org/packages/2.10/bioc/src/contrib/Rsubre ad_1.6.3.tar.gz' > > Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb) > > opened URL > > ================================================== > > downloaded 20.9 Mb > > > > WARNING: ignoring environment value of R_HOME > > * installing *source* package ?Rsubread? ... > > ** libs > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c R_wrapper.c -o R_wrapper.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c SNP_calling.c -o SNP_calling.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c aligner.c -o aligner.o > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c atgcContent.c -o atgcContent.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c detectionCall.c -o detectionCall.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c detectionCallAnnotation.c -o detectionCallAnnotation.o > > detectionCallAnnotation.c: In function ?calculateExonGCContent?: > > detectionCallAnnotation.c:175: warning: ignoring return value of > > ?fgets?, declared with attribute warn_unused_result > > detectionCallAnnotation.c: In function ?calculateIRGCContent?: > > detectionCallAnnotation.c:262: warning: ignoring return value of > > ?fgets?, declared with attribute warn_unused_result > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c exon-algorithms.c -o exon-algorithms.o > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c exon-align.c -o exon-align.o > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c fullscan.c -o fullscan.o > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c gene-algorithms.c -o gene-algorithms.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c gene-value-index.c -o gene-value-index.o > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c hashtable.c -o hashtable.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c index-builder.c -o index-builder.o > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c input-files.c -o input-files.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c processExons.c -o processExons.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c propmapped.c -o propmapped.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c qualityScores.c -o qualityScores.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c readSummary.c -o readSummary.o > > readSummary.c: In function ?readSummary?: > > readSummary.c:122: warning: format ?%d? expects type ?int?, but argument > > 5 has type ?long int? > > readSummary.c:122: warning: format ?%d? expects type ?int?, but argument > > 6 has type ?long int? > > readSummary.c:39: warning: ignoring return value of ?getline?, declared > > with attribute warn_unused_result > > readSummary.c:52: warning: ignoring return value of ?getline?, declared > > with attribute warn_unused_result > > readSummary.c:55: warning: ignoring return value of ?getline?, declared > > with attribute warn_unused_result > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c removeDuplicatedReads.c -o removeDuplicatedReads.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c sam2bed.c -o sam2bed.o > > gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic > > -O3 -pipe -g -c sorted-hashtable.c -o sorted-hashtable.o > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? > > declared but never defined > > gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared > > but never defined > > gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o > > aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o > > exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o > > gene-value-index.o hashtable.o index-builder.o input-files.o > > processExons.o propmapped.o qualityScores.o readSummary.o > > removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread > > -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR > > ^Cmake: *** Deleting file `Rsubread.so' > > make: *** [Rsubread.so] Interrupt > > ** R > > ** inst > > ** preparing package for lazy loading > > ** help > > *** installing help indices > > ** building package indices > > ** installing vignettes > > ?Rsubread.Rnw? > > ** testing if installed package can be loaded > > Error in library.dynam(lib, package, package.lib) : > > shared object ?Rsubread.so? not found > > Error: loading failed > > Execution halted > > -------------------------------------------------------------------- > >> sessionInfo() > > R version 2.15.0 (2012-03-30) > > Platform: x86_64-pc-linux-gnu (64-bit) > > > > locale: > > [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C > > [3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8 > > [5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8 > > [7] LC_PAPER=C LC_NAME=C > > [9] LC_ADDRESS=C LC_TELEPHONE=C > > [11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C > > > > attached base packages: > > [1] stats graphics grDevices utils datasets methods base > > -------------------------------------------------------------------- > > $ gcc -v > > Using built-in specs. > > Target: x86_64-linux-gnu > > Configured with: ../src/configure -v --with-pkgversion='Ubuntu > > 4.4.3-4ubuntu5.1' > > --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs > > --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr > > --enable-shared --enable-multiarch --enable-linker-build-id > > --with-system-zlib --libexecdir=/usr/lib --without-included- gettext > > --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 > > --program-suffix=-4.4 --enable-nls --enable-clocale=gnu > > --enable-libstdcxx-debug --enable-plugin --enable-objc-gc > > --disable-werror --with-arch-32=i486 --with-tune=generic > > --enable-checking=release --build=x86_64-linux-gnu > > --host=x86_64-linux-gnu --target=x86_64-linux-gnu > > Thread model: posix > > gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1) > > -------------------------------------------------------------------- > > > > On Wed, 2012-06-06 at 20:10 +1000, Wei Shi wrote: > >> Dear Dan, > >> > >> It didn't seem to be problem of requesting a continuous 1GB block in our > >> investigation. We tracked the memory usage of buildindex() function when > >> running it on yeast genome using a 32-bit VM, and found that the segfault > >> happened right after a request of a few KB of memory was sent to the > >> system when the memory parameter was set to 2500. However, the problem was > >> gone when the memory parameter was changed to 1000. > >> > >> Removing highly repetitive 16 mers required a continuous 1GB block of > >> memory, but this step was always executed successfully. This step also > >> included in the old version of Rsubread (1.1.1), and it did not have > >> problem there either. > >> > >> Could you please provide us your complete code for running your test and > >> also session info? This will help us to diagnose what the problem could be > >> because we couldn't reproduce what you saw from our end. > >> > >> For the compilation issue on your 64bit laptop, could you provide us more > >> details as well, including the message output from gcc? > >> > >> Thanks, > >> Wei > >> > >>> Dear Wei, > >>> > >>> Unfortunately reducing the memory parameter to 1000, still causes the > >>> segfault. I guess with 3g ram limit on a 32bit system, there is still a > >>> fat chance that you can not request a continuous 1g block. > >>> > >>> For that 64bit laptop, it is still strange about the 6g memory draining. > >>> It is happing during the installation when compiling the shared library > >>> Rsubread.so, not running the buildindex function. Btw, the gcc version > >>> is 4.4.3. > >>> > >>> Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. > >>> > >>> Regards, > >>> Dan > >>> > >>> On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: > >>>> Dear Dan, > >>>> > >>>> It is probably because including genome sequences into the index slowed > >>>> down your laptop. But I believe it should be alleviated if you give > >>>> smaller values to the 'memory ' parameter of the buildindex() function. > >>>> Also, the index building is an one-off operation, you do not need to > >>>> redo it even when new releases come. > >>>> > >>>> For your 32-bit opensuse box, I guess the problem will be solved if you > >>>> change the amount of memory requested to be 1000MB. > >>>> > >>>> Cheers, > >>>> Wei > >>>> > >>>> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: > >>>> > >>>>> Hi Robert, > >>>>> > >>>>> I have been experiencing something else, possibly related to yours, > >>>>> on a 64bit ubuntu laptop with 6g of ram. > >>>>> > >>>>> As I recall, when bumping to Bioc 2.10, the Rsubread installation kind > >>>>> of ate all the memory, basically froze the system so I had to call it > >>>>> off, yet building it on the server side turned out fine. So I think I > >>>>> just accepted that the new version may be 'computationally heavy' thus > >>>>> not suitable for a normal pc, though I did not find any mentioning of > >>>>> this increased memory requirement in the NEWS file. > >>>>> > >>>>> So currently Rsubread stays at 1.4.4 on that pc, all subsequent > >>>> versions > >>>>> of Rsubread drain the memory in the same way when compiling > >>>> Rsubread.so. > >>>>> > >>>>> Now I think I can confirm this on a 32-bit opensuse box, it did > >>>>> successfully built, but when running the example code in the manual, > >>>>> same segfault happens. > >>>>> > >>>>> > >>>>>> library(Rsubread) > >>>>>> ref <- system.file("extdata","reference.fa",package="Rsubread") > >>>>>> path <- system.file("extdata",package="Rsubread") > >>>>>> buildindex(basename=file.path(path,"reference_index"),reference=ref) > >>>>> > >>>>> Building a base-space index. > >>>>> Size of memory used=3700 MB > >>>>> Base name of the built index > >>>>> = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index > >>>>> > >>>>> *** caught segfault *** > >>>>> address 0xdf03ee80, cause 'memory not mapped' > >>>>> > >>>>> Traceback: > >>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = > >>>>> as.character(cmd), PACKAGE = "Rsubread") > >>>>> 2: buildindex(basename = file.path(path, "reference_index"), reference > >>>>> = ref) > >>>>> > >>>>>> sessionInfo() > >>>>> R version 2.15.0 Patched (2012-06-04 r59517) > >>>>> Platform: i686-pc-linux-gnu (32-bit) > >>>>> > >>>>> locale: > >>>>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > >>>>> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > >>>>> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > >>>>> [7] LC_PAPER=C LC_NAME=C > >>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C > >>>>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > >>>>> > >>>>> attached base packages: > >>>>> [1] stats graphics grDevices utils datasets methods > >>>>> base > >>>>> > >>>>> other attached packages: > >>>>> [1] Rsubread_1.6.3 > >>>>> > >>>>> > >>>>> Regards, > >>>>> Dan > >>>>> > >>>>> On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: > >>>>>> hi, > >>>>>> > >>>>>> the computer room at my university where we do practicals on R & > >>>> Bioconductor runs a 32bit linux distribution and when i tried to run > >>>> the latest version of the Rsubread package (1.6.3) it crashes when > >>>> calling the buildindex() function on a multifasta file with the yeast > >>>> genome. this does *not* happen under a 64bit linux distribution. > >>>>>> > >>>>>> i have verified that installing the version before (1.4.4) on the > >>>> current R 2.15 it also crashes (on the 32bit), but two versions > >>>> before, the 1.1.1, it does *not* and it works smoothly on this 32bit > >>>> linux distribution. > >>>>>> > >>>>>> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 > >>>> where allChr.fa is the multifasta file with the yeast genome. > >>>>>> > >>>>>> so i can manage by now the problem by using the 1.1.1 version on R > >>>> 2.15 for my teaching but i wonder whether there would be some easy > >>>> solution for this, or even if it could be a symptom of something else > >>>> that the Rsubread developers should worry about. i know that using a > >>>> 32bit system nowadays is quite obsolete but this is what i got for > >>>> teaching :( and i would be happy to let my students play with the > >>>> latest version of Rsubread in the future. > >>>>>> > >>>>>> > >>>>>> thanks!!! > >>>>>> robert. > >>>>>> > >>>>>> ======================Rsubread 1.6.3 on R 2.15======================= > >>>>>> > >>>>>>> library(Rsubread) > >>>>>>> sessionInfo() > >>>>>> R version 2.15.0 (2012-03-30) > >>>>>> Platform: i686-pc-linux-gnu (32-bit) > >>>>>> > >>>>>> locale: > >>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > >>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > >>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > >>>>>> [7] LC_PAPER=C LC_NAME=C > >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C > >>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > >>>>>> > >>>>>> attached base packages: > >>>>>> [1] stats graphics grDevices utils datasets methods base > >>>>>> > >>>>>> other attached packages: > >>>>>> [1] Rsubread_1.6.3 > >>>>>> > >>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", > >>>> memory=2500) > >>>>>> > >>>>>> Building a base-space index. > >>>>>> Size of memory used=2500 MB > >>>>>> Base name of the built index = subreadindex > >>>>>> > >>>>>> *** caught segfault *** > >>>>>> address 0xdf670cc0, cause 'memory not mapped' > >>>>>> > >>>>>> Traceback: > >>>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = > >>>> as.character(cmd), PACKAGE = "Rsubread") > >>>>>> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", > >>>> memory = 2500) > >>>>>> > >>>>>> Possible actions: > >>>>>> 1: abort (with core dump, if enabled) > >>>>>> 2: normal R exit > >>>>>> 3: exit R without saving workspace > >>>>>> 4: exit R saving workspace > >>>>>> Selection: > >>>>>> > >>>>>> > >>>>>> ======================Rsubread 1.1.1 on R 2.15======================= > >>>>>> > >>>>>>> library(Rsubread) > >>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", > >>>> memory=2500) > >>>>>> > >>>>>> Building the index in the base space. > >>>>>> Size of memory requested=2500 MB > >>>>>> Index base name = subreadindex > >>>>>> INDEX ITEMS PER PARTITION = 275940352 > >>>>>> > >>>>>> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps > >>>> completed=81.76%; time used=2.4s; rate=4111.8k > >>>> bps/s; total=12m bps > >>>>>> All the chromosome files are processed. > >>>>>> | Dumping index > >>>> [===========================================================>] > >>>>>> Index subreadindex is successfully built. > >>>>>>> sessionInfo() > >>>>>> R version 2.15.0 (2012-03-30) > >>>>>> Platform: i686-pc-linux-gnu (32-bit) > >>>>>> > >>>>>> locale: > >>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > >>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > >>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > >>>>>> [7] LC_PAPER=C LC_NAME=C > >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C > >>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > >>>>>> > >>>>>> attached base packages: > >>>>>> [1] stats graphics grDevices utils datasets methods base > >>>>>> > >>>>>> other attached packages: > >>>>>> [1] Rsubread_1.1.1 > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Bioconductor mailing list > >>>>>> Bioconductor at r-project.org > >>>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor > >>>>>> Search the archives: > >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor > >>>>> > >>>>> _______________________________________________ > >>>>> Bioconductor mailing list > >>>>> Bioconductor at r-project.org > >>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor > >>>>> Search the archives: > >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor > >>>> > >>>> > >>>> ______________________________________________________________________ > >>>> The information in this email is confidential and intended solely for > >>>> the addressee. > >>>> You must not disclose, forward, print or use it without the permission > >>>> of the sender. > >>>> ______________________________________________________________________ > >>> > >>> > >>> > >>> > >> > >> > >> > >> ______________________________________________________________________ > >> The information in this email is confidential and intended solely for the addressee. > >> You must not disclose, forward, print or use it without the permission of the sender. > >> ______________________________________________________________________ > > > > > > > ______________________________________________________________________ > The information in this email is confidential and inte...{{dropped:5}}
ADD REPLYlink written 5.4 years ago by Dan Du210
Dear Dan, Thanks for letting us know. That's great! Cheers, Wei On Jun 8, 2012, at 4:09 PM, Dan Du wrote: > Dear Wei, > > Good work, memory freed. Just checkout the devel version, package built > and installed successfully with no hiccup, test codes runs fine. > > Regards, > Dan > > On Fri, 2012-06-08 at 12:10 +1000, Wei Shi wrote: >> Dear Robert, Dan and Peter, >> >> We have made changes to a number of functions in the package to reduce the memory allocated to Rsubread by the operating system when it was loaded. The new version has been committed to both bioc release (Rsubread 1.6.4) and bioc devel (Rsubread 1.7.4). They should be available to you in a day or two. >> >> Also, the buildindex() function no longer needs the allocation of 1GB continuous memory region. But it will still consume at least 1GB of memory when it is running, no matter what the given value of the 'memory' parameter is. >> >> We have tested the new version on our 32-bit VM machine (it has 3GB of memory and the value of 'memory' parameter used by buildindex was 2500) and it solves all the reported problems, so we are pretty happy with it. I hope the new version works in your computers/laptops, but please do let us know if it doesn't. >> >> Sorry about the problems you have encountered. It's always a challenge to develop a R package with so much C code in it! >> >> Cheers, >> Wei >> >> >> >> On Jun 6, 2012, at 9:08 PM, Dan Du wrote: >> >>> Dear Wei, >>> >>> Here is a standard bioclite update, I think it is at the last step when >>> compiling Rsubread.so, the memory usage exceeds 5.5g, then system freeze >>> and I have to call it off. Same result when runing 'R CMD INSTALL >>> Rsubread_1.6.3.tar.gz' from shell, or manually compile all .c file and >>> run the last gcc statement. I guess there might just be a minimum ram >>> requirement somewhere higher than 6g... I will do some more poking when >>> I have time. >>> >>> 'gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o >>> aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o >>> exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o >>> gene-value-index.o hashtable.o index-builder.o input-files.o >>> processExons.o propmapped.o qualityScores.o readSummary.o >>> removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread >>> -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR' >>> >>> Also down there are the sessionInfo and full gcc version, please let me >>> know if you need more information. >>> >>> Regards, >>> Dan >>> -------------------------------------------------------------------- >>>> source('http://www.bioconductor.org/biocLite.R') >>>> biocLite('') >>> BioC_mirror: http://bioconductor.org >>> Using R version 2.15, BiocInstaller version 1.4.6. >>> Installing package(s) '' >>> Old packages: 'Rsubread' >>> Update all/some/none? [a/s/n]: a >>> trying URL >>> 'http://www.bioconductor.org/packages/2.10/bioc/src/contrib/Rsubre ad_1.6.3.tar.gz' >>> Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb) >>> opened URL >>> ================================================== >>> downloaded 20.9 Mb >>> >>> WARNING: ignoring environment value of R_HOME >>> * installing *source* package ?Rsubread? ... >>> ** libs >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c R_wrapper.c -o R_wrapper.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c SNP_calling.c -o SNP_calling.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c aligner.c -o aligner.o >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c atgcContent.c -o atgcContent.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c detectionCall.c -o detectionCall.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c detectionCallAnnotation.c -o detectionCallAnnotation.o >>> detectionCallAnnotation.c: In function ?calculateExonGCContent?: >>> detectionCallAnnotation.c:175: warning: ignoring return value of >>> ?fgets?, declared with attribute warn_unused_result >>> detectionCallAnnotation.c: In function ?calculateIRGCContent?: >>> detectionCallAnnotation.c:262: warning: ignoring return value of >>> ?fgets?, declared with attribute warn_unused_result >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c exon-algorithms.c -o exon-algorithms.o >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c exon-align.c -o exon-align.o >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c fullscan.c -o fullscan.o >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c gene-algorithms.c -o gene-algorithms.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c gene-value-index.c -o gene-value-index.o >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c hashtable.c -o hashtable.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c index-builder.c -o index-builder.o >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c input-files.c -o input-files.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c processExons.c -o processExons.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c propmapped.c -o propmapped.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c qualityScores.c -o qualityScores.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c readSummary.c -o readSummary.o >>> readSummary.c: In function ?readSummary?: >>> readSummary.c:122: warning: format ?%d? expects type ?int?, but argument >>> 5 has type ?long int? >>> readSummary.c:122: warning: format ?%d? expects type ?int?, but argument >>> 6 has type ?long int? >>> readSummary.c:39: warning: ignoring return value of ?getline?, declared >>> with attribute warn_unused_result >>> readSummary.c:52: warning: ignoring return value of ?getline?, declared >>> with attribute warn_unused_result >>> readSummary.c:55: warning: ignoring return value of ?getline?, declared >>> with attribute warn_unused_result >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c removeDuplicatedReads.c -o removeDuplicatedReads.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c sam2bed.c -o sam2bed.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >>> -O3 -pipe -g -c sorted-hashtable.c -o sorted-hashtable.o >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >>> declared but never defined >>> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >>> but never defined >>> gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o >>> aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o >>> exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o >>> gene-value-index.o hashtable.o index-builder.o input-files.o >>> processExons.o propmapped.o qualityScores.o readSummary.o >>> removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread >>> -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR >>> ^Cmake: *** Deleting file `Rsubread.so' >>> make: *** [Rsubread.so] Interrupt >>> ** R >>> ** inst >>> ** preparing package for lazy loading >>> ** help >>> *** installing help indices >>> ** building package indices >>> ** installing vignettes >>> ?Rsubread.Rnw? >>> ** testing if installed package can be loaded >>> Error in library.dynam(lib, package, package.lib) : >>> shared object ?Rsubread.so? not found >>> Error: loading failed >>> Execution halted >>> -------------------------------------------------------------------- >>>> sessionInfo() >>> R version 2.15.0 (2012-03-30) >>> Platform: x86_64-pc-linux-gnu (64-bit) >>> >>> locale: >>> [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C >>> [3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8 >>> [5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8 >>> [7] LC_PAPER=C LC_NAME=C >>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>> [11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C >>> >>> attached base packages: >>> [1] stats graphics grDevices utils datasets methods base >>> -------------------------------------------------------------------- >>> $ gcc -v >>> Using built-in specs. >>> Target: x86_64-linux-gnu >>> Configured with: ../src/configure -v --with-pkgversion='Ubuntu >>> 4.4.3-4ubuntu5.1' >>> --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs >>> --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr >>> --enable-shared --enable-multiarch --enable-linker-build-id >>> --with-system-zlib --libexecdir=/usr/lib --without-included- gettext >>> --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 >>> --program-suffix=-4.4 --enable-nls --enable-clocale=gnu >>> --enable-libstdcxx-debug --enable-plugin --enable-objc-gc >>> --disable-werror --with-arch-32=i486 --with-tune=generic >>> --enable-checking=release --build=x86_64-linux-gnu >>> --host=x86_64-linux-gnu --target=x86_64-linux-gnu >>> Thread model: posix >>> gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1) >>> -------------------------------------------------------------------- >>> >>> On Wed, 2012-06-06 at 20:10 +1000, Wei Shi wrote: >>>> Dear Dan, >>>> >>>> It didn't seem to be problem of requesting a continuous 1GB block in our >>>> investigation. We tracked the memory usage of buildindex() function when >>>> running it on yeast genome using a 32-bit VM, and found that the segfault >>>> happened right after a request of a few KB of memory was sent to the >>>> system when the memory parameter was set to 2500. However, the problem was >>>> gone when the memory parameter was changed to 1000. >>>> >>>> Removing highly repetitive 16 mers required a continuous 1GB block of >>>> memory, but this step was always executed successfully. This step also >>>> included in the old version of Rsubread (1.1.1), and it did not have >>>> problem there either. >>>> >>>> Could you please provide us your complete code for running your test and >>>> also session info? This will help us to diagnose what the problem could be >>>> because we couldn't reproduce what you saw from our end. >>>> >>>> For the compilation issue on your 64bit laptop, could you provide us more >>>> details as well, including the message output from gcc? >>>> >>>> Thanks, >>>> Wei >>>> >>>>> Dear Wei, >>>>> >>>>> Unfortunately reducing the memory parameter to 1000, still causes the >>>>> segfault. I guess with 3g ram limit on a 32bit system, there is still a >>>>> fat chance that you can not request a continuous 1g block. >>>>> >>>>> For that 64bit laptop, it is still strange about the 6g memory draining. >>>>> It is happing during the installation when compiling the shared library >>>>> Rsubread.so, not running the buildindex function. Btw, the gcc version >>>>> is 4.4.3. >>>>> >>>>> Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. >>>>> >>>>> Regards, >>>>> Dan >>>>> >>>>> On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: >>>>>> Dear Dan, >>>>>> >>>>>> It is probably because including genome sequences into the index slowed >>>>>> down your laptop. But I believe it should be alleviated if you give >>>>>> smaller values to the 'memory ' parameter of the buildindex() function. >>>>>> Also, the index building is an one-off operation, you do not need to >>>>>> redo it even when new releases come. >>>>>> >>>>>> For your 32-bit opensuse box, I guess the problem will be solved if you >>>>>> change the amount of memory requested to be 1000MB. >>>>>> >>>>>> Cheers, >>>>>> Wei >>>>>> >>>>>> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: >>>>>> >>>>>>> Hi Robert, >>>>>>> >>>>>>> I have been experiencing something else, possibly related to yours, >>>>>>> on a 64bit ubuntu laptop with 6g of ram. >>>>>>> >>>>>>> As I recall, when bumping to Bioc 2.10, the Rsubread installation kind >>>>>>> of ate all the memory, basically froze the system so I had to call it >>>>>>> off, yet building it on the server side turned out fine. So I think I >>>>>>> just accepted that the new version may be 'computationally heavy' thus >>>>>>> not suitable for a normal pc, though I did not find any mentioning of >>>>>>> this increased memory requirement in the NEWS file. >>>>>>> >>>>>>> So currently Rsubread stays at 1.4.4 on that pc, all subsequent >>>>>> versions >>>>>>> of Rsubread drain the memory in the same way when compiling >>>>>> Rsubread.so. >>>>>>> >>>>>>> Now I think I can confirm this on a 32-bit opensuse box, it did >>>>>>> successfully built, but when running the example code in the manual, >>>>>>> same segfault happens. >>>>>>> >>>>>>> >>>>>>>> library(Rsubread) >>>>>>>> ref <- system.file("extdata","reference.fa",package="Rsubread") >>>>>>>> path <- system.file("extdata",package="Rsubread") >>>>>>>> buildindex(basename=file.path(path,"reference_index"),reference=ref) >>>>>>> >>>>>>> Building a base-space index. >>>>>>> Size of memory used=3700 MB >>>>>>> Base name of the built index >>>>>>> = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index >>>>>>> >>>>>>> *** caught segfault *** >>>>>>> address 0xdf03ee80, cause 'memory not mapped' >>>>>>> >>>>>>> Traceback: >>>>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >>>>>>> as.character(cmd), PACKAGE = "Rsubread") >>>>>>> 2: buildindex(basename = file.path(path, "reference_index"), reference >>>>>>> = ref) >>>>>>> >>>>>>>> sessionInfo() >>>>>>> R version 2.15.0 Patched (2012-06-04 r59517) >>>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>>> >>>>>>> locale: >>>>>>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >>>>>>> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >>>>>>> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >>>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >>>>>>> >>>>>>> attached base packages: >>>>>>> [1] stats graphics grDevices utils datasets methods >>>>>>> base >>>>>>> >>>>>>> other attached packages: >>>>>>> [1] Rsubread_1.6.3 >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Dan >>>>>>> >>>>>>> On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: >>>>>>>> hi, >>>>>>>> >>>>>>>> the computer room at my university where we do practicals on R & >>>>>> Bioconductor runs a 32bit linux distribution and when i tried to run >>>>>> the latest version of the Rsubread package (1.6.3) it crashes when >>>>>> calling the buildindex() function on a multifasta file with the yeast >>>>>> genome. this does *not* happen under a 64bit linux distribution. >>>>>>>> >>>>>>>> i have verified that installing the version before (1.4.4) on the >>>>>> current R 2.15 it also crashes (on the 32bit), but two versions >>>>>> before, the 1.1.1, it does *not* and it works smoothly on this 32bit >>>>>> linux distribution. >>>>>>>> >>>>>>>> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 >>>>>> where allChr.fa is the multifasta file with the yeast genome. >>>>>>>> >>>>>>>> so i can manage by now the problem by using the 1.1.1 version on R >>>>>> 2.15 for my teaching but i wonder whether there would be some easy >>>>>> solution for this, or even if it could be a symptom of something else >>>>>> that the Rsubread developers should worry about. i know that using a >>>>>> 32bit system nowadays is quite obsolete but this is what i got for >>>>>> teaching :( and i would be happy to let my students play with the >>>>>> latest version of Rsubread in the future. >>>>>>>> >>>>>>>> >>>>>>>> thanks!!! >>>>>>>> robert. >>>>>>>> >>>>>>>> ======================Rsubread 1.6.3 on R 2.15======================= >>>>>>>> >>>>>>>>> library(Rsubread) >>>>>>>>> sessionInfo() >>>>>>>> R version 2.15.0 (2012-03-30) >>>>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>>>> >>>>>>>> locale: >>>>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >>>>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >>>>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >>>>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >>>>>>>> >>>>>>>> attached base packages: >>>>>>>> [1] stats graphics grDevices utils datasets methods base >>>>>>>> >>>>>>>> other attached packages: >>>>>>>> [1] Rsubread_1.6.3 >>>>>>>> >>>>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", >>>>>> memory=2500) >>>>>>>> >>>>>>>> Building a base-space index. >>>>>>>> Size of memory used=2500 MB >>>>>>>> Base name of the built index = subreadindex >>>>>>>> >>>>>>>> *** caught segfault *** >>>>>>>> address 0xdf670cc0, cause 'memory not mapped' >>>>>>>> >>>>>>>> Traceback: >>>>>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >>>>>> as.character(cmd), PACKAGE = "Rsubread") >>>>>>>> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", >>>>>> memory = 2500) >>>>>>>> >>>>>>>> Possible actions: >>>>>>>> 1: abort (with core dump, if enabled) >>>>>>>> 2: normal R exit >>>>>>>> 3: exit R without saving workspace >>>>>>>> 4: exit R saving workspace >>>>>>>> Selection: >>>>>>>> >>>>>>>> >>>>>>>> ======================Rsubread 1.1.1 on R 2.15======================= >>>>>>>> >>>>>>>>> library(Rsubread) >>>>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", >>>>>> memory=2500) >>>>>>>> >>>>>>>> Building the index in the base space. >>>>>>>> Size of memory requested=2500 MB >>>>>>>> Index base name = subreadindex >>>>>>>> INDEX ITEMS PER PARTITION = 275940352 >>>>>>>> >>>>>>>> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps >>>>>> completed=81.76%; time used=2.4s; rate=4111.8k >>>>>> bps/s; total=12m bps >>>>>>>> All the chromosome files are processed. >>>>>>>> | Dumping index >>>>>> [===========================================================>] >>>>>>>> Index subreadindex is successfully built. >>>>>>>>> sessionInfo() >>>>>>>> R version 2.15.0 (2012-03-30) >>>>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>>>> >>>>>>>> locale: >>>>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >>>>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >>>>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >>>>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >>>>>>>> >>>>>>>> attached base packages: >>>>>>>> [1] stats graphics grDevices utils datasets methods base >>>>>>>> >>>>>>>> other attached packages: >>>>>>>> [1] Rsubread_1.1.1 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Bioconductor mailing list >>>>>>>> Bioconductor at r-project.org >>>>>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>>>>> Search the archives: >>>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Bioconductor mailing list >>>>>>> Bioconductor at r-project.org >>>>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>>>> Search the archives: >>>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>>>> >>>>>> >>>>>> ______________________________________________________________________ >>>>>> The information in this email is confidential and intended solely for >>>>>> the addressee. >>>>>> You must not disclose, forward, print or use it without the permission >>>>>> of the sender. >>>>>> ______________________________________________________________________ >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> ______________________________________________________________________ >>>> The information in this email is confidential and intended solely for the addressee. >>>> You must not disclose, forward, print or use it without the permission of the sender. >>>> ______________________________________________________________________ >>> >>> >> >> >> ______________________________________________________________________ >> The information in this email is confidential and intended solely for the addressee. >> You must not disclose, forward, print or use it without the permission of the sender. >> ______________________________________________________________________ > > ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:6}}
ADD REPLYlink written 5.4 years ago by Wei Shi2.7k
Dear Wei, it also works on my side, the 32 bit machines do not crash anymore. thanks for solving this so quickly! robert. On 06/08/2012 04:10 AM, Wei Shi wrote: > Dear Robert, Dan and Peter, > > We have made changes to a number of functions in the package to reduce the memory allocated to Rsubread by the operating system when it was loaded. The new version has been committed to both bioc release (Rsubread 1.6.4) and bioc devel (Rsubread 1.7.4). They should be available to you in a day or two. > > Also, the buildindex() function no longer needs the allocation of 1GB continuous memory region. But it will still consume at least 1GB of memory when it is running, no matter what the given value of the 'memory' parameter is. > > We have tested the new version on our 32-bit VM machine (it has 3GB of memory and the value of 'memory' parameter used by buildindex was 2500) and it solves all the reported problems, so we are pretty happy with it. I hope the new version works in your computers/laptops, but please do let us know if it doesn't. > > Sorry about the problems you have encountered. It's always a challenge to develop a R package with so much C code in it! > > Cheers, > Wei > > > > On Jun 6, 2012, at 9:08 PM, Dan Du wrote: > >> Dear Wei, >> >> Here is a standard bioclite update, I think it is at the last step when >> compiling Rsubread.so, the memory usage exceeds 5.5g, then system freeze >> and I have to call it off. Same result when runing 'R CMD INSTALL >> Rsubread_1.6.3.tar.gz' from shell, or manually compile all .c file and >> run the last gcc statement. I guess there might just be a minimum ram >> requirement somewhere higher than 6g... I will do some more poking when >> I have time. >> >> 'gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o >> aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o >> exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o >> gene-value-index.o hashtable.o index-builder.o input-files.o >> processExons.o propmapped.o qualityScores.o readSummary.o >> removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread >> -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR' >> >> Also down there are the sessionInfo and full gcc version, please let me >> know if you need more information. >> >> Regards, >> Dan >> -------------------------------------------------------------------- >>> source('http://www.bioconductor.org/biocLite.R') >>> biocLite('') >> BioC_mirror: http://bioconductor.org >> Using R version 2.15, BiocInstaller version 1.4.6. >> Installing package(s) '' >> Old packages: 'Rsubread' >> Update all/some/none? [a/s/n]: a >> trying URL >> 'http://www.bioconductor.org/packages/2.10/bioc/src/contrib/Rsubrea d_1.6.3.tar.gz' >> Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb) >> opened URL >> ================================================== >> downloaded 20.9 Mb >> >> WARNING: ignoring environment value of R_HOME >> * installing *source* package ?Rsubread? ... >> ** libs >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c R_wrapper.c -o R_wrapper.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c SNP_calling.c -o SNP_calling.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c aligner.c -o aligner.o >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c atgcContent.c -o atgcContent.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c detectionCall.c -o detectionCall.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c detectionCallAnnotation.c -o detectionCallAnnotation.o >> detectionCallAnnotation.c: In function ?calculateExonGCContent?: >> detectionCallAnnotation.c:175: warning: ignoring return value of >> ?fgets?, declared with attribute warn_unused_result >> detectionCallAnnotation.c: In function ?calculateIRGCContent?: >> detectionCallAnnotation.c:262: warning: ignoring return value of >> ?fgets?, declared with attribute warn_unused_result >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c exon-algorithms.c -o exon-algorithms.o >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c exon-align.c -o exon-align.o >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c fullscan.c -o fullscan.o >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c gene-algorithms.c -o gene-algorithms.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c gene-value-index.c -o gene-value-index.o >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c hashtable.c -o hashtable.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c index-builder.c -o index-builder.o >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c input-files.c -o input-files.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c processExons.c -o processExons.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c propmapped.c -o propmapped.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c qualityScores.c -o qualityScores.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c readSummary.c -o readSummary.o >> readSummary.c: In function ?readSummary?: >> readSummary.c:122: warning: format ?%d? expects type ?int?, but argument >> 5 has type ?long int? >> readSummary.c:122: warning: format ?%d? expects type ?int?, but argument >> 6 has type ?long int? >> readSummary.c:39: warning: ignoring return value of ?getline?, declared >> with attribute warn_unused_result >> readSummary.c:52: warning: ignoring return value of ?getline?, declared >> with attribute warn_unused_result >> readSummary.c:55: warning: ignoring return value of ?getline?, declared >> with attribute warn_unused_result >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c removeDuplicatedReads.c -o removeDuplicatedReads.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c sam2bed.c -o sam2bed.o >> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic >> -O3 -pipe -g -c sorted-hashtable.c -o sorted-hashtable.o >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gene-algorithms.h:23: warning: inline function ?add_gene_vote_weighted? >> declared but never defined >> gene-algorithms.h:22: warning: inline function ?add_gene_vote? declared >> but never defined >> gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o >> aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o >> exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o >> gene-value-index.o hashtable.o index-builder.o input-files.o >> processExons.o propmapped.o qualityScores.o readSummary.o >> removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread >> -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR >> ^Cmake: *** Deleting file `Rsubread.so' >> make: *** [Rsubread.so] Interrupt >> ** R >> ** inst >> ** preparing package for lazy loading >> ** help >> *** installing help indices >> ** building package indices >> ** installing vignettes >> ?Rsubread.Rnw? >> ** testing if installed package can be loaded >> Error in library.dynam(lib, package, package.lib) : >> shared object ?Rsubread.so? not found >> Error: loading failed >> Execution halted >> -------------------------------------------------------------------- >>> sessionInfo() >> R version 2.15.0 (2012-03-30) >> Platform: x86_64-pc-linux-gnu (64-bit) >> >> locale: >> [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C >> [3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8 >> [5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8 >> [7] LC_PAPER=C LC_NAME=C >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> [11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> -------------------------------------------------------------------- >> $ gcc -v >> Using built-in specs. >> Target: x86_64-linux-gnu >> Configured with: ../src/configure -v --with-pkgversion='Ubuntu >> 4.4.3-4ubuntu5.1' >> --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs >> --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr >> --enable-shared --enable-multiarch --enable-linker-build-id >> --with-system-zlib --libexecdir=/usr/lib --without-included-gettext >> --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 >> --program-suffix=-4.4 --enable-nls --enable-clocale=gnu >> --enable-libstdcxx-debug --enable-plugin --enable-objc-gc >> --disable-werror --with-arch-32=i486 --with-tune=generic >> --enable-checking=release --build=x86_64-linux-gnu >> --host=x86_64-linux-gnu --target=x86_64-linux-gnu >> Thread model: posix >> gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1) >> -------------------------------------------------------------------- >> >> On Wed, 2012-06-06 at 20:10 +1000, Wei Shi wrote: >>> Dear Dan, >>> >>> It didn't seem to be problem of requesting a continuous 1GB block in our >>> investigation. We tracked the memory usage of buildindex() function when >>> running it on yeast genome using a 32-bit VM, and found that the segfault >>> happened right after a request of a few KB of memory was sent to the >>> system when the memory parameter was set to 2500. However, the problem was >>> gone when the memory parameter was changed to 1000. >>> >>> Removing highly repetitive 16 mers required a continuous 1GB block of >>> memory, but this step was always executed successfully. This step also >>> included in the old version of Rsubread (1.1.1), and it did not have >>> problem there either. >>> >>> Could you please provide us your complete code for running your test and >>> also session info? This will help us to diagnose what the problem could be >>> because we couldn't reproduce what you saw from our end. >>> >>> For the compilation issue on your 64bit laptop, could you provide us more >>> details as well, including the message output from gcc? >>> >>> Thanks, >>> Wei >>> >>>> Dear Wei, >>>> >>>> Unfortunately reducing the memory parameter to 1000, still causes the >>>> segfault. I guess with 3g ram limit on a 32bit system, there is still a >>>> fat chance that you can not request a continuous 1g block. >>>> >>>> For that 64bit laptop, it is still strange about the 6g memory draining. >>>> It is happing during the installation when compiling the shared library >>>> Rsubread.so, not running the buildindex function. Btw, the gcc version >>>> is 4.4.3. >>>> >>>> Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. >>>> >>>> Regards, >>>> Dan >>>> >>>> On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: >>>>> Dear Dan, >>>>> >>>>> It is probably because including genome sequences into the index slowed >>>>> down your laptop. But I believe it should be alleviated if you give >>>>> smaller values to the 'memory ' parameter of the buildindex() function. >>>>> Also, the index building is an one-off operation, you do not need to >>>>> redo it even when new releases come. >>>>> >>>>> For your 32-bit opensuse box, I guess the problem will be solved if you >>>>> change the amount of memory requested to be 1000MB. >>>>> >>>>> Cheers, >>>>> Wei >>>>> >>>>> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: >>>>> >>>>>> Hi Robert, >>>>>> >>>>>> I have been experiencing something else, possibly related to yours, >>>>>> on a 64bit ubuntu laptop with 6g of ram. >>>>>> >>>>>> As I recall, when bumping to Bioc 2.10, the Rsubread installation kind >>>>>> of ate all the memory, basically froze the system so I had to call it >>>>>> off, yet building it on the server side turned out fine. So I think I >>>>>> just accepted that the new version may be 'computationally heavy' thus >>>>>> not suitable for a normal pc, though I did not find any mentioning of >>>>>> this increased memory requirement in the NEWS file. >>>>>> >>>>>> So currently Rsubread stays at 1.4.4 on that pc, all subsequent >>>>> versions >>>>>> of Rsubread drain the memory in the same way when compiling >>>>> Rsubread.so. >>>>>> >>>>>> Now I think I can confirm this on a 32-bit opensuse box, it did >>>>>> successfully built, but when running the example code in the manual, >>>>>> same segfault happens. >>>>>> >>>>>> >>>>>>> library(Rsubread) >>>>>>> ref<- system.file("extdata","reference.fa",package="Rsubread") >>>>>>> path<- system.file("extdata",package="Rsubread") >>>>>>> buildindex(basename=file.path(path,"reference_index"),reference=ref) >>>>>> >>>>>> Building a base-space index. >>>>>> Size of memory used=3700 MB >>>>>> Base name of the built index >>>>>> = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index >>>>>> >>>>>> *** caught segfault *** >>>>>> address 0xdf03ee80, cause 'memory not mapped' >>>>>> >>>>>> Traceback: >>>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >>>>>> as.character(cmd), PACKAGE = "Rsubread") >>>>>> 2: buildindex(basename = file.path(path, "reference_index"), reference >>>>>> = ref) >>>>>> >>>>>>> sessionInfo() >>>>>> R version 2.15.0 Patched (2012-06-04 r59517) >>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>> >>>>>> locale: >>>>>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >>>>>> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >>>>>> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >>>>>> >>>>>> attached base packages: >>>>>> [1] stats graphics grDevices utils datasets methods >>>>>> base >>>>>> >>>>>> other attached packages: >>>>>> [1] Rsubread_1.6.3 >>>>>> >>>>>> >>>>>> Regards, >>>>>> Dan >>>>>> >>>>>> On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: >>>>>>> hi, >>>>>>> >>>>>>> the computer room at my university where we do practicals on R& >>>>> Bioconductor runs a 32bit linux distribution and when i tried to run >>>>> the latest version of the Rsubread package (1.6.3) it crashes when >>>>> calling the buildindex() function on a multifasta file with the yeast >>>>> genome. this does *not* happen under a 64bit linux distribution. >>>>>>> >>>>>>> i have verified that installing the version before (1.4.4) on the >>>>> current R 2.15 it also crashes (on the 32bit), but two versions >>>>> before, the 1.1.1, it does *not* and it works smoothly on this 32bit >>>>> linux distribution. >>>>>>> >>>>>>> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 >>>>> where allChr.fa is the multifasta file with the yeast genome. >>>>>>> >>>>>>> so i can manage by now the problem by using the 1.1.1 version on R >>>>> 2.15 for my teaching but i wonder whether there would be some easy >>>>> solution for this, or even if it could be a symptom of something else >>>>> that the Rsubread developers should worry about. i know that using a >>>>> 32bit system nowadays is quite obsolete but this is what i got for >>>>> teaching :( and i would be happy to let my students play with the >>>>> latest version of Rsubread in the future. >>>>>>> >>>>>>> >>>>>>> thanks!!! >>>>>>> robert. >>>>>>> >>>>>>> ======================Rsubread 1.6.3 on R 2.15======================= >>>>>>> >>>>>>>> library(Rsubread) >>>>>>>> sessionInfo() >>>>>>> R version 2.15.0 (2012-03-30) >>>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>>> >>>>>>> locale: >>>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >>>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >>>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >>>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >>>>>>> >>>>>>> attached base packages: >>>>>>> [1] stats graphics grDevices utils datasets methods base >>>>>>> >>>>>>> other attached packages: >>>>>>> [1] Rsubread_1.6.3 >>>>>>> >>>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", >>>>> memory=2500) >>>>>>> >>>>>>> Building a base-space index. >>>>>>> Size of memory used=2500 MB >>>>>>> Base name of the built index = subreadindex >>>>>>> >>>>>>> *** caught segfault *** >>>>>>> address 0xdf670cc0, cause 'memory not mapped' >>>>>>> >>>>>>> Traceback: >>>>>>> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >>>>> as.character(cmd), PACKAGE = "Rsubread") >>>>>>> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", >>>>> memory = 2500) >>>>>>> >>>>>>> Possible actions: >>>>>>> 1: abort (with core dump, if enabled) >>>>>>> 2: normal R exit >>>>>>> 3: exit R without saving workspace >>>>>>> 4: exit R saving workspace >>>>>>> Selection: >>>>>>> >>>>>>> >>>>>>> ======================Rsubread 1.1.1 on R 2.15======================= >>>>>>> >>>>>>>> library(Rsubread) >>>>>>>> buildindex(basename="subreadindex", reference="allChr.fa", >>>>> memory=2500) >>>>>>> >>>>>>> Building the index in the base space. >>>>>>> Size of memory requested=2500 MB >>>>>>> Index base name = subreadindex >>>>>>> INDEX ITEMS PER PARTITION = 275940352 >>>>>>> >>>>>>> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps >>>>> completed=81.76%; time used=2.4s; rate=4111.8k >>>>> bps/s; total=12m bps >>>>>>> All the chromosome files are processed. >>>>>>> | Dumping index >>>>> [===========================================================>] >>>>>>> Index subreadindex is successfully built. >>>>>>>> sessionInfo() >>>>>>> R version 2.15.0 (2012-03-30) >>>>>>> Platform: i686-pc-linux-gnu (32-bit) >>>>>>> >>>>>>> locale: >>>>>>> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >>>>>>> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >>>>>>> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >>>>>>> [7] LC_PAPER=C LC_NAME=C >>>>>>> [9] LC_ADDRESS=C LC_TELEPHONE=C >>>>>>> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >>>>>>> >>>>>>> attached base packages: >>>>>>> [1] stats graphics grDevices utils datasets methods base >>>>>>> >>>>>>> other attached packages: >>>>>>> [1] Rsubread_1.1.1 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Bioconductor mailing list >>>>>>> Bioconductor at r-project.org >>>>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>>>> Search the archives: >>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>>>> >>>>>> _______________________________________________ >>>>>> Bioconductor mailing list >>>>>> Bioconductor at r-project.org >>>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>>> Search the archives: >>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>>> >>>>> >>>>> ______________________________________________________________________ >>>>> The information in this email is confidential and intended solely for >>>>> the addressee. >>>>> You must not disclose, forward, print or use it without the permission >>>>> of the sender. >>>>> ______________________________________________________________________ >>>> >>>> >>>> >>>> >>> >>> >>> >>> ______________________________________________________________________ >>> The information in this email is confidential and intended solely for the addressee. >>> You must not disclose, forward, print or use it without the permission of the sender. >>> ______________________________________________________________________ >> >> > > > ______________________________________________________________________ > The information in this email is confidential and intended solely for the addressee. > You must not disclose, forward, print or use it without the permission of the sender. > ______________________________________________________________________ > > -- Robert Castelo, PhD Associate Professor Dept. of Experimental and Health Sciences Universitat Pompeu Fabra (UPF) Barcelona Biomedical Research Park (PRBB) Dr Aiguader 88 E-08003 Barcelona, Spain telf: +34.933.160.514 fax: +34.933.160.550
ADD REPLYlink written 5.4 years ago by Robert Castelo2.0k
Dear Robert, That's great. Thanks for letting us know. Cheers, Wei > Dear Wei, > > it also works on my side, the 32 bit machines do not crash anymore. > > thanks for solving this so quickly! > robert. > > On 06/08/2012 04:10 AM, Wei Shi wrote: >> Dear Robert, Dan and Peter, >> >> We have made changes to a number of functions in the package to reduce >> the memory allocated to Rsubread by the operating system when it was >> loaded. The new version has been committed to both bioc release >> (Rsubread 1.6.4) and bioc devel (Rsubread 1.7.4). They should be >> available to you in a day or two. >> >> Also, the buildindex() function no longer needs the allocation of 1GB >> continuous memory region. But it will still consume at least 1GB of >> memory when it is running, no matter what the given value of the >> 'memory' parameter is. >> >> We have tested the new version on our 32-bit VM machine (it has 3GB of >> memory and the value of 'memory' parameter used by buildindex was 2500) >> and it solves all the reported problems, so we are pretty happy with it. >> I hope the new version works in your computers/laptops, but please do >> let us know if it doesn't. >> >> Sorry about the problems you have encountered. It's always a challenge >> to develop a R package with so much C code in it! >> >> Cheers, >> Wei >> >> >> >> On Jun 6, 2012, at 9:08 PM, Dan Du wrote: >> >>> Dear Wei, >>> >>> Here is a standard bioclite update, I think it is at the last step when >>> compiling Rsubread.so, the memory usage exceeds 5.5g, then system >>> freeze >>> and I have to call it off. Same result when runing 'R CMD INSTALL >>> Rsubread_1.6.3.tar.gz' from shell, or manually compile all .c file and >>> run the last gcc statement. I guess there might just be a minimum ram >>> requirement somewhere higher than 6g... I will do some more poking when >>> I have time. >>> >>> 'gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o >>> aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o >>> exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o >>> gene-value-index.o hashtable.o index-builder.o input-files.o >>> processExons.o propmapped.o qualityScores.o readSummary.o >>> removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread >>> -DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR' >>> >>> Also down there are the sessionInfo and full gcc version, please let me >>> know if you need more information. >>> >>> Regards, >>> Dan >>> -------------------------------------------------------------------- >>>> source('http://www.bioconductor.org/biocLite.R') >>>> biocLite('') >>> BioC_mirror: http://bioconductor.org >>> Using R version 2.15, BiocInstaller version 1.4.6. >>> Installing package(s) '' >>> Old packages: 'Rsubread' >>> Update all/some/none? [a/s/n]: a >>> trying URL >>> 'http://www.bioconductor.org/packages/2.10/bioc/src/contrib/Rsubre ad_1.6.3.tar.gz' >>> Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb) >>> opened URL >>> ================================================== >>> downloaded 20.9 Mb >>> >>> WARNING: ignoring environment value of R_HOME >>> * installing *source* package ?Rsubread? ... >>> ** libs >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON >>> -fpic >>> -O3 -pipe -g -c R_wrapper.c -o R_wrapper.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON >>> -fpic >>> -O3 -pipe -g -c SNP_calling.c -o SNP_calling.o >>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON >> ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:4}}
ADD REPLYlink written 5.4 years ago by Wei Shi2.7k
0
gravatar for Wei Shi
5.5 years ago by
Wei Shi2.7k
Australia
Wei Shi2.7k wrote:
Dear Robert, We do not have a 32bit linux machine here, but we managed to reproduce the problem you have encountered using a 32bit Virtual Machine running on a 64bit linux machine. It turned out that the problem occurred when one of our calls to the malloc() function was unsuccessful in requesting memories from the system, which means the system runs out of memory and can not allocate more memories to the buildindex() function. We tried to let buildindex() function request a small amount of memory (1000MB), which was found to be able to solve this problem. So I recommend you to give value of 1000 to the 'memory' parameter of buildindex() function. The yeast genome is very tiny, so you do not need 2500MB of memory to build the index for it. The buildindex() function requires at least 1000MB of memory no matter how big or small the reference genome is, to build hash tables and remove highly repetitive 16 mers. Also note that the mapping results are not affected by the amount of memory requested in the index building step. The amount of memory used will only affect the running time. For example, using 8GB of memory to build index for mouse genome will give you a mapping speed twice as fast as that from using 4GB of memory. But for the yeast, the entire index will always be loaded into the memory in one go, because its genome size is tiny and the minimal memory used by buildindex() is 1GB which is big enough to accommodate the hash table, the genome sequences and other related information. Finally, the reason why the problem you encountered did not happen in version 1.1.1 was because genome sequences were not included in the built index by default in that version, however, they are included in the index in the newer versions. Hope this can solve your problem. But please let us know if it doesn't. Cheers, Wei On Jun 5, 2012, at 5:45 PM, Robert Castelo wrote: > hi, > > the computer room at my university where we do practicals on R & Bioconductor runs a 32bit linux distribution and when i tried to run the latest version of the Rsubread package (1.6.3) it crashes when calling the buildindex() function on a multifasta file with the yeast genome. this does *not* happen under a 64bit linux distribution. > > i have verified that installing the version before (1.4.4) on the current R 2.15 it also crashes (on the 32bit), but two versions before, the 1.1.1, it does *not* and it works smoothly on this 32bit linux distribution. > > i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 where allChr.fa is the multifasta file with the yeast genome. > > so i can manage by now the problem by using the 1.1.1 version on R 2.15 for my teaching but i wonder whether there would be some easy solution for this, or even if it could be a symptom of something else that the Rsubread developers should worry about. i know that using a 32bit system nowadays is quite obsolete but this is what i got for teaching :( and i would be happy to let my students play with the latest version of Rsubread in the future. > > > thanks!!! > robert. > > ======================Rsubread 1.6.3 on R 2.15======================= > >> library(Rsubread) >> sessionInfo() > R version 2.15.0 (2012-03-30) > Platform: i686-pc-linux-gnu (32-bit) > > locale: > [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > [7] LC_PAPER=C LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] Rsubread_1.6.3 > >> buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) > > Building a base-space index. > Size of memory used=2500 MB > Base name of the built index = subreadindex > > *** caught segfault *** > address 0xdf670cc0, cause 'memory not mapped' > > Traceback: > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = as.character(cmd), PACKAGE = "Rsubread") > 2: buildindex(basename = "subreadindex", reference = "allChr.fa", memory = 2500) > > Possible actions: > 1: abort (with core dump, if enabled) > 2: normal R exit > 3: exit R without saving workspace > 4: exit R saving workspace > Selection: > > > ======================Rsubread 1.1.1 on R 2.15======================= > >> library(Rsubread) >> buildindex(basename="subreadindex", reference="allChr.fa", memory=2500) > > Building the index in the base space. > Size of memory requested=2500 MB > Index base name = subreadindex > INDEX ITEMS PER PARTITION = 275940352 > > completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps completed=81.76%; time used=2.4s; rate=4111.8k bps/s; total=12m bps > All the chromosome files are processed. > | Dumping index [===========================================================>] > Index subreadindex is successfully built. >> sessionInfo() > R version 2.15.0 (2012-03-30) > Platform: i686-pc-linux-gnu (32-bit) > > locale: > [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C > [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 > [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 > [7] LC_PAPER=C LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] Rsubread_1.1.1 > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:6}}
ADD COMMENTlink written 5.5 years ago by Wei Shi2.7k
0
gravatar for Wei Shi
5.5 years ago by
Wei Shi2.7k
Australia
Wei Shi2.7k wrote:
Dear Peter, It looks like your installation of Rsubread was unsuccessfully, therefore you got an error when you loaded it into your R session. This issue is different from the issue with running buildindex() function on a 32-bit machine. Did you notice any errors when it was being installed? You can reinstall it using the following commands in your R session: source("http://bioconductor.org/biocLite.R") biocLite("Rsubread") Cheers, Wei > Dear Wei, > I also encounter a (perhaps related) problem using Rsubread (see below). > Kind regards > Peter > > >> library(Rsubread) > Error in dyn.load(file, DLLpath = DLLpath, ...) : > unable to load shared object > '/opt/ghpc/R-2.15.0/lib64/R/library/Rsubread/libs/Rsubread.so': > /opt/ghpc/R-2.15.0/lib64/R/library/Rsubread/libs/Rsubread.so: cannot map > zero-fill pages: Cannot allocate memory > Error: package/namespace load failed for 'Rsubread' >> ref <- system.file("extdata","reference.fa",package="Rsubread") >> path <- system.file("extdata",package="Rsubread") >> buildindex(basename=file.path(path,"reference_index"),reference=ref) > Error: could not find function "buildindex" >> sessionInfo() > R version 2.15.0 (2012-03-30) > Platform: x86_64-unknown-linux-gnu (64-bit) > locale: > [1] LC_CTYPE=en_US LC_NUMERIC=C LC_TIME=en_US > [4] LC_COLLATE=en_US LC_MONETARY=en_US LC_MESSAGES=en_US > [7] LC_PAPER=C LC_NAME=C LC_ADDRESS=C > [10] LC_TELEPHONE=C LC_MEASUREMENT=en_US LC_IDENTIFICATION=C > attached base packages: > [1] stats graphics grDevices utils datasets methods base >> > > > ________________________________________ > From: bioconductor-bounces at r-project.org > [bioconductor-bounces at r-project.org] On Behalf Of Dan Du > [tooyoung at gmail.com] > Sent: Wednesday, June 06, 2012 8:50 AM > To: Wei Shi > Cc: Yang Liao; bioconductor at r-project.org mailman > Subject: Re: [BioC] Rsubread crashes in 32bit linux > > Dear Wei, > > Unfortunately reducing the memory parameter to 1000, still causes the > segfault. I guess with 3g ram limit on a 32bit system, there is still a > fat chance that you can not request a continuous 1g block. > > For that 64bit laptop, it is still strange about the 6g memory draining. > It is happing during the installation when compiling the shared library > Rsubread.so, not running the buildindex function. Btw, the gcc version > is 4.4.3. > > Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. > > Regards, > Dan > > On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: >> Dear Dan, >> >> It is probably because including genome sequences into the index slowed >> down your laptop. But I believe it should be alleviated if you give >> smaller values to the 'memory ' parameter of the buildindex() function. >> Also, the index building is an one-off operation, you do not need to >> redo it even when new releases come. >> >> For your 32-bit opensuse box, I guess the problem will be solved if you >> change the amount of memory requested to be 1000MB. >> >> Cheers, >> Wei >> >> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: >> >> > Hi Robert, >> > >> > I have been experiencing something else, possibly related to yours, >> > on a 64bit ubuntu laptop with 6g of ram. >> > >> > As I recall, when bumping to Bioc 2.10, the Rsubread installation kind >> > of ate all the memory, basically froze the system so I had to call it >> > off, yet building it on the server side turned out fine. So I think I >> > just accepted that the new version may be 'computationally heavy' thus >> > not suitable for a normal pc, though I did not find any mentioning of >> > this increased memory requirement in the NEWS file. >> > >> > So currently Rsubread stays at 1.4.4 on that pc, all subsequent >> versions >> > of Rsubread drain the memory in the same way when compiling >> Rsubread.so. >> > >> > Now I think I can confirm this on a 32-bit opensuse box, it did >> > successfully built, but when running the example code in the manual, >> > same segfault happens. >> > >> > >> >> library(Rsubread) >> >> ref <- system.file("extdata","reference.fa",package="Rsubread") >> >> path <- system.file("extdata",package="Rsubread") >> >> buildindex(basename=file.path(path,"reference_index"),reference=ref) >> > >> > Building a base-space index. >> > Size of memory used=3700 MB >> > Base name of the built index >> > = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index >> > >> > *** caught segfault *** >> > address 0xdf03ee80, cause 'memory not mapped' >> > >> > Traceback: >> > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >> > as.character(cmd), PACKAGE = "Rsubread") >> > 2: buildindex(basename = file.path(path, "reference_index"), reference >> > = ref) >> > >> >> sessionInfo() >> > R version 2.15.0 Patched (2012-06-04 r59517) >> > Platform: i686-pc-linux-gnu (32-bit) >> > >> > locale: >> > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >> > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >> > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >> > [7] LC_PAPER=C LC_NAME=C >> > [9] LC_ADDRESS=C LC_TELEPHONE=C >> > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >> > >> > attached base packages: >> > [1] stats graphics grDevices utils datasets methods >> > base >> > >> > other attached packages: >> > [1] Rsubread_1.6.3 >> > >> > >> > Regards, >> > Dan >> > >> > On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: >> >> hi, >> >> >> >> the computer room at my university where we do practicals on R & >> Bioconductor runs a 32bit linux distribution and when i tried to run >> the latest version of the Rsubread package (1.6.3) it crashes when >> calling the buildindex() function on a multifasta file with the yeast >> genome. this does *not* happen under a 64bit linux distribution. >> >> >> >> i have verified that installing the version before (1.4.4) on the >> current R 2.15 it also crashes (on the 32bit), but two versions >> before, the 1.1.1, it does *not* and it works smoothly on this 32bit >> linux distribution. >> >> >> >> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 >> where allChr.fa is the multifasta file with the yeast genome. >> >> >> >> so i can manage by now the problem by using the 1.1.1 version on R >> 2.15 for my teaching but i wonder whether there would be some easy >> solution for this, or even if it could be a symptom of something else >> that the Rsubread developers should worry about. i know that using a >> 32bit system nowadays is quite obsolete but this is what i got for >> teaching :( and i would be happy to let my students play with the >> latest version of Rsubread in the future. >> >> >> >> >> >> thanks!!! >> >> robert. >> >> >> >> ======================Rsubread 1.6.3 on R 2.15======================= >> >> >> >>> library(Rsubread) >> >>> sessionInfo() >> >> R version 2.15.0 (2012-03-30) >> >> Platform: i686-pc-linux-gnu (32-bit) >> >> >> >> locale: >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >> >> [7] LC_PAPER=C LC_NAME=C >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >> >> >> >> attached base packages: >> >> [1] stats graphics grDevices utils datasets methods base >> >> >> >> other attached packages: >> >> [1] Rsubread_1.6.3 >> >> >> >>> buildindex(basename="subreadindex", reference="allChr.fa", >> memory=2500) >> >> >> >> Building a base-space index. >> >> Size of memory used=2500 MB >> >> Base name of the built index = subreadindex >> >> >> >> *** caught segfault *** >> >> address 0xdf670cc0, cause 'memory not mapped' >> >> >> >> Traceback: >> >> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >> as.character(cmd), PACKAGE = "Rsubread") >> >> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", >> memory = 2500) >> >> >> >> Possible actions: >> >> 1: abort (with core dump, if enabled) >> >> 2: normal R exit >> >> 3: exit R without saving workspace >> >> 4: exit R saving workspace >> >> Selection: >> >> >> >> >> >> ======================Rsubread 1.1.1 on R 2.15======================= >> >> >> >>> library(Rsubread) >> >>> buildindex(basename="subreadindex", reference="allChr.fa", >> memory=2500) >> >> >> >> Building the index in the base space. >> >> Size of memory requested=2500 MB >> >> Index base name = subreadindex >> >> INDEX ITEMS PER PARTITION = 275940352 >> >> >> >> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps >> completed=81.76%; time used=2.4s; rate=4111.8k >> bps/s; total=12m bps >> >> All the chromosome files are processed. >> >> | Dumping index >> [===========================================================>] >> >> Index subreadindex is successfully built. >> >>> sessionInfo() >> >> R version 2.15.0 (2012-03-30) >> >> Platform: i686-pc-linux-gnu (32-bit) >> >> >> >> locale: >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >> >> [7] LC_PAPER=C LC_NAME=C >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >> >> >> >> attached base packages: >> >> [1] stats graphics grDevices utils datasets methods base >> >> >> >> other attached packages: >> >> [1] Rsubread_1.1.1 >> >> >> >> _______________________________________________ >> >> Bioconductor mailing list >> >> Bioconductor at r-project.org >> >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> >> Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor >> > >> > _______________________________________________ >> > Bioconductor mailing list >> > Bioconductor at r-project.org >> > https://stat.ethz.ch/mailman/listinfo/bioconductor >> > Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor >> >> >> ______________________________________________________________________ >> The information in this email is confidential and inte...{{dropped:6}} > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:4}}
ADD COMMENTlink written 5.5 years ago by Wei Shi2.7k
Hi Wei, I have tried to install Rsubread locally in my own directory, but I was not successfull. The error/warning messages below. regards Peter R version 2.15.0 (2012-03-30) Copyright (C) 2012 The R Foundation for Statistical Computing ISBN 3-900051-07-0 Platform: x86_64-unknown-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > source("http://bioconductor.org/biocLite.R") BiocInstaller version 1.4.6, ?biocLite for help > biocLite("Rsubread") BioC_mirror: http://bioconductor.org Using R version 2.15, BiocInstaller version 1.4.6. Installing package(s) 'Rsubread' Warning in install.packages(pkgs = pkgs, lib = lib, repos = repos, ...) : 'lib = "/usr/hag/biogen/pso/R/x86_64-unknown-linux-gnu- library/2.15"' is not writable Would you like to create a personal library ~/R/x86_64-unknown-linux-gnu-library/2.15 to install packages into? (y/n) y trying URL 'http://www.bioconductor.org/packages/2.10/bioc/src/contrib /Rsubread_1.6.3.tar.gz' Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb) opened URL ================================================== downloaded 20.9 Mb * installing *source* package 'Rsubread' ... ** libs gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c R_wrapper.c -o R_wrapper.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c SNP_calling.c -o SNP_calling.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c aligner.c -o aligner.o gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c atgcContent.c -o atgcContent.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c detectionCall.c -o detectionCall.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c detectionCallAnnotation.c -o detectionCallAnnotation.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c exon- algorithms.c -o exon-algorithms.o gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c exon-align.c -o exon-align.o gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c fullscan.c -o fullscan.o gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c gene- algorithms.c -o gene-algorithms.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c gene-value- index.c -o gene-value-index.o gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c hashtable.c -o hashtable.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c index- builder.c -o index-builder.o gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c input-files.c -o input-files.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c processExons.c -o processExons.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c propmapped.c -o propmapped.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c qualityScores.c -o qualityScores.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c readSummary.c -o readSummary.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c removeDuplicatedReads.c -o removeDuplicatedReads.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c sam2bed.c -o sam2bed.o gcc -std=gnu99 -I/opt/ghpc/R-2.15.0/lib64/R/include -DNDEBUG -I/opt/ghpc/include -DMAKE_FOR_EXON -fpic -g -O2 -c sorted- hashtable.c -o sorted-hashtable.o gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gene-algorithms.h:23:13: warning: inline function 'add_gene_vote_weighted' declared but never defined gene-algorithms.h:22:13: warning: inline function 'add_gene_vote' declared but never defined gcc -std=gnu99 -shared -L/opt/ghpc/lib64 -L/opt/ghpc/lib -o Rsubread.so R_wrapper.o SNP_calling.o aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o exon-algorithms.o exon- align.o fullscan.o gene-algorithms.o gene-value-index.o hashtable.o index-builder.o input-files.o processExons.o propmapped.o qualityScores.o readSummary.o removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread -DMAKE_FOR_EXON -L/opt/ghpc/R-2.15.0/lib64/R/lib -lR installing to /usr/hag/biogen/pso/R/x86_64-unknown-linux-gnu- library/2.15/Rsubread/libs ** R ** inst ** preparing package for lazy loading ** help *** installing help indices ** building package indices ** installing vignettes 'Rsubread.Rnw' ** testing if installed package can be loaded Error in dyn.load(file, DLLpath = DLLpath, ...) : unable to load shared object '/usr/hag/biogen/pso/R/x86_64-unknown- linux-gnu-library/2.15/Rsubread/libs/Rsubread.so': /usr/hag/biogen/pso/R/x86_64-unknown-linux-gnu- library/2.15/Rsubread/libs/Rsubread.so: cannot map zero-fill pages: Cannot allocate memory Error: loading failed Execution halted ERROR: loading failed * removing '/usr/hag/biogen/pso/R/x86_64-unknown-linux-gnu- library/2.15/Rsubread' The downloaded source packages are in '/tmp/Rtmp5zvTug/downloaded_packages' Warning messages: 1: In install.packages(pkgs = pkgs, lib = lib, repos = repos, ...) : installation of package 'Rsubread' had non-zero exit status 2: installed directory not writable, cannot update packages 'gdata', 'GEOquery', 'Rgraphviz' > ________________________________________ From: Wei Shi [shi@wehi.EDU.AU] Sent: Wednesday, June 06, 2012 12:56 PM To: Peter S?rensen (HAG) Cc: Dan Du; Wei Shi; Yang Liao; bioconductor at r-project.org mailman Subject: RE: [BioC] Rsubread crashes in 32bit linux Dear Peter, It looks like your installation of Rsubread was unsuccessfully, therefore you got an error when you loaded it into your R session. This issue is different from the issue with running buildindex() function on a 32-bit machine. Did you notice any errors when it was being installed? You can reinstall it using the following commands in your R session: source("http://bioconductor.org/biocLite.R") biocLite("Rsubread") Cheers, Wei > Dear Wei, > I also encounter a (perhaps related) problem using Rsubread (see below). > Kind regards > Peter > > >> library(Rsubread) > Error in dyn.load(file, DLLpath = DLLpath, ...) : > unable to load shared object > '/opt/ghpc/R-2.15.0/lib64/R/library/Rsubread/libs/Rsubread.so': > /opt/ghpc/R-2.15.0/lib64/R/library/Rsubread/libs/Rsubread.so: cannot map > zero-fill pages: Cannot allocate memory > Error: package/namespace load failed for 'Rsubread' >> ref <- system.file("extdata","reference.fa",package="Rsubread") >> path <- system.file("extdata",package="Rsubread") >> buildindex(basename=file.path(path,"reference_index"),reference=ref) > Error: could not find function "buildindex" >> sessionInfo() > R version 2.15.0 (2012-03-30) > Platform: x86_64-unknown-linux-gnu (64-bit) > locale: > [1] LC_CTYPE=en_US LC_NUMERIC=C LC_TIME=en_US > [4] LC_COLLATE=en_US LC_MONETARY=en_US LC_MESSAGES=en_US > [7] LC_PAPER=C LC_NAME=C LC_ADDRESS=C > [10] LC_TELEPHONE=C LC_MEASUREMENT=en_US LC_IDENTIFICATION=C > attached base packages: > [1] stats graphics grDevices utils datasets methods base >> > > > ________________________________________ > From: bioconductor-bounces at r-project.org > [bioconductor-bounces at r-project.org] On Behalf Of Dan Du > [tooyoung at gmail.com] > Sent: Wednesday, June 06, 2012 8:50 AM > To: Wei Shi > Cc: Yang Liao; bioconductor at r-project.org mailman > Subject: Re: [BioC] Rsubread crashes in 32bit linux > > Dear Wei, > > Unfortunately reducing the memory parameter to 1000, still causes the > segfault. I guess with 3g ram limit on a 32bit system, there is still a > fat chance that you can not request a continuous 1g block. > > For that 64bit laptop, it is still strange about the 6g memory draining. > It is happing during the installation when compiling the shared library > Rsubread.so, not running the buildindex function. Btw, the gcc version > is 4.4.3. > > Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively. > > Regards, > Dan > > On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote: >> Dear Dan, >> >> It is probably because including genome sequences into the index slowed >> down your laptop. But I believe it should be alleviated if you give >> smaller values to the 'memory ' parameter of the buildindex() function. >> Also, the index building is an one-off operation, you do not need to >> redo it even when new releases come. >> >> For your 32-bit opensuse box, I guess the problem will be solved if you >> change the amount of memory requested to be 1000MB. >> >> Cheers, >> Wei >> >> On Jun 5, 2012, at 11:43 PM, Dan Du wrote: >> >> > Hi Robert, >> > >> > I have been experiencing something else, possibly related to yours, >> > on a 64bit ubuntu laptop with 6g of ram. >> > >> > As I recall, when bumping to Bioc 2.10, the Rsubread installation kind >> > of ate all the memory, basically froze the system so I had to call it >> > off, yet building it on the server side turned out fine. So I think I >> > just accepted that the new version may be 'computationally heavy' thus >> > not suitable for a normal pc, though I did not find any mentioning of >> > this increased memory requirement in the NEWS file. >> > >> > So currently Rsubread stays at 1.4.4 on that pc, all subsequent >> versions >> > of Rsubread drain the memory in the same way when compiling >> Rsubread.so. >> > >> > Now I think I can confirm this on a 32-bit opensuse box, it did >> > successfully built, but when running the example code in the manual, >> > same segfault happens. >> > >> > >> >> library(Rsubread) >> >> ref <- system.file("extdata","reference.fa",package="Rsubread") >> >> path <- system.file("extdata",package="Rsubread") >> >> buildindex(basename=file.path(path,"reference_index"),reference=ref) >> > >> > Building a base-space index. >> > Size of memory used=3700 MB >> > Base name of the built index >> > = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index >> > >> > *** caught segfault *** >> > address 0xdf03ee80, cause 'memory not mapped' >> > >> > Traceback: >> > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >> > as.character(cmd), PACKAGE = "Rsubread") >> > 2: buildindex(basename = file.path(path, "reference_index"), reference >> > = ref) >> > >> >> sessionInfo() >> > R version 2.15.0 Patched (2012-06-04 r59517) >> > Platform: i686-pc-linux-gnu (32-bit) >> > >> > locale: >> > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >> > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >> > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >> > [7] LC_PAPER=C LC_NAME=C >> > [9] LC_ADDRESS=C LC_TELEPHONE=C >> > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >> > >> > attached base packages: >> > [1] stats graphics grDevices utils datasets methods >> > base >> > >> > other attached packages: >> > [1] Rsubread_1.6.3 >> > >> > >> > Regards, >> > Dan >> > >> > On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote: >> >> hi, >> >> >> >> the computer room at my university where we do practicals on R & >> Bioconductor runs a 32bit linux distribution and when i tried to run >> the latest version of the Rsubread package (1.6.3) it crashes when >> calling the buildindex() function on a multifasta file with the yeast >> genome. this does *not* happen under a 64bit linux distribution. >> >> >> >> i have verified that installing the version before (1.4.4) on the >> current R 2.15 it also crashes (on the 32bit), but two versions >> before, the 1.1.1, it does *not* and it works smoothly on this 32bit >> linux distribution. >> >> >> >> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15 >> where allChr.fa is the multifasta file with the yeast genome. >> >> >> >> so i can manage by now the problem by using the 1.1.1 version on R >> 2.15 for my teaching but i wonder whether there would be some easy >> solution for this, or even if it could be a symptom of something else >> that the Rsubread developers should worry about. i know that using a >> 32bit system nowadays is quite obsolete but this is what i got for >> teaching :( and i would be happy to let my students play with the >> latest version of Rsubread in the future. >> >> >> >> >> >> thanks!!! >> >> robert. >> >> >> >> ======================Rsubread 1.6.3 on R 2.15======================= >> >> >> >>> library(Rsubread) >> >>> sessionInfo() >> >> R version 2.15.0 (2012-03-30) >> >> Platform: i686-pc-linux-gnu (32-bit) >> >> >> >> locale: >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >> >> [7] LC_PAPER=C LC_NAME=C >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >> >> >> >> attached base packages: >> >> [1] stats graphics grDevices utils datasets methods base >> >> >> >> other attached packages: >> >> [1] Rsubread_1.6.3 >> >> >> >>> buildindex(basename="subreadindex", reference="allChr.fa", >> memory=2500) >> >> >> >> Building a base-space index. >> >> Size of memory used=2500 MB >> >> Base name of the built index = subreadindex >> >> >> >> *** caught segfault *** >> >> address 0xdf670cc0, cause 'memory not mapped' >> >> >> >> Traceback: >> >> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv = >> as.character(cmd), PACKAGE = "Rsubread") >> >> 2: buildindex(basename = "subreadindex", reference = "allChr.fa", >> memory = 2500) >> >> >> >> Possible actions: >> >> 1: abort (with core dump, if enabled) >> >> 2: normal R exit >> >> 3: exit R without saving workspace >> >> 4: exit R saving workspace >> >> Selection: >> >> >> >> >> >> ======================Rsubread 1.1.1 on R 2.15======================= >> >> >> >>> library(Rsubread) >> >>> buildindex(basename="subreadindex", reference="allChr.fa", >> memory=2500) >> >> >> >> Building the index in the base space. >> >> Size of memory requested=2500 MB >> >> Index base name = subreadindex >> >> INDEX ITEMS PER PARTITION = 275940352 >> >> >> >> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps >> completed=81.76%; time used=2.4s; rate=4111.8k >> bps/s; total=12m bps >> >> All the chromosome files are processed. >> >> | Dumping index >> [===========================================================>] >> >> Index subreadindex is successfully built. >> >>> sessionInfo() >> >> R version 2.15.0 (2012-03-30) >> >> Platform: i686-pc-linux-gnu (32-bit) >> >> >> >> locale: >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8 >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8 >> >> [7] LC_PAPER=C LC_NAME=C >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C >> >> >> >> attached base packages: >> >> [1] stats graphics grDevices utils datasets methods base >> >> >> >> other attached packages: >> >> [1] Rsubread_1.1.1 >> >> >> >> _______________________________________________ >> >> Bioconductor mailing list >> >> Bioconductor at r-project.org >> >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> >> Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor >> > >> > _______________________________________________ >> > Bioconductor mailing list >> > Bioconductor at r-project.org >> > https://stat.ethz.ch/mailman/listinfo/bioconductor >> > Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor >> >> >> ______________________________________________________________________ >> The information in this email is confidential and inte...{{dropped:6}} > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:6}}
ADD REPLYlink written 5.5 years ago by Peter Sørensen (HAG)60
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.2.0
Traffic: 145 users visited in the last hour