Hi Thomas,

I had a quick look at your example, and it appears that very few guides can be matched (0.032% if I don't allow shifting, 0.059% with shifting and 0.083% if both shifting and mismatches are allowed - see code below).

I guess this is reasonable given that your demo provides 99 sgRNAs out of 120,000, which is about 0.08% of the total library. 

For screens I've analysed, this step is done in 5-10 mins depending upon the number of reads. Maybe try re-running with allowMismatch=FALSE . According to system.time on this example, it takes around twice as long to allow both shifting and mismatches.

Cheers,

Matt 

> system.time(processAmplicons("plasmidpart.fastq", readfile2=NULL, "Samples1.txt", "tinysgset_gene.txt",

+                                barcodeStart=1, barcodeEnd=9, hairpinStart=34, hairpinEnd=53, verbose=TRUE))

 -- Number of Barcodes : 4

 -- Number of Hairpins : 99

Processing reads in plasmidpart.fastq.

 -- Processing 10 million reads

Number of reads in file plasmidpart.fastq : 37500

 

The input run parameters are: 

 -- Barcode: start position 1    end position 9  length 9

 -- Hairpin: start position 34   end position 53         length 20

 -- Hairpin sequences need to match at specified positions. 

 -- Mismatch in barcode/hairpin sequences not allowed. 

 

Total number of read is 37500 

There are 37178 reads (99.1413 percent) with barcode matches

There are 12 reads (0.0320 percent) with hairpin matches

There are 12 reads (0.0320 percent) with both barcode and hairpin matches

   user  system elapsed 

  0.056   0.008   0.330 

> system.time(processAmplicons("plasmidpart.fastq", readfile2=NULL, "Samples1.txt", "tinysgset_gene.txt",

+                                barcodeStart=1, barcodeEnd=9, hairpinStart=34, hairpinEnd=53, allowShifting=TRUE, shiftingBase=2,

+                                verbose=TRUE))

 -- Number of Barcodes : 4

 -- Number of Hairpins : 99

Processing reads in plasmidpart.fastq.

 -- Processing 10 million reads

Number of reads in file plasmidpart.fastq : 37500

 

The input run parameters are: 

 -- Barcode: start position 1    end position 9  length 9

 -- Hairpin: start position 34   end position 53         length 20

 -- Allow hairpin sequences to be matched to a shifted position, <= 2 base left or right of the specified positions. 

 -- Mismatch in barcode/hairpin sequences not allowed. 

 

Total number of read is 37500 

There are 37178 reads (99.1413 percent) with barcode matches

There are 22 reads (0.0587 percent) with hairpin matches

There are 22 reads (0.0587 percent) with both barcode and hairpin matches

   user  system elapsed 

  0.060   0.011   0.202 

> system.time(processAmplicons("plasmidpart.fastq", readfile2=NULL, "Samples1.txt", "tinysgset_gene.txt",

+                                barcodeStart=1, barcodeEnd=9, hairpinStart=34, hairpinEnd=53, allowShifting=TRUE, shiftingBase=2,

+                                allowMismatch=TRUE, hairpinMismatchBase=2, allowShiftedMismatch=TRUE, verbose=TRUE))

 -- Number of Barcodes : 4

 -- Number of Hairpins : 99

Processing reads in plasmidpart.fastq.

 -- Processing 10 million reads

Number of reads in file plasmidpart.fastq : 37500

 

The input run parameters are: 

 -- Barcode: start position 1    end position 9  length 9

 -- Hairpin: start position 34   end position 53         length 20

 -- Allow hairpin sequences to be matched to a shifted position, <= 2 base left or right of the specified positions. 

 -- Allow sequence mismatch, <= 1 base in barcode sequence and <= 2 base in hairpin sequence. 

 

Total number of read is 37500 

There are 37347 reads (99.5920 percent) with barcode matches

There are 31 reads (0.0827 percent) with hairpin matches

There are 31 reads (0.0827 percent) with both barcode and hairpin matches

   user  system elapsed 

  0.763   0.015   1.175

 

> sessionInfo()

R version 3.2.0 (2015-04-16)

Platform: x86_64-unknown-linux-gnu (64-bit)

Running under: CentOS release 6.4 (Final)

 

locale:

 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C              

 [3] LC_TIME=en_AU.UTF-8        LC_COLLATE=en_AU.UTF-8    

 [5] LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   

 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                 

 [9] LC_ADDRESS=C               LC_TELEPHONE=C            

[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

 

attached base packages:

[1] stats     graphics  grDevices utils     datasets  methods   base     

 

other attached packages:

[1] edgeR_3.11.5 limma_3.24.4

 

loaded via a namespace (and not attached):

[1] tools_3.2.0