You likely may be running an outdated version of Bioconductor, based on the R version you reported. You should consider updating to the currently supported Bioconductor version, and hence of its packages. With the current version of SomaticSignatures, I cannot reproduce the error message that you report.
In your example, it seems that you have two defined 2 groups, but want to decompose the motif matrix into 5 signatures. Please note that the upper bound for the number of possible signatures is the minimum of the rows and columns of the motif matrix M, i.e. in your case 2 due to the 2 groups. This is by design of the underlying approach, and its matrix decomposition. You can have a look at the package vignette, and especially at the cited papers to learn about the background of this.
What is the output of your sessionInfo?
This is the required output of sessioninfo -
> sessionInfo()
R version 3.2.4 (2016-03-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.4 LTS
locale:
[1] LC_CTYPE=en_IN LC_NUMERIC=C LC_TIME=en_IN
[4] LC_COLLATE=en_IN LC_MONETARY=en_IN LC_MESSAGES=en_IN
[7] LC_PAPER=en_IN LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=en_IN LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
Swati