I would like to do pfam enrichment analysis using Fishers.test function in R. I have ~2500 Pfam domains and want to compare counts across 10 taxa, and want to calculate pvalues of each Pfam domain. The error is that "out of workspace", how do I increase the workspace? 

I am not sure there is an error that says you are out of workspace. Anyway, your question has nothing to do with Bioconductor packages, so perhaps you should ask on R-help (r-help@r-project.org). If you do so, make sure to include your code and the actual error message.