Hi Nikolai,
Here are the steps I followed to get a function running via BiocParallel (using SGE) . Please try to mimic this and tell us if you’re getting expected output or what’s failing:

# Make a directory to contain a Python virtual environment :
mkdir starcluster-experiment
cd starcluster-experiment
# Create a virtual environment for StarCluster
virtualenv venv
# Activate the virtualenv 
source venv/bin/activate

# Install StarCluster
sudo easy_install StarCluster

# Modify StarCluster configuration ( ~/.starcluster/config ) with the appropriate values.
vim ~/.starcluster/config

#
# I only needed to modify the following lines: 
#
# Format is <line number="">:<text> 

4:[global]
8:DEFAULT_TEMPLATE=smallcluster
17:
21:[aws info]
25:AWS_ACCESS_KEY_ID = << CHANGE TO YOUR ACCESS KEY ID >>
26:AWS_SECRET_ACCESS_KEY= << CHANGE TO YOUR SECRET ACCESS KEY >>
28:AWS_USER_ID= << CHANGE TO YOUR USER ID, SHOULD BE A NUMBER NOT A STRING OF CHARACTERS >>

# Note the name you gave your keypair from https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#KeyPairs:sort=desc:keyName 
# You should be using your ssh key ( ~/.ssh/id_rsa ) and you should have uploaded your public key to AWS ( ~/.ssh/id_rsa.pub )

49:[key keypair-brian]
50:KEY_LOCATION=~/.ssh/id_rsa
51:
72:
73:[cluster smallcluster]
75:KEYNAME = keypair-brian
77:CLUSTER_SIZE = 2
79:CLUSTER_USER = ubuntu
82:CLUSTER_SHELL = bash
87:DNS_PREFIX = True

# This may be a discrepancy, I used the devel version of Bioconductor.  Via: http://www.bioconductor.org/help/bioconductor-cloud-ami/#ami_ids 
93:NODE_IMAGE_ID = ami-1f2fe074

96:NODE_INSTANCE_TYPE = m3.medium

# Ensure StarCluster can use HTTP
129:permissions = http
138:
206:
208:[permission http]
209:IP_PROTOCOL = tcp
210:FROM_PORT = 80
211:TO_PORT = 80
212:



# Once all of that is configured, run the starcluster:
starcluster start smallcluster	

# Prove that the cluster is properly configured 
starcluster listclusters

# Your output should be like this, but it should show 50 nodes 
(venv)blong@work:~/Documents/Work/REPOS__git/b-long/starcluster-experiment$ starcluster listclusters
StarCluster - (http://star.mit.edu/cluster) (v. 0.95.6)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

-----------------------------------------------
smallcluster (security group: @sc-smallcluster)
-----------------------------------------------
Launch time: 2015-10-06 14:37:10
Uptime: 0 days, 02:08:32
Zone: us-east-1e
Keypair: keypair-brian
EBS volumes: N/A
Cluster nodes:
    smallcluster-master running i-e738fc45 ec2-54-158-152-91.compute-1.amazonaws.com
    smallcluster-node001 running i-e838fc4a ec2-54-145-219-171.compute-1.amazonaws.com
Total nodes: 2


# Connect to the master node as the Ubuntu user
starcluster sshmaster --user=ubuntu smallcluster

# You should be able to start R and run the following (although be sure to use 32 CPUs rather than 1) : 
ubuntu@smallcluster-master:~$ R

> library(BatchJobs)
> library(BiocParallel)
> param <- BatchJobsParam(2, resources=list(ncpus=1))
> register(param)
> FUN <- function(i) system("hostname", intern=TRUE)
> xx <- bplapply(1:100, FUN)
> table(unlist(xx))


Output should be similar to : 

> library(BatchJobs)
Loading required package: BBmisc
Sourcing configuration file: '/home/ubuntu/R-libs/BatchJobs/etc/BatchJobs_global_config.R'
Sourcing configuration file: '/home/ubuntu/.BatchJobs.R'
BatchJobs configuration:
  cluster functions: SGE
  mail.from: 
  mail.to: 
  mail.start: none
  mail.done: none
  mail.error: none
  default.resources: 
  debug: FALSE
  raise.warnings: FALSE
  staged.queries: TRUE
  max.concurrent.jobs: Inf
  fs.timeout: NA

> library(BiocParallel)
'BiocParallel' did not register default BiocParallelParams:
  invalid class “SnowParam” object: 'workers' must be integer(1) and >= 0
> param <- BatchJobsParam(2, resources=list(ncpus=1))
> register(param)
> FUN <- function(i) system("hostname", intern=TRUE)
> xx <- bplapply(1:100, FUN)
SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++++| 100% (00:00:00)
Waiting [S:0 D:100 E:0 R:0] |++++++++++++++++++++++++++++++++++| 100% (00:00:00)0)

> table(unlist(xx))

 smallcluster-master smallcluster-node001 
                  50                   50 

Hi Dan,

thank you very much for your help. I will try out a SSH cluster, the next time I am working with my StarCluster+Bioconductor setup, and report any issues.

Regarding the SGE issue I reported above: I was in a lot of hurry to finish my simulation on that day, as they were part of my now finished Bachelor Thesis and I was way behind on schedule already. So I implemented an ad hoc version in which I made the BatchJobs parameter connect to only one core on all of the 50 instances and then on those 50 cores (each on a seperate instance) start a function which uses the foreach (from the foreach package) function made to work on the 32 cores on every machine. Here is an example code to understand it better:

library(BatchJobs)
library(BiocParallel)
param <- BatchJobsParam(50, resources=list(ncpus=1))
FUN <- function(X) {
  library(foreach)
  library(doParallel)
  cl<-makeCluster(detectCores())
  registerDoParallel(cl)
  answer2 <- foreach(1:X} %dopar% {
     ### Actual Calculations
}}
answer <- bplapply(1:1600, FUN, BPPARAM=param)

Of course, I made sure that detectCores() actually detected all of the 32 cores on the 50 machines - which it did.

To my suprise it worked. Some hasty benchmarking showed that it was significantly faster than just using

param <- BatchJobsParam(50, resources=list(ncpus=32))

and the CPU workload was at 100%. All this information is a bit subjective and not 100% indicative that all the 1600 cores are working, but it had to work for me and it did. One definit caveat lies in the functionality of

BatchJobsParam(50, resources=list(ncpus=32))

As if it does not work as intended in recognising the 32 cores on the 50 machines, I cannot guarantee that it works perfectly with my ad hoc setup. Not that I advise anyone to use it.