Question: Using AMI with StarCluster - error authenticating
gravatar for spollack
20 months ago by
Biostatistics and Computational Biology, Dana-Farber Cancer Institute, Boston
spollack0 wrote:

Dear Bioconductor,

I am attempting to follow the instructions here to configure a StarCluster for running the Bioconductor AMI as a cluster.

But I am getting an error when I try to start a cluster. The error is a Python stack trace which seems to indicate AWS cloud refused to authenticate my key. I tried creating a key with "starcluster createkey" and the same error turned up.

(I couldn't create a key, so I tried using my regular AWS key and neither seems to work.)

Is this a StarCluster problem? Should I send them a question? Am I think only one it happens to? Do you still recommend using StarCluster? Is it still an active project? (The fact that I had to install Python 2.7 to build it is not encouraging.) Is there a newer better way to do this?

- Sam



ADD COMMENTlink modified 20 months ago by Dan Tenenbaum ♦♦ 8.2k • written 20 months ago by spollack0
gravatar for nitesh.turaga
20 months ago by
United States
nitesh.turaga130 wrote:

Hi Sam,

I suspect a configuration error, could you post the stack trace or your starcluster config file (without AWS secret/access keys). 


ADD COMMENTlink modified 20 months ago • written 20 months ago by nitesh.turaga130

Sorry, just saw this. I got the AMI to work with the latest StarCluster version 0.95.6 yesterday.

But now I have a problem with connecting to Rstudio Server.

I can connect to rstudio server on a slave node just fine, but I can't connect to it on the master node. On the master node, I get the login screen and I enter ubuntu/bioc and it spins its wheel for a bit, then it says "Studio initialization error" and "Unable to connect to  service."

So I logged into the master node and tried to restart. But I am getting an error with the /etc/rstudio/rserver.conf file. When I do "rstudio-server verify-installation" it says:

"Error reading /etc/rstudio/rserver.conf: option 'path' cannot be specified more than once"

The issue is that there are multiple identical lines in /etc/rstudio/rserver.conf that look like this:


(It looks like every time I attempt to connect, a few more of these lines show up.)

Do you have any insight on this? 

- Sam







ADD REPLYlink modified 19 months ago • written 19 months ago by spollack0
gravatar for Dan Tenenbaum
20 months ago by
Dan Tenenbaum ♦♦ 8.2k
United States
Dan Tenenbaum ♦♦ 8.2k wrote:

StarCluster is a little old. Look info CfnCluster: .


ADD COMMENTlink written 20 months ago by Dan Tenenbaum ♦♦ 8.2k

Thanks for the advice. I tried to build a cluster with CfnCluster and the Bioconductor AMI but I never did get it to work. My best attempt was a config file with this:

master_instance_type = t2.large
slave_instance_type = t2.large
master_root_volume_size = 500
slave_root_volume_size = 500
base_os = ubuntu1404
custom_ami = ami-abd0b3bc

But something is still generating an error when the master node is instantiated. CfnCluster fills in values to a four-thousand-line CloudFormation template. Somewhere in that template is the reason my master node is not instantiating. I sure would like to get it to work, though, because it looks like StarCluster is kind of moribund. 

ADD REPLYlink written 19 months ago by spollack0

Since the last time I contributed to this thread, I've become aware of AWS Batch. You might want to look at that.

Whatever you do, I'd focus on keeping things simple. If you try to get CfnCluster to work, I'd start by just trying to get a simple cluster to work (with a basic AMI like one of the Alestic Ubuntu 16.04 AMIs). Then maybe install Docker on the AMI and run your job as a simple batch command using the Bioconductor Docker containers.



ADD REPLYlink written 19 months ago by Dan Tenenbaum ♦♦ 8.2k

OK, thanks, good idea (Bioconductor Docker), I'll try that. 
- S

ADD REPLYlink written 19 months ago by spollack0

We are finding that cfncluster creation of clusters in EC2 is extremely slow, relative to the solution available a few years through starcluster.  Any other approaches in use in our community?

ADD REPLYlink written 19 months ago by Vincent J. Carey, Jr.6.2k
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.2.0
Traffic: 230 users visited in the last hour