Using AMI with StarCluster - error authenticating
2
0
Entering edit mode
spollack • 0
@spollack-12387
Last seen 5.4 years ago
Biostatistics and Computational Biology…

Dear Bioconductor,

I am attempting to follow the instructions here https://www.bioconductor.org/help/bioconductor-cloud-ami/#configuring_starcluster to configure a StarCluster for running the Bioconductor AMI as a cluster.

But I am getting an error when I try to start a cluster. The error is a Python stack trace which seems to indicate AWS cloud refused to authenticate my key. I tried creating a key with "starcluster createkey" and the same error turned up.

(I couldn't create a key, so I tried using my regular AWS key and neither seems to work.)

Is this a StarCluster problem? Should I send them a question? Am I think only one it happens to? Do you still recommend using StarCluster? Is it still an active project? (The fact that I had to install Python 2.7 to build it is not encouraging.) Is there a newer better way to do this?

thanks,
- Sam

 

 

AMI • 2.2k views
ADD COMMENT
0
Entering edit mode
nitesh.turaga ▴ 140
@niteshturaga-8667
Last seen 2.2 years ago
United States

Hi Sam,

I suspect a configuration error, could you post the stack trace or your starcluster config file (without AWS secret/access keys). 

 

ADD COMMENT
0
Entering edit mode

Sorry, just saw this. I got the AMI to work with the latest StarCluster version 0.95.6 yesterday.

But now I have a problem with connecting to Rstudio Server.

I can connect to rstudio server on a slave node just fine, but I can't connect to it on the master node. On the master node, I get the login screen and I enter ubuntu/bioc and it spins its wheel for a bit, then it says "Studio initialization error" and "Unable to connect to  service."

So I logged into the master node and tried to restart. But I am getting an error with the /etc/rstudio/rserver.conf file. When I do "rstudio-server verify-installation" it says:

"Error reading /etc/rstudio/rserver.conf: option 'path' cannot be specified more than once"

The issue is that there are multiple identical lines in /etc/rstudio/rserver.conf that look like this:

rsession-path=/usr/local/bin/setup_r_mpi.sh

(It looks like every time I attempt to connect, a few more of these lines show up.)

Do you have any insight on this? 

thanks,
- Sam

 


 


 

 

 

 

ADD REPLY
0
Entering edit mode
Dan Tenenbaum ★ 8.2k
@dan-tenenbaum-4256
Last seen 3.1 years ago
United States

StarCluster is a little old. Look info CfnCluster: https://github.com/awslabs/cfncluster .

 

ADD COMMENT
0
Entering edit mode

Thanks for the advice. I tried to build a cluster with CfnCluster and the Bioconductor AMI but I never did get it to work. My best attempt was a config file with this:

master_instance_type = t2.large
slave_instance_type = t2.large
master_root_volume_size = 500
slave_root_volume_size = 500
base_os = ubuntu1404
custom_ami = ami-abd0b3bc
 

But something is still generating an error when the master node is instantiated. CfnCluster fills in values to a four-thousand-line CloudFormation template. Somewhere in that template is the reason my master node is not instantiating. I sure would like to get it to work, though, because it looks like StarCluster is kind of moribund. 
 

ADD REPLY
0
Entering edit mode

Since the last time I contributed to this thread, I've become aware of AWS Batch. You might want to look at that.

Whatever you do, I'd focus on keeping things simple. If you try to get CfnCluster to work, I'd start by just trying to get a simple cluster to work (with a basic AMI like one of the Alestic Ubuntu 16.04 AMIs). Then maybe install Docker on the AMI and run your job as a simple batch command using the Bioconductor Docker containers.

 

 

ADD REPLY
0
Entering edit mode

OK, thanks, good idea (Bioconductor Docker), I'll try that. 
- S

ADD REPLY
0
Entering edit mode

We are finding that cfncluster creation of clusters in EC2 is extremely slow, relative to the solution available a few years through starcluster.  Any other approaches in use in our community?

ADD REPLY

Login before adding your answer.

Traffic: 764 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6