Wednesday, December 26, 2012

SSH connection refused on Amazon EC2 instance

Does this sound familiar?
ssh: connect to host ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com
port 22: Connection refused
A few days back, one of my friends was having an issue with his Amazon EC2 production instance. He was able to SSH into it a few days back, but all of a sudden, he was not able to get into it.

Now thats a big problem. You cannot do anything with an amazon EC2 instance if you cannot SSH into it. Well, i started troubleshooting.
First thing i tried was:

telnet hostname 22
which gave me an error. That tells me that the port is not open. That means sshd has crashed for some reason or is not running.

Then my friend recalls that he rebooted the instance a day back. This led me into a different direction. I stopped the instance, detached the EBS root volume from it, fired up a temporary micro instance, attached the bad instance's root volume to this temporary instance and mounted it. Unluckily, the logs did not tell me anything as SSH debugging was not turned on. I started to look around in various config files that run on boot. A glance at /etc/fstab file led me to my solution to this problem.

The instance was not able to mount his secondary EBS volume for some reason. Well, but that should not have anything to do with SSH. Should it? Well, lets see. Further diving into
/etc/init/sshd.conf
to see what the SSHD startup process is. Looking at the first few lines gave me my answer.


"OpenSSH server"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]
So this upstart file tells me that SSHD service starts after all entries in /etc/fstab have been mounted.
I commented the lines in fstab file, detached the volume, attached it back to the original instance and booted it up. Boom...SSH starts working again.

Morale of the story

Its not a good idea to mount your EBS volumes in /etc/fstab file. A better option is to make an RC init script that mounts an EBS volume for you. That way you will be able to SSH into the instance in case of failiure.

I hope you guys enjoyed this post.

5 comments:

  1. Replies
    1. I have got the same problem and followed the given procedure but i couldn't see any changes can you please help me?

      Delete
  2. Please email me the exact nature of your problem. I will try to do my best.

    ReplyDelete