Header RSS Feed
 
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

AWS EC2 instance unreachable after reboot ([Errno 101])
Friday - Jun 16th 2017 - by - (0 comments)

Already two weeks ago I had a problem with a newly installed EC2 instance launched from an AMI image on the AWS cloud (Amazon Web Services). After I rebooted the instance, an updated Ubuntu 16.04 Xenial machine, the system didn't come up anymore. Even the ping from another instance in the same subnet didn't work. 

Today I came across the same problem. Different region (this time Europe-Ireland, the one from two weeks ago was Europe-Frankfurt), different VPC, different network segments, but the exact same problem. The instance didn't come up anymore after a reboot. Just prior to this instance I have set up two other instances the exact same way (except a different Availability Zone was used) and neither of these had a problem after a reboot.

By using Instance "Settings -> Get System Log" I was able to get some information whats going on:

 [[0;32m  OK  [0m] Reached target Network.
[    7.511380] cloud-init[770]: Cloud-init v. 0.7.9 running 'init' at Fri, 16 Jun 2017 11:46:18 +0000. Up 7.41 seconds.
[    7.515990] cloud-init[770]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[    7.520087] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.524021] cloud-init[770]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
[    7.528206] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.532219] cloud-init[770]: ci-info: |  ens3  | False |     .     |     .     |   .   | 0a:76:72:96:c7:52 |
[    7.536186] cloud-init[770]: ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
[    7.540244] cloud-init[770]: ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |
[    7.544243] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.548428] cloud-init[770]: 2017-06-16 13:46:18,547 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[    8.514438] cloud-init[770]: 2017-06-16 13:46:19,550 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[    9.517358] cloud-init[770]: 2017-06-16 13:46:20,553 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[   10.520266] cloud-init[770]: 2017-06-16 13:46:21,555 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [3/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[... messages repeating ...]

For some reason, the network interface ens3 didn't come up:

| Device |   Up  |  Address  |    Mask   | Scope | Hw-Address        |  
|  ens3  | False |     .     |     .     |   .   | 0a:76:72:96:c7:52 |

No reason whatsoever in the log file.

Even in the EC2 administration UI the status check was at 1/2, so something's definitely not working correctly:

AWS EC2 instance unreachable after reboot

As there is no console terminal access in AWS, I had no other choices than to:

  • reboot the instance in the EC2 administration -> didn't help
  • stop the instance, then start the instance -> didn't help
  • stop the instance, change the instance type -> didn't help

All the tools given by AWS didn't help. As the instance is not usable anymore, I terminated it and created a new instance from the same AMI image. Same instance type, same network settings, everything exactly the same. This time, the instance was able to reboot with no problem whatsoever. I came across a question on serverfault (amazon ec2 - Unable to connect to EC2 instance after "reboot") which describes the same problem as I experienced (already twice now in two weeks), but unfortunately without solution. For now the most plausible explanation to me is a bug in the AWS infrastructure which hits randomly (?). I didn't find a way to reproduce this problem.

Summary: You rebooted your EC2 instance (from within the guest OS)? It doesn't come up afterwards? Even pings from the same subnet don't work? You get the same message as above in the system log? Recreate the instance from scratch. :-(

 

Add a comment

Show form to leave a comment

Comments (newest first):

No comments yet.

Go to Homepage home
Linux Howtos how to's
Monitoring Plugins monitoring plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

7423 Days
until Death of Computers
Why?