AWS EC2 instance unreachable after reboot ([Errno 101])

Written by - 4 comments

Published on - Listed in Linux Internet AWS


Already two weeks ago I had a problem with a newly installed EC2 instance launched from an AMI image on the AWS cloud (Amazon Web Services). After I rebooted the instance, an updated Ubuntu 16.04 Xenial machine, the system didn't come up anymore. Even the ping from another instance in the same subnet didn't work. 

Today I came across the same problem. Different region (this time Europe-Ireland, the one from two weeks ago was Europe-Frankfurt), different VPC, different network segments, but the exact same problem. The instance didn't come up anymore after a reboot. Just prior to this instance I have set up two other instances the exact same way (except a different Availability Zone was used) and neither of these had a problem after a reboot.

By using Instance "Settings -> Get System Log" I was able to get some information whats going on:

 [[0;32m  OK  [0m] Reached target Network.
[    7.511380] cloud-init[770]: Cloud-init v. 0.7.9 running 'init' at Fri, 16 Jun 2017 11:46:18 +0000. Up 7.41 seconds.
[    7.515990] cloud-init[770]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[    7.520087] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.524021] cloud-init[770]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
[    7.528206] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.532219] cloud-init[770]: ci-info: |  ens3  | False |     .     |     .     |   .   | 0a:76:72:96:c7:52 |
[    7.536186] cloud-init[770]: ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
[    7.540244] cloud-init[770]: ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |
[    7.544243] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.548428] cloud-init[770]: 2017-06-16 13:46:18,547 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[    8.514438] cloud-init[770]: 2017-06-16 13:46:19,550 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[    9.517358] cloud-init[770]: 2017-06-16 13:46:20,553 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[   10.520266] cloud-init[770]: 2017-06-16 13:46:21,555 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [3/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[... messages repeating ...]

For some reason, the network interface ens3 didn't come up:

| Device |   Up  |  Address  |    Mask   | Scope | Hw-Address        |  
|  ens3  | False |     .     |     .     |   .   | 0a:76:72:96:c7:52 |

No reason whatsoever in the log file.

Even in the EC2 administration UI the status check was at 1/2, so something's definitely not working correctly:

AWS EC2 instance unreachable after reboot

As there is no console terminal access in AWS, I had no other choices than to:

  • reboot the instance in the EC2 administration -> didn't help
  • stop the instance, then start the instance -> didn't help
  • stop the instance, change the instance type -> didn't help

All the tools given by AWS didn't help. As the instance is not usable anymore, I terminated it and created a new instance from the same AMI image. Same instance type, same network settings, everything exactly the same. This time, the instance was able to reboot with no problem whatsoever. I came across a question on serverfault (amazon ec2 - Unable to connect to EC2 instance after "reboot") which describes the same problem as I experienced (already twice now in two weeks), but unfortunately without solution. For now the most plausible explanation to me is a bug in the AWS infrastructure which hits randomly (?). I didn't find a way to reproduce this problem.

Summary: You rebooted your EC2 instance (from within the guest OS)? It doesn't come up afterwards? Even pings from the same subnet don't work? You get the same message as above in the system log? Recreate the instance from scratch. :-(


Add a comment

Show form to leave a comment

Comments (newest first)

Sergio from wrote on May 30th, 2022:

Julio's workaround worked great, thanks!.


Julio Lemus from GT wrote on Apr 20th, 2022:

How about I hope it helps you, I had to restart my instance and I had the same error, and I couldn't access it either by ssh, or by serial console, I didn't have any access to the instance and I did the following:
I tell you that I had to restart my instance and I had the same error, I restarted several times and it did not pass that check, what I did was stop the instance and then add an interface with private IPv4, turned it on it did not work, I stopped it again and I disassociated the interface, I stopped the instance again for 5 minutes and then started it and it gave me a successful result, it worked for me and passed the verification of the two states


ck from St. Gallen wrote on Sep 2nd, 2019:

Thanks, Sushant, for that hint. Your suggestion might help for systems using netplan, but in my case it was a Ubuntu 16.04 which didn't have netplan as network management then.


Sushant from wrote on Sep 2nd, 2019:

I was facing the same issue. I solved it by editing /etc/netplan/50-cloud-init.yaml

And pasting following code:

network:
ethernets:
ens3:
dhcp4: true
version: 2

This will give dhcp server to set IPv4 settings.


RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Icingaweb   Icingaweb2   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder   


Update cookies preferences