Header RSS Feed
 
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Nagios/Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

What are the tasks of a DevOp engineer?
Wednesday - May 27th 2015 - by - (0 comments)

I was asked that question today and these are the tasks I understand to be my duty as a DevOp engineer.

- systems administration
- security
- operations and monitoring
- (re-) engineering of infrastructure and system environments
- automation
- scripting/coding
- understanding applications from developers
- make developers understand security and systems
- being the bridge between code and system
- quality assurance
- use Agile methods to build and deploy new systems (we're talking about Continuous Deployments here)
- use centralized/programmable configuration management (Ansible, chef, puppet, ..)
- troubleshoot incidents (performance issues, hack attacks, etc)
- write and review documentations

As mentioned above, a DevOp is the bridge between the code and the systems. This image (found on http://stackify.com/defining-the-ops-in-devops/) shows that part very well:

Devops

 

Network connectivity problems when running LXC (with veth) in VMware VM
Tuesday - May 26th 2015 - by - (0 comments)

And then there was this saying "you never stop learning". Indeed.

I've been using VMware since 2005 and Linux containers (LXC) since 2013 and have managed hundreds of machines, VM's running on VMware ESX/i and containers. But I had yet to combine the two of them and run containers on a VMware VM. And promptly I ran into problems.

Let me first explain the (basic) setup:

VMware VM1 (.51)          VMware VM2 (.52)
    |_ lxc1 (.53)              |_ lxc3 (.54)   
    |_ lxc2 (.55)              |_ lxc4 (.56)

There is one interface configured on the VM which is declared as virbr0 and is being used by the containers as link. The VM itself has its ip address configured on the virbr0, so there is this communication as well. The virtual switch was already configured to allow multiple MAC addresses behind an interface (see Problems on LXC in same network as its host which is a VMware VM).

The vms have no connectiviy issue at all and are able to communicate correctly with each other. However as soon as the connection should go to (or from) an lxc container, there are connectivity issues.

Ping tests from VM's (lxc hosts):

VM1 -> VM2  : success
VM2 -> VM1  : success
VM1 -> lxc1 : fail (12 packets transmitted, 0 received, 100% packet loss, time 11087ms)
VM2 -> lxc1 : fail (12 packets transmitted, 0 received, 100% packet loss, time 10999ms)
VM1 -> lxc2 : success (12 packets transmitted, 12 received, 0% packet loss, time 10999ms)
VM2 -> lxc2 : success (12 packets transmitted, 12 received, 0% packet loss, time 10997ms)
VM1 -> lxc3 : fail (12 packets transmitted, 0 received, 100% packet loss, time 11086ms)
VM2 -> lxc3 : fail (12 packets transmitted, 2 received, 83% packet loss, time 11070ms)
VM1 -> lxc4 : fail (13 packets transmitted, 0 received, 100% packet loss, time 12095ms)
VM2 -> lxc4 : fail (13 packets transmitted, 4 received, 69% packet loss, time 11998ms)

This is very surprising. Even the VMs' which is the host of its containers gets packet losses on them.
An exception is lxc2. For some reason the connectivity there works.

arp -a shows the following:

VM1:
? (192.168.253.56) at 00:16:3e:8f:5d:39 [ether] on virbr0
? (192.168.253.52) at 00:50:56:99:06:b8 [ether] on virbr0
? (192.168.253.53) at 00:16:3e:bf:72:12 [ether] on virbr0
? (192.168.253.54) at 00:16:3e:ef:70:14 [ether] on virbr0
? (192.168.253.55) at 00:16:3e:bf:76:71 [ether] on virbr0

VM2:
? (192.168.253.54) at 00:16:3e:ef:70:14 [ether] on virbr0
? (192.168.253.51) at 00:50:56:99:55:89 [ether] on virbr0
? (192.168.253.56) at 00:16:3e:8f:5d:39 [ether] on virbr0
? (192.168.253.53) at 00:16:3e:bf:72:12 [ether] on virbr0
? (192.168.253.55) at 00:16:3e:bf:76:71 [ether] on virbr0

So the arp resolving worked...

A completely different view is when the pings are launched from within a container (tested on lxc1 and lxc3). Here we even get ping errors:

lxc1 -> VM1 : success
lxc3 -> VM1 : fail (12 packets transmitted, 0 received, +12 errors, 100% packet loss, time 10999ms)
lxc1 -> VM2 : success
lxc3 -> VM2 : success
lxc1 -> lxc3: fail (12 packets transmitted, 0 received, +12 errors, 100% packet loss, time 11055ms)
lxc3 -> lxc1: fail (12 packets transmitted, 0 received, 100% packet loss, time 11086ms)
lxc1 -> lxc2: fail (12 packets transmitted, 0 received, 100% packet loss, time 11063ms)
lxc3 -> lxc2: fail (12 packets transmitted, 0 received, 100% packet loss, time 10999ms)
lxc1 -> lxc4: fail (12 packets transmitted, 0 received, +12 errors, 100% packet loss, time 11004ms)
lxc3 -> lxc4: success

The arp table on the containers lxc1 and lxc3 showed incomplete entries:

lxc1:
? (192.168.253.56) at on eth0
? (192.168.253.51) at 00:50:56:99:55:89 [ether] on eth0
? (192.168.253.52) at 00:50:56:99:06:b8 [ether] on eth0
? (192.168.253.55) at 00:16:3e:bf:76:71 [ether] on eth0
? (192.168.253.54) at 00:16:3e:ef:70:14 [ether] on eth0

lxc3:
? (192.168.253.53) at 00:16:3e:bf:72:12 [ether] on eth0
? (192.168.253.52) at 00:50:56:99:06:b8 [ether] on eth0
? (192.168.253.51) at on eth0
? (192.168.253.56) at 00:16:3e:8f:5d:39 [ether] on eth0
? (192.168.253.55) at 00:16:3e:bf:76:71 [ether] on eth0

I expected an issue with the arp announcements from the host (vm) to the network. According to the kernel documentation of arp_accept and arp_notify I did another set of tests where I set both values to 1:

sysctl -w net.ipv4.conf.all.arp_accept=1
sysctl -w net.ipv4.conf.all.arp_notify=1

But this didn't change anything either. There were still packet losses, however some LXC's responded to the ping. Meaning that communication worked.
I also saw very strange results where I launched another set of ping tests and the first few pings failed, then suddenly the ping worked. After I waited a few seconds and re-launched the same ping, again packet losses. Call that a stable network communication...

After a lot of troubleshooting, it seems that it actually has nothing to do with the kernel settings for arp but that this is rather a limitation of ESXi to accept bridges within the guest vm. After a while, I finally found some important information in an old LXC-Users mailing list post.
The author of the post, Olivier Mauras, mentions that he needed to use MACVLAN (instead of veth) virtual interfaces for the containers and that he needed to configure two interfaces on the LXC Host (the vm). The first nic is used for the VM communication itself, the second nic is used for the bridge.

Good hint. Let's try it with MACVLAN interfaces then.
The original configuration used veth interfaces and used virbr0 (the vm's one and only nic) as link:

grep network /var/lib/lxc/lxc3/config
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = virbr0
lxc.network.ipv4 = 192.168.168.54/24
lxc.network.hwaddr = 00:16:3e:ef:70:54
lxc.network.ipv4.gateway = 192.168.168.1

I added a new virtual nic to the VM and edited /etc/network/interfaces in the VM. The virtual bridge should be using eth1 (the new added nic).

# The secondary network interface is used as bridge
iface eth1 inet manual

# Virtual bridge
auto virbr1
iface virbr1 inet manual
    bridge_ports eth1
    # bridge control
    pre-up brctl addbr virbr1
    pre-down brctl delbr virbr1

Then I modified the LXC configurations to use the macvlan interface type and define macvlan as bridge modus and use the newly created virbr1:

grep network /var/lib/lxc/lxc3/config
lxc.network.type = macvlan
lxc.network.macvlan.mode = bridge
lxc.network.flags = up
lxc.network.link = virbr1
lxc.network.ipv4 = 192.168.168.54/24
lxc.network.hwaddr = 00:16:3e:ef:70:54
lxc.network.ipv4.gateway = 192.168.168.1

After a reboot of the vms, make the ping tests again:

VM1 -> VM2  : success
VM2 -> VM1  : success
VM1 -> lxc1 : success
VM2 -> lxc1 : success
VM1 -> lxc2 : success
VM2 -> lxc2 : success
VM1 -> lxc3 : success
VM2 -> lxc3 : success
VM1 -> lxc4 : success
VM2 -> lxc4 : success

From the vms it looks good - all containers were pingable. Now inside the containers:

lxc1 -> VM1  : success
lxc1 -> VM2  : success
lxc3 -> VM1  : success
lxc3 -> VM2  : success
lxc1 -> lxc3 : success
lxc3 -> lxc1 : success
lxc1 -> lxc2 : success
lxc3 -> lxc2 : success
lxc1 -> lxc4 : success
lxc3 -> lxc4 : success

Do not use veth interfaces on LXC in VM 

Finally!! All containers are now reachable and the communications between them work as it should.


Now I should get a tattoo on my arm saying "Do not use veth interfaces when running Linux Containers on VMware".

That will spare me of several hours troubleshooting next time.


 

Allow user to execute eval() command on MongoDB 3.x
Tuesday - May 26th 2015 - by - (0 comments)

User wants to execute eval functions in mongodb, but doesnt have the rights and got the following error message:

"errmsg" : "not authorized on LOG to execute command { $eval: \"Test(2)\" }",

The mongodb user has the following roles assigned:

> use LOG
switched to db LOG
> db.getUser("dbuser")
{
    "_id" : "LOG.dbuser",
    "user" : "dbuser",
    "db" : "LOG",
    "roles" : [
        {
            "role" : "readWrite",
            "db" : "LOG"
        },
        {
            "role" : "dbAdmin",
            "db" : "LOG"
        },
        {
            "role" : "dbOwner",
            "db" : "LOG"
        }
    ]
}

But that's not enough. According to the MongoDB documentation for db.eval command:

If authorization is enabled, you must have access to all actions on all resources in order to run eval. Providing such access is not recommended, but if your organization requires a user to run eval, create a role that grants anyAction on anyResource.

The "anyAction" privilege action is, next to "internal", the only privilege action which isn't assigned to a Built-In Role.
To be able to assign this privilege action to the final user, a new role must be created and this new role has to be assigned to the db user.

So lets create this new role, which I called "executeFunctions". It is important to create this role in the admin db, because the role requires "anyAction" to "anyResource" as stated in the documentation.
Therefore the role cannot be created on the user's db (LOG) but must be created in the admin db.

> use admin
switched to db admin
> db.createRole( { role: "executeFunctions", privileges: [ { resource: { anyResource: true }, actions: [ "anyAction" ] } ], roles: [] } )
{
    "role" : "executeFunctions",
    "privileges" : [
        {
            "resource" : {
                "anyResource" : true
            },
            "actions" : [
                "anyAction"
            ]
        }
    ],
    "roles" : [ ]
}

Afterwards I assigned this new role as an additional role to the db user.
Important here is also that the new role must be assigned on the admin database (you'd get an error otherwise):

> use LOG
switched to db LOG
> db.grantRolesToUser("dbuser", [ { role: "executeFunctions", db: "admin" } ])

The new created role "executeFunctions" should be assigned to the db user now. Verification with db.getUser again:

> db.getUser("dbuser")
{
    "_id" : "LOG.dbuser",
    "user" : "dbuser",
    "db" : "LOG",
    "roles" : [
        {
            "role" : "executeFunctions",
            "db" : "admin"
        },
        {
            "role" : "readWrite",
            "db" : "LOG"
        },
        {
            "role" : "dbAdmin",
            "db" : "LOG"
        },
        {
            "role" : "dbOwner",
            "db" : "LOG"
        }
    ]
}

Voilą. Afterwards the db user was able to run eval.

 

HAProxy: Forward based on string in url combined with existing acl
Friday - May 22nd 2015 - by - (0 comments)

I got a special request to configure a HAProxy load balancer to use a different backend when a certain string was found in the URL.
There are already some very good answers how to do that (for example http://serverfault.com/questions/127491/haproxy-forward-to-a-different-web-server-based-on-uri) but I had to adapt the config for my environment which includes a combination of already existing acl's.

To show the current architecture:

test.example.com -> backend1 -> internalserver1:8888
stage.example.com -> backend2 -> internalserver2:8888

Because the HAProxy listens on the same IP address, both domains test.example.com and stage.example.com are handled by the same HAProxy. But depending on which domain was called, the requests should be forwarded to a different backend.
To be able to handle different domains, an acl which checks the requested host address does the job:

frontend main_http
    bind *:80

    # Define ACL based on host names
    acl test_host hdr(host) -i test.example.com
    acl stage_host hdr(host) -i stage.example.com

    # Default Route to normal backends
    use_backend backend_test_8888 if test_host
    use_backend backend_stage_8888 if stage_host

The ACL's "test_host" and "stage_host" check which host name was requested by using hdr(host). If it matches test.example.com, the acl "test_host" is set to true.
Depending which ACL is true (e.g. which host name was requested), a different backend is used (backend_test_8888 or backend_stage_8888).

Now I need an additional ACL which handles a special string (blog) in the requested URL. These requests should be sent to another backend.

test.example.com -> backend1 -> internalserver1:8888
test.example.com/bla/blog/page3 -> backend3 -> internalserver1:8889
stage.example.com -> backend2 -> internalserver2:8888
stage.example.com/bla/blog/page3 -> backend4 -> internalserver2:8889

In the serverfault.com example from above the answer was to use "path_beg" but that doesn't work for my case because I need to make sure that the string is not only at the beginning of the URI, it could appear literally anywhere in the whole URI.

To solve this, I created the following ACL which uses the sub(-string) rather than beg lookup method:

    # Define ACL based on string in URI
    acl blog_in_uri path_sub blog

So the ACL "blog_in_uri" is set to true, as soon as the string "blog" appears in the requested URI.

How to combine these two ACL's now? First I needed to create separate backends to handle the different destination ports:

#---------------------------------------------------------------------
# backend_test_8889
#---------------------------------------------------------------------
backend backend_test_8889
    balance     source
    server      intenralserver1 192.168.44.10:8889 check inter 2000 fall 1 rise 3
    server      intenralserver11 192.168.44.11:8889 check inter 2000 fall 1 rise 3

#---------------------------------------------------------------------
# backend_stage_8889
#---------------------------------------------------------------------
backend backend_stage_8889
    balance     source
    server      intenralserver2 192.168.44.20:8889 check inter 2000 fall 1 rise 3
    server      intenralserver22 192.168.44.22:8889 check inter 2000 fall 1 rise 3

Then I combined the ACL's in the "use_backend" syntax. Important is the following information from the HAProxy config documentation:

There may be as many "use_backend" rules as desired. All of these rules are evaluated in their declaration order, and the first one which matches will assign the backend.

This means that in my scenario I have to use the ACL combination (hostname + blog in uri) first. If it doesn't match, it will use the second ACL (hostname only).
To combine two or more ACL's just write both one after another. HAProxy will understand this as logical AND (do not add the word ADD into the config, this is considered a syntax error!):

frontend main_http
    bind *:80

    # Define ACL based on host names
    acl test_host hdr(host) -i test.example.com
    acl stage_host hdr(host) -i stage.example.com

    # Define ACL based on string in URI
    acl blog_in_uri path_sub blog

    # Special route which matches both hostname and string acl
    use_backend backend_test_8889 if test_host blog_in_uri
    use_backend backend_stage_8889 if stage_host blog_in_uri

    # Default Route to normal backends
    use_backend backend_test_8888 if test_host
    use_backend backend_stage_8888 if stage_host

Now all the requests which contain the string "blog" in the URI will be forwarded to a different backend rather than using the default backend for the requested domain.

 

Authentication on MongoDB 3.x fails with mechanism MONGODB-CR
Wednesday - May 20th 2015 - by - (0 comments)

On a fresh MongoDB 3.0.3 installation, with authentication enabled, a user needed to connect with a third party tool to his Mongo database. The connection itself worked, but the authentication didn't. The following error message appeared in the MongoDB log (/var/log/mongodb/mongod.log):

2015-05-20T11:32:28.797+0200 I NETWORK  [initandlisten] connection accepted from remote.example.com:58977 #70 (5 connections now open)
2015-05-20T11:32:28.803+0200 I ACCESS   [conn70]  authenticate db: LOG { authenticate: 1, nonce: "xxx", user: "dbuser", key: "xxx" }
2015-05-20T11:32:28.804+0200 I ACCESS   [conn70] Failed to authenticate dbuser@LOG with mechanism MONGODB-CR: AuthenticationFailed MONGODB-CR credentials missing in the user document
2015-05-20T11:32:28.805+0200 I NETWORK  [conn70] end connection remote.example.com:58977 (4 connections now open)

It seems that since MongoDB 3.0 there is a new authentication mechanism: SCRAM-SHA-1:

MongoDB’s default authentication method is a challenge and response mechanism (SCRAM-SHA-1).
Previously, MongoDB used MongoDB Challenge and Response (MONGODB-CR) as the default.

So this explains the log entries that MONGODB-CR credentials are missing.

Unfortunately the third party tool from the user did not support another authentication mechanism than MONGODB-CR, so it was required to change the mechanism back to the defaults of pre MongoDB 3.x.
This problem did not only occur to me, a MongoDB issue was recently (March 2015) opened (SERVER-17459: Users can't authenticate with authenticationMechanisms=MONGODB-CR) and proves that third party tools are not aware (yet) of a change of the authentication mechanism.

To my big surprise there seems to be lack of downward compatiblity, however if this would have been an older MongoDB 2.x server and it was upgraded to MongoDB 3.x, the db user would have been compatible to both mechanisms. Yet on a new and fresh install only SCRAM-SHA-1 works. Great... >.<

OK, back to solving the issue. The goal was to change the authentication mechanism from SCRAM-SHA-1 to MONGODB-CR.
Funnily the authentication mechanism (called authSchema in MongoDB) cannot be changed while authentication is active, the following error appeared when I tried:

"errmsg" : "not authorized on admin to execute command { update: \"system.version\", updates: [ { q: { _id: \"authSchema\" }, u: { _id: \"authSchema\", currentVersion: 3 }, multi: false, upsert: true } ], ordered: true }"

So I first disabled the authentication, restarted MongoDB server and then changed the authSchema value to 3 (which stands for MONGODB-CR or better said was the "old default"):

grep auth /etc/mongod.conf
#noauth = true
#auth = true
# path to a key file storing authentication info for connections

/etc/init.d/mongod restart

mongo
MongoDB shell version: 3.0.2
connecting to: test
> use admin
switched to db admin
var schema = db.system.version.findOne({"_id" : "authSchema"})
> schema.currentVersion = 3
3
> db.system.version.save(schema)
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> exit
bye

Now enabled authentication in /etc/mongod.conf and restart again:

sed -i "s/#auth/auth/" /etc/mongod.conf

grep auth /etc/mongod.conf
#noauth = true
auth = true
# path to a key file storing authentication info for connections

/etc/init.d/mongod restart

I actually thought this was it - but yet again the authentication failed:

2015-05-20T11:57:58.755+0200 I ACCESS   [conn7] Failed to authenticate dbuser@LOG with mechanism MONGODB-CR: AuthenticationFailed MONGODB-CR credentials missing in the user document

Back in MongoDB I verified if there was still some wrong reference to the SCRAM-SHA-1 mechanism, and indeed I found it:

> use admin
switched to db admin
> db.system.users.find()
[...]
{ "_id" : "LOG.dbuser", "user" : "dbuser", "db" : "LOG", "credentials" : { "SCRAM-SHA-1" : { "iterationCount" : 10000, "salt" : "XXXXXXXXXXXXXXXXXXXXXXXX", "storedKey" : "XXXXXXXXXXXXXXXXXXXXXXXXXXX", "serverKey" : "XXXXXXXXXXXXXXXXXXXXXXXXXXX" } }, "roles" : [ { "role" : "readWrite", "db" : "LOG" }, { "role" : "dbAdmin", "db" : "LOG" }, { "role" : "dbOwner", "db" : "LOG" } ] }

Because the default authentication mechanism is now set to MONGODB-CR, I decided to drop the user and recreate it:

> use LOG
switched to db LOG
> db.dropUser("dbuser")
true

> db.createUser ( { user: "dbuser", pwd: "XXXXXXXX", roles: [ "readWrite", "dbAdmin", "dbOwner" ] } )
Successfully added user: { "user" : "dbuser", "roles" : [ "readWrite", "dbAdmin", "dbOwner" ] }

The system user entry in the admin db looked a bit different now and contained the credentials with the MONGODB-CR schema:

> use admin
switched to db admin
> db.system.users.find()
[...]
{ "_id" : "LOG.dbuser", "user" : "dbuser", "db" : "LOG", "credentials" : { "MONGODB-CR" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" }, "roles" : [ { "role" : "readWrite", "db" : "LOG" }, { "role" : "dbAdmin", "db" : "LOG" }, { "role" : "dbOwner", "db" : "LOG" } ] }
> exit

After this change, the user was finally able to log in with the third party tool:

2015-05-20T12:57:22.219+0200 I NETWORK  [initandlisten] connection accepted from remote.example.com:63726 #9 (1 connection now open)
2015-05-20T12:57:22.226+0200 I ACCESS   [conn9]  authenticate db: LOG { authenticate: 1, nonce: "xxx", user: "dbuser", key: "xxx" }
2015-05-20T12:57:22.228+0200 I NETWORK  [conn9] end connection remote.example.com:63726 (0 connections now open)

Once again: Too bad MongoDB didn't supply a backward compatibility to support both authentication methods in Mongo 3.x.
Now I had to set the default to a lower security standard to guarantee compatibility with the user's third party tools.

 

Finally a good example of a long lasting SATA drive
Wednesday - May 20th 2015 - by - (0 comments)

Two years ago I wrote a post (Seagate ST3000DM001-9YN166: A catastrophic disk) about massive failings of a Seagate hard drive model. One of them didn't even last 4 months and already failed.

Now to a good example. In March 2010 I ordered two Western Digital Caviar Blue (7200rpm, 500GB, SATA-II) and used them together as RAID-1 on a self-built NAS server.

Now, five years later, one of the two drives failed. After a runtime of about 40'000 hours. That's quite a good life time for a SATA hard drive being online 24/7!

Here's the smartctl output of the failing drive:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Blue Serial ATA family
Device Model:     WDC WD5000AAKS-00V1A0
Serial Number:    XXXXXXXXXXXXX
Firmware Version: 05.01D05
User Capacity:    500,107,862,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Wed May 20 15:25:40 2015 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

[...]

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   177   138   021    Pre-fail  Always       -       2150
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       88
  5 Reallocated_Sector_Ct   0x0033   151   151   140    Pre-fail  Always       -       391
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   045   045   000    Old_age   Always       -       40361
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       84
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       29
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       56
194 Temperature_Celsius     0x0022   100   093   000    Old_age   Always       -       43
196 Reallocated_Event_Count 0x0032   001   001   000    Old_age   Always       -       359
197 Current_Pending_Sector  0x0032   001   001   000    Old_age   Always       -       32916
198 Offline_Uncorrectable   0x0030   199   194   000    Old_age   Offline      -       151
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   197   000    Old_age   Offline      -       13

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     40174         919347518

As you can see, the drive first re-allocated defect sectors and has no spare sectors left anymore. Another 32k defect sectors are pending to be reallocated - well, that won't happen anymore ;-).

Thanks to the RAID-1 configured in the NAS, everything continues to work. However it is strange that mdadm didn't detect the drive as failed. Both disks seem to be active in /proc/mdstat. Might be because of the old and unpatched SLES11 from anno 2010 though.

 

Problems on LXC in same network as its host which is a VMware VM
Monday - May 11th 2015 - by - (0 comments)

I almost spent the whole day debugging and trying to find the reason, why a plain simple LXC setup on Ubuntu 14..04 LTS wasn't working correctly. 

LXC itself was working fine, the container was created. But as soon as the container wanted to communicate with other servers in the same subnet, every connection attempt failed. Even the arp address could not be determinated.

On the LXC host, which is a virtual machine running on a VMware ESXi host, a very basic network configuration using the single network interface (eth0) as bridge was defined:

# The primary network interface
iface eth0 inet manual

# Virtual bridge
auto virbr0
iface virbr0 inet static
        address 192.168.168.51
        netmask 255.255.255.0
        network 192.168.168.0
        broadcast 192.168.168.255
        gateway 192.168.168.1
        # bridge control
        bridge_ports eth0
        bridge_fd 0
        pre-up brctl addbr virbr0
        pre-down brctl delbr virbr0
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 192.168.168.10
        dns-search claudiokuenzler.com

The LXC configuration file (/var/lib/lxc/mylxc/config) was using this defined virbr0 as link:

# Network configuration
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = virbr0
lxc.network.hwaddr = 00:FF:AA:00:00:53
lxc.network.ipv4 = 192.168.168.53/24
lxc.network.ipv4.gateway = 192.168.168.1

But after I started the container and tried to ping it from other servers in this subnet, I never got a response. In general, all network traffic did not work. Within the container I was not even able to ping the gateway.
By using tcpdump I however saw continous requests to find out the MAC address for the IP 192.168.168.53:

tcpdump host 192.168.168.53
14:20:51.672635 ARP, Request who-has 192.168.168.53 tell 192.168.168.52, length 46
14:20:51.672679 ARP, Reply 192.168.168.53 is-at 0a:2a:2a:d0:8f:00 (oui Unknown), length 28
14:20:51.743872 ARP, Request who-has 192.168.168.53 tell 192.168.168.52, length 46
14:20:51.743910 ARP, Reply 192.168.168.53 is-at 0a:2a:2a:d0:8f:00 (oui Unknown), length 28
14:20:51.774195 ARP, Request who-has 192.168.168.11 tell 192.168.168.53, length 28
14:20:52.673736 ARP, Request who-has 192.168.168.53 tell 192.168.168.52, length 46
14:20:52.673781 ARP, Reply 192.168.168.53 is-at 0a:2a:2a:d0:8f:00 (oui Unknown), length 28
14:20:52.761621 ARP, Request who-has 192.168.168.53 tell 192.168.168.52, length 46
14:20:52.761668 ARP, Reply 192.168.168.53 is-at 0a:2a:2a:d0:8f:00 (oui Unknown), length 28
14:20:52.774229 ARP, Request who-has 192.168.168.11 tell 192.168.168.53, length 28

On the remote server (192.168.168.52) the MAC address was never received (marked as incomplete):

root@192.168.168.52:~# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.168.53                   (incomplete)                              eth0
192.168.168.12           ether   00:1c:7f:61:38:35   C                     eth0
192.168.168.1            ether   02:00:5e:37:41:01   C                     eth0

At first I suspected arp issues when the container is running in the same subnet as the host. In all my previous setups the LXC host usually had an IP address of a different network than the containers, so this could be it. But all hints, configuration modifications and changing the kernel parameters for arp handling did not resolve the problem. After hours of research and debugging, I changed the focus of my search terms from something like "lxc same subnet as host" or "lxc bridge arp incomplete" to more VMware specific. The keywords "running lxc in vmware network issue" pointed to me an interesting thread which did point me to the correct information: promiscuous mode.

According to VMware KB 1002934, the following description explains what happens on ESXi's virtual switch level:

Promiscuous mode is a security policy which can be defined at the virtual switch or portgroup level in vSphere ESX/ESXi.
A virtual machine, Service Console or VMkernel network interface in a portgroup which allows use of promiscuous mode can see all network traffic traversing the virtual switch.
By default, a guest operating system's virtual network adapter only receives frames that are meant for it.
Placing the guest's network adapter in promiscuous mode causes it to receive all frames passed on the virtual switch that are allowed under the VLAN policy for the associated portgroup.

 That's exactly the case with LXC. As the virtual machine is a LXC host it is therefore also a bridge for its containers. As long as the promiscuous mode is disabled (which seems to be the VMware default), then the containers cannot do any communication. This also explains why the arp answers of the LXC host were never "saved" from requesters but still being asked for several times per second.

I checked the current settings on the virtual distributed switch (in vSphere client -> Inventory -> Networking -> Right click on the "managed distributed port group" which is used by the VM (LXC host) -> Edit Settings -> Security:

VMware Distrubted Port Group Security 

No big surprise - promiscuous mode is rejected. Which explains everything. I already thought I was going nuts after so many LXC installations (on physical machines though) and suddenly a very basic LXC setup wouldn't work anymore...

Update May 12th 2015:
It turned out setting Promiscuous Mode to Accept was not enough. The setting of "Forged Transmits" also needed to be set to Accept. After having set both Promiscuous Mode and Forged Transmits to Accept, the network communication with the containers worked.

 

Firefox cannot save passwords, store ssl certificate exceptions anymore
Thursday - May 7th 2015 - by - (0 comments)

Yesterday evening I tried to install a simple PDF printer (pdfforge PDFCreator) on my more or less new notebook. During the installation the installer seemed hanging somewhere and could not be cancelled. I also saw that the installer installed additional software (Anti Ad-Ware) which I didn't want at all. So I killed the installer process in the task manager.

Shortly after, when I used Firefox to search for alternative pdf printers, I saw that every new tab is now a Yahoo! search tab. Furthermore my search engine settings were emptied and only Yahoo! was in the list. A lot of other settings (like default home page) were changed. Anger!

Additionally to all that frustration it was not possible anymore to save any passwords in Firefox. In the Settings->Security window, the "Remember passwords for sites" box was even greyed out. Also the self-signed SSL certificates, which show up with a warning "Untrusted Connection", could not be added to the ssl exception list anymore (checkbox was greyed out, too).

After a Firefox reset the Yahoo! and other setting changes were gone and back to normal. However I was still unable to save passwords and whenever I restarted Firefox, I was logged out of websites I just logged in (and ticked"Remember me"). All this used to work before.

The final solution to this is to uninstall Firefox, manually delete the Firefox folder (in C:\Users\myusername\App Data\Roaming\Mozilla\) and then install Firefox from the setup exe again. Everything is back to normal now.

Important note: Hands off pdfforge PDFCreator!! It messed around with the Firefox profile and even worse, corrupted it.

 

xfs anger: repair not working (inodes not freed) and slow ls
Wednesday - May 6th 2015 - by - (0 comments)

In the past days I experienced a lot of problems with newly installed CentOS 7 machines. Besides having to use a ridiculous installer and systemd (but I'll keep that for another post) I came across problems with the new default filesystem CentOS 7 has chosen for its installations: xfs.

A few months ago I already became aware of xfs problems when an online resize didn't work as it should have - so I've been a xfs skeptical since then.

Now a couple of days ago I had an application file system running full on two machines which are being synchronized constantly. Due to weird circumstances (call it internal communication issues before I took over the machines) both machines, although intended to be installed the same way, were set up with different file system types: Machine1 running with xfs, machine2 running with ext4.

When the application, running on both machine1 and machine2, had an issue, there were a lot of inodes which needed to be cleared and the file system repaired. On machine2 (running with ext4) this happened automatically. Dmesg showed entries for orphaned inodes which were cleared. Afterwards the system rebooted itself. Not great (for the reboot), but 30s later the machine was up and running again.

On machine1 however, the inodes were never freed, nor did dmesg or any log show issues with the file system. Instead the file system was filling up until it was 100% used. Remember, machine1 and machine2 are being synchronized on the application file system, so the usage should be exactly the same. Yet while ext4 showed a usage of around 40%, xfs was full at 100%.

I decided to run xfs_repair to "force" xfs to free the used inodes. So I disabled machine1 from the load balancer, rebooted with grml and launched xfs_repair (first with the dry-run option -n):

root@grml ~ # xfs_repair -n /dev/vgapp/lvapp
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
agi unlinked bucket 0 is 863680 in ag 3 (inode=51195328)
agi unlinked bucket 1 is 3085889 in ag 3 (inode=53417537)
agi unlinked bucket 2 is 3085890 in ag 3 (inode=53417538)
agi unlinked bucket 3 is 3085891 in ag 3 (inode=53417539)
agi unlinked bucket 4 is 3719876 in ag 3 (inode=54051524)
[...]

The list went on and on... I was kind of "relieved" to see that xfs_repair has found some "unlinked" buckets, which means still referenced but undeleted inodes, according to the xfs documentation:

The di_next_unlinked value in the inode is used to track inodes that have been unlinked (deleted) but which are still referenced. 
[...]
The only time the unlinked fields can be seen to be used on disk is either on an active filesystem or a crashed system. A cleanly unmounted or recovered filesystem will not have any inodes in these unlink hash chains. 

So I ran xfs_repair without the dry-run:

root@grml ~ # xfs_repair /dev/vgapp/lvapp
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
agi unlinked bucket 0 is 11247360 in ag 1 (inode=28024576)
agi unlinked bucket 1 is 2556417 in ag 1 (inode=19333633)
agi unlinked bucket 2 is 11247042 in ag 1 (inode=28024258)
agi unlinked bucket 3 is 11247043 in ag 1 (inode=28024259)
agi unlinked bucket 4 is 11980292 in ag 1 (inode=28757508)
agi unlinked bucket 5 is 11247365 in ag 1 (inode=28024581)
[...]

It ran through until the end and it seemed to me that now the filesystem should be working again. Meaning that the occupied inodes were freed. I rebooted into CentOS but found that the application file system was still 100% fully used.

Back in grml I started another xfs_repair:

root@grml ~ # xfs_repair /dev/vgapp/lvapp
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes
Phase 4 - check for duplicated blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
Phase 5 - rebuld AG header and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done

All looks good, right? At least I thought so. Back to a booted CentOS, the file system was still at 100% full.

Besides the issue with the full file system, I also found that xfs is much slower in accessing the application files. On a cloned environment where both machines are in a healthy state, I compared the output of ls -l.
Remember, the application file system is synchronized... The same size and the same number of files (4905) are in the application folder. Yet xfs is almost 3x slower:

Machine 1 with xfs:

[root@machine1 backup]# time ls -la
[...]
real    0m0.679s
user    0m0.526s
sys    0m0.137s

Machine 2 with ext4:

[root@machine2 backup]# time ls -la
[...]
real    0m0.240s
user    0m0.198s
sys    0m0.039s

After having already spent too much time on it and not seeing any reason (performance wise) to continue using xfs, I finally decided to reinstall machine1 with ext4. Since then I got no problems anymore.

 

Changed internet access from DSL to Cable - gained a bit of speed
Saturday - May 2nd 2015 - by - (0 comments)

Changed Internet access and switched from a DSL to a Cable provider. At the end gained around 500kbit/s... ;-)

First the old speed test (with Speedtest.net) on the VDSL connection:

VDSL Speedtest Wil Switzerland

Then connected the new modem on the cable plug and connected to the same computer:

Cable Speedtest.net Wil Switzerland

 

 


Go to Homepage home
Linux Howtos how to's
Nagios Plugins nagios plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

8272 Days
until Death of Computers
Why?