Over the past few hours I've been trying to find the cause for the following log entries on a SmartOS running PHP-FPM:

tail /var/log/php-fpm.log
[19-Jun-2013 17:12:07] WARNING:  /dev/poll: ioctl() returns 22
[19-Jun-2013 17:12:07] WARNING:  /dev/poll: ioctl() returns 22
[19-Jun-2013 17:12:07] WARNING:  /dev/poll: ioctl() returns 22
[19-Jun-2013 17:12:07] WARNING:  /dev/poll: ioctl() returns 22
[19-Jun-2013 17:12:07] WARNING:  /dev/poll: ioctl() returns 22
[19-Jun-2013 17:12:07] WARNING:  /dev/poll: ioctl() returns 22
[...]

SmartOS version: 5.11 joyent_20120614T184600Z i86pc i386 i86pc Solaris
PHP-FPM version: php53-fpm-5.3.10

These entries were written up to 52'000 times per second (!!!!!).

At the begin I thought a mal-programmed website was the cause so I started to disable one fpm config after another. And suddenly the entries were gone! I thought I found the responsible web-host and re-activated all php-fpm configs again. As soon as I did that, the log entries appeared again. Over quite some time I tried to manually find which webs (obviously there must be several bad customer hostings... at least that's what I thought) are causing the issue.

...Until a strange idea hit me. What if this depends on the number of php-fpm configs in /opt/local/etc/fpm.d/? I didn't get any errors when I moved most of them to a temporary folder, so that would explain it...

So I re-activated all configs and started to move away one after another (again - but this time with another goal). After each move I restarted the PHP SMF service to see if the entries stopped.
I hit the magic number 35 after some time. As soon as there were no more than 35 fpm config files, the mass logging of these warnings stopped.

Summary: SmartOS PHP-FPM can only handle up to 35 php-fpm configs in /opt/local/etc/fpm.d/. At least so it seems after my tests.

Looks like the madness is always looking for me...

May daily schedule for a work-day looks somewhat like this: Get up, catch the train and read the paper, work, catch the train and watch an episode (currently Stargate SG1), get home, work some more, sleep.

Since Monday my schedule is disturbed - I cannot watch Stargate anymore! It all started Monday evening when I launched the "File Explorer" app as always and wanted to launch the video with "MX Player". Instead of instant video play, MX Player was just loading.. and loading.. and loading the video. During that time the phone (Samsung Galaxy S2 I9100) got very hot and I had to force a power off.

My first guess was that I didn't correctly finish the transfer of the video file to the phone but when I tried to play another episode, which I knew worked before, the same happened. So I took a look at the Android internal logs (using the app "Get Logs") and found some weird  entries:

06-17 17:52:08.745 D/VoldCmdListener( 1868): volume mount /mnt/emmc
06-17 17:52:08.745 D/DirectVolume( 1868): Checking for bad partition major number
06-17 17:52:08.745 I/Vold    ( 1868): /dev/block/vold/179:25 being considered for volume sdcard
06-17 17:52:08.745 D/Vold    ( 1868): Volume sdcard state changing 1 (Idle-Unmounted) -> 3 (Checking)
06-17 17:52:08.750 V/WiredAccessoryObserver( 1992): init()
06-17 17:52:08.755 D/MountService( 1992): volume state changed for /mnt/emmc (unmounted -> checking)
06-17 17:52:08.760 D/MountService( 1992): sendStorageIntent Intent { act=android.intent.action.MEDIA_CHECKING dat=file:///mnt/emmc (has extras) }
[...]
06-17 17:52:09.890 I/StorageNotification( 2113): Media {/mnt/emmc} state changed from {unmounted} -> {checking} (primary = false)
[...]
06-17 17:52:10.115 I//system/bin/fsck_msdos( 1868): ** /dev/block/vold/179:25
[...]
06-17 17:52:10.225 I//system/bin/fsck_msdos( 1868): ** Phase 1 - Read and Compare FATs
06-17 17:52:10.235 I//system/bin/fsck_msdos( 1868): Attempting to allocate 1872 KB for FAT
[...]
06-17 17:52:10.945 I//system/bin/fsck_msdos( 1868): Attempting to allocate 1872 KB for FAT
[...]
06-17 17:52:13.420 I//system/bin/fsck_msdos( 1868): ** Phase 2 - Check Cluster Chains
[...]
06-17 17:52:13.780 I//system/bin/fsck_msdos( 1868): ** Phase 3 - Checking Directories
[...]
06-17 17:52:33.250 I//system/bin/fsck_msdos( 1868): 9305 files, 3249472 free (232618 clusters)
[...]
06-17 17:52:33.615 I/Vold    ( 1868): Filesystem check completed OK
06-17 17:52:33.625 I/Vold    ( 1868): Device /dev/block/vold/179:25, target /mnt/emmc mounted @ /mnt/secure/staging
06-17 17:52:33.630 D/Vold    ( 1868): Volume sdcard state changing 3 (Checking) -> 4 (Mounted)
06-17 17:52:33.630 D/MountService( 1992): volume state changed for /mnt/emmc (checking -> mounted)
06-17 17:52:33.630 D/VoldCmdListener( 1868): volume mount /mnt/sdcard
06-17 17:52:33.630 D/DirectVolume( 1868): Checking for bad partition major number
06-17 17:52:33.630 D/DirectVolume( 1868): Found bad partition
06-17 17:52:33.630 E/DirectVolume( 1868): Fixing major number from 179 to 259 for partition 11
06-17 17:52:33.630 I/Vold    ( 1868): /dev/block/vold/259:3 being considered for volume emmc
06-17 17:52:33.630 D/Vold    ( 1868): Volume emmc state changing 1 (Idle-Unmounted) -> 3 (Checking)
06-17 17:52:33.635 I/StorageNotification( 2113): Media {/mnt/emmc} state changed from {checking} -> {mounted} (primary = false)
06-17 17:52:33.635 D/MountService( 1992): sendStorageIntent Intent { act=android.intent.action.MEDIA_MOUNTED dat=file:///mnt/emmc (has extras) }
06-17 17:52:33.635 D/MountService( 1992): volume state changed for /mnt/sdcard (unmounted -> checking)
06-17 17:52:33.640 D/MountService( 1992): sendStorageIntent Intent { act=android.intent.action.MEDIA_CHECKING dat=file:///mnt/sdcard (has extras) }
06-17 17:52:33.655 I//system/bin/fsck_msdos( 1868): ** /dev/block/vold/259:3
06-17 17:52:33.660 I//system/bin/fsck_msdos( 1868): ** Phase 1 - Read and Compare FATs
06-17 17:52:33.665 I//system/bin/fsck_msdos( 1868): Attempting to allocate 1473 KB for FAT
06-17 17:52:33.665 I/StorageNotification( 2113): Media {/mnt/sdcard} state changed from {unmounted} -> {checking} (primary = true)
[...]
06-17 17:52:33.920 I//system/bin/fsck_msdos( 1868): Attempting to allocate 1473 KB for FAT
06-17 17:52:34.405 I//system/bin/fsck_msdos( 1868): ** Phase 2 - Check Cluster Chains
06-17 17:52:34.435 D/MediaScannerReceiver( 3575): action: android.intent.action.MEDIA_MOUNTED path: /mnt/emmc
[...]
06-17 17:52:35.280 I/k9      ( 3261): storage path "/mnt/emmc" mounted readOnly=true
[...]
06-17 17:52:36.150 I//system/bin/fsck_msdos( 1868): ** Phase 4 - Checking for Lost Files
06-17 17:52:36.165 I//system/bin/fsck_msdos( 1868): Next free cluster in FSInfo block (1262) not free
06-17 17:52:36.165 I//system/bin/fsck_msdos( 1868): FIXED
06-17 17:52:36.165 I//system/bin/fsck_msdos( 1868): 5884 files, 540544 free (147964 clusters)
06-17 17:52:36.235 I/Vold    ( 1868): Filesystem check completed OK
06-17 17:52:36.245 I/Vold    ( 1868): Device /dev/block/vold/259:3, target /mnt/sdcard mounted @ /mnt/secure/staging
06-17 17:52:36.250 I/PackageManager( 1992): Updating external media status from unmounted to mounted
06-17 17:52:36.250 D/Vold    ( 1868): Volume emmc state changing 3 (Checking) -> 4 (Mounted)
06-17 17:52:36.250 D/MountService( 1992): volume state changed for /mnt/sdcard (checking -> mounted)
06-17 17:52:36.250 I/StorageNotification( 2113): Media {/mnt/sdcard} state changed from {checking} -> {mounted} (primary = true)
06-17 17:52:36.255 D/MountService( 1992): sendStorageIntent Intent { act=android.intent.action.MEDIA_MOUNTED dat=file:///mnt/sdcard (has extras) }
06-17 17:52:36.275 D/MediaScannerReceiver( 3575): action: android.intent.action.MEDIA_MOUNTED path: /mnt/sdcard
[...]
06-17 17:52:36.415 I/k9      ( 3261): storage path "/mnt/sdcard" mounted readOnly=true
[...]
06-17 17:52:38.665 E/Vold    ( 1868): ASEC mount failed (No such file or directory)
06-17 17:52:38.685 I/PackageHelper( 1992): Failed to mount container com.google.android.apps.unveil-2 rc : -1
06-17 17:52:38.685 E/PackageManager( 1992): Failed to mount cid : com.google.android.apps.unveil-2 when installing from sdcard

The most important entry of all these lines:

06-17 17:52:33.630 D/DirectVolume( 1868): Found bad partition

So something must be terribly wrong with my external SDCard (because the Samsung Galaxy S2 also contains an internal SDCard).

Today - a few days and reboots later - I got the confirmation of data corruption just by opening "File Explorer" again:

The file names became garbled and the files, of course, can still not be opened.

I will now try to boot into Clockworkmod and manually format /dev/emmc and hope this will fix it. But I honestly doubt that. It is likely that the SDCard is definitely grilled.. :-/

I had to solve a strange MySQL replication issue today. All of a sudden, the Slave_SQL_Running changed to No and showed the following error message:

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: XX.XX.XX.XX
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: bin.000481
          Read_Master_Log_Pos: 344928978
               Relay_Log_File: relay-log.001382
                Relay_Log_Pos: 293565267
        Relay_Master_Log_File: bin.000481
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1146
                   Last_Error: Error 'Table 'mydb.mytable_temp' doesn't exist' on query. Default database: 'mydb'. Query:
'CREATE TABLE `mytable_temp` (
[...]

After some investigation on the slave server it turned out that there was already a table called mydb.mytable_temp. Therefore the CREATE TABLE returned an error.

As a first step I disabled the SLAVE process and skipped the query causing the error (the mentioned CREATE TABLE query). Then I started the SLAVE again to see what happens afterwards:

mysql> STOP SLAVE;
mysql> SET GLOBAL sql_slave_skip_counter = 1;
mysql> START SLAVE;

As soon as the slave was started again, the next query was again resulting in an error, because it was an INSERT INTO mytable_temp. Although the table already existed (seen in 'show tables') the INSERT didn't work. 

I decided to manually drop the table mytable_temp and re-create it with the query from the previous slave error, but funnily the table couldn't be dropped:

mysql> use mydb;
mysql> DROP TABLE mytable_temp;
ERROR 1051 (42S02) at line 1: Unknown table 'mytable_temp'

Well... that is strange. But although an error appeared right after the DROP, a new 'show tables' didn't show mytable_temp anymore. So it's gone. Right?

Now that the table seems to be gone, nevermind the error before, I wanted to create the table manually with the CREATE TABLE query seen in the first slave error:

mysql> CREATE TABLE mytable_temp (
    -> `entry_id` INT(11) NOT NULL,
[...]
    -> key text_index(data_text(4096))
    -> );
ERROR 1005 (HY000): Can't create table 'mydb.mytable_temp' (errno: -1)

What the hell?! I checked out the mysql error log and found the following entry:

# tail error.log
InnoDB: Error number 17 means 'File exists'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
InnoDB: The file already exists though the corresponding table did not
InnoDB: exist in the InnoDB data dictionary. Have you moved InnoDB
InnoDB: .ibd files around without using the SQL commands
InnoDB: DISCARD TABLESPACE and IMPORT TABLESPACE, or did
InnoDB: mysqld crash in the middle of CREATE TABLE? You can
InnoDB: resolve the problem by removing the file './mydb/mytable_temp.ibd'
InnoDB: under the 'datadir' of MySQL.

Oh wow... Incredible! There seems to be a problem with the ibd file - which exists although the table wasn't created successfully. The suggestion in the error log is to remove that particular file:

# rm /var/lib/mysql/mydb/mydb_temp.ibd

Right after that I relaunched the CREATE TABLE query and this time ... it worked! (Well no surprise given the new information)

All what was left to do was to finally activate the slave again:

mysql> START SLAVE;

... and the slave was catching up just fine by doing all the INSERTs into that temp table.

One question remains however: What's the reason for this "corrupted" ibd file? Did MySQL try to create the table but had a process crash in the middle of it? Was this table already there from an earlier manual replication sync? Had the file system an issue?

According to the mysql error log, the MySQL slave server seems to have crashed just before or during that operation:

130618 14:01:48  InnoDB: Assertion failure in thread 14 in file buf0buf.c line 4166
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:01:48 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=16777216
read_buffer_size=1048576
max_used_connections=3
max_threads=5000
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10316071 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
/mysql/bin/mysqld'my_print_stacktrace+0x1e [0x8d214e]
/mysql/bin/mysqld'handle_fatal_signal+0x359 [0x7b7b79]
/lib/amd64/libc.so.1'__sighndlr+0x6 [0xfffffd7fff2412b6]
/lib/amd64/libc.so.1'call_user_handler+0x1c3 [0xfffffd7fff234db3]
/lib/amd64/libc.so.1'_lwp_kill+0xa [0xfffffd7fff247e3a] [Signal 6 (ABRT)]
/lib/amd64/libc.so.1'raise+0x19 [0xfffffd7fff1e6d79]
/lib/amd64/libc.so.1'abort+0x5d [0xfffffd7fff1c3a2d]
/mysql/bin/mysqld'buf_page_io_complete+0x41f [0x970f8f]
/mysql/bin/mysqld'fil_aio_wait+0x144 [0x9b12e4]
/mysql/bin/mysqld'io_handler_thread+0x48 [0x921538]
/lib/amd64/libc.so.1'_thrp_setup+0x83 [0xfffffd7fff240ef3]
/lib/amd64/libc.so.1'_lwp_start+0x0 [0xfffffd7fff2411f0]
Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
and follow instructions on how to resolve the stack trace.
Resolved stack trace is much more helpful in diagnosing the
problem, so please do resolve it
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
130618 14:02:08 mysqld_safe mysqld restarted
130618 14:02:08 [Note] Plugin 'FEDERATED' is disabled.
130618 14:02:08 InnoDB: The InnoDB memory heap is disabled
130618 14:02:08 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130618 14:02:08 InnoDB: Compressed tables use zlib 1.2.3
130618 14:02:08 InnoDB: Initializing buffer pool, size = 32.0G
130618 14:02:15 InnoDB: Completed initialization of buffer pool
130618 14:02:15 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 777018804236
130618 14:02:16  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 777024047104
InnoDB: Doing recovery: scanned up to log sequence number 777029289984
InnoDB: Doing recovery: scanned up to log sequence number 777034532864
InnoDB: Doing recovery: scanned up to log sequence number 777039775744
InnoDB: Doing recovery: scanned up to log sequence number 777042170082
130618 14:02:21  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 293564055, file name bin.000481
InnoDB: and relay log file
InnoDB: position 0 293564195, file name /var/mysql/logs/relay-log.001382
InnoDB: Last MySQL binlog file position 0 107087858, file name /var/mysql/logs/bin.000021
130618 14:02:24 InnoDB: Restoring buffer pool pages from ib_lru_dump
130618 14:02:24  InnoDB: Waiting for the background threads to start
130618 14:02:25 Percona XtraDB (http://www.percona.com) 1.1.8-27.0 started; log sequence number 777042170082
handlersocket: not listening for reads
handlersocket: not listening for writes
handlersocket: initialized
130618 14:02:25 [Note] Recovering after a crash using /var/mysql/logs/bin
130618 14:02:25 [Note] Starting crash recovery...
130618 14:02:25 [Note] Crash recovery finished.
130618 14:02:25 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130618 14:02:25 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
130618 14:02:25 [Note] Server socket created on IP: '0.0.0.0'.
130618 14:02:26 [Note] Slave SQL thread initialized, starting replication in log 'bin.000481' at position 293565127, relay log '/var/mysql/logs/relay-log.001382' position: 293565267
130618 14:02:26 [Note] Slave I/O thread: connected to master 'repl@xx.xx.xx.xx:3306',replication started in log 'bin.000481' at position 293648217
130618 14:02:26 [ERROR] Cannot find or open table mydb/mytable_temp from
the internal data dictionary of InnoDB though the .frm file for the
table exists. Maybe you have deleted and recreated InnoDB data
files but have forgotten to delete the corresponding .frm files
of InnoDB tables, or you have moved .frm files to another database?
or, the table contains indexes that this version of the engine
doesn't support.
See http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html
how you can resolve the problem.

So this is confirmed. The corruption was caused by a crashed MySQL (Percona) process.

In a previous article I explained how a defect hard drive can be replaced in a HP Proliant Server (using a HP Raid Controller) and running on Solaris.

This time I had to add new disks to an existing zfs pool (zpool). There are a lot of howtos on the Internet (a good one is http://docs.oracle.com/cd/E19253-01/819-5461/gazgw/index.html), however almost all howtos don't mention the hpacucli command which must be used when the HP Raid Controller presents the drives to the Operating System.

Once the new disks are physically installed, their presence can be verified in hpacucli:

/opt/HPQacucli/sbin/hpacucli
HP Array Configuration Utility CLI 8.0-14.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.

=> ctrl slot=1 show config

Smart Array P400 in Slot 1    (sn: PA2240J9STL1WU)

   array A (SAS, Unused Space: 0 MB)
      logicaldrive 1 (68.3 GB, RAID 0, OK)
      physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 72 GB, OK)

   array B (SAS, Unused Space: 0 MB)
      logicaldrive 2 (68.3 GB, RAID 0, OK)
      physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 72 GB, OK)

   array C (SAS, Unused Space: 0 MB)
      logicaldrive 3 (68.3 GB, RAID 0, OK)
      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SAS, 72 GB, OK)

   array D (SAS, Unused Space: 0 MB)
      logicaldrive 4 (68.3 GB, RAID 0, OK)
      physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SAS, 72 GB, OK)

   unassigned
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 146 GB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 146 GB, OK)

The new drives appear as "unassigned" at the bottom.

Now these drives must be configured like the other drives, as single logical drives with a raid 0 (sounds strange, indeed).
So still in the hpacucli, launch the following commands with the ID's of your new physical drives:

=> ctrl slot=1 create type=ld drives=1I:1:5 raid=0
=> ctrl slot=1 create type=ld drives=1I:1:6 raid=0

Now verify the config again and then exit hpacucli:

 => ctrl slot=1 show config

Smart Array P400 in Slot 1    (sn: PA2240J9STL1WU)

   array A (SAS, Unused Space: 0 MB)
      logicaldrive 1 (68.3 GB, RAID 0, OK)
      physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 72 GB, OK)

   array B (SAS, Unused Space: 0 MB)
      logicaldrive 2 (68.3 GB, RAID 0, OK)
      physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 72 GB, OK)

   array C (SAS, Unused Space: 0 MB)
      logicaldrive 3 (68.3 GB, RAID 0, OK)
      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SAS, 72 GB, OK)

   array D (SAS, Unused Space: 0 MB)
      logicaldrive 4 (68.3 GB, RAID 0, OK)
      physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SAS, 72 GB, OK)

   array E (SAS, Unused Space: 0 MB)
      logicaldrive 5 (136.7 GB, RAID 0, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 146 GB, OK)

   array F (SAS, Unused Space: 0 MB)
      logicaldrive 6 (136.7 GB, RAID 0, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 146 GB, OK)

=> exit

The disks should now be seen in Solaris. Their new "Solaris device names" can be seen with the command format:

format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <DEFAULT cyl 8917 alt 2 hd 255 sec 63>
          /pci@0,0/pci8086,25e3@3/pci103c,3234@0/sd@0,0
       1. c0t1d0 <DEFAULT cyl 8917 alt 2 hd 255 sec 63>
          /pci@0,0/pci8086,25e3@3/pci103c,3234@0/sd@1,0
       2. c0t2d0 <DEFAULT cyl 8917 alt 2 hd 255 sec 63>
          /pci@0,0/pci8086,25e3@3/pci103c,3234@0/sd@2,0
       3. c0t3d0 <DEFAULT cyl 8917 alt 2 hd 255 sec 63>
          /pci@0,0/pci8086,25e3@3/pci103c,3234@0/sd@3,0
       4. c0t4d0 <HP-LOGICAL VOLUME-1.18-136.70GB>
          /pci@0,0/pci8086,25e3@3/pci103c,3234@0/sd@4,0
       5. c0t5d0 <HP-LOGICAL VOLUME-1.18-136.70GB>
          /pci@0,0/pci8086,25e3@3/pci103c,3234@0/sd@5,0

So the new devices are called c0t4d0 and c0t5d0. The disks can now be added to the existing pool.
The command to add new drives to a zpool allows the -n option, which allows a test add (so it's just a test-run):

zpool add -n zonepool mirror c0t4d0 c0t5d0
would update 'zonepool' to the following configuration:
        zonepool
          mirror
            c0t2d0s0
            c0t3d0s0
          mirror
            c0t4d0
            c0t5d0

If there are no errors appearing, the drives can definitely be added to the zpool:

zpool add zonepool mirror c0t4d0 c0t5d0

The zpool status then shows the new attached mirror:

zpool status

  pool: zonepool
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        zonepool      ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c0t2d0s0  ONLINE       0     0     0
            c0t3d0s0  ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c0t4d0    ONLINE       0     0     0
            c0t5d0    ONLINE       0     0     0

errors: No known data errors

Over the past few years I've settled with two major Linux distros: openSuSE (and SLES) and Debian. While the package management in Debian was using apt for a long time, SuSE's zypper alternative is relatively new but nonetheless very helpful. 

The following table shows a comparison of the most important and most used commands for package/software management in SuSE and Debian systems.

zypper vs. apt command comparison

	zypper	apt
Search package	zypper se < pkgname >	apt-cache search < pkgname >
Install package	zypper in < pkgname >	apt-get install < pkgname >
Update package list	zypper refresh	apt-get update
Update all packages	zypper update	apt-get upgrade
Upgrade all packages to new major version*	N/A	apt-get dist-upgrade
Update a single package	zypper update < pkgname >	apt-get upgrade < pkgname >
Uninstall/remove a package	zypper remove < pkgname >	apt-get remove < pkgname >
Show all packages found in repo(s)	zypper packages	apt-cache pkgnames
Show updateable packages	zypper list-updates	apt-show-versions -u
Show dependencies of a package	N/A	apt-cache depends < pkgname >
Show description of a package	zypper info < pkgname >	apt-cache show < pkgname >

 rpm vs. dpkg command comparison

	rpm	dpkg
List installed packages	rpm -ql	dpkg -l
Install package	rpm -i < pkgname >.rpm	dpkg -i < pkgname >.deb
Update a package	rpm -U < pkgname >.rpm	N/A
Uninstall/remove a package	rpm -e < pkgname >	dpkg -r < pkgname >

If you know additional zypper/apt/rpm/dpkg commands which are worth to be compared and mentioned here, please drop a comment.

*Update June 14th, 2013: Added apt-get dist-upgrade to the table. This command allows to upgrade to a newer major software version (e.g. MySQL 5.0 -> 5.1). However it should be mentioned, that although this command exists in Debian, it is unlikely that it's going be used (except for a distribution upgrade, e.g. lenny -> squeeze). The reason for this is the "stable software version" mind of Debian. Ubuntu handles this approach differently and frequently adds newer major versions to their repositories.

In the past few months I've had a lot of failing hard drives to compensate... While I was writing a summary of all disk replace actions, I got aware that it's always the same model failing: ST3000DM001-9YN166.

November 3rd, 2012 - runtime 4391 hours (182 days + 23 hours):

=== START OF INFORMATION SECTION ===
Device Model:     ST3000DM001-9YN166
Serial Number:    XXXXXXXX
Firmware Version: CC4C
User Capacity:    3,000,592,982,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Sat Nov  3 11:51:24 2012 CET

February 28th, 2013 - runtime 2728 hours (113 days + 16 hours):

=== START OF INFORMATION SECTION ===
Device Model:     ST3000DM001-9YN166
Serial Number:    XXXXXXXX
Firmware Version: CC4B
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Thu Feb 28 01:20:07 2013 CET

May 27th, 2013 - runtime 2593 hours (108 days + 1 hours):

=== START OF INFORMATION SECTION ===
Device Model:     ST3000DM001-9YN166
Serial Number:    XXXXXXXX
Firmware Version: CC4H
User Capacity:    3,000,592,982,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Mon May 27 11:59:17 2013 CEST

June 11th, 2013 - runtime 10528 hours (438 days + 16 hours):

=== START OF INFORMATION SECTION ===
Device Model:     ST3000DM001-9YN166
Serial Number:    XXXXXXXX
Firmware Version: CC4C
User Capacity:    3,000,592,982,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Jun 11 14:12:53 2013 CEST

So I'd say hands off this disk...

One could be surprised when  opening a closet in a data center ...

Yeah, these are some really old packages back from the days when you actually bought software in boxes...

Many USB drives come by default pre-formatted in either FAT32 (which can be read by, to my knowledge, all operating systems) or NTFS, which is of course the Windows file system. 

But what if the drive is formatted in another file system type, let's say ext3? You may plug it to a Windows machine and the drive will not be recognized - therefore you cannot re-format the drive from within the Windows machine to re-use it.

In this case you need to format the drive on your Linux machine with the following commands.

First install the package dosfstools, which contains the program to format a file system in msdos (fat):

apt-get install dosfstools

Then delete existing partitions on the drive (/dev/sdd in my case):

fdisk /dev/sdd

Command (m for help): p
Disk /dev/sdd: 2004 MB, 2004877312 bytes
Disk identifier: 0x00090d91
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         590     1955840   83  Linux

Command (m for help): d
Selected partition 1

Command (m for help): p
Disk /dev/sdd: 2004 MB, 2004877312 bytes
Disk identifier: 0x00090d91
   Device Boot      Start         End      Blocks   Id  System

Followed by creating a new partition (still in fdisk):

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-590, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-590, default 590):
Using default value 590

... followed by setting the partition type to FAT32 (see fdisk option l to list the available types):

 Command (m for help): t
Selected partition 1
Hex code (type L to list codes): b
Changed system type of partition 1 to b (W95 FAT32)

Save your changes to the drive:

Command (m for help): w
The partition table has been altered!

fdisk -l /dev/sdd
Disk /dev/sdd: 2004 MB, 2004877312 bytes
Disk identifier: 0x00090d91
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         590     1957599    b  W95 FAT32

Now format the new partition with FAT32 using mkfs.msdos:

mkfs.msdos -F 32 /dev/sdd1
mkfs.msdos 3.0.9 (31 Jan 2010)

And done! Your Linux and your Windows machine can now read and write on this drive.

If someone wondered, how long it takes to copy a 3TB SATA drive to another identical SATA drive with 'dd', here's your fact:

dd if=/dev/sdb3 of=/dev/sda3 conv=noerror,sync
5855286143+16 records in
5855286159+0 records out
2997906513408 bytes (3.0 TB) copied, 106038 s, 28.3 MB/s

So it took 106038 seconds = 1767.3 minutes = 29.455 hours.

15 sectors of the input drive (/dev/sdb3) were defect, that's why I used "conv=noerror,sync" which allows to ignore the read errors.

Just tried to update ILO3 on a HP DL 380 G7 from version 1.16 (Dec 17 2010) to one of the latest version 1.55 (Feb 19 2013) but got the following error message in the browser:

The last firmware update attempt was not successful. Ready for the next update.

A quick look in the readme file of the ILO3 1.55 package reveals:

PART NUMBER:  592410-013
FILE NAME:    ilo3_155.bin
TITLE:        Integrated Lights-Out 3 Firmware 1.55
[...]
PREREQUISITE:  iLO 3 v1.20
[...]
FIRMWARE DEPENDENCY: iLO 3 should be upgraded to v1.20 before upgrading to this version.

But it also works with version 1.28 as intermediate version.