またデータベースサーバがダウンしてました

なあんかVMware ESXiサーバのRAIDカードがダウンしてるっぽい。
以下は、今回はMySQLサーバが起動しなくなったので、その対応内容の殴り書き

1.ログの確認

# cat /var/log/mysqld.log
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x29) [0x84c049]
/usr/libexec/mysqld(handle_fatal_signal+0x483) [0x6a0fa3]
/lib64/libpthread.so.0() [0x3eb760f500]
/lib64/libc.so.6(gsignal+0x35) [0x3eb6e328a5]
/lib64/libc.so.6(abort+0x175) [0x3eb6e34085]
/usr/libexec/mysqld() [0x736b98]
/usr/libexec/mysqld(btr_cur_search_to_nth_level+0xa95) [0x737ae5]
/usr/libexec/mysqld(row_search_index_entry+0x5b) [0x7af12b]
/usr/libexec/mysqld() [0x7adea5]
/usr/libexec/mysqld(row_purge_step+0x5ae) [0x7aefbe]
/usr/libexec/mysqld(que_run_threads+0x55b) [0x79df8b]
/usr/libexec/mysqld(trx_purge+0x332) [0x7c82a2]
/usr/libexec/mysqld(srv_master_thread+0x718) [0x7c0ca8]
/lib64/libpthread.so.0() [0x3eb7607851]
/lib64/libc.so.6(clone+0x6d) [0x3eb6ee76dd]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120817 18:59:30 mysqld_safe Number of processes running now: 0
120817 18:59:30 mysqld_safe mysqld restarted
120817 18:59:30 [Warning] '--default-character-set' is deprecated and will be removed in a future release. Please use '--character-set-server' instead.
120817 18:59:30  InnoDB: Initializing buffer pool, size = 2.0G
120817 18:59:30  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 12 2441123190
120817 18:59:30  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 12 2444809486
120817 18:59:30  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
120817 18:59:33  InnoDB: Started; log sequence number 12 2444809486
120817 18:59:33 [Note] Event Scheduler: Loaded 0 events
120817 18:59:33 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
120817 18:59:34  InnoDB: Assertion failure in thread 139913521219328 in file btr/btr0cur.c line 196
InnoDB: Failing assertion: btr_page_get_prev(get_page, mtr) == buf_frame_get_page_no(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:59:34 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

2.強制リカバリモードの設定

# vi /etc/my.cnf
innodb_force_recovery = 3 ←追記

3.mysqldサービスの停止・起動

# /etc/rc.d/init.d/mysqld restart
mysqld を停止中:                                           [  OK  ]
mysqld を起動中:                                           [  OK  ]
　　※停止しない場合は"kill -9"、"killall mysqld"後にstart

4.ダンプを取得

# mysqldump -u root -p -x --all-database > all-database.sql 

5.新しいデータディレクトリの作成、強制リカバリモードの停止とその反映

# mkdir /var/lib/mysqld
# chown mysql:mysql /var/lib/mysqld 

# vi /etc/my.cnf
#innodb_force_recovery = 3 ←コメントアウト
datadir=/var/lib/mysqld ←追記

6. mysqldサービスの再起動とダンプのインポート

# /etc/rc.d/init.d/mysqld restart
mysqld を停止中:                                           [  OK  ]
mysqld を起動中:                                           [  OK  ]

# mysql -u root < all-database.sql 

==============================
13.5.8.1. InnoDB 復旧の強制
http://dev.mysql.com/doc/refman/5.1/ja/forcing-recovery.html
innodb_force_recovery のゼロではない許容値が続きます。大きい数字は小さい数字の全ての予防策を含んでいます。もし最大4のオプション値を利用してテーブルをダンプする事ができれば、破損した独立ページ上のいくつかのデータが失われるだけなので、比較的に安全です。データベース ページは既に廃止された状態で残されるので、6の値はさらに徹底的であり、Bツリーやその他のデータベース構造に更なる破損を引き起こす可能性があります。
1 (SRV_FORCE_IGNORE_CORRUPT)
破損ページを検出したとしてもサーバを起動させてください。テーブルをダンプする助けになるので、SELECT * FROM tbl_name が破損したインデックス レコードとページを飛び越えるようにして下さい。

6 (SRV_FORCE_NO_LOG_REDO)
復旧と共にログ前進を接続内で行わないでください。

1の利便性と10の学習機会を求めて100の苦労を覚悟するのが自宅サーバ