- PR -

mysqlダウンの原因を突き止めたい

1
投稿者投稿内容
T3
会議室デビュー日: 2009/04/03
投稿数: 3
投稿日時: 2009-04-03 13:37
はじめまして。Linux運用初心者のT3です。
七日三晩ＷＥＢとバイブルを調べたものの、原因の特定に至れないため、お知恵をお借りしたく投稿致しました。

弊部署で使用しているサーバは以下の仕様です。
■マシン名：DELL PowerEdge SC430
■搭載物理メモリ： 1024MB + 256MB
■スワップ領域：2GB

CentOS5.2をクリーンインストールし、yumを使って
# yum -y install httpd
# yum -y install php php-mbstring
# yum -y install mysql-server
とし、phpとmysqlを利用できるwebサーバを運用しております。
/var/lib/mysql(データベースのデータが格納されたフォルダ)は99MBありました。

最近、毎日のようにmysqld_safeを含むmysqlサーバがダウンします。
mysqlダウン直後には、いつもコンソールに見たこともないメッセージが表示されています。
そのメッセージと同じ内容が/var/log/messagesに書かれていました。
内容の一部は、以下の通りです。
可能性として考えられることをご存じでしたら、どうかご教授の程よろしくお願いいたします。
--------------------------------------
Apr 2 23:35:31 localhost kernel: Mem-info:
Apr 2 23:35:31 localhost kernel: DMA per-cpu:
Apr 2 23:35:31 localhost kernel: cpu 0 hot: high 0, batch 1 used:0
Apr 2 23:35:31 localhost kernel: cpu 0 cold: high 0, batch 1 used:0
Apr 2 23:35:31 localhost kernel: DMA32 per-cpu: empty
Apr 2 23:35:31 localhost kernel: Normal per-cpu:
Apr 2 23:35:31 localhost kernel: cpu 0 hot: high 186, batch 31 used:33
Apr 2 23:35:31 localhost kernel: cpu 0 cold: high 62, batch 15 used:15
Apr 2 23:35:31 localhost kernel: HighMem per-cpu:
Apr 2 23:35:31 localhost kernel: cpu 0 hot: high 186, batch 31 used:48
Apr 2 23:35:31 localhost kernel: cpu 0 cold: high 62, batch 15 used:43
Apr 2 23:35:31 localhost kernel: Free pages: 21484kB (376kB HighMem)
Apr 2 23:35:31 localhost kernel: Active:169152 inactive:141862 dirty:0 writeback:0 unstable:0 free:5371 slab:2296 mapped-file:882 mapped-anon:311312 pagetables:1645
Apr 2 23:35:31 localhost kernel: DMA free:5116kB min:68kB low:84kB high:100kB active:3860kB inactive:3048kB present:16384kB pages_scanned:11969 all_unreclaimable? yes
Apr 2 23:35:31 localhost kernel: lowmem_reserve[]: 0 0 880 1262
Apr 2 23:35:31 localhost kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 2 23:35:31 localhost kernel: lowmem_reserve[]: 0 0 880 1262
Apr 2 23:35:31 localhost kernel: Normal free:15992kB min:3756kB low:4692kB high:5632kB active:450564kB inactive:404996kB present:901120kB pages_scanned:1656293 all_unreclaimable? yes
Apr 2 23:35:31 localhost kernel: lowmem_reserve[]: 0 0 0 3060
Apr 2 23:35:31 localhost kernel: HighMem free:376kB min:380kB low:788kB high:1196kB active:222184kB inactive:159404kB present:391728kB pages_scanned:777519 all_unreclaimable? yes
Apr 2 23:35:31 localhost kernel: lowmem_reserve[]: 0 0 0 0
Apr 2 23:35:31 localhost kernel: DMA: 1*4kB 1*8kB 9*16kB 13*32kB 7*64kB 4*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 5116kB
Apr 2 23:35:31 localhost kernel: DMA32: empty
Apr 2 23:35:31 localhost kernel: Normal: 26*4kB 440*8kB 453*16kB 18*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 15992kB
Apr 2 23:35:31 localhost kernel: HighMem: 26*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 376kB
Apr 2 23:35:31 localhost kernel: 1348 pagecache pages
Apr 2 23:35:31 localhost kernel: Swap cache: add 42194729, delete 42194476, find 5087837/9690160, race 9+140
Apr 2 23:35:31 localhost kernel: Free swap = 0kB
Apr 2 23:35:31 localhost kernel: Total swap = 2031608kB
Apr 2 23:35:31 localhost kernel: Free swap: 0kB
Apr 2 23:35:31 localhost kernel: 327308 pages of RAM
Apr 2 23:35:31 localhost kernel: 97932 pages of HIGHMEM
Apr 2 23:35:31 localhost kernel: 3819 reserved pages
Apr 2 23:35:31 localhost kernel: 8952 pages shared
Apr 2 23:35:31 localhost kernel: 253 pages swap cached
Apr 2 23:35:31 localhost kernel: 0 pages dirty
Apr 2 23:35:31 localhost kernel: 0 pages writeback
Apr 2 23:35:31 localhost kernel: 882 pages mapped
Apr 2 23:35:31 localhost kernel: 2296 pages slab
Apr 2 23:35:31 localhost kernel: 1645 pages pagetables
Apr 2 23:35:31 localhost kernel: Out of memory: Killed process 4995 (mysqld).
Apr 2 23:35:31 localhost kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Apr 2 23:35:31 localhost kernel: [<c0458895>] out_of_memory+0x72/0x1a5
Apr 2 23:35:31 localhost kernel: [<c0459d6a>] __alloc_pages+0x216/0x297
Apr 2 23:35:31 localhost kernel: [<c045afd7>] __do_page_cache_readahead+0xc4/0x1c6
Apr 2 23:35:31 localhost kernel: [<c05a7550>] sys_recvfrom+0xcf/0x130
Apr 2 23:35:31 localhost kernel: [<c04385e0>] ktime_get_ts+0x16/0x44
Apr 2 23:35:31 localhost kernel: [<c04505ba>] delayacct_end+0x58/0x7a
Apr 2 23:35:31 localhost kernel: [<f88b6c5c>] dm_any_congested+0x2f/0x35 [dm_mod]
Apr 2 23:35:31 localhost kernel: [<c0457e47>] filemap_nopage+0x151/0x312
Apr 2 23:35:31 localhost kernel: [<c0460c8e>] __handle_mm_fault+0x178/0x8e6
Apr 2 23:35:31 localhost kernel: [<c060acb3>] do_page_fault+0x20a/0x4b8
Apr 2 23:35:31 localhost kernel: [<c060aaa9>] do_page_fault+0x0/0x4b8
Apr 2 23:35:31 localhost kernel: [<c0405a71>] error_code+0x39/0x40
Apr 2 23:35:31 localhost kernel: =======================
Apr 2 23:35:31 localhost kernel: Mem-info:
Apr 2 23:35:31 localhost kernel: DMA per-cpu:
Apr 2 23:35:31 localhost kernel: cpu 0 hot: high 0, batch 1 used:0
Apr 2 23:35:31 localhost kernel: cpu 0 cold: high 0, batch 1 used:0
Apr 2 23:35:31 localhost kernel: DMA32 per-cpu: empty
Apr 2 23:35:31 localhost kernel: Normal per-cpu:
Apr 2 23:35:31 localhost kernel: cpu 0 hot: high 186, batch 31 used:20
Apr 2 23:35:31 localhost kernel: cpu 0 cold: high 62, batch 15 used:14
Apr 2 23:35:31 localhost kernel: HighMem per-cpu:
Apr 2 23:35:31 localhost kernel: cpu 0 hot: high 186, batch 31 used:74
Apr 2 23:35:31 localhost kernel: cpu 0 cold: high 62, batch 15 used:14
Apr 2 23:35:31 localhost kernel: Free pages: 21444kB (352kB HighMem)
Apr 2 23:35:31 localhost kernel: Active:169524 inactive:142516 dirty:0 writeback:0 unstable:0 free:5361 slab:2335 mapped-file:846 mapped-anon:311076 pagetables:1599
Apr 2 23:35:31 localhost kernel: DMA free:5116kB min:68kB low:84kB high:100kB active:3640kB inactive:3232kB present:16384kB pages_scanned:12323 all_unreclaimable? yes
Apr 2 23:35:31 localhost kernel: lowmem_reserve[]: 0 0 880 1262
Apr 2 23:35:31 localhost kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 2 23:35:31 localhost kernel: lowmem_reserve[]: 0 0 880 1262
Apr 2 23:35:31 localhost kernel: Normal free:15976kB min:3756kB low:4692kB high:5632kB active:453812kB inactive:402904kB present:901120kB pages_scanned:1837921 all_unreclaimable? yes
Apr 2 23:35:31 localhost kernel: lowmem_reserve[]: 0 0 0 3060
Apr 2 23:35:31 localhost kernel: HighMem free:352kB min:380kB low:788kB high:1196kB active:220644kB inactive:163928kB present:391728kB pages_scanned:704354 all_unreclaimable? yes
Apr 2 23:35:31 localhost kernel: lowmem_reserve[]: 0 0 0 0
Apr 2 23:35:31 localhost kernel: DMA: 1*4kB 1*8kB 9*16kB 13*32kB 7*64kB 4*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 5116kB
Apr 2 23:35:31 localhost kernel: DMA32: empty
Apr 2 23:35:31 localhost kernel: Normal: 24*4kB 439*8kB 453*16kB 18*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 15976kB
Apr 2 23:35:31 localhost kernel: HighMem: 20*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 352kB
Apr 2 23:35:31 localhost kernel: 1698 pagecache pages
Apr 2 23:35:31 localhost kernel: Swap cache: add 42224287, delete 42224051, find 5090054/9695722, race 9+151
Apr 2 23:35:31 localhost kernel: Free swap = 0kB
Apr 2 23:35:31 localhost kernel: Total swap = 2031608kB
Apr 2 23:35:31 localhost kernel: Free swap: 0kB
Apr 2 23:35:31 localhost kernel: 327308 pages of RAM
Apr 2 23:35:31 localhost kernel: 97932 pages of HIGHMEM
Apr 2 23:35:31 localhost kernel: 3819 reserved pages
Apr 2 23:35:31 localhost kernel: 8672 pages shared
Apr 2 23:35:31 localhost kernel: 236 pages swap cached
Apr 2 23:35:31 localhost kernel: 0 pages dirty
Apr 2 23:35:31 localhost kernel: 0 pages writeback
Apr 2 23:35:31 localhost kernel: 846 pages mapped
Apr 2 23:35:31 localhost kernel: 2335 pages slab
Apr 2 23:35:31 localhost kernel: 1599 pages pagetables
Apr 2 23:35:31 localhost kernel: Out of memory: Killed process 4754 (httpd).
デューン
大ベテラン
会議室デビュー日: 2004/04/21
投稿数: 174
お住まい・勤務地: Tokyo
投稿日時: 2009-04-03 14:14
あまり詳しくありませんが、

プロセス停止の原因はOOM Killer？


OOM killerとの危険な関係
http://www.atmarkit.co.jp/flinux/rensai/watch2008/watch02a.html

OOM Killer
http://itpro.nikkeibp.co.jp/article/COLUMN/20061117/254053/

Google検索「OOM Killer」
http://www.google.co.jp/search?rlz=1C1GGLS_jaJP300JP309&sourceid=chrome&ie=UTF-8&q=OOM-Killer


# ちなみに最初は「all_unreclaimable? yes」ググりました。
# Out Of Memoryの原因はわかりません
T3
会議室デビュー日: 2009/04/03
投稿数: 3
投稿日時: 2009-04-04 12:04
デューン様
ご指摘ありがとうございます。

Free swap = 0kB
Out of memory: Killed process 4995

のような症状を鑑みると、ご紹介いただきましたサイトの説明の症状に合致します。
端的に言うとメモリの使いすぎという可能性なので、httpdとmysqlの接続数やプロセス数を制限し、不要なデーモンを停止して様子を見てみたいと思います。
T3
会議室デビュー日: 2009/04/03
投稿数: 3
投稿日時: 2009-04-11 17:22
apacheとmysqldに接続制限を掛けることで、過度なメモリの消費を抑えた結果、スワップを頻繁に利用する症状が出なくなりました。
今は安定して稼働しております。

本当にありがとうございました。
1

スキルアップ／キャリアアップ（JOB@IT）