Ok long story short, a month or so ago we got a power outage, the UPS kicked in properly and eventually shut down the system properly. The outage lasted several hours. When the power came back I turned everything back on.
As I'm loading my VMs, I lost a drive... then lost another, the raid 5 was now broken. After screwing around with it for a while I managed to get it back online, but I did lose one drive, then it stayed that way.
I ordered new backplanes, and all new drives, and changed all the cables. This seemed to have fixed the issues for about a month. Now today, it started crashing again, with tons of these errors:
I was getting those previously as well after I changed the backplanes and drives but it just went away. Now it's back. This causes all Linux VMs to crap out with disk I/O errors. The windows vms are fine. The system is Fedora Core 9 running Virtualbox 3.2.12. I know there's newer versions, but that's besides the point. Everything was ok before the power outage.
I put over 1k into replacing everything I can in hopes it goes away, and no luck I guess. I thought I was doing good given it's been running ok for a while.
. Any suggestions on what else to try? Motherboard? Sata controller? Right now I'm using the mobo + 2 pcie controllers. Now they make 8 port + controllers so I'm wondering if I should get one of those. It's hard to find ones that don't do raid though. I had bought one but learned the hard way that if it has raid you HAVE to use it, it wont present the drives normally to the OS before they are initialized. I'm running out of money here, but this system just has to work.
As I'm loading my VMs, I lost a drive... then lost another, the raid 5 was now broken. After screwing around with it for a while I managed to get it back online, but I did lose one drive, then it stayed that way.
I ordered new backplanes, and all new drives, and changed all the cables. This seemed to have fixed the issues for about a month. Now today, it started crashing again, with tons of these errors:
Code:
INFO: task kjournald:1202 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88021a845040 0 1202 2
ffff8802188a5ca0 0000000000000046 0000000000000086 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff88021956adc0 ffff8801001b2dc0
ffff88021956b108 00000003a0286a3c ffff88021aad96c8 ffff88021956b108
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff8113cb81>] ? submit_bio+0xe0/0xe9
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffffa0024deb>] wait_on_buffer+0x41/0x45 [jbd]
[<ffffffffa002540e>] journal_commit_transaction+0x55d/0xf2f [jbd]
[<ffffffff81049a0d>] ? try_to_del_timer_sync+0x58/0x63
[<ffffffffa0028ad8>] kjournald+0xe3/0x23a [jbd]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffffa00289f5>] ? kjournald+0x0/0x23a [jbd]
[<ffffffff810534bf>] kthread+0x49/0x76
[<ffffffff81011719>] child_rip+0xa/0x11
[<ffffffff81010a37>] ? restore_args+0x0/0x30
[<ffffffff81053476>] ? kthread+0x0/0x76
[<ffffffff8101170f>] ? child_rip+0x0/0x11
INFO: task VirtualBox:25932 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D 0000000000000002 0 25932 25560
ffff8801d9e51928 0000000000000086 0000000000000096 ffff88021b4505d0
ffffffff8162a500 ffffffff8162a500 ffff8800c2a20000 ffff88021b9fadc0
ffff8800c2a20348 00000000a0286a3c ffff88021aad96c8 ffff8800c2a20348
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff8113cb81>] ? submit_bio+0xe0/0xe9
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffff810e154f>] wait_on_buffer+0x41/0x45
[<ffffffff810e1b53>] sync_dirty_buffer+0x58/0xa7
[<ffffffffa0023eec>] journal_dirty_data+0xed/0x1cb [jbd]
[<ffffffffa00375d4>] ext3_journal_dirty_data+0x18/0x40 [ext3]
[<ffffffffa003557b>] walk_page_buffers+0x6e/0x96 [ext3]
[<ffffffffa00375bc>] ? ext3_journal_dirty_data+0x0/0x40 [ext3]
[<ffffffffa0036f75>] ext3_ordered_write_end+0x7d/0x127 [ext3]
[<ffffffff8108e167>] generic_file_buffered_write+0x1b8/0x643
[<ffffffff810d623b>] ? mnt_drop_write+0x82/0x143
[<ffffffff810d453d>] ? mnt_want_write+0x77/0x8d
[<ffffffff8108e9e7>] __generic_file_aio_write_nolock+0x25e/0x292
[<ffffffff8108f233>] generic_file_aio_write+0x67/0xc3
[<ffffffffa003443f>] ext3_file_write+0x1e/0x9f [ext3]
[<ffffffff810beb88>] do_sync_write+0xe7/0x12d
[<ffffffffa03d88a2>] ? RTMemFree+0x1e/0x20 [vboxdrv]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff812c20fa>] ? _spin_lock+0x9/0xc
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf444>] vfs_write+0xab/0x105
[<ffffffff810bf562>] sys_write+0x47/0x6f
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task VirtualBox:25970 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D 0000000000000002 0 25970 3154
ffff8801d6ed9948 0000000000000086 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8801f8c75b80 ffff88021b9fadc0
ffff8801f8c75ec8 00000000a0286a3c ffff88021aad96c8 ffff8801f8c75ec8
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa0024149>] do_get_write_access+0x72/0x404 [jbd]
[<ffffffffa0027cfb>] ? journal_add_journal_head+0xc7/0x14e [jbd]
[<ffffffffa00244fd>] journal_get_write_access+0x22/0x33 [jbd]
[<ffffffffa0042806>] __ext3_journal_get_write_access+0x1f/0x48 [ext3]
[<ffffffffa0036391>] ext3_reserve_inode_write+0x3f/0x76 [ext3]
[<ffffffffa0036403>] ext3_mark_inode_dirty+0x3b/0x58 [ext3]
[<ffffffffa0036564>] ext3_dirty_inode+0x6c/0x83 [ext3]
[<ffffffff810dc8f7>] __mark_inode_dirty+0x33/0x190
[<ffffffff810d13e0>] file_update_time+0xbe/0x102
[<ffffffff8108e8f1>] __generic_file_aio_write_nolock+0x168/0x292
[<ffffffff8108f233>] generic_file_aio_write+0x67/0xc3
[<ffffffffa003443f>] ext3_file_write+0x1e/0x9f [ext3]
[<ffffffff810beb88>] do_sync_write+0xe7/0x12d
[<ffffffffa03d88a2>] ? RTMemFree+0x1e/0x20 [vboxdrv]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf444>] vfs_write+0xab/0x105
[<ffffffff810bf562>] sys_write+0x47/0x6f
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task VirtualBox:25990 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D ffff880183ffd6c0 0 25990 3145
ffff880179ba1948 0000000000000086 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff88021151adc0 ffff8801001b2dc0
ffff88021151b108 00000003a0286a3c ffff88021aad96c8 ffff88021151b108
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa0024149>] do_get_write_access+0x72/0x404 [jbd]
[<ffffffffa0027cfb>] ? journal_add_journal_head+0xc7/0x14e [jbd]
[<ffffffffa00244fd>] journal_get_write_access+0x22/0x33 [jbd]
[<ffffffffa0042806>] __ext3_journal_get_write_access+0x1f/0x48 [ext3]
[<ffffffffa0036391>] ext3_reserve_inode_write+0x3f/0x76 [ext3]
[<ffffffffa0036403>] ext3_mark_inode_dirty+0x3b/0x58 [ext3]
[<ffffffffa0036564>] ext3_dirty_inode+0x6c/0x83 [ext3]
[<ffffffff810dc8f7>] __mark_inode_dirty+0x33/0x190
[<ffffffff810d13e0>] file_update_time+0xbe/0x102
[<ffffffff8108e8f1>] __generic_file_aio_write_nolock+0x168/0x292
[<ffffffff8108f233>] generic_file_aio_write+0x67/0xc3
[<ffffffffa003443f>] ext3_file_write+0x1e/0x9f [ext3]
[<ffffffff810beb88>] do_sync_write+0xe7/0x12d
[<ffffffffa03d88a2>] ? RTMemFree+0x1e/0x20 [vboxdrv]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf444>] vfs_write+0xab/0x105
[<ffffffff810bf562>] sys_write+0x47/0x6f
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task VirtualBox:20126 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D 0000000000000002 0 20126 3145
ffff880090a2b9a8 0000000000000082 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8801d6075b80 ffff88021b9fadc0
ffff8801d6075ec8 00000000a0286a3c ffff88021aad96c8 ffff8801d6075ec8
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa0024149>] do_get_write_access+0x72/0x404 [jbd]
[<ffffffffa0027cfb>] ? journal_add_journal_head+0xc7/0x14e [jbd]
[<ffffffffa00244fd>] journal_get_write_access+0x22/0x33 [jbd]
[<ffffffffa0042806>] __ext3_journal_get_write_access+0x1f/0x48 [ext3]
[<ffffffffa0036391>] ext3_reserve_inode_write+0x3f/0x76 [ext3]
[<ffffffffa0036403>] ext3_mark_inode_dirty+0x3b/0x58 [ext3]
[<ffffffffa0036564>] ext3_dirty_inode+0x6c/0x83 [ext3]
[<ffffffff810dc8f7>] __mark_inode_dirty+0x33/0x190
[<ffffffff810d1536>] touch_atime+0x112/0x11d
[<ffffffff8108ef55>] generic_file_aio_read+0x53a/0x595
[<ffffffff810becb5>] do_sync_read+0xe7/0x12d
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff8103c847>] ? finish_task_switch+0x31/0xc9
[<ffffffff812c0812>] ? thread_return+0xab/0xd9
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf632>] vfs_read+0xa8/0x102
[<ffffffff810bf750>] sys_read+0x47/0x6e
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task rsync:11446 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsync D ffff8801ef08d380 0 11446 11438
ffff880111159948 0000000000000082 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff880201ec16e0 ffff8801006c96e0
ffff880201ec1a28 00000003a0286a3c ffff88021aad96c8 ffff880201ec1a28
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa0024149>] do_get_write_access+0x72/0x404 [jbd]
[<ffffffffa0027cfb>] ? journal_add_journal_head+0xc7/0x14e [jbd]
[<ffffffffa00244fd>] journal_get_write_access+0x22/0x33 [jbd]
[<ffffffffa0042806>] __ext3_journal_get_write_access+0x1f/0x48 [ext3]
[<ffffffffa0036391>] ext3_reserve_inode_write+0x3f/0x76 [ext3]
[<ffffffffa0036403>] ext3_mark_inode_dirty+0x3b/0x58 [ext3]
[<ffffffffa0036564>] ext3_dirty_inode+0x6c/0x83 [ext3]
[<ffffffff810dc8f7>] __mark_inode_dirty+0x33/0x190
[<ffffffff810d13e0>] file_update_time+0xbe/0x102
[<ffffffff8108e8f1>] __generic_file_aio_write_nolock+0x168/0x292
[<ffffffff812342ca>] ? __sock_recvmsg+0x6d/0x7a
[<ffffffff8108f233>] generic_file_aio_write+0x67/0xc3
[<ffffffffa003443f>] ext3_file_write+0x1e/0x9f [ext3]
[<ffffffff810beb88>] do_sync_write+0xe7/0x12d
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff8103c847>] ? finish_task_switch+0x31/0xc9
[<ffffffff812c20fa>] ? _spin_lock+0x9/0xc
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf444>] vfs_write+0xab/0x105
[<ffffffff810bf562>] sys_write+0x47/0x6f
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task smbd:26021 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000002 0 26021 2498
ffff88017e9a5bb8 0000000000000086 ffff88021a8c3a00 0000000000000000
ffffffff8162a500 ffffffff8162a500 ffff88019eceadc0 ffff88021b9fadc0
ffff88019eceb108 000000022801e568 ffff88017e9a5b88 ffff88019eceb108
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff8108d285>] sync_page+0x51/0x58
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff8108d234>] ? sync_page+0x0/0x58
[<ffffffff8108d4f1>] wait_on_page_bit+0x6e/0x75
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff810965d2>] wait_on_page_writeback+0x2a/0x2e
[<ffffffff81096e3c>] truncate_inode_pages_range+0x2e1/0x361
[<ffffffff8109e06e>] ? unmap_mapping_range+0x21c/0x22c
[<ffffffff810b7ac7>] ? virt_to_head_page+0x31/0x41
[<ffffffff81096ec9>] truncate_inode_pages+0xd/0x10
[<ffffffff8109e1f3>] vmtruncate+0x96/0xe4
[<ffffffff810d29c8>] inode_setattr+0x2b/0x125
[<ffffffffa003670b>] ext3_setattr+0x190/0x1f7 [ext3]
[<ffffffff810d2c58>] notify_change+0x196/0x2ee
[<ffffffff810be03d>] do_truncate+0x63/0x81
[<ffffffff810be13c>] sys_ftruncate+0xe1/0xfe
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task smbd:26023 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D ffff8801c5ab5a00 0 26023 2498
ffff8801d28c1af8 0000000000000086 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8801232516e0 ffff8800617444a0
ffff880123251a28 00000002a0286a3c ffff88021aad96c8 ffff880123251a28
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff81031dd9>] ? __wake_up_common+0x46/0x76
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa00236bb>] journal_invalidatepage+0x97/0x2a6 [jbd]
[<ffffffffa003736f>] ext3_invalidatepage+0x3c/0x3e [ext3]
[<ffffffff8109656d>] do_invalidatepage+0x20/0x22
[<ffffffff81096b31>] truncate_complete_page+0x2a/0x54
[<ffffffff81096c4f>] truncate_inode_pages_range+0xf4/0x361
[<ffffffff8109e06e>] ? unmap_mapping_range+0x21c/0x22c
[<ffffffff810b7ac7>] ? virt_to_head_page+0x31/0x41
[<ffffffff81096ec9>] truncate_inode_pages+0xd/0x10
[<ffffffff8109e1f3>] vmtruncate+0x96/0xe4
[<ffffffff810d29c8>] inode_setattr+0x2b/0x125
[<ffffffffa003670b>] ext3_setattr+0x190/0x1f7 [ext3]
[<ffffffff810d2c58>] notify_change+0x196/0x2ee
[<ffffffff810be03d>] do_truncate+0x63/0x81
[<ffffffff810be13c>] sys_ftruncate+0xe1/0xfe
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task spamd:8628 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
spamd D ffff88005c966700 0 8628 8626
ffff880101fad6d8 0000000000000086 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff88010060db80 ffff8800012396e0
ffff88010060dec8 00000000a0286a3c ffff88021aad96c8 ffff88010060dec8
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff8113cb81>] ? submit_bio+0xe0/0xe9
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffff810e154f>] wait_on_buffer+0x41/0x45
[<ffffffff810e34fc>] __bread+0x5e/0x7e
[<ffffffffa0036911>] ext3_get_branch+0x76/0xea [ext3]
[<ffffffffa00376ca>] ext3_get_blocks_handle+0x9d/0x858 [ext3]
[<ffffffff810935af>] ? __alloc_pages_internal+0xfe/0x457
[<ffffffff810b1323>] ? alloc_page_vma+0xc1/0xc6
[<ffffffff81099ca5>] ? __inc_zone_page_state+0x25/0x27
[<ffffffff810a6dfd>] ? page_add_new_anon_rmap+0x4c/0x4e
[<ffffffff8109ec50>] ? handle_mm_fault+0x856/0x890
[<ffffffff810448a3>] ? raise_softirq+0x41/0x58
[<ffffffffa0037f43>] ext3_get_block+0xbe/0xfc [ext3]
[<ffffffff811504a3>] ? __up_read+0x7a/0x83
[<ffffffff810e7c1e>] do_mpage_readpage+0x1a8/0x4d5
[<ffffffff8114fafd>] ? radix_tree_insert+0x186/0x1ca
[<ffffffff81099ca5>] ? __inc_zone_page_state+0x25/0x27
[<ffffffff8108d627>] ? add_to_page_cache_locked+0x9a/0xae
[<ffffffff81096290>] ? lru_cache_add+0x2b/0x5c
[<ffffffff810e804e>] mpage_readpages+0xb1/0xf4
[<ffffffffa0037e85>] ? ext3_get_block+0x0/0xfc [ext3]
[<ffffffff810935af>] ? __alloc_pages_internal+0xfe/0x457
[<ffffffffa0037e85>] ? ext3_get_block+0x0/0xfc [ext3]
[<ffffffff810d518e>] ? mntput_no_expire+0x31/0x144
[<ffffffffa00375a5>] ext3_readpages+0x1a/0x1c [ext3]
[<ffffffff81095490>] __do_page_cache_readahead+0xfc/0x172
[<ffffffff81095804>] ondemand_readahead+0x178/0x18a
[<ffffffff810958af>] page_cache_sync_readahead+0x17/0x1c
[<ffffffff8108ec4a>] generic_file_aio_read+0x22f/0x595
[<ffffffff810becb5>] do_sync_read+0xe7/0x12d
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff812c20fa>] ? _spin_lock+0x9/0xc
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf632>] vfs_read+0xa8/0x102
[<ffffffff810bf6e8>] sys_pread64+0x5c/0x7d
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task pdflush:10196 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pdflush D ffff8801c5ab4000 0 10196 2
ffff8801d71f1ad0 0000000000000046 ffff88021a8c3a00 0000000000000000
ffffffff8162a500 ffffffff8162a500 ffff8802114f96e0 ffff8801f8c0db80
ffff8802114f9a28 000000022801b628 ffff8801d71f1aa0 ffff8802114f9a28
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff8108d285>] sync_page+0x51/0x58
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff8108d234>] ? sync_page+0x0/0x58
[<ffffffff8108d1e7>] __lock_page+0x63/0x6a
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff810943e2>] write_cache_pages+0x1eb/0x3b4
[<ffffffff81010a37>] ? restore_args+0x0/0x30
[<ffffffff81093998>] ? __writepage+0x0/0x2f
[<ffffffff810945ca>] generic_writepages+0x1f/0x25
[<ffffffff810945ff>] do_writepages+0x2f/0x38
[<ffffffff810dbf95>] __writeback_single_inode+0x1a2/0x332
[<ffffffff81033569>] ? __dequeue_entity+0x61/0x6a
[<ffffffff8100e717>] ? __switch_to+0xb9/0x3e0
[<ffffffff810dc526>] generic_sync_sb_inodes+0x245/0x390
[<ffffffff810dc86b>] writeback_inodes+0xa4/0xfd
[<ffffffff8109474e>] wb_kupdate+0xa3/0x119
[<ffffffff81095163>] pdflush+0x16e/0x231
[<ffffffff810946ab>] ? wb_kupdate+0x0/0x119
[<ffffffff81094ff5>] ? pdflush+0x0/0x231
[<ffffffff81094ff5>] ? pdflush+0x0/0x231
[<ffffffff810534bf>] kthread+0x49/0x76
[<ffffffff81011719>] child_rip+0xa/0x11
[<ffffffff81010a37>] ? restore_args+0x0/0x30
[<ffffffff81053476>] ? kthread+0x0/0x76
[<ffffffff8101170f>] ? child_rip+0x0/0x11
INFO: task VirtualBox:25951 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D ffff8800709fc680 0 25951 25560
ffff880148fc16e8 0000000000000082 0000000000000096 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8801e514adc0 ffff8801f8c0db80
ffff8801e514b108 00000003a0286a3c ffff88021aad96c8 ffff8801e514b108
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff8113cb81>] ? submit_bio+0xe0/0xe9
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffff810e154f>] wait_on_buffer+0x41/0x45
[<ffffffff810e34fc>] __bread+0x5e/0x7e
[<ffffffffa0036911>] ext3_get_branch+0x76/0xea [ext3]
[<ffffffffa00376ca>] ext3_get_blocks_handle+0x9d/0x858 [ext3]
[<ffffffffa0037f43>] ext3_get_block+0xbe/0xfc [ext3]
[<ffffffff810e7c1e>] do_mpage_readpage+0x1a8/0x4d5
[<ffffffff8114fafd>] ? radix_tree_insert+0x186/0x1ca
[<ffffffff81099ca5>] ? __inc_zone_page_state+0x25/0x27
[<ffffffff8108d627>] ? add_to_page_cache_locked+0x9a/0xae
[<ffffffff81096290>] ? lru_cache_add+0x2b/0x5c
[<ffffffff810e804e>] mpage_readpages+0xb1/0xf4
[<ffffffffa0037e85>] ? ext3_get_block+0x0/0xfc [ext3]
[<ffffffff810935af>] ? __alloc_pages_internal+0xfe/0x457
[<ffffffffa0037e85>] ? ext3_get_block+0x0/0xfc [ext3]
[<ffffffffa00375a5>] ext3_readpages+0x1a/0x1c [ext3]
[<ffffffff81095490>] __do_page_cache_readahead+0xfc/0x172
[<ffffffff81095804>] ondemand_readahead+0x178/0x18a
[<ffffffff810958af>] page_cache_sync_readahead+0x17/0x1c
[<ffffffff8108ec4a>] generic_file_aio_read+0x22f/0x595
[<ffffffff810becb5>] do_sync_read+0xe7/0x12d
[<ffffffffa03d88a2>] ? RTMemFree+0x1e/0x20 [vboxdrv]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf632>] vfs_read+0xa8/0x102
[<ffffffff810bf750>] sys_read+0x47/0x6e
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task kjournald:1202 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff88021a845040 0 1202 2
ffff8802188a5ca0 0000000000000046 0000000000000086 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff88021956adc0 ffff8801001b2dc0
ffff88021956b108 00000003a0286a3c ffff88021aad96c8 ffff88021956b108
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff8113cb81>] ? submit_bio+0xe0/0xe9
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffffa0024deb>] wait_on_buffer+0x41/0x45 [jbd]
[<ffffffffa002540e>] journal_commit_transaction+0x55d/0xf2f [jbd]
[<ffffffff81049a0d>] ? try_to_del_timer_sync+0x58/0x63
[<ffffffffa0028ad8>] kjournald+0xe3/0x23a [jbd]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffffa00289f5>] ? kjournald+0x0/0x23a [jbd]
[<ffffffff810534bf>] kthread+0x49/0x76
[<ffffffff81011719>] child_rip+0xa/0x11
[<ffffffff81010a37>] ? restore_args+0x0/0x30
[<ffffffff81053476>] ? kthread+0x0/0x76
[<ffffffff8101170f>] ? child_rip+0x0/0x11
INFO: task VirtualBox:25932 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D 0000000000000002 0 25932 25560
ffff8801d9e51928 0000000000000086 0000000000000096 ffff88021b4505d0
ffffffff8162a500 ffffffff8162a500 ffff8800c2a20000 ffff88021b9fadc0
ffff8800c2a20348 00000000a0286a3c ffff88021aad96c8 ffff8800c2a20348
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff8113cb81>] ? submit_bio+0xe0/0xe9
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffff810e154f>] wait_on_buffer+0x41/0x45
[<ffffffff810e1b53>] sync_dirty_buffer+0x58/0xa7
[<ffffffffa0023eec>] journal_dirty_data+0xed/0x1cb [jbd]
[<ffffffffa00375d4>] ext3_journal_dirty_data+0x18/0x40 [ext3]
[<ffffffffa003557b>] walk_page_buffers+0x6e/0x96 [ext3]
[<ffffffffa00375bc>] ? ext3_journal_dirty_data+0x0/0x40 [ext3]
[<ffffffffa0036f75>] ext3_ordered_write_end+0x7d/0x127 [ext3]
[<ffffffff8108e167>] generic_file_buffered_write+0x1b8/0x643
[<ffffffff810d623b>] ? mnt_drop_write+0x82/0x143
[<ffffffff810d453d>] ? mnt_want_write+0x77/0x8d
[<ffffffff8108e9e7>] __generic_file_aio_write_nolock+0x25e/0x292
[<ffffffff8108f233>] generic_file_aio_write+0x67/0xc3
[<ffffffffa003443f>] ext3_file_write+0x1e/0x9f [ext3]
[<ffffffff810beb88>] do_sync_write+0xe7/0x12d
[<ffffffffa03d88a2>] ? RTMemFree+0x1e/0x20 [vboxdrv]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff812c20fa>] ? _spin_lock+0x9/0xc
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf444>] vfs_write+0xab/0x105
[<ffffffff810bf562>] sys_write+0x47/0x6f
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task VirtualBox:25970 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D 0000000000000002 0 25970 3154
ffff8801d6ed9948 0000000000000086 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8801f8c75b80 ffff88021b9fadc0
ffff8801f8c75ec8 00000000a0286a3c ffff88021aad96c8 ffff8801f8c75ec8
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa0024149>] do_get_write_access+0x72/0x404 [jbd]
[<ffffffffa0027cfb>] ? journal_add_journal_head+0xc7/0x14e [jbd]
[<ffffffffa00244fd>] journal_get_write_access+0x22/0x33 [jbd]
[<ffffffffa0042806>] __ext3_journal_get_write_access+0x1f/0x48 [ext3]
[<ffffffffa0036391>] ext3_reserve_inode_write+0x3f/0x76 [ext3]
[<ffffffffa0036403>] ext3_mark_inode_dirty+0x3b/0x58 [ext3]
[<ffffffffa0036564>] ext3_dirty_inode+0x6c/0x83 [ext3]
[<ffffffff810dc8f7>] __mark_inode_dirty+0x33/0x190
[<ffffffff810d13e0>] file_update_time+0xbe/0x102
[<ffffffff8108e8f1>] __generic_file_aio_write_nolock+0x168/0x292
[<ffffffff8108f233>] generic_file_aio_write+0x67/0xc3
[<ffffffffa003443f>] ext3_file_write+0x1e/0x9f [ext3]
[<ffffffff810beb88>] do_sync_write+0xe7/0x12d
[<ffffffffa03d88a2>] ? RTMemFree+0x1e/0x20 [vboxdrv]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf444>] vfs_write+0xab/0x105
[<ffffffff810bf562>] sys_write+0x47/0x6f
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task VirtualBox:25990 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D ffff880183ffd6c0 0 25990 3145
ffff880179ba1948 0000000000000086 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff88021151adc0 ffff8801001b2dc0
ffff88021151b108 00000003a0286a3c ffff88021aad96c8 ffff88021151b108
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa0024149>] do_get_write_access+0x72/0x404 [jbd]
[<ffffffffa0027cfb>] ? journal_add_journal_head+0xc7/0x14e [jbd]
[<ffffffffa00244fd>] journal_get_write_access+0x22/0x33 [jbd]
[<ffffffffa0042806>] __ext3_journal_get_write_access+0x1f/0x48 [ext3]
[<ffffffffa0036391>] ext3_reserve_inode_write+0x3f/0x76 [ext3]
[<ffffffffa0036403>] ext3_mark_inode_dirty+0x3b/0x58 [ext3]
[<ffffffffa0036564>] ext3_dirty_inode+0x6c/0x83 [ext3]
[<ffffffff810dc8f7>] __mark_inode_dirty+0x33/0x190
[<ffffffff810d13e0>] file_update_time+0xbe/0x102
[<ffffffff8108e8f1>] __generic_file_aio_write_nolock+0x168/0x292
[<ffffffff8108f233>] generic_file_aio_write+0x67/0xc3
[<ffffffffa003443f>] ext3_file_write+0x1e/0x9f [ext3]
[<ffffffff810beb88>] do_sync_write+0xe7/0x12d
[<ffffffffa03d88a2>] ? RTMemFree+0x1e/0x20 [vboxdrv]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf444>] vfs_write+0xab/0x105
[<ffffffff810bf562>] sys_write+0x47/0x6f
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task VirtualBox:20126 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D 0000000000000002 0 20126 3145
ffff880090a2b9a8 0000000000000082 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8801d6075b80 ffff88021b9fadc0
ffff8801d6075ec8 00000000a0286a3c ffff88021aad96c8 ffff8801d6075ec8
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa0024149>] do_get_write_access+0x72/0x404 [jbd]
[<ffffffffa0027cfb>] ? journal_add_journal_head+0xc7/0x14e [jbd]
[<ffffffffa00244fd>] journal_get_write_access+0x22/0x33 [jbd]
[<ffffffffa0042806>] __ext3_journal_get_write_access+0x1f/0x48 [ext3]
[<ffffffffa0036391>] ext3_reserve_inode_write+0x3f/0x76 [ext3]
[<ffffffffa0036403>] ext3_mark_inode_dirty+0x3b/0x58 [ext3]
[<ffffffffa0036564>] ext3_dirty_inode+0x6c/0x83 [ext3]
[<ffffffff810dc8f7>] __mark_inode_dirty+0x33/0x190
[<ffffffff810d1536>] touch_atime+0x112/0x11d
[<ffffffff8108ef55>] generic_file_aio_read+0x53a/0x595
[<ffffffff810becb5>] do_sync_read+0xe7/0x12d
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff8103c847>] ? finish_task_switch+0x31/0xc9
[<ffffffff812c0812>] ? thread_return+0xab/0xd9
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf632>] vfs_read+0xa8/0x102
[<ffffffff810bf750>] sys_read+0x47/0x6e
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task smbd:32220 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D ffff880183ffc340 0 32220 2498
ffff8800c3771b78 0000000000000082 0000000000000096 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8800010ddb80 ffff8801001b2dc0
ffff8800010ddec8 00000003a0286a3c ffff88021aad96c8 ffff8800010ddec8
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffffa0035cc7>] wait_on_buffer+0x41/0x45 [ext3]
[<ffffffffa0038160>] ext3_bread+0x4c/0x76 [ext3]
[<ffffffffa003cb48>] htree_dirblock_to_tree+0x33/0x13d [ext3]
[<ffffffffa003ccc4>] ext3_htree_fill_tree+0x72/0x1ea [ext3]
[<ffffffff810c817b>] ? path_walk+0xb7/0xc4
[<ffffffff810b7ac7>] ? virt_to_head_page+0x31/0x41
[<ffffffff810b7ac7>] ? virt_to_head_page+0x31/0x41
[<ffffffffa0033e6b>] ext3_readdir+0x188/0x5c8 [ext3]
[<ffffffff810caa56>] ? filldir+0x0/0xc5
[<ffffffff810ccdcd>] ? locks_free_lock+0x4a/0x4e
[<ffffffff810cdd41>] ? fcntl_setlk+0x29a/0x2ae
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff810caa56>] ? filldir+0x0/0xc5
[<ffffffff810cac3d>] vfs_readdir+0x79/0xaf
[<ffffffff810cadb2>] sys_getdents+0x7d/0xc4
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task rsync:11446 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsync D ffff8801ef08d380 0 11446 11438
ffff880111159948 0000000000000082 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff880201ec16e0 ffff8801006c96e0
ffff880201ec1a28 00000003a0286a3c ffff88021aad96c8 ffff880201ec1a28
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa0024149>] do_get_write_access+0x72/0x404 [jbd]
[<ffffffffa0027cfb>] ? journal_add_journal_head+0xc7/0x14e [jbd]
[<ffffffffa00244fd>] journal_get_write_access+0x22/0x33 [jbd]
[<ffffffffa0042806>] __ext3_journal_get_write_access+0x1f/0x48 [ext3]
[<ffffffffa0036391>] ext3_reserve_inode_write+0x3f/0x76 [ext3]
[<ffffffffa0036403>] ext3_mark_inode_dirty+0x3b/0x58 [ext3]
[<ffffffffa0036564>] ext3_dirty_inode+0x6c/0x83 [ext3]
[<ffffffff810dc8f7>] __mark_inode_dirty+0x33/0x190
[<ffffffff810d13e0>] file_update_time+0xbe/0x102
[<ffffffff8108e8f1>] __generic_file_aio_write_nolock+0x168/0x292
[<ffffffff812342ca>] ? __sock_recvmsg+0x6d/0x7a
[<ffffffff8108f233>] generic_file_aio_write+0x67/0xc3
[<ffffffffa003443f>] ext3_file_write+0x1e/0x9f [ext3]
[<ffffffff810beb88>] do_sync_write+0xe7/0x12d
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff8103c847>] ? finish_task_switch+0x31/0xc9
[<ffffffff812c20fa>] ? _spin_lock+0x9/0xc
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf444>] vfs_write+0xab/0x105
[<ffffffff810bf562>] sys_write+0x47/0x6f
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task smbd:26021 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000002 0 26021 2498
ffff88017e9a5bb8 0000000000000086 ffff88021a8c3a00 0000000000000000
ffffffff8162a500 ffffffff8162a500 ffff88019eceadc0 ffff88021b9fadc0
ffff88019eceb108 000000022801e568 ffff88017e9a5b88 ffff88019eceb108
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff8108d285>] sync_page+0x51/0x58
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff8108d234>] ? sync_page+0x0/0x58
[<ffffffff8108d4f1>] wait_on_page_bit+0x6e/0x75
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff810965d2>] wait_on_page_writeback+0x2a/0x2e
[<ffffffff81096e3c>] truncate_inode_pages_range+0x2e1/0x361
[<ffffffff8109e06e>] ? unmap_mapping_range+0x21c/0x22c
[<ffffffff810b7ac7>] ? virt_to_head_page+0x31/0x41
[<ffffffff81096ec9>] truncate_inode_pages+0xd/0x10
[<ffffffff8109e1f3>] vmtruncate+0x96/0xe4
[<ffffffff810d29c8>] inode_setattr+0x2b/0x125
[<ffffffffa003670b>] ext3_setattr+0x190/0x1f7 [ext3]
[<ffffffff810d2c58>] notify_change+0x196/0x2ee
[<ffffffff810be03d>] do_truncate+0x63/0x81
[<ffffffff810be13c>] sys_ftruncate+0xe1/0xfe
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task smbd:26023 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D ffff8801c5ab5a00 0 26023 2498
ffff8801d28c1af8 0000000000000086 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8801232516e0 ffff8800617444a0
ffff880123251a28 00000002a0286a3c ffff88021aad96c8 ffff880123251a28
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0d8d>] out_of_line_wait_on_bit_lock+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff81031dd9>] ? __wake_up_common+0x46/0x76
[<ffffffff810e1a7c>] __lock_buffer+0x25/0x27
[<ffffffffa002361f>] lock_buffer+0x35/0x3a [jbd]
[<ffffffffa00236bb>] journal_invalidatepage+0x97/0x2a6 [jbd]
[<ffffffffa003736f>] ext3_invalidatepage+0x3c/0x3e [ext3]
[<ffffffff8109656d>] do_invalidatepage+0x20/0x22
[<ffffffff81096b31>] truncate_complete_page+0x2a/0x54
[<ffffffff81096c4f>] truncate_inode_pages_range+0xf4/0x361
[<ffffffff8109e06e>] ? unmap_mapping_range+0x21c/0x22c
[<ffffffff810b7ac7>] ? virt_to_head_page+0x31/0x41
[<ffffffff81096ec9>] truncate_inode_pages+0xd/0x10
[<ffffffff8109e1f3>] vmtruncate+0x96/0xe4
[<ffffffff810d29c8>] inode_setattr+0x2b/0x125
[<ffffffffa003670b>] ext3_setattr+0x190/0x1f7 [ext3]
[<ffffffff810d2c58>] notify_change+0x196/0x2ee
[<ffffffff810be03d>] do_truncate+0x63/0x81
[<ffffffff810be13c>] sys_ftruncate+0xe1/0xfe
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task spamd:8628 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
spamd D ffff88005c966700 0 8628 8626
ffff880101fad6d8 0000000000000086 0000000000000092 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff88010060db80 ffff8800012396e0
ffff88010060dec8 00000000a0286a3c ffff88021aad96c8 ffff88010060dec8
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff8113cb81>] ? submit_bio+0xe0/0xe9
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffff810e154f>] wait_on_buffer+0x41/0x45
[<ffffffff810e34fc>] __bread+0x5e/0x7e
[<ffffffffa0036911>] ext3_get_branch+0x76/0xea [ext3]
[<ffffffffa00376ca>] ext3_get_blocks_handle+0x9d/0x858 [ext3]
[<ffffffff810935af>] ? __alloc_pages_internal+0xfe/0x457
[<ffffffff810b1323>] ? alloc_page_vma+0xc1/0xc6
[<ffffffff81099ca5>] ? __inc_zone_page_state+0x25/0x27
[<ffffffff810a6dfd>] ? page_add_new_anon_rmap+0x4c/0x4e
[<ffffffff8109ec50>] ? handle_mm_fault+0x856/0x890
[<ffffffff810448a3>] ? raise_softirq+0x41/0x58
[<ffffffffa0037f43>] ext3_get_block+0xbe/0xfc [ext3]
[<ffffffff811504a3>] ? __up_read+0x7a/0x83
[<ffffffff810e7c1e>] do_mpage_readpage+0x1a8/0x4d5
[<ffffffff8114fafd>] ? radix_tree_insert+0x186/0x1ca
[<ffffffff81099ca5>] ? __inc_zone_page_state+0x25/0x27
[<ffffffff8108d627>] ? add_to_page_cache_locked+0x9a/0xae
[<ffffffff81096290>] ? lru_cache_add+0x2b/0x5c
[<ffffffff810e804e>] mpage_readpages+0xb1/0xf4
[<ffffffffa0037e85>] ? ext3_get_block+0x0/0xfc [ext3]
[<ffffffff810935af>] ? __alloc_pages_internal+0xfe/0x457
[<ffffffffa0037e85>] ? ext3_get_block+0x0/0xfc [ext3]
[<ffffffff810d518e>] ? mntput_no_expire+0x31/0x144
[<ffffffffa00375a5>] ext3_readpages+0x1a/0x1c [ext3]
[<ffffffff81095490>] __do_page_cache_readahead+0xfc/0x172
[<ffffffff81095804>] ondemand_readahead+0x178/0x18a
[<ffffffff810958af>] page_cache_sync_readahead+0x17/0x1c
[<ffffffff8108ec4a>] generic_file_aio_read+0x22f/0x595
[<ffffffff810becb5>] do_sync_read+0xe7/0x12d
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff812c20fa>] ? _spin_lock+0x9/0xc
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf632>] vfs_read+0xa8/0x102
[<ffffffff810bf6e8>] sys_pread64+0x5c/0x7d
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1b
INFO: task pdflush:10196 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pdflush D ffff8801c5ab4000 0 10196 2
ffff8801d71f1ad0 0000000000000046 ffff88021a8c3a00 0000000000000000
ffffffff8162a500 ffffffff8162a500 ffff8802114f96e0 ffff8801f8c0db80
ffff8802114f9a28 000000022801b628 ffff8801d71f1aa0 ffff8802114f9a28
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff8108d285>] sync_page+0x51/0x58
[<ffffffff812c0cdc>] __wait_on_bit_lock+0x45/0x8c
[<ffffffff8108d234>] ? sync_page+0x0/0x58
[<ffffffff8108d1e7>] __lock_page+0x63/0x6a
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff810943e2>] write_cache_pages+0x1eb/0x3b4
[<ffffffff81010a37>] ? restore_args+0x0/0x30
[<ffffffff81093998>] ? __writepage+0x0/0x2f
[<ffffffff810945ca>] generic_writepages+0x1f/0x25
[<ffffffff810945ff>] do_writepages+0x2f/0x38
[<ffffffff810dbf95>] __writeback_single_inode+0x1a2/0x332
[<ffffffff81033569>] ? __dequeue_entity+0x61/0x6a
[<ffffffff8100e717>] ? __switch_to+0xb9/0x3e0
[<ffffffff810dc526>] generic_sync_sb_inodes+0x245/0x390
[<ffffffff810dc86b>] writeback_inodes+0xa4/0xfd
[<ffffffff8109474e>] wb_kupdate+0xa3/0x119
[<ffffffff81095163>] pdflush+0x16e/0x231
[<ffffffff810946ab>] ? wb_kupdate+0x0/0x119
[<ffffffff81094ff5>] ? pdflush+0x0/0x231
[<ffffffff81094ff5>] ? pdflush+0x0/0x231
[<ffffffff810534bf>] kthread+0x49/0x76
[<ffffffff81011719>] child_rip+0xa/0x11
[<ffffffff81010a37>] ? restore_args+0x0/0x30
[<ffffffff81053476>] ? kthread+0x0/0x76
[<ffffffff8101170f>] ? child_rip+0x0/0x11
INFO: task VirtualBox:25951 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
VirtualBox D ffff8800709fc680 0 25951 25560
ffff880148fc16e8 0000000000000082 0000000000000096 ffff8802195e8000
ffffffff8162a500 ffffffff8162a500 ffff8801e514adc0 ffff8801f8c0db80
ffff8801e514b108 00000003a0286a3c ffff88021aad96c8 ffff8801e514b108
Call Trace:
[<ffffffff8101686f>] ? read_tsc+0xe/0x24
[<ffffffff810590b6>] ? getnstimeofday+0x54/0xb0
[<ffffffff812c08a3>] io_schedule+0x63/0xa5
[<ffffffff810e15e1>] sync_buffer+0x3b/0x3f
[<ffffffff812c0de1>] __wait_on_bit+0x47/0x79
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff810e15a6>] ? sync_buffer+0x0/0x3f
[<ffffffff812c0e7d>] out_of_line_wait_on_bit+0x6a/0x77
[<ffffffff81053861>] ? wake_bit_function+0x0/0x2a
[<ffffffff8113cb81>] ? submit_bio+0xe0/0xe9
[<ffffffff810e150a>] __wait_on_buffer+0x36/0x3a
[<ffffffff810e154f>] wait_on_buffer+0x41/0x45
[<ffffffff810e34fc>] __bread+0x5e/0x7e
[<ffffffffa0036911>] ext3_get_branch+0x76/0xea [ext3]
[<ffffffffa00376ca>] ext3_get_blocks_handle+0x9d/0x858 [ext3]
[<ffffffffa0037f43>] ext3_get_block+0xbe/0xfc [ext3]
[<ffffffff810e7c1e>] do_mpage_readpage+0x1a8/0x4d5
[<ffffffff8114fafd>] ? radix_tree_insert+0x186/0x1ca
[<ffffffff81099ca5>] ? __inc_zone_page_state+0x25/0x27
[<ffffffff8108d627>] ? add_to_page_cache_locked+0x9a/0xae
[<ffffffff81096290>] ? lru_cache_add+0x2b/0x5c
[<ffffffff810e804e>] mpage_readpages+0xb1/0xf4
[<ffffffffa0037e85>] ? ext3_get_block+0x0/0xfc [ext3]
[<ffffffff810935af>] ? __alloc_pages_internal+0xfe/0x457
[<ffffffffa0037e85>] ? ext3_get_block+0x0/0xfc [ext3]
[<ffffffffa00375a5>] ext3_readpages+0x1a/0x1c [ext3]
[<ffffffff81095490>] __do_page_cache_readahead+0xfc/0x172
[<ffffffff81095804>] ondemand_readahead+0x178/0x18a
[<ffffffff810958af>] page_cache_sync_readahead+0x17/0x1c
[<ffffffff8108ec4a>] generic_file_aio_read+0x22f/0x595
[<ffffffff810becb5>] do_sync_read+0xe7/0x12d
[<ffffffffa03d88a2>] ? RTMemFree+0x1e/0x20 [vboxdrv]
[<ffffffff81053829>] ? autoremove_wake_function+0x0/0x38
[<ffffffff81031103>] ? need_resched+0x1e/0x28
[<ffffffff81120e84>] ? security_file_permission+0x11/0x13
[<ffffffff810bf632>] vfs_read+0xa8/0x102
[<ffffffff810bf750>] sys_read+0x47/0x6e
[<ffffffff8101027a>] system_call_fastpath+0x16/0x1bI was getting those previously as well after I changed the backplanes and drives but it just went away. Now it's back. This causes all Linux VMs to crap out with disk I/O errors. The windows vms are fine. The system is Fedora Core 9 running Virtualbox 3.2.12. I know there's newer versions, but that's besides the point. Everything was ok before the power outage.
I put over 1k into replacing everything I can in hopes it goes away, and no luck I guess. I thought I was doing good given it's been running ok for a while.
. Any suggestions on what else to try? Motherboard? Sata controller? Right now I'm using the mobo + 2 pcie controllers. Now they make 8 port + controllers so I'm wondering if I should get one of those. It's hard to find ones that don't do raid though. I had bought one but learned the hard way that if it has raid you HAVE to use it, it wont present the drives normally to the OS before they are initialized. I'm running out of money here, but this system just has to work.
Facebook
Twitter