人人网 - 抱歉
哦，抱歉，好像看不到了
现在你可以:
北京千橡网景科技发展有限公司：
文网文[号??京公网安备号?甲测资字
文化部监督电子邮箱：wlwh@??
文明办网文明上网举报***： 举报邮箱：&&&&&&&&&&&&Re: [PATCH v5 7/8] mm: Only IPI CPUs to drain local pages if they exist
[][][][][][]
To: Mel Gorman &&
Subject: Re: [PATCH v5 7/8] mm: Only IPI CPUs to drain local pages if they exist
From: &Srivatsa S. Bhat& &&
Date: Fri, 06 Jan :11 +0530
Cc: &Paul E. McKenney& &&,
Russell King - ARM Linux &&,
KOSAKI Motohiro &&,
Gilad Ben-Yossef &&, ,
Chris Metcalf &&,
Peter Zijlstra &&,
Frederic Weisbecker &&, ,
Pekka Enberg &&, Matt Mackall &&,
Sasha Levin &&, Rik van Riel &&,
Andi Kleen &&,
Andrew Morton &&,
Alexander Viro &&, Greg KH &&,
, Avi Kivity &&
In-reply-to: &&
User-agent: Mozilla/5.0 (X11; Linux i686; rv:7.0) Gecko/ Thunderbird/7.0
On 01/06/ AM, Mel Gorman wrote:
& (Adding Greg to cc to see if he recalls seeing issues with sysfs dentry
& suffering from recursive locking recently)
& On Thu, Jan 05, 2012 at 10:35:04AM -0800, Paul E. McKenney wrote:
&& On Thu, Jan 05, 2012 at 04:35:29PM +0000, Russell King - ARM Linux wrote:
&&& On Thu, Jan 05, 2012 at 04:17:39PM +0000, Mel Gorman wrote:
&&&& Link please?
&&& Forwarded, as its still in my mailbox.
&&&& I'm including a patch below under development that is
&&&& intended to only cope with the page allocator case under heavy memory
&&&& pressure. Currently it does not pass testing because eventually RCU
&&&& gets stalled with the following trace
[&ffffffff&] arch_trigger_all_cpu_backtrace+0x87/0xa0
[&ffffffff810c4779&] __rcu_pending+0x149/0x260
[&ffffffff810c48ef&] rcu_check_callbacks+0x5f/0x110
[&ffffffff81068d7f&] update_process_times+0x3f/0x80
[&ffffffff8108c4eb&] tick_sched_timer+0x5b/0xc0
[&ffffffff8107f28e&] __run_hrtimer+0xbe/0x1a0
[&ffffffff&] hrtimer_interrupt+0xc1/0x1e0
[&ffffffff81020ef3&] smp_apic_timer_interrupt+0x63/0xa0
[&ffffffff&] apic_timer_interrupt+0x13/0x20
[&ffffffff&] vfsmount_lock_local_lock+0x25/0x30
[&ffffffff&] path_init+0x2d5/0x370
[&ffffffff8115eecd&] path_lookupat+0x2d/0x620
[&ffffffff8115f4ef&] do_path_lookup+0x2f/0xd0
[&ffffffff811602af&] user_path_at_empty+0x9f/0xd0
[&ffffffff81154e7b&] vfs_fstatat+0x4b/0x90
[&ffffffff81154f4f&] sys_newlstat+0x1f/0x50
[&ffffffff&] system_call_fastpath+0x16/0x1b
&&&& It might be a separate bug, don't know for sure.
& I rebased the patch on top of 3.2 and tested again with a bunch of
& debugging options set (PROVE_RCU, PROVE_LOCKING etc). Same results. CPU
& hotplug is a lot more reliable and less likely to hang but eventually
& gets into trouble.
Hi everyone,
I was running some CPU hotplug stress tests recently and found it to be
problematic too. Mel, I have some logs from those tests which appear very
relevant to the &IPI to offline CPU& issue that has been discussed in this
Kernel: 3.2-rc7
Here is the log:
(Unfortunately I couldn't capture the log intact, due to some annoying
serial console issues, but I hope this log is good enough to analyze.)
907.825267] Booting Node 1 Processor 15 APIC 0x17
907.830117] smpboot cpu 15: start_ip = 97000
906.104006] Calibrating delay loop (skipped) already calibrated this CPU
907.860875] NMI watchdog enabled, takes one hw-pmu counter.
907.898899] Broke affinity for irq 81
907.904539] CPU 1 is now offline
907.912891] CPU 9 MCA banks CMCI:2 CMCI:3 CMCI:5
907.929462] CPU 2 is now offline
907.939573] CPU 10 MCA banks CMCI:2 CMCI:3 CMCI:5
907.969514] CPU 3 is now offline
907.978644] CPU 11 MCA banks CMCI:2 CMCI:3 CMCI:5
908.021903] Broke affinity for irq 74
908.024021] ------------[ cut here ]------------
908.024021] WARNING: at kernel/smp.c:258 generic_smp_call_function_single_interrupt+0x109/0x120()
908.024021] Hardware name: IBM System x -[7870C4Q]-
908.024021] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
908.024021] Pid: 22076, comm: migration/4 Not tainted 3.2.0-rc7-0.0.0.28.36b5ec9-default #1
908.024021] Call Trace:
908.024021]
[&ffffffff&] ? generic_smp_call_function_single_interrupt+0x109/0x120
908.024021]
[&ffffffff8105441a&] warn_slowpath_common+0x7a/0xb0
908.024021]
[&ffffffff&] warn_slowpath_null+0x15/0x20
908.024021]
[&ffffffff&] generic_smp_call_function_single_interrupt+0x109/0x120
908.024021]
[&ffffffff8101ffa2&] smp_call_function_single_interrupt+0x22/0x40
908.024021]
[&ffffffff8146afb3&] call_function_single_interrupt+0x73/0x80
908.024021]
[&ffffffff810ba86a&] ? stop_machine_cpu_stop+0xda/0x130
908.024021]
[&ffffffff810ba790&] ? stop_one_cpu_nowait+0x50/0x50
908.024021]
[&ffffffff810ba4ea&] cpu_stopper_thread+0xba/0x180
908.024021]
[&ffffffff8146077f&] ? _raw_spin_unlock_irqrestore+0x3f/0x70
908.024021]
[&ffffffff810ba430&] ? res_counter_init+0x50/0x50
908.024021]
[&ffffffff8109141d&] ? trace_hardirqs_on_caller+0x12d/0x1b0
908.024021]
[&ffffffff810914ad&] ? trace_hardirqs_on+0xd/0x10
908.024021]
[&ffffffff810ba430&] ? res_counter_init+0x50/0x50
908.024021]
[&ffffffff81078cf6&] kthread+0x96/0xa0
908.024021]
[&ffffffff&] kernel_thread_helper+0x4/0x10
908.024021]
[&ffffffff81460ab4&] ? retint_restore_args+0x13/0x13
908.024021]
[&ffffffff81078c60&] ? __init_kthread_worker+0x70/0x70
908.024021]
[&ffffffff&] ? gs_change+0x13/0x13
908.024021] ---[ end trace f4c7a25be63a672a ]---
908.328208] CPU 4 is now offline
908.332730] CPU 5 MCA banks CMCI:6 CMCI:8
908.337074] CPU 12 MCA banks CMCI:2 CMCI:3 CMCI:5
908.349270] CPU 5 is now offline
908.353888] CPU 6 MCA banks CMCI:6 CMCI:8
908.376131] CPU 13 MCA banks CMCI:2 CMCI:3 CMCI:5
908.391939] CPU 6 is now offline
908.413193] CPU 7 MCA banks CMCI:6 CMCI:8
908.443245] CPU 14 MCA banks CMCI:2 CMCI:3 CMCI:5
908.475871] CPU 7 is now offline
908.481601] CPU 12 MCA banks CMCI:6 CMCI:8
908.485923] CPU 15 MCA banks CMCI:2 CMCI:3 CMCI:5
908.519889] CPU 8 is now offline
908.565926] CPU 9 is now offline
908.602874] CPU 10 is now offline
908.634696] CPU 11 is now offline
908.674735] CPU 12 is now offline
908.680343] CPU 13 MCA banks CMCI:6 CMCI:8
908.721887] CPU 13 is now offline
908.728086] CPU 14 MCA banks CMCI:6 CMCI:8
908.789105] CPU 14 is now offline
908.794969] CPU 15 MCA banks CMCI:6 CMCI:8
908.881878] CPU 15 is now offline
908.885301] lockdep: fixing up alternatives.
908.889663] SMP alternatives: switching to UP code
909.140900] lockdep: fixing up alternatives.
909.145281] SMP alternatives: switching to SMP code
909.153536] Booting Node 0 Processor 1 APIC 0x2
909.158157] smpboot cpu 1: start_ip = 97000
907.900022] Calibrating delay loop (skipped) already calibrated this CPU
909.181323] NMI watchdog enabled, takes one hw-pmu counter.
909.275696] lockdep: fixing up alternatives.
909.280106] Booting Node 0 Processor 2 APIC 0x4
909.280107] smpboot cpu 2: start_ip = 97000
907.928015] Calibrating delay loop (skipped) already calibrated this CPU
909.308538] NMI watchdog enabled, takes one hw-pmu counter.
909.376170] lockdep: fixing up alternatives.
909.380589] Booting Node 0 Processor 3 APIC 0x6
[ ] Booting Node 1 Processor 14 APIC 0x15
[ ] smpboot cpu 14: start_ip = 97000
[ ] Calibrating delay loop (skipped) already calibrated this CPU
[ ] NMI watchdog enabled, takes one hw-pmu counter.
[ ] lockdep: fixing up alternatives.
[ ] Booting Node 1 Processor 15 APIC 0x17
[ ] smpboot cpu 15: start_ip = 97000
[ ] Calibrating delay loop (skipped) already calibrated this CPU
[ ] NMI watchdog enabled, takes one hw-pmu counter.
[ ] CPU 1 is now offline
[ ] CPU 9 MCA banks CMCI:2 CMCI:3 CMCI:5
[ ] CPU 2 is now offline
[ ] CPU 10 MCA banks CMCI:2 CMCI:3 CMCI:5
[ ] CPU 3 is now offline
[ ] CPU 11 MCA banks CMCI:2 CMCI:3 CMCI:5
[ ] CPU 4 is now offline
[ ] CPU 5 MCA banks CMCI:6 CMCI:8
[ ] CPU 12 MCA banks CMCI:2 CMCI:3 CMCI:5
[ ] CPU 5 is now offline
[ ] CPU 6 MCA banks CMCI:6 CMCI:8
[ ] CPU 13 MCA banks CMCI:2 CMCI:3 CMCI:5
[ ] CPU 6 is now offline
[ ] CPU 7 MCA banks CMCI:6 CMCI:8
[ ] CPU 14 MCA banks CMCI:2 CMCI:3 CMCI:5
[ ] CPU 7 is now offline
[ ] CPU 12 MCA banks CMCI:6 CMCI:8
[ ] CPU 15 MCA banks CMCI:2 CMCI:3 CMCI:5
[ ] CPU 8 is now offline
[ ] CPU 9 is now offline
[ ] CPU 10 is now offline
[ ] CPU 11 is now offline
[ ] CPU 12 is now offline
[ ] CPU 13 MCA banks CMCI:6 CMCI:8
[ ] CPU 13 is now offline
[ ] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff81033de8&]
[&ffffffff81033de8&] flush_tlb_others_ipi+0x108/0x140
[ ] RSP: 0000:ffff881147bfdc48
[ ] RAX: 0000 RBX: ffffffff81460ab4 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc78 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffff881147bfdbb8
[ ] R13: ffffc80 R14: ffff881147bfc000 R15: 0000
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffff881147bfdcb8 ffffd80 00007fff56156db8 ffffde0
ffff ffffab0 ffff881147bfdc88 ffffffff81033e29
ffff881147bfdcb8 ffffffff81033f2a ffffdf8 0001
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 8b 35 00 4f 9b 00 &4c& 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66 2e 0f 1f 84 00 00
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] INFO: rcu_sched detected stall on CPU 14 (t=16250 jiffies)
[ ] sending NMI to all CPUs:
[ ] INFO: rcu_sched detected stalls on CPUs/tasks: { 14} (detected by 15, t=16252 jiffies)
[ ] NMI backtrace for cpu 0
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 24575, comm: cc1 Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0033:[&00002abf6f4f7c48&]
[&00002abf6f4f7c48&] 0x2abf6f4f7c47
[ ] RSP: 002b:00007fffa24587e8
[ ] RAX: 002b RBX: 00007fffa24587f0 RCX: 0004
[ ] RDX: 0000 RSI: 00002abf RDI: 00007fffa24587f0
[ ] RBP: 00007fffa: fffffffffffffffc R09: 00007fffa245892e
[ ] R10: 00002abf6ee686c0 R11: 00002abf6f4fa1c6 R12: 0000
[ ] R13: 00002abf: 007fffa245892f
00002abf6f7d87e0(0000) GS:ffff8808ffc) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 0033
[ ] CR2: 00002abf CR3: e4000 CR4: 06f0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process cc1 (pid: 24575, threadinfo ffff0, task ffff0)
[ ] Call Trace:
[ ] NMI backtrace for cpu 14
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff81258d62&]
[&ffffffff81258d62&] delay_tsc+0x42/0xa0
[ ] RSP: 0000:ffff8
[ ] RAX: 00d4 RBX: 70af RCX: ffffffff8ac50e81
[ ] RDX: ac50e81 RSI: 000f RDI: 70af
[ ] RBP: ffff8 R08: 0010 R09: ffffffff819e7660
[ ] R10: 03 R12: 1000
[ ] R13: 0e R15: 000e
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffffffff8ac50dad
00000cbc0 0002 ffff8
ffffffff81258e1f ffff8 ffffffff81021aaa 000f
[ ] Call Trace:
[&ffffffff81258e1f&] __const_udelay+0x2f/0x40
[&ffffffff81021aaa&] native_safe_apic_wait_icr_idle+0x1a/0x50
[&ffffffff81021fbd&] default_send_IPI_mask_sequence_phys+0xdd/0x130
[&ffffffff&] physflat_send_IPI_all+0x14/0x20
[&ffffffff&] arch_trigger_all_cpu_backtrace+0x67/0xb0
[&ffffffff810cfca9&] __rcu_pending+0x119/0x280
[&ffffffff810cfeba&] rcu_check_callbacks+0xaa/0x1b0
[&ffffffff&] update_process_times+0x41/0x80
[&ffffffff8108b05f&] tick_sched_timer+0x5f/0xc0
[&ffffffff&] ? tick_nohz_handler+0x100/0x100
[&ffffffff&] __run_hrtimer+0xd1/0x1d0
[&ffffffff8107dc97&] hrtimer_interrupt+0xc7/0x1f0
[&ffffffff81021a34&] smp_apic_timer_interrupt+0x64/0xa0
[&ffffffff81469db3&] apic_timer_interrupt+0x73/0x80
[&ffffffff81033de2&] ? flush_tlb_others_ipi+0x102/0x140
[&ffffffff81033df0&] ? flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: 34 25 b0 cb 00 00 66 66 90 0f ae e8 e8 e8 15 db ff 66 90 48 98 44 89 75 d4 48 89 45 c8 eb 1b 66 2e 0f 1f 84 00 00 00 00 00 f3 90 &65& 44 8b 3c 25 b0 cb 00 00 44 3b 7d d4 75 2b 66 66 90 0f ae e8
[ ] Call Trace:
[&ffffffff81258e1f&] __const_udelay+0x2f/0x40
[&ffffffff81021aaa&] native_safe_apic_wait_icr_idle+0x1a/0x50
[&ffffffff81021fbd&] default_send_IPI_mask_sequence_phys+0xdd/0x130
[&ffffffff&] physflat_send_IPI_all+0x14/0x20
[&ffffffff&] arch_trigger_all_cpu_backtrace+0x67/0xb0
[&ffffffff810cfca9&] __rcu_pending+0x119/0x280
[&ffffffff810cfeba&] rcu_check_callbacks+0xaa/0x1b0
[&ffffffff&] update_process_times+0x41/0x80
[&ffffffff8108b05f&] tick_sched_timer+0x5f/0xc0
[&ffffffff&] ? tick_nohz_handler+0x100/0x100
[&ffffffff&] __run_hrtimer+0xd1/0x1d0
[&ffffffff8107dc97&] hrtimer_interrupt+0xc7/0x1f0
[&ffffffff81021a34&] smp_apic_timer_interrupt+0x64/0xa0
[&ffffffff81469db3&] apic_timer_interrupt+0x73/0x80
[&ffffffff81033de2&] ? flush_tlb_others_ipi+0x102/0x140
[&ffffffff81033df0&] ? flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] NMI backtrace for cpu 15
[ ] CPU 15
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 24569, comm: sh Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0033:[&0c59&]
[&0c59&] 0x2b292b250c58
[ ] RSP: 002b:00007fff6bb94b70
[ ] RAX: 0580 RBX: b0907 RCX: 00007fff6bb94c20
[ ] RDX: 00007fff6bb94bd0 RSI: b0907 RDI: 00007fff6bb94bd0
[ ] RBP: 001b R08: b01
[ ] R10: 007fff6bb94c20 R12: 0000
[ ] R13: b007fff6bb94bd0 R15: eba0
eba0(0000) GS:ffff88117fdc) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 0033
[ ] CR2: b1110 CR3: d000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process sh (pid: 24569, threadinfo ffff, task ffff)
[ ] Call Trace:
[ ] BUG: soft lockup - CPU#14 stuck for 23s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff81033de2&]
[&ffffffff81033de2&] flush_tlb_others_ipi+0x102/0x140
[ ] RSP: 0000:ffff881147bfdc48
[ ] RAX: 0000 RBX: ffffffff81460ab4 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc78 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffff881147bfdbb8
[ ] R13: ffffc80 R14: ffff881147bfc000 R15: 0000
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffff881147bfdcb8 ffffd80 00007fff56156db8 ffffde0
ffff ffffab0 ffff881147bfdc88 ffffffff81033e29
ffff881147bfdcb8 ffffffff81033f2a ffffdf8 0001
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: c3 0f 1f 84 00 00 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 &8b& 35 00 4f 9b 00 4c 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] BUG: soft lockup - CPU#14 stuck for 23s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff&]
[&ffffffff&] __bitmap_empty+0x13/0x90
[ ] RSP: 0000:ffff881147bfdc38
[ ] RAX: 004f RBX: ffffffff81460ab4 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc78 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffff881147bfdba8
[ ] R13: ffffc80 R14: ffff881147bfc000 R15: 0000
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffff881147bfdc78 ffffffff81033df0 ffff881147bfdcb8 ffffd80
00007fff56156db8 ffffde0 ffff ffffab0
ffff881147bfdc88 ffffffff81033e29 ffff881147bfdcb8 ffffffff81033f2a
[ ] Call Trace:
[&ffffffff81033df0&] ? flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: c7 45 b0 10 00 00 00 48 89 45 c0 e8 38 ff ff ff c9 c3 90 90 90 90 90 90 8d 46 3f 85 f6 41 89 f0 55 44 0f 48 c0 31 d2 41 c1 f8 06 &48& 89 e5 45 85 c0 7e 2a 31 d2 48 83 3f 00 48 89 f9 74 17 eb 60
[ ] Call Trace:
[&ffffffff81033df0&] ? flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff81033de2&]
[&ffffffff81033de2&] flush_tlb_others_ipi+0x102/0x140
[ ] RSP: 0000:ffff881147bfdc48
[ ] RAX: 0000 RBX: ffffffff81460ab4 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc78 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffff881147bfdbb8
[ ] R13: ffffc80 R14: ffff881147bfc000 R15: 0000
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffff881147bfdcb8 ffffd80 00007fff56156db8 ffffde0
ffff ffffab0 ffff881147bfdc88 ffffffff81033e29
ffff881147bfdcb8 ffffffff81033f2a ffffdf8 0001
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: c3 0f 1f 84 00 00 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 &8b& 35 00 4f 9b 00 4c 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff81033de2&]
[&ffffffff81033de2&] flush_tlb_others_ipi+0x102/0x140
[ ] RSP: 0000:ffff881147bfdc48
[ ] RAX: 0000 RBX: ffffffff81460ab4 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc78 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffff881147bfdbb8
[ ] R13: ffffc80 R14: ffff881147bfc000 R15: 0000
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffff881147bfdcb8 ffffd80 00007fff56156db8 ffffde0
ffff ffffab0 ffff881147bfdc88 ffffffff81033e29
ffff881147bfdcb8 ffffffff81033f2a ffffdf8 0001
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: c3 0f 1f 84 00 00 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 &8b& 35 00 4f 9b 00 4c 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff&]
[&ffffffff&] __bitmap_empty+0x77/0x90
[ ] RSP: 0000:ffff881147bfdc38
[ ] RAX: ffff RBX: ffff881147bfdbb8 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc38 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffff881147bfc000
[ ] R13: 01 R15: 0000
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffff881147bfdc78 ffffffff81033df0 ffff881147bfdcb8 ffffd80
00007fff56156db8 ffffde0 ffff ffffab0
ffff881147bfdc88 ffffffff81033e29 ffff881147bfdcb8 ffffffff81033f2a
[ ] Call Trace:
[&ffffffff81033df0&] flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: 00 00 75 08 c9 c3 66 0f 1f 44 00 00 89 f0 48 63 d2 c1 f8 1f c1 e8 1a 8d 0c 06 83 e1 3f 29 c1 b8 01 00 00 00 48 d3 e0 48 83 e8 01 &48& 85 04 d7 c9 0f 94 c0 0f b6 c0 c3 0f 1f 44 00 00 31 c0 c9 c3
[ ] Call Trace:
[&ffffffff81033df0&] flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff81033de8&]
[&ffffffff81033de8&] flush_tlb_others_ipi+0x108/0x140
[ ] RSP: 0000:ffff881147bfdc48
[ ] RAX: 0000 RBX: ffffffff81460ab4 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc78 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffff881147bfdbb8
[ ] R13: ffffc80 R14: ffff881147bfc000 R15: 0000
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffff881147bfdcb8 ffffd80 00007fff56156db8 ffffde0
ffff ffffab0 ffff881147bfdc88 ffffffff81033e29
ffff881147bfdcb8 ffffffff81033f2a ffffdf8 0001
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 8b 35 00 4f 9b 00 &4c& 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66 2e 0f 1f 84 00 00
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff&]
[&ffffffff&] __bitmap_empty+0x82/0x90
[ ] RSP: 0000:ffff881147bfdc40
[ ] RAX: 0000 RBX: ffffffff81460ab4 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc78 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffff881147bfdbb8
[ ] R13: ffffc80 R14: ffff881147bfc000 R15: 0000
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffffffff81033df0 ffff881147bfdcb8 ffffd80 00007fff56156db8
ffffde0 ffff ffffab0 ffff881147bfdc88
ffffffff81033e29 ffff881147bfdcb8 ffffffff81033f2a ffffdf8
[ ] Call Trace:
[&ffffffff81033df0&] ? flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: 00 89 f0 48 63 d2 c1 f8 1f c1 e8 1a 8d 0c 06 83 e1 3f 29 c1 b8 01 00 00 00 48 d3 e0 48 83 e8 01 48 85 04 d7 c9 0f 94 c0 0f b6 c0 &c3& 0f 1f 44 00 00 31 c0 c9 c3 0f 1f 40 00 8d 46 3f 85 f6 41 89
[ ] Call Trace:
[&ffffffff81033df0&] ? flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] INFO: rcu_sched detected stall on CPU 14 (t=65032 jiffies)
[ ] sending NMI to all CPUs:
[ ] INFO: rcu_sched detected stalls on CPUs/tasks: { 14} (detected by 0, t=65034 jiffies)
[ ] NMI backtrace for cpu 0
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 9736, comm: make Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff810ffe52&]
[&ffffffff810ffe52&] generic_file_aio_read+0x62/0x280
[ ] RSP: 0018:ffff
[ ] RAX: ffff RBX: ffff RCX: ffffab0
[ ] RDX: ffffac0 RSI: ffffab0 RDI: ffffaa8
[ ] RBP: ffff R08: 0040 R09: ffffb8
[ ] R10: 02 R12: 0001
[ ] R13: 01f8 R14: ffffc0 R15: ffffad8
bbe700(0000) GS:ffff8808ffc) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 0033
[ ] CR2: 0280 CR3: 3000 CR4: 06f0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process make (pid: 9736, threadinfo ffff, task ffff)
[ ] Stack:
ffff ffff ffffc0 ffffca0
ffff ffff ffffffff81160f79 ffffb8
ffff 0000 ffffffff ffffc0
[ ] Call Trace:
[&ffffffff81160f79&] do_sync_read+0xd9/0x120
[&ffffffff81221b1d&] ? common_file_perm+0x8d/0x110
[&ffffffff811fc373&] ? security_file_permission+0x93/0xa0
[&ffffffff&] vfs_read+0xc8/0x130
[&ffffffff&] kernel_read+0x44/0x60
[&ffffffff811b5aa3&] load_elf_binary+0x173/0x1060
[&ffffffff&] ? __lock_acquire+0x301/0x520
[&ffffffff811695cc&] ? search_binary_handler+0xfc/0x360
[&ffffffff811b5930&] ? load_elf_interp+0x5e0/0x5e0
[&ffffffff811b5930&] ? load_elf_interp+0x5e0/0x5e0
[&ffffffff&] search_binary_handler+0x106/0x360
[&ffffffff8116951e&] ? search_binary_handler+0x4e/0x360
[&ffffffff81169d2d&] do_execve_common+0x27d/0x320
[&ffffffff81169e5a&] do_execve+0x3a/0x40
[&ffffffff8100aac9&] sys_execve+0x49/0x70
[&ffffffff8146972c&] stub_execve+0x6c/0xc0
[ ] Code: c8 4c 8b 77 20 4c 89 e7 48 89 85 58 ff ff ff 48 c7 45 c8 00 00 00 00 e8 8d d8 ff ff 4c 63 e8 4d 85 ed 74 15 48 81 c4 88 00 00 00 &4c& 89 e8 5b 41 5c 41 5d 41 5e 41 5f c9 c3 48 8d bd 70 ff ff ff
[ ] Call Trace:
[&ffffffff81160f79&] do_sync_read+0xd9/0x120
[&ffffffff81221b1d&] ? common_file_perm+0x8d/0x110
[&ffffffff811fc373&] ? security_file_permission+0x93/0xa0
[&ffffffff&] vfs_read+0xc8/0x130
[&ffffffff&] kernel_read+0x44/0x60
[&ffffffff811b5aa3&] load_elf_binary+0x173/0x1060
[&ffffffff&] ? __lock_acquire+0x301/0x520
[&ffffffff811695cc&] ? search_binary_handler+0xfc/0x360
[&ffffffff811b5930&] ? load_elf_interp+0x5e0/0x5e0
[&ffffffff811b5930&] ? load_elf_interp+0x5e0/0x5e0
[&ffffffff&] search_binary_handler+0x106/0x360
[&ffffffff8116951e&] ? search_binary_handler+0x4e/0x360
[&ffffffff81169d2d&] do_execve_common+0x27d/0x320
[&ffffffff81169e5a&] do_execve+0x3a/0x40
[&ffffffff8100aac9&] sys_execve+0x49/0x70
[&ffffffff8146972c&] stub_execve+0x6c/0xc0
[ ] NMI backtrace for cpu 14
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff81258d71&]
[&ffffffff81258d71&] delay_tsc+0x51/0xa0
[ ] RSP: 0000:ffff8
[ ] RAX: 0042 RBX: 70af RCX: ffffffffd032bab4
[ ] RDX: bab4 RSI: 000f RDI: 70af
[ ] RBP: ffff8 R08: 0010 R09: ffffffff819e7660
[ ] R10: 03 R12: 1000
[ ] R13: 0e R15: 000e
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffffffffd032ba72
00000cbc0 0002 ffff8
ffffffff81258e1f ffff8 ffffffff81021aaa 000f
[ ] Call Trace:
[&ffffffff81258e1f&] __const_udelay+0x2f/0x40
[&ffffffff81021aaa&] native_safe_apic_wait_icr_idle+0x1a/0x50
[&ffffffff81021fbd&] default_send_IPI_mask_sequence_phys+0xdd/0x130
[&ffffffff&] physflat_send_IPI_all+0x14/0x20
[&ffffffff&] arch_trigger_all_cpu_backtrace+0x67/0xb0
[&ffffffff810cfca9&] __rcu_pending+0x119/0x280
[&ffffffff810cfeba&] rcu_check_callbacks+0xaa/0x1b0
[&ffffffff&] update_process_times+0x41/0x80
[&ffffffff8108b05f&] tick_sched_timer+0x5f/0xc0
[&ffffffff&] ? tick_nohz_handler+0x100/0x100
[&ffffffff&] __run_hrtimer+0xd1/0x1d0
[&ffffffff8107dc97&] hrtimer_interrupt+0xc7/0x1f0
[&ffffffff81021a34&] smp_apic_timer_interrupt+0x64/0xa0
[&ffffffff81469db3&] apic_timer_interrupt+0x73/0x80
[&ffffffff8125b8eb&] ? __bitmap_empty+0x7b/0x90
[&ffffffff81033df0&] flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: db ff 66 90 48 98 44 89 75 d4 48 89 45 c8 eb 1b 66 2e 0f 1f 84 00 00 00 00 00 f3 90 65 44 8b 3c 25 b0 cb 00 00 44 3b 7d d4 75 2b &66& 66 90 0f ae e8 e8 b4 15 db ff 66 90 48 63 c8 48 89 c8 48 2b
[ ] Call Trace:
[&ffffffff81258e1f&] __const_udelay+0x2f/0x40
[&ffffffff81021aaa&] native_safe_apic_wait_icr_idle+0x1a/0x50
[&ffffffff81021fbd&] default_send_IPI_mask_sequence_phys+0xdd/0x130
[&ffffffff&] physflat_send_IPI_all+0x14/0x20
[&ffffffff&] arch_trigger_all_cpu_backtrace+0x67/0xb0
[&ffffffff810cfca9&] __rcu_pending+0x119/0x280
[&ffffffff810cfeba&] rcu_check_callbacks+0xaa/0x1b0
[&ffffffff&] update_process_times+0x41/0x80
[&ffffffff8108b05f&] tick_sched_timer+0x5f/0xc0
[&ffffffff&] ? tick_nohz_handler+0x100/0x100
[&ffffffff&] __run_hrtimer+0xd1/0x1d0
[&ffffffff8107dc97&] hrtimer_interrupt+0xc7/0x1f0
[&ffffffff81021a34&] smp_apic_timer_interrupt+0x64/0xa0
[&ffffffff81469db3&] apic_timer_interrupt+0x73/0x80
[&ffffffff8125b8eb&] ? __bitmap_empty+0x7b/0x90
[&ffffffff81033df0&] flush_tlb_others_ipi+0x110/0x140
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] NMI backtrace for cpu 15
[ ] CPU 15
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 9869, comm: sh Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff&]
[&ffffffff&] restore+0xe/0x3a
[ ] RSP: 0000:ffff8810760aff00
[ ] RAX: 0dbe RBX: 0000 RCX: 0000
[ ] RDX: ffff8810760dcbc0 RSI: cd50 RDI: ffffffff81460f06
[ ] RBP: 00007fff2d19ee70 R08: 05
[ ] R10: 8df0 R11: ffffffffffffffff R12: cd50
[ ] R13: cd50 R14: ce00
fc1ba0(0000) GS:ffff88117fdc) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 0033
[ ] CR2: c1e0 CR3: d000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process sh (pid: 9869, threadinfo ffff0, task ffff8810760dcbc0)
[ ] Stack:
ffffffffffffffff 8df0
c1e0 ffffffff81460f06 ffffffff81460caf ce00
[ ] Call Trace:
[&ffffffff81460f06&] ? error_sti+0x5/0x6
[&ffffffff81460caf&] ? page_fault+0xf/0x30
[ ] Code: 48 89 4c 24 28 48 89 44 24 20 4c 89 44 24 18 4c 89 4c 24 10 4c 89 54 24 08 4c 89 1c 24 4c 8b 1c 24 4c 8b 54 24 08 4c 8b 4c 24 10 &4c& 8b 44 24 18 48 8b 44 24 20 48 8b 4c 24 28 48 8b 54 24 30 48
[ ] Call Trace:
[&ffffffff81460f06&] ? error_sti+0x5/0x6
[&ffffffff81460caf&] ? page_fault+0xf/0x30
[ ] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068]
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] irq event stamp:
[ ] hardirqs last
enabled at (): [&ffffffff81460ab4&] restore_args+0x0/0x30
[ ] hardirqs last disabled at (): [&ffffffff81469dae&] apic_timer_interrupt+0x6e/0x80
[ ] softirqs last
enabled at (): [&ffffffff8105bc31&] __do_softirq+0x1a1/0x200
[ ] softirqs last disabled at (): [&ffffffff8146b53c&] call_softirq+0x1c/0x30
[ ] CPU 14
[ ] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
[ ] Pid: 1068, comm: udevd Tainted: G
3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033
[ ] RIP: 0010:[&ffffffff81033de2&]
[&ffffffff81033de2&] flush_tlb_others_ipi+0x102/0x140
[ ] RSP: 0000:ffff881147bfdc48
[ ] RAX: 0000 RBX: 0002 RCX: 0010
[ ] RDX: 0000 RSI: 0010 RDI: ffffffff81bbcfc8
[ ] RBP: ffff881147bfdc78 R08: 00
[ ] R10: 0002 R11: ffff0 R12: ffffffff81460ab4
[ ] R13: 000e R14: ffff881147bfdba8 R15: ffffc80
00007fbaa8d) GS:ffff8(0000) knlGS:0000
0010 DS: 0000 ES: 0000 CR0: 003b
[ ] CR2: 00007fff56156db8 CR3: a0000 CR4: 06e0
[ ] DR0: 0000 DR1: 0000 DR2: 0000
[ ] DR3: 0000 DR6: ffff0ff0 DR7: 0400
[ ] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffffc0)
[ ] Stack:
ffff881147bfdcb8 ffffd80 00007fff56156db8 ffffde0
ffff ffffab0 ffff881147bfdc88 ffffffff81033e29
ffff881147bfdcb8 ffffffff81033f2a ffffdf8 0001
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] Code: c3 0f 1f 84 00 00 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 &8b& 35 00 4f 9b 00 4c 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66
[ ] Call Trace:
[&ffffffff81033e29&] native_flush_tlb_others+0x9/0x10
[&ffffffff81033f2a&] flush_tlb_page+0x5a/0xa0
[&ffffffff81032a4d&] ptep_set_access_flags+0x4d/0x70
[&ffffffff&] do_wp_page+0x469/0x7e0
[&ffffffff81127acd&] handle_pte_fault+0x19d/0x1e0
[&ffffffff81127c88&] handle_mm_fault+0x178/0x2e0
[&ffffffff&] do_page_fault+0x1e5/0x490
[&ffffffff8125a17d&] ? trace_hardirqs_off_thunk+0x3a/0x3c
[&ffffffff81460cc5&] page_fault+0x25/0x30
[ ] CPU 14 MCA banks CMCI:6 CMCI:8
[ ] Broke affinity for irq 74
[ ] Broke affinity for irq 80
[ ] CPU 14 is now offline
[ ] CPU 15 MCA banks CMCI:6 CMCI:8
[ ] Broke affinity for irq 76
[ ] CPU 15 is now offline
[ ] lockdep: fixing up alternatives.
[ ] SMP alternatives: switching to UP code
[ ] lockdep: fixing up alternatives.
[ ] SMP alternatives: switching to SMP code
[ ] Booting Node 0 Processor 1 APIC 0x2
[ ] smpboot cpu 1: start_ip = 97000
[ ] Calibrating delay loop (skipped) already calibrated this CPU
[ ] NMI watchdog enabled, takes one hw-pmu counter.
[ ] lockdep: fixing up alternatives.
[ ] Booting Node 0 Processor 2 APIC 0x4
[ ] smpboot cpu 2: start_ip = 97000
[ ] Calibrating delay loop (skipped) already calibrated this CPU
[ ] NMI watchdog enabled, takes one hw-pmu counter.
[ ] lockdep: fixing up alternatives.
[ ] Booting Node 0 Processor 3 APIC 0x6
[ ] smpboot cpu 3: start_ip = 97000
[ ] Calibrating delay loop (skipped) already calibrated this CPU
[ ] NMI watchdog enabled, takes one hw-pmu counter.
[ ] lockdep: fixing up alternatives.
Srivatsa S. Bhat
To unsubscribe from this list: send the line &unsubscribe linux-fsdevel& in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Follow-Ups:
From: Mel Gorman
References:
From: Gilad Ben-Yossef
From: KOSAKI Motohiro
From: Mel Gorman
From: Russell King - ARM Linux
From: Mel Gorman
From: Russell King - ARM Linux
From: Paul E. McKenney
From: Mel Gorman
Prev by Date:
Next by Date:
Previous by thread:
Next by thread:
Index(es):