1 =========================
2 CPU hotplug in the Kernel
3 =========================
6 :Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
7 Rusty Russell <rusty@rustcorp.com.au>,
8 Srivatsa Vaddagiri <vatsa@in.ibm.com>,
9 Ashok Raj <ashok.raj@intel.com>,
10 Joel Schopp <jschopp@austin.ibm.com>,
11 Thomas Gleixner <tglx@linutronix.de>
16 Modern advances in system architectures have introduced advanced error
17 reporting and correction capabilities in processors. There are couple OEMS that
18 support NUMA hardware which are hot pluggable as well, where physical node
19 insertion and removal require support for CPU hotplug.
21 Such advances require CPUs available to a kernel to be removed either for
22 provisioning reasons, or for RAS purposes to keep an offending CPU off
23 system execution path. Hence the need for CPU hotplug support in the
26 A more novel use of CPU-hotplug support is its use today in suspend resume
27 support for SMP. Dual-core and HT support makes even a laptop run SMP kernels
28 which didn't support these methods.
34 Restrict boot time CPUs to *n*. Say if you have four CPUs, using
35 ``maxcpus=2`` will only boot two. You can choose to bring the
36 other CPUs later online.
39 Restrict the total amount of CPUs the kernel will support. If the number
40 supplied here is lower than the number of physically available CPUs, then
41 those CPUs can not be brought online later.
44 Use this to limit hotpluggable CPUs. This option sets
45 ``cpu_possible_mask = cpu_present_mask + additional_cpus``
47 This option is limited to the IA64 architecture.
50 This option sets ``possible_cpus`` bits in ``cpu_possible_mask``.
52 This option is limited to the X86 and S390 architecture.
55 Allow to shutdown CPU0.
57 This option is limited to the X86 architecture.
63 Bitmap of possible CPUs that can ever be available in the
64 system. This is used to allocate some boot time memory for per_cpu variables
65 that aren't designed to grow/shrink as CPUs are made available or removed.
66 Once set during boot time discovery phase, the map is static, i.e no bits
67 are added or removed anytime. Trimming it accurately for your system needs
68 upfront can save some boot time memory.
71 Bitmap of all CPUs currently online. Its set in ``__cpu_up()``
72 after a CPU is available for kernel scheduling and ready to receive
73 interrupts from devices. Its cleared when a CPU is brought down using
74 ``__cpu_disable()``, before which all OS services including interrupts are
75 migrated to another target CPU.
78 Bitmap of CPUs currently present in the system. Not all
79 of them may be online. When physical hotplug is processed by the relevant
80 subsystem (e.g ACPI) can change and new bit either be added or removed
81 from the map depending on the event is hot-add/hot-remove. There are currently
82 no locking rules as of now. Typical usage is to init topology during boot,
83 at which time hotplug is disabled.
85 You really don't need to manipulate any of the system CPU maps. They should
86 be read-only for most use. When setting up per-cpu resources almost always use
87 ``cpu_possible_mask`` or ``for_each_possible_cpu()`` to iterate. To macro
88 ``for_each_cpu()`` can be used to iterate over a custom CPU mask.
90 Never use anything other than ``cpumask_t`` to represent bitmap of CPUs.
96 The kernel option *CONFIG_HOTPLUG_CPU* needs to be enabled. It is currently
97 available on multiple architectures including ARM, MIPS, PowerPC and X86. The
98 configuration is done via the sysfs interface::
100 $ ls -lh /sys/devices/system/cpu
102 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu0
103 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu1
104 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu2
105 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu3
106 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu4
107 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu5
108 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu6
109 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu7
110 drwxr-xr-x 2 root root 0 Dec 21 16:33 hotplug
111 -r--r--r-- 1 root root 4.0K Dec 21 16:33 offline
112 -r--r--r-- 1 root root 4.0K Dec 21 16:33 online
113 -r--r--r-- 1 root root 4.0K Dec 21 16:33 possible
114 -r--r--r-- 1 root root 4.0K Dec 21 16:33 present
116 The files *offline*, *online*, *possible*, *present* represent the CPU masks.
117 Each CPU folder contains an *online* file which controls the logical on (1) and
118 off (0) state. To logically shutdown CPU4::
120 $ echo 0 > /sys/devices/system/cpu/cpu4/online
121 smpboot: CPU 4 is now offline
123 Once the CPU is shutdown, it will be removed from */proc/interrupts*,
124 */proc/cpuinfo* and should also not be shown visible by the *top* command. To
125 bring CPU4 back online::
127 $ echo 1 > /sys/devices/system/cpu/cpu4/online
128 smpboot: Booting Node 0 Processor 4 APIC 0x1
130 The CPU is usable again. This should work on all CPUs, but CPU0 is often special
131 and excluded from CPU hotplug.
133 The CPU hotplug coordination
134 ============================
139 Once a CPU has been logically shutdown the teardown callbacks of registered
140 hotplug states will be invoked, starting with ``CPUHP_ONLINE`` and terminating
141 at state ``CPUHP_OFFLINE``. This includes:
143 * If tasks are frozen due to a suspend operation then *cpuhp_tasks_frozen*
145 * All processes are migrated away from this outgoing CPU to new CPUs.
146 The new CPU is chosen from each process' current cpuset, which may be
147 a subset of all online CPUs.
148 * All interrupts targeted to this CPU are migrated to a new CPU
149 * timers are also migrated to a new CPU
150 * Once all services are migrated, kernel calls an arch specific routine
151 ``__cpu_disable()`` to perform arch specific cleanup.
157 CPU hotplug state machine
158 -------------------------
160 CPU hotplug uses a trivial state machine with a linear state space from
161 CPUHP_OFFLINE to CPUHP_ONLINE. Each state has a startup and a teardown
164 When a CPU is onlined, the startup callbacks are invoked sequentially until
165 the state CPUHP_ONLINE is reached. They can also be invoked when the
166 callbacks of a state are set up or an instance is added to a multi-instance
169 When a CPU is offlined the teardown callbacks are invoked in the reverse
170 order sequentially until the state CPUHP_OFFLINE is reached. They can also
171 be invoked when the callbacks of a state are removed or an instance is
172 removed from a multi-instance state.
174 If a usage site requires only a callback in one direction of the hotplug
175 operations (CPU online or CPU offline) then the other not-required callback
176 can be set to NULL when the state is set up.
178 The state space is divided into three sections:
180 * The PREPARE section
182 The PREPARE section covers the state space from CPUHP_OFFLINE to
185 The startup callbacks in this section are invoked before the CPU is
186 started during a CPU online operation. The teardown callbacks are invoked
187 after the CPU has become dysfunctional during a CPU offline operation.
189 The callbacks are invoked on a control CPU as they can't obviously run on
190 the hotplugged CPU which is either not yet started or has become
191 dysfunctional already.
193 The startup callbacks are used to setup resources which are required to
194 bring a CPU successfully online. The teardown callbacks are used to free
195 resources or to move pending work to an online CPU after the hotplugged
196 CPU became dysfunctional.
198 The startup callbacks are allowed to fail. If a callback fails, the CPU
199 online operation is aborted and the CPU is brought down to the previous
200 state (usually CPUHP_OFFLINE) again.
202 The teardown callbacks in this section are not allowed to fail.
204 * The STARTING section
206 The STARTING section covers the state space between CPUHP_BRINGUP_CPU + 1
209 The startup callbacks in this section are invoked on the hotplugged CPU
210 with interrupts disabled during a CPU online operation in the early CPU
211 setup code. The teardown callbacks are invoked with interrupts disabled
212 on the hotplugged CPU during a CPU offline operation shortly before the
213 CPU is completely shut down.
215 The callbacks in this section are not allowed to fail.
217 The callbacks are used for low level hardware initialization/shutdown and
222 The ONLINE section covers the state space between CPUHP_AP_ONLINE + 1 and
225 The startup callbacks in this section are invoked on the hotplugged CPU
226 during a CPU online operation. The teardown callbacks are invoked on the
227 hotplugged CPU during a CPU offline operation.
229 The callbacks are invoked in the context of the per CPU hotplug thread,
230 which is pinned on the hotplugged CPU. The callbacks are invoked with
231 interrupts and preemption enabled.
233 The callbacks are allowed to fail. When a callback fails the hotplug
234 operation is aborted and the CPU is brought back to the previous state.
236 CPU online/offline operations
237 -----------------------------
239 A successful online operation looks like this::
242 [CPUHP_OFFLINE + 1]->startup() -> success
243 [CPUHP_OFFLINE + 2]->startup() -> success
244 [CPUHP_OFFLINE + 3] -> skipped because startup == NULL
246 [CPUHP_BRINGUP_CPU]->startup() -> success
247 === End of PREPARE section
248 [CPUHP_BRINGUP_CPU + 1]->startup() -> success
250 [CPUHP_AP_ONLINE]->startup() -> success
251 === End of STARTUP section
252 [CPUHP_AP_ONLINE + 1]->startup() -> success
254 [CPUHP_ONLINE - 1]->startup() -> success
257 A successful offline operation looks like this::
260 [CPUHP_ONLINE - 1]->teardown() -> success
262 [CPUHP_AP_ONLINE + 1]->teardown() -> success
263 === Start of STARTUP section
264 [CPUHP_AP_ONLINE]->teardown() -> success
266 [CPUHP_BRINGUP_ONLINE - 1]->teardown()
268 === Start of PREPARE section
269 [CPUHP_BRINGUP_CPU]->teardown()
270 [CPUHP_OFFLINE + 3]->teardown()
271 [CPUHP_OFFLINE + 2] -> skipped because teardown == NULL
272 [CPUHP_OFFLINE + 1]->teardown()
275 A failed online operation looks like this::
278 [CPUHP_OFFLINE + 1]->startup() -> success
279 [CPUHP_OFFLINE + 2]->startup() -> success
280 [CPUHP_OFFLINE + 3] -> skipped because startup == NULL
282 [CPUHP_BRINGUP_CPU]->startup() -> success
283 === End of PREPARE section
284 [CPUHP_BRINGUP_CPU + 1]->startup() -> success
286 [CPUHP_AP_ONLINE]->startup() -> success
287 === End of STARTUP section
288 [CPUHP_AP_ONLINE + 1]->startup() -> success
290 [CPUHP_AP_ONLINE + N]->startup() -> fail
291 [CPUHP_AP_ONLINE + (N - 1)]->teardown()
293 [CPUHP_AP_ONLINE + 1]->teardown()
294 === Start of STARTUP section
295 [CPUHP_AP_ONLINE]->teardown()
297 [CPUHP_BRINGUP_ONLINE - 1]->teardown()
299 === Start of PREPARE section
300 [CPUHP_BRINGUP_CPU]->teardown()
301 [CPUHP_OFFLINE + 3]->teardown()
302 [CPUHP_OFFLINE + 2] -> skipped because teardown == NULL
303 [CPUHP_OFFLINE + 1]->teardown()
306 A failed offline operation looks like this::
309 [CPUHP_ONLINE - 1]->teardown() -> success
311 [CPUHP_ONLINE - N]->teardown() -> fail
312 [CPUHP_ONLINE - (N - 1)]->startup()
314 [CPUHP_ONLINE - 1]->startup()
317 Recursive failures cannot be handled sensibly. Look at the following
318 example of a recursive fail due to a failed offline operation: ::
321 [CPUHP_ONLINE - 1]->teardown() -> success
323 [CPUHP_ONLINE - N]->teardown() -> fail
324 [CPUHP_ONLINE - (N - 1)]->startup() -> success
325 [CPUHP_ONLINE - (N - 2)]->startup() -> fail
327 The CPU hotplug state machine stops right here and does not try to go back
328 down again because that would likely result in an endless loop::
330 [CPUHP_ONLINE - (N - 1)]->teardown() -> success
331 [CPUHP_ONLINE - N]->teardown() -> fail
332 [CPUHP_ONLINE - (N - 1)]->startup() -> success
333 [CPUHP_ONLINE - (N - 2)]->startup() -> fail
334 [CPUHP_ONLINE - (N - 1)]->teardown() -> success
335 [CPUHP_ONLINE - N]->teardown() -> fail
337 Lather, rinse and repeat. In this case the CPU left in state::
339 [CPUHP_ONLINE - (N - 1)]
341 which at least lets the system make progress and gives the user a chance to
342 debug or even resolve the situation.
347 There are two ways to allocate a CPU hotplug state:
351 Static allocation has to be used when the subsystem or driver has
352 ordering requirements versus other CPU hotplug states. E.g. the PERF core
353 startup callback has to be invoked before the PERF driver startup
354 callbacks during a CPU online operation. During a CPU offline operation
355 the driver teardown callbacks have to be invoked before the core teardown
356 callback. The statically allocated states are described by constants in
357 the cpuhp_state enum which can be found in include/linux/cpuhotplug.h.
359 Insert the state into the enum at the proper place so the ordering
360 requirements are fulfilled. The state constant has to be used for state
363 Static allocation is also required when the state callbacks are not set
364 up at runtime and are part of the initializer of the CPU hotplug state
365 array in kernel/cpu.c.
369 When there are no ordering requirements for the state callbacks then
370 dynamic allocation is the preferred method. The state number is allocated
371 by the setup function and returned to the caller on success.
373 Only the PREPARE and ONLINE sections provide a dynamic allocation
374 range. The STARTING section does not as most of the callbacks in that
375 section have explicit ordering requirements.
377 Setup of a CPU hotplug state
378 ----------------------------
380 The core code provides the following functions to setup a state:
382 * cpuhp_setup_state(state, name, startup, teardown)
383 * cpuhp_setup_state_nocalls(state, name, startup, teardown)
384 * cpuhp_setup_state_cpuslocked(state, name, startup, teardown)
385 * cpuhp_setup_state_nocalls_cpuslocked(state, name, startup, teardown)
387 For cases where a driver or a subsystem has multiple instances and the same
388 CPU hotplug state callbacks need to be invoked for each instance, the CPU
389 hotplug core provides multi-instance support. The advantage over driver
390 specific instance lists is that the instance related functions are fully
391 serialized against CPU hotplug operations and provide the automatic
392 invocations of the state callbacks on add and removal. To set up such a
393 multi-instance state the following function is available:
395 * cpuhp_setup_state_multi(state, name, startup, teardown)
397 The @state argument is either a statically allocated state or one of the
398 constants for dynamically allocated states - CPUHP_BP_PREPARE_DYN,
399 CPUHP_AP_ONLINE_DYN - depending on the state section (PREPARE, ONLINE) for
400 which a dynamic state should be allocated.
402 The @name argument is used for sysfs output and for instrumentation. The
403 naming convention is "subsys:mode" or "subsys/driver:mode",
404 e.g. "perf:mode" or "perf/x86:mode". The common mode names are:
406 ======== =======================================================
407 prepare For states in the PREPARE section
409 dead For states in the PREPARE section which do not provide
412 starting For states in the STARTING section
414 dying For states in the STARTING section which do not provide
417 online For states in the ONLINE section
419 offline For states in the ONLINE section which do not provide
421 ======== =======================================================
423 As the @name argument is only used for sysfs and instrumentation other mode
424 descriptors can be used as well if they describe the nature of the state
425 better than the common ones.
427 Examples for @name arguments: "perf/online", "perf/x86:prepare",
428 "RCU/tree:dying", "sched/waitempty"
430 The @startup argument is a function pointer to the callback which should be
431 invoked during a CPU online operation. If the usage site does not require a
432 startup callback set the pointer to NULL.
434 The @teardown argument is a function pointer to the callback which should
435 be invoked during a CPU offline operation. If the usage site does not
436 require a teardown callback set the pointer to NULL.
438 The functions differ in the way how the installed callbacks are treated:
440 * cpuhp_setup_state_nocalls(), cpuhp_setup_state_nocalls_cpuslocked()
441 and cpuhp_setup_state_multi() only install the callbacks
443 * cpuhp_setup_state() and cpuhp_setup_state_cpuslocked() install the
444 callbacks and invoke the @startup callback (if not NULL) for all online
445 CPUs which have currently a state greater than the newly installed
446 state. Depending on the state section the callback is either invoked on
447 the current CPU (PREPARE section) or on each online CPU (ONLINE
448 section) in the context of the CPU's hotplug thread.
450 If a callback fails for CPU N then the teardown callback for CPU
451 0 .. N-1 is invoked to rollback the operation. The state setup fails,
452 the callbacks for the state are not installed and in case of dynamic
453 allocation the allocated state is freed.
455 The state setup and the callback invocations are serialized against CPU
456 hotplug operations. If the setup function has to be called from a CPU
457 hotplug read locked region, then the _cpuslocked() variants have to be
458 used. These functions cannot be used from within CPU hotplug callbacks.
460 The function return values:
461 ======== ===================================================================
462 0 Statically allocated state was successfully set up
464 >0 Dynamically allocated state was successfully set up.
466 The returned number is the state number which was allocated. If
467 the state callbacks have to be removed later, e.g. module
468 removal, then this number has to be saved by the caller and used
469 as @state argument for the state remove function. For
470 multi-instance states the dynamically allocated state number is
471 also required as @state argument for the instance add/remove
475 ======== ===================================================================
477 Removal of a CPU hotplug state
478 ------------------------------
480 To remove a previously set up state, the following functions are provided:
482 * cpuhp_remove_state(state)
483 * cpuhp_remove_state_nocalls(state)
484 * cpuhp_remove_state_nocalls_cpuslocked(state)
485 * cpuhp_remove_multi_state(state)
487 The @state argument is either a statically allocated state or the state
488 number which was allocated in the dynamic range by cpuhp_setup_state*(). If
489 the state is in the dynamic range, then the state number is freed and
490 available for dynamic allocation again.
492 The functions differ in the way how the installed callbacks are treated:
494 * cpuhp_remove_state_nocalls(), cpuhp_remove_state_nocalls_cpuslocked()
495 and cpuhp_remove_multi_state() only remove the callbacks.
497 * cpuhp_remove_state() removes the callbacks and invokes the teardown
498 callback (if not NULL) for all online CPUs which have currently a state
499 greater than the removed state. Depending on the state section the
500 callback is either invoked on the current CPU (PREPARE section) or on
501 each online CPU (ONLINE section) in the context of the CPU's hotplug
504 In order to complete the removal, the teardown callback should not fail.
506 The state removal and the callback invocations are serialized against CPU
507 hotplug operations. If the remove function has to be called from a CPU
508 hotplug read locked region, then the _cpuslocked() variants have to be
509 used. These functions cannot be used from within CPU hotplug callbacks.
511 If a multi-instance state is removed then the caller has to remove all
514 Multi-Instance state instance management
515 ----------------------------------------
517 Once the multi-instance state is set up, instances can be added to the
520 * cpuhp_state_add_instance(state, node)
521 * cpuhp_state_add_instance_nocalls(state, node)
523 The @state argument is either a statically allocated state or the state
524 number which was allocated in the dynamic range by cpuhp_setup_state_multi().
526 The @node argument is a pointer to an hlist_node which is embedded in the
527 instance's data structure. The pointer is handed to the multi-instance
528 state callbacks and can be used by the callback to retrieve the instance
531 The functions differ in the way how the installed callbacks are treated:
533 * cpuhp_state_add_instance_nocalls() and only adds the instance to the
534 multi-instance state's node list.
536 * cpuhp_state_add_instance() adds the instance and invokes the startup
537 callback (if not NULL) associated with @state for all online CPUs which
538 have currently a state greater than @state. The callback is only
539 invoked for the to be added instance. Depending on the state section
540 the callback is either invoked on the current CPU (PREPARE section) or
541 on each online CPU (ONLINE section) in the context of the CPU's hotplug
544 If a callback fails for CPU N then the teardown callback for CPU
545 0 .. N-1 is invoked to rollback the operation, the function fails and
546 the instance is not added to the node list of the multi-instance state.
548 To remove an instance from the state's node list these functions are
551 * cpuhp_state_remove_instance(state, node)
552 * cpuhp_state_remove_instance_nocalls(state, node)
554 The arguments are the same as for the cpuhp_state_add_instance*()
557 The functions differ in the way how the installed callbacks are treated:
559 * cpuhp_state_remove_instance_nocalls() only removes the instance from the
562 * cpuhp_state_remove_instance() removes the instance and invokes the
563 teardown callback (if not NULL) associated with @state for all online
564 CPUs which have currently a state greater than @state. The callback is
565 only invoked for the to be removed instance. Depending on the state
566 section the callback is either invoked on the current CPU (PREPARE
567 section) or on each online CPU (ONLINE section) in the context of the
568 CPU's hotplug thread.
570 In order to complete the removal, the teardown callback should not fail.
572 The node list add/remove operations and the callback invocations are
573 serialized against CPU hotplug operations. These functions cannot be used
574 from within CPU hotplug callbacks and CPU hotplug read locked regions.
579 Setup and teardown a statically allocated state in the STARTING section for
580 notifications on online and offline operations::
582 ret = cpuhp_setup_state(CPUHP_SUBSYS_STARTING, "subsys:starting", subsys_cpu_starting, subsys_cpu_dying);
586 cpuhp_remove_state(CPUHP_SUBSYS_STARTING);
588 Setup and teardown a dynamically allocated state in the ONLINE section
589 for notifications on offline operations::
591 state = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "subsys:offline", NULL, subsys_cpu_offline);
595 cpuhp_remove_state(state);
597 Setup and teardown a dynamically allocated state in the ONLINE section
598 for notifications on online operations without invoking the callbacks::
600 state = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "subsys:online", subsys_cpu_online, NULL);
604 cpuhp_remove_state_nocalls(state);
606 Setup, use and teardown a dynamically allocated multi-instance state in the
607 ONLINE section for notifications on online and offline operation::
609 state = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "subsys:online", subsys_cpu_online, subsys_cpu_offline);
613 ret = cpuhp_state_add_instance(state, &inst1->node);
617 ret = cpuhp_state_add_instance(state, &inst2->node);
621 cpuhp_remove_instance(state, &inst1->node);
623 cpuhp_remove_instance(state, &inst2->node);
625 remove_multi_state(state);
628 Testing of hotplug states
629 =========================
631 One way to verify whether a custom state is working as expected or not is to
632 shutdown a CPU and then put it online again. It is also possible to put the CPU
633 to certain state (for instance *CPUHP_AP_ONLINE*) and then go back to
634 *CPUHP_ONLINE*. This would simulate an error one state after *CPUHP_AP_ONLINE*
635 which would lead to rollback to the online state.
637 All registered states are enumerated in ``/sys/devices/system/cpu/hotplug/states`` ::
639 $ tail /sys/devices/system/cpu/hotplug/states
640 138: mm/vmscan:online
641 139: mm/vmstat:online
642 140: lib/percpu_cnt:online
643 141: acpi/cpu-drv:online
644 142: base/cacheinfo:online
645 143: virtio/net:online
651 To rollback CPU4 to ``lib/percpu_cnt:online`` and back online just issue::
653 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
655 $ echo 140 > /sys/devices/system/cpu/cpu4/hotplug/target
656 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
659 It is important to note that the teardown callback of state 140 have been
660 invoked. And now get back online::
662 $ echo 169 > /sys/devices/system/cpu/cpu4/hotplug/target
663 $ cat /sys/devices/system/cpu/cpu4/hotplug/state
666 With trace events enabled, the individual steps are visible, too::
668 # TASK-PID CPU# TIMESTAMP FUNCTION
670 bash-394 [001] 22.976: cpuhp_enter: cpu: 0004 target: 140 step: 169 (cpuhp_kick_ap_work)
671 cpuhp/4-31 [004] 22.977: cpuhp_enter: cpu: 0004 target: 140 step: 168 (sched_cpu_deactivate)
672 cpuhp/4-31 [004] 22.990: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0
673 cpuhp/4-31 [004] 22.991: cpuhp_enter: cpu: 0004 target: 140 step: 144 (mce_cpu_pre_down)
674 cpuhp/4-31 [004] 22.992: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0
675 cpuhp/4-31 [004] 22.993: cpuhp_multi_enter: cpu: 0004 target: 140 step: 143 (virtnet_cpu_down_prep)
676 cpuhp/4-31 [004] 22.994: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0
677 cpuhp/4-31 [004] 22.995: cpuhp_enter: cpu: 0004 target: 140 step: 142 (cacheinfo_cpu_pre_down)
678 cpuhp/4-31 [004] 22.996: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0
679 bash-394 [001] 22.997: cpuhp_exit: cpu: 0004 state: 140 step: 169 ret: 0
680 bash-394 [005] 95.540: cpuhp_enter: cpu: 0004 target: 169 step: 140 (cpuhp_kick_ap_work)
681 cpuhp/4-31 [004] 95.541: cpuhp_enter: cpu: 0004 target: 169 step: 141 (acpi_soft_cpu_online)
682 cpuhp/4-31 [004] 95.542: cpuhp_exit: cpu: 0004 state: 141 step: 141 ret: 0
683 cpuhp/4-31 [004] 95.543: cpuhp_enter: cpu: 0004 target: 169 step: 142 (cacheinfo_cpu_online)
684 cpuhp/4-31 [004] 95.544: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0
685 cpuhp/4-31 [004] 95.545: cpuhp_multi_enter: cpu: 0004 target: 169 step: 143 (virtnet_cpu_online)
686 cpuhp/4-31 [004] 95.546: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0
687 cpuhp/4-31 [004] 95.547: cpuhp_enter: cpu: 0004 target: 169 step: 144 (mce_cpu_online)
688 cpuhp/4-31 [004] 95.548: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0
689 cpuhp/4-31 [004] 95.549: cpuhp_enter: cpu: 0004 target: 169 step: 145 (console_cpu_notify)
690 cpuhp/4-31 [004] 95.550: cpuhp_exit: cpu: 0004 state: 145 step: 145 ret: 0
691 cpuhp/4-31 [004] 95.551: cpuhp_enter: cpu: 0004 target: 169 step: 168 (sched_cpu_activate)
692 cpuhp/4-31 [004] 95.552: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0
693 bash-394 [005] 95.553: cpuhp_exit: cpu: 0004 state: 169 step: 140 ret: 0
695 As it an be seen, CPU4 went down until timestamp 22.996 and then back up until
696 95.552. All invoked callbacks including their return codes are visible in the
699 Architecture's requirements
700 ===========================
702 The following functions and configurations are required:
704 ``CONFIG_HOTPLUG_CPU``
705 This entry needs to be enabled in Kconfig
708 Arch interface to bring up a CPU
711 Arch interface to shutdown a CPU, no more interrupts can be handled by the
712 kernel after the routine returns. This includes the shutdown of the timer.
715 This actually supposed to ensure death of the CPU. Actually look at some
716 example code in other arch that implement CPU hotplug. The processor is taken
717 down from the ``idle()`` loop for that specific architecture. ``__cpu_die()``
718 typically waits for some per_cpu state to be set, to ensure the processor dead
719 routine is called to be sure positively.
721 User Space Notification
722 =======================
724 After CPU successfully onlined or offline udev events are sent. A udev rule like::
726 SUBSYSTEM=="cpu", DRIVERS=="processor", DEVPATH=="/devices/system/cpu/*", RUN+="the_hotplug_receiver.sh"
728 will receive all events. A script like::
732 if [ "${ACTION}" = "offline" ]
734 echo "CPU ${DEVPATH##*/} offline"
736 elif [ "${ACTION}" = "online" ]
738 echo "CPU ${DEVPATH##*/} online"
742 can process the event further.
744 When changes to the CPUs in the system occur, the sysfs file
745 /sys/devices/system/cpu/crash_hotplug contains '1' if the kernel
746 updates the kdump capture kernel list of CPUs itself (via elfcorehdr),
747 or '0' if userspace must update the kdump capture kernel list of CPUs.
749 The availability depends on the CONFIG_HOTPLUG_CPU kernel configuration
752 To skip userspace processing of CPU hot un/plug events for kdump
753 (i.e. the unload-then-reload to obtain a current list of CPUs), this sysfs
754 file can be used in a udev rule as follows:
756 SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
758 For a CPU hot un/plug event, if the architecture supports kernel updates
759 of the elfcorehdr (which contains the list of CPUs), then the rule skips
760 the unload-then-reload of the kdump capture kernel.
762 Kernel Inline Documentations Reference
763 ======================================
765 .. kernel-doc:: include/linux/cpuhotplug.h