Messages in this thread | | | From | John Ogness <> | Subject | [PATCH printk v3 00/14] fix console flushing | Date | Thu, 14 Dec 2023 22:47:47 +0106 |
| |
Hi,
While testing various flushing scenarios, I stumbled on a few issues that cause console flushing to fail. While at LPC2023 in Richmond, I sat down with Petr Mladek and we reviewed the v2 [0] series. This v3 series is the result of that offline discussion.
This series addresses the following issues:
1. The prb_next_seq() optimization caused inconsistent return values. Fix prb_next_seq() to the originally intended behavior but keep an optimization.
2. pr_flush() might not wait until the most recently stored printk() message if non-finalized records precede it. Fix pr_flush() to wait for all records to print that are at least reserved at the time of the call.
3. In panic, the panic messages will not print if non-finalized records precede them. Add a special condition so that readers on the panic CPU will drop records that are not in a consistent state.
4. It is possible (and easy to reproduce) a scenario where the console on the panic CPU hands over to a waiter of a stopped CPU. Do not use the handover feature in panic.
5. If messages are being dropped during panic, non-panic CPUs are silenced. But by then it is already too late and most likely the panic messages have been dropped. Change the non-panic CPU silencing logic to _immediately_ silence non-panic CPUs during panic. This also leads to clean panic output when many CPUs are blasting the kernel log.
6. If a panic occurs in a context where printk() calls defer printing (NMI or printk_safe section), the printing of the final panic messages rely on irq_work. If that mechanism is not available, the final panic messages are not seen (even though they are finalized in the ringbuffer). Add one last explicit flush after all printk() calls are finished to ensure all available messages in the kernel log are printed.
This series also performs some minor cleanups to remove open coded checks about the panic context and improve documentation language regarding data-less records.
Because of multiple refactoring done in recent history, it would be helpful to provide the LTS maintainers with the proper backported patches. I am happy to do this.
The changes since v2:
- When on the panic CPU, allow desc_read_finalized_seq() to read records in the committed state.
- Fix the off-by-one return value of the new function prb_next_reserve_seq(). [1]
- Remove the suppress_panic_printk feature and instead simply silently drop all printk() messages from non-panic CPUs.
- Remove the __seq_to_nbcon_seq()/__nbcon_seq_to_seq() macros and use the new __u64seq_to_ulseq()/__ulseq_to_u64seq() macros for nbcon as well.
- Adjust the new __u64seq_to_ulseq()/__ulseq_to_u64seq() macros to support seq values above the base value. This is necessary for the new last_finalized_seq feature. [2]
- Relocate the new __u64seq_to_ulseq()/__ulseq_to_u64seq() macros to printk_ringbuffer.h so they can be shared by the printk_ringbuffer and nbcon code.
- Call console_flush_on_panic() one more time at the end of panic().
- Add and cleanup many comments and commit messages to clarify reasoning and variable meanings.
John Ogness
[0] https://lore.kernel.org/lkml/20231106210730.115192-1-john.ogness@linutronix.de [1] https://lore.kernel.org/lkml/87h6lwcxix.fsf@jogness.linutronix.de [2] https://lore.kernel.org/lkml/87lear2i8w.fsf@jogness.linutronix.de
John Ogness (13): printk: nbcon: Relocate 32bit seq macros printk: Adjust mapping for 32bit seq macros printk: Use prb_first_seq() as base for 32bit seq macros printk: ringbuffer: Do not skip non-finalized records with prb_next_seq() printk: ringbuffer: Clarify special lpos values printk: For @suppress_panic_printk check for other CPU in panic printk: Add this_cpu_in_panic() printk: ringbuffer: Cleanup reader terminology printk: Wait for all reserved records with pr_flush() printk: ringbuffer: Skip non-finalized records in panic printk: ringbuffer: Consider committed as finalized in panic printk: Avoid non-panic CPUs writing to ringbuffer panic: Flush kernel log buffer at the end
Petr Mladek (1): printk: Disable passing console lock owner completely during panic()
kernel/panic.c | 8 + kernel/printk/internal.h | 1 + kernel/printk/nbcon.c | 41 +--- kernel/printk/printk.c | 101 +++++---- kernel/printk/printk_ringbuffer.c | 360 +++++++++++++++++++++++++----- kernel/printk/printk_ringbuffer.h | 54 ++++- 6 files changed, 427 insertions(+), 138 deletions(-)
base-commit: 6c3a34e38436a2a3f7a1fa764c108ee19b05b893 -- 2.39.2
| |