Messages in this thread | | | Date | Mon, 20 Feb 2023 14:10:00 +0100 | From | Mirsad Todorovac <> | Subject | Re: INFO: REPRODUCED: memory leak in gpio device in 6.2-rc6 |
| |
On 2/16/23 15:16, Bartosz Golaszewski wrote: > On Wed, Feb 15, 2023 at 3:45 PM Mirsad Goran Todorovac > <mirsad.todorovac@alu.unizg.hr> wrote: >> >> On 15.2.2023. 11:53, Andy Shevchenko wrote: >>> On Tue, Feb 14, 2023 at 07:19:16PM +0100, Mirsad Goran Todorovac wrote: >>>> On 12. 02. 2023. 15:19, Andy Shevchenko wrote: >>>>> On Wed, Feb 08, 2023 at 08:55:24PM +0100, Mirsad Goran Todorovac wrote: >>>>>> On 31. 01. 2023. 10:36, Mirsad Goran Todorovac wrote: >>>>>>> I came across this memory leak apparently in the GPIO device driver. >>>>>>> It is still present in 6.2-rc6 release candidate kernel (just ran kselftest). >>>>>>> >>>>>>> This is a vanilla Torvalds tree kernel with MGLRU and KMEMLEAK (obviously) >>>>>>> enabled. >>>>>>> >>>>>>> If you think this bug is significant, I can attempt the bug bisect in the >>>>>>> environment that triggered it (Lenovo LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB) >>>>>>> with BIOS M22KT49A from 11/10/2022 and AlmaLinux 8.7. >>>>>>> >>>>>>> Here is the /sys/kernel/debug/kmemleak output: >>>>>>> >>>>>>> unreferenced object 0xffff9e67ad71f160 (size 32): >>>>>>> comm "gpio-sim.sh", pid 208926, jiffies 4372229685 (age 2101.564s) >>>>>>> hex dump (first 32 bytes): >>>>>>> 67 70 69 6f 2d 73 69 6d 2e 30 2d 6e 6f 64 65 30 gpio-sim.0-node0 >>>>>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >>>>>>> backtrace: >>>>>>> [<0000000098bf3d1b>] slab_post_alloc_hook+0x91/0x320 >>>>>>> [<00000000da3205c5>] __kmem_cache_alloc_node+0x1bf/0x2b0 >>>>>>> [<00000000aa51a58a>] __kmalloc_node_track_caller+0x55/0x140 >>>>>>> [<00000000bd682ecc>] kvasprintf+0x6b/0xd0 >>>>>>> [<00000000a3431d55>] kasprintf+0x4e/0x70 >>>>>>> [<00000000f52d2629>] gpio_sim_device_config_live_store+0x401/0x59d [gpio_sim] >>>>>>> [<00000000673fc6df>] configfs_write_iter+0xcc/0x130 >>>>>>> [<000000001d5d0829>] vfs_write+0x2b4/0x3d0 >>>>>>> [<00000000d2336251>] ksys_write+0x61/0xe0 >>>>>>> [<00000000f7015bb1>] __x64_sys_write+0x1a/0x20 >>>>>>> [<000000008ac743d2>] do_syscall_64+0x58/0x80 >>>>>>> [<000000004d7b7d50>] entry_SYSCALL_64_after_hwframe+0x63/0xcd >>> >>>> Sorry, but unfortunately this patch didn't fix the memleak. Please see the result: >>> >>> Thank you for give a try! >> >> No sweat. It was worth the effort. >> >>> Yeah, that's why I put that I'm skeptical, because while patch is correct per >>> se it wouldn't prevent the initial leakage (it seems it happens due to other >>> circumstances). >> >> I must admit that is looks like hieroglyphs to me. My learning curve had not >> yet reached that point of debugging memory leaks in the kernel drivers ... >> >>>> root@marvin-IdeaPad-3-15ITL6:/home/marvin/linux/kernel/linux_torvalds/tools/testing/selftests/gpio# echo clear > /sys/kernel/debug/kmemleak >>>> root@marvin-IdeaPad-3-15ITL6:/home/marvin/linux/kernel/linux_torvalds/tools/testing/selftests/gpio# time ./gpio-sim.sh >>>> trap: SIGTERM: bad trap >>>> 1. chip_name and dev_name attributes >>>> 1.1. Chip name is communicated to user >>>> 1.2. chip_name returns 'none' if the chip is still pending >>>> 1.3. Device name is communicated to user >>>> 2. Creating and configuring simulated chips >>>> 2.1. Default number of lines is 1 >>>> 2.2. Number of lines can be specified >>>> 2.3. Label can be set >>>> 2.4. Label can be left empty >>>> 2.5. Line names can be configured >>>> 2.6. Line config can remain unused if offset is greater than number of lines >>>> 2.7. Line configfs directory names are sanitized >>>> 2.8. Multiple chips can be created >>>> 2.9. Can't modify settings when chip is live >>>> 2.10. Can't create line items when chip is live >>>> 2.11. Probe errors are propagated to user-space >>>> 2.12. Cannot enable a chip without any GPIO banks >>>> 2.13. Duplicate chip labels are not allowed >>>> 2.14. Lines can be hogged >>>> 3. Controlling simulated chips >>>> 3.1. Pull can be set over sysfs >>>> 3.2. Pull can be read from sysfs >>>> 3.3. Incorrect input in sysfs is rejected >>>> 3.4. Can't write to value >>>> 4. Simulated GPIO chips are functional >>>> 4.1. Values can be read from sysfs >>>> 4.2. Bias settings work correctly >>>> GPIO gpio-sim test PASS >>>> >>>> real 0m1.120s >>>> user 0m0.283s >>>> sys 0m0.842s >>>> root@marvin-IdeaPad-3-15ITL6:/home/marvin/linux/kernel/linux_torvalds/tools/testing/selftests/gpio# echo scan > /sys/kernel/debug/kmemleak >>>> root@marvin-IdeaPad-3-15ITL6:/home/marvin/linux/kernel/linux_torvalds/tools/testing/selftests/gpio# cat /sys/kernel/debug/kmemleak >>>> unreferenced object 0xffff91f019219660 (size 32): >>>> comm "gpio-sim.sh", pid 11223, jiffies 4295028142 (age 87.304s) >>>> hex dump (first 32 bytes): >>>> 67 70 69 6f 2d 73 69 6d 2e 30 2d 6e 6f 64 65 30 gpio-sim.0-node0 >>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >>>> backtrace: >>>> [<ffffffff86fcd408>] __kmem_cache_alloc_node+0x1d8/0x330 >>>> [<ffffffff86f4c271>] __kmalloc_node_track_caller+0x51/0xd0 >>>> [<ffffffff872fb359>] kvasprintf+0x69/0xd0 >>>> [<ffffffff872fb4ce>] kasprintf+0x4e/0x70 >>>> [<ffffffffc1864553>] gpio_sim_device_config_live_store+0x483/0x6b5 [gpio_sim] >>>> [<ffffffff870dde9c>] configfs_write_iter+0xcc/0x130 >>>> [<ffffffff87016869>] vfs_write+0x1f9/0x3b0 >>>> [<ffffffff87016dab>] ksys_write+0x6b/0xf0 >>>> [<ffffffff87016e59>] __x64_sys_write+0x19/0x20 >>>> [<ffffffff87b91188>] do_syscall_64+0x58/0x80 >>>> [<ffffffff87c000aa>] entry_SYSCALL_64_after_hwframe+0x72/0xdc >>>> root@marvin-IdeaPad-3-15ITL6:/home/marvin/linux/kernel/linux_torvalds/tools/testing/selftests/gpio# uname -rms >>>> Linux 6.2.0-rc8-lru-km-andy-00015-gf6feea56f66d-dirty x86_64 >>>> root@marvin-IdeaPad-3-15ITL6:/home/marvin/linux/kernel/linux_torvalds/tools/testing/selftests/gpio# >>>> >>>> I can verify that I build the right patch: >>>> >>>> marvin@marvin-IdeaPad-3-15ITL6:~/linux/kernel/linux_torvalds$ git diff >>>> diff --git a/drivers/gpio/gpio-sim.c b/drivers/gpio/gpio-sim.c >>>> index 60514bc5454f..7f79e49b23d7 100644 >>>> --- a/drivers/gpio/gpio-sim.c >>>> +++ b/drivers/gpio/gpio-sim.c >>>> @@ -954,9 +954,9 @@ static void gpio_sim_device_deactivate_unlocked(struct gpio_sim_device *dev) >>>> >>>> swnode = dev_fwnode(&dev->pdev->dev); >>>> platform_device_unregister(dev->pdev); >>>> + gpio_sim_remove_hogs(dev); >>>> gpio_sim_remove_swnode_recursive(swnode); >>>> dev->pdev = NULL; >>>> - gpio_sim_remove_hogs(dev); >>>> } >>>> >>>> static ssize_t >>>> marvin@marvin-IdeaPad-3-15ITL6:~/linux/kernel/linux_torvalds$ >>>> >>>> Alternatively, I could try to bisect if you think it's prudent to try that. >>>> But first I need a stroll after this kernel build :-) >>>> >>>> Do you think that knowing when the bug was introduced might help find the culprit?
> Thanks for the report Mirsad, just sent out a fix.
Hi, Bart,
It is really nothing. The reproducer shows that the leak is apparently gone.
As Mr. McKenney once said, a bunch of monkeys with keyboard could have done it in a considerable number of trials and errors ;-)
But here I have something that could potentially leak as well. I could not devise a reproducer due to the leak being lightly triggered only in extreme memory contention.
See it for yourself:
drivers/gpio/gpio-sim.c: 301 static int gpio_sim_setup_sysfs(struct gpio_sim_chip *chip) 302 { 303 struct device_attribute *val_dev_attr, *pull_dev_attr; 304 struct gpio_sim_attribute *val_attr, *pull_attr; 305 unsigned int num_lines = chip->gc.ngpio; 306 struct device *dev = chip->gc.parent; 307 struct attribute_group *attr_group; 308 struct attribute **attrs; 309 int i, ret; 310 311 chip->attr_groups = devm_kcalloc(dev, sizeof(*chip->attr_groups), 312 num_lines + 1, GFP_KERNEL); 313 if (!chip->attr_groups) 314 return -ENOMEM; 315 316 for (i = 0; i < num_lines; i++) { 317 attr_group = devm_kzalloc(dev, sizeof(*attr_group), GFP_KERNEL); 318 attrs = devm_kcalloc(dev, GPIO_SIM_NUM_ATTRS, sizeof(*attrs), 319 GFP_KERNEL); 320 val_attr = devm_kzalloc(dev, sizeof(*val_attr), GFP_KERNEL); 321 pull_attr = devm_kzalloc(dev, sizeof(*pull_attr), GFP_KERNEL); 322 if (!attr_group || !attrs || !val_attr || !pull_attr) 323 return -ENOMEM; 324 325 attr_group->name = devm_kasprintf(dev, GFP_KERNEL, 326 "sim_gpio%u", i); 327 if (!attr_group->name) 328 return -ENOMEM;
Apparently, if the memory allocation only partially succeeds, in the theoretical case that the system is close to its kernel memory exhaustion, `return -ENOMEM` would not free the partially succeeded allocs, would it?
To explain it better, I tried a version that is not yet full doing "all or nothing" memory allocation for the gpio-sim driver, because I am not that familiar with the driver internals.
I give this just as an illustration to what I mean to say (one line of code speaks like thousand words):
301 static int gpio_sim_setup_sysfs(struct gpio_sim_chip *chip) 302 { 303 struct device_attribute *val_dev_attr, *pull_dev_attr; 304 struct gpio_sim_attribute *val_attr, *pull_attr; 305 unsigned int num_lines = chip->gc.ngpio; 306 struct device *dev = chip->gc.parent; 307 struct attribute_group *attr_group; 308 struct attribute **attrs; 309 int i, ret; 310 311 chip->attr_groups = devm_kcalloc(dev, sizeof(*chip->attr_groups), 312 num_lines + 1, GFP_KERNEL); 313 if (!chip->attr_groups) 314 return -ENOMEM; 315 316 for (i = 0; i < num_lines; i++) { 317 attr_group = devm_kzalloc(dev, sizeof(*attr_group), GFP_KERNEL); 318 attrs = devm_kcalloc(dev, GPIO_SIM_NUM_ATTRS, sizeof(*attrs), 319 GFP_KERNEL); 320 val_attr = devm_kzalloc(dev, sizeof(*val_attr), GFP_KERNEL); 321 pull_attr = devm_kzalloc(dev, sizeof(*pull_attr), GFP_KERNEL); 322 if (!attr_group || !attrs || !val_attr || !pull_attr) { 323 if (attr_group) 324 devm_kfree(dev, attr_group); 325 if (attrs) 326 devm_kfree(dev, attrs); 327 if (val_attr) 328 devm_kfree(dev, val_attr); 329 if (pull_attr) 330 devm_kfree(dev, pull_attr); 331 return -ENOMEM; 332 } 333 334 attr_group->name = devm_kasprintf(dev, GFP_KERNEL, 335 "sim_gpio%u", i); 336 if (!attr_group->name) { 337 devm_kfree(dev, attr_group); 338 devm_kfree(dev, attrs); 339 devm_kfree(dev, val_attr); 340 devm_kfree(dev, pull_attr); 341 return -ENOMEM; 342 } 343 344 val_attr->offset = pull_attr->offset = i;
This is, of course, far from perfect, and it seems that the driver will still leak memory in case of partially successful allocation because of the for (i = 0; i < num_lines; i++) { ... } loop. :-/
I hope this makes any sense to you.
Thanks, Mirsad
-- Mirsad Goran Todorovac Sistem inženjer Grafički fakultet | Akademija likovnih umjetnosti Sveučilište u Zagrebu
System engineer Faculty of Graphic Arts | Academy of Fine Arts University of Zagreb, Republic of Croatia
| |