Messages in this thread | | | Date | Thu, 13 Jan 2022 11:58:16 +0100 | From | Greg KH <> | Subject | Re: [PATCH v3 2/2] kernfs: Reduce contention around global per-fs kernfs_rwsem. |
| |
On Thu, Jan 13, 2022 at 09:42:59PM +1100, Imran Khan wrote: > Right now a global per file system based rwsem (kernfs_rwsem) > synchronizes multiple kernfs operations. On a large system with > few hundred CPUs and few hundred applications simultaenously trying > to access sysfs, this results in multiple sys_open(s) contending on > kernfs_rwsem via kernfs_iop_permission and kernfs_dop_revalidate. > > - 21.42% 21.34% showgids [kernel.kallsyms] [k] up_read > 21.34% __libc_start_main > __GI___libc_open > entry_SYSCALL_64_after_hwframe > do_syscall_64 > sys_open > do_sys_open > do_filp_open > - path_openat > - 20.05% link_path_walk > - 9.76% walk_component > lookup_fast > - d_revalidate.part.24 > - 9.75% kernfs_dop_revalidate > up_read > - 9.46% inode_permission > - __inode_permission > - 9.46% kernfs_iop_permission > up_read > - 0.83% kernfs_iop_get_link > up_read > - 0.80% lookup_fast > d_revalidate.part.24 > kernfs_dop_revalidate > up_read > > - 21.31% 21.21% showgids [kernel.kallsyms] [k] down_read > 21.21% __libc_start_main > __GI___libc_open > entry_SYSCALL_64_after_hwframe > do_syscall_64 > sys_open > do_sys_open > do_filp_open > - path_openat > - 19.78% link_path_walk > - 10.62% inode_permission > - __inode_permission > - 10.62% kernfs_iop_permission > down_read > - 8.45% walk_component > lookup_fast > - d_revalidate.part.24 > - 8.45% kernfs_dop_revalidate > down_read > - 0.71% kernfs_iop_get_link > down_read > - 0.72% lookup_fast > - d_revalidate.part.24 > - 0.72% kernfs_dop_revalidate > down_read > - 0.71% may_open > inode_permission > __inode_permission > kernfs_iop_permission > down_read > > Since permission is specific to a kernfs_node we can use a hashed > lock to access/modify permission. Also use kernfs reference counting > to ensure we are accessing/modifying permissions for an existing > kernfs_node object. > > Using this change brings down the above mentioned down_read/up_read > numbers to ~8%, thus indicating that contention around kernfs_rwsem > has reduced to about 1/3rd of earlier value.
Ah, nevermind, you do post the results here, I should have kept reading.
Nice work!
I'll look at these after 5.17-rc1 is out, thanks!
greg k-h
| |