Hi John and Georgia,

I found a repeatable AF_UNIX datagram send slowdown caused by the exact source
delta of 6456ccbd2ff7 ("apparmor: fix regression in fs based unix sockets when
using old abi"). In a bare-metal parent/child/parent test, a standalone
sendmmsg() workload was 15.30% slower after the change. An independent larger
runner reproduced the slowdown with both sendmmsg() and io_uring SEND, so this
is not specific to io_uring.

#regzbot introduced: 6456ccbd2ff7
#regzbot title: apparmor AF_UNIX datagram send slowdown

The standalone workload uses socketpair(AF_UNIX, SOCK_DGRAM). Each sendmmsg()
call sends 32 messages of 128 bytes. Peer draining and payload validation are
outside the timed region. Each point had 3 warm-up rounds followed by 15
measured rounds of 65,536 messages.

Because 6456ccbd2ff7 was part of an AppArmor series, I isolated its source
delta on one controlled baseline:

  parent 0bfa1c2da7a8: AppArmor topic prefix through 50d56a1a366a
  child  30cb02a874b4: parent plus only the exact two-file 6456ccbd2ff7 diff

The child tree matches the same baseline with the AppArmor prefix advanced
through 6456ccbd2ff7. I used a fresh boot for each point:

  point       mean ns/message
  parent A          343.382
  child             395.831
  parent B          343.259

The child was 15.295% slower than the parent midpoint. Dropping the first
measured round gave 15.289%; parent drift was -0.036%, and the maximum CV was
0.137%. All 45 measured rows passed the payload, length, count, and empty-queue
checks.

The machine was an Intel Core i7-12700KF system with 32 GiB RAM. The workload
was pinned to P-core CPU 2, with governor and EPP set to performance and Turbo
disabled. All three runs used actual preempt=none and reported
/proc/self/attr/current as unconfined.

As a separate check, the larger original runner used matched actual
preempt=full. Its direct sendmmsg() profile was 14.841% slower, and its
io_uring SEND profile was 17.953% slower. I am keeping these results separate
from the standalone percentage.

A separate v6.16.12 -> v6.17.13 perf comparison was directionally consistent.
The children overhead reported for security_unix_may_send was 0.27% in both
v6.16.12 controls and 6.77% at v6.17.13. The latter included the path through
apparmor_unix_may_send(), aa_unix_peer_perm(), and unix_peer_perm(). This was
a release-endpoint comparison rather than the exact source pair above, so I
use it only as supporting path evidence, not as an internal decomposition of
the 15.295% slowdown.

I understand that 6456ccbd2ff7 fixes a real old AppArmor policy ABI issue,
so I am not suggesting a revert. Would it be possible to retain that fix while
avoiding the added cost on this unconfined AF_UNIX datagram send path?

Evidence bundle:

  https://github.com/lcf0399/linux-regression-evidence/tree/7b9862c7ac2d5f9622725912d7f0df95011bcbfd/apparmor-af-unix-send-old-abi-6456cc

Thanks,
Chengfeng