lkml.org 
[lkml]   [2020]   [Dec]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [PATCH -tip 00/32] Core scheduling (v9)
    From
    Date

    On 2020/11/24 23:08, Joel Fernandes wrote:
    >>>
    >>> Core-Scheduling
    >>> ===============
    >>> Enclosed is series v9 of core scheduling.
    >>> v9 is rebased on tip/master (fe4adf6f92c4 ("Merge branch 'irq/core'"))..
    >>> I hope that this version is acceptable to be merged (pending any new review
    >>> comments that arise) as the main issues in the past are all resolved:
    >>> 1. Vruntime comparison.
    >>> 2. Documentation updates.
    >>> 3. CGroup and per-task interface developed by Google and Oracle.
    >>> 4. Hotplug fixes.
    >>> Almost all patches also have Reviewed-by or Acked-by tag. See below for full
    >>> list of changes in v9.
    >>>
    >>> Introduction of feature
    >>> =======================
    >>> Core scheduling is a feature that allows only trusted tasks to run
    >>> concurrently on cpus sharing compute resources (eg: hyperthreads on a
    >>> core). The goal is to mitigate the core-level side-channel attacks
    >>> without requiring to disable SMT (which has a significant impact on
    >>> performance in some situations). Core scheduling (as of v7) mitigates
    >>> user-space to user-space attacks and user to kernel attack when one of
    >>> the siblings enters the kernel via interrupts or system call.
    >>>
    >>> By default, the feature doesn't change any of the current scheduler
    >>> behavior. The user decides which tasks can run simultaneously on the
    >>> same core (for now by having them in the same tagged cgroup). When a tag
    >>> is enabled in a cgroup and a task from that cgroup is running on a
    >>> hardware thread, the scheduler ensures that only idle or trusted tasks
    >>> run on the other sibling(s). Besides security concerns, this feature can
    >>> also be beneficial for RT and performance applications where we want to
    >>> control how tasks make use of SMT dynamically.
    >>>
    >>> Both a CGroup and Per-task interface via prctl(2) are provided for configuring
    >>> core sharing. More details are provided in documentation patch. Kselftests are
    >>> provided to verify the correctness/rules of the interface.
    >>>
    >>> Testing
    >>> =======
    >>> ChromeOS testing shows 300% improvement in keypress latency on a Google
    >>> docs key press with Google hangout test (the maximum latency drops from 150ms
    >>> to 50ms for keypresses).
    >>>
    >>> Julien: TPCC tests showed improvements with core-scheduling as below. With kernel
    >>> protection enabled, it does not show any regression. Possibly ASI will improve
    >>> the performance for those who choose kernel protection (can be toggled through
    >>> sched_core_protect_kernel sysctl).
    >>> average stdev diff
    >>> baseline (SMT on) 1197.272 44.78312824
    >>> core sched ( kernel protect) 412.9895 45.42734343 -65.51%
    >>> core sched (no kernel protect) 686.6515 71.77756931 -42.65%
    >>> nosmt 408.667 39.39042872 -65.87%
    >>> (Note these results are from v8).
    >>>
    >>> Vineeth tested sysbench and does not see any regressions.
    >>> Hong and Aubrey tested v9 and see results similar to v8. There is a known issue
    >>> with uperf that does regress. This appears to be because of ksoftirq heavily
    >>> contending with other tasks on the core. The consensus is this can be improved
    >>> in the future.
    >>>
    >>> Changes in v9
    >>> =============
    >>> - Note that the vruntime snapshot change is written in 2 patches to show the
    >>> progression of the idea and prevent merge conflicts:
    >>> sched/fair: Snapshot the min_vruntime of CPUs on force idle
    >>> sched: Improve snapshotting of min_vruntime for CGroups
    >>> Same with the RT priority inversion change:
    >>> sched: Fix priority inversion of cookied task with sibling
    >>> sched: Improve snapshotting of min_vruntime for CGroups
    >>> - Disable coresched on certain AMD HW.
    >>>

    Adding workloads and negative case test results for core scheduling v9 posted:

    - kernel under test:
    -- coresched community v9 posted from https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=sched/coresched-v9-posted (tag: sched/coresched-v9-posted)
    -- latest commit: d48636e429de (HEAD -> coresched-v9-posted, tag: sched/coresched-v9-posted) sched: Debug bits...
    -- coresched=on kernel parameter applied
    - workloads:
    -- A. sysbench cpu (192 threads) + sysbench cpu (192 threads)
    -- B. sysbench cpu (192 threads) + sysbench mysql (192 threads, mysqld forced into the same cgroup)
    -- C. uperf netperf.xml (192 threads over TCP or UDP protocol separately)
    -- D. will-it-scale context_switch via pipe (192 threads)
    - negative case:
    -- A. continuously toggle cpu.core_tag, during full loading uperf workload running with cs_on
    -- B. continuously toggle smt setting via /sys/devices/system/cpu/smt/control, during full loading uperf workload running with cs_on
    -- C. continuously cgroup switch between cs_on cgroup and cs_off cgroup via cgclassify, during full loading uperf workload running
    - test machine setup:
    CPU(s): 192
    On-line CPU(s) list: 0-191
    Thread(s) per core: 2
    Core(s) per socket: 48
    Socket(s): 2
    NUMA node(s): 4
    - test results of workloads, no obvious performance drop compared to community v8 build:
    -- workload A:
    +----------------------+------+----------------------+------------------------+
    | workloads | ** | sysbench cpu * 192 | sysbench cpu * 192 |
    +======================+======+======================+========================+
    | cgroup | ** | cg_sysbench_cpu_0 | cg_sysbench_cpu_1 |
    +----------------------+------+----------------------+------------------------+
    | record_item | ** | Tput_avg (events/s) | Tput_avg (events/s) |
    +----------------------+------+----------------------+------------------------+
    | coresched_normalized | ** | 0.97 | 1.02 |
    +----------------------+------+----------------------+------------------------+
    | default_normalized | ** | 1.00 | 1.00 |
    +----------------------+------+----------------------+------------------------+
    | smtoff_normalized | ** | 0.60 | 0.60 |
    +----------------------+------+----------------------+------------------------+

    -- workload B:
    +----------------------+------+----------------------+------------------------+
    | workloads | ** | sysbench cpu * 192 | sysbench mysql * 192 |
    +======================+======+======================+========================+
    | cgroup | ** | cg_sysbench_cpu_0 | cg_sysbench_mysql_0 |
    +----------------------+------+----------------------+------------------------+
    | record_item | ** | Tput_avg (events/s) | Tput_avg (events/s) |
    +----------------------+------+----------------------+------------------------+
    | coresched_normalized | ** | 0.94 | 0.88 |
    +----------------------+------+----------------------+------------------------+
    | default_normalized | ** | 1.00 | 1.00 |
    +----------------------+------+----------------------+------------------------+
    | smtoff_normalized | ** | 0.56 | 0.84 |
    +----------------------+------+----------------------+------------------------+

    -- workload C:
    +----------------------+------+---------------------------+---------------------------+
    | workloads | ** | uperf netperf TCP * 192 | uperf netperf UDP * 192 |
    +======================+======+===========================+===========================+
    | cgroup | ** | cg_uperf | cg_uperf |
    +----------------------+------+---------------------------+---------------------------+
    | record_item | ** | Tput_avg (Gb/s) | Tput_avg (Gb/s) |
    +----------------------+------+---------------------------+---------------------------+
    | coresched_normalized | ** | 0.64 | 0.68 |
    +----------------------+------+---------------------------+---------------------------+
    | default_normalized | ** | 1.00 | 1.00 |
    +----------------------+------+---------------------------+---------------------------+
    | smtoff_normalized | ** | 0.92 | 0.89 |
    +----------------------+------+---------------------------+---------------------------+

    -- workload D:
    +----------------------+------+-------------------------------+
    | workloads | ** | will-it-scale * 192 |
    | | | (pipe based context_switch) |
    +======================+======+===============================+
    | cgroup | ** | cg_will-it-scale |
    +----------------------+------+-------------------------------+
    | record_item | ** | threads_avg |
    +----------------------+------+-------------------------------+
    | coresched_normalized | ** | 0.30 |
    +----------------------+------+-------------------------------+
    | default_normalized | ** | 1.00 |
    +----------------------+------+-------------------------------+
    | smtoff_normalized | ** | 0.87 |
    +----------------------+------+-------------------------------+

    -- notes on record_item:
    * coresched_normalized: smton, cs enabled, test result normalized by default value
    * default_normalized: smton, cs disabled, test result normalized by default value
    * smtoff_normalized: smtoff, test result normalized by default value

    - test results of negative case, all as expected, no kernel panic or system hang observed

    Hongyu

    \
     
     \ /
      Last update: 2020-12-03 07:20    [W:4.214 / U:0.128 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site