lkml.org 
[lkml]   [2016]   [Jan]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [RFD] CAT user space interface revisited
On Thu, Dec 31, 2015 at 11:30:57PM +0100, Thomas Gleixner wrote:
> Marcelo,
>
> On Thu, 31 Dec 2015, Marcelo Tosatti wrote:
>
> First of all thanks for the explanation.
>
> > There is one directory structure in this topic, CAT. That is the
> > directory structure which is exposed to userspace to control the
> > CAT HW.
> >
> > With the current patchset posted by Intel ("Subject: [PATCH V16 00/11]
> > x86: Intel Cache Allocation Technology Support"), the directory
> > structure there (the files and directories exposed by that patchset)
> > (*1) does not allow one to configure different CBM masks on each socket
> > (that is, it forces the user to configure the same mask CBM on every
> > socket). This is a blocker for us, and it is one of the points in your
> > proposal.
> >
> > There was a call between Red Hat and Intel where it was communicated
> > to Intel, and Intel agreed, that it was necessary to fix this (fix this
> > == allow different CBM masks on different sockets).
> >
> > Now, that is one change to the current directory structure (*1).
>
> I don't have an idea how that would look like. The current structure is a
> cgroups based hierarchy oriented approach, which does not allow simple things
> like
>
> T1 00001111
> T2 00111100
>
> at least not in a way which is natural to the problem at hand.



cgroupA/

cbm_mask (if set, set for all CPUs)

socket1/cbm_mask
socket2/cbm_mask
...
socketN/cbm_mask (if set, overrides global
cbm_mask).

Something along those lines.

Do you see any problem with it?

> > (*1) modified to allow for different CBM masks on different sockets,
> > lets say (*2), is what we have been waiting for Intel to post.
> > It would handle our usecase, and all use-cases which the current
> > patchset from Intel already handles (Vikas posted emails mentioning
> > there are happy users of the current interface, feel free to ask
> > him for more details).
>
> I cannot imagine how that modification to the current interface would solve
> that. Not to talk about per CPU associations which are not related to tasks at
> all.

Not sure what you mean by per CPU associations.

If you fix a cbmmask on a given pCPU, say CPU1, and control which tasks
run on that pCPU, then you control the cbmmask for all tasks (say
tasklist-1) on that CPU, fine.

Can achieve the same by putting all tasks from tasklist-1 into a
cgroup.

> > What i have asked you, and you replied "to go Google read my previous
> > post" is this:
> > What are the advantages over you proposal (which is a completely
> > different directory structure, requiring a complete rewrite),
> > over (*2) ?
> >
> > (what is my reason behind this: the reason is that if you, with
> > maintainer veto power, forces your proposal to be accepted, it will be
> > necessary to wait for another rewrite (a new set of problems, fully
> > think through your proposal, test it, ...) rather than simply modify an
> > already known, reviewed, already used directory structure.
> >
> > And functionally, your proposal adds nothing to (*2) (other than, well,
> > being a different directory structure).
>
> Sorry. I cannot see at all how a modification to the existing interface would
> cover all the sensible use cases I described in a coherent way. I really want
> to see a proper description of the interface before people start hacking on it
> in a frenzy. What you described is: "let's say (*2)" modification. That's
> pretty meager.
>
> > If Fenghua or you post a patchset, say in 2 weeks, with your proposal,
> > i am fine with that. But i since i doubt that will be the case, i am
> > pushing for the interface which requires the least amount of changes
> > (and therefore the least amount of time) to be integrated.
> >
> > >From your email:
> >
> > "It would even be sufficient for particular use cases to just associate
> > a piece of cache to a given CPU and do not bother with tasks at all.
> >
> > We really need to make this as configurable as possible from userspace
> > without imposing random restrictions to it. I played around with it on
> > my new intel toy and the restriction to 16 COS ids (that's 8 with CDP
> > enabled) makes it really useless if we force the ids to have the same
> > meaning on all sockets and restrict it to per task partitioning."
> >
> > Yes, thats the issue we hit, that is the modification that was agreed
> > with Intel, and thats what we are waiting for them to post.
>
> How do you implement the above - especially that part:
>
> "It would even be sufficient for particular use cases to just associate a
> piece of cache to a given CPU and do not bother with tasks at all."
>
> as a "simple" modification to (*1) ?

As noted above.
>
> > > I described a directory structure for that qos/cat stuff in my proposal and
> > > that's complete AFAICT.
> >
> > Ok, lets make the job for the submitter easier. You are the maintainer,
> > so you decide.
> >
> > Is it enough for you to have (*2) (which was agreed with Intel), or
> > would you rather prefer to integrate the directory structure at
> > "[RFD] CAT user space interface revisited" ?
>
> The only thing I care about as a maintainer is, that we merge something which
> actually reflects the properties of the hardware and gives the admin the
> required flexibility to utilize it fully. I don't care at all if it's my
> proposal or something else which allows to do the same.
>
> Let me copy the relevant bits from my proposal here once more and let me ask
> questions to the various points so you can tell me how that modification to
> (*1) is going to deal with that.
>
> >> At top level:
> >>
> >> xxxxxxx/cat/max_cosids <- Assume that all CPUs are the same
> >> xxxxxxx/cat/max_maskbits <- Assume that all CPUs are the same

This can be exposed to userspace via a file.

> >> xxxxxxx/cat/cdp_enable <- Depends on CDP availability
>
> Where is that information in (*2) and how is that related to (*1)? If you
> think it's not required, please explain why.

Intel has come up with a scheme to implement CDP. I'll go read
that and reply to this email afterwards.

> >> Per socket data:
> >>
> >> xxxxxxx/cat/socket-0/
> >> ...
> >> xxxxxxx/cat/socket-N/l3_size
> >> xxxxxxx/cat/socket-N/hwsharedbits
>
> Where is that information in (*2) and how is that related to (*1)? If you
> think it's not required, please explain why.

l3_size: userspace can figure that by itself (exposed somewhere in
sysfs).

hwsharedbits: All userspace needs to know is
which bits are shared with HW, to decide whether or not to use that
region of a given socket for a given cbmmask.

So expose that userspace, fine. Can do that in cgroups.

> >> Per socket mask data:
> >>
> >> xxxxxxx/cat/socket-N/cos-id-0/
> >> ...
> >> xxxxxxx/cat/socket-N/cos-id-N/inuse
> >> /cat_mask
> >> /cdp_mask <- Data mask if CDP enabled
>
> Where is that information in (*2) and how is that related to (*1)? If you
> think it's not required, please explain why.

Unsure - will reply in next email (but per-socket information seems
independent of that).

>
> >> Per cpu default cos id for the cpus on that socket:
> >>
> >> xxxxxxx/cat/socket-N/cpu-x/default_cosid
> >> ...
> >> xxxxxxx/cat/socket-N/cpu-N/default_cosid
> >>
> >> The above allows a simple cpu based partitioning. All tasks which do
> >> not have a cache partition assigned on a particular socket use the
> >> default one of the cpu they are running on.
>
> Where is that information in (*2) and how is that related to (*1)? If you
> think it's not required, please explain why.

Not required because with current Intel patchset you'd do:


# mount | grep rdt
cgroup on /sys/fs/cgroup/intel_rdt type cgroup
(rw,nosuid,nodev,noexec,relatime,intel_rdt)
# cd /sys/fs/cgroup/intel_rdt
# ls
cgroupALL cgroup.procs cgroup.sane_behavior
notify_on_release tasks
cgroup.clone_children cgroupRSVD intel_rdt.l3_cbm release_agent
# cat tasks
1042
1066
1067
1069
...
# cd cgroupALL/
ps auxw | while read i; do echo $i ; done
| cut -f 2 -d " " | grep -v PID | while read x ; do echo $x > tasks;
done
-bash: echo: write error: No such process
-bash: echo: write error: No such process
-bash: echo: write error: No such process
-bash: echo: write error: No such process

# cat ../tasks | while read i; do echo $i > tasks; done
# cat ../tasks | wc -l
0
(no tasks on root cgroup)

# cd ../cgroupRSVD
# cgroupRSVD]# cat tasks
# ps auxw | grep postfix
root 1942 0.0 0.0 91136 4860 ? Ss Nov25 0:00
/usr/libexec/postfix/master -w
postfix 1981 0.0 0.0 91308 6520 ? S Nov25 0:00 qmgr
-l -t unix -u
postfix 4416 0.0 0.0 91240 6296 ? S 17:05 0:00 pickup
-l -t unix -u
root 4486 0.0 0.0 112652 2304 pts/0 S+ 17:31 0:00 grep
--color=auto postfix
# echo 4416 > tasks
# cat intel_rdt.l3_cbm
000ffff0
# cat ../cgroupALL/intel_rdt.l3_cbm
000000ff

Bits f0 are shared between cgroupRSVD and cgroupALL. Lets change:
# echo 0xf > ../cgroupALL/intel_rdt.l3_cbm
# cat ../cgroupALL/intel_rdt.l3_cbm
0000000f

Now they share none.

------

So i have created "cgroupALL" (what you call default_cosid) and
"cgroupRSVD".

>
> >> Now for the task(s) partitioning:
> >>
> >> xxxxxxx/cat/partitions/
> >>
> >> Under that directory one can create partitions
> >>
> >> xxxxxxx/cat/partitions/p1/tasks
> >> /socket-0/cosid
> >> ...
> >> /socket-n/cosid
> >>
> >> The default value for the per socket cosid is COSID_DEFAULT, which
> >> causes the task(s) to use the per cpu default id.
>
> Where is that information in (*2) and how is that related to (*1)? If you
> think it's not required, please explain why.
>
> Yes. I ask the same question several times and I really want to see the
> directory/interface structure which solves all of the above before anyone
> starts to implement it.

I don't see the problem, have a sequence of commands above which shows
to set a directory structure which is useful and does what the HW
interface is supposed to do.

> We already have a completely useless interface (*1)
> and there is no point to implement another one based on it (*2) just because
> it solves your particular issue and is the fastest way forward. User space
> interfaces are hard and we really do not need some half baken solution which
> we have to support forever.

Fine. Can you please tell me what i can't do with the current interface?
AFAICS everything can be done (except missing support for (*2)).

>
> Let me enumerate the required points again:
>
> 1) Information about the hardware properties

Fine. Intel should expose that information.

> 2) Integration of CAT and CDP

Fine, Intel has comeup with a directory structure for that,
let me read the patchset again and i'll reply to you.

> 3) Per socket cos-id partitioning

(*2) as listed in the beginning of this e-mail.

> 4) Per cpu default cos-id association

This already exists, and as noted in the command sequence above,
works just fine. Please explain what problem are you seeing.

> 5) Task association to cos-id

Not sure what that means. Please explain.

>
> Can you please explain in a simple directory based scheme, like the one I gave
> you how all of these points are going to be solved with a modification to (*1)?
>
> Thanks,
>
> tglx

Thanks Thomas, this style discussion is quite useful.



\
 
 \ /
  Last update: 2016-01-04 19:01    [W:0.120 / U:0.420 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site