lkml.org 
[lkml]   [2021]   [Jan]   [28]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [PATCH v9 01/17] Documentation: PCI: Add specification for the *PCI NTB* function device
    From
    Date
    On 1/4/21 7:28 AM, Kishon Vijay Abraham I wrote:
    > Add specification for the *PCI NTB* function device. The endpoint function
    > driver and the host PCI driver should be created based on this
    > specification.
    >
    > Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com>
    > ---
    > Documentation/PCI/endpoint/index.rst | 1 +
    > .../PCI/endpoint/pci-ntb-function.rst | 351 ++++++++++++++++++
    > 2 files changed, 352 insertions(+)
    > create mode 100644 Documentation/PCI/endpoint/pci-ntb-function.rst


    > diff --git a/Documentation/PCI/endpoint/pci-ntb-function.rst b/Documentation/PCI/endpoint/pci-ntb-function.rst
    > new file mode 100644
    > index 000000000000..a57908be4047
    > --- /dev/null
    > +++ b/Documentation/PCI/endpoint/pci-ntb-function.rst
    > @@ -0,0 +1,351 @@
    > +.. SPDX-License-Identifier: GPL-2.0
    > +
    > +=================
    > +PCI NTB Function
    > +=================
    > +
    > +:Author: Kishon Vijay Abraham I <kishon@ti.com>
    > +
    > +PCI Non Transparent Bridges (NTB) allow two host systems to communicate

    preferably
    Non-Transparent

    > +with each other by exposing each host as a device to the other host.
    > +NTBs typically support the ability to generate interrupts on the remote
    > +machine, expose memory ranges as BARs and perform DMA. They also support
    > +scratchpads which are areas of memory within the NTB that are accessible
    > +from both machines.
    > +
    > +PCI NTB Function allows two different systems (or hosts) to communicate
    > +with each other by configurig the endpoint instances in such a way that
    > +transactions from one system is routed to the other system.
    > +
    > +In the below diagram, PCI NTB function configures the SoC with multiple
    > +PCIe Endpoint (EP) instances in such a way that transaction from one EP
    > +controller is routed to the other EP controller. Once PCI NTB function
    > +configures the SoC with multiple EP instances, HOST1 and HOST2 can
    > +communicate with each other using SoC as a bridge.
    > +
    > +.. code-block:: text
    > +
    > + +-------------+ +-------------+
    > + | | | |
    > + | HOST1 | | HOST2 |
    > + | | | |
    > + +------^------+ +------^------+
    > + | |
    > + | |
    > + +---------|-------------------------------------------------|---------+
    > + | +------v------+ +------v------+ |
    > + | | | | | |
    > + | | EP | | EP | |
    > + | | CONTROLLER1 | | CONTROLLER2 | |
    > + | | <-----------------------------------> | |
    > + | | | | | |
    > + | | | | | |
    > + | | | SoC With Multiple EP Instances | | |
    > + | | | (Configured using NTB Function) | | |
    > + | +-------------+ +-------------+ |
    > + +---------------------------------------------------------------------+
    > +
    > +Constructs used for Implementing NTB
    > +====================================
    > +
    > + 1) Config Region
    > + 2) Self Scratchpad Registers
    > + 3) Peer Scratchpad Registers
    > + 4) Doorbell Registers
    > + 5) Memory Window
    > +
    > +
    > +Config Region:
    > +--------------
    > +
    > +Config Region is a construct that is specific to NTB implemented using NTB
    > +Endpoint Function Driver. The host and endpoint side NTB function driver will
    > +exchange information with each other using this region. Config Region has
    > +Control/Status Registers for configuring the Endpoint Controller. Host can
    > +write into this region for configuring the outbound ATU and to indicate the

    what is ATU?

    > +link status. Endpoint can indicate the status of commands issued be host in

    by ??

    > +this region. Endpoint can also indicate the scratchpad offset, number of
    > +memory windows to the host using this region.
    > +
    > +The format of Config Region is given below. Each of the fields here are 32

    Each is

    > +bits.
    > +
    > +.. code-block:: text
    > +
    > + +------------------------+
    > + | COMMAND |
    > + +------------------------+
    > + | ARGUMENT |
    > + +------------------------+
    > + | STATUS |
    > + +------------------------+
    > + | TOPOLOGY |
    > + +------------------------+
    > + | ADDRESS (LOWER 32) |
    > + +------------------------+
    > + | ADDRESS (UPPER 32) |
    > + +------------------------+
    > + | SIZE |
    > + +------------------------+
    > + | NO OF MEMORY WINDOW |
    > + +------------------------+
    > + | MEMORY WINDOW1 OFFSET |
    > + +------------------------+
    > + | SPAD OFFSET |
    > + +------------------------+
    > + | SPAD COUNT |
    > + +------------------------+
    > + | DB ENTRY SIZE |
    > + +------------------------+
    > + | DB DATA |
    > + +------------------------+
    > + | : |
    > + +------------------------+
    > + | : |
    > + +------------------------+
    > + | DB DATA |
    > + +------------------------+
    > +
    > +
    > + COMMAND:
    > +
    > + NTB function supports three commands:
    > +
    > + CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
    > + invoking this command, the host should allocate and initialize
    > + MSI/MSI-X vectors (i.e initialize the MSI/MSI-X capability in the

    i.e.

    > + Endpoint). The endpoint on receiving this command will configure
    > + the outbound ATU such that transaction to DB BAR will be routed
    > + to the MSI/MSI-X address programmed by the host. The ARGUMENT
    > + register should be populated with number of DBs to configure (in the
    > + lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
    > + (TODO: Add support for MSI-X).
    > +
    > + CMD_CONFIGURE_MW (0x2): Command to configure memory window. The
    > + host invokes this command after allocating a buffer that can be
    > + accessed by remote host. The allocated address should be programmed
    > + in the ADDRESS register (64 bit), the size should be programmed in
    > + the SIZE register and the memory window index should be programmed
    > + in the ARGUMENT register. The endpoint on receiving this command
    > + will configure the outbound ATU such that trasaction to MW BAR

    transaction

    > + will be routed to the address provided by the host.
    > +
    > + CMD_LINK_UP (0x3): Command to indicate an NTB application is
    > + bound to the EP device on the host side. Once the endpoint
    > + receives this command from both the hosts, the endpoint will
    > + raise an LINK_UP event to both the hosts to indicate the hosts
    > + can start communicating with each other.
    > +
    > + ARGUMENT:
    > +
    > + The value of this register is based on the commands issued in
    > + command register. See COMMAND section for more information.
    > +
    > + TOPOLOGY:
    > +
    > + Set to NTB_TOPO_B2B_USD for Primary interface
    > + Set to NTB_TOPO_B2B_DSD for Secondary interface
    > +
    > + ADDRESS/SIZE:
    > +
    > + Address and Size to be used while configuring the memory window.
    > + See "CMD_CONFIGURE_MW" for more info.
    > +
    > + MEMORY WINDOW1 OFFSET:
    > +
    > + Memory Window 1 and Doorbell registers are packed together in the
    > + same BAR. The initial portion of the region will have doorbell
    > + registers and the latter portion of the region is for memory window 1.
    > + This register will specify the offset of the memory window 1.
    > +
    > + NO OF MEMORY WINDOW:
    > +
    > + Specifies the number of memory windows supported by the NTB device.
    > +
    > + SPAD OFFSET:
    > +
    > + Self scratchpad region and config region are packed together in the
    > + same BAR. The initial portion of the region will have config region
    > + and the latter portion of the region is for self scratchpad. This
    > + register will specify the offset of the self scratchpad registers.
    > +
    > + SPAD COUNT:
    > +
    > + Specifies the number of scratchpad registers supported by the NTB
    > + device.
    > +
    > + DB ENTRY SIZE:
    > +
    > + Used to determine the offset within the DB BAR that should be written
    > + in order to raise doorbell. EPF NTB can use either MSI/MSI-X to
    > + ring doorbell (MSI-X support will be added later). MSI uses same
    > + address for all the interrupts and MSI-X can provide different
    > + addresses for different interrupts. The MSI/MSI-X address is provided
    > + by the host and the address it gives is based on the MSI/MSI-X
    > + implementation supported by the host. For instance, ARM platform
    > + using GIC ITS will have same MSI-X address for all the interrupts.
    > + In order to support all the combinations and use the same mechanism
    > + for both MSI and MSI-X, EPF NTB allocates separate region in the
    > + Outbound Address Space for each of the interrupts. This region will
    > + be mapped to the MSI/MSI-X address provided by the host. If a host
    > + provides the same address for all the interrupts, all the regions
    > + will be translated to the same address. If a host provides different
    > + address, the regions will be translated to different address. This
    > + will ensure there is no difference while raising the doorbell.
    > +
    > + DB DATA:
    > +
    > + EPF NTB supports 32 interrupts. So there are 32 DB DATA registers.
    > + This holds the MSI/MSI-X data that has to be written to MSI address
    > + for raising doorbell interrupt. This will be populated by EPF NTB
    > + while invoking CMD_CONFIGURE_DOORBELL.
    > +
    > +Scratchpad Registers:
    > +---------------------
    > +
    > + Each host has it's own register space allocated in the memory of NTB EPC.

    its
    [it's means "it is"]

    > + They are both readable and writable from both sides of the bridge. They
    > + are used by applications built over NTB and can be used to pass control
    > + and status information between both sides of a device.
    > +
    > + Scratchpad registers has 2 parts
    > + 1) Self Scratchpad: Host's own register space
    > + 2) Peer Scratchpad: Remote host's register space.
    > +
    > +Doorbell Registers:
    > +-------------------
    > +
    > + Registers using which one host can interrupt the other host.

    eh? ENOPARSE.

    > +
    > +Memory Window:
    > +--------------
    > +
    > + Actual transfer of data between the two hosts will happen using the
    > + memory window.
    > +
    > +Modeling Constructs:
    > +====================
    > +
    > +There are 5 or more distinct regions (config, self scratchpad, peer
    > +scratchpad, doorbell, one or more memory windows) to be modeled to achieve
    > +NTB functionality. Atleast one memory window is required while more than

    At least

    > +one is permitted. All these regions should be mapped to BAR for hosts to
    > +access these regions.
    > +
    > +If one 32-bit BAR is allocated for each of these regions, the scheme would
    > +look like
    > +
    > +====== ===============
    > +BAR NO CONSTRUCTS USED
    > +====== ===============
    > +BAR0 Config Region
    > +BAR1 Self Scratchpad
    > +BAR2 Peer Scratchpad
    > +BAR3 Doorbell
    > +BAR4 Memory Window 1
    > +BAR5 Memory Window 2
    > +====== ===============
    > +
    > +However if we allocate a separate BAR for each of the region, there would not

    regions,

    > +be enough BARs for all the regions in a platform that supports only 64-bit
    > +BAR.
    > +
    > +In order to be supported by most of the platforms, the regions should be
    > +packed and mapped to BARs in a way that provides NTB functionality and
    > +also making sure the hosts doesn't access any region that it is not supposed
    > +to.
    > +
    > +The following scheme is used in EPF NTB Function
    > +
    > +====== ===============================
    > +BAR NO CONSTRUCTS USED
    > +====== ===============================
    > +BAR0 Config Region + Self Scratchpad
    > +BAR1 Peer Scratchpad
    > +BAR2 Doorbell + Memory Window 1
    > +BAR3 Memory Window 2
    > +BAR4 Memory Window 3
    > +BAR5 Memory Window 4
    > +====== ===============================
    > +
    > +With this scheme, for the basic NTB functionality 3 BARs should be sufficient.
    > +
    > +Modeling Config/Scratchpad Region:
    > +----------------------------------
    > +
    > +.. code-block:: text
    > +
    > + +-----------------+------->+------------------+ +-----------------+
    > + | BAR0 | | CONFIG REGION | | BAR0 |
    > + +-----------------+----+ +------------------+<-------+-----------------+
    > + | BAR1 | | |SCRATCHPAD REGION | | BAR1 |
    > + +-----------------+ +-->+------------------+<-------+-----------------+
    > + | BAR2 | Local Memory | BAR2 |
    > + +-----------------+ +-----------------+
    > + | BAR3 | | BAR3 |
    > + +-----------------+ +-----------------+
    > + | BAR4 | | BAR4 |
    > + +-----------------+ +-----------------+
    > + | BAR5 | | BAR5 |
    > + +-----------------+ +-----------------+
    > + EP CONTROLLER 1 EP CONTROLLER 2
    > +
    > +Above diagram shows Config region + Scratchpad region for HOST1 (connected to
    > +EP controller 1) allocated in local memory. The HOST1 can access the config
    > +region and scratchpad region (self scratchpad) using BAR0 of EP controller 1.
    > +The peer host (HOST2 connected to EP controller 2) can also access this
    > +scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This
    > +diagram shows the case where Config region and Scratchpad region is allocated
    > +for HOST1, however the same is applicable for HOST2.
    > +
    > +Modeling Doorbell/Memory Window 1:
    > +----------------------------------
    > +
    > +.. code-block:: text
    > +
    > + +-----------------+ +----->+----------------+-----------+-----------------+
    > + | BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 |
    > + +-----------------+ | +----------------+ +-----------------+
    > + | BAR1 | | | Doorbell 2 +---------+ | |
    > + +-----------------+----+ +----------------+ | | |
    > + | BAR2 | | Doorbell 3 +-------+ | +-----------------+
    > + +-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 |
    > + | BAR3 | | | Doorbell 4 +-----+ | +-----------------+
    > + +-----------------+ | |----------------+ | | | |
    > + | BAR4 | | | | | | +-----------------+
    > + +-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3||
    > + | BAR5 | | | | | | +-----------------+
    > + +-----------------+ +----->-----------------+ | | | |
    > + EP CONTROLLER 1 | | | | +-----------------+
    > + | | | +---->+ MSI-X ADDRESS 4 |
    > + +----------------+ | +-----------------+
    > + EP CONTROLLER 2 | | |
    > + (OB SPACE) | | |
    > + +-------> MW1 |
    > + | |
    > + | |
    > + +-----------------+
    > + | |
    > + | |
    > + | |
    > + | |
    > + | |
    > + +-----------------+
    > + PCI Address Space
    > + (Managed by HOST2)
    > +
    > +Above diagram shows how the doorbell and memory window 1 is mapped so that
    > +HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access
    > +buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and
    > +memory window 1 regions are allocated in EP controller 2 outbound (OB) address
    > +space. Allocating and configuring BARs for doorbell and memory window1
    > +is done during the initialization phase of NTB endpoint function driver.
    > +Mapping from EP controller 2 OB space to PCI address space is done when HOST2
    > +sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL. The commands are explained
    > +below.

    below??

    > +
    > +Modeling Optional Memory Windows:
    > +---------------------------------
    > +
    > +This is modeled the same was as MW1 but each of the additional memory windows
    > +is mapped to separate BARs.
    >

    Is all of this register/memory space mapping defined in some PCI NTB spec
    or is this specific to some hardware/SoC implementation?


    HTH.
    --
    ~Randy

    \
     
     \ /
      Last update: 2021-01-29 02:11    [W:4.937 / U:0.748 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site