lkml.org 
[lkml]   [2017]   [Dec]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: [PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview
On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> Add a document for Intel FPGA driver overview.
>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
> updated open discussion per current implementation.
> fixed some typos.
> v3: use FPGA base region as container device instead of fpga-dev class.
> split common enumeration code from pcie driver to functions exposed by
> device feature list framework.
> update FME performance reporting which supports both integrated (iperf/)
> and discrete (dperf/) FPGA solutions.
> ---
> Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++
> 1 file changed, 261 insertions(+)
> create mode 100644 Documentation/fpga/intel-fpga.txt
>
> diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> new file mode 100644
> index 0000000..0754733
> --- /dev/null
> +++ b/Documentation/fpga/intel-fpga.txt
> @@ -0,0 +1,261 @@
> +===============================================================================
> + Intel FPGA driver Overview

This doesn't look Intel specific to me. This could all be 'DFL FPGA Framework'

> +-------------------------------------------------------------------------------
> + Enno Luebbers <enno.luebbers@intel.com>
> + Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + Wu Hao <hao.wu@intel.com>
> +
> +The Intel FPGA driver provides interfaces for userspace applications to
> +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> +with Intel(R) FPGA PCIe based solutions and enables system level management
> +functions such as FPGA reconfiguration, power management, and virtualization.
> +
> +HW Architecture
> +===============
> +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> +The FPGA device memory is organized using a predefined data structure (Device
> +Feature List). Features supported by the particular FPGA device are exposed
> +through these data structures, as illustrated below:
> +
> + +-------------------------------+ +-------------+
> + | PF | | VF |
> + +-------------------------------+ +-------------+
> + ^ ^ ^ ^
> + | | | |
> ++-----|------------|---------|--------------|-------+
> +| | | | | |
> +| +-----+ +-------+ +-------+ +-------+ |
> +| | FME | | Port0 | | Port1 | | Port2 | |
> +| +-----+ +-------+ +-------+ +-------+ |
> +| ^ ^ ^ |
> +| | | | |
> +| +-------+ +------+ +-------+ |
> +| | AFU | | AFU | | AFU | |
> +| +-------+ +------+ +-------+ |
> +| |
> +| FPGA PCIe Device |
> ++---------------------------------------------------+
> +
> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> +used to assign individual accelerators to virtual machines.
> +
> +FME (FPGA Management Engine)
> +============================
> +The FPGA Management Engine performs power and thermal management, error
> +reporting, reconfiguration, performance reporting for integrated and discrete
> +solution, and other infrastructure functions. Each FPGA has one FME, which is
> +always accessed through the physical function (PF).
> +
> +User-space applications can acquire exclusive access to the FME using open(),
> +and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> + Get driver API version (FPGA_GET_API_VERSION)
> + Check for extensions (FPGA_CHECK_EXTENSION)
> + Assign port to PF (FPGA_FME_PORT_ASSIGN)
> + Release port from PF (FPGA_FME_PORT_RELEASE)
> + Program bitstream (FPGA_FME_PORT_PR)
> +
> +More functions are exposed through sysfs
> +(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/):

I see that /sys/class/fpga/* has changed to /sys/class/fpga_region/*
now as requested (thanks!). It looks like it ended up being pretty
straightforward (so far, just diffing this doc with the previous v2).

> +
> + Read bitstream ID (bitstream_id)
> + Read bitstream metadata (bitstream_metadata)
> + Read number of ports (ports_num)
> + Read socket ID (socket_id)
> + Read performance counters (iperf/ and dperf/)
> + Power management (power_mgmt/)
> + Thermal management (thermal_mgmt/)
> + Error reporting (errors/)
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.
> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +
> +AFU
> +===
> +An AFU is attached to a port and exposes a 256k MMIO region to be used for
> +accelerator-specific control registers.
> +
> +User-space applications can acquire exclusive access to an AFU attached to a
> +port by using open() on the port device node, and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> + Get driver API version (FPGA_GET_API_VERSION)
> + Check for extensions (FPGA_CHECK_EXTENSION)
> + Get port info (FPGA_PORT_GET_INFO)
> + Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> + Map DMA buffer (FPGA_PORT_DMA_MAP)
> + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> + Reset AFU (FPGA_PORT_RESET)
> + Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> + Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> +
> +User-space applications can also mmap() accelerator MMIO regions.
> +
> +More functions are exposed through sysfs:
> +(/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/):
> +
> + Read Accelerator GUID (afu_id)
> + Error reporting (errors/)
> +
> +Partial Reconfiguration
> +=======================
> +As mentioned above, accelerators can be reconfigured through partial
> +reconfiguration of a green bitstream file (GBS). The green bitstream must have
> +been generated for the exact blue bitstream and targeted reconfigurable region
> +(port) of the FPGA; otherwise, the reconfiguration operation will fail and
> +possibly cause system instability. This compatibility can be checked by
> +comparing the interface ID noted in the GBS header against the interface ID
> +exposed by the FME through sysfs (see above). This check is usually done by
> +user-space before calling the reconfiguration IOCTL.
> +
> +FPGA virtualization
> +===================
> +To enable accessing an accelerator from applications running in a VM, the
> +respective AFU's port needs to be assigned to a VF using the following steps:
> +
> + a) The PF owns all AFU ports by default. Any port that needs to be reassigned
> + to a VF must first be released through the FPGA_FME_PORT_RELEASE ioctl on the
> + FME device.
> +
> + b) Once N ports are released from PF, then user can use command below to
> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> +
> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> +
> + c) Pass through the VFs to VMs
> +
> + d) The AFU under VF is accessible from applications in VM (using the same
> + driver inside the VF).
> +
> +Note that an FME can't be assigned to a VF, thus PR and other management
> +functions are only available via the PF.
> +
> +
> +Driver organization
> +===================
> +
> + +-------++------++------+ |
> + | FME || FME || FME | |
> + | FPGA || FPGA || FPGA | |
> + |Manager||Bridge||Region| |
> + +-------++------++------+ |
> + +-----------------------+ +--------+ | +--------+
> + | FME | | AFU | | | AFU |
> + | Module | | Module | | | Module |
> + +-----------------------+ +--------+ | +--------+
> + +-----------------------+ | +-----------------------+
> + | FPGA Container Device | | | FPGA Container Device |
> + | (FPGA Base Region) | | | (FPGA Base Region) |
> + +-----------------------+ | +-----------------------+
> + +------------------+ | +------------------+
> + | FPGA PCIE Module | | Virtual | FPGA PCIE Module |
> + +------------------+ Host | Machine +------------------+
> + -------------------------------------- | ------------------------------
> + +---------------+ | +---------------+
> + | PCI PF Device | | | PCI VF Device |
> + +---------------+ | +---------------+
> +
> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
> +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
> +driver plays an infrastructural role in the driver architecture. It:
> +
> + a) locates the Device Feature Lists in PCIE device BAR memory, handles
> + them and related resources to common interfaces from DFL framework
> + for enumeration.
> + b) supports SRIOV.
> +
> +The feature device infrastructure provides common interfaces to create container
> +device (FPGA base region), discover feature devices and their sub features from
> +the given Device Feature Lists, and create platform devices for feature devices
> +with related resources under the container device. It also abstracts operations
> +for sub features and exposes common interfaces to feature device drivers.
> +
> +The FPGA Management Engine (FME) driver is a platform driver which is loaded
> +automatically after FME platform device creation from the PCIE driver. It
> +provides the key features for FPGA management, including:
> +
> + a) Power and thermal management, error reporting, performance reporting
> + and other infrastructure functions. Users can access these functions
> + via sysfs interfaces exposed by FME driver.
> + b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
> + bridges and FPGA regions during PR sub feature initialization; Once
> + it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
> + common interface function from FPGA Region to complete the partial
> + reconfiguration of the bitstream to the given port.
> + c) Port management for virtualization. The FME driver introduces two
> + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
> + released from the PF, it can be assigned to the VF through the SRIOV
> + interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
> + for more details).
> +
> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
> +is probed once the AFU platform device is created. The main function of this
> +module is to provide an interface for userspace applications to access the
> +individual accelerators, including basic reset control on port, AFU MMIO region
> +export, dma buffer mapping service, UMsg notification, and remote debug
> +functions (see above).
> +
> +
> +Device enumeration
> +==================
> +This section introduces how applications enumerate the fpga device from
> +the sysfs hierarchy under /sys/class/fpga_region.
> +
> +In the example below, two Intel(R) FPGA devices are installed in the host. Each
> +fpga device has one FME and two ports (AFUs).
> +
> +FPGA regions are created under /sys/class/fpga_region/
> +
> + /sys/class/fpga_region/region0
> + /sys/class/fpga_region/region1
> + /sys/class/fpga_region/region2
> + ...
> +
> +Application needs to search each regionX folder, if feature device is found,
> +(e.g "fpga-dfl-port.n" or "fpga-dfl-fme.m" is found), then it's the base
> +fpga region which represents the FPGA device.
> +
> +Each base region has one FME and two ports (AFUs) as child devices:
> +
> + /sys/class/fpga_region/region0/fpga-dfl-fme.0
> + /sys/class/fpga_region/region0/fpga-dfl-port.0
> + /sys/class/fpga_region/region0/fpga-dfl-port.1
> + ...
> +
> + /sys/class/fpga_region/region3/fpga-dfl-fme.1
> + /sys/class/fpga_region/region3/fpga-dfl-port.2
> + /sys/class/fpga_region/region3/fpga-dfl-port.3
> + ...
> +
> +In general, the FME/AFU sysfs interfaces are named as follows:
> +
> + /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/
> + /sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/
> +
> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
> +ports.
> +
> +The device nodes used for ioctl() or mmap() can be referenced through:
> +
> + /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/dev
> + /sys/class/fpga_region/<regionX>/<fpga-dfl-port.n>/dev
> +
> +Open discussion
> +===============
> +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
> +user now. In the future, if unified user interfaces for reconfiguration are
> +added, FME driver should switch to them from ioctl interface.
> --
> 1.8.3.1
>

\
 
 \ /
  Last update: 2017-12-04 20:57    [W:0.364 / U:0.032 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site