lkml.org 
[lkml]   [2017]   [Jul]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: [PATCH v2 01/22] docs: fpga: add a document for Intel FPGA driver overview
On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

> Add a document for Intel FPGA driver overview.
>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
> updated open discussion per current implementation.
> fixed some typos.
> ---
> Documentation/fpga/intel-fpga.txt | 256 ++++++++++++++++++++++++++++++++++++++
> 1 file changed, 256 insertions(+)
> create mode 100644 Documentation/fpga/intel-fpga.txt
>
> diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> new file mode 100644
> index 0000000..4a29470
> --- /dev/null
> +++ b/Documentation/fpga/intel-fpga.txt
> @@ -0,0 +1,256 @@
> +===============================================================================
> + Intel FPGA driver Overview
> +-------------------------------------------------------------------------------
> + Enno Luebbers <enno.luebbers@intel.com>
> + Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + Wu Hao <hao.wu@intel.com>
> +
> +The Intel FPGA driver provides interfaces for userspace applications to
> +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> +with Intel(R) FPGA PCIe based solutions and enables system level management
> +functions such as FPGA reconfiguration, power management, and virtualization.
> +
> +HW Architecture
> +===============
> +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> +The FPGA device memory is organized using a predefined data structure (Device
> +Feature List). Features supported by the particular FPGA device are exposed
> +through these data structures, as illustrated below:
> +
> + +-------------------------------+ +-------------+
> + | PF | | VF |
> + +-------------------------------+ +-------------+
> + ^ ^ ^ ^
> + | | | |
> ++-----|------------|---------|--------------|-------+
> +| | | | | |
> +| +-----+ +-------+ +-------+ +-------+ |
> +| | FME | | Port0 | | Port1 | | Port2 | |
> +| +-----+ +-------+ +-------+ +-------+ |
> +| ^ ^ ^ |
> +| | | | |
> +| +-------+ +------+ +-------+ |
> +| | AFU | | AFU | | AFU | |
> +| +-------+ +------+ +-------+ |
> +| |
> +| FPGA PCIe Device |
> ++---------------------------------------------------+
> +
> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> +used to assign individual accelerators to virtual machines.
> +
> +FME (FPGA Management Engine)
> +============================
> +The FPGA Management Engine performs power and thermal management, error
> +reporting, reconfiguration, performance reporting, and other infrastructure
> +functions. Each FPGA has one FME, which is always accessed through the physical
> +function (PF).
> +
> +User-space applications can acquire exclusive access to the FME using open(),
> +and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> + Get driver API version (FPGA_GET_API_VERSION)
> + Check for extensions (FPGA_CHECK_EXTENSION)
> + Assign port to PF (FPGA_FME_PORT_ASSIGN)
> + Release port from PF (FPGA_FME_PORT_RELEASE)
> + Program bitstream (FPGA_FME_PORT_PR)
> +

I was hoping the API mailing list might have an opinion about this,
but I think adding ioctls to the kernel is discouraged. Could these
be sysfs?

> +More functions are exposed through sysfs
> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> +
> + Read bitstream ID (bitstream_id)
> + Read bitstream metadata (bitstream_metadata)
> + Read number of ports (ports_num)
> + Read socket ID (socket_id)
> + Read performance counters (perf/)
> + Power management (power_mgmt/)
> + Thermal management (thermal_mgmt/)
> + Error reporting (errors/)
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.
> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +
> +AFU
> +===
> +An AFU is attached to a port and exposes a 256k MMIO region to be used for
> +accelerator-specific control registers.
> +
> +User-space applications can acquire exclusive access to an AFU attached to a
> +port by using open() on the port device node, and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> + Get driver API version (FPGA_GET_API_VERSION)
> + Check for extensions (FPGA_CHECK_EXTENSION)
> + Get port info (FPGA_PORT_GET_INFO)
> + Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> + Map DMA buffer (FPGA_PORT_DMA_MAP)
> + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> + Reset AFU (FPGA_PORT_RESET)
> + Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> + Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> +
> +User-space applications can also mmap() accelerator MMIO regions.
> +
> +More functions are exposed through sysfs:
> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
> +
> + Read Accelerator GUID (afu_id)
> + Error reporting (errors/)
> +
> +Partial Reconfiguration
> +=======================
> +As mentioned above, accelerators can be reconfigured through partial
> +reconfiguration of a green bitstream file (GBS). The green bitstream must have
> +been generated for the exact blue bitstream and targeted reconfigurable region
> +(port) of the FPGA; otherwise, the reconfiguration operation will fail and
> +possibly cause system instability. This compatibility can be checked by
> +comparing the interface ID noted in the GBS header against the interface ID
> +exposed by the FME through sysfs (see above). This check is usually done by
> +user-space before calling the reconfiguration IOCTL.
> +
> +FPGA virtualization
> +===================
> +To enable accessing an accelerator from applications running in a VM, the
> +respective AFU's port needs to be assigned to a VF using the following steps:
> +
> + a) The PF owns all AFU ports by default. Any port that needs to be reassigned
> + to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE
> + ioctl on the FME device.
> +
> + b) Once N ports are released from PF, then user can use below command to
> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> +
> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> +
> + c) Pass through the VFs to VMs
> +
> + d) The AFU under VF is accessible from applications in VM (using the same
> + driver inside the VF).
> +
> +Note the an FME can't be assigned to a VF, thus PR and other management
> +functions are only available via the PF.
> +
> +
> +Driver organization
> +===================
> +
> + +-------++------++------+ |
> + | FME || FME || FME | |
> + | FPGA || FPGA || FPGA | |
> + |Manager||Bridge||Region| |
> + +-------++------++------+ |
> + +-----------------------+ +--------+ | +--------+
> + | FME | | AFU | | | AFU |
> + | Module | | Module | | | Module |
> + +-----------------------+ +--------+ | +--------+
> + +-----------------------+ | +-----------------------+
> + | FPGA Container Device | | | FPGA Container Device |
> + +-----------------------+ | +-----------------------+
> + +------------------+ | +------------------+
> + | FPGA PCIE Module | | Virtual | FPGA PCIE Module |
> + +------------------+ Host | Machine +------------------+
> + -------------------------------------- | ------------------------------
> + +---------------+ | +---------------+
> + | PCI PF Device | | | PCI VF Device |
> + +---------------+ | +---------------+
> +
> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
> +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
> +driver plays an infrastructural role in the driver architecture. It:
> +
> + a) creates FPGA container device as parent of the feature devices.
> + b) walks through the Device Feature List, which is implemented in PCIE
> + device BAR memory, to discover feature devices and their sub features
> + and create platform device for them under the container device.
> + c) supports SRIOV.
> + d) introduces the feature device infrastructure, which abstracts
> + operations for sub features and exposes common functions to feature
> + device drivers.
> +
> +The FPGA Management Engine (FME) driver is a platform driver which is loaded
> +automatically after FME platform device creation from the PCIE driver. It
> +provides the key features for FPGA management, including:
> +
> + a) Power and thermal management, error reporting, performance reporting
> + and other infrastructure functions. Users can access these functions
> + via sysfs interfaces exposed by FME driver.
> + b) Partial Reconfiguration. The FME driver creates platform devices
> + for FPGA manager, FPGA bridges and FPGA regions during PR sub
> + feature initialization; Once it receives an FPGA_FME_PORT_PR ioctl
> + from user, it invokes the common interface function from FPGA Region
> + to complete the partial reconfiguration of the bitstream to the given
> + port.
> + c) Port management for virtualization. The FME driver introduces two
> + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
> + released from the PF, it can be assigned to the VF through the SRIOV
> + interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
> + for more details).
> +
> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
> +is probed once the AFU platform device is created. The main function of this
> +module is to provide an interface for userspace applications to access the
> +individual accelerators, including basic reset control on port, AFU MMIO region
> +export, dma buffer mapping service, UMsg notification, and remote debug
> +functions (see above).
> +
> +
> +Device enumeration
> +==================
> +This section introduces how applications enumerate the fpga device from
> +the sysfs hierarchy under /sys/class/fpga.
> +
> +In the example below, two Intel(R) FPGA devices are installed in the host. Each
> +fpga device has one FME and two ports (AFUs).
> +
> +For each FPGA device, a device director is created under /sys/class/fpga/:
> +
> + /sys/class/fpga/fpga.0
> + /sys/class/fpga/fpga.1
> +
> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name.
> +Application can retrieve name information via the sysfs interface:
> +
> + /sys/class/fpga/fpga.0/name
> +
> +Each node has one FME and two ports (AFUs) as child devices:
> +
> + /sys/class/fpga/fpga.0/intel-fpga-fme.0
> + /sys/class/fpga/fpga.0/intel-fpga-port.0
> + /sys/class/fpga/fpga.0/intel-fpga-port.1
> +
> + /sys/class/fpga/fpga.1/intel-fpga-fme.1
> + /sys/class/fpga/fpga.1/intel-fpga-port.2
> + /sys/class/fpga/fpga.1/intel-fpga-port.3
> +
> +In general, the FME/AFU sysfs interfaces are named as follows:
> +
> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
> +
> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
> +ports.
> +
> +The device nodes used for ioctl() or mmap() can be referenced through:
> +
> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
> +
> +Open discussion
> +===============
> +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
> +user now. In the future, if unified user interfaces for reconfiguration are
> +added, FME driver should switch to them from ioctl interface.

Adding an ioctl that will go away in the future?

I think we are going to have to find a different interface for doing
partial reconfiguration. There was a recent conversation on the
mailing list regarding adding a header to the raw bitstream and
probably using sysfs to present that to the kernel. I'm working on
getting an RFC together for that.

Alan Tull

\
 
 \ /
  Last update: 2017-07-12 16:52    [W:0.514 / U:0.260 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site