On the FPGA side of SNAP diagram, there are three parts that need to consider when moving to a new FPGA card. They are (a) PSL, (b) PSL/AXI bridge (snap_core), (c) DDR memory controller (mig). And there are also some components in SNAP need to be updated for a new FPGA card. The following sections introduced the the structure of SNAP folders and scripts and the steps.

Firstly, clone the repository: git clone https://github.com/open-power/snap git submodule init git submodule update

All of the user-developed accelerators should be put in "actions" directory. There are already some examples there. Each "action" has its "sw", "hw", "tests", and other sub-directories. Then back to ${SNAP_ROOT}, "software" directory includes libsnap, header files and some tools. "hardware" directory is the main focus. deconfig has the config files for silent testing purpose, and scripts has the menu settings and other scripts. How does SNAP work and what are the files used in each step? make snap_config: The menu to select cards and other options is controlled by "script/Kconfig" make model: This step creates a Vivado project. It firstly calls "hardware/setup/create_snap_ip.tcl" to generate the IP files in use, then calls "hardware/setup/create_framework.tcl" to build the project. About create_framework.tcl: It adds BSP (board support package). In CAPI1.0, it is also called PSL Checkpoint file (b_route_design.dcp) or base_image. It uses the path pointed to b_route_design.dcp and adds it into the design. In CAPI2.0, it will call the make process in capi2-bsp submodule. Submodule "capi2-bsp" reads the encrypted PSL source files, adds PCIe and Flash logic, packs them into capi2_bsp_wrap.xcix (IP container file). Then "create_framework.tcl" adds the capi2_bsp_wrap.xcix into the design. It adds FPGA top files and snap_core files (in hardware/hdl/core). It adds constrain files: in hardware/setup/${FPGACARD} or in hardware/capi2-bsp/${FPGACARD} It adds user files (in actions/${ACTION_NAME}/hw). User's action hardware uses top file named "action_wrapper.vhd" It adds simulation files (in hardware/sim/core) including simulation top files and simulation models. (If "no_sim" is selected in snap_config menu, this step is skipped.) After above steps, "viv_project" is created. You can open it with Vivado GUI, and check the design hierarchy. And it will call the selected simulator to compile the simulation model. make image: This step runs synthesis, implementation and bitstream generation. It calls "hardware/setup/snap_build.tcl" and also uses some related tcl scripts to work on "viv_project". In this step, "hardware/build" will be created and the output products like bit images, checkpoints (middle products for debugging) and reports (reports of timing, clock, IO, utilization, etc.) If everything runs well and timing passes, user will get the bitstream files (in "Images" sub directory) to program the FPGA card.

For CAPI1.0, base_image contains surrounding logic and the kernel logic: PCIe hard IP core (pcie3_ultrascale_0) Flash Controller (psl_flash) VSEC: Vendor Specific Extended Capability (psl_vsec) Xilinx MultiBoot control logic (psl_xilmltbt) PSL kernel logic (psl) The interface between base_image and AFU(psl_accel) has 5 groups of signals, described in PSL spec CAPI1.0 PSL/AFU interface Spec. The interface between base_image and Chip IOs are card specific, and the information need to be provided by Card Vendor. Generally, they include: Flash interface (usually DPIx16) PCIe interface: perst, refclk, TX and RX data lanes Peripheral IPs: I2C, LED, DDR, Ethernet, etc. Marked in light orange color, you can download the entire base_image (b_route_design.dcp) from OpenPower Portal.

For CAPI2.0, the structure is similar, but the PSL9 logic (marked in light orange color) is provided as an encrypted Zip package. It can be downloaded from OpenPower Portal and put in "capi2-bsp/psl" directory. Then it uses the make process in capi2-bsp to generate an IP container file (capi_bsp_wrap.xcix). Please refer to the README file at https://github.com/open-power/capi2-bsp for more details. CAPI2.0 cards are using SPI Flash interface: SPIx4 or dual SPIx4 (also mentioned as SPIx8). For PCIe Gen3, it uses 16 lanes. For PCIe Gen4, it uses 8 lanes. The interface of PSL9 has 6 groups of signals. Please refer to CAPI2.0 PSL/AFU interface Spec for the details. The logic in snap_core (CAPI2.0) implements the data path with DMA interface. Buffer interface is not used. The above two figures apply to both HDK development and SNAP framework. The difference is, for HDK developers, they work on the AFU by themselves. For SNAP developers, they make use of the snap_core logic and only work on action_wrapper. The AFU part for SNAP developers contains following blocks:

AFU logic RTL files are open-sourced. Developer can make modifications for their own purpose, like adding multiple DDR channels, adding NVMe and Ethernet controllers.

For a new FPGA card, the detailed items to update are: Preparations Hardware RTL, setup, simulation Software and tools Testing Publishing

First, give a FPGACARD name. It should start from the company's name, following with the card ID and be short. For example. ADKU3 = Alpha-Data ADM-PCIE-KU3. Get follow information from the card vendor. Item Description FPGACARD Short card name used in SNAP FPGACHIP FPGA part name, for example, xcvu9p-fsgd2104-2L-e Flash Type Flash chip that attached to FPGA, for example mt28gu01gaax1e-bpi-x16. And the related xdc files for FPGA config. DDR MC IP Short card name used in SNAP FPGACARD DDR memory controller Vivado IP tcl/xdc file. Other peripherals NVMe IP, Ethernet IP and so on (Optional) IO pins PACKAGE_PIN for base_image or bsp: flash, pcie, i2c etc. PACKAGE_PIN for peripheral IPs.

The best way is to grep some keywords like "S241" or "AD8K5" under the directories and look for the locations that need modifications. If you meet files ending with "_source", like "psl_fpga.vhd_source", that means this file will be pre-processed to generate the output file without "_source" suffix, like "psl_fpga.vhd". There are #ifdef macros or comments like -- only for NVME_USED=TRUE. They help to create a target VHDL/Verilog file with different configurations. Below lists the files to change. There may be some differences with new commits in SNAP git repository. Keep in mind they include: snap_config and environmental files Hardware: psl_accel and psl_fpga (top) RTL files Hardware: tcl files for the workflow Hardware: Board: xdc files for IO/floorplan/clock/bitstream Hardware: DDR: create_ip, sim model, xdc files Hardware: Other IP: create_ip, sim model, xdc files Software: New card type, register definition Testing: jenkins Readme and Documents For CAPI1.0, you need to generate a new PSL checkpoint file and upload it to OpenPower Portal. Section describes the details. For CAPI2.0, you need to add a ${FPGACARD} directory in capi2-bsp git repository. Copy an existing folder as a start and follow the README file. Make sure the information in xdc/tcl files are permitted to be open-source. Send email to OpenPower Acceleration Workgroup or contact your representative to apply for a subsystem device ID for the new card. For example, ADKU3 uses 0x0605. S241 uses 0x0660. You also need to update https://github.com/ibm-capi/capi-utils to allow capi-flash-script to program this new card. Subsystem ID will be used there. It is also used in snap/software/tools/snap_find_card. File name Changes done scripts/Kconfig adding card to the Kconfig menu. Provide Flash information (size/type/user address) hardware/doc/SNAP-Registers.md SNAP registers for new card - doc hardware/setup/snap_config.sh SNAP registers - setting File name Changes done hardware/hdl/core/psl_accel_${FPGACARD}.vhd_source specific to cardhardware/hdl/core/psl_accel_types.vhd_sourcespecific to cardhardware/hdl/core/psl_fpga_${FPGACARD}.vhd_source specific to cardhardware/setup/${FPGACARD}/capi_bsp_pblock.xdc specific to cardhardware/setup/${FPGACARD}/snap_${FPGACARD}.xdc specific to cardhardware/setup/${FPGACARD}/snap_ddr4pins.xdc specific to cardhardware/setup/build_mcs.tcldeclare card namehardware/setup/create_framework.tcldeclare card namehardware/setup/create_snap_ip.tcldeclare card name and its IPhardware/setup/flash_mcs.tcldeclare card namehardware/setup/snap_bitstream_post.tcldeclare card namehardware/setup/snap_bitstream_pre.tcldeclare card namehardware/setup/snap_bitstream_step.tcldeclare card namehardware/setup/snap_impl_step.tcldeclare card namehardware/snap_check_psldeclare card name File name Changes done software/lib/snap.cdeclare card namesoftware/tools/snap_find_carddeclare card name + idsoftware/include/snap_regs.hSNAP registers - setting File name Changes done actions/scripts/snap_jenkins.shjenkins tests (optional)defconfig/{FPGACARD}*.defconfigFor silent jenkins testing (optional)README.mdAnnounce a new card is supported

To enable a new card on SNAP, please take following tasks one by one.

Make modifications to snap git repository (and capi2-bsp) as described above. Select an action example without DDR, for example: hls_helloworld. Go through the "make model" and "make image" processes and get the bitstream files. Plug the card onto Power8/Power9 server and power on. Use Jtag to program the generated bitstream files (bin or mcs) to the card. You need a laptop or workstation installed Vivado Lab Edition, and connect a JTAG/USB cable to the card. Open Hardware Manager, open target, select the FPGA chip and right-click, choose "Add Configuration Memory Device..." and program the bitstream files. See in picture and Wait it done, unplug the JTAG/USB cable, reboot the server. When the server is booted, install snap, capi-utils, libcxl. Run lspci to see if the card is there. (Usually with ID 0x0477). Then go to snap directory, make apps and run the application.

When you download and install Vivado Lab Edition, please pick up as same version as the Vivado (SDx) that you are using to build images.

Use capi-utils to program the bitstream files. If it succeeds, it proves that the Flash interface has been configured correctly.

Select another action example (hdl_example with DDR) or hls_memcopy. "make model" and "make sim". Make sure the DDR simulation model works well. "make image" to generate the bitstream files. Use capi-utils to program the bitstream files to the card. Run the application to see if it works.

This step is decided by the card vendor and the specific IOs that the card provide.

You can check the result of "snap/actions/hls_memcopy/tests/test_*_throughput.sh" for bandwidth and "snap/actions/hls_latency_eval/test/test*.sh" for latency.

Prepare bitstream files for basic tests, throughput tests, latency tests, max-power tests. Adding image flashing tests, card reset tests and others. Run them intensively.