diff --git a/enable_capi_snap/app_template.xml b/enable_capi_snap/app_template.xml
new file mode 100644
index 0000000..f4b7ae4
--- /dev/null
+++ b/enable_capi_snap/app_template.xml
@@ -0,0 +1,30 @@
+
+
+
+
+ Appendix template
+ This is the first paragraph of a new appendix...
+
+ Section title
+ Section text...
+
+
diff --git a/enable_capi_snap/bk_main.xml b/enable_capi_snap/bk_main.xml
new file mode 100644
index 0000000..74f7426
--- /dev/null
+++ b/enable_capi_snap/bk_main.xml
@@ -0,0 +1,113 @@
+
+
+
+
+
+]>
+
+
+
+
+ Enable FPGA for SNAP
+
+ For CAPI2.0 and CAPI1.0
+
+
+
+
+
+ Acceleration Workgroup
+
+ aclwg-chair@openpowerfoundation.org
+
+ OpenPower Foundation
+
+
+
+ 2018
+ OpenPOWER Foundation
+
+
+ Revision 1.0_pre1
+ OpenPOWER
+
+
+
+
+
+
+ Copyright details are filled in by the template.
+
+
+
+
+
+
+ The purpose of this document is to describe how to enable a new customer card on CAPI SNAP framework. SNAP is a open-sourced programming framework for FPGA Acclerations. Its homepage is https://github.com/open-power/snap. With it, you can develop accelerations with Power and CAPI technology easily.
+
+ This document describes when you get a PCIe FPGA card not listed in today's "SNAP enabled cards" (On the homepage README of SNAP github website), how do you get it enabled. Since all of the files are open-sourced, you can create a board support package (bsp) similar to the existing one and walk through the entire working flow with the help of this document.
+
+ This document is a Standard Track, Workgroup Specification work product owned by the Acceleration Workgroup and handled in compliance with the requirements outlined in the
+ OpenPOWER Foundation Work Group (WG) Process document. It was
+ created using the Master Template Guide version &template_version;.
+ Comments, questions, etc. can be submitted to the
+ public mailing list for the parent specification at
+ tbd@mailinglist.openpowerfoundation.org.
+
+
+
+
+
+ 2019-03-29
+
+
+
+ Start from the original Word document
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/enable_capi_snap/ch_enable_snap.xml b/enable_capi_snap/ch_enable_snap.xml
new file mode 100644
index 0000000..fe94044
--- /dev/null
+++ b/enable_capi_snap/ch_enable_snap.xml
@@ -0,0 +1,355 @@
+
+
+
+
+ Enable a FPGA card in SNAP
+ On the FPGA side of SNAP diagram, there are three parts that need to consider when moving to a new FPGA card. They are (a) PSL, (b) PSL/AXI bridge (snap_core), (c) DDR memory controller (mig). And there are also some components in SNAP need to be updated for a new FPGA card. The following sections introduced the the structure of SNAP folders and scripts and the steps.
+ SNAP structure
+ Firstly, clone the repository:
+
+ git clone https://github.com/open-power/snap
+ git submodule init
+ git submodule update
+
+
+
+ SNAP structure
+
+
+
+
+
+
+
+ All of the user-developed accelerators should be put in "actions" directory. There are already some examples there. Each "action" has its "sw", "hw", "tests", and other sub-directories.
+ Then back to ${SNAP_ROOT}, "software" directory includes libsnap, header files and some tools. "hardware" directory is the main focus. deconfig has the config files for silent testing purpose, and scripts has the menu settings and other scripts.
+
+ How does SNAP work and what are the files used in each step?
+
+
+ make snap_config: The menu to select cards and other options is controlled by "script/Kconfig"
+
+ make model: This step creates a Vivado project. It firstly calls "hardware/setup/create_snap_ip.tcl" to generate the IP files in use, then calls "hardware/setup/create_framework.tcl" to build the project. About create_framework.tcl:
+
+
+ It adds BSP (board support package). In CAPI1.0, it is also called PSL Checkpoint file (b_route_design.dcp) or base_image. It uses the path pointed to b_route_design.dcp and adds it into the design. In CAPI2.0, it will call the make process in capi2-bsp submodule. Submodule "capi2-bsp" reads the encrypted PSL source files, adds PCIe and Flash logic, packs them into capi2_bsp_wrap.xcix (IP container file). Then "create_framework.tcl" adds the capi2_bsp_wrap.xcix into the design.
+
+
+ It adds FPGA top files and snap_core files (in hardware/hdl/core).
+
+
+ It adds constrain files: in hardware/setup/${FPGACARD} or in hardware/capi2-bsp/${FPGACARD}
+
+
+ It adds user files (in actions/${ACTION_NAME}/hw). User's action hardware uses top file named "action_wrapper.vhd"
+
+
+ It adds simulation files (in hardware/sim/core) including simulation top files and simulation models. (If "no_sim" is selected in snap_config menu, this step is skipped.)
+
+
+ After above steps, "viv_project" is created. You can open it with Vivado GUI, and check the design hierarchy. And it will call the selected simulator to compile the simulation model.
+
+
+ make image: This step runs synthesis, implementation and bitstream generation. It calls "hardware/setup/snap_build.tcl" and also uses some related tcl scripts to work on "viv_project". In this step, "hardware/build" will be created and the output products like bit images, checkpoints (middle products for debugging) and reports (reports of timing, clock, IO, utilization, etc.) If everything runs well and timing passes, user will get the bitstream files (in "Images" sub directory) to program the FPGA card.
+
+
+
+
+ BSP (board support package) module
+
+ CAPI1.0: base_image (b_route_design.dcp)
+
+
+
+
+
+
+ For CAPI1.0, base_image contains surrounding logic and the kernel logic:
+
+ PCIe hard IP core (pcie3_ultrascale_0)
+ Flash Controller (psl_flash)
+ VSEC: Vendor Specific Extended Capability (psl_vsec)
+ Xilinx MultiBoot control logic (psl_xilmltbt)
+ PSL kernel logic (psl)
+
+ The interface between base_image and AFU(psl_accel) has 5 groups of signals, described in PSL spec CAPI1.0 PSL/AFU interface Spec.
+ The interface between base_image and Chip IOs are card specific, and the information need to be provided by Card Vendor. Generally, they include:
+
+ Flash interface (usually DPIx16)
+ PCIe interface: perst, refclk, TX and RX data lanes
+ Peripheral IPs: I2C, LED, DDR, Ethernet, etc.
+
+ Marked in light orange color, you can download the entire base_image (b_route_design.dcp) from OpenPower Portal.
+
+
+ CAPI2.0: capi2-bsp (capi_bsp_wrap.xcix)
+
+
+
+
+
+
+ For CAPI2.0, the structure is similar, but the PSL9 logic (marked in light orange color) is provided as an encrypted Zip package. It can be downloaded from OpenPower Portal and put in "capi2-bsp/psl" directory. Then it uses the make process in capi2-bsp to generate an IP container file (capi_bsp_wrap.xcix). Please refer to the README file at https://github.com/open-power/capi2-bsp for more details.
+ CAPI2.0 cards are using SPI Flash interface: SPIx4 or dual SPIx4 (also mentioned as SPIx8). For PCIe Gen3, it uses 16 lanes. For PCIe Gen4, it uses 8 lanes. The interface of PSL9 has 6 groups of signals. Please refer to CAPI2.0 PSL/AFU interface Spec for the details.
+ The logic in snap_core (CAPI2.0) implements the data path with DMA interface. Buffer interface is not used.
+ The above two figures apply to both HDK development and SNAP framework. The difference is, for HDK developers, they work on the AFU by themselves. For SNAP developers, they make use of the snap_core logic and only work on action_wrapper. The AFU part for SNAP developers contains following blocks:
+
+ AFU diagram in SNAP framework
+
+
+
+
+
+
+
+ AFU logic RTL files are open-sourced. Developer can make modifications for their own purpose, like adding multiple DDR channels, adding NVMe and Ethernet controllers.
+
+
+
+
+ Enable a new card in SNAP
+ For a new FPGA card, the detailed items to update can be classified into following sections:
+
+ Preparations
+ Hardware RTL, setup, simulation
+ Software and tools
+ Testing
+ Publishing
+
+
+ Preppartions
+ First, give a FPGACARD name. It should start from the company's name, following with the card ID and be short. For example. ADKU3 = Alpha-Data ADM-PCIE-KU3. Get follow information from the card vendor. (You can check the "Status" column to trace the progress.)
+
+
+ Information to collect
+
+
+
+
+
+
+
+ Item
+
+
+
+
+ Description
+
+
+
+
+
+
+ FPGACARD
+ Short card name used in SNAP
+
+
+ FPGACHIP
+ FPGA part name, for example, xcvu9p-fsgd2104-2L-e
+
+
+ Flash Type
+ Flash chip that attached to FPGA, for example mt28gu01gaax1e-bpi-x16. And the related xdc files for FPGA config.
+
+
+ DDR MC IP
+ Short card name used in SNAP
+
+
+ FPGACARD
+ DDR memory controller Vivado IP tcl/xdc file.
+
+
+ Other peripherals
+ NVMe IP, Ethernet IP and so on (Optional)
+
+
+ IO pins
+ PACKAGE_PIN for base_image or bsp: flash, pcie, i2c etc.
+ PACKAGE_PIN for peripheral IPs.
+
+
+
+
+
+ SNAP environment updates
+ The best way is to grep some keywords like "S241" or "AD8K5" under the directories and look for the locations that need modifications.
+
+ If you meet files ending with "_source", like "psl_fpga.vhd_source", that means this file will be pre-processed to generate the output file without "_source" suffix, like "psl_fpga.vhd". There are #ifdef macros or comments like -- only for NVME_USED=TRUE. They help to create a target VHDL/Verilog file with different configurations.
+
+ Below lists the files to change. There may be some differences with new commits in SNAP git repository. Keep in mind they include:
+
+ snap_config and environmental files
+ Hardware: psl_accel and psl_fpga (top) RTL files
+ Hardware: tcl files for the workflow
+ Hardware: Board: xdc files for IO/floorplan/clock/bitstream
+ Hardware: DDR: create_ip, sim model, xdc files
+ Hardware: Other IP: create_ip, sim model, xdc files
+ Software: New card type, register definition
+ Testing: jenkins
+ Readme and Documents
+
+
+
+ For CAPI1.0, you need to generate a new PSL checkpoint file and upload it to OpenPower Portal. Chapter TODO describes the details.
+ For CAPI2.0, you need to add a ${FPGACARD} directory in capi2-bsp git repository. Copy an existing folder as a start and follow the README file.
+ Make sure the information in xdc/tcl files are permitted to be open-source.
+ Send email to OpenPower Acceleration Workgroup or contact your representative to apply for a subsystem device ID for the new card. For example, ADKU3 uses 0x0605. S241 uses 0x0660.
+ You also need to update https://github.com/ibm-capi/capi-utils to allow capi-flash-script to program this new card. Subsystem ID will be used there. It is also used in snap/software/tools/snap_find_card.
+
+
+
+
+ Config files to change
+
+
+
+
+
+
+
+ File name
+
+
+
+
+ Changes done
+
+
+
+
+
+
+ scripts/Kconfig
+ adding card to the Kconfig menu. Provide Flash information (size/type/user address)
+
+
+ hardware/doc/SNAP-Registers.md
+ SNAP registers for new card - doc
+
+
+ hardware/setup/snap_config.sh
+ SNAP registers - setting
+
+
+
+
+
+
+ RTL/xdc/tcl files to change
+
+
+
+
+
+
+
+ File name
+
+
+
+
+ Changes done
+
+
+
+
+
+ hardware/hdl/core/psl_accel_${FPGACARD}.vhd_source specific to card
+hardware/hdl/core/psl_accel_types.vhd_sourcespecific to card
+hardware/hdl/core/psl_fpga_${FPGACARD}.vhd_source specific to card
+hardware/setup/${FPGACARD}/capi_bsp_pblock.xdc specific to card
+hardware/setup/${FPGACARD}/snap_${FPGACARD}.xdc specific to card
+hardware/setup/${FPGACARD}/snap_ddr4pins.xdc specific to card
+hardware/setup/build_mcs.tcldeclare card name
+hardware/setup/create_framework.tcldeclare card name
+hardware/setup/create_snap_ip.tcldeclare card name and its IP
+hardware/setup/flash_mcs.tcldeclare card name
+hardware/setup/snap_bitstream_post.tcldeclare card name
+hardware/setup/snap_bitstream_pre.tcldeclare card name
+hardware/setup/snap_bitstream_step.tcldeclare card name
+hardware/setup/snap_impl_step.tcldeclare card name
+hardware/snap_check_psldeclare card name
+
+
+
+
+
+ Software files to change
+
+
+
+
+
+
+
+ File name
+
+
+
+
+ Changes done
+
+
+
+
+
+software/lib/snap.cdeclare card name
+software/tools/snap_find_carddeclare card name + id
+software/include/snap_regs.hSNAP registers - setting
+
+
+
+
+
+ Other files to change
+
+
+
+
+
+
+
+ File name
+
+
+
+
+ Changes done
+
+
+
+
+
+actions/scripts/snap_jenkins.shjenkins tests (optional)
+defconfig/{FPGACARD}*.defconfigFor silent jenkins testing (optional)
+README.mdAnnounce a new card is supported
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/enable_capi_snap/ch_example.xml b/enable_capi_snap/ch_example.xml
new file mode 100644
index 0000000..1ae70dd
--- /dev/null
+++ b/enable_capi_snap/ch_example.xml
@@ -0,0 +1,323 @@
+
+
+
+
+ Enable CAPI2 card
+
+
+
+ Section Title goes here
+ This Section covers something of interest to a limited number of people and shows a 1st level section
+
+
+ Example Itemized List
+
+ Here is an example of an itemized list
+
+ A list title is completely optional
+
+
+ Item you don't care about
+
+
+ Perhaps you'd like a sub-list
+
+
+ Oooh, here's about another
+
+
+
+
+
+
+
+ Item you might care about
+
+
+
+ Item you do care about
+
+
+
+
+ Example ordered list
+
+ All good documents need ordered lists.
+
+ Another purely optional title
+
+ First item
+
+
+ Second item
+
+
+ first indented item
+
+
+ second indented item
+
+
+
+
+ Third item
+
+
+
+
+
+ Example figure with embedded graphic
+
+ Here is how you embed a graphic.
+
+ Example figure
+
+
+
+
+
+
+ Raw images such as the bitmap (bmp) file above may become blurry as they are scaled.
+ Scalable graphic formats like SVG (Scalable Vector Graphics) embed and scale the best.
+
+
+
+ Example table
+ Of course all good documents need tables. Here's how you build a basic table.
+
+
+ Example Table Title
+
+
+
+
+
+
+
+
+
+ 1st Column Heading
+
+
+
+
+ 2nd Column Heading
+
+
+
+
+ 3rd Column Heading
+
+
+
+
+ 4th Column Heading
+
+
+
+
+
+
+
+ Yes
+
+
+ Red
+ Green
+ Blue
+ Custom (Amber)
+
+
+ MAIN_Junk
+
+
+ More_Junk
+
+
+
+
+ merged cells horizontal
+
+
+ cell_stuff
+
+
+
+
+ Merge cells vertical
+
+
+ filler
+
+
+ merge cells both ways
+
+
+
+
+ filler 2
+
+
+
+
+ How about we put a list in the table cell
+
+
+ item 1
+
+
+ item 2
+
+
+ item 2
+
+
+
+
+ Another Cell
+
+
+ Yet Another Cell
+
+
+ Finally the last cell
+
+
+
+
+
+
+
+ Example of crossreferences and footnotes
+ To reference another section or table is pretty easy. For example: see for how tables look.
+ Lists are shown in and if you need to make a footnote
+ The footnote text goes here and can reference something like for additional explanation.
+ For clarification that is easy. Of course you might want an additional reference to the footnote which can also be done easily.
+ Lastly you probably want to mark text by making it italic text example or Bold Text Example.
+
+
+ Example of code citations and user input
+ When showing user input, you want a nice sceen-looking layout, a prompt, monospace text, and a way to differentiate input from output. Here's an example:
+ $ echo "Hello world"
+Hello world
+$
+
+ Docbook also allows for formatting and display of common languages, allowing for whitespace
+ and line returns just as they are written. Here's a sample snippet of C code with line numbering enabled:
+main()
+{
+ printf("Hello world\n");
+}]]>
+ If code formatting is not quite what you need, simply displaying text "literally" may suffice as follows: This is my literal
+text. It ignores whitespace.
+
+
+ Example of special characters in text
+ Sometimes in text you need special characters. These can be provided using their UNICODE values such as ≠ (≠),
+ Ω (Ω), and ∆ (∆).
+ These can be "coded" using the form &#ddddd; where ddddd is
+ the up to five digit decimal representation of the character. The form &#xhhhh; where
+ hhhh is the up to 4 digit hexidecimal representation of the character.
+ This formatting works well as long as the symbol to which you are referring is contained in the font set
+ used for the document -- Arimo for standard text and Cousine for monospace. If when building a document, you see a message like
+ "WARNING, Glyph...not available in font 'Arimo',"
+ see in for details on using the provided symbol fonts explicitly.
+
+
+
+
+
+ Examples of OpenPOWER Foundation Docbook extensions
+
+ The OpenPOWER Foundation Maven Plugin supports a number of extensions that are not pure Docbook. These are:
+
+
+ Setting text color explicitly
+
+ Text color can be controlled using <phrase role="color:color_name">
+ tag where color_name contains the color setting. For example, this
+ text:A red sentence contains a blue word.]]> produces this sentence:
+ A red sentence contains a blue word.
+ Valid colors include either a keyword color name or a numerical RGB specification. Keyword names are common with the HTML 4 specificiation:
+ aqua, black, blue, fuchsia, gray,
+ green, lime, maroon, navy, olive,
+ purple, red, silver, teal, white,
+ and yellow. Additionally, RGB values can be #nnnnnn where nnnnnn is a hexidecimal color value or
+ rgb(n1, n2, n3) where n1, n2, and n3 are integers 0-255.
+ This tag has also been implemented on the following tags: <thead>,
+ <tbody>, and <tfoot>.
+ This parameter should only be used for tags listed above.
+
+
+
+ Inserting line breaks
+ Line breaks can be introduced using <?linebreak?> tags. For example, this
+ text:A line break in the middle of text]]> produces this sentence:
+ A line break in the middle of text
+ This tag becomes useful in table text spacing.
+
+
+
+ Inserting page breaks
+ Page breaks can be introduced using <?hard-pagebreak?> tags. For example, this
+ text:A page breakBetween two paragraphs]]> produces this output:
+ A page breakBetween two paragraphs
+ This tag becomes useful in placing tables on page. Placing this statement before a large table may prevent it from spanning a page.
+ Because the XSL template behind the Processing Instruction generates
+ a ]]> in
+ the book FO output, this instruction should be used in the outer most blocks of a section to work effectively. Use inside lists and other structural
+ components may result in the text after the break being dropped. User beware!.
+
+
+
+ Varying the font size
+ Font sizes can also be set using the
+ <phrase role="font-size:size">
+ tag where size contains a size value such as "6pt" or "50%" or "1.5em".
+ For example, a paragraph can be made to be 6 point as follows:A sentence that contains some 6pt font,
+50% font, and
+1.5em font in it.]]> produces this output:
+ A sentence that contains some 6pt font,
+ 50% font, and 1.5em font in it.
+ This tag has also been implemented on the following tags: <para>,
+ <thead>, <tbody>, and <tfoot>.
+ This parameter should only be used for tags listed above.
+
+
+
+ Using additional symbols
+ If you find that the Arimo and Cousine fonts do not contain the special symbol you need
+ for your document, you may use the additional symbol font provided for document (STIX Two Math).
+ Due to an unimplemented feature in the Apach FO Processor, selection of this
+ font needs to be explicitly performed using the
+ <symbol role="symbolfont"> wrapper around your symbol value.
+
+ For example, the symbol coding of should produce
+ a circle with a cross in here "⨁", but instead creates a "Glyph...not available in font 'Arimo'" error
+ on document build and the PDF renders as a "#".
+
+ Re-coding this to use ⨁]]> produces
+ the correct symbole here "⨁".
+ If this still does not provide the symbol you expected, double check the code and the font maps found at
+ http://www.stixfonts.org/charactertable.html.
+
+
+
+
+
diff --git a/enable_capi_snap/ch_genpsl_capi10.xml b/enable_capi_snap/ch_genpsl_capi10.xml
new file mode 100644
index 0000000..768bb63
--- /dev/null
+++ b/enable_capi_snap/ch_genpsl_capi10.xml
@@ -0,0 +1,207 @@
+
+
+
+
+
+ Work with CAPI1.0 HDK project
+
+
+
+ Ask your contact representative or write to capi@us.ibm.com to get a CAPI1.0 HDK project to start. This chapter only works for CAPI1.0, running on Power8.
+
+ Steps and directory structure
+ We use an "Out-of-context" flow to generate a PSL dcp file. For a new FPGA card, following steps need to be done:
+
+ Four steps to build a PSL checkpoint
+
+
+
+
+
+
+
+ We use an "Out-of-context" flow to generate a PSL dcp file. The directory structure is as following:
+
+ HDK directory structure
+
+
+
+
+
+
+
+ In build_dir/Sources
+
+
+ The "prj" directory includes the source file lists.
+ The "top" directory includes the top design file "psl_fpga.vhdl" and the wrapper for AFU "psl_accel.vhdl"
+ PSL source files are in "psl" directory.
+ AFU source files are in "afu" directory.
+ "cores" includes 4 Xilinx IP cores used by PSL.
+ "xdc" are the constraint files used by PSL and the top design.
+
+ In build_dir, psl_fpga.tcl is the script "entrance". It assigns the FPGA chip information, and the build flow.
+ FPGA chip information is needed for a new FPGA card, for example:
+ set device "xcku115"
+set package "-flva1517"
+set speed "-2-e"
+ And some controlling bits are for two build flows:
+ 1. Build a PSL checkpoint.
+ 2. Build a whole FPGA image (including AFU).
+
+
+ Generate PSL Checkpoint (b_route_design.dcp)
+ In this section, we just talk about the first build flow - "build a PSL checkpoint". Read it when you need to enable a FPGA card on CAPI1.0 or to fix a bug and update b_routed_design.dcp. The controlling bits should be set as:
+
+ ####flow control
+set run.topSynth 1
+set run.oocSynth 1
+set run.tdImpl 0
+set run.oocImpl 1
+set run.topImpl 0
+set run.flatImpl 0
+ The outfile file will be placed in "Checkpoint" directory, the file name is "b_route_design.dcp".
+
+ Upgrade Xilinx IP cores
+ When a FPGA chip type is changed, or the Vivado tool version has been upgraded, you need to upgrade the Xilinx IP cores that are used in PSL module. PSL module has instantiated four Xilinx IP cores (in Sources/cores):
+
+ pcie3_ultrascale_0
+ sem_ultra_0 (Soft Error Migration)
+ clk_wiz_0
+ tx_wr_fifo
+
+ Steps to upgrade them:
+
+ Open Vivado GUI
+ Create a new project. For the second time, just open the project with the four IP cores.
+ Import IP cores (by importing *.xci files under "Source/cores/xxx" directory). For the second time, this step is not needed.
+ Set FPGA type in Project Settings.
+ Run "Tools->Report->Report IP Status"
+ "Upgrade All" and read the upgrade log.
+
+
+ Small project to update IPs
+
+
+
+
+
+
+
+ Report IP Status
+
+
+
+
+
+
+
+ For PCIe IP, you need to change subsystem_id for a new card. Right click pcie3_ultrascale_0 -> Reconfig IP and change the subsystem ID field.
+ Ask an IBM representative for the subsystem ID.
+
+
+ Input xdc files
+ The IO pin package information for the new card should be provided by card vendor. Generally, they include Flash Interface, PCIe Interface and other interfaces like I2C and LED. Sample code with IO pins in b_phys.xdc:
+ Refer to Xilinx document UG575 for detailed pin package information.
+ Example:
+ set_property PACKAGE_PIN AJ15 [get_ports {o_flash_a[1]}]
+set_property PACKAGE_PIN AK15 [get_ports {o_flash_a[2]}]
+set_property PACKAGE_PIN AH14 [get_ports {o_flash_a[3]}]
+
+set_property IOSTANDARD LVCMOS18 [get_ports {o_flash_a[1]}]
+set_property IOSTANDARD LVCMOS18 [get_ports {o_flash_a[2]}]
+set_property IOSTANDARD LVCMOS18 [get_ports {o_flash_a[3]}]
+ Some other constraints also must be updated for the new selection of FPGA chip. It defines the floorplan for PSL.
+
+ There is also a patch to keep VSEC address for Vivado2017.4 and newer Vivado version:
+ set_property PF0_SECONDARY_PCIE_CAP_NEXTPTR 12'h400 [get_cells *pcihip0/psl_pcihip0_inst/inst/pcie3_uscale_top_inst/pcie3_uscale_wrapper_inst/PCIE_3_1_inst]
+ This is the base address for VSEC registers. capi-utils uses it to set the register address to send bitstream data to flash controller.
+ Update:This step is not needed after "ECAP update (#28)" commit of capi-utils in Feb 2018.
+
+
+
+ Run Vivado to generate PSL checkpoint
+ vivado -mode batch -source psl_fpga.tcl -notrace
+ The checkpoint file b_route_design.dcp will be generated and put in "Checkpoint" directory. With this checkpoint file, we can continue to build a full FPGA bit image and validate it on hardware.
+
+
+
+
+ Generate full FPGA image
+ Steps
+
+ Steps to build the full FPGA image
+
+
+
+
+
+
+ This time the controlling bits should be set to:
+ ####flow control
+set run.topSynth 1
+set run.oocSynth 0
+set run.tdImpl 0
+set run.oocImpl 0
+set run.topImpl 1
+set run.flatImpl 0
+
+ Check top design file psl_fpga.vhdl
+ For a new card, the IO pins and functions may be different to your reference card design. So the logic in top file psl_fpga.vhdl needs to be updated.
+ Similarly, the xdc file "……topimp.xdc" also needs to be updated.
+
+ Prepare filelist for psl_fpga.prj
+ The "prj" file is a file list. It should contain all the AFU design files. Edit it.
+
+ Run Vivado
+ Two sub steps are here.
+ vivado -mode batch -source psl_fpga.tcl -notrace
+ Now the bit file is generated
+ vivado -mode batch -source write_bitstream.tcl -notrace
+ Now you get the bin files to be program to the FPGA flash.
+
+ For more information about FPGA configuration, please refer to Xilinx Document UG570.
+ Then you can program the generated bin file to FPGA either by JTAG or on-line programming tools capi-utils
+
+
+
+
+
+
+
+
+
+
+
diff --git a/enable_capi_snap/ch_introduction.xml b/enable_capi_snap/ch_introduction.xml
new file mode 100644
index 0000000..d66f975
--- /dev/null
+++ b/enable_capi_snap/ch_introduction.xml
@@ -0,0 +1,94 @@
+
+
+
+
+
+ Introduction
+
+
+ What is CAPI
+ CAPI stands for "Coherent Accelerator Processor Interface" which enables FPGA to access Host memory by virtual address. You can find more introduction about this interface on https://developer.ibm.com/linuxonpower/capi/. It is an important feature to develop hardware accelerators in heterogeneous computing. In this document, the "hardware accelerators" are built on FPGA.
+
+ CAPI(1.0 and 2.0) basic concept
+
+
+
+
+
+
+
+ A complete accelerator has software part (APP, or Applications) running on CPU Processor and the hardware part (AFU, Acceleration Function Unit) running on FPGA chip. APP and AFU are sharing host memory, that means, they both can read and write the 2^64 range of virtual memory address. To make it happen, CAPI technology has a CAPP (Coherent Acceleration Processor Proxy) logic unit in Processor chip, and also needs a PSL (Processor Service Layer) logic unit in FPGA chip. For CAPI1.0 and CAPI2.0, the interconnection between processor and FPGA is using PCIe physical links and PCIe form factor. CAPI1.0 uses PCIe Gen3x8 and CAPI2.0 uses PCIe Gen4x8 or Gen3x16. (For OpenCAPI, the physical links and the connected datalink layer and transportation layer all change to OpenCAPI. Please check https://opencapi.org for more information. It is not covered in this document.)
+
+
+
+ HDK and SNAP
+ Let's focus on the FPGA side.
+ A customer FPGA card needs to have a PSL module (Processor Service Interface) to become a "CAPI-enabled" card. This PSL module is provided by IBM. For CAPI1.0, it is in the form of a post-implemented dcp file (Xilinx Vivado design checkpoint). For CAPI2.0, it is encrypted source code. They can be downloaded at https://www.ibm.com/systems/power/openpower. From the menu, select "CAPI","Coherent Accelerator Processor Interface (CAPI)" or directly click the "CAPI" icon to go to the CAPI section. Then download the appropriate files depending on your target system being POWER8 (CAPI 1.0) or POWER9 (CAPI 2.0). You need to register an IBM ID to download them.
+ A project from FPGA Vendors (i.e, a Xilinx Vivado project), including PSL module and sample user logic (AFU), is delivered to acceleration developers. This project is called HDK (Hardware Development Kit).
+
+ Develop an acceleration on HDK
+
+
+
+
+
+
+
+ Working on HDK, developers need to know the details about PSL interface specifications. You can find CAPI1.0 PSL Spec and CAPI2.0 PSL Spec or search "PSL/AFU interface" in your web browser. But they have differences. To hide the differences on the interface and provide an industry standard interface protocol (AXI), we also created SNAP framework.
+
+ SNAP is the abbreviation of Storage, Networking and Analytics Programming. It is an open-source framework https://github.com/open-power/snap. On the FPGA side, SNAP framework adds a PSL/AXI bridge, a DDR SDRAM controller and an optional NVMe controller. Thus, the developer can focus on their acceleration kernel logic (here we call it hardware action) and interface the framework via several AXI ports.
+
+ Develop an acceleration on SNAP
+
+
+
+
+
+
+
+ For both CAPI1.0 and CAPI2.0, people can choose to work on HDK or on SNAP. The preferred way is to work on SNAP. In the following chapters, we will introduce:
+
+ Enable a FPGA card in SNAP
+ Generate a PSL Checkpoint (CAPI1.0 only)
+ Work with HDK (CAPI1.0)
+ Work with HDK (CAPI2.0)
+
+
+ For most of the new cards on Power9, just reading the chapter of "Enable a FPGA card in SNAP" is enough. You can find abundant materials on how to develop an accelerator with SNAP (Training videos, "docs" folder on github, or other webpages) so they are not discussed in this document.
+
+
+
+
+
diff --git a/enable_capi_snap/figures/.DS_Store b/enable_capi_snap/figures/.DS_Store
new file mode 100644
index 0000000..5008ddf
Binary files /dev/null and b/enable_capi_snap/figures/.DS_Store differ
diff --git a/enable_capi_snap/figures/afu.png b/enable_capi_snap/figures/afu.png
new file mode 100644
index 0000000..811cbf5
Binary files /dev/null and b/enable_capi_snap/figures/afu.png differ
diff --git a/enable_capi_snap/figures/base_image.png b/enable_capi_snap/figures/base_image.png
new file mode 100644
index 0000000..6841471
Binary files /dev/null and b/enable_capi_snap/figures/base_image.png differ
diff --git a/enable_capi_snap/figures/bsp.png b/enable_capi_snap/figures/bsp.png
new file mode 100644
index 0000000..e686d9f
Binary files /dev/null and b/enable_capi_snap/figures/bsp.png differ
diff --git a/enable_capi_snap/figures/capi1.png b/enable_capi_snap/figures/capi1.png
new file mode 100644
index 0000000..305a80b
Binary files /dev/null and b/enable_capi_snap/figures/capi1.png differ
diff --git a/enable_capi_snap/figures/full-steps.png b/enable_capi_snap/figures/full-steps.png
new file mode 100644
index 0000000..6fb3481
Binary files /dev/null and b/enable_capi_snap/figures/full-steps.png differ
diff --git a/enable_capi_snap/figures/hdk-structure.png b/enable_capi_snap/figures/hdk-structure.png
new file mode 100644
index 0000000..184df7e
Binary files /dev/null and b/enable_capi_snap/figures/hdk-structure.png differ
diff --git a/enable_capi_snap/figures/hdk.png b/enable_capi_snap/figures/hdk.png
new file mode 100644
index 0000000..21b3905
Binary files /dev/null and b/enable_capi_snap/figures/hdk.png differ
diff --git a/enable_capi_snap/figures/ip-update.png b/enable_capi_snap/figures/ip-update.png
new file mode 100644
index 0000000..665c2c0
Binary files /dev/null and b/enable_capi_snap/figures/ip-update.png differ
diff --git a/enable_capi_snap/figures/psl-4steps.png b/enable_capi_snap/figures/psl-4steps.png
new file mode 100644
index 0000000..b02c0ab
Binary files /dev/null and b/enable_capi_snap/figures/psl-4steps.png differ
diff --git a/enable_capi_snap/figures/report-ip.png b/enable_capi_snap/figures/report-ip.png
new file mode 100644
index 0000000..81995d3
Binary files /dev/null and b/enable_capi_snap/figures/report-ip.png differ
diff --git a/enable_capi_snap/figures/snap-structure.png b/enable_capi_snap/figures/snap-structure.png
new file mode 100644
index 0000000..5ba9af7
Binary files /dev/null and b/enable_capi_snap/figures/snap-structure.png differ
diff --git a/enable_capi_snap/figures/snap-structure_white.png b/enable_capi_snap/figures/snap-structure_white.png
new file mode 100644
index 0000000..857981e
Binary files /dev/null and b/enable_capi_snap/figures/snap-structure_white.png differ
diff --git a/enable_capi_snap/figures/snap.png b/enable_capi_snap/figures/snap.png
new file mode 100644
index 0000000..6d36264
Binary files /dev/null and b/enable_capi_snap/figures/snap.png differ
diff --git a/enable_capi_snap/pom.xml b/enable_capi_snap/pom.xml
new file mode 100644
index 0000000..7366c1f
--- /dev/null
+++ b/enable_capi_snap/pom.xml
@@ -0,0 +1,161 @@
+
+
+
+
+
+ org.openpowerfoundation.docs
+ workgroup-pom
+ 1.0.0-SNAPSHOT
+ ../pom.xml
+
+ 4.0.0
+
+
+ todo-artifact_id
+
+ jar
+
+
+ enable-fpga-to-capi-snap
+
+
+
+
+ 0
+
+
+
+
+
+
+
+
+ org.openpowerfoundation.docs
+
+ openpowerdocs-maven-plugin
+
+
+
+ generate-webhelp
+
+ generate-webhelp
+
+ generate-sources
+
+
+ ${comments.enabled}
+ openpower-template-guide
+ 1
+ UA-17511903-1
+
+ appendix toc,title
+ article/appendix nop
+ article toc,title
+ book toc,title,figure,table,example,equation
+ book/appendix nop
+ book/chapter nop
+ chapter toc,title
+ chapter/section nop
+ section toc
+ part toc,title
+ reference toc,title
+ set toc,title
+
+
+ 1
+ 1
+ 1
+
+
+ enable-fpga-to-capi-snap
+
+
+ enable-fpga-to-capi-snap
+
+
+
+ workgroupSpecification
+
+
+
+
+ workgroupConfidential
+
+
+
+
+ draft
+
+
+
+
+
+
+
+
+ true
+ .
+
+
+ bk_main.xml
+
+
+
+
+ ${basedir}/../glossary/glossary-terms.xml
+ 1
+ www.openpowerfoundation.org
+
+
+
+
+
diff --git a/enable_capi_snap/sec_example.xml b/enable_capi_snap/sec_example.xml
new file mode 100644
index 0000000..8bd05dd
--- /dev/null
+++ b/enable_capi_snap/sec_example.xml
@@ -0,0 +1,25 @@
+
+
+
+ Sample section include
+ This section was developed in a separate file but included in the document by using the following
+ text:]]>
+ where sec_example.xml is the source file name.
+
+
diff --git a/pom.xml b/pom.xml
index 7b370ef..830cb70 100644
--- a/pom.xml
+++ b/pom.xml
@@ -17,8 +17,6 @@
- doc_dev_guide
- doc_template
- errata_template
+ enable_capi_snap