Change document to public

Change document from Workgroup Confidential to Public

Signed-off-by: Klaus Heinrich Kiwi <klaus@linux.vnet.ibm.com>

.gitignore

@@ -0,0 +1,2 @@
+*~
+target/

LICENSE

@@ -0,0 +1,176 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+

README.md

@@ -1,2 +1,91 @@
-# master_template
-This repository is for the development and maintenance of the Docbook common files and template used for OPF Workgroup documents
+# OpenPOWER Foundation CAPI SNAP Enablement Documentation
+This repository holds the source for the *Enable new FPGA Cards for CAPI2.0*
+for OpenPOWER Foundation. 
+
+To build this project, one must ensure that the Docs-Master project has
+also been cloned at the same directory level as the Docs-Template project.
+This can be accomplished with the following steps:
+
+1. Clone the master documentation project (Docs-Master) using the following command:
+
+  ```
+  $ git clone https://github.com/OpenPOWERFoundation/Docs-Master.git
+  ```
+  
+2. Clone this project (CAPI-SNAP-Doc) using the following command:
+
+  ```
+  $ git clone https://github.com/OpenPOWERFoundation/CAPI-SNAP-Doc.git
+  ```
+  
+3. Build the project with these commands:
+  ```
+  $ cd CAPI-SNAP-Doc
+  $ mvn clean generate-sources
+  ```
+
+For more information about with the document generation, please refer to [Documentation Development Guide](https://openpowerfoundation.org/?resource_lib=openpower-foundation-documentation-development-guide)
+
+The online version of this document can be found in the OpenPOWER Foundation
+Document library at [https://openpowerfoundation.org/?resource_lib=enable-new-fpga-cards-for-capi-2-0](https://openpowerfoundation.org/?resource_lib=enable-new-fpga-cards-for-capi-2-0) and also this github repository's release tab.   
+
+
+## License
+This project is licensed under the Apache V2 license.  More information
+can be found in the LICENSE file or online at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+## Community
+To comment on, propose changes, and engage in community dialogue, you can open issues 
+in the [Project Issues](https://github.com/OpenPOWERFoundation/CAPI-SNAP-Doc/issues), post to
+the community mailing list \([capi-snap-doc@mailinglist.openpowerfoundation.org](mailto://capi-snap-doc@mailinglist.openpowerfoundation.org)\).
+
+## Contributions
+Contributions to this project should conform to the `Developer Certificate
+of Origin` as defined at http://elinux.org/Developer_Certificate_Of_Origin.
+Commits to this project need to contain the following line to indicate
+the submitter accepts the DCO:
+```
+Signed-off-by: Your Name <your_email@domain.com>
+```
+By contributing in this way, you agree to the terms as follows:
+```
+Developer Certificate of Origin
+Version 1.1
+
+Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+660 York Street, Suite 102,
+San Francisco, CA 94110 USA
+
+Everyone is permitted to copy and distribute verbatim copies of this
+license document, but changing it is not allowed.
+
+
+Developer's Certificate of Origin 1.1
+
+By making a contribution to this project, I certify that:
+
+(a) The contribution was created in whole or in part by me and I
+    have the right to submit it under the open source license
+    indicated in the file; or
+
+(b) The contribution is based upon previous work that, to the best
+    of my knowledge, is covered under an appropriate open source
+    license and I have the right under that license to submit that
+    work with modifications, whether created in whole or in part
+    by me, under the same open source license (unless I am
+    permitted to submit under a different license), as indicated
+    in the file; or
+
+(c) The contribution was provided directly to me by some other
+    person who certified (a), (b) or (c) and I have not modified
+    it.
+
+(d) I understand and agree that this project and the contribution
+    are public and that a record of the contribution (including all
+    personal information I submit with it, including my sign-off) is
+    maintained indefinitely and may be redistributed consistent with
+    this project or the open source license(s) involved.
+```
+

enable_capi_snap/app_template.xml

@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Copyright (c) 2016 OpenPOWER Foundation
+  
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  
+-->
+<appendix xmlns="http://docbook.org/ns/docbook"
+  xmlns:xi="http://www.w3.org/2001/XInclude"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  version="5.0"
+  xml:id="app_template">
+    <?dbhtml stop-chunking?>
+    <title>Appendix template</title>
+    <para>This is the first paragraph of a new appendix...</para>
+    <section xml:id="sample-app-section">
+        <title>Section title</title>
+        <para>Section text...</para>
+    </section>
+</appendix>

enable_capi_snap/bk_main.xml

@@ -0,0 +1,114 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Copyright (c) 2016 OpenPOWER Foundation
+  
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  
+-->
+
+<!-- The following entity variable is used to reflect the version of the 
+     template document master used for building a document.  This value
+     should be set by copy of the of template used to create a new
+     document and should not be changed.  Use of this value is in
+     in the Abstract section in this file. --> 
+<!DOCTYPE book [
+<!ENTITY template_version "1.0.0">
+]>
+
+<book xmlns="http://docbook.org/ns/docbook"
+  xmlns:xi="http://www.w3.org/2001/XInclude"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  version="5.0"
+  xml:id="bk_main">
+
+    <!-- TODO: Pick a Title for the new document -->
+    <title>Enable new FPGA cards for CAPI2.0</title>
+    <!-- TODO: Either add a subtitle or remove the following line -->
+    <subtitle>BSP and SNAP</subtitle>
+
+    <info>
+    <author>
+      <!-- TODO: Set the correct Work Group Name and email id for WG Chair -->
+      <personname>
+          System Software Workgroup
+      </personname>
+      <email>syssw-chair@openpowerfoundation.org</email>
+      <affiliation>
+          <orgname>OpenPower Foundation</orgname>
+      </affiliation>
+    </author>
+    <copyright>
+      <year>2020</year>
+      <holder>OpenPOWER Foundation</holder>
+    </copyright>
+    <!-- TODO: Set the correct document releaseinfo -->
+    <releaseinfo>Revision 1.0</releaseinfo>
+    <productname>OpenPOWER</productname>
+    <pubdate/>
+
+    <legalnotice role="apache2">
+    <!--legalnotice role="opfExternal"-->
+
+      <annotation>
+          <remark>Copyright details are filled in by the template.</remark>
+      </annotation>
+    </legalnotice>
+    
+    <!-- TODO: Update the following text with the correct document description (first paragraph),
+               Work Group name, and Work Product track (both in second paragraph). -->
+    <abstract>
+      <!-- TODO: remove "phrase" tags (2) and text below and insert proper information -->
+      <para>The purpose of this document is to describe how to enable a new customer card to support CAPI SNAP framework. SNAP is a open-source programming framework for FPGA Accelerations. Its homepage is <link xlink:href="https://github.com/open-power/snap">https://github.com/open-power/snap</link>. With it, you can develop accelerators with CAPI technology easily.</para>
+      
+      <para>This document describes the flow and steps to enable a new PCIe FPGA card to have CAPI2.0 features, and to support SNAP developing framework. If your PCIe FPGA card is not listed on today's available "SNAP enabled cards" (On the homepage README of <link xlink:href="https://github.com/open-power/snap">SNAP Github</link>), this document will guide you on <emphasis role="bold">how to enable it</emphasis>. Since all of the project files are open-source, you can create a Github repository fork, and create a new board support package (BSP) and walk through the working flow to enable SNAP. </para>
+
+      <para>This document is a Workgroup Note owned by the System Software Workgroup and handled in compliance with the requirements outlined in the
+      <citetitle>OpenPOWER Foundation Work Group (WG) Process</citetitle> document.  It was
+      created using the <citetitle>Master Template Guide</citetitle> version &template_version;.
+      Comments, questions, etc. can be submitted to the
+      public mailing list for this document at 
+      <email>capi-snap-doc@mailinglist.openpowerfoundation.org</email>.</para>
+    </abstract>
+
+    <revhistory>
+      <revision>
+          <!-- TODO: Set correct date and remove "phrase" tags (1) and text below and insert proper information -->
+          <date>2019-03-29</date>
+          <revdescription>
+            <itemizedlist spacing="compact">
+              <listitem>
+                <para>Start from the original Word document</para>
+              </listitem>
+            </itemizedlist>
+          </revdescription>
+      </revision>
+     </revhistory>
+    </info>
+
+  <!-- The ch_preface.xml file is required by all documents -->
+  <xi:include href="../../Docs-Master/common/ch_preface.xml"/>
+
+  <!-- TODO: Add your chapter heading files here.  Remove both files and insert your own. -->
+  <!--       See the template document for naming conventions and location of files.      -->
+  <xi:include href="ch_introduction.xml"/>
+  <xi:include href="ch_capi20_bsp.xml"/>
+  <xi:include href="ch_capi20_snap.xml"/>
+
+  <!-- The app_foundation.xml appendix file is required by all documents. -->
+  <xi:include href="../../Docs-Master/common/app_foundation.xml"/>
+
+  <!-- TODO: The following template document may be modified to create additional appendices as needed. -->
+  <!-- <xi:include href="app_template.xml"/>
+  -->
+
+</book>

enable_capi_snap/ch_capi20_bsp.xml

@@ -0,0 +1,191 @@
+<!--
+    Copyright (c) 2016 OpenPOWER Foundation
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<chapter version="5.0" xml:lang="en" xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"
+                                     xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="chapter_capi20_bsp">
+
+<!-- Chapter Title goes here. -->
+<title  xml:id="chapter_capi20_bsp_title">Enable CAPI2.0 BSP</title>
+<section><title>Structure</title>
+    <para>Each card supplier may design their FPGA board with different FPGA chips, circuit components, memory and IOs, so BSP (Board support package) is different from card to card. That's why an open-sourced project is so helpful: It allows card supplier and every developer to explore the functions of the card freely, and get benefits from CAPI technology. </para>
+    <figure pgwide="1" xml:id="capi2-bsp">
+        <title>Project hierarchy for HDK mode</title>
+        <mediaobject>
+        <imageobject>
+        <imagedata fileref="figures/bsp.png" format="PNG" scalefit="1" width="90%" align="center" />
+        </imageobject>
+        </mediaobject>
+    </figure>
+    <para>CAPI2.0 BSP (capi_bsp_wrap) contains following modules:</para>
+    <itemizedlist>
+    <listitem><para>PSL9 (PSL9_WRAP)</para></listitem> 
+    <listitem><para>PCIe hard IP core (pcie3/4_ultrascale_plus_0)</para></listitem> 
+    <listitem><para>Flash Controller (capi_flash_spi_mt25qt)</para></listitem> 
+    <listitem><para>VSEC: Vendor Specific Extended Capability (capi_vsec)</para></listitem> 
+    <listitem><para>Xilinx MultiBoot control logic (capi_xilmltbt)</para></listitem> 
+    <listitem><para>Other miscelleneous logic</para></listitem> 
+    </itemizedlist>
+    <para>In this diagram, psl_fpga.vhdl is the top design. How to connect the ports to FPGA pins is different from card to card, and the information is provided by Card Supplier. They may include:</para>
+    <itemizedlist>
+    <listitem><para>Flash interface (usually SPIx4 or dual SPIx4)</para></listitem>
+    <listitem><para>PCIe interface: perst, refclk, TX and RX data lanes (PCIe Gen3.0x16, or Gen4.0x8)</para></listitem>
+    <listitem><para>Peripheral IPs: I2C, LED, DDR, Ethernet, etc. </para></listitem>
+    </itemizedlist>
+    <para> At least Flash interface pins and PCIe interface pins are required to be assigned in xdc files precisely.</para>
+
+    <para> Between capi_bsp_wrap and user AFU (psl_accel), there are 6 groups of signals: Command, DMA, Buffer, Response, MMIO and Control. Please refer to <link xlink:href="http://openpowerfoundation.org/wp-content/uploads/resources/v2-psl-afu-spec/content/ch_preface.html"> CAPI2.0 PSL/AFU interface Spec</link> for the details. </para>
+   </section>
+
+<section><title>Step-by-step guidance</title>
+    <section><title>Work on github</title>
+        <para>capi2-bsp is a public Github repository. You need to have a Github account first. Then create a "fork" (Click the "fork" button) on <link xlink:href="https://github.com/open-power/capi2-bsp">https://github.com/open-power/capi2-bsp</link>.</para>
+        <screen>git clone https://github.com/[YOUR_USERNAME]/capi2-bsp</screen>
+        <note><para>capi2-bsp is also a submodule of snap.</para></note>
+        <para>Keep working on your own capi2-bsp fork, when it has been validated to work well, submit a pull request to "open-power/capi2-bsp" and request merging into the public upstream.</para>
+    </section>
+    <section><title>Preparations</title>
+        <para>First, define a FPGACARD name. It can start from the company's name, following with the card name and be short. For example, ADKU3 = Alpha-Data ADM-PCIE-KU3. Get information from the card supplier.</para>
+        <!-- A table starts -->
+        <table frame="all" pgwide="1" xml:id="supplier-info">
+            <title>Information to collect</title>
+            <tgroup cols="2">
+            <colspec colname="c2" colwidth="15*" />
+            <colspec colname="c3" colwidth="35*" />
+            <thead>
+                <row>
+                <entry>
+                <para>
+                <emphasis role="bold">Item</emphasis>
+                </para>
+                </entry>
+                <entry>
+                <para>
+                <emphasis role="bold">Description</emphasis>
+                </para>
+                </entry>
+                </row>
+            </thead>
+            <tbody>
+                <row>
+                <entry><para>FPGACARD</para></entry>
+                <entry><para>Short card name</para></entry>
+                </row>
+                <row>
+                <entry><para>FPGACHIP</para></entry>
+                <entry><para>FPGA part name, for example, xcvu9p-fsgd2104-2L-e</para></entry>
+                </row>
+                <row>
+                <entry><para>Flash Type and IO pins</para></entry>
+                <entry><para>Flash chip that attached to FPGA, for example mt28gu01gaax1e-bpi-x16. And the related xdc files for FPGA config.</para></entry>
+                </row>
+                <row>
+                <entry><para>PCIe Config and IO pins</para></entry>
+                <entry><para>The tcl/xdc information about the PCIe hardware IP for this card. </para></entry>
+                </row>
+                <row>
+                <entry><para>DDR MC IP</para></entry>
+                <entry><para>DDR memory controller Vivado IP tcl/xdc file. </para></entry>
+                </row>
+                <row>
+                <entry><para>Other peripherals</para></entry>
+                <entry><para>NVMe IP, Ethernet IP and so on (Optional)</para></entry>
+                </row>
+            </tbody>
+            </tgroup>
+        </table>
+        <note><para>DDR MC and other peripheral IP's configurations are not needed at the beginning. They are not in the capi_bsp_wrap diagram. However, when you query information from the FPGA Supplier, it's wise to include them also. </para></note>
+    </section>
+    <section><title>Download PSL Zip package</title>
+        <para>Download PSL9 Zip package from <link xlink:href="https://www-355.ibm.com/systems/power/openpower/posting.xhtml?postingId=1BED44BCA884D845852582B70076A89A">OpenPower Portal</link> and put it in "capi2-bsp/psl" directory. </para>
+    </section>
+    <section><title>Copy and modify</title>
+        <para>Choose an existing card as a base. Copy the folder to your new FPGACARD name. Then modify the contents according to the information collected from FPGA supplier.</para>
+        <important><para> Make sure the information in xdc/tcl files are permitted to be open-source. </para></important>
+        <para>There are some other modifications you should pay attention to:</para>
+        <orderedlist>
+                <listitem><para><emphasis role="underline">PCIe core IP creation:</emphasis></para>
+            <itemizedlist>
+                <listitem><para>"Vendor ID" and "Device ID" have to be 0x1014 and 0x0477, so kernel module cxl can recognize the card as a CAPI device. (See in <link xlink:href="https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/tree/drivers/misc/cxl/pci.c">pci.c</link>)</para></listitem>
+                <listitem><para>If the card vendor has a code allocated by PCISIG (See in <link xlink:href="https://pcisig.com/membership/member-companies">PCISIG Member companies</link>), use it as "Subsystem Vendor ID". "Subsystem Device ID" can be chosen freely. </para></listitem>
+                <listitem><para>If the card vendor doesn't have a code allocated by PCISIG, or just for testing and evaluation purpose, please use default "Subsystem Vendor ID" = 0x1014, and send email to <email>aclwg-chair@openpowerfoundation.org</email> to get a distinct "Subsystem Device ID" to differentiate this card from others.</para>
+                <para>Example: (in create_ip.tcl)</para>
+                <screen>create_ip -name pcie4_uscale_plus -vendor xilinx.com -library ip -module_name pcie4_uscale_plus_0 -dir $ip_dir >> $log_file
+set_property -dict [list                                               \
+                    CONFIG.PF0_CLASS_CODE {1200ff}                     \
+                    CONFIG.PF0_REVISION_ID {02}                        \
+                    CONFIG.VENDOR_ID {1014}                            \
+                    CONFIG.PF0_DEVICE_ID {0477}                        \
+                    CONFIG.PF0_SUBSYSTEM_VENDOR_ID {1014}              \
+                    CONFIG.PF0_SUBSYSTEM_ID {0661}                     \
+                    ......                                             \
+                    ......                                             \
+                   ] [get_ips pcie4_uscale_plus_0] >> $log_file</screen>
+           <para>The corresponding  "Subsytem Vendor ID" and "Subsystem Device ID" need to be added into <link xlink:href="https://github.com/ibm-capi/capi-utils">capi-utils</link>, file "psl-devices".</para>
+                </listitem>
+            </itemizedlist>
+        </listitem>
+        <listitem><para><emphasis role="underline">Product Family support:</emphasis></para>
+         <para>If the FPGA chip types are Xilinx VU33P or VU37P who have HBM, this is actually a new FPGA family <userinput>virtexuplushbm</userinput>. For a new FPGA Production family, additional steps need to take:</para>
+        <itemizedlist>
+                <listitem><para>"capi2-bsp/psl/create_ip.tcl": <userinput>set_property supported_families ...</userinput>, add new family name like "virtexuplushbm Production"</para></listitem>
+                <listitem><para>"capi2-bsp/common/tcl/create_capi_bsp.tcl": <userinput>set_property supported_families ...</userinput>, do the same as above.</para></listitem>
+        <listitem><para>Add family support to PSL9 ZIP package: unzip the package, do the modifications, and zip them back again. Commands:</para>
+        <screen>$ unzip ibm.com_CAPI_PSL9_WRAP_2.00.zip
+(modify compnent.xml to add new family name, search "supportedFamilies")
+$ zip -r ibm.com_CAPI_PSL9_WRAP_2.00.zip component.xml src/ xgui/
+$ rm -fr component.xml src/ xgui/</screen>
+        </listitem>
+        </itemizedlist>
+        </listitem>
+
+        <listitem><para><emphasis role="underline">VSEC starting address: </emphasis></para>
+                <para>VSEC (Vendor Specific Extended Capability Structure) is a part of PCIe capability list architecture. It needs to be properly linked in PCIe config space. </para> <para>"capi2-bsp/[FPGACARD]/src/capi_vsec.vhdl": <userinput>vsec_addr[21:32]</userinput> defines the address for VSEC. It should be matched with PCIe core value <userinput>PF0_SECONDARY_PCIE_CAP_NEXTPTR</userinput>. Take card U200 for example, its <userinput>vsec_addr[21:32]</userinput> starts from 12'h400 (12'b0100_0000_0000), and "tcl/patch_ip.tcl" modifies it from default value 12'h480 to 12'h400." </para>
+                <screen>exec /bin/bash -c "sed -i \"s/PF0_SECONDARY_PCIE_CAP_NEXTPTR=0x480/PF0_SECONDARY_PCIE_CAP_NEXTPTR=0x400/\" $pcie_source"
+exec /bin/bash -c "sed -i \"s/PF0_SECONDARY_PCIE_CAP_NEXTPTR('H480)/PF0_SECONDARY_PCIE_CAP_NEXTPTR('H400)/\" $pcie_source"</screen>
+                <para>Xilinx PCIe code information for extended configuration space can be found on <link xlink:href="https://www.xilinx.com/support/documentation/ip_documentation/pcie3_ultrascale/v4_4/pg156-ultrascale-pcie-gen3.pdf">PG156 (for Ultrascale Device)</link> or <link xlink:href="https://www.xilinx.com/support/documentation/ip_documentation/pcie4_uscale_plus/v1_3/pg213-pcie4-ultrascale-plus.pdf">PG213 (for Ultrascale+ Device)</link>.</para>
+                <para>For Ultrascale+ HBM device's pcie4c core, the VSEC starts from 12'hE80. At this time <userinput>vsec_addr[21:32]</userinput> must be changed in "capi_vsec.vhdl". And the above two lines in "patch_ip.tcl" are not needed anymore.</para>
+
+        </listitem>
+        <listitem>
+                <para><emphasis role="underline">Vital Product Data: </emphasis><emphasis  role="bold">This step is optional.</emphasis></para>
+                <para>"capi2-bsp/[FPGACARD]/src/capi_vsec.vhdl": Edit the hardcoded <userinput>vpd44data_be</userinput> to add VPD (Vital Product Data) information. Ideally this information should be read from an I2C EEPROM. The FPGA supplier wrote the content of EEPROM before shipping. Today the simpliest way is taken -- writing some hard coded value. "capi2-bsp/common/script" has a script "gen_vsec.sh" to do this. </para>
+        </listitem>
+
+        <listitem>
+                <para><emphasis role="underline">User Image Address: </emphasis></para>
+                <para>"capi2-bsp/[FPGACARD]/src/capi_xilmltbt.vhdl": Edit the User image starting address <userinput>wbstart_addr</userinput>. </para>
+        <screen>wbstart_addr &lt;=  "User_image_address"  when (cpld_user_bs_req = '1') else "00000000000000000000000000000000";</screen>
+        <para>"capi_xilmltbt.vhdl" has a Xilinx multi-boot core. That means you can create two kinds of images: Factory image and User image. Factory images will be placed at address 0 of FPGA Flash, and User image will be placed at "User_image_address" on the flash. When power-on or the FPGA card is reset, the multiboot core knows where to load the image. Usually a Golden factory image is put on address 0 and is never changed. Multiboot core always tries to load user image first. If the user image has something wrong, multiboot logic will tell the FPGA to "fallback" to factory image. You still see the card in the system and you can just program a new user image to try again. </para>
+        </listitem>
+
+        <listitem>
+                <para><emphasis role="underline">Check Vivado Version:</emphasis></para>
+                <para>Make sure this version of Vivado tool supports the FPGA part name you have assigned in "capi2-bsp/[FPGACARD]/Makefile". For some very new FPGA chip types, in one Vivado version they may have a suffix of "es" (engineering sample), and in a newer Vivado version the "es" suffix is removed.</para>
+        </listitem>
+        </orderedlist>
+    </section>
+    <section><title>Generate capi_bsp_wrap</title>
+        <screen>cd capi2-bsp
+make [FPGACARD]</screen>
+        <para>If it is successfully done, the generation of BSP for CAPI2.0 is completed. Developers using HDK mode can create their own Vivado project and import capi_bsp_wrap as an IP. But for SNAP developers there are some other work to do, see in next chapter.</para>
+    </section>
+</section>
+
+</chapter>
+
+
+
+

enable_capi_snap/ch_capi20_snap.xml

@@ -0,0 +1,432 @@
+<!--
+    Copyright (c) 2016 OpenPOWER Foundation
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<chapter version="5.0" xml:lang="en" xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"
+                                     xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="chapter_capi20_snap">
+
+<!-- Chapter Title goes here. -->
+<title  xml:id="chapter_capi20_snap_title">Enable CAPI2.0 SNAP</title>
+
+<section><title>Work on github</title>
+    <para>Snap is also a public Github repository. Create a "fork" (Click the "fork" button) on <link xlink:href="https://github.com/open-power/snap">https://github.com/open-power/snap</link>. Keep working on your own snap fork, when it works, submit a pull request to "open-power/snap" and require merging into the public upstream.</para>
+    <screen>git clone https://github.com/[YOUR_USERNAME]/snap</screen>
+    <note>
+    <para>capi2-bsp is a submodule of snap. It is shown in ".gitmodules" file (this is a hidden file). Please point it to your own capi2-bsp fork. Then </para>
+    <screen>git submodule init
+git submodule update</screen>
+    <para>Anyway, make sure that "hardware/capi2-bsp" is the one just generated in last chapter.</para>
+    </note>
+</section>
+
+<section><title>SNAP structure</title>
+    <para>On the FPGA side, there are three parts that need to consider when moving to a new FPGA card. They are (a) BSP, (b) snap_core, (c) DDR memory controller (mig). And there are also some components in SNAP need to be updated for a new FPGA card. </para>
+    <figure pgwide="1" xml:id="psl_fpga">
+        <title>Project hierarchy for SNAP</title>
+        <mediaobject>
+        <imageobject>
+        <imagedata fileref="figures/psl_fpga.png" format="PNG" scalefit="1" width="90%" align="center" />
+        </imageobject>
+        </mediaobject>
+</figure>
+    <note><para>Module <userinput>snap_core</userinput> on CAPI2.0 implemented the data path with DMA interface. Buffer interface is not used. </para></note>
+
+    <para>The following picture shows the SNAP github repository folders and files.</para>
+    <figure pgwide="1" xml:id="snap-str">
+        <title>Repository structure</title>
+        <mediaobject>
+        <imageobject>
+        <imagedata fileref="figures/snap-structure_white.png" format="PNG" scalefit="1" width="90%" align="center" />
+        </imageobject>
+        </mediaobject>
+    </figure> 
+    <para>All of the user-developed accelerators are in "<emphasis role="bold">actions</emphasis>" directory. There are already some examples there. Each "action" has its "sw", "hw", "tests", and other sub-directories. The hardware part uses "action_wrapper" as its top.</para>
+    <para>"<emphasis role="bold">software</emphasis>" directory includes libsnap, header files and some tools. "<emphasis role="bold">hardware</emphasis>" directory is the main focus. "<emphasis role="bold">deconfig</emphasis>" has the config files for silent testing purpose, and "<emphasis role="bold">scripts</emphasis>" has the menu settings and other scripts. </para>
+    <para>
+    How does SNAP work and what are the files used in each step? 
+    </para>
+    <itemizedlist>
+    <listitem><para><userinput>make snap_config</userinput>: The menu to select cards and other options is controlled by "script/Kconfig"</para></listitem>
+    <listitem>
+            <para><userinput>make model</userinput>: This step creates a Vivado project. It firstly calls "hardware/setup/create_snap_ip.tcl" to generate the IP files in use, then calls "hardware/setup/create_framework.tcl" to build the project. About "create_framework.tcl": </para>
+    <itemizedlist>
+    <listitem>
+    <para>It adds BSP (board support package). In CAPI1.0, it is also called PSL Checkpoint file (b_route_design.dcp) or base_image. It uses the path pointed to b_route_design.dcp and adds it into the design. In CAPI2.0, it will call the make process in capi2-bsp submodule to generate "capi_bsp_wrap" if it doesn't exist. This step is skipped if "capi_bsp_wrap" is already generated. Then "create_framework.tcl"  adds the capi_bsp_wrap (xcix or xci file) into the design.</para>
+    </listitem>
+    <listitem>
+    <para>It adds FPGA top files and snap_core files (in hardware/hdl/core).</para>
+    </listitem>
+    <listitem>
+    <para>It adds constrain files: in "hardware/setup/[FPGACARD]" or in "hardware/capi2-bsp/[FPGACARD]"</para>
+    </listitem>
+    <listitem>
+    <para>It adds user files (in "actions/[ACTION_NAME]/hw"). User's action hardware uses top file named "action_wrapper.vhd"</para>
+    </listitem>
+    <listitem>
+            <para>It adds simulation files (in "hardware/sim/core") including simulation top files and simulation models. (If <userinput>no_sim</userinput> is selected in snap_config menu, this step is skipped.)</para>
+    </listitem>
+    </itemizedlist>
+    <para>After above steps, "<emphasis role="bold">hardware/viv_project</emphasis>" is created. Open it with Vivado GUI, and check the design hierarchy. And it will call the selected simulator to compile the simulation model.</para>
+    </listitem>
+    <listitem>
+            <para><userinput>make image</userinput>: This step runs synthesis, implementation and bitstream generation. It calls "hardware/setup/snap_build.tcl" and also uses some related tcl scripts to work together. In this step, "<emphasis role="bold">hardware/build</emphasis>" will be created and the output products like bit images, checkpoints (middle products for debugging) and reports (reports of timing, clock, IO, utilization, etc.) If everything runs well and timing passes, user will get the bitstream files (in "build/Images" sub directory) to program the FPGA card. </para>
+    </listitem>
+    </itemizedlist>
+</section>
+<section><title>Modifications to snap git repositories</title>
+    <para>For a new FPGA card, the detailed items to update are listed as below.</para>
+    <itemizedlist>
+    <listitem><para>Hardware RTL, setup, simulation</para></listitem>
+    <listitem><para>Software and tools</para></listitem>
+    <listitem><para>Testing</para></listitem>
+    <listitem><para>Publishing</para></listitem>
+    </itemizedlist>
+
+        <para>The best way is to grep some keywords like "S241" or "AD8K5" under the directories and look for the locations that need modifications.</para>
+        <note>
+        <para>A file ending with "_source", like "psl_fpga.vhd_source", means this file will be pre-processed to generate the output file without "_source" suffix, like "psl_fpga.vhd". There are <userinput>#ifdef</userinput> macros or comments like <userinput>-- only for NVME_USED=TRUE</userinput>. They help to create a target VHDL/Verilog file with different configurations.</para>
+        </note>
+        <para>Below lists the files to change:</para>
+        <itemizedlist>
+        <listitem><para>snap_config and environmental files</para></listitem>
+        <listitem><para>Hardware: psl_accel and psl_fpga (top) RTL files</para></listitem>
+        <listitem><para>Hardware: tcl files for the workflow</para></listitem>
+        <listitem><para>Hardware: xdc files for IO, floorplan, clock and bitstream settings</para></listitem>
+        <listitem><para>Hardware: create DDR Memory controller IP (mig) in create_snap_ip.tcl, create DDR memory sim model, and other xdc files</para></listitem>
+        <listitem><para>Hardware: create_ip, sim model and xdc files for other IPs</para></listitem>
+        <listitem><para>Software: New card type, register definition</para></listitem>
+        <listitem><para>Testing: jenkins</para></listitem>
+        <listitem><para>Readme and Documents</para></listitem>
+        </itemizedlist>
+
+        <table frame="all" pgwide="1" xml:id="filelist">
+            <title>Config files to change</title>
+            <tgroup cols="2">
+            <colspec colname="c2" colwidth="25*" />
+            <colspec colname="c3" colwidth="25*" />
+            <thead>
+                <row>
+                <entry>
+                <para>
+                <emphasis role="bold">File name</emphasis>
+                </para>
+                </entry>
+                <entry>
+                <para>
+                <emphasis role="bold">Changes to do</emphasis>
+                </para>
+                </entry>
+                </row>
+            </thead>
+            <tbody>
+                <row>
+                <entry><para>scripts/Kconfig</para></entry>
+                <entry><para>adding card to the Kconfig menu. Provide Flash information (size/type/user address)</para></entry>
+                </row>
+                <row>
+                <entry><para>hardware/doc/SNAP-Registers.md</para></entry>
+                <entry><para>SNAP registers for new card - doc</para></entry>
+                </row>
+                <row>
+                <entry><para>hardware/setup/snap_config.sh</para></entry>
+                <entry><para>SNAP registers - setting</para></entry>
+                </row>
+            </tbody>
+            </tgroup>
+        </table>
+
+        <table frame="all" pgwide="1" xml:id="rtlchange">
+            <title>RTL/xdc/tcl files to change</title>
+            <tgroup cols="2">
+            <colspec colname="c2" colwidth="25*" />
+            <colspec colname="c3" colwidth="25*" />
+            <thead>
+                <row>
+                <entry>
+                <para>
+                <emphasis role="bold">File name</emphasis>
+                </para>
+                </entry>
+                <entry>
+                <para>
+                <emphasis role="bold">Changes to do</emphasis>
+                </para>
+                </entry>
+                </row>
+            </thead>
+            <tbody>
+                <row><entry><para>hardware/hdl/core/psl_accel_${FPGACARD}.vhd_source</para></entry><entry><para> specific to card</para></entry></row>
+                <row><entry><para>hardware/hdl/core/psl_accel_types.vhd_source</para></entry><entry><para>specific to card</para></entry></row>
+                <row><entry><para>hardware/hdl/core/psl_fpga_${FPGACARD}.vhd_source</para></entry><entry><para> specific to card</para></entry></row>
+                <row><entry><para>hardware/setup/${FPGACARD}/capi_bsp_pblock.xdc</para></entry><entry><para> specific to card</para></entry></row>
+                <row><entry><para>hardware/setup/${FPGACARD}/snap_${FPGACARD}.xdc</para></entry><entry><para> specific to card</para></entry></row>
+                <row><entry><para>hardware/setup/${FPGACARD}/snap_ddr4pins.xdc</para></entry><entry><para> specific to card</para></entry></row>
+                <row><entry><para>hardware/setup/build_mcs.tcl</para></entry><entry><para>declare card name</para></entry></row>
+                <row><entry><para>hardware/setup/create_framework.tcl</para></entry><entry><para>declare card name</para></entry></row>
+                <row><entry><para>hardware/setup/create_snap_ip.tcl</para></entry><entry><para>declare card name and the IPs in use</para></entry></row>
+                <row><entry><para>hardware/setup/flash_mcs.tcl</para></entry><entry><para>declare card name</para></entry></row>
+                <row><entry><para>hardware/setup/snap_bitstream_post.tcl</para></entry><entry><para>declare card name</para></entry></row>
+                <row><entry><para>hardware/setup/snap_bitstream_pre.tcl</para></entry><entry><para>declare card name</para></entry></row>
+                <row><entry><para>hardware/setup/snap_bitstream_step.tcl</para></entry><entry><para>declare card name</para></entry></row>
+                <row><entry><para>hardware/setup/snap_impl_step.tcl</para></entry><entry><para>declare card name</para></entry></row>
+                <row><entry><para>hardware/sim/ddr4_dimm_???.sv</para></entry><entry><para>DDR memory model for simulation. Please get the information about how many DDR chips are connected together, the density and data width of each chip, and whether there is one chip is used for ECC (redundant). Take an existing one as a template and modify.</para></entry></row>
+                <row><entry><para>hardware/sim/top_capi?0.sv_source</para></entry><entry><para>Instantiate the DDR memory model</para></entry></row>
+                <row><entry><para>hardware/snap_check_psl (Only for CAPI1.0)</para></entry><entry><para>declare card name</para></entry></row>
+            </tbody>
+            </tgroup>
+        </table>
+
+        <table frame="all" pgwide="1" xml:id="swchange">
+            <title>Software files to change</title>
+            <tgroup cols="2">
+            <colspec colname="c2" colwidth="25*" />
+            <colspec colname="c3" colwidth="25*" />
+            <thead>
+                <row>
+                <entry>
+                <para>
+                <emphasis role="bold">File name</emphasis>
+                </para>
+                </entry>
+                <entry>
+                <para>
+                <emphasis role="bold">Changes to do</emphasis>
+                </para>
+                </entry>
+                </row>
+            </thead>
+            <tbody>
+                <row><entry><para>software/lib/snap.c</para></entry><entry><para>declare card name</para></entry></row>
+                <row><entry><para>software/tools/snap_find_card</para></entry><entry><para>declare card name + SUBSYSTEM_ID</para></entry></row>
+                <row><entry><para>software/include/snap_regs.h</para></entry><entry><para>SNAP registers - setting</para></entry></row>
+            </tbody>
+            </tgroup>
+        </table>
+
+        <table frame="all" pgwide="1" xml:id="otherchange">
+            <title>Other files to change</title>
+            <tgroup cols="2">
+            <colspec colname="c2" colwidth="25*" />
+            <colspec colname="c3" colwidth="25*" />
+            <thead>
+                <row>
+                <entry>
+                <para>
+                <emphasis role="bold">File name</emphasis>
+                </para>
+                </entry>
+                <entry>
+                <para>
+                <emphasis role="bold">Changes to do</emphasis>
+                </para>
+                </entry>
+                </row>
+            </thead>
+            <tbody>
+                <row><entry><para>actions/scripts/snap_jenkins.sh</para></entry><entry><para>jenkins tests (optional)</para></entry></row>
+                <row><entry><para>defconfig/${FPGACARD}*.defconfig</para></entry><entry><para>For silent jenkins testing (optional)</para></entry></row>
+                <row><entry><para>README.md</para></entry><entry><para>Announce a new card is supported </para></entry></row>
+            </tbody>
+            </tgroup>
+        </table>
+</section>
+<section><title>Update capi-utils</title>
+    <para>capi-utils is the third git repository that needs a few modifications. Same as before, fork it, make the modifications and submit a pull request.</para>
+    <screen>git clone https://github.com/[YOUR_USERNAME]/capi-utils</screen>
+    <para>There is only one file to be modified: "psl-devices". Add a new line, for example</para>
+    <screen>0x1014 0x0665 U200 Xilinx 0x1002000 64 SPIx4</screen>
+    <para>It lists the Subsystem Vendor ID, Subsystem Device ID, Card name, FPGA chip, then it is the "User_image_address" on the flash. For SPI device, size of block is 64Bytes. "SPIx4" is the flash interface type. It may also be "DPIx16" or "SPIx8". </para>
+    <para>"SPIx8" uses two bitstreams so another starting address also needs to be provided. Script "capi-flash-script" needs two input bitstream files (primary and secondary) to program the flash.</para>
+
+</section>
+<section><title>Strategy to enable a new card</title>
+    <para>
+    To enable a new card on SNAP, complete following tasks one by one.
+    </para>
+    <section><title>Stage 1: Verify PCIe interface</title>
+        <orderedlist>
+        <listitem><para>Generate capi_bsp_wrap in capi2-bsp.</para></listitem>
+        <listitem><para>Make modifications to snap git repository as described above.</para></listitem>
+        <listitem><para>Select an action example without DDR, for example: hls_helloworld. </para></listitem>
+        <listitem><para>Go through the <userinput>make model</userinput> and <userinput>make image</userinput> processes and build the bitstream files. </para></listitem>
+        <listitem><para>Plug the card onto Power9 server and connect a JTAG/USB cable to a laptop. Install Vivado Lab on this laptop (it requires Windows or Linux operating system). Start Vivado Lab tool, open Hardware manager.</para></listitem>
+        <listitem><para>Power on the server. Soon the FPGA target is recognized by Vivado Lab tool.</para></listitem>
+        <listitem><para>Program the generated bitstream files (bin or mcs) to the card. On Vivado Lab tool, select the FPGA chip and right-click, choose "Add Configuration Memory Device..." and program the bin or mcs files to the flash. See in picture <xref linkend="vivado-lab"/> and <xref linkend="jtag"/> </para></listitem>
+        <listitem><para>Wait it done (It may take 10 minutes). Unplug the JTAG/USB cable, reboot the server.</para></listitem>
+        <listitem><para>After the server is booted, log into OS, run <userinput>lspci</userinput> to see if the card is there. (Usually with Device ID 0x0477). Then download snap, capi-utils, libcxl (from github). Go to snap directory, <userinput>make apps</userinput> and run the application. </para></listitem>
+        </orderedlist>
+        <note><para>There is another way to replace step 6 to 8 which is called <emphasis role="underline">"Fast program bit-file when power on"</emphasis>. Prepare the <emphasis role="bold">bit</emphasis> file on laptop in advance. Not like bin/mcs files which are for the flash, the bit file is used to program the FPGA chip directly. When the server is powered on, after Vivado Lab sees the FPGA, right click the device, "program device..." and select the bit file <emphasis role="bold">immediately</emphasis>. This action only takes about 10 seconds and can be done before skiboot on the server starts to scan PCIe devices.</para>
+        <para>Be aware of the fact that now only FPGA chip is programmed, (the flash memory is still empty or holding old data), so when the server is powered off or reboot the recent programming to FPGA chip will be lost.</para>
+        </note>
+
+        <figure pgwide="1" xml:id="vivado-lab">
+            <title>Vivado Lab Edition</title>
+            <mediaobject>
+            <imageobject>
+            <imagedata fileref="figures/vivado-lab.png" format="PNG" scalefit="1" width="70%" align="center" />
+            </imageobject>
+            </mediaobject>
+        </figure> 
+        <figure pgwide="1" xml:id="jtag">
+            <title>Add Configuration Memory Device and Program the flash</title>
+            <mediaobject>
+            <imageobject>
+            <imagedata fileref="figures/jtag.png" format="PNG" scalefit="1" width="100%" align="center" />
+            </imageobject>
+            </mediaobject>
+        </figure> 
+
+        <important>
+        <para>When installing <emphasis role="bold">Vivado Lab</emphasis>, please choose as same version as the Vivado tool which was used to build images. </para>
+        </important>
+        <para> <emphasis role="underline">Tips to debugging:</emphasis></para>
+        <orderedlist>
+                <listitem><para>Seeing 0477 by <userinput>lspci</userinput> is the most important milestone. If not, please check file "<emphasis role="bold">/sys/firmware/opal/msglog</emphasis>" to see whether there are link training failed messages. A successful message looks like this, which means this PCIe device has been scanned and recognized. The number followed "PHB#" is the PCIe device identifier in the format of "domain:bus:slot.func":</para>
+<screen>[   63.403485191,5] PHB#0000:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..01 SLOT=CPU1 Slot2 (16x)
+[   63.403572553,5] PHB#0000:01:00.0 [EP  ] 1014 0477 R:02 C:1200ff (        device) LOC_CODE=CPU1 Slot2 (16x)</screen>
+                </listitem>
+                <listitem> <para> Check <userinput>dmesg</userinput>. Run "dmesg &gt; dmesg.log" and search "cxl" in "dmesg.log" file. A normal output should be look like this:</para>
+                        <screen>[    9.301403] cxl-pci 0000:01:00.0: Device uses a PSL9
+[    9.301523] cxl-pci 0000:01:00.0: enabling device (0140 -> 0142)
+[    9.303327] cxl-pci 0000:01:00.0: PCI host bridge to bus 0006:00
+[    9.306749] cxl afu0.0: Activating AFU directed mode</screen>
+                </listitem>
+                <listitem><para>Today most of the linux kernel versions already include cxl module. Doublecheck it by:</para>
+                        <screen>modinfo cxl</screen>
+                </listitem>
+
+        <listitem>
+         <para>If the PCIe device has been recognized as CAPI, <prompt>ls /dev/cxl</prompt> and "afu*" devices should be there. Then application software can open the device like operating an ordinary file.</para>
+        <screen>ls /dev/cxl
+afu0.0m  afu0.0s</screen>
+        <para>Some other useful commands to check PCIe config (with the right PCIe identifier "domain:bus:slot.func") </para>
+        <screen>sudo lspci -s 0000:01:00.0 -vvv</screen>
+        <para>It shows the settings coded in Xilinx PCIe core, like Subsystem Device ID:</para>
+        <screen>0000:01:00.0 Processing accelerators: IBM Device 0477 (rev 02) (prog-if ff)
+	Subsystem: IBM Device 0660</screen>
+        <para>Link Speed</para>
+        <screen>LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-</screen>
+        <para>Vital Product Data which was coded in capi_vsec.vhdl</para>
+        <screen>Capabilities: [b0] Vital Product Data
+Product Name: U200 PCIe CAPI2 Adapter
+Read-only fields:
+	[PN] Part number: Xilinx.U200
+	[V1] Vendor specific: 0000000000000000
+	[V2] Vendor specific: 0000000000000000
+	[V3] Vendor specific: 0000000000000000
+	[V4] Vendor specific: 0000000000000000
+	[RV] Reserved: checksum good, 3 byte(s) reserved
+End</screen>
+        <para>And see VSEC and kernel module:</para>
+        <screen>Capabilities: [400 v1] Vendor Specific Information: ID=1280 Rev=0 Len=080 &lt;?&gt;
+Kernel driver in use: cxl-pci
+Kernel modules: cxl</screen>
+</listitem>
+        <listitem>
+                <para>If nothing shows by <userinput>ls /dev/cxl</userinput>, check PCIe config space:</para>
+        <screen>sudo hexdump /sys/bus/pci/devices/0000\:00\:00.1/config
+        </screen>
+        <para>Please pick up the correct PCIe device identifier (0000:00:00.1). Make sure the VSEC is properly linked. If not, go back to check "capi_vsec.vhdl".</para>
+        <screen>0000000 1014 0477 0146 0010 ff02 1200 0000 0000
+0000010 000c 0000 2200 0006 000c 1000 2200 0006
+0000020 000c 0000 0000 0002 0000 0000 1014 0668
+0000030 0000 0000 0040 0000 0000 0000 00ff 0000
+0000040 4801 0003 0008 0000 7005 0080 0000 0000
+0000050 0000 0000 0000 0000 0000 0000 0000 0000
+0000060 7011 0000 0000 0000 0000 0000 0000 0000
+0000070 b010 0002 8022 0000 2950 0000 f103 0043
+0000080 0000 1103 0000 0000 0000 0000 0000 0000
+0000090 0000 0000 0016 0000 0010 0000 000e 0000
+00000a0 0003 001e 0000 0000 0000 0000 0000 0000
+00000b0 0003 0000 2082 5300 0000 0000 0000 0000
+00000c0 0000 0000 0000 0000 0000 0000 0000 0000
+*
+0000100 0001 1c01 0000 0000 0000 0044 2030 0046 --> next_ptr: 1c0
+0000110 0000 0000 e000 0000 0000 0000 0000 0000
+0000120 0000 0000 0000 0000 0000 0000 0000 0000
+*
+00001c0 0019 1f01 0000 0000 0000 0000 0000 0000 --> next_ptr: 1f0
+00001d0 0000 0000 0000 0000 0000 0000 0000 0000
+*
+00001f0 0002 e801 0000 0000 0000 3100 0000 0000 --> e80 (or 400) points to VSEC
+0000200 0000 0000 00ff 8000 0000 0000 0000 0000
+0000210 0000 0000 0000 0000 0000 0000 0000 0000
+*
+0000e80 000b 0001 1280 0800 0801 0021 0006 0200 --> VSEC starts from e80 (or 400)
+0000e90 0000 b000 0000 0000 0000 0000 0000 0000
+0000ea0 0100 0000 0040 0000 0200 0000 0400 0000
+0000eb0 0000 0000 0000 0000 0000 0000 0000 0000
+*
+0000ed0 0000 0000 0000 0000 0000 8000 0000 0000
+0000ee0 0000 0000 0000 0000 0000 0000 0000 0000
+*
+0001000</screen>
+        </listitem>
+        </orderedlist>
+
+    </section>
+    <section><title>Stage 2: Verify Flash interface</title>
+        <para>Use <link xlink:href="https://github.com/ibm-capi/capi-utils">capi-utils</link> to program the bitstream files. If it succeeds, it proves that the Flash interface has been configured correctly. After this step, JTAG connector is not needed anymore. Use "capi-flash-script" to program the FPGA bitstreams. </para>
+        <para>The mechanic behind "capi-flash-script" is: </para>
+        <para>There is a flash controller on FPGA (in capi_bsp_wrap), and it connects to PCIe config space. The flash controller exposes four VSEC registers to allow host system to control. They are: </para>
+        <itemizedlist>
+                <listitem><para>Flash Address Register</para></listitem>
+                <listitem><para>Flash Size Register</para></listitem>
+                <listitem><para>Flash Status/Control Register</para></listitem>
+                <listitem><para>Flash Data Port</para></listitem>
+        </itemizedlist>
+        <para>The details are decribed in <link xlink:href="http://openpowerfoundation.org/wp-content/uploads/resources/hwarch-caia-spec/content/ch_preface.html">Coherent Accelerator Interface Architecture</link>, Chapter 12.3, "CAIA Vendor-Specific Extended Capability Structure". So the C file in <link xlink:href="https://github.com/ibm-capi/capi-utils">capi-utils</link> reads FPGA bitstream "bin" file, and writes the data to VSEC "Flash Data Port" register. So the bytes are sent through PCIe, to Flash controller and finally arrive to flash memory on the card.</para>
+    </section>
+
+    <section><title>Stage 3: Verify DDR interface</title>
+        <orderedlist>
+        <listitem><para>Select another action example (hdl_example with DDR) or hls_memcopy.</para></listitem>
+        <listitem><para><userinput>make model</userinput> and <userinput>make sim</userinput>. Make sure the DDR simulation model works well.</para></listitem>
+        <listitem><para><userinput>make image</userinput> to generate the bitstream files.</para></listitem>
+        <listitem><para>Use capi-utils to program the bitstream "bin" file to the card.</para></listitem>
+        <listitem><para>Run the application to see whether it works.</para></listitem>
+        </orderedlist>
+        <para>Basically SNAP only implemented one DDR Bank (or channel) while most cards have two to four banks. (N250S+ is one of the rare card which has only one DDR bank). To implement more DDR channels, depending on user's needs, there are two options: the first is to just extend the size of the first bank by adding this second bank on the same DDR memory controller. The other option is to use two (or more) memory controllers in parallel to have a higher throughput. This later option means duplicating the DDR memory controller in place and this will take twice the place in the design. In this case, the action_wrapper also requires change to add the additional DDR ports. For HLS design, another HLS DDR port should be added into "actions/[YOUR_ACTION]/hw/XXX.CPP". As for an opensource project, everyone is welcomed to add your contribution by implementing it and add it to the SNAP design.</para>
+    </section>
+
+
+
+    <section><title>Stage 4: Verify Other IO interface</title>
+        <para>This step is decided by the card's capabilities and the specific IOs that the card can provide. Like the second or more DDR channels, user can add their code for other IO interface freely. </para>
+    </section>
+
+
+    <section><title>Stage 5: Performance Validation</title>
+        <para>Check the result of "snap/actions/hls_memcopy/tests/test_*_throughput.sh" for bandwidth and "snap/actions/hls_latency_eval/test/test*.sh" for latency.  </para>
+    </section>
+
+    <section><title>Stage 6: Pressure Test</title>
+        <para>Prepare bitstream files for basic tests, throughput tests, latency tests, max-power tests. Adding image flashing tests, card reset tests and others. Run them intensively.</para>
+    </section>
+</section>
+<section><title>Cleanup and submit</title>
+    <para>Now a new FPGA card has been enabled to CAPI2.0 SNAP. Cleanup your workspace, check files and submit your work!</para> 
+    <itemizedlist>
+    <listitem><para>Submit the pull request of your "capi2-bsp fork" before "snap fork". Assign the reviewer and wait capi2-bsp to be merged into https://github.com/open-power/capi2-bsp master branch</para></listitem>
+    <listitem><para>Update the submodule pointer to the latest "open-power/capi2-bsp" master and then submit the pull request of your forked snap.</para></listitem>
+    <listitem><para>Capi-utils is independent. Just create a pull request and assign a reviewer. It can only been merged into master branch after having been reviewed.</para></listitem>
+    </itemizedlist>
+    
+</section>
+
+</chapter>
+
+
+

enable_capi_snap/ch_example.xml

@@ -0,0 +1,323 @@
+<!--
+  Copyright (c) 2016 OpenPOWER Foundation
+  
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  
+-->
+<chapter version="5.0" xml:lang="en" xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"
+xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="section_template_examples">
+  
+  <!-- Chapter Title goes here. -->
+  <title>Enable CAPI2 card</title>
+ 
+  
+  <section>
+    <title>Section Title goes here</title>
+    <para>This Section covers something of interest to a limited number of people and shows a 1st level section</para>
+
+    <section xml:id="list_example_label">
+      <title>Example Itemized List</title>
+      <para>
+      Here is an example of an itemized list</para>
+      <itemizedlist>
+        <title>A list title is completely optional</title>
+        <listitem>
+          <para>
+          Item you don't care about</para>
+          <itemizedlist>
+            <listitem>
+              <para>Perhaps you'd like a sub-list</para>
+              <itemizedlist>
+                <listitem>
+                  <para>Oooh, here's about another</para>
+                </listitem>
+              </itemizedlist>
+            </listitem>
+          </itemizedlist>
+        </listitem>
+        <listitem>
+          <para>
+          Item you might care about </para>
+        </listitem>
+        <listitem>
+          <para>
+          Item you do care about </para>
+        </listitem>
+      </itemizedlist>
+    </section>
+    <section>
+      <title>Example ordered list</title>
+      <para>
+      All good documents need ordered lists.</para>
+      <orderedlist>
+        <title>Another purely optional title</title>
+        <listitem>
+          <para>First item</para>
+        </listitem>
+        <listitem>
+          <para>Second item</para>
+          <orderedlist numeration="loweralpha">
+            <listitem>
+              <para>first indented item</para>
+            </listitem>
+            <listitem>
+              <para> second indented item</para>
+            </listitem>
+          </orderedlist>
+        </listitem>          
+        <listitem>
+          <para>Third item</para>
+        </listitem>
+      </orderedlist>
+    </section>
+
+    <section>
+      <title>Example figure with embedded graphic</title>
+      <para>
+      Here is how you embed a graphic.</para>
+      <figure pgwide="1" xml:id="figure_label">
+        <title>Example figure</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/example_graphic.bmp" format="BMP" scalefit="1" width="100%" />
+          </imageobject>
+        </mediaobject>
+      </figure> 
+      <note><para>Raw images such as the bitmap (bmp) file above may become blurry as they are scaled.
+      Scalable graphic formats like SVG (Scalable Vector Graphics) embed and scale the best.</para></note>
+    </section>
+
+    <section>
+      <title>Example table</title>
+      <para>Of course all good documents need tables.  Here's how you build a basic table.</para>
+
+      <table frame="all" pgwide="1" xml:id="table_label">
+        <title>Example Table Title</title>
+        <tgroup cols="4">
+          <colspec colname="c1" colwidth="25*" />
+          <colspec colname="c2" colwidth="25*" />
+          <colspec colname="c3" colwidth="25*" />
+          <colspec colname="c4" colwidth="25*" />
+          <thead>
+            <row>
+              <entry>
+                <para>
+                  <emphasis role="bold">1st Column Heading</emphasis>
+                </para>
+              </entry>
+              <entry>
+                <para>
+                  <emphasis role="bold">2nd Column Heading</emphasis>
+                </para>
+              </entry>
+              <entry>
+                <para>
+                  <emphasis role="bold">3rd Column Heading</emphasis>
+                </para>
+              </entry>
+              <entry>
+                <para>
+                  <emphasis role="bold">4th Column Heading</emphasis>
+                </para>
+              </entry>
+            </row>
+          </thead>
+          <tbody>
+            <row>
+              <entry>
+                <para>Yes</para>
+              </entry>
+              <entry>
+                <para><phrase role="color:red">Red</phrase></para>
+                <para><phrase role="color:green">Green</phrase></para>
+                <para><phrase role="color:blue">Blue</phrase></para>           
+                <para><phrase role="color:#FFBF00">Custom (Amber)</phrase></para>
+              </entry>
+              <entry>
+                <para>MAIN_Junk</para>
+              </entry>
+              <entry>
+                <para>More_Junk</para>
+              </entry>
+            </row>
+            <row>
+              <entry namest='c1' nameend='c3' align='center'>
+                <para>merged cells horizontal</para>
+              </entry>
+              <entry>
+                <para>cell_stuff</para>
+              </entry>
+            </row>
+            <row>
+              <entry morerows='1'>
+                <para>Merge cells vertical</para>
+              </entry>
+              <entry>
+                <para>filler</para>         
+              </entry>
+              <entry namest='c3' nameend='c4' morerows='1' align='center'>
+                <para>merge cells both ways</para>
+              </entry>             
+            </row>
+            <row>
+              <entry>
+                <para>filler 2</para>         
+              </entry>             
+            </row>
+            <row>
+              <entry>
+                <para>How about we put a list in the table cell</para>
+                <itemizedlist>
+	          <listitem>
+                    <para>item 1</para>
+                  </listitem>
+	          <listitem>
+                    <para>item 2</para>
+                  </listitem>
+	          <listitem>
+                    <para>item 2</para>
+                  </listitem>
+                </itemizedlist>
+              </entry>
+              <entry>
+                <para>Another Cell</para>
+              </entry>
+              <entry>
+                <para>Yet Another Cell</para>
+              </entry>
+              <entry>
+                <para>Finally the last cell</para>
+              </entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+    </section> 
+    <section>
+      <title>Example of crossreferences and footnotes</title>
+      <para>To reference another section or table is pretty easy.  For example: see <xref linkend="table_label" /> for how tables look.</para>
+      <para>Lists are shown in <xref linkend="list_example_label" /> and if you need to make a footnote 
+        <footnote xml:id="foot_id"><para>The footnote text goes here and can reference something like <xref linkend="figure_label" /> for additional explanation.</para></footnote> 
+        For clarification that is easy.  Of course you might want an additional reference to the footnote <footnoteref linkend="foot_id"/> which can also be done easily.</para>
+      <para>Lastly you probably want to mark text by making it <emphasis>italic text example</emphasis> or <emphasis role='bold'>Bold Text Example</emphasis>.</para>
+    </section>
+    <section>
+      <title>Example of code citations and user input</title>
+      <para>When showing user input, you want a nice sceen-looking layout, a prompt, monospace text, and a way to differentiate input from output.  Here's an example:
+      <screen><prompt>$ </prompt><userinput>echo "Hello world"</userinput>
+Hello world
+<prompt>$ </prompt></screen>
+      </para>
+      <para>Docbook also allows for formatting and display of common languages, allowing for whitespace
+            and line returns just as they are written. Here's a sample snippet of C code with line numbering enabled:<programlisting linenumbering="numbered"><![CDATA[#include<stdio.h>
+main()
+{
+    printf("Hello world\n");
+}]]></programlisting></para>
+      <para>If code formatting is not quite what you need, simply displaying text "literally" may suffice as follows: <literal>This is my literal 
+text.         It ignores      whitespace.</literal></para>
+    </section>
+    <section>
+      <title>Example of special characters in text</title>
+      <para>Sometimes in text you need special characters.  These can be provided using their UNICODE values such as &#8800; (&amp;#8800), 
+      &#x2126; (&amp;#x2126), and &#8710; (&amp;#8710;).
+      These can be "coded" using the form <literal>&amp;#</literal><emphasis>ddddd</emphasis><literal>;</literal> where <emphasis>ddddd</emphasis> is 
+      the up to five digit decimal representation of the character. The form <literal>&amp;#x</literal><emphasis>hhhh</emphasis><literal>;</literal> where 
+      <emphasis>hhhh</emphasis> is the up to 4 digit hexidecimal representation of the character.</para>
+      <para>This formatting works well as long as the symbol to which you are referring is contained in the font set
+      used for the document -- Arimo for standard text and Cousine for monospace.  If when building a document, you see a message like 
+      "WARNING, Glyph...not available in font 'Arimo',"
+      see <xref linkend="symbol_font" /> in <xref linkend="docbook_extensions" /> for details on using the provided symbol fonts explicitly.</para>
+    </section>
+    
+    <xi:include href="sec_example.xml"/>
+    
+    <section xml:id="docbook_extensions">
+      <title>Examples of OpenPOWER Foundation Docbook extensions</title>
+      
+      <para>The OpenPOWER Foundation Maven Plugin supports a number of extensions that are not pure Docbook.  These are:</para>
+
+      <simplesect>
+        <title>Setting text color explicitly</title>
+             
+          <para>Text color can be controlled using <literal>&lt;phrase role="color:</literal><emphasis>color_name</emphasis><literal>"&gt;</literal> 
+          tag  where <emphasis>color_name</emphasis> contains the color setting.  For example, this 
+          text:<programlisting><![CDATA[<para role="color:red">A red sentence contains a <phrase role="color:blue">blue</phrase> word.</para>]]></programlisting> produces this sentence:</para>
+          <para role="color:red">A red sentence contains a <phrase role="color:blue">blue</phrase> word.</para>
+          <para>Valid colors include either a keyword color name or a numerical RGB specification.  Keyword names are common with the HTML 4 specificiation:
+          <literal>aqua</literal>, <literal>black</literal>, <literal>blue</literal>, <literal>fuchsia</literal>, <literal>gray</literal>, 
+          <literal>green</literal>, <literal>lime</literal>, <literal>maroon</literal>, <literal>navy</literal>, <literal>olive</literal>, 
+          <literal>purple</literal>, <literal>red</literal>, <literal>silver</literal>, <literal>teal</literal>, <literal>white</literal>, 
+          and <literal>yellow</literal>.  Additionally, RGB values can be <literal>#nnnnnn</literal> where <literal>nnnnnn</literal> is a hexidecimal color value or
+          <literal>rgb(n1, n2, n3)</literal> where <literal>n1</literal>, <literal>n2</literal>, and <literal>n3</literal> are integers 0-255.</para>
+          <para>This tag has also been implemented on the following tags: <literal>&lt;thead&gt;</literal>, 
+          <literal>&lt;tbody&gt;</literal>, and <literal>&lt;tfoot&gt;</literal>.</para>
+          <warning><para>This parameter should only be used for tags listed above.</para></warning>
+      </simplesect>
+      
+      <simplesect>
+        <title>Inserting line breaks</title>
+          <para>Line breaks can be introduced using <literal>&lt;?linebreak?&gt;</literal> tags.  For example, this 
+          text:<programlisting><![CDATA[<para>A line break <?linebreak?> in the middle of text</para>]]></programlisting> produces this sentence:</para>
+          <para>A line break <?linebreak?> in the middle of text</para>
+          <para>This tag becomes useful in table text spacing.</para>
+      </simplesect>
+      
+      <simplesect>
+        <title>Inserting page breaks</title>
+          <para>Page breaks can be introduced using <literal>&lt;?hard-pagebreak?&gt;</literal> tags.  For example, this 
+          text:<programlisting><![CDATA[<para>A page break</para> <?hard-pagebreak?> <para>Between two paragraphs</para>]]></programlisting> produces this output:</para>
+          <para>A page break</para> <?hard-pagebreak?> <para>Between two paragraphs</para>
+          <para>This tag becomes useful in placing tables on page.  Placing this statement before a large table may prevent it from spanning a page.</para>
+          <warning><para>Because the XSL template behind the Processing Instruction generates 
+          a <programlisting><![CDATA[<fo:block break-after='page'/>]]></programlisting> in
+          the book FO output, this instruction should be used in the outer most blocks of a section to work effectively.  Use inside lists and other structural
+          components may result in the text after the break being dropped. <emphasis role="bold">User beware!</emphasis>.</para></warning>
+      </simplesect>
+        
+      <simplesect>
+        <title>Varying the font size</title>
+          <para>Font sizes can also be set using the 
+          <literal>&lt;phrase role="font-size:</literal><emphasis>size</emphasis><literal>"&gt;</literal> 
+          tag where <emphasis>size</emphasis> contains a size value such as "6pt" or "50%" or "1.5em".</para>
+          <para>For example, a paragraph can be made to be 6 point as follows:<programlisting><![CDATA[<para>A sentence that contains some <phrase role="font-size:6pt">6pt font</phrase>, 
+<phrase role="font-size:50%">50% font</phrase>, and 
+<phrase role="font-size:1.5em">1.5em font</phrase> in it.</para>]]></programlisting> produces this output:</para>
+          <para>A sentence that contains some <phrase role="font-size:6pt">6pt font</phrase>, 
+          <phrase role="font-size:50%">50% font</phrase>, and <phrase role="font-size:1.5em">1.5em font</phrase> in it.</para>
+          <para>This tag has also been implemented on the following tags: <literal>&lt;para&gt;</literal>, 
+          <literal>&lt;thead&gt;</literal>, <literal>&lt;tbody&gt;</literal>, and <literal>&lt;tfoot&gt;</literal>.</para>
+          <warning><para>This parameter should only be used for tags listed above.</para></warning>
+      </simplesect>
+        
+      <simplesect xml:id="symbol_font">
+        <title>Using additional symbols</title>
+          <para>If you find that the Arimo and Cousine fonts do not contain the special symbol you need
+          for your document, you may use the additional symbol font provided for document (STIX Two Math).
+          Due to an unimplemented feature in the Apach FO Processor, selection of this 
+          font needs to be explicitly performed using the
+          <literal>&lt;symbol role="symbolfont"&gt;</literal> wrapper around your symbol value.</para>
+          
+          <para>For example, the symbol coding of <programlisting><![CDATA[&#x2A01;]]></programlisting> should produce
+          a circle with a cross in here "&#x2A01;", but instead creates a "Glyph...not available in font 'Arimo'" error
+          on document build and the PDF renders as a "#".</para>
+          
+          <para>Re-coding this to use <programlisting><![CDATA[<symbol role="symbolfont">&#x2A01;</symbol>]]></programlisting> produces
+          the correct symbole here "<symbol role="symbolfont">&#x2A01;</symbol>".</para>
+          <para>If this still does not provide the symbol you expected, double check the code and the font maps found at 
+          <link xlink:href="http://www.stixfonts.org/charactertable.html">http://www.stixfonts.org/charactertable.html</link>.</para>
+      </simplesect>
+    </section>
+
+  </section>
+</chapter>

enable_capi_snap/ch_genpsl_capi10.xml

@@ -0,0 +1,207 @@
+<!--
+  Copyright (c) 2016 OpenPOWER Foundation
+  
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  
+-->
+<!--
+About the XML editing: 
+It's really hard. I am using gvim (Macvim)
+I downloaded a filetype plugin from https://github.com/othree/xml.vim.git
+(git clone it, and copy the ftplugin folder into your ~/.vim folder. )
+It helps a little bit. use ':help xml-plugin' to see what it has. 
+
+
+BTW, I also turned on spelling checking by ':set spell spelllang=en_us' in vim. 
+You can put it into your ~/.vimrc (no comma needed). 
+If you want to disable the spell checking,just ':set nospell'
+
+There is another trick. You will find that the spelling checker doesn't work 
+for your paragraphs. I found an alternative way is to ':set filetype=html'
+
+
+-->
+<chapter version="5.0" xml:lang="en" xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"
+xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="chapter_psl">
+  
+  <!-- Chapter Title goes here. -->
+  <title>Work with CAPI1.0 HDK project</title>
+
+  <note>
+	  <para>
+		  Ask your contact representative or write to capi@us.ibm.com to get a CAPI1.0 HDK project to start. This chapter only works for CAPI1.0, running on Power8. 	  </para>
+  </note> 
+  <section> <title>Steps and directory structure</title>
+    <para>We use an "Out-of-context" flow to generate a PSL dcp file. For a new FPGA card, following steps need to be done:</para>
+    <figure pgwide="1" xml:id="step4">
+        <title>Four steps to build a PSL checkpoint</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/psl-4steps.png" format="PNG" scalefit="1" width="80%" align="center" />
+          </imageobject>
+        </mediaobject>
+    </figure> 
+   
+    <para>We use an "Out-of-context" flow to generate a PSL dcp file. The directory structure is as following:</para>
+    <figure pgwide="1" xml:id="hdk-dir">
+        <title>HDK directory structure</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/hdk-structure.png" format="PNG" scalefit="1" width="90%" align="center" />
+          </imageobject>
+        </mediaobject>
+  </figure> 
+  <para>
+	  In <emphasis role="bold">build_dir/Sources</emphasis>
+  </para>
+  <itemizedlist>
+  <listitem><para>The "prj" directory includes the source file lists.</para></listitem>
+  <listitem><para>The "top" directory includes the top design file "psl_fpga.vhdl" and the wrapper for AFU "psl_accel.vhdl"</para></listitem>
+  <listitem><para>PSL source files are in "psl" directory.</para></listitem>
+  <listitem><para>AFU source files are in "afu" directory.</para></listitem>
+  <listitem><para>"cores" includes 4 Xilinx IP cores used by PSL.</para></listitem>
+  <listitem><para>"xdc" are the constraint files used by PSL and the top design.</para></listitem>
+  </itemizedlist>
+  <para>In build_dir, psl_fpga.tcl is the script "entrance". It assigns the FPGA chip information, and the build flow. </para>
+  <para>FPGA chip information is needed for a new FPGA card, for example:</para>
+  <screen>set device       "xcku115"
+set package      "-flva1517"
+set speed        "-2-e"</screen> 
+  <para>And some controlling bits are for two build flows:</para>
+  <para>1. Build a PSL checkpoint.</para> 
+  <para>2. Build a whole FPGA image (including AFU).</para>
+
+  </section>
+  <section xml:id="section_gen_psl_chkpt"><title xml:id="section_gen_psl_chkpt_title">Generate PSL Checkpoint (b_route_design.dcp)</title>
+  <para>In this section, we just talk about the first build flow - "build a PSL checkpoint". Read it when you need to enable a FPGA card on CAPI1.0 or to fix a bug and update b_routed_design.dcp. The controlling bits should be set as:</para>
+	  
+  <screen>####flow control
+set run.topSynth   1
+set run.oocSynth   1
+set run.tdImpl     0
+set run.oocImpl    1
+set run.topImpl    0
+set run.flatImpl   0</screen>
+  <para>The outfile file will be placed in "Checkpoint" directory, the file name is "b_route_design.dcp".</para>
+
+  <section><title>Upgrade Xilinx IP cores</title>
+	  <para>When a FPGA chip type is changed, or the Vivado tool version has been upgraded, you need to upgrade the Xilinx IP cores that are used in PSL module. PSL module has instantiated four Xilinx IP cores (in Sources/cores):</para>
+	  <itemizedlist>
+		  <listitem><para>pcie3_ultrascale_0</para></listitem>
+		  <listitem><para>sem_ultra_0 (Soft Error Migration)</para></listitem>
+		  <listitem><para>clk_wiz_0</para></listitem>
+		  <listitem><para>tx_wr_fifo</para></listitem>
+	  </itemizedlist>
+	  <para>Steps to upgrade them:</para>
+	  <orderedlist>
+		  <listitem><para>Open Vivado GUI</para></listitem>
+		  <listitem><para>Create a new project. For the second time, just open the project with the four IP cores.</para></listitem>
+		  <listitem><para>Import IP cores (by importing *.xci files under "Source/cores/xxx" directory). For the second time, this step is not needed.</para></listitem>
+		  <listitem><para>Set FPGA type in Project Settings.</para></listitem>
+		  <listitem><para>Run "Tools->Report->Report IP Status"</para></listitem>
+		  <listitem><para>"Upgrade All" and read the upgrade log.</para></listitem>
+	  </orderedlist>
+    <figure pgwide="1" xml:id="ip-update">
+        <title>Small project to update IPs</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/ip-update.png" format="PNG" scalefit="1" width="100%" align="center" />
+          </imageobject>
+        </mediaobject>
+    </figure> 
+    <figure pgwide="1" xml:id="report-ip">
+        <title>Report IP Status</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/report-ip.png" format="PNG" scalefit="1" width="70%" align="center" />
+          </imageobject>
+        </mediaobject>
+    </figure>
+    <note>
+	    <para>For PCIe IP, you need to change subsystem_id for a new card. Right click pcie3_ultrascale_0 -> Reconfig IP and change the subsystem ID field. </para>
+	    <para>Ask an IBM representative for the subsystem ID. </para>
+    </note>
+  </section>
+  <section><title>Input xdc files</title>
+	  <para>The IO pin package information for the new card should be provided by card vendor. Generally, they include Flash Interface, PCIe Interface and other interfaces like I2C and LED. Sample code with IO pins in b_phys.xdc:</para> 
+	  <para>Refer to Xilinx document UG575 for detailed pin package information.</para>
+	  <para>Example:</para>
+	  <screen>set_property PACKAGE_PIN AJ15  [get_ports {o_flash_a[1]}]
+set_property PACKAGE_PIN AK15  [get_ports {o_flash_a[2]}]
+set_property PACKAGE_PIN AH14  [get_ports {o_flash_a[3]}]
+
+set_property IOSTANDARD LVCMOS18 [get_ports {o_flash_a[1]}]
+set_property IOSTANDARD LVCMOS18 [get_ports {o_flash_a[2]}]
+set_property IOSTANDARD LVCMOS18 [get_ports {o_flash_a[3]}]</screen>
+	  <para>Some other constraints also must be updated for the new selection of FPGA chip. It defines the floorplan for PSL. </para>
+	  <note>
+		  <para>There is also a patch to keep VSEC address for Vivado2017.4 and newer Vivado version:</para>
+		  <screen>set_property PF0_SECONDARY_PCIE_CAP_NEXTPTR 12'h400 [get_cells *pcihip0/psl_pcihip0_inst/inst/pcie3_uscale_top_inst/pcie3_uscale_wrapper_inst/PCIE_3_1_inst]</screen>
+		  <para>This is the base address for VSEC registers. capi-utils uses it to set the register address to send bitstream data to flash controller.</para>
+		  <para><emphasis role="bold">Update:</emphasis>This step is not needed after "ECAP update (#28)" commit of capi-utils in Feb 2018.</para>
+	  </note>
+
+  </section>
+  <section><title>Run Vivado to generate PSL checkpoint</title>
+	  <screen>vivado -mode batch -source psl_fpga.tcl -notrace</screen>
+	  <para>The checkpoint file b_route_design.dcp will be generated and put in "Checkpoint" directory. With this checkpoint file, we can continue to build a full FPGA bit image and validate it on hardware.</para>
+	  
+  </section> 
+  </section>
+
+  <section><title>Generate full FPGA image</title>
+	  <section><title>Steps</title>
+  <figure pgwide="1" xml:id="full-steps">
+        <title>Steps to build the full FPGA image</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/full-steps.png" format="PNG" scalefit="1" width="70%" align="center" />
+          </imageobject>
+        </mediaobject>
+  </figure>  
+  <para>This time the controlling bits should be set to:</para>
+  <screen>####flow control
+set run.topSynth   1
+set run.oocSynth   0
+set run.tdImpl     0
+set run.oocImpl    0
+set run.topImpl    1
+set run.flatImpl   0</screen>
+	  </section>
+	  <section><title>Check top design file psl_fpga.vhdl</title>
+		  <para>For a new card, the IO pins and functions may be different to your reference card design. So the logic in top file psl_fpga.vhdl needs to be updated.</para>
+		  <para>Similarly, the xdc file "……topimp.xdc" also needs to be updated.</para>
+	  </section>
+	  <section><title>Prepare filelist for psl_fpga.prj</title>
+		  <para>The "prj" file is a file list. It should contain all the AFU design files. Edit it.</para>
+	  </section>
+	  <section><title>Run Vivado</title>
+		  <para>Two sub steps are here.</para>
+		  <screen>vivado -mode batch -source psl_fpga.tcl -notrace</screen>
+		  <para>Now the bit file is generated</para>
+		  <screen>vivado -mode batch -source write_bitstream.tcl -notrace</screen>
+		  <para>Now you get the bin files to be program to the FPGA flash.</para>
+
+		  <para>For more information about FPGA configuration, please refer to Xilinx Document UG570.</para>
+		  <para>Then you can program the generated bin file to FPGA either by JTAG or on-line programming tools  <link xlink:href="https://github.com/ibm-capi/capi-utils">capi-utils</link></para>
+	  </section>
+	  
+  </section>
+
+
+  
+  
+</chapter>
+
+
+

enable_capi_snap/ch_introduction.xml

@@ -0,0 +1,111 @@
+<!--
+  Copyright (c) 2016 OpenPOWER Foundation
+  
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  
+-->
+<!--
+About the XML editing: 
+It's really hard. I am using gvim (Macvim)
+I downloaded a filetype plugin from https://github.com/othree/xml.vim.git
+(git clone it, and copy the ftplugin folder into your ~/.vim folder. )
+It helps a little bit. use ':help xml-plugin' to see what it has. 
+
+
+BTW, I also turned on spelling checking by ':set spell spelllang=en_us' in vim. 
+You can put it into your ~/.vimrc (no comma needed). 
+If you want to disable the spell checking,just ':set nospell'
+
+There is another trick. You will find that the spelling checker doesn't work 
+for your paragraphs. I found an alternative way is to ':set filetype=html'
+
+
+-->
+<chapter version="5.0" xml:lang="en" xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"
+xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="chapter_introduction">
+  
+  <!-- Chapter Title goes here. -->
+  <title>Introduction</title>
+
+  
+  <section> <title>What is CAPI</title>
+    <para>CAPI stands for "Coherent Accelerator Processor Interface" which enables FPGA to access Host memory by virtual address. You can find more introduction about this interface on <link xlink:href="https://developer.ibm.com/linuxonpower/capi/">https://developer.ibm.com/linuxonpower/capi/</link>. It is an important feature to develop hardware accelerators in heterogeneous computing. In this document, the "hardware accelerators" are built on FPGA. </para>
+    <figure pgwide="1" xml:id="capi1">
+        <title>CAPI basic concept</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/capi1.png" format="PNG" scalefit="1" width="70%" />
+          </imageobject>
+        </mediaobject>
+    </figure> 
+   
+    <para>A complete accelerator has software part (APP, or Applications) running on CPU Processor and the hardware part (AFU, Acceleration Function Unit) running on FPGA chip. APP and AFU are sharing host memory, that means, they both can read and write the 2^64 range of virtual memory address. To make it happen, CAPI technology has a CAPP (Coherent Acceleration Processor Proxy) logic unit in Processor chip, and also needs a PSL (Processor Service Layer) logic unit in FPGA chip. For CAPI1.0 and CAPI2.0, the interconnection between processor and FPGA is using PCIe physical links and PCIe form factor.</para>
+    <para> CAPI1.0 uses PCIe Gen3x8. </para>
+    <para> CAPI2.0 uses PCIe Gen4x8 or Gen3x16.</para>
+    <para> OpenCAPI is not covered in this document. Visit <link xlink:href="https://opencapi.org">https://opencapi.org</link> for more information. </para>
+  </section>
+  
+  
+  <section> <title>Enable PSL IP on FPGA</title>
+  <para>This document only applies to the cards using Xilinx FPGA chips.</para>
+  <para>A customer FPGA card needs to have a PSL module (Processor Service Interface) to become a "CAPI-enabled" card. This PSL module is provided by OpenPower Foundation. </para>
+  <itemizedlist>
+     <listitem><para> For CAPI1.0, PSL module and the surrounding board specific modules are provided in the form of a routed dcp file (Xilinx Vivado design checkpoint). It's usually called b_route_design.dcp. </para></listitem>
+     <listitem><para> For CAPI2.0, PSL is an IP package with encrypted source code. It's named like ibm.com_CAPI_PSL9_WRAP_2.00.zip.</para></listitem>
+   </itemizedlist> 
+   <para> They can be downloaded at <link xlink:href="https://www.ibm.com/systems/power/openpower">https://www.ibm.com/systems/power/openpower</link>. From the menu, select "CAPI","Coherent Accelerator Processor Interface (CAPI)" or directly click the "CAPI" icon to go to the CAPI section. Then download the appropriate files depending on your target system being POWER8 (CAPI 1.0) or POWER9 (CAPI 2.0). You need to register an IBM ID to download them.</para>
+   <para>Users can develop CAPI accelerators in two modes: HDK and SNAP.</para>
+   <para>HDK is the abbreviation of Hardware Development Kit. As shown in the diagram below, on the FPGA side, the Xilinx Vivado project includes two parts: BSP (Board Supporting Package, containing PSL module) and AFU (Acceleration Function Unit). How to generate BSP will be introduced in Chapter <xref linkend="chapter_capi20_bsp" endterm="chapter_capi20_bsp_title"/></para>   
+    <figure pgwide="1" xml:id="hdk1">
+        <title>Develop an accelerator in HDK mode</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/hdk.png" format="PNG" scalefit="1" width="80%" align="center"/>
+          </imageobject>
+        </mediaobject>
+    </figure> 
+
+    <para>AFU is where to implement user-defined functions. The developer working on AFU needs to understand the protocol between AFU and BSP, which is called PSL/AFU interface specification. Please refer to <link xlink:href="http://openpowerfoundation.org/wp-content/uploads/resources/psl-afu-spec/content/go01.html"> CAPI1.0 PSL Spec</link> and <link xlink:href="http://openpowerfoundation.org/wp-content/uploads/resources/v2-psl-afu-spec/content/ch_preface.html"> CAPI2.0 PSL Spec</link> or search "PSL/AFU interface" in your web browser. </para>
+    <para>When developing an acceleration, <link xlink:href="https://github.com/ibm-capi/pslse">PSLSE</link> (PSL Simulation Engine) is also needed to make a software-hardware co-simulation which guarantees the correctness of accelerator design. </para>
+    <para>When deploying the acceleration to OpenPower servers, it requires user library <link xlink:href="https://github.com/ibm-capi/libcxl">libcxl</link> and kernel module <userinput>cxl</userinput> to run the application. </para>
+    <para>In all, HDK mode will provide the maximum control, utilization of resources and shortest latency. However, SNAP mode simplifies and standardizes the application development significantly and is more recommended.</para>
+   </section>
+
+  <section> <title>SNAP</title>
+          <para>SNAP is the abbreviation of Storage, Networking and Analytics Programming. It is an open-source acceleration development framework <link xlink:href="https://github.com/open-power/snap"> https://github.com/open-power/snap</link>. The benefits are: </para>
+          <orderedlist>
+          <listitem><para>On the FPGA side, SNAP framework adds a bridge to provide AXI interface to developers. So the developer can focus on acceleration function logic design, and doesn't need to study the details of PSL interface specification. AXI is the defacto industry standard for on-chip bus interconnections and is part of <link xlink:href="https://www.arm.com/products/silicon-ip-system/embedded-system-design/amba-specifications">AMBA (Advanced Microcontroller Bus Architecture)</link>.</para></listitem>
+          <listitem><para>It also provides DDR SDRAM controller and an optional NVMe controller. The developer can use the card memory or storage directly.</para></listitem>
+          <listitem><para>SNAP supports using HLS (High Level Synthesis) to develop the acceleration functional unit ("Hardware Action" in yellow box). Developers can write C++ functions and Vivado HLS will compile/convert them to Verilog or VHDL design automatically.</para></listitem>
+          <listitem><para>A new layer of user library "libsnap" provides more convenient APIs.</para></listitem>
+          <listitem><para>SNAP is an integrated developing environment that the developer can configure, create Vivado project, run co-simulation or build bitstream with simple commands.</para></listitem>
+          <listitem><para>Many action examples help new developers to get started.</para></listitem>
+          </orderedlist>
+     <figure pgwide="1" xml:id="snap1">
+        <title>Develop an acceleration on SNAP</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="figures/snap.png" format="PNG" scalefit="1" width="90%" align="center" />
+          </imageobject>
+        </mediaobject>
+    </figure> 
+    <para>Equipping the new FPGA card with SNAP framework needs a few additional steps and is introduced in Chapter <xref linkend="chapter_capi20_snap" endterm="chapter_capi20_snap_title"/></para>
+    <note><para>This document focuses on CAPI2.0. For CAPI1.0 enablement, please contact <email>capi-snap-doc@mailinglist.openpowerfoundation.org</email> for more information.</para>
+   <para>It is assumed the reader knows how to work on Vivado Project and SNAP already. Many materials on how to develop an accelerator with SNAP can be found in "docs" folder of <link xlink:href="https://github.com/open-power/snap"> snap github</link> or other webpages so they are not discussed in this document.</para></note>
+
+  </section>
+ 
+</chapter>
+
+
+

enable_capi_snap/pom.xml

@@ -0,0 +1,161 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Copyright (c) 2016 OpenPOWER Foundation
+  
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <parent>
+
+        <groupId>org.openpowerfoundation.docs</groupId>
+        <artifactId>workgroup-pom</artifactId>
+        <version>1.0.0-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <!-- TODO: Rename the artifactID field to some appropriate for your new document -->
+    <artifactId>todo-artifact_id</artifactId>
+
+    <packaging>jar</packaging>
+ 
+    <!-- TODO: Rename the name field to some appropriate for your new document -->
+    <name>enable-fpga-to-capi-snap</name>
+
+    <properties>
+        <!-- This is set by Jenkins according to the branch. -->
+        <release.path.name></release.path.name>
+        <comments.enabled>0</comments.enabled>
+    </properties>
+    <!-- ################################################ -->
+    <!-- USE "mvn clean generate-sources" to run this POM -->
+    <!-- ################################################ -->
+    <build>
+        <plugins>
+            <plugin>
+
+                <groupId>org.openpowerfoundation.docs</groupId>
+
+                <artifactId>openpowerdocs-maven-plugin</artifactId>
+                <!-- version set in ../pom.xml -->
+                <executions>
+                    <execution>
+                        <id>generate-webhelp</id>
+                        <goals>
+                            <goal>generate-webhelp</goal>
+                        </goals>
+                        <phase>generate-sources</phase>
+                        <configuration>
+                            <!-- These parameters only apply to webhelp -->
+                            <enableDisqus>${comments.enabled}</enableDisqus>
+                            <disqusShortname>openpower-template-guide</disqusShortname>
+                            <enableGoogleAnalytics>1</enableGoogleAnalytics>
+                            <googleAnalyticsId>UA-17511903-1</googleAnalyticsId>
+                            <generateToc>
+                                appendix		toc,title
+                                article/appendix	nop
+                                article   		toc,title
+                                book      		toc,title,figure,table,example,equation
+                				book/appendix		nop
+                                book/chapter		nop
+                                chapter   		toc,title
+                				chapter/section 	nop
+                                section   		toc
+                                part      		toc,title
+                                reference 		toc,title
+                                set       		toc,title
+                            </generateToc>
+                            <!-- The following elements sets the autonumbering of sections in output for chapter numbers but no numbered sections-->
+                            <sectionAutolabel>1</sectionAutolabel>
+                            <tocSectionDepth>1</tocSectionDepth>
+                            <sectionLabelIncludesComponentLabel>1</sectionLabelIncludesComponentLabel>
+
+                            <!-- TODO: Rename the webhelpDirname field to the new directory for new document -->
+                            <webhelpDirname>enable-fpga-to-capi-snap</webhelpDirname>
+
+                            <!-- TODO: Rename the pdfFilenameBase field to the PDF name for new document -->
+                            <pdfFilenameBase>enable-fpga-to-capi-snap</pdfFilenameBase>
+
+                            <!-- TODO: Define the appropriate work product type.  These values are defined by the IPR Policy.
+                                       Consult with the Work Group Chair or a Technical Steering Committee member if you have
+                                       questions about which value to select.
+                                       
+                                       If no value is provided below, the document will default to "Work Group Notes".-->
+                            <workProduct>workgroupNotes</workProduct>
+                            <!--workProduct>workgroupSpecification</workProduct -->
+                            <!-- workProduct>candidateStandard</workProduct -->
+                            <!-- workProduct>openpowerStandard</workProduct -->
+
+                            <!-- TODO: Set the appropriate security policy for the document.  For documents
+                                       which are not "public" this will affect the document title page and
+                                       create a vertical running ribbon on the internal margin of the
+                                       security status in all CAPS.  Values and definitions are formally
+                                       defined by the IPR policy.  A layman's definition follows:
+
+                                       public =	                this document may be shared outside the
+                                                                foundation and thus this setting must be
+                                                                used only when completely sure it allowed
+                                       foundationConfidential = this document may be shared freely with
+                                                                OpenPOWER Foundation members but may not be
+                                                                shared publicly
+                                       workgroupConfidential =  this document may only be shared within the
+                                                                work group and should not be shared with
+                                                                other Foundation members or the public
+
+                                       The appropriate starting security for a new document is "workgroupConfidential". -->
+                            <!-- security>workgroupConfidential</security -->
+                            <!-- security>foundationConfidential</security -->
+                            <security>public</security>
+
+                            <!-- TODO: Set the appropriate work flow status for the document.  For documents
+                                       which are not "published" this will affect the document title page
+                                       and create a vertical running ribbon on the internal margin of the 
+                                       security status in all CAPS.  Values and definitions are formally
+                                       defined by the IPR policy.  A layman's definition follows:
+
+                                           published = this document has completed all reviews and has
+                                                       been published
+                                           draft =     this document is actively being updated and has
+                                                       not yet been reviewed
+                                           review =    this document is presently being reviewed
+
+                                       The appropriate starting security for a new document is "draft". -->
+                            <documentStatus>published</documentStatus>
+                            <!-- documentStatus>review</documentStatus -->
+                            <!-- documentStatus>published</documentStatus -->                            
+
+                        </configuration>
+                    </execution>
+                </executions>
+                <configuration>
+                    <!-- These parameters apply to pdf and webhelp -->
+                    <xincludeSupported>true</xincludeSupported>
+                    <sourceDirectory>.</sourceDirectory>
+                    <includes>
+                       <!-- TODO: If you desire, you may change the following filename to something more appropriate for the new document -->
+                       bk_main.xml
+                    </includes>
+
+                    <!-- **TODO: Set to the correct project URL.  This likely needs input from the TSC.  -->
+                    <!-- canonicalUrlBase>http://openpowerfoundation.org/docs/template-guide/content</canonicalUrlBase -->
+                    <glossaryCollection>${basedir}/../glossary/glossary-terms.xml</glossaryCollection>
+                    <includeCoverLogo>1</includeCoverLogo>
+                    <coverUrl>www.openpowerfoundation.org</coverUrl>
+                </configuration>
+            </plugin>
+        </plugins>
+    </build>
+</project>

enable_capi_snap/sec_example.xml

@@ -0,0 +1,25 @@
+<!--
+  Copyright (c) 2016 OpenPOWER Foundation
+  
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  
+-->
+<section version="5.0" xml:lang="en" xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"
+xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="section_example">
+
+    <title>Sample section include  </title>
+    <para>This section was developed in a separate file but included in the document by using the following 
+    text:<programlisting><![CDATA[  <xi:include href="sec_example.xml"/>]]></programlisting>
+    where <literal>sec_example.xml</literal> is the source file name.</para>
+
+</section>

pom.xml

@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project  xmlns="http://maven.apache.org/POM/4.0.0"
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+  <parent>
+     <groupId>org.openpowerfoundation.docs</groupId>
+     <artifactId>master-pom</artifactId>
+     <version>1.0.0-SNAPSHOT</version>
+     <relativePath>../Docs-Master/pom.xml</relativePath>
+  </parent>
+  <modelVersion>4.0.0</modelVersion>
+
+  <artifactId>workgroup-pom</artifactId>
+  <packaging>pom</packaging>
+
+  <modules>
+    <!-- TODO: Add new documents are build in the project, add their directories to this list to
+               enable all document builds from the top level -->
+    <module>enable_capi_snap</module>
+  </modules>
+</project>