# libreBMC This repository contains the high-level documentation for libreBMC - an open source hardware, gateware and software project to implement a fully open Baseboard Management Controller to boot a high power, modern server. This work was conducted as a collaboration between the OpenPOWER foundation and [Code Construct](https://codeconstruct.com.au); this documentation is based on the [original article](https://codeconstruct.com.au/docs/dcscm-openbmc) on the Code Construct docs site. We are building on a lot of prior work from IBM and Antmicro, mainly: * The [Antmicro Artix DC-SCM](https://github.com/antmicro/artix-dc-scm) hardware - an open hardware implementation of the Open Compute DC-SCM module, using an Artix FPGA as the core * The "[AC922 Interposer](https://git.openpower.foundation/librebmc/ac922interposer)" - an adapter between the AC922's baseboard and a DC-SCI connector. * Anton Blanchard's [Microwatt](https://github.com/antonblanchard/microwatt) project, implementing a fully open powerpc core, as gateware suitable for the Artix. * System-on-Chip peripherals using gateware from the [LiteX](https://github.com/enjoy-digital/) project - `litesdcard`, `liteeth`, and `litedram`. * [LPC controller gateware](https://git.openpower.foundation/librebmc/lpcperipheral), from Michael Neuling and Anton Blanchard, as part of the OpenPOWER Foundation's LibreBMC project. * A [Linux kernel](https://kernel.org) with a few extra drivers and platform configuration for Microwatt and the DC-SCM board. * An [OpenBMC](https://github.com/openbmc/) userspace, ported from the existing witherspoon platform definition. Our work here was mainly to integrate these parts; a few fixes here and there, plus a few new features at various places in the stack. There are a few other projects implementing a similar platform: * The original "boxarty" project from Joel Stanley, Paul Mackerras and Michael Neuling from IBM, an original prototype replacing the AC922's AST2500-based BMC board (codenamed "boxelder") with an Arty A7 FPGA prototyping board with Microwatt gateware. This was running a set of scripts to boot the AC922 host. * The [Axiado](https://axiado.com/) / [Antmicro](https://antmicro.com) boxarty updates, which built the original boxarty into a more reproducible embedded system using a [buildroot port](https://github.com/Axiado/boxarty-buildroot/tree/antmicro). However, this is the first implementation (that we're aware of!) that runs a close-to-production OpenBMC stack, and uses a DC-SCM module. Finally, this document provides an early snapshot of the project; we will keep things up-to-date as things progress, and/or as issues are fixed or the source code is updated. # Hardware The hardware components for this are: * The Antmicro DC-SCM module * An Antmicro Root of Trust (RoT) [bypass board](https://github.com/antmicro/dc-scm-rot-jumper-card) (we're not using a RoT on this platform). * The AC922 interposer module * The AC922 server itself If you're following along at home, you'll also need: * Some industrial hearing protection \- because we're working on pre-fan-control firmware, and the AC922 fans running at full speed are, well, _industrial_. We needed a few reworks to the hardware to fix a few bugs and incompatibilities. Those are, in roughly-decreasing order of complexity: ## Rework 1: LPC connectivity The version of the DC-SCM board that we're using does not connect the DC-SCI's ESPI channel to the FPGA, due to an intentionally non-populated component: a voltage translator between the 3.3V signals of the FPGA, and the 1.8V signals of the ESPI bus. We have documented this at [issue #2](https://github.com/antmicro/artix-dc-scm/issues/2) of the DC-SCM repo. If/when that issue is fixed, this rework will not be necessary. On the POWER9 platform, we use LPC rather than ESPI, and re-use those ESPI pins for the LPC connection to the host. In its current state, we can't interface the Artix FPGA to the LPC bus due to the unpopulated voltage translator. Fortunately, we don't actually need voltage translation here, and can connect the 3.3V FPGA IOs to the DC-SCI directly. This does require a rework though, bypassing the missing translator (refdes U21 in the design): ![DCSCM U21 rework detail](images/dc-scm-rework-1-detail.png) We have a printable version of the rework instructions at [dc-scm-rework-1.v1.pdf](resources/dc-scm-rework-1.v1.pdf) if you want to replicate these modifications, and/or take to a fabricator. ## Rework 2: DC-SCM i2c5 routing There's an off-by-two error in the DC-SCM board, where pins for I2C bus 5 are out-of-place, and are instead connected to PCIe signals on the Artix. We have documented this at [issue #1](https://github.com/antmicro/artix-dc-scm/issues/1) of the DC-SCM project, which is now fixed in the design. In the meantime, we have cut the two traces for the A11 and A12 lines on the DC-SCM board, just inboard of the DC-SCI connector. ![DC-SCM i2c5 rework detail](images/dc-scm-rework-2.png) Note that these traces are on the back side of the board; the board diagram is flipped to show the traces as they appear on this side. Since we still need the i2c5 bus connected, we have used some spare IO pins on the TPM header, and routed those to the i2c5 SDA and SCL signals. There are corresponding gateware and device tree updates to use those TPM pins as GPIOs (for i2c5), covered below. ## Rework 3: Interposer FSI mux signal routing The AC922 interposer connects the baseboard's FSI mux input to pin B51 of the DC-SCI connector, which is a USB signal, and so not routed to the FPGA. We have filed [an issue](https://git.openpower.foundation/librebmc/ac922interposer/issues/2) on the interposer design for this. To work around the issue, we have connected pin E4 of the molex connector to another FPGA IO on the TPM header. However, the USB controller still drives this signal, so we need to isolate the USB controller device from the trace. There are a couple of options here: masking the pins on the DC-SCI connector, or isolating a single trace on the DC-SCM board. We went for the former, as it's less intrusive, and the issue is with the interposer rather than the DC-SCM. Since none of the signals in that same section of the DC-SCI edge connector are used, we can just mask the entire section with ESD-safe insulating tape: ![DCSCM FSI connector masking](images/dc-scm-rework-4.jpg) Alternatively, you may isolate this one signal by cutting the USB trace on the DC-SCM board, near U18. ![DCSCM FSI mux rework detail](images/dc-scm-rework-3.png) This is quite close to the `USB_DP` trace, but that is unused on the AC922 hardware - so any "collateral damage" from cutting both traces is likely to be fine - provided you're not planning to use the DC-SCM board with other hardware. ## Replacement IO In implementing the workaround above, we're replacing some existing (incorrect) connections with other FPGA IO lines. Since the TPM header is available, and connected directly to the FPIO IO banks, this gives us a convenient spot for our four replacement IO signals: two for i2c SDA/SCL, and two for the FSI mux/trans signals. We have used the following pins on the board: ![DCSCM IOs on TPM header](images/tpm-ios.png) These can either be routed to corresponding pins on the back of the interposer's pin grid, or directly to the AC922 baseboard. ## Other reworks We had a couple of issues with missing pins on the AC922 interposer board, which we have worked-around by using other spare FPGA IO pins (also on the DC-SCM TPM header), routed directly to the AC922 baseboard. These should be contained to our particular hardware, but do check your own interposer for full pins on the 20x12 connector grid before going too far with debugging! # Gateware Microwatt gateware for the FPGA on the DC-SCM board can be found in the [`dev/dcscm`](https://github.com/CodeConstruct/microwatt/commits/dev/dcscm) branch of the Code Construct microwatt repo on github. Building microwatt for the Artix FPGA is mainly described in Microwatt's [README](https://github.com/antonblanchard/microwatt#synthesis-on-xilinx-fpgas-using-vivado). In short, you'll need to set up `fusesoc`, and install Xilinx Vivado. In our case, we used Vivado Free Edition 2022.1, but older versions should also work. Build with: ```sh $ fusesoc run --target=antmicro-artix-dc-scm microwatt --no_bram --memory_size=0 ``` The build will take around 10 minutes, and the resultant bitstream will `build/microwatt_0/antmicro-artix-dc-scm-vivado/microwatt_0.bit`, ready to load on the board. Alternatively, try our pre-built bitstream: [`microwatt-dcscm-405c16a924.bit`](https://codeconstruct.com.au/docs/dcscm-openbmc/microwatt-dcscm-405c16a924.bit). The micro-USB connector on the DC-SCM board provides JTAG access to the FPGA, and a Microwatt serial console. Connect this to your development machine to upload the bitstream to your DC-SCM board. One of the board's 4MB NOR SPI flash chips is used to store the FPGA bitstream and early software - this can be flashed using OpenOCD for JTAG, wrapped in the `flash-arty` script to handle configuration. Behind the scenes `flash-arty` loads a small [JTAG-SPI proxy bitstream](https://github.com/quartiq/bscan_spi_bitstreams) on the FPGA, to write to the flash chip. ```sh $ ./openocd/flash-arty -c antmicro-artix-dc-scm -f a100 -t bin \ build/microwatt_0/antmicro-artix-dc-scm-vivado/microwatt_0.bit ``` If you specify the `-l` argument, the bitstream will be loaded directly to the FPGA (rather than to flash), for a one-time boot. Use `-x` for a FPGA reset. A serial console will show Microwatt's first stage loader running: ``` $ screen /dev/serial/by-id/usb-FTDI_Quad_RS232-HS-if02-port0 115200 Welcome to Microwatt ! Soc signature: f00daa5500010001 HDL Git SHA1: 405c16a924694a Soc features: UART DRAM SPIFLASH ETHERNET SDCARD DRAM: 512 MB ... Trying flash... Doesn't look like an elf64 HDR: ff ff ff ff ff ff ff ff Copying payload to DRAM... Booting from DRAM... ``` In this case, microwatt is up and running, but can't yet find software to load from flash - that needs `u-boot`, which we'll add in the later [u-boot section](#u-boot). The changes implemented in our microwatt port are: ## LiteEth We have regenerated the [LiteEth](https://github.com/enjoy-digital/liteeth/) ethernet with a hardware reset to fix intermittent startup. This also required patching newer LiteEth to have stable register bank offsets (these are both ready pull requests). The RGMII PHY chip on the DC-SCM also needs a zero delay, so a separate Microwatt config. ## Extra GPIOs The BMC uses several I2C busses from the host for monitoring and control. We use the Linux GPIO bitbanging I2C driver for this, taking a SDA and SCL GPIO line per I2C bus. Since there are 32 pins per GPIO bank, we added a second (`GPIOB`) bank at `0xc0008000` on the wishbone MMIO bus. In future, the I2C functionality could be moved to gateware to avoid inefficient polling of GPIO by the kernel driver. The interposer only connects a subset of the platform GPIO signals to the DC-SCI connector, so there are some input signals that OpenBMC expects and are not implemented in hardware. To work around this, we have added a few special GPIO lines with fixed `0` or `1` values, so that certain missing hardware presence lines can be emulated. ## LPC gateware A LPC bus between host and BMC provides low-level interactions between these two components, including host firmware load, host serial console, and an IPMI channel. The BMC LPC client is implemented through [LPC gateware](https://github.com/OpenPOWERFoundation/lpcperipheral), originally provided by Michael Neuling and Anton Blanchard. Amaranth was used to generate the verilog, no changes were required. ## GPIO interrupts OpenBMC monitors GPIO interrupts, so we added support to Microwatt's GPIO implementation to trigger interrupts on input edges/levels, configurable by hardware registers. ## SoC reset We have connected the SoC reset signal in Microwatt, triggerable by a syscon register. ## LiteSDCard 1-bit data SD and MMC cards normally start in 1-bit data width mode and transition to wider modes after MMC host initialisation. [LiteSDCard](https://github.com/enjoy-digital/litesdcard) previously only supported 4 bit mode, so required driver workarounds to send an early "switch to 4 bit" command. We have added support to LiteSDCard for 1-bit mode and switching at runtime, so the normal SD/MMC infrastructure can be used in u-boot and Linux. This should also support the 8-bit data width on the DC-SCM's eMMC, pending further testing. # Software The software architecture for the DC-SCM + OpenBMC setup is based on an fairly plain OpenBMC image, using the "witherspoon" platform. This is the production AC922 stack, using an AST2500 BMC. We have modified this for the Microwatt environment, plus hardware definitions for the DC-SCM board. There are a couple of areas of storage available: * A small amount of SPI flash; we use this for the FPGA bitstream and the u-boot binary * A larger eMMC device; we use this for the OpenBMC image, including kernel and PNOR (host firmware) storage The boot flow for the device is: * The FPGA bitstream is read from the on-board SPI flash * The bitstream contains a tiny loader implemented in BRAM (block RAM in the FPGA). This loader initialises the DRAM code, reads the u-boot image (also from SPI flash) into DRAM, and jumps to the `u-boot` entry point * The `u-boot` binary has a driver for the eMMC device, plus support for reading a kernel from the `ext4` filesystem * `u-boot` loads the Linux kernel `dtbImage` from a pre-defined path on an eMMC partition, and executes the kernel * The Linux kernel has drivers for the eMMC device, which contains the OpenBMC root filesystem. * AC922 host firmware is kept on a separate partition on the eMMC device, as a simple 64MB flat file. This is loaded during the host power-on procedure. All of the components for this project are available on github: | Component | Repository & Branch | |-----------|--------| | u-boot | [u-boot `dev/dcscm`](https://github.com/CodeConstruct/u-boot/tree/dev/dcscm) | | Linux | [linux `dev/dcscm`](https://github.com/CodeConstruct/linux/tree/dev/dcscm) | | OpenBMC | [openbmc `dev/dcscm`](https://github.com/CodeConstruct/openbmc/tree/dev/dcscm) | Note that these branches are still in-development, and may be updated and rebased. The general trend will be for patches to go upstream, and we will update this document to correspond to any changes as they happen. ## u-boot The u-boot port for the DC-SCM hardware is based on the general `microwatt` port, by Paul Mackerras and Joel Stanley. This gives a base microwatt port, including support for ethernet and SPI peripherals. On top of this base, we have added support for the on-board eMMC device (to allow booting the kernel), added a general platform definition (a device tree and defconfig), and added some small features to make development easier. The code is up at the [`dev/dcscm` branch](https://github.com/CodeConstruct/u-boot/tree/dev/dcscm) of the Code Construct `u-boot` tree on github. Building should just be a matter of: ```sh $ make CROSS_COMPILE=powerpc64le-linux-gnu- O=obj.microwatt \ microwatt_antmicro_artix_dcscm_defconfig $ make CROSS_COMPILE=powerpc64le-linux-gnu- O=obj.microwatt -j$(nproc) ``` the resulting image will be an ELF binary in `obj.microwatt/u-boot`; the gateware can load and execute this directly from SPI flash. To write this to flash, use the `flash-arty` script in the microwatt repository, at the base address of `0x300000` ``` $ ./openocd/flash-arty -c antmicro-artix-dc-scm -f a100 -a 0x300000 -t bin \ ../u-boot/obj.microwatt/u-boot ``` ## Linux kernel The kernel port for the DC-SCM board is based on the general microwatt bringup work, plus a few OpenBMC-specific patches. We have published our changes, based on a v6.0 upstream kernel, at the [`dev/dcscm` branch](https://github.com/CodeConstruct/linux/tree/dev/dcscm) of the Code Construct `linux` tree on github. The kernel `.config` is available here too: [`linux-v6.0-dcscm.config`](resources/linux-v6.0-dcscm.config). This also includes support for a NFS root, which can be handy for initial development work - particularly setting up the initial filesystem on the MMC device. To build: ```sh $ mkdir obj.microwatt $ cp linux-v6.0-dcscm.config obj.microwatt/.config $ make ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- O=obj.microwatt oldconfig $ make ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- O=obj.microwatt -j$(nproc) ``` The DC-SCM kernel will build into a combined zImage + device-tree blob (a `dtbImage`), located at `obj.microwatt/arch/powerpc/boot/dtbImage.microwatt-dcscm.elf`. If you need to alter the boot arguments, they are located in the `/chosen/bootargs` property of the device tree source file, at `arch/powerpc/boot/dts/microwatt-dcscm.dts`: ```dts chosen { bootargs = "root=/dev/mmcblk0p1"; ibm,architecture-vec-5 = [19 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 40]; stdout-path = &UART0; }; ``` for NFS booting, try something like: ```dts bootargs = "root=/dev/nfs nfsroot=192.168.0.1:/srv/nfs/dcscm ip=dhcp"; ``` \- substituting IP and paths for your own environment, of course. ## OpenBMC In order to get OpenBMC running on the DC-SCM board, we've added a new `witherspoon-microwatt` platform definition, based on the original `witherspoon` (ie., production AC922) platform. Anton Blanchard had created the base support, and we have added further porting work to make a usable BMC image. However: our main goal here was to prove the DC-SCM + microwatt components; this is still a long way from an upstreamable OpenBMC implementation. There are changes that are not suitable for the general repo, as they alter existing (working) components. So: treat this as reference code, rather than production! The OpenBMC implementation is available from the [`dev/dcscm` branch](https://github.com/CodeConstruct/openbmc/tree/dev/dcscm) of the Code Construct `openbmc` tree on github. The build process is standard for an OpenBMC tree, just using the `witherspoon-microwat` configuration: ```sh $ . setup witherspoon-microwatt $ bitbake obmc-phosphor-image ``` The build infrastructure will download and build the OpenBMC components, and assemble into an `ext4` filesystem image, at: ``` tmp/deploy/images/witherspoon-microwatt/obmc-phosphor-image-witherspoon-microwatt.ext4 ``` # Deploying From here, we have a few components: * The gateware, already flashed to the SPI device on the DC-SCM * A u-boot build, also flashed to SPI * A kernel, as a `dtbImage.elf` * An OpenBMC image, as an `ext4` filesystem image The easiest way to assemble this into a full BMC will be to netboot the kernel, using an embedded initramfs or small NFS root filesystem, and use that to set up the MMC device. I would suggest a buildroot image for this; Joel Stanley has a suitable tree in his [`microwatt` branch of buildroot](https://github.com/shenki/buildroot/tree/microwatt). From there, you can: * Create two partitions on the eMMC device (`/dev/mmcblk0`): * One large one for the OpenBMC root filesystem * One small (say, 200MB) for the host firmware * Write the OpenBMC `.ext4` file to the first MMC partition * Mount the new OpenBMC partition (`/dev/mmcblk0p1`) * Add the new kernel to the new partition: say, as `/boot/dtbImage.elf` * Create a new filesystem for host firmware on the second partition: ``` mkfs.ext4 -m 0 /dev/mmcblk0p1 ``` * Add a recent witherspoon PNOR to that new filesystem * Create a symlink from the OpenBMC `/var/pnor.img` to the PNOR image on the second partition Using the two partitions will save time should you want to re-write the OpenBMC filesystem after a rebuild. Then, set up `u-boot` to boot automatically: ``` => setenv mmcboot 'ext4load mmc 0 0xc000000 /boot/dtbImage.elf; bootelf 0xc000000' => setenv bootcmd 'run mmcboot' => saveenv ``` And you should be good to go! Of course, if you have any issues or queries, please get in touch with the LibreBMC SIG, and/or Jeremy Kerr . # Future work This project was originally intended as a proof-of-concept of a full BMC running on the open source microwatt core, on the open DC-SCM hardware. We've reached that goal, but there are still a few components that would get us closer to a production-suitable BMC implementation. ## General cleanups and upstreaming As this is mainly a proof-of-concept, some of the changes are a little rough-around-the-edges, so could benefit from a review and rework. There are also areas that can be optimised - both in the new code and the underlying codebases. The main components of this work would be suitable for submitting upstream. We have a number of pull requests pending already, which will reduce the amount DC-SCM specific code needed. ## Software optimisation In working through the stack for this implementation, we have identified a few areas that may benefit from some performance optimisation work. Parts of the OpenBMC stack are certainly designed for a higher-performance machine than the (current) microwatt core on the Artix FPGA, and there are definitely areas that would be fairly straightforward to start an optimisation effort on. ## Gateware additions Currently, the FSI and I2C interfaces are implemented in software, so can be taxing on the CPU when there is traffic over those busses. With the system running, the full fan control implementation may use around 60%-80% of CPU just in FSI and I2C code. The next step here would be to add hardware/gateware implementations of the FSI and I2C controllers; that would go a long way to reducing system load. ## Easier deployment The current process requires a lot of manual setup to get an OpenBMC image constructed on the on-board storage. There could certainly be improvements to this flow; for example, implementing a small bootable image that automates flashing and initialising the OpenBMC system, or allowing easier access to the eMMC device for initial image loads. If there's any area of development you would be interested in seeing, or participating in, please let us know!