Compare commits
2 Commits
master
...
jamesk-ocp
Author | SHA1 | Date |
---|---|---|
James Kulina | 0c500407a1 | 1 year ago |
James Kulina | bfccd47360 | 1 year ago |
Before Width: | Height: | Size: 207 KiB |
Before Width: | Height: | Size: 71 KiB |
Before Width: | Height: | Size: 105 KiB |
Before Width: | Height: | Size: 427 KiB |
Before Width: | Height: | Size: 406 KiB |
Before Width: | Height: | Size: 396 KiB |
Before Width: | Height: | Size: 330 KiB |
Before Width: | Height: | Size: 362 KiB |
Before Width: | Height: | Size: 321 KiB |
Before Width: | Height: | Size: 286 KiB |
Before Width: | Height: | Size: 198 KiB |
Before Width: | Height: | Size: 101 KiB |
Before Width: | Height: | Size: 393 KiB |
Before Width: | Height: | Size: 438 KiB |
Before Width: | Height: | Size: 237 KiB |
Before Width: | Height: | Size: 205 KiB |
Before Width: | Height: | Size: 54 KiB |
Before Width: | Height: | Size: 22 KiB |
Before Width: | Height: | Size: 37 KiB After Width: | Height: | Size: 24 KiB |
Before Width: | Height: | Size: 31 KiB |
Before Width: | Height: | Size: 10 KiB |
Before Width: | Height: | Size: 76 KiB |
Before Width: | Height: | Size: 44 KiB |
Before Width: | Height: | Size: 8.5 KiB |
Before Width: | Height: | Size: 3.1 KiB |
@ -1,11 +1,11 @@
|
|||||||
#!parms.toml
|
#!parms.toml
|
||||||
[forms.contact]
|
[forms.contact]
|
||||||
URI = "http://localhost:8080/forms/contactus/opf"
|
URI = "http://localhost:8080/forms/contactus/test/opf"
|
||||||
[forms.hub]
|
[forms.hub]
|
||||||
URI = "http://localhost:8080/forms/hub/opf"
|
URI = "http://localhost:8080/forms/hub/test/opf"
|
||||||
[forms.isarfc]
|
[forms.isarfc]
|
||||||
URI = "http://localhost:8080/forms/isarfc/opf"
|
URI = "http://localhost:8080/forms/isarfc/test/opf"
|
||||||
[forms.passport]
|
[forms.passport]
|
||||||
URI = "http://localhost:8080/forms/passport/opf"
|
URI = "http://localhost:8080/forms/passport/test/opf"
|
||||||
[forms.mailmonk]
|
[forms.mailmonk]
|
||||||
URI = "http://localhost:9000/subscription/form"
|
URI = "http://localhost:9000/subscription/form"
|
||||||
|
@ -1,315 +0,0 @@
|
|||||||
---
|
|
||||||
title: coreboot and Heads as an alternative firmware for OpenPOWER Talos II
|
|
||||||
author: Krystian Hebel (3mdeb)
|
|
||||||
tags:
|
|
||||||
- openpower
|
|
||||||
- power9
|
|
||||||
- firmware
|
|
||||||
- coreboot
|
|
||||||
- hostboot
|
|
||||||
date: 2024-10-08
|
|
||||||
draft: false
|
|
||||||
---
|
|
||||||
|
|
||||||
This blog post presents coreboot ([spelled in lower case characters](https://doc.coreboot.org/#spelling-of-coreboot),
|
|
||||||
even when it is the first word in a sentence) and Heads as an alternative to
|
|
||||||
Hostboot and Skiroot/Petitboot, respectively. The project was extensive, a list
|
|
||||||
of changes (incomplete!) can be found in [this issue](https://github.com/linuxboot/heads/issues/1729).
|
|
||||||
The work was paid through [Open Collective funds](https://opencollective.com/3mdeb_com)
|
|
||||||
and Insurgo personal investments, as well as 3mdeb investment and [NGI0 PET fund](https://nlnet.nl/project/AccessibleSecurity/).
|
|
||||||
It took 3 years so far, with results available as part of [Dasharo firmware distribution](https://www.dasharo.com/).
|
|
||||||
|
|
||||||
As described on the [project's page](https://coreboot.org),
|
|
||||||
|
|
||||||
> coreboot is an extended firmware platform that delivers a lightning fast and
|
|
||||||
> secure boot experience on modern computers and embedded systems.
|
|
||||||
|
|
||||||
{{< image src="blog/coreboot.png" width="900" height="900" >}}
|
|
||||||
|
|
||||||
It aims to do the bare minimum required to make the hardware usable and pass
|
|
||||||
the control to next program, called the payload. In case of Talos II, that
|
|
||||||
payload is Skiboot, with [few changes on top](https://github.com/Dasharo/skiboot/tree/raptor-cs_talos-2)
|
|
||||||
to make it play along with Heads.
|
|
||||||
|
|
||||||
Speaking of [Heads](https://github.com/linuxboot/heads), it is:
|
|
||||||
|
|
||||||
> a minimal Linux that (...) provides a secure, flexible boot environment for
|
|
||||||
laptops, workstations and servers.
|
|
||||||
|
|
||||||
Heads provides a bootloader menu that starts final operating system through
|
|
||||||
kexec call. This is very similar to what Petitboot does, but Heads puts
|
|
||||||
security above everything else.
|
|
||||||
|
|
||||||
It is possible to use coreboot without Heads, but not the other way around.
|
|
||||||
Heads depends on structures created by coreboot, which just aren't present when
|
|
||||||
booting with Hostboot.
|
|
||||||
|
|
||||||
## Building and flashing coreboot
|
|
||||||
|
|
||||||
To build coreboot image, follow the steps below:
|
|
||||||
|
|
||||||
1. Clone the coreboot repository:
|
|
||||||
|
|
||||||
```
|
|
||||||
git clone https://github.com/Dasharo/coreboot.git \
|
|
||||||
--depth=1 -b raptor-cs_talos-2/rel_v0.7.0
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Start docker container:
|
|
||||||
|
|
||||||
```
|
|
||||||
cd coreboot
|
|
||||||
docker run --rm -it \
|
|
||||||
-v $PWD:/home/coreboot/coreboot \
|
|
||||||
-w /home/coreboot/coreboot \
|
|
||||||
-u "$(id -u):$(id -g)" \
|
|
||||||
coreboot/coreboot-sdk:0ad5fbd48d /bin/bash
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Configure and start the build process inside of the container:
|
|
||||||
|
|
||||||
```
|
|
||||||
(docker) cp configs/config.raptor-cs-talos-2 .config
|
|
||||||
(docker) make olddefconfig
|
|
||||||
(docker) make
|
|
||||||
```
|
|
||||||
|
|
||||||
After image is built you can exit the container, either with `exit` or Ctrl+D.
|
|
||||||
To flash it to your platform:
|
|
||||||
|
|
||||||
0. Make sure you're running System Package v2.00, if not, get it from
|
|
||||||
[here](https://wiki.raptorcs.com/wiki/Talos_II/Firmware) and
|
|
||||||
[update/downgrade](https://wiki.raptorcs.com/wiki/Updating_Firmware#Updating_the_OpenPOWER_firmware).
|
|
||||||
Start the platform once so SEEPROM is also updated, then power off.
|
|
||||||
|
|
||||||
1. Copy images to BMC:
|
|
||||||
|
|
||||||
```
|
|
||||||
scp -O build/bootblock.signed.ecc root@<BMC_IP>:/tmp/bootblock.signed.ecc
|
|
||||||
scp -O build/coreboot.rom.signed.ecc root@<BMC_IP>:/tmp/coreboot.rom.signed.ecc
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Log in to BMC through SSH:
|
|
||||||
|
|
||||||
```
|
|
||||||
ssh root@<BMC_IP>
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Flash both partitions:
|
|
||||||
|
|
||||||
```
|
|
||||||
pflash -e -P HBB -p /tmp/bootblock.signed.ecc
|
|
||||||
pflash -e -P HBI -p /tmp/coreboot.rom.signed.ecc
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Boot the platform as usual and enjoy coreboot running on Talos II:
|
|
||||||
|
|
||||||
[![asciicast](https://asciinema.org/a/zkQV1KhxY4n6IrlzssuvFHHS5.svg)]https://asciinema.org/a/zkQV1KhxY4n6IrlzssuvFHHS5
|
|
||||||
|
|
||||||
## Building and flashing Heads
|
|
||||||
|
|
||||||
Reminder: Heads requires coreboot. Instructions above **must** be performed
|
|
||||||
before flashing Heads. It also requires a [compatible USB security dongle](https://osresearch.net/Prerequisites#usb-security-dongles-aka-security-token-aka-smartcard)
|
|
||||||
and TPM (more about it later).
|
|
||||||
|
|
||||||
> Technically, TPM isn't a hard requirement of Heads, however its usefulness
|
|
||||||
> without it is very limited, up to a point where it doesn't have any advantages
|
|
||||||
> over Petitboot.
|
|
||||||
|
|
||||||
1. Just as earlier, start with cloning the repository:
|
|
||||||
|
|
||||||
```
|
|
||||||
git clone https://github.com/Dasharo/heads.git \
|
|
||||||
--depth=1 -b raptor-cs_talos-2/release
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Start docker container:
|
|
||||||
|
|
||||||
```
|
|
||||||
cd heads
|
|
||||||
docker run --rm -it \
|
|
||||||
-v $PWD:/home/heads/heads \
|
|
||||||
-w /home/heads/heads \
|
|
||||||
-u "$(id -u):$(id -g)" \
|
|
||||||
3mdeb/heads-docker:2.4.0 /bin/bash
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Build:
|
|
||||||
|
|
||||||
```
|
|
||||||
(docker) make BOARD=talos-2
|
|
||||||
```
|
|
||||||
|
|
||||||
This will take a while, wait for it to finish and then exit the container. In
|
|
||||||
the process, a coreboot image will also be built, but with slightly different
|
|
||||||
configuration. For security and reproducible images, `BUILD_TIMELESS` is always
|
|
||||||
enabled. While it actually strips file paths, it also removes file names and
|
|
||||||
line numbers from asserts in the code. It makes reporting and debugging
|
|
||||||
potential issues harder, so we suggest using coreboot built manually, at least
|
|
||||||
for the time being.
|
|
||||||
|
|
||||||
Steps for flashing Heads are similar to those done for coreboot.
|
|
||||||
|
|
||||||
1. Copy the Heads binary to the BMC (assuming in the Heads root directory):
|
|
||||||
|
|
||||||
```
|
|
||||||
scp -O build/zImage.bundled root@<BMC_IP>:/tmp/zImage.bundled
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Log in to the BMC:
|
|
||||||
|
|
||||||
```
|
|
||||||
ssh root@<BMC_IP>
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Flash the BOOTKERNEL partition with Heads:
|
|
||||||
|
|
||||||
```
|
|
||||||
pflash -e -P BOOTKERNEL -p /tmp/zImage.bundled
|
|
||||||
```
|
|
||||||
|
|
||||||
Answer yes to the prompt and wait for the process to finish. After that, start
|
|
||||||
the platform and begin [configuring Heads](https://osresearch.net/Configuring-Keys/).
|
|
||||||
|
|
||||||
## PNOR emulation
|
|
||||||
|
|
||||||
Flash device can be emulated by BMC, which is something we were often using for
|
|
||||||
development and testing. This saves a lot of time which would be spent flashing,
|
|
||||||
as well as reduces the wear of flash device.
|
|
||||||
|
|
||||||
However, this still requires System Package v2.00, and if this is different than
|
|
||||||
what real flash holds, SEEPROM will have to be updated when switching between
|
|
||||||
physical and emulated image. Also, this approach doesn't survive BMC reboots and
|
|
||||||
power losses. BMC doesn't have enough space to keep full PNOR image in
|
|
||||||
non-volatile memory, so `tmpfs` must be used for emulation. Don't try to put
|
|
||||||
more than one image in `tmpfs` or BMC **will** run out of RAM, which most likely
|
|
||||||
will require manual power cycle.
|
|
||||||
|
|
||||||
To start, obtain full flash image, either by downloading it from
|
|
||||||
[RaptorCS release page](https://wiki.raptorcs.com/wiki/Talos_II/Firmware) or
|
|
||||||
reading from existing image on BMC with:
|
|
||||||
|
|
||||||
```
|
|
||||||
pflash -r /tmp/talos.pnor
|
|
||||||
```
|
|
||||||
|
|
||||||
After that, you can "flash" the partitions mentioned earlier by adding
|
|
||||||
additional parameters to use the file instead of physical flash:
|
|
||||||
|
|
||||||
```
|
|
||||||
pflash -f -P <partition> -p <image_file> -F /tmp/talos.pnor
|
|
||||||
```
|
|
||||||
|
|
||||||
Change `<partition>` to one of `HBB`, `HBI`, `BOOTKERNEL` and `<image_file>` to
|
|
||||||
`/tmp/bootblock.signed.ecc`, `/tmp/coreboot.rom.signed.ecc` or
|
|
||||||
`/tmp/zImage.bundled`, respectively.
|
|
||||||
|
|
||||||
To tell BMC to present the contents of this file as flash, run:
|
|
||||||
|
|
||||||
```
|
|
||||||
mboxctl --backend file:/tmp/talos.pnor
|
|
||||||
```
|
|
||||||
|
|
||||||
Sometimes this command fails with a timeout, in that case run it again until it
|
|
||||||
succeeds.
|
|
||||||
|
|
||||||
> We've noticed that sometimes, despite no error message printed, physical flash
|
|
||||||
> was used anyway. It is easy to spot when one copy has Hostboot and the other
|
|
||||||
> has coreboot, but it can be missed when both images have different versions of
|
|
||||||
> coreboot. It caused us few hours of unnecessary debugging of issues that were
|
|
||||||
> already fixed...
|
|
||||||
|
|
||||||
With the file now mounted, platform can be started. Host firmware and OS
|
|
||||||
shouldn't be able to tell the difference, except for different reported erase
|
|
||||||
block size and maybe different access times.
|
|
||||||
|
|
||||||
To get back to original flash, run:
|
|
||||||
|
|
||||||
```
|
|
||||||
mboxctl --backend vpnor
|
|
||||||
```
|
|
||||||
|
|
||||||
It will report an error (`Failed to post message: Connection timed out`), but
|
|
||||||
will revert to physical device nonetheless. This can be confirmed by checking
|
|
||||||
the output of `mboxctl --lpc-state`:
|
|
||||||
|
|
||||||
```
|
|
||||||
root@talos:~# mboxctl --lpc-state
|
|
||||||
LPC Bus Maps: Flash Device
|
|
||||||
```
|
|
||||||
|
|
||||||
Since the file is now the full image with coreboot (and optionally Heads), it
|
|
||||||
can be simply written to flash, should you choose to accept it:
|
|
||||||
|
|
||||||
```
|
|
||||||
pflash -E -p /tmp/talos.pnor
|
|
||||||
```
|
|
||||||
|
|
||||||
## Noticeable differences between Hostboot and coreboot
|
|
||||||
|
|
||||||
For those wondering why we even started this project, here are some of the
|
|
||||||
differences between Hostboot and coreboot.
|
|
||||||
|
|
||||||
First of all, coreboot uses C, while Hostboot was written in C++. The latter can
|
|
||||||
be viewed as a complete operating system - it can use multiple threads
|
|
||||||
simultaneously, manages virtual memory and uses memory swapping (even before RAM
|
|
||||||
is trained). Each major [istep](https://wiki.raptorcs.com/w/images/b/bd/IPL-Flow-POWER9.pdf)
|
|
||||||
(IPL Step, which in turn stands for Initial Program Load) is a separate
|
|
||||||
application, with some common dynamically loaded libraries. coreboot, on the
|
|
||||||
other hand, runs all of the code in just 3 separate stages - bootblock, romstage
|
|
||||||
and ramstage. This allows for tighter linking process, which reduces the final
|
|
||||||
size of the code.
|
|
||||||
|
|
||||||
Another significant difference is reduced amount of RAS (Reliability,
|
|
||||||
Availability, Serviceability) features enabled in coreboot. Talos II is often
|
|
||||||
used as a workstation, and while RAS has its uses in servers (it is preferred
|
|
||||||
to start with partially working hardware than not starting at all), for home
|
|
||||||
users booting fast is usually more important. Because of that coreboot doesn't
|
|
||||||
support bad DQ masking for DRAM, it can also optionally skip initial RAM
|
|
||||||
scrubbing. Because of these reasons, as well as smaller size of code in general,
|
|
||||||
booting coreboot is significantly faster than Hostboot - some preliminary
|
|
||||||
results can be found [here](https://github.com/3mdeb/openpower-coreboot-docs/blob/main/devnotes/user_perspective.md).
|
|
||||||
|
|
||||||
Another technical difference is the way data is passed to Skiboot. Hostboot uses
|
|
||||||
HDAT - a format specific to this particular firmware. For coreboot, a device
|
|
||||||
tree conforming to a well-defined specification is used. In fact, Skiboot
|
|
||||||
internally converts HDAT to device tree anyway. At the moment, some information
|
|
||||||
is not presented by coreboot (https://github.com/Dasharo/dasharo-issues/issues/446,
|
|
||||||
https://github.com/Dasharo/dasharo-issues/issues/32), but those seem to be
|
|
||||||
rather cosmetic than anything else - if you know about something that requires
|
|
||||||
those pieces of information to be present, let us know.
|
|
||||||
|
|
||||||
## TPM
|
|
||||||
|
|
||||||
TPM is an integral part of security mechanisms added by Heads. As existing I2C
|
|
||||||
TPMs were [hard to obtain](https://github.com/3mdeb/openpower-coreboot-docs/blob/main/devnotes/tpm_over_i2c.md)
|
|
||||||
at the time we were working on this part of the project, and LPC TPMs couldn't
|
|
||||||
be used [due to the way POWER9 processor exposed access to LPC bus](https://github.com/3mdeb/openpower-coreboot-docs/blob/main/devnotes/tpm_over_lpc.md#tpm-over-lpc-interface),
|
|
||||||
we had to [consider other options](https://github.com/3mdeb/openpower-coreboot-docs/blob/main/devnotes/tpm.md).
|
|
||||||
|
|
||||||
The solution we ended up with was to create our own [I2C TPM 1.2 module](https://docs.dasharo.com/variants/talos_2/tpm-support/),
|
|
||||||
based on Infineon SLB9645TT1.2 chip. This chip isn't supported by drivers in
|
|
||||||
Hostboot and upstream Skiboot, to make use of it you have to use coreboot and
|
|
||||||
Dasharo's fork of Skiboot.
|
|
||||||
|
|
||||||
{{< image src="blog/TPM-1.2-Talos-II.jpg" >}}
|
|
||||||
|
|
||||||
## Links and references
|
|
||||||
|
|
||||||
Here are some links to documentation related to this project:
|
|
||||||
|
|
||||||
- [main user documentation page](https://docs.dasharo.com/variants/talos_2/overview/)
|
|
||||||
- [release notes and binaries](https://docs.dasharo.com/variants/talos_2/releases/)
|
|
||||||
- [list of known issues](https://github.com/Dasharo/dasharo-issues/labels/raptor-cs_talos-2),
|
|
||||||
if you want to create new issue remember to add proper tag
|
|
||||||
- [scripts for dumping debug data and logs](https://github.com/3mdeb/openpower-coreboot-docs/tree/main/devnotes/scripts)
|
|
||||||
- [dump of SCOM accesses and other debug output divided by isteps](https://github.com/3mdeb/openpower-coreboot-docs/tree/main/logs/scom_dumps)
|
|
||||||
- [other uncategorized developer notes](https://github.com/3mdeb/openpower-coreboot-docs/blob/main/devnotes)
|
|
||||||
- [Open Source Firmware Slack channel dedicated to coreboot on OpenPOWER](https://osfw.slack.com/archives/C01BHE47JSW)
|
|
||||||
|
|
||||||
We invite you to test for yourselves and share the results, both good and bad,
|
|
||||||
either on channels listed above or in issue.
|
|
||||||
|
|
||||||
The OpenPOWER Talos II running coreboot and heads was presented on Open Source
|
|
||||||
Firmware Conference 2024 in Bochum, Germany. Video from the presentaion is
|
|
||||||
availabel at [Vimeo](https://vimeo.com/1007701325).
|
|
@ -1,174 +0,0 @@
|
|||||||
---
|
|
||||||
title: OpenROAD and OpenPOWER Microwatt core for low-power ASIC applications
|
|
||||||
author: Ganesan Narayanasamy
|
|
||||||
tags:
|
|
||||||
- openpower
|
|
||||||
- adg
|
|
||||||
- academic
|
|
||||||
- openroad
|
|
||||||
date: 2024-01-25
|
|
||||||
draft: false
|
|
||||||
---
|
|
||||||
# OpenROAD and OpenPOWER - Microwatt core for low-power ASIC applications
|
|
||||||
|
|
||||||
## OpenPOWER for Energy-efficient applications
|
|
||||||
|
|
||||||
The [OpenPOWER](https://openpowerfoundation.org) microwatt processor is an open-source soft process core implementation of the IBM [POWER ISA](https://openpowerfoundation.org/specifications/isa/) architecture that is designed to be highly energy-efficient. This processor is an example of the growing trend of designing low-power processors that can be used in a variety of applications, including mobile devices, Internet of Things (IoT) devices, and more.
|
|
||||||
|
|
||||||
### Microwatt
|
|
||||||
Microwatt is a multi-core processor capable of running at up to 2.3 GHz and using the Power ISA v3.0 specification. It can run on lower cost FPGAs. It has a memory management unit, privilege modes, interrupts, a floating-point unit, an interrupt controller and a level-2 cache, so that it can boot Linux.
|
|
||||||
|
|
||||||
The Microwatt processor was implemented using the [OpenROAD](https://theopenroadproject.org/) native design flow - [OpenROAD-flow-scripts](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts) . It is available as part of the EDA cloud native solution in OpenPOWER. OpenROAD delivers an autonomous, No-Human-In-Loop, fully-integrated RTL-GDSII flow for the design of digital integrated circuits across a range of technologies. OpenROAD includes specific features for low power design that enabled the design to specification for Microwatt.
|
|
||||||
|
|
||||||
## Microwatt Design Flow using OpenROAD
|
|
||||||
|
|
||||||
The figures below show the OpenROAD flow using [OpenROAD-flow-scripts](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts)
|
|
||||||
|
|
||||||
### OpenROAD High-level Flow
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image1.png" >}}
|
|
||||||
|
|
||||||
### OpenROAD Flow Scripts - A detailed Physical Implementation Flow
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image2.png" >}}
|
|
||||||
|
|
||||||
## Design Methodology
|
|
||||||
|
|
||||||
Microwatt was implemented using Skywater130nm (What production versions are supported- Is there a link to share about it? It consists of 6 macros and over 800K standard cells.
|
|
||||||
|
|
||||||
The hardware description for Microwatt was defined using Verilog in OpenPOWER. The RTL was synthesized in OpenROAD using Yosys to obtain a power-efficient mapping. [OpenSTA](https://github.com/The-OpenROAD-Project/OpenSTA) is an integrated and incremental timing and power analysis engine built into OpenROAD.
|
|
||||||
|
|
||||||
### Synthesis
|
|
||||||
The OpenROAD flow uses a synthesized netlist from Yosys to transform the RTL into an elaborated design with corresponding technology mapped primitives to generate an optimized netlist based on the input on specified timing constraints. This netlist was used to create the floorplan for detailed physical implementation.
|
|
||||||
|
|
||||||
### Placement and Routing
|
|
||||||
|
|
||||||
#### Floorplan Import design
|
|
||||||
|
|
||||||
Microwatt has 6 macros that were placed using the automatic macro placement in ORFS.
|
|
||||||
|
|
||||||
- RAM512 1 392.925
|
|
||||||
- RAM32 1RW1R
|
|
||||||
- Multiply_add_64x64
|
|
||||||
- RAM32_1RW1R
|
|
||||||
- Microwatt_FP_DFFRFile
|
|
||||||
- Multiply_add_64x64
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image3.png" >}}
|
|
||||||
|
|
||||||
macro placement
|
|
||||||
|
|
||||||
### Power Distribution
|
|
||||||
|
|
||||||
There are 5 metal layers in total in Microwatt. Metal 1 is horizontal and M2 is vertical followed by M3, M4 and M5 in horizontal direction. Power distribution is using M1, M4 and M5. The M1 layer is horizontal on the standard cell rails to enable power for the standard cells. VIA2 and VIA3 power vias are dropped. M4 power stripes and M5 power stripes are drawn. An alternate pattern of VSS and VDD pattern is followed
|
|
||||||
|
|
||||||
The figure below shows the PDN grid generated.
|
|
||||||
|
|
||||||
This is the complete PDN of the floor plan.
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image4.png" >}}
|
|
||||||
|
|
||||||
### Placement
|
|
||||||
|
|
||||||
Global Placement places standard cells to minimize wire length and avoid congestion during routing. Detailed placement further legalizes standard cell placement to remove overlaps and fix orientation for optimal routing. This step also includes the use of the resizer to optimize timing using techniques like buffer insertion, sizing, pin swapping etc.
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image5.png" >}}
|
|
||||||
Figure: Placement Density View
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image6.png" >}}
|
|
||||||
|
|
||||||
Figure: Placement Optimization by Resizer
|
|
||||||
|
|
||||||
### Clock Tree Synthesis
|
|
||||||
|
|
||||||
ORFS automatically synthesizes clock trees to meet timing constraints and inserts buffers as needed to balance skew on high fanout nets. Filler cells are also added to meet manufacturing density requirements.
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image7.png" >}}
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image8.png" >}}
|
|
||||||
Figure: ext_clk Clock Tree Viewer
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image9.png" >}}
|
|
||||||
|
|
||||||
Figure: jtag_tck Clock Tree Viewer
|
|
||||||
|
|
||||||
### Timing Analysis
|
|
||||||
|
|
||||||
OpenROAD includes a built-in STA engine that is used to incrementally analyze timing at various flow stages. The GUI enables easy visualization of timing paths and setup/hold checks.
|
|
||||||
|
|
||||||
Shown below are post-CTS timing views for setup and hold times.
|
|
||||||
|
|
||||||
#### Setup Timing Path
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image10.png" >}}
|
|
||||||
Figure: Setup Timing Path
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image11.png" >}}
|
|
||||||
|
|
||||||
Figure: Hold Timing Path
|
|
||||||
|
|
||||||
#### Routing
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image12.png" >}}
|
|
||||||
Figure: Routing Congestion View
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image13.png" >}}
|
|
||||||
|
|
||||||
Figure: Detail Routing
|
|
||||||
|
|
||||||
#### Show CTS after detailed placement.
|
|
||||||
|
|
||||||
CTS is shown below by highlighting the clock nets before and after CTS
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image14.png" >}}
|
|
||||||
Figure: ext_clk source CTS
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image15.png" >}}
|
|
||||||
|
|
||||||
Figure: jtag tck source CTS
|
|
||||||
|
|
||||||
#### How was timing optimization done using the resizer repair commands?
|
|
||||||
|
|
||||||
Using resizer commands the following is done at the placement step. The details are available in 3_4_resizer.log
|
|
||||||
|
|
||||||
1) 51 input buffers are added
|
|
||||||
2) 87 output buffers are added
|
|
||||||
3) 7237 buffers are added in 9711 nets
|
|
||||||
4) 23377 instances are resized
|
|
||||||
|
|
||||||
#### What tradeoffs were made for timing vs power?
|
|
||||||
|
|
||||||
We optimize for timing and less so for power. We do have some power recovery features in development that should be available soon.
|
|
||||||
|
|
||||||
#### Where is the static IR estimation report?
|
|
||||||
|
|
||||||
In 6_report.log when using ORFS
|
|
||||||
|
|
||||||
#### Did you run a simulation for power analysis?
|
|
||||||
|
|
||||||
We can provide a VCD (toggle counts) from simulation for improved accuracy. OR doesn't do simulation itself.
|
|
||||||
|
|
||||||
#### What kind of signoff was used to verify final timing and power (PrimeTime, Calibre etc.)
|
|
||||||
|
|
||||||
We have not done any signoff for final timing and power using Prime Time & Calibre.Only OpenROAD flow is being executed.
|
|
||||||
|
|
||||||
### Physical Verification
|
|
||||||
|
|
||||||
Once the physical layout (i.e., the GDS file) is complete, the design is ready for final verification and sign-off. This involves running a series of tests to ensure that the design meets all of the necessary requirements and specifications. Any issues that are found can be addressed, and the design can be re-synthesized and re-implemented until it meets all of the necessary criteria.
|
|
||||||
|
|
||||||
Finally, the design is ready for fabrication. The GDSII layout file is sent to a foundry, where the physical chip is manufactured. Once the chip is fabricated, it can be assembled into a larger system or device, such as a mobile phone or IoT device.
|
|
||||||
|
|
||||||
{{< image src="blog/openroad_image16.png" >}}
|
|
||||||
|
|
||||||
Figure: Microwatt Layout generated through OpenROAD
|
|
||||||
|
|
||||||
## Results
|
|
||||||
|
|
||||||
Showcase results here. Final GDSII. Power and Timing achieved, run times and state the potential for such application development for the future.
|
|
||||||
|
|
||||||
|
|
||||||
Worstcase voltage:1.80e+00 V
|
|
||||||
Average IR Drop : 4.62e-04 V
|
|
||||||
Worstcase IR Drop: 1.95e-03 V
|
|
||||||
|
|
||||||
As a result, it is possible to create highly efficient and effective microwatt processors that can be used in a variety of different devices and applications, from mobile phones to IoT devices and beyond. The combination of the OpenPOWER microwatt processor and the OpenROAD design flow represents a powerful toolset for designers looking to create low-power, high-performance processors for modern electronics.
|
|
@ -1,35 +0,0 @@
|
|||||||
---
|
|
||||||
title: Solid Silicon Joins OpenPOWER Foundation
|
|
||||||
categories:
|
|
||||||
- blogs
|
|
||||||
tags:
|
|
||||||
- openpower
|
|
||||||
- openpower-foundation
|
|
||||||
- linux-foundation
|
|
||||||
- open-source
|
|
||||||
- cloud
|
|
||||||
- open-source-hardware
|
|
||||||
- member
|
|
||||||
date: 2024-06-13
|
|
||||||
draft: false
|
|
||||||
---
|
|
||||||
|
|
||||||
Today, we are pleased to announce that Solid Silicon, a fabless semiconductor company based in Colorado, US, has joined the OpenPOWER Foundation as a Platinum member.
|
|
||||||
|
|
||||||
The OpenPOWER Foundation is committed to an open, community-driven technology development process, which is critical to the continued growth of open collaboration around the POWER architecture.
|
|
||||||
|
|
||||||
"_OpenPOWER Foundation is pleased to welcome Solid Silicon to the foundation and our community, and looks forward to our collaboration in community-driven open development that broadens customer computing choice_", said [James Kulina](https://www.linkedin.com/in/james-kulina/), Executive Director, OpenPOWER Foundation.
|
|
||||||
|
|
||||||
Solid Silicon Corporation is developing open source software and secure chip designs based on standardized, self-sovereign, future-proof silicon to give developers the ability to own, control, and trust their computing devices down to the silicon.
|
|
||||||
|
|
||||||
Solid Silicon currently is [developing](https://www.linkedin.com/posts/solid-silicon_raptor-computing-sys-on-x-activity-7121183443475992576-kq_A?utm_source=share&utm_medium=member_desktop) two new silicon devices based on the OpenPOWER ISA. The first is the X-class chip line that targets Edge/IoT applications while the second line, the S-class chip, targets Data Center applications. Both chip lines will adopt the latest POWER ISA 3.1 specifications leading to new product capabilities within the OpenPOWER ecosystem.
|
|
||||||
|
|
||||||
"_Solid Silicon's OpenPOWER CPUs are built on and fully comply with the open, standardized POWER ISA administered by the OpenPOWER Foundation. This commitment to open standards ensures that our customers benefit from the latest advancements in processor technology, while also providing the transparency and security that comes from an open and unified software ecosystem. In our view, the vendor-neutral approach to ISA standardization that the OpenPOWER Foundation provides is crucial to both avoiding vendor lock-in and ensuring a diverse software ecosystem while simultaneously ensuring long-term application compatibility._", said [Tim Pearson](https://www.linkedin.com/company/solid-silicon/), CTO and Founder.
|
|
||||||
|
|
||||||
"_At a fundamental level, akin to public cryptography, the entire OpenPOWER architecture and standard are open, auditable, and completely verifiable. With Solid Silicon's firmware, software, and tools also being fully open source, our customers benefit immensely by deploying self-sovereign silicon solutions from the edge to the data center. We are thrilled to partner with the OpenPOWER Foundation to foster and expand this incredible developer community_", says [Todd Rooke](https://www.linkedin.com/in/toddrooke/), CEO and Founder.
|
|
||||||
|
|
||||||
The OpenPOWER Foundation looks forward to collaborating with Solid Silicon
|
|
||||||
in developing new and exciting POWER based silicon through open standards and community driven collaboration.
|
|
||||||
|
|
||||||
Learn more about [Solid Silicon](https://solidsilicon.com).
|
|
||||||
|
|
@ -1,9 +1,9 @@
|
|||||||
---
|
---
|
||||||
title: "Timothy Pearson"
|
title: "Timothy Pearson"
|
||||||
position: "Chairman"
|
position: "Director"
|
||||||
weight: 1
|
weight: 3000
|
||||||
date: 2024-06-27
|
date: 2021-03-03
|
||||||
start: 2024-06-14
|
start: 2021-01-30
|
||||||
end: 2026-06-14
|
end: 2023-03-27
|
||||||
draft: false
|
draft: false
|
||||||
---
|
---
|
||||||
|
@ -1,25 +0,0 @@
|
|||||||
---
|
|
||||||
title: "IBM TechXchange 2024"
|
|
||||||
eventdates:
|
|
||||||
from: 2024-10-21
|
|
||||||
eventplace: San Jose, CA
|
|
||||||
register: https://reg.tools.ibm.com/flow/ibm/techxchange24/reg/
|
|
||||||
schedule: https://reg.tools.ibm.com/flow/ibm/techxchange24/sessioncatalog/page/sessioncatalog?tab.sessioncatalogtabs=option_1601178495160
|
|
||||||
date: 2024-10-17
|
|
||||||
draft: false
|
|
||||||
---
|
|
||||||
|
|
||||||
The OpenPOWER Foundation is participating at [IBM TechXchange 2024 event](https://www.ibm.com/community/ibm-techxchange-conference/). We will be giving a talk on in the OSS track with our member company [Solid Silicon](https://solidsilicon.com/desktop/index.html), who is developing next generation POWER processors.
|
|
||||||
|
|
||||||
Come to our talk on Monday Oct. 21st.
|
|
||||||
[Open Comuting Built on OpenPOWER](https://reg.tools.ibm.com/flow/ibm/techxchange24/sessioncatalog/page/sessioncatalog?search=open%20source&tab.sessioncatalogtabs=option_1601178495160#:~:text=computing%20built%20on-,OpenPOWER,-%5B4158%5D)
|
|
||||||
|
|
||||||
OpenPOWER will also be showcasing within the IBM POWER booth. Please stop by and say Hello!
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1,23 +0,0 @@
|
|||||||
---
|
|
||||||
title: "Open Compute North American Summit 2024"
|
|
||||||
eventdates:
|
|
||||||
from: 2024-10-15
|
|
||||||
eventplace: San Jose, CA
|
|
||||||
register: https://2024ocpregional.fnvirtual.app/
|
|
||||||
schedule: https://2024ocpregional.fnvirtual.app/a/schedule/
|
|
||||||
date: 2024-09-16
|
|
||||||
draft: false
|
|
||||||
---
|
|
||||||
|
|
||||||
The OpenPOWER Foundation is a sponsor with booth #C52. We will be sharing our booth with [Solid Silicon](https://solidsilicon.com/desktop/index.html), who is developing next generation POWER processors. Stop by and say hello!
|
|
||||||
|
|
||||||
[OCP24 registration link](https://www.opencompute.org/summit/global-summit/registration)
|
|
||||||
|
|
||||||
|
|
||||||
The OCP Summit is the premier event uniting the most forward-thinking minds in open IT Ecosystem development. The Summit presents a unique platform for our Community from around the globe to share their insights, foster partnerships and showcase cutting-edge advancements in open hardware and software.
|
|
||||||
|
|
||||||
The 2024 OCP Global Summit theme is "From Ideas to Impact". This encapsulates the transformative journey at the heart of the Open Compute Project. This year's theme reflects OCP's commitment to fostering innovation that transcends theoretical discussions and manifests into real-world solutions. As the pace of technological evolution accelerates and development cycles shorten, our industry is forced to rapidly respond to emerging trends and needs. By harnessing the collective expertise of our global community, we turn visionary ideas into groundbreaking technologies that drive openness, efficiency, sustainability, scalability and growth in the data center industry. Our focus honors the relentless pursuit of progress and the profound impact that OCP's community-driven innovation can achieve. OCP transforms concepts into impactful advancements.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1,10 +0,0 @@
|
|||||||
---
|
|
||||||
title: Loyola Institute of Technology
|
|
||||||
image: loyolainstitute.png
|
|
||||||
country: India
|
|
||||||
link: https://www.lit.edu.in/
|
|
||||||
level: academic
|
|
||||||
joined: 2017
|
|
||||||
date: 2021-03-31
|
|
||||||
draft: false
|
|
||||||
---
|
|
@ -1,10 +0,0 @@
|
|||||||
---
|
|
||||||
title: Power Maroc
|
|
||||||
image: powerm.png
|
|
||||||
country: Morocco
|
|
||||||
link: https://powerm.ma/
|
|
||||||
level: silver
|
|
||||||
joined: 2024
|
|
||||||
date: 2024-01-01
|
|
||||||
draft: false
|
|
||||||
---
|
|
@ -1,12 +0,0 @@
|
|||||||
---
|
|
||||||
title: Solid Silicon
|
|
||||||
image: solidsilicon.png
|
|
||||||
country: USA
|
|
||||||
link: https://solidsilicon.com
|
|
||||||
level: platinum
|
|
||||||
joined: 2024
|
|
||||||
date: 2024-05-18
|
|
||||||
draft: false
|
|
||||||
---
|
|
||||||
|
|
||||||
Solid Silicon Corporation is a fabless silicon company developing open source software and secure chip designs based on standardized, sovereign, future-proof silicon, which gives developers the ability to own, control, and trust their computing devices down to the silicon.
|
|
@ -1,10 +0,0 @@
|
|||||||
---
|
|
||||||
title: University of California - CHIPS Lab
|
|
||||||
image: ucla.png
|
|
||||||
country: USA
|
|
||||||
link: https://www.chips.ucla.edu/
|
|
||||||
level: academic
|
|
||||||
joined: 2021
|
|
||||||
date: 2021-08-31
|
|
||||||
draft: false
|
|
||||||
---
|
|
@ -1,7 +0,0 @@
|
|||||||
---
|
|
||||||
title: "Brad Frey"
|
|
||||||
#image:
|
|
||||||
member: ibm
|
|
||||||
date: 2021-08-05
|
|
||||||
draft: false
|
|
||||||
---
|
|
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Ganesan Narayanasamy
|
title: Ganesan Narayanasamy
|
||||||
member: ganesannarayansamy
|
member: ibm
|
||||||
date: 2023-12-04
|
date: 2022-02-04
|
||||||
draft: false
|
draft: false
|
||||||
---
|
---
|
||||||
|
@ -1,7 +0,0 @@
|
|||||||
---
|
|
||||||
title: "Jeremy Kerr"
|
|
||||||
#image:
|
|
||||||
member: codeconstruct
|
|
||||||
date: 2021-08-05
|
|
||||||
draft: false
|
|
||||||
---
|
|
@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
title: "Paul Mackerras"
|
title: "Paul Mackerras"
|
||||||
image:
|
image:
|
||||||
member: paulmackerras
|
member: ibm
|
||||||
date: 2023-12-06
|
date: 2021-03-06
|
||||||
draft: false
|
draft: false
|
||||||
---
|
---
|
||||||
|
@ -1,11 +1,12 @@
|
|||||||
---
|
---
|
||||||
title: "Timothy Pearson"
|
title: "Timothy Pearson"
|
||||||
image: "timothypearson-200.jpg"
|
image: "timothypearson-200.jpg"
|
||||||
member: solidsilicon
|
member: raptorcomputingsystems
|
||||||
date: 2024-06-14
|
date: 2021-03-03
|
||||||
draft: false
|
draft: false
|
||||||
---
|
---
|
||||||
|
|
||||||
Timothy has worked in multiple areas of high performance systems design over the past 15 years,
|
Timothy has worked in multiple areas of high performance systems design over the past 15 years,
|
||||||
from hardware / firmware to kernel and application development, with a strong emphasis on open solutions and system security.
|
from hardware / firmware to kernel and application development, with a strong emphasis on open solutions and system security.
|
||||||
|
In his time at Raptor he has been heavily involved in the inception and creation of the Talos and Blackbird lines of secure,
|
||||||
|
owner-controlled OpenPOWER machines, along with the FlexVer remote attestation technology.
|
||||||
|
@ -1,32 +0,0 @@
|
|||||||
---
|
|
||||||
title: PowerVM on POWER Architecture Reference
|
|
||||||
group: IBM
|
|
||||||
tags:
|
|
||||||
- powervm
|
|
||||||
- hypervisor
|
|
||||||
- software
|
|
||||||
- linux
|
|
||||||
- unix
|
|
||||||
- operatingsystem
|
|
||||||
- distribution
|
|
||||||
- architecture
|
|
||||||
feedback:
|
|
||||||
queue: PAPR
|
|
||||||
date: 2024-10-28
|
|
||||||
draft: false
|
|
||||||
---
|
|
||||||
|
|
||||||
|
|
||||||
## Power Architecture Platform Requirements. ##
|
|
||||||
|
|
||||||
The purpose of this document is to detail a stable platform architecture to be used by:
|
|
||||||
|
|
||||||
- Platforms defined by the POWER ISA Specification that desire to be compatible with operating systems that run on the PowerVM Hypervisor.
|
|
||||||
- Operating systems that desire to be compatible with running on the PowerVM Hypervisor.
|
|
||||||
|
|
||||||
This architecture specification provides a comprehensive computer system platform-to-software interface definition.
|
|
||||||
It documents minimum system requirements, enabling the development and porting of software to a range of compatible industry-standard computer systems from workstations through servers based on the PowerISA.
|
|
||||||
|
|
||||||
|
|
||||||
This document is maintained by IBM and is made available through the OpenPOWER Foundation to support the work of the OpenPOWER Foundation and open source communities in supporting the POWER platform.
|
|
||||||
Comments, questions, and suggestions can be submitted through the OpenPOWER Foundation Request Tracker.
|
|
@ -1,9 +0,0 @@
|
|||||||
members:
|
|
||||||
- ibm
|
|
||||||
- openpowerfoundation
|
|
||||||
- oregonstateuniversity
|
|
||||||
- vantosh
|
|
||||||
nextmeeting :
|
|
||||||
date: 2024-05-21
|
|
||||||
time: 21:00
|
|
||||||
|
|
@ -1,5 +1,8 @@
|
|||||||
members:
|
members:
|
||||||
- ibm
|
- ibm
|
||||||
|
- redsemiconductor
|
||||||
- vantosh
|
- vantosh
|
||||||
- hexintechnologies
|
nextmeeting :
|
||||||
|
date: 2022-06-09
|
||||||
|
time: 07:00
|
||||||
|
|
||||||
|
@ -1,9 +1,10 @@
|
|||||||
members:
|
members:
|
||||||
|
- antmicro
|
||||||
- google
|
- google
|
||||||
- ibm
|
- ibm
|
||||||
- oregonstateuniversity
|
- oregonstateuniversity
|
||||||
- vantosh
|
- vantosh
|
||||||
nextmeeting :
|
nextmeeting :
|
||||||
date: 2023-03-09
|
date: 2022-06-09
|
||||||
time: 16:00
|
time: 15:00
|
||||||
|
|
||||||
|
@ -1,4 +0,0 @@
|
|||||||
versions:
|
|
||||||
- number: "10.60"
|
|
||||||
date: 2024-11-04
|
|
||||||
download: https://files.openpower.foundation/s/XFgfMaqLMD5Bcm8
|
|
@ -1,55 +0,0 @@
|
|||||||
<script>
|
|
||||||
var formname = "form.contactForm";
|
|
||||||
var submiturl = "{{ .Site.Params.forms.contact.URI }}";
|
|
||||||
{{ if .Site.Params.forms.contact.goal }}
|
|
||||||
var goal = "{{ .Site.Params.forms.contact.goal }}";
|
|
||||||
{{ end }}
|
|
||||||
{{ $feedbackqueue := .Param "feedback.queue" }}
|
|
||||||
</script>
|
|
||||||
<section id="contact">
|
|
||||||
<div class="container"><div class="row wow fadeInUp">
|
|
||||||
<div class="col-lg-12 col-md-12"><div class="form">
|
|
||||||
<div id="sendmessage">Your message has been sent. Thank you!</div>
|
|
||||||
<div id="errormessage">Error</div>
|
|
||||||
<form role="form" name="contactForm" class="contactForm" id="contactForm" action="" method="post">
|
|
||||||
<div class="form-row">
|
|
||||||
<div class="form-group col-lg-12">
|
|
||||||
<input type="text" name="name" class="form-control" id="name" placeholder="Your Name" data-rule="minlen:4" data-msg="Please enter at least 4 characters" />
|
|
||||||
<div class="validation"></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div class="form-row">
|
|
||||||
<div class="form-group col-lg-12">
|
|
||||||
<input type="email" class="form-control" name="email" id="email" placeholder="Your Email" data-rule="email" data-msg="Please enter a valid email address" />
|
|
||||||
<div class="validation"></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div class="form-row">
|
|
||||||
<div class="form-group col-lg-12">
|
|
||||||
<input type="text" class="form-control" name="subject" id="subject" placeholder="Subject" data-rule="minlen:4" data-msg="Please enter at least 8 characters" />
|
|
||||||
<div class="validation"></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div class="form-row">
|
|
||||||
<div class="form-group col-lg-12">
|
|
||||||
<textarea class="form-control" name="message" rows="5" data-rule="required" data-msg="Please write something for us" placeholder="Message"></textarea>
|
|
||||||
<div class="validation"></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div class="form-row">
|
|
||||||
<div class="form-group col-lg-12">
|
|
||||||
By submitting this form, I acknowledge that my information is subject
|
|
||||||
to the <a href="{{ "/policy/privacy" | relURL }}">OpenPOWER Foundation's Privacy Policy</a>.
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div class="form-row">
|
|
||||||
<div class="form-group col-lg-12 text-center">
|
|
||||||
<input type="hidden" name="contact" value="{{ $feedbackqueue }}" />
|
|
||||||
<input type="hidden" name="url" value="{{ $.Page.RelPermalink | absURL }}" />
|
|
||||||
<button class="btn btn-outline-primary mb-4 mt-4" type="submit" title="Send Message">Send Message</button>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</form>
|
|
||||||
</div></div>
|
|
||||||
</div></div>
|
|
||||||
</section>
|
|