Skip to content

Commit

Permalink
Add dpu-tools
Browse files Browse the repository at this point in the history
  • Loading branch information
bn222 committed Aug 20, 2024
1 parent d6ace38 commit b865769
Show file tree
Hide file tree
Showing 3 changed files with 234 additions and 0 deletions.
9 changes: 9 additions & 0 deletions dpu-tools/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM quay.io/centos/centos:stream9
RUN dnf install -y \
minicom python39 pciutils lshw && \
dnf clean all && \
rm -rf /var/cache/* && \
ln -s /usr/bin/pip3.9 /usr/bin/pip && \
ln -s /usr/bin/python3.9 /usr/bin/python
COPY * /
ENTRYPOINT ["python3", "/dpu-tools"]
113 changes: 113 additions & 0 deletions dpu-tools/RHEL-on-ipu.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
Installing RHEL on the Intel IPU
================================

This guide outlines the process of installing Red Hat Enterprise Linux (RHEL) on an Intel Infrastructure Processing Unit (IPU). Please note that these instructions involve various steps, primarily carried out on an ARM server and an x86 system with access to the Intel Management Controller (IMC).

Prerequisites
-------------

* x86 system with access to the IMC

Step-by-Step Installation Process
---------------------------------


In case anything has been left behind from previous attempts, run the following commands to clean up mount points:

```bash
mkdir -p /mnt/p4
umount /mnt/p4
kpartx -dv /dev/loop3
losetup -d /dev/loop3
```

After shutting down the virtual machine, proceed to mount the raw image to access its contents. Note that you need to replace the parition number p3 below with the root partition from the Virtual Machine:

```bash
losetup /dev/loop3 /root/disk-3.raw
kpartx -a /dev/loop3
mount /dev/mapper/loop3p4 /mnt/p4
```

Create an archive of the root filesystem:

```bash
tar cf rootfs.tar /mnt
```

### Copying and Preparing Root Filesystem on x86 System

Copy the resulting `rootfs.tar` to an x86 system that has access to the IMC:

```bash
scp rootfs.tar <x86-system>`
```

From the x86 system, transfer the `rootfs.tar` to the IMC:

```bash
scp rootfs.tar [email protected]:/work
```

### Configuring the IMC

```bash
ssh [email protected] "date -s '$(date)'"
```

Connect to the IMC via SSH:

```bash
ssh [email protected]
date -s ...
```

Prepare the `p4` partition, assuming it's already formatted as ext4:
```bash
mkdir -p /mnt/p4
mount /dev/nvme0n1p4 /mnt/p4
rm -rf /mnt/p4
tar xf /work/rootfs.tar -C /mnt/p4
umount /mnt/p4
```
Modify `/etc/fstab` to ensure it points to the correct device:
```bash
echo "/dev/sda / ext4 defaults 0 0" > /mnt/p4/etc/fstab
```
### Modifying Kernel Arguments
Add `selinux=0` to the kernel arguments in `/mnt/imc/acc_variable/acc-config.json`:
```bash
{ "acc_watchdog_timer": 60, "kernel": { "boot_params" : "ip=192.168.0.2::192.168.0.1:255.255.255.0::enp0s1f0:off root=/dev/sda rw netroot=iscsi:192.168.0.1::::iqn.e2000:acc initrdmem=0x440c000000,0x8000000 acpi=force selinux=0" } }
```
### Preparing the kernel and initrd
```bash
git clone https://github.com/coccinelle/coccinelle.git
cd coccinelle/
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)"
opam init
opam install Ocaml
eval $(opam env --switch=default)
./autogen
./configure
source env.sh
make clean
make
make install
```
### Finalizing
Reboot the IMC to cause the ACC to reboot too:
```bash
reboot
```
This process should result in the successful installation RHEL on the Intel IPU.
112 changes: 112 additions & 0 deletions dpu-tools/dpu-tools
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
#!/usr/bin/env python3

import os
import argparse
import tempfile
import shutil
import shlex
from collections import namedtuple
import subprocess
import re


def run(cmd: str, env: dict = os.environ.copy()):
Result = namedtuple("Result", "out err returncode")
args = shlex.split(cmd)
pipe = subprocess.PIPE
with subprocess.Popen(args, stdout=pipe, stderr=pipe, env=env) as proc:
out = proc.stdout.read().decode("utf-8")
err = proc.stderr.read().decode("utf-8")
proc.communicate()
ret = proc.returncode
return Result(out, err, ret)


def reset(args):
run("ssh [email protected] sudo reboot")


def console(args):
if args.target == "imc":
minicom_cmd = "minicom -b 460800 -D /dev/ttyUSB2"
else:
minicom_cmd = "minicom -b 115200 -D /dev/ttyUSB0"

minirc_path = "/root/.minirc.dfl"
if os.path.exists(minirc_path):
backed_up = True
with tempfile.NamedTemporaryFile(delete=False) as temp_file:
temp_file_path = temp_file.name
shutil.move(minirc_path, temp_file_path)
else:
backed_up = False
temp_file_path = ""

with open(minirc_path, 'w') as new_file:
new_file.write("pu rtscts No\n")
os.system(minicom_cmd)

if backed_up:
shutil.move(temp_file_path, minirc_path)


def find_bus_pci_address(address: str) -> str:
pattern = r"(\d+):(\d+)\.(\d+)"

match = re.match(pattern, address)

if match:
bus = match.group(1)
device = match.group(2)
function = match.group(3)

new_address = f"{bus}:00.0"
return new_address
else:
return "Invalid PCI address format"


def list_dpus(args):
del args
devs = {}
for e in run("lspci").out.split("\n"):
if "Intel Corporation Device 1453" in e:
addr = find_bus_pci_address(e.split()[0])
for line in run("lshw -c network -businfo").out.split("\n"):
if addr in line:
dev = line.split()[1]
devs[dev] = (addr, "IPU")

print("ID netdev PCI-Address Kind")
print("----- -------- ------------ ------")
for i, (k, (d, kind)) in enumerate(devs.items()):
print(f"{i: 5d} {k.ljust(8)} {d.ljust(12)} {kind}")



def main():
parser = argparse.ArgumentParser(description='Tools to interact with an IPU')
subparsers = parser.add_subparsers(title='subcommands', description='Valid subcommands', dest='subcommand')

reset_parser = subparsers.add_parser('reset', help='Reset the IPU')
reset_parser.set_defaults(func=reset)

list_parser = subparsers.add_parser('list', help='list devices')
list_parser.set_defaults(func=list_dpus)

console_parser = subparsers.add_parser('console', help='Open console for the IPU')
console_parser.set_defaults(func=console)
console_parser.add_argument('target', choices=['imc', 'acc'], help="Specify imc or acc as the target")


args = parser.parse_args()
if hasattr(args, 'func'):
args.func(args)
else:
parser.print_help()


if __name__ == "__main__":
main()


0 comments on commit b865769

Please sign in to comment.