Network Switches

A distributed KV benchmark moves KV cache between machines through the switches they are cabled to. The machines' own counters show what each network card sent and received. The switch shows what happened in between: congestion marks, pause frames, drops, buffer peaks and the power it drew. This page shows you how to connect your switches so that every benchmark reads them.

You do three things once per switch, and Metrum AI Bench does the rest:

1. Find. Your machines find the switches they are cabled to. You do not type anything in.
2. Connect. You give each switch a read-only login, once. One login can connect many switches.
3. Test. A machine cabled to the switch logs in and checks which counters it can read.

After that, every distributed KV benchmark on those machines reads the switches it runs through, from start to end of every job.

Before you start

You need the following:

• The Metrum AI Bench agent on every machine, version 4.2.0 or later.
• Passwordless sudo for tcpdump on each machine, so the agent can listen for the switch's LLDP announcement. Most GPU machines already allow it for the agent's user.
• LLDP turned on at the switch. SONiC sends LLDP by default.
• A read-only login on each switch. On Dell Enterprise SONiC, create a user with the operator role. Metrum AI Bench only reads counters and never changes the switch configuration.
• A route from the machines to the switch's management address. The machine that reads the switch reaches it on SSH (port 22), RESTCONF (port 443) or gNMI (port 8080). Metrum AI Bench itself never connects to your switch.

Supported operating systems:

Operating system	How it is read
Dell Enterprise SONiC 4.x	gNMI or RESTCONF with a password login, or the switch's command line over SSH with a password or an SSH key. Verified on a Dell Z9864F-O64 running 4.4.0.
Community SONiC	sonic-gnmi, or the counters database over SSH.

A switch on any other operating system is listed as Not supported. It never blocks a benchmark or the publishing of its results.

Find your switches

Open Infrastructure, and then click the Network tab.

Each machine listens on every network card for one LLDP window (about 35 seconds) when its agent starts, and again every six hours. Every switch a machine hears appears in the list, with the switch port each network card is cabled to. Switches found this way arrive in the Needs login state.

• If a switch is missing, click Scan again. Every online machine listens once more. The list under the table shows what each machine heard, and why when it heard nothing.
• If the switch does not announce itself, click Add a switch by address and enter its management address. The first test fills in the rest.
• If a switch is not one you want read (a management switch, for example), click Ignore. It stays hidden until you show ignored switches.

A switch cabled to another switch, such as a leaf to a spine, reports the far switch during its test. That switch then appears in the list too.

Connect a switch

1. Select one or more switches in Needs login.
2. Click Connect.
3. Choose where the login is kept:

   • Keep it encrypted in Metrum AI Bench. Enter the username, and a password or an SSH private key. The Metrum AI Bench web server encrypts it before it is saved, and decrypts it only for the one machine that reads the switch, for one benchmark or one test at a time. Every release is recorded and shown to team admins under Login releases.

   • Keep it only on our machines. The login never leaves your machines. Run this command on every machine that can lead a benchmark, and enter the password when it asks:

     insights-cli switch login <switch> --username <user>

     To use an SSH key instead of a password, add --key <path-to-key>. The file is saved with mode 600 under ~/.metrum/switches/. The dialog lists the exact command for each switch and the machines that still lack the file.

4. Click Connect. Each switch moves to Testing, and the dialog names the machine that tests it.

A password works with every reader. An SSH key works with the command line only, which is read every 60 seconds instead of every 10.

Only a team owner or team admin can connect, test, disconnect or ignore a switch. Every team member can see the switches and their counters.

Read the test result

A test takes about a minute. The machine logs in once, identifies the switch, and tries each way of reading it in this order: gNMI, SONiC gNMI, RESTCONF, the SONiC counters database over SSH, and the command line. Each counter goes to the first reader that read it.

State	What it means	What to do
Ready	Every benchmark on the machines cabled to it reads it.	Nothing.
Login failed	The switch refused the login, or a machine lacks its login file.	Check the username and password and connect again, or run the command the message names.
Unreachable	No machine could reach the switch on its management address.	Check the address, and that SSH, RESTCONF or gNMI is enabled. Change the address or ports under Settings.
Certificate changed	The switch presented a different TLS certificate from the one pinned at its first test.	Confirm the change on the switch, for example a certificate renewal, and then click Accept new certificate.
Not supported	Metrum AI Bench cannot read this switch yet.	Nothing. It never blocks a benchmark or its publishing.

Open a switch to see every counter, whether it can be read, through which reader, and why a counter cannot. A counter marked Unconfirmed field name is read through a name that has not yet been confirmed on a live switch of that model. Its figures are shown, but a result that uses them cannot be published.

The certificate section shows the pinned certificate's fingerprint and expiry date. Dell Enterprise SONiC ships a certificate valid for one year. Renew it before it expires, and then accept the new one.

What a benchmark reads

When a distributed KV pool starts, its head machine reads every Ready switch that a pool machine's RDMA network card is cabled to, until the pool ends. When the pool's machines sit on more than one switch, it also reads the uplink ports between those switches, because KV cache crosses them. Switches you ignored are never read. The report then shows, per job:

• Switch counters: bytes and packets per port, ECN-marked packets, PFC pause frames per priority, drops and discards per queue, buffer peaks, FEC errors, optical power and switch power.
• Switch reads: which reader read each switch, how often, how many readings failed, and the switch clock's offset from the head machine.
• Timeline: any counter over the job, with the moments the switch cleared a counter marked.
• Link agreement: for every network card, what it sent against what its switch port received, and the other way round.
• Network energy: the energy each switch drew and the pool's share of it, with a separate tokens per joule with network figure.

The Excel export and the PDF report carry the same figures.

A result cannot be published until every switch its pool's RDMA network cards are cabled to was read, apart from switches you ignored and switches Metrum AI Bench cannot read. A switch in Needs login therefore holds results back until you connect it or ignore it. Ignore a switch only when it carries no benchmark traffic. Before you run, the distributed KV setup and the scenario pre-checks show which switches will be read, and what to fix for any that will not.

Disconnect a switch

Open the switch, and then click Disconnect. Metrum AI Bench deletes the stored login and the last test's findings at once, and the switch returns to Needs login. The switch, its cabling and the counters that past benchmarks recorded from it stay.

A login kept on your machines stays there until you run insights-cli switch logout <switch> on each machine.