View source for CSIT/fdio csit lab ext lld draft

== FD.io CSIT Lab Extension - LLD Draft ==

This is a (proposed) low-level design for the extensions of FD.io CSIT lab to accommodate the new Intel Xeon Skylake, Arm AArch64 and Atom devices. The current version is available on [https://wiki.fd.io/view/CSIT/fdio_csit_lab_ext_lld_draft CSIT wiki].

== Testbeds Overview ==

=== Testbeds Type Breakdown ===

<pre>
 #.  CSIT_tb.           Purpose.    SUT.  TG.   #tb.  #SUTs.  #TGs. #skx_node.
 1.  1-node Xeon.       func.       skx.  n/a.  2.    2.      0.    2.
 2.  2-node Xeon.       perf.       skx.  skx.  4.    4.      4.    8.
 3.  3-node Xeon.       perf.       skx.  skx.  2.    4.      2.    6.
 4.  tcp-l47.           tcp-stack.  skx.  ps1.  2.    1.      1.    1.
 5.  atom-netgate.      perf+func.  net.  skx.  1.    3.      1.    1.
 6.  aarch64.           perf+func.  arm.  skx.  >1.   >1.     2.    2.
                                                 Total skx_node:   20.
</pre>

=== 1-Node Xeon Testbeds ===

One 1-node Xeon testbed for VPP_Device tests is built using one SUT (Type-6 server), with NIC ports connected back-to-back.

=== 2-Node Xeon Testbeds ===

Four 2-node Xeon testbeds (are expected to be built|are built), with each testbed using one SUTs (Type-1 server) and one TG (Type-2 server) connected back-to-back. NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in next sections.

=== 3-Node Xeon Testbeds ===

Two 3-node Xeon testbeds (are expected to be built|are built), with each testbed using two SUTs (Type-1 server) and one TG (Type-2 server) connected in full-mesh triangle. NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in next sections.

=== Arm Testbeds ===

One 3-node Huawei testbeds (are expected to be built|are built), with each testbed using two SUTs (Type-3 server) and one TG (Type-2 server) connected in full-mesh triangle.

One 3-node Marvell testbeds (are expected to be built|are built), with each testbed using two SUTs (Type-4 server) and one TG (Type-2 server) connected in full-mesh triangle.

=== TCP/IP and L47 Testbeds ===

<..>

=== Atom Testbeds ===

One 3-node Atom (Netgate based) testbed is built consisting of three SUTs (Type-5 Netgate device.) NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in the next section.

== Inventory ==

=== Servers ===

<pre>
1. Intel Xeon servers:
    - 20 * SuperMicro SYS-7049GP-TRT with Xeon Skylake processors.
        - Chassis: SuperMicro SYS-7049GP-TRT.
        - Motherboard: SuperMicro X11DPG-QT.
        - Processors: 2* Intel Platinum 8180 2.3 GHz.
        - RAM Memory: 16* 16GB DDR4-2666MHz.
        - Disks: 2* 1.6TB 6G SATA SSD.
2. Arm Cortex A-72 servers
    - 2 * Huawei TaiShan 2280.
        - Chassis: Huawei TaiShan 2280.
        - Processors: 1* hip07-d05 ~ 64* Arm Cortex-A72.
        - RAM Memory: 8* 16GB DDR4-2400MT/s.
        - Disks: 1* 4TB SATA HDD.
    - 3 * MACCHIATObin
        - Chassis: MACCHIATObin.
        - Processors: 1* Armada 8040 ~ 4* Arm Cortex-A72.
        - RAM Memory: 1* 16GB DDR4.
        - Disks: 1* 128GB(?) SATA SDD.
3. Intel Atom servers with Rangely processors.
    - 3 * Netgate XG-2758-1u
        - Chassis: Netgate XG-2758-1u
        - Processors: 1* Rangely (Atom) C2758 2.4 GHz
        - RAM Memory: 16GB ECC
        - Disks: 150 GB
</pre>

=== Network Interface Cards ===

<pre>
1. 10GE NICs
    - 30 * X710-DA4.
    - 6 * X710-DA2.
    - 6 * X520-DA2.
2. 25GE NICs
    - 20 * XXV710-DA2.
3. 40GE NICs
    - 2 * XL710-QDA2.
4. 100GE NICs
    - none.
</pre>

=== Pluggables and Cables ===

Pluggables:

<pre>
1. 10GE SFP+
    - 80 * Intel Ethernet SFP+ SR.
2. 25GE SFP28
    - None?
3. 40GE QSFP+
    - 20 * Intel® Ethernet QSFP+ Twinaxial Cable, 5 meters.
4. 100GE
    - None?
</pre>

Standalone cables:

<pre>
1. 10GE
    - None?
2. 25GE
    - None?
3. 40GE
    - None?
4. 100GE
    - None?
</pre>

=== Other Network Cards ===

Any QATs?

== Installation Status ==

Lab installation status is tracked by LF IT team in [https://docs.google.com/document/d/16TdvGC73wuNQjkP355MTXRckv7yqnaxJkWwX7G7izEo/edit?ts=5b10411b#heading=h.dprb64shku8u FD.io Server Status].

== Server/Device Management and Naming ==

=== Server Management Requirements ===

Total of 20 SM SYS-7049GP-TRT servers are made available for FD.IO CSIT testbed.
For management purposes, each server must have following two ports connected to the management network:

* 1GE IPMI port
** IPMI - Intelligent Platform Management Interface.
** Required for access to embedded server management with WebUI, CLI, SNMPv3, IPMIv2.0, for firmware (BIOS) and OS updates.
* 1GE/10GE management port
** hostOS management port for general system management.

=== Server and Port Naming Convention ===

Following naming convention is used within this page to specify physical connectivity and wiring across defined CSIT testbeds:

<pre>
- testbedname: testbedN.
- hostname:
    - traffic-generator: tN-tgW.
    - system-under-testX: tN-sutX.
- portnames:
    - tN-tgW-cY/pZ.
    - tN-sutX-cY/pZ.
- where:
    - N - testbed number.
    - tgW - server acts as traffic-generator with W index.
    - sutX - server acts as system-under-test with X index.
    - Y - PCIe slot number denoting a NIC card number within the host.
        - Y=2,4,9 - slots connected to NUMA node 0.
        - Y=6,8,10 - slots connected to NUMA node 1.
    - Z - port number on the NIC card.
</pre>

=== Server Management - Addressing ===

Each server has a LOM (Lights-Out-Management e.g. SM IPMI) and a Management port, which are connected to two different VLANs.

<pre>
1. LOM (IPMI) VLAN:
    - Subnet: 10.30.50.0/24
    - Gateway: 10.30.50.1
    - Broadcast: 10.30.50.255
    - DNS1: 199.204.44.24
    - DNS2: 199.204.47.54
2. Management Vlan:
    - Subnet: 10.30.51.0/24
    - Gateway: 10.30.51.1
    - Broadcast: 10.30.51.255
    - DNS1: 199.204.44.24
    - DNS2: 199.204.47.54
</pre>

To access these hosts, an VPN connection is required.

=== LOM (IPMI) VLAN IP Addresses ===

<..>

=== Management VLAN IP Addresses ===

<..>

== Testbeds Specification - Target Build ==

=== Server/Ports Naming, NIC Placement ===

==== 1-Node Xeon ====

Each server in 1-node Xeon topology has its NIC cards placed, and NIC cards and ports indexed per following specification:

<pre>
- Server1 [Type-6]:
    - testbedname: testbed11.
    - hostname: s1-t11-sut1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s1-t11-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s1-t11-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s1-t11-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s1-t11-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s1-t11-sut1-c4/p1 - 10GE-port1 x710-4p10GE.
        - s1-t11-sut1-c4/p2 - 10GE-port2 x710-4p10GE.
        - s1-t11-sut1-c4/p3 - 10GE-port3 x710-4p10GE.
        - s1-t11-sut1-c4/p4 - 10GE-port4 x710-4p10GE.
- Server2 [Type-6]:
    - testbedname: testbed12.
    - hostname: s2-t12-sut1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s2-t12-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s2-t12-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s2-t12-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s2-t12-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s2-t12-sut1-c4/p1 - 10GE-port1 x710-4p10GE.
        - s2-t12-sut1-c4/p2 - 10GE-port2 x710-4p10GE.
        - s2-t12-sut1-c4/p3 - 10GE-port3 x710-4p10GE.
        - s2-t12-sut1-c4/p4 - 10GE-port4 x710-4p10GE.
</pre>

==== 2-Node Xeon ====

Each server in 2-node Xeon topology has its NIC cards placed, and NIC cards and ports indexed per following specification:

<pre>
- Server3 [Type-1]:
    - testbedname: testbed21.
    - hostname: s3-t21-sut1.
    - IPMI IP: 10.30.50.41
    - Host IP: 10.30.51.44
    - portnames:
        - s3-t21-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s3-t21-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s3-t21-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s3-t21-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s3-t21-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s3-t21-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s3-t21-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s3-t21-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server4 [Type-2]:
    - testbedname: testbed21.
    - hostname: s4-t21-tg1.
    - IPMI IP: 10.30.50.42
    - Host IP: 10.30.51.45
    - portnames:
        - s4-t21-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s4-t21-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s4-t21-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s4-t21-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s4-t21-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s4-t21-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s4-t21-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s4-t21-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server5 [Type-1]:
    - testbedname: testbed22.
    - hostname: s5-t22-sut1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s5-t22-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s5-t22-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s5-t22-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s5-t22-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s5-t22-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s5-t22-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s5-t22-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s5-t22-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server6 [Type-2]:
    - testbedname: testbed22.
    - hostname: s6-t22-tg1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s6-t22-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s6-t22-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s6-t22-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s6-t22-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s6-t22-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s6-t22-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s6-t22-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s6-t22-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server7 [Type-1]:
    - testbedname: testbed23.
    - hostname: s7-t23-sut1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s7-t23-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s7-t23-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s7-t23-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s7-t23-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s7-t23-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s7-t23-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s7-t23-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s7-t23-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server8 [Type-2]:
    - testbedname: testbed23.
    - hostname: s8-t23-tg1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s8-t23-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s8-t23-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s8-t23-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s8-t23-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s8-t23-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s8-t23-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s8-t23-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s8-t23-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server9 [Type-1]:
    - testbedname: testbed24.
    - hostname: s9-t24-sut1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s9-t24-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s9-t24-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s9-t24-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s9-t24-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s9-t24-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s9-t24-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s9-t24-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s9-t24-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server10 [Type-2]:
    - testbedname: testbed21.
    - hostname: s10-t24-tg1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s10-t24-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s10-t24-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s10-t24-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s10-t24-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s10-t24-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s10-t24-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s10-t24-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s10-t24-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
</pre>

==== 3-Node Xeon ====

Each server in 3-node Xeon topology has its NIC cards placed, and NIC cards and ports indexed per following specification:

<pre>
- Server11 [Type-1]:
    - testbedname: testbed31.
    - hostname: s11-t31-sut1.
    - IPMI IP: 10.30.50.43
    - Host IP: 10.30.51.46
    - portnames:
        - s11-t31-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s11-t31-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s11-t31-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s11-t31-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s11-t31-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s11-t31-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s11-t31-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s11-t31-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server12 [Type-1]:
    - testbedname: testbed31.
    - hostname: s12-t31-sut2.
    - IPMI IP: 10.30.50.44
    - Host IP: 10.30.51.47
    - portnames:
        - s12-t31-sut2-c2/p1 - 10GE-port1 x710-4p10GE.
        - s12-t31-sut2-c2/p2 - 10GE-port2 x710-4p10GE.
        - s12-t31-sut2-c2/p3 - 10GE-port3 x710-4p10GE.
        - s12-t31-sut2-c2/p4 - 10GE-port4 x710-4p10GE.
        - s12-t31-sut2-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s12-t31-sut2-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s12-t31-sut2-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s12-t31-sut2-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server13 [Type-2]:
    - testbedname: testbed31.
    - hostname: s13-t31-tg1.
    - IPMI IP: 10.30.50.45
    - Host IP: 10.30.51.48
    - portnames:
        - s13-t31-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s13-t31-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s13-t31-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s13-t31-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s13-t31-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s13-t31-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s13-t31-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s13-t31-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server14 [Type-1]:
    - testbedname: testbed32.
    - hostname: s14-t32-sut1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s14-t32-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s14-t32-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s14-t32-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s14-t32-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s14-t32-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s14-t32-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s14-t32-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s14-t32-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server15 [Type-1]:
    - testbedname: testbed32.
    - hostname: s15-t32-sut2.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s15-t32-sut2-c2/p1 - 10GE-port1 x710-4p10GE.
        - s15-t32-sut2-c2/p2 - 10GE-port2 x710-4p10GE.
        - s15-t32-sut2-c2/p3 - 10GE-port3 x710-4p10GE.
        - s15-t32-sut2-c2/p4 - 10GE-port4 x710-4p10GE.
        - s15-t32-sut2-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s15-t32-sut2-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s15-t32-sut2-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s15-t32-sut2-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
- Server16 [Type-2]:
    - testbedname: testbed32.
    - hostname: s16-t32-tg1.
    - IPMI IP: <TODO(lf)>
    - Host IP: <TODO(lf)>
    - portnames:
        - s16-t32-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s16-t32-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s16-t32-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s16-t32-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s16-t32-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s16-t32-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
        - s16-t32-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
        - s16-t32-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
</pre>

==== 3-Node Arm ====

Note: Server19 (TG) is shared between testbed33 & testbed34

<pre>
- Server17 [Type-3]:
    - testbedname: testbed33.
    - hostname: s17-t33-sut1.
    - IPMI IP: 10.30.50.36
    - Host IP: 10.30.51.36
    - portnames:
        - s17-t33-sut1-ca/p1 - 10GE-port1 82599-2p10GE.
        - s17-t33-sut1-ca/p2 - 10GE-port2 82599-2p10GE.
        - s17-t33-sut1-c2/p1 - 25GE-port1 cx4-2p10GE.
        - s17-t33-sut1-c2/p2 - 25GE-port2 cx4-2p10GE.
- Server18 [Type-3]:
    - testbedname: testbed33.
    - hostname: s18-t33-sut2.
    - IPMI IP: 10.30.50.37
    - Host IP: 10.30.51.37
    - portnames:
        - s18-t33-sut2-ca/p1 - 10GE-port1 82599-2p10GE.
        - s18-t33-sut2-ca/p2 - 10GE-port2 82599-2p10GE.
        - s18-t33-sut2-c2/p1 - 25GE-port1 cx4-2p10GE.
        - s18-t33-sut2-c2/p2 - 25GE-port2 cx4-2p10GE.
- Server19 [Type-2]:
    - testbedname: testbed33.
    - hostname: s19-t33-tg1.
    - IPMI IP: 10.30.50.46
    - Host IP: 10.30.51.49
    - portnames:
        - s19-t33-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s19-t33-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s19-t33-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s19-t33-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s19-t33-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s19-t33-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
- Server19 [Type-2]:
    - testbedname: testbed34.
    - hostname: s19-t34-tg1.
    - IPMI IP: 10.30.50.46
    - Host IP: 10.30.51.49
    - portnames:
        - s19-t33-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
        - s19-t33-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
        - s19-t33-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
        - s19-t33-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
        - s19-t33-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
        - s19-t33-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
- Server20 [Type-4]:
    - testbedname: testbed34.
    - hostname: s20-t34-sut1.
    - IPMI IP: N/A
    - Host IP: 10.30.51.41
    - portnames:
        - s20-t34-sut1-ca/p1 - 10GE-port1 Marvell.
        - s20-t34-sut1-ca/p2 - 10GE-port2 Marvell.
- Server21 [Type-4]:
    - testbedname: testbed34.
    - hostname: s21-t34-sut2.
    - IPMI IP: N/A
    - Host IP: 10.30.51.42
    - portnames:
        - s21-t34-sut2-ca/p1 - 10GE-port1 Marvell.
        - s21-t34-sut2-ca/p2 - 10GE-port2 Marvell.
</pre>

==== TCP/IP and L47 ====

<..>

==== 3-Node Atom ====

Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB.

<pre>
- Server22 [Type-5]:
    - testbedname: testbed35.
    - hostname: s22-t35-sut1 (vex-yul-rot-netgate-1).
    - IPMI IP: 10.30.51.29 - screen -r /dev/ttyUSB0
    - Host IP: 10.30.51.9
    - portnames:
        - s22-t35-sut1-p1 - 10GE-port1 ix0 82599.
        - s22-t35-sut1-p2 - 10GE-port2 ix1 82599.
    - 1GB ports (tbd)
- Server23 [Type-5]:
    - testbedname: testbed35.
    - hostname: s23-t35-sut2 (vex-yul-rot-netgate-2).
    - IPMI IP: 10.30.51.30 - screen -r /dev/ttyUSB1
    - Host IP: 10.30.51.10
    - portnames:
        - s23-t35-sut1-p1 - 10GE-port1 ix0 82599.
        - s23-t35-sut1-p2 - 10GE-port2 ix1 82599.
    - 1GB ports (tbd)
- Server24 [Type-5]:
    - testbedname: testbed35.
    - hostname: s24-t35-sut3 (vex-yul-rot-netgate-3).
    - IPMI IP: 10.30.51.30 - screen -r /dev/ttyUSB2
    - Host IP: 10.30.51.11
    - portnames:
        - s24-t35-sut1-p1 - 10GE-port1 ix0 82599.
        - s24-t35-sut1-p2 - 10GE-port2 ix1 82599.
    - 1GB ports (tbd)
</pre>

=== Physical Connectivity within Testbeds ===

==== 1-Node Xeon ====

Two 1-Node testbeds are constructed by connecting 2 Xeon servers using below specification:

<pre>
- testbed11:
    - ring1 10GE-ports x710-4p10GE:
        - s1-t11-sut1-c2/p1 to s1-t11-sut1-c4/p1.
    - ring2 10GE-ports x710-4p10GE:
        - s1-t11-sut1-c2/p2 to s1-t11-sut1-c4/p2.
    - ring3 10GE-ports x710-4p10GE:
        - s1-t11-sut1-c2/p3 to s1-t11-sut1-c4/p3.
    - ring4 10GE-ports x710-4p10GE:
        - s1-t11-sut1-c2/p3 to s1-t11-sut1-c4/p3.
- testbed12:
    - ring1 10GE-ports x710-4p10GE:
        - s2-t12-sut1-c2/p1 to s2-t12-sut1-c4/p1.
    - ring2 10GE-ports x710-4p10GE:
        - s2-t12-sut1-c2/p2 to s2-t12-sut1-c4/p2.
    - ring3 10GE-ports x710-4p10GE:
        - s2-t12-sut1-c2/p3 to s2-t12-sut1-c4/p3.
    - ring4 10GE-ports x710-4p10GE:
        - s2-t12-sut1-c2/p3 to s2-t12-sut1-c4/p3.
</pre>

==== 2-Node Xeon ====

Four 2-Node testbeds are constructed by connecting 8 Xeon servers using below specification:

<pre>
- testbed21:
    - ring1 10GE-ports x710-4p10GE on SUT:
        - t21-tg1-c2/p1 to t21-sut1-c2/p1.
        - t21-sut1-c2/p2 to t21-tg1-c2/p2.
    - ring2 10GE-ports x710-4p10GE on SUT:
        - t21-tg1-c2/p3 to t21-sut1-c2/p3.
        - t21-sut1-c2/p4 to t21-tg1-c2/p4.
    - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
        - t21-tg1-c4/p1 to t21-sut1-c4/p1.
        - t21-sut1-c4/p2 to t21-tg1-c4/p2.
    - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
        - t21-tg1-c9/p1 to t21-sut1-c9/p1.
        - t21-sut1-c9/p2 to t21-tg1-c9/p2.
    - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
        - t21-tg1-c10/p1 to t21-tg1-c10/p2.
        - t21-tg1-c10/p3 to t21-tg1-c10/p4.
- testbed22:
    - ring1 10GE-ports x710-4p10GE on SUT:
        - t22-tg1-c2/p1 to t22-sut1-c2/p1.
        - t22-sut1-c2/p2 to t22-tg1-c2/p2.
    - ring2 10GE-ports x710-4p10GE on SUT:
        - t22-tg1-c2/p3 to t22-sut1-c2/p3.
        - t22-sut1-c2/p4 to t22-tg1-c2/p4.
    - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
        - t22-tg1-c4/p1 to t22-sut1-c4/p1.
        - t22-sut1-c4/p2 to t22-tg1-c4/p2.
    - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
        - t22-tg1-c9/p1 to t22-sut1-c9/p1.
        - t22-sut1-c9/p2 to t22-tg1-c9/p2.
    - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
        - t22-tg1-c10/p1 to t22-tg1-c10/p2.
        - t22-tg1-c10/p3 to t22-tg1-c10/p4.
- testbed23:
    - ring1 10GE-ports x710-4p10GE on SUT:
        - t23-tg1-c2/p1 to t23-sut1-c2/p1.
        - t23-sut1-c2/p2 to t23-tg1-c2/p2.
    - ring2 10GE-ports x710-4p10GE on SUT:
        - t23-tg1-c2/p3 to t23-sut1-c2/p3.
        - t23-sut1-c2/p4 to t23-tg1-c2/p4.
    - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
        - t23-tg1-c4/p1 to t23-sut1-c4/p1.
        - t23-sut1-c4/p2 to t23-tg1-c4/p2.
    - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
        - t23-tg1-c9/p1 to t23-sut1-c9/p1.
        - t23-sut1-c9/p2 to t23-tg1-c9/p2.
    - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
        - t23-tg1-c10/p1 to t23-tg1-c10/p2.
        - t23-tg1-c10/p3 to t23-tg1-c10/p4.
- testbed24:
    - ring1 10GE-ports x710-4p10GE on SUT:
        - t24-tg1-c2/p1 to t24-sut1-c2/p1.
        - t24-sut1-c2/p2 to t24-tg1-c2/p2.
    - ring2 10GE-ports x710-4p10GE on SUT:
        - t24-tg1-c2/p3 to t24-sut1-c2/p3.
        - t24-sut1-c2/p4 to t24-tg1-c2/p4.
    - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
        - t24-tg1-c4/p1 to t24-sut1-c4/p1.
        - t24-sut1-c4/p2 to t24-tg1-c4/p2.
    - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
        - t24-tg1-c9/p1 to t24-sut1-c9/p1.
        - t24-sut1-c9/p2 to t24-tg1-c9/p2.
    - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
        - t24-tg1-c10/p1 to t24-tg1-c10/p2.
        - t24-tg1-c10/p3 to t24-tg1-c10/p4.
</pre>

==== 3-Node Xeon ====

Two 3-Node testbeds are constructed by connecting 6 Xeon servers using below specification:

<pre>
- testbed31:
    - ring1 10GE-ports x710-4p10GE on SUTs:
        - t31-tg1-c2/p1 to t31-sut1-c2/p1.
        - t31-sut1-c2/p2 to t31-sut2-c2/p2.
        - t31-sut2-c2/p1 to t31-tg1-c2/p2.
    - ring2 10GE-ports x710-4p10GE on SUT:
        - t31-tg1-c2/p3 to t31-sut1-c2/p3.
        - t31-sut1-c2/p4 to t31-sut2-c2/p4.
        - t31-sut2-c2/p3 to t31-tg1-c2/p4.
    - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
        - t31-tg1-c4/p1 to t31-sut1-c4/p1.
        - t31-sut1-c4/p2 to t31-sut2-c4/p2.
        - t31-sut2-c4/p1 to t31-tg1-c4/p2.
    - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
        - t31-tg1-c9/p1 to t31-sut1-c9/p1.
        - t31-sut1-c9/p2 to t31-sut2-c9/p2.
        - t31-sut2-c9/p1 to t31-tg1-c9/p2.
    - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
        - t31-tg1-c10/p1 to t31-tg1-c10/p2.
        - t31-tg1-c10/p3 to t31-tg1-c10/p4.
- testbed32:
    - ring1 10GE-ports x710-4p10GE on SUTs:
        - t32-tg1-c2/p1 to t32-sut1-c2/p1.
        - t32-sut1-c2/p2 to t32-sut2-c2/p2.
        - t32-sut2-c2/p1 to t32-tg1-c2/p2.
    - ring2 10GE-ports x710-4p10GE on SUT:
        - t32-tg1-c2/p3 to t32-sut1-c2/p3.
        - t32-sut1-c2/p4 to t32-sut2-c2/p4.
        - t32-sut2-c2/p3 to t32-tg1-c2/p4.
    - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
        - t32-tg1-c4/p1 to t32-sut1-c4/p1.
        - t32-sut1-c4/p2 to t32-sut2-c4/p2.
        - t32-sut2-c4/p1 to t32-tg1-c4/p2.
    - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
        - t32-tg1-c9/p1 to t32-sut1-c9/p1.
        - t32-sut1-c9/p2 to t32-sut2-c9/p2.
        - t32-sut2-c9/p1 to t32-tg1-c9/p2.
    - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
        - t32-tg1-c10/p1 to t32-tg1-c10/p2.
        - t32-tg1-c10/p3 to t32-tg1-c10/p4.
</pre>

==== 3-Node Arm ====

<pre>
- testbed33:
    - ring1 10GE-ports 82599-2p10GE on SUTs:
        - t33-tg1-c2/p1 - t33-sut1-ca/p1.
        - t33-sut1-ca/p2 - t33-sut2-ca/p1.
        - t33-sut2-ca/p2 - t33-tg1-c2/p2.
- testbed34:
    - ring1 10GE-ports Marvell on SUTs:
        - t33-tg1-c2/p3 - t34-sut1-ca/p1.
        - t34-sut1-ca/p2 - t34-sut2-ca/p1.
        - t34-sut2-ca/p2 - t33-tg1-c2/p4.
</pre>

==== TCP/IP and L47 ====

<..>

==== 3-Node Atom ====

<..>

=== Host OS and configuration ===

<..>

==== SUTs ====

<..>

==== TG ====

<..>

== Server Specification ==

The new FD.io CSIT-CPL lab (is expected to contain|contains) following hardware server configurations:

<pre>
1. Type-1: Purpose - (Intel Xeon Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
    - Quantity: TBD based on testbed allocation.
    - Physical connectivity:
        - IPMI and host management ports.
        - NIC ports connected into 2-node and 3-node topologies.
    - Main HW configuration:
        - Chassis: SuperMicro SYS-7049GP-TRT.
        - Motherboard: SuperMicro X11DPG-QT.
        - Processors: 2* Intel Platinum 8180 2.3 GHz.
        - RAM Memory: 16* 16GB DDR4-2666MHz.
        - Disks: 2* 1.6TB 6G SATA SSD.
    - NICs configuration:
        - Numa0: (x16, x16, x16 PCIe3.0 lanes)
            - PCIe Slot2 18:00.xx: x710-4p10GE Intel.
            - PCIe Slot4 3b:00.xx: xxv710-DA2-2p25GE Intel.
            - PCIe Slot9 5e:00.xx: FUTURE ConnectX5-2p100GE Mellanox.
        - Numa1: (x16, x16, x16 PCIe3.0 lanes)
            - PCIe Slot6 86:00.xx: empty.
            - PCIe Slot8 af:00.xx: empty.
            - PCIe Slot10 d8:00.xx: empty.
2. Type-2: Purpose - (Intel Xeon Processor) TG for T-Rex.
    - Quantity: TBD based on testbed allocation.
    - Physical connectivity:
        - IPMI and host management ports.
        - NIC ports connected into 2-node and 3-node topologies.
    - Main HW configuration:
        - Chassis: SuperMicro SYS-7049GP-TRT.
        - Motherboard: SuperMicro X11DPG-QT.
        - Processors: 2* Intel Platinum 8180 2.3 GHz.
        - RAM Memory: 16* 16GB DDR4-2666MHz.
        - Disks: 2* 1.6TB 6G SATA SSD.
    - NICs configuration:
        - Numa0: (x16, x16, x16 PCIe3.0 lanes)
            - PCIe Slot2 18:00.xx: x710-4p10GE Intel.
            - PCIe Slot4 3b:00.xx: xxv710-DA2 2p25GE Intel.
            - PCIe Slot9 5e:00.xx: FUTURE ConnectX5-2p100GE Mellanox.
        - Numa1: (x16, x16, x16 PCIe3.0 lanes)
            - PCIe Slot6 86:00.xx: empty.
            - PCIe Slot8 af:00.xx: empty.
            - PCIe Slot10 d8:00.xx: x710-4p10GE Intel.
3. Type-3: Purpose - (Arm hip07-d05 Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
    - Quantity: 2
    - Physical connectivity:
        - IPMI(?) and host management ports.
        - NIC ports connected into 3-node topology.
    - Main HW configuration:
        - Chassis: Huawei TaiShan 2280.
        - Processors: 1* hip07-d05 ~ 64* Arm Cortex-A72
        - RAM Memory: 8* 16GB DDR4-2400MT/s
        - Disks: 1* 4TB SATA HDD
    - NICs configuration:
        - PCIe Slot2 e9:00.xx: connectx4-2p25GE Mellanox.
        - PCIe Slota 11:00.xx: 82599-2p10GE Intel.
4. Type-4: Purpose - (Arm Armada 8040 Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
    - Quantity: 3
    - Physical connectivity:
        - Host management ports.
        - NIC ports connected into 2-node and 3-node topologies.
    - Main HW configuration:
        - Chassis: MACCHIATObin.
        - Processors: 1* Armada 8040 ~ 4* Arm Cortex-A72
        - RAM Memory: 1* 16GB DDR4
        - Disks: 1* 128GB(?) SATA SDD
    - NICs configuration:
        - pp2-2p10GE Marvell (on-chip Ethernet ports ; marvell plugin in VPP)
5. Type-5: Purpose - (Intel Atom Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
    - Quantity: TBD based on testbed allocation.
    - Physical connectivity:
        - Management: serial Port (usb) for console
        - NIC ports connected into 2-node.
    - Main HW configuration:
        - Chassis: Netgate XG-2758-1u
        - Processors: 1* Rangely (Atom) C2758 2.4 GHz
        - RAM Memory: 16GB ECC
        - Disks: 150 GB
    - NICs configuration:
        - 2x 10Gb Intel 82599ES
        - 4x 1GB Intel I354
6. Type-6: Purpose - (Intel Xeon Processor) SUT for VPP_Device functional tests.
    - Quantity: 1.
    - Physical connectivity:
        - IPMI and host management ports.
        - NIC ports connected into 2-node and 3-node topologies.
    - Main HW configuration:
        - Chassis: SuperMicro SYS-7049GP-TRT.
        - Motherboard: SuperMicro X11DPG-QT.
        - Processors: 2* Intel Platinum 8180 2.3 GHz.
        - RAM Memory: 16* 16GB DDR4-2666MHz.
        - Disks: 2* 1.6TB 6G SATA SSD.
    - NICs configuration:
        - Numa0: (x16, x16, x16 PCIe3.0 lanes)
            - PCIe Slot2 18:00.xx: x710-4p10GE Intel.
            - PCIe Slot4 3b:00.xx: x710-4p10GE Intel.
            - PCIe Slot9 5e:00.xx: empty.
        - Numa1: (x16, x16, x16 PCIe3.0 lanes)
            - PCIe Slot6 86:00.xx: empty.
            - PCIe Slot8 af:00.xx: empty.
            - PCIe Slot10 d8:00.xx: empty.
</pre>

== Initial Build (Deprecated) ==

=== Physical Connectivity within Testbeds ===

==== 2-Node Xeon ====

One 2-Node testbeds are constructed by connecting 8 Xeon servers using below specification:

 1. testbed21:
    - ring3 40GE-ports xl710-2p40GE on SUT
        - t21-tg1-c4/p1 to t21-sut1-c4/p1.
        - t21-sut1-c4/p2 to t21-tg1-c4/p2.

==== 3-Node Xeon ====

 1. testbed31:
    - ring1 10GE-ports x710-2p10GE on SUTs:
        - t31-tg1-c2/p1 to t31-sut1-c2/p1.
        - t31-sut1-c2/p2 to t31-sut2-c2/p2.
        - t31-sut2-c2/p1 to t31-tg1-c2/p2.
    - ring2 10GE-ports x710-2p10GE on SUT:
        - t31-tg1-c4/p1 to t31-sut1-c4/p1.
        - t31-sut1-c4/p2 to t31-sut2-c4/p2.
        - t31-sut2-c4/p1 to t31-tg1-c4/p2.

==== 2-Node Atom ====

 1. testbed35:
    - (tbd)

==== 3-Node Atom ====

 1. testbed35:
    - (tbd)