Difference between revisions of "CSIT/fdio csit lab ext lld draft"

From fd.io
Jump to: navigation, search
(LOM (IPMI) and Management networks)
 
(48 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== FD.io CSIT Lab Extension - LLD Draft ==
+
This page has been deprecated. Content has been moved to [[CSIT/Testbeds:_Xeon_Skx,_Arm,_Atom. | FD.io CSIT testbeds - Xeon Skylake, Arm, Atom]].
 
+
This is a (proposed) low-level design for the extensions of FD.io CSIT lab to accommodate the new Intel Xeon Skylake, Arm AArch64 and Atom devices. The current version is available on [https://wiki.fd.io/view/CSIT/fdio_csit_lab_ext_lld_draft CSIT wiki].
+
 
+
== Server Specs ==
+
 
+
The new FD.io CSIT-CPL lab (is expected to contain|contains) following hardware server configurations:
+
 
+
1. Type-1: Purpose - SUT for SW Data Plane Workload i.e. VPP, testpmd.
+
    - Quantity: TBD based on testbed allocation.
+
    - Physical connectivity:
+
        - IPMI and host management ports.
+
        - NIC ports connected into 2-node and 3-node topologies.
+
    - Main HW configuration:
+
        - Chassis: SuperMicro SYS-7049GP-TRT.
+
        - Motherboard: SuperMicro X11DPG-QT.
+
        - Processors: 2* Intel Platinum 8180 2.3 GHz.
+
        - RAM Memory: 16* 16GB DDR4-2666MHz.
+
        - Disks: 2* 1.6TB 6G SATA SSD.
+
    - NICs configuration:
+
        - Numa0: (x16, x16, x16 PCIe3.0 lanes)
+
            - PCIe Slot2 18:00.xx: TBD e.g. x710-4p10GE Intel.
+
            - PCIe Slot4 3b:00.xx: TBD e.g. (xxv710-DA2|xl710)-(2p25GE|2p40GE) Intel.
+
            - PCIe Slot9 5e:00.xx: TBD e.g. ConnectX5-2p100GE Mellanox.
+
        - Numa1: (x16, x16, x16 PCIe3.0 lanes)
+
            - PCIe Slot6 86:00.xx: empty.
+
            - PCIe Slot8 af:00.xx: empty.
+
            - PCIe Slot10 d8:00.xx: empty.
+
2. Type-2: Purpose - TG for T-Rex.
+
    - Quantity: TBD based on testbed allocation.
+
    - Physical connectivity:
+
        - IPMI and host management ports.
+
        - NIC ports connected into 2-node and 3-node topologies.
+
    - Main HW configuration:
+
        - Chassis: SuperMicro SYS-7049GP-TRT.
+
        - Motherboard: SuperMicro X11DPG-QT.
+
        - Processors: 2* Intel Platinum 8180 2.3 GHz.
+
        - RAM Memory: 16* 16GB DDR4-2666MHz.
+
        - Disks: 2* 1.6TB 6G SATA SSD.
+
    - NICs configuration:
+
        - Numa0: (x16, x16, x16 PCIe3.0 lanes)
+
            - PCIe Slot2 18:00.xx: TBD e.g. x710-4p10GE Intel.
+
            - PCIe Slot4 3b:00.xx: TBD e.g. xl710 2p40GE or xxv710-DA2 2p25GE Intel.
+
            - PCIe Slot9 5e:00.xx: TBD e.g. ConnectX5-2p100GE Mellanox.
+
        - Numa1: (x16, x16, x16 PCIe3.0 lanes)
+
            - PCIe Slot6 86:00.xx: empty.
+
            - PCIe Slot8 af:00.xx: empty.
+
            - PCIe Slot10 d8:00.xx: TBD e.g. x710-4p10GE Intel.
+
3. Type-3: Purpose - SUT for SW Data Plane Workload i.e. VPP, testpmd.
+
    - Quantity: 2
+
    - Physical connectivity:
+
        - IPMI(?) and host management ports.
+
        - NIC ports connected into 3-node topology.
+
    - Main HW configuration:
+
        - Chassis: Huawei TaiShan 2280.
+
        - Processors: 1* hip07-d05 ~ 64* Arm Cortex-A72
+
        - RAM Memory: 8* 16GB DDR4-2400MT/s
+
        - Disks: 1* 4TB SATA HDD
+
    - NICs configuration:
+
        - PCIe Slot2 11:00.xx: 82599-2p10GE Intel.
+
        - PCIe Slota e9:00.xx: connectx4-2p25GE Mellanox.
+
4. Type-4: Purpose - SUT for SW Data Plane Workload i.e. VPP, testpmd.
+
    - Quantity: 3
+
    - Physical connectivity:
+
        - Host management ports.
+
        - NIC ports connected into 2-node and 3-node topologies.
+
    - Main HW configuration:
+
        - Chassis: MACCHIATObin.
+
        - Processors: 1* Armada 8040 ~ 4* Arm Cortex-A72
+
        - RAM Memory: 1* 16GB DDR4
+
        - Disks: 1* 128GB(?) SATA SDD
+
    - NICs configuration:
+
        - pp2-2p10GE Marvell (on-chip Ethernet ports ; marvell plugin in VPP)
+
5. Type-5: Purpose - SUT for SW Data Plane Workload i.e. VPP, testpmd.
+
    - Quantity: TBD based on testbed allocation.
+
    - Physical connectivity:
+
        - Management: serial Port (usb) for console
+
        - NIC ports connected into 2-node.
+
    - Main HW configuration:
+
        - Chassis: Netgate XG-2758-1u
+
        - Processors: 1* Rangely (Atom) C2758 2.4 GHz
+
        - RAM Memory: 16GB ECC
+
        - Disks: 150 GB
+
    - NICs configuration:
+
        - 2x 10Gb Intel 82599ES
+
        - 4x 1GB Intel I354
+
 
+
  Current NIC equipment in storage includes:
+
  - 2x XL710-QDA2
+
  - 6x X710-DA2
+
  - 6x X520-DA2
+
 
+
== Server Management Requirements ==
+
 
+
Total of 20 SM SYS-7049GP-TRT servers are made available for FD.IO CSIT testbed.
+
For management purposes, each server must have following two ports connected to the management network:
+
 
+
1. 1GE IPMI port
+
    - IPMI - Intelligent Platform Management Interface.
+
    - Required for access to embedded server management with WebUI, CLI, SNMPv3, IPMIv2.0, for firmware (BIOS) and OS updates.
+
2. 1GE/10GE management port
+
    - hostOS management port for general system management.
+
 
+
== Server and Port Naming Convention ==
+
 
+
Following naming convention is used within this page to specify physical connectivity and wiring across defined CSIT testbeds:
+
 
+
- testbedname: testbedN.
+
- hostname:
+
    - traffic-generator: tN-tgW.
+
    - system-under-testX: tN-sutX.
+
- portnames:
+
    - tN-tgW-cY/pZ.
+
    - tN-sutX-cY/pZ.
+
- where:
+
    - N - testbed number.
+
    - tgW - server acts as traffic-generator with W index.
+
    - sutX - server acts as system-under-test with X index.
+
    - Y - PCIe slot number denoting a NIC card number within the host.
+
        - Y=2,4,9 - slots connected to NUMA node 0.
+
        - Y=6,8,10 - slots connected to NUMA node 1.
+
    - Z - port number on the NIC card.
+
 
+
== Testbeds Overview ==
+
 
+
=== Testbeds Type Breakdown ===
+
 
+
#.  CSIT_tb.      Purpose.    SUT.  TG.  #tb.  #SUTs.  #TGs. #skx_node.
+
1.  2-node Xeon.  perf+func.  skx.  skx.  4.    4.      4.    8.
+
2.  3-node Xeon.  perf.      skx.  skx.  2.    4.      2.    6.
+
3.  tcp-l47.      tcp-stack.  skx.  ps1.  2.    2.      1.    2.
+
4.  atom-netgate. perf+func.  net.  skx.  1.    3.      1.    1.
+
5.  aarch64.      perf+func.  arm.  skx.  >1.  >1.    2.    2.
+
                                            Total skx_node:  19.
+
 
+
=== 2-Node Xeon Testbeds ===
+
 
+
Four 2-node Xeon testbeds (are expected to be built|are built), with each testbed using one SUTs (Type-1 server) and one TG (Type-2 server) connected back-to-back. NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in next sections.
+
 
+
=== 3-Node Xeon Testbeds ===
+
 
+
Two 3-node Xeon testbeds (are expected to be built|are built), with each testbed using two SUTs (Type-1 server) and one TG (Type-2 server) connected in full-mesh triangle. NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in next sections.
+
 
+
=== Arm Testbeds ===
+
 
+
One 3-node Huawei testbeds (are expected to be built|are built), with each testbed using two SUTs (Type-3 server) and one TG (Type-2 server) connected in full-mesh triangle.
+
 
+
One 3-node Marvell testbeds (are expected to be built|are built), with each testbed using two SUTs (Type-4 server) and one TG (Type-2 server) connected in full-mesh triangle.
+
 
+
=== TCP/IP and L47 Testbeds ===
+
 
+
<..>
+
 
+
=== AToM Testbeds ===
+
 
+
1 3-node AToM (Netgate based) testbed is built consisting of three SUTs (Type-5 Netgate device.) NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in the next section.
+
 
+
== Server Management Addressing ==
+
 
+
=== LOM (IPMI) and Management networks ===
+
 
+
Each server has a LOM (Lights-Out-Management e.g. SM IPMI) and a Management port, which are connected to two different VLANs.
+
 
+
1. LOM (IPMI) VLAN:
+
    - Subnet: 10.30.50.0/24
+
    - Gateway: 10.30.50.1
+
    - Broadcast: 10.30.50.255
+
    - DNS1: 199.204.44.24
+
    - DNS2: 199.204.47.54
+
2. Management Vlan:
+
    - Subnet: 10.30.51.0/24
+
    - Gateway: 10.30.51.1
+
    - Broadcast: 10.30.51.255
+
    - DNS1: 199.204.44.24
+
    - DNS2: 199.204.47.54
+
 
+
To access these hosts, an VPN connection is required.
+
 
+
=== LOM (IPMI) VLAN IP Addresses allocation ===
+
 
+
<..>
+
 
+
=== Management VLAN IP Addresses allocation ===
+
 
+
<..>
+
 
+
== Initial Build ==
+
 
+
Based on server hardware available on site as of 26-Apr-2018 (6 of Type-1/-2 servers), following are the initial testbeds to be built initially:
+
 
+
  1. 1 of 2-node Xeon ring3 40GE => totals: 2 * xeon servers, 2 * XL710-QDA2
+
  2. 1 of 3-node Xeon ring1+ring2 10GE => totals: 3 * xeon servers, 6 * X710-DA2
+
  3. 2 of 3-node Arm ring1 10GE => totals: 1 * xeon servers, 2 * X520-DA2
+
  4. 1 of 3-node Atom ring1 10GE => totals: 3 * netgate Atom servers 2 * 10GE 82599
+
 
+
=== Server/Ports Naming, NIC Placement ===
+
 
+
==== 2-Node Xeon ====
+
 
+
1. Server1 of Type-1:
+
    - testbedname: testbed21.
+
    - hostname: s1-t21-sut1.
+
    - IPMI IP: 10.30.50.41
+
    - Host IP: <..>
+
    - portnames:
+
        - s1-t21-sut1-c2/p1 - empty.
+
        - s1-t21-sut1-c2/p2 - empty.
+
        - s1-t21-sut1-c2/p3 - empty.
+
        - s1-t21-sut1-c2/p4 - empty.
+
        - s1-t21-sut1-c4/p1 - 40GE-port1 xl710-2p40GE.
+
        - s1-t21-sut1-c4/p2 - 40GE-port2 xl710-2p40GE.
+
        - s1-t21-sut1-c9/p1 - empty.
+
        - s1-t21-sut1-c9/p2 - empty.
+
2. Server2 of Type-2:
+
    - testbedname: testbed21.
+
    - hostname: s2-t21-tg1.
+
    - IPMI IP: 10.30.50.42
+
    - Host IP: <..>
+
    - portnames:
+
        - s2-t21-tg1-c2/p1 - empty.
+
        - s2-t21-tg1-c2/p2 - empty.
+
        - s2-t21-tg1-c2/p3 - empty.
+
        - s2-t21-tg1-c2/p4 - empty.
+
        - s2-t21-tg1-c4/p1 - 40GE-port1 xl710-2p40GE.
+
        - s2-t21-tg1-c4/p2 - 40GE-port2 xl710-2p40GE.
+
        - s2-t21-tg1-c9/p1 - empty.
+
        - s2-t21-tg1-c9/p2 - empty.
+
 
+
==== 3-Node Xeon ====
+
 
+
9. Server9 of Type-1:
+
    - testbedname: testbed31.
+
    - hostname: s9-t31-sut1.
+
    - IPMI IP: 10.30.50.43
+
    - Host IP: <..>
+
    - portnames:
+
        - s9-t31-sut1-c2/p1 - 10GE-port1 x710-2p10GE.
+
        - s9-t31-sut1-c2/p2 - 10GE-port2 x710-2p10GE.
+
        - s9-t31-sut1-c2/p3 - empty.
+
        - s9-t31-sut1-c2/p4 - empty.
+
        - s9-t31-sut1-c4/p1 - 10GE-port1 x710-2p10GE.
+
        - s9-t31-sut1-c4/p2 - 10GE-port1 x710-2p10GE.
+
        - s9-t31-sut1-c9/p1 - empty.
+
        - s9-t31-sut1-c9/p2 - empty.
+
10. Server10 of Type-1:
+
    - testbedname: testbed31.
+
    - hostname: s10-t31-sut2.
+
    - IPMI IP: 10.30.50.44
+
    - Host IP: <..>
+
    - portnames:
+
        - s10-t31-sut2-c2/p1 - 10GE-port1 x710-2p10GE.
+
        - s10-t31-sut2-c2/p2 - 10GE-port2 x710-2p10GE.
+
        - s10-t31-sut2-c2/p3 - empty.
+
        - s10-t31-sut2-c2/p4 - empty.
+
        - s10-t31-sut2-c4/p1 - 10GE-port1 x710-2p10GE.
+
        - s10-t31-sut2-c4/p2 - 10GE-port1 x710-2p10GE.
+
        - s10-t31-sut2-c9/p1 - empty.
+
        - s10-t31-sut2-c9/p2 - empty.
+
11. Server11 of Type-2:
+
    - testbedname: testbed31.
+
    - hostname: s5-t31-tg1.
+
    - IPMI IP: 10.30.50.45
+
    - Host IP: <..>
+
    - portnames:
+
        - s11-t31-tg1-c2/p1 - 10GE-port1 x710-2p10GE.
+
        - s11-t31-tg1-c2/p2 - 10GE-port2 x710-2p10GE.
+
        - s11-t31-tg1-c2/p3 - empty.
+
        - s11-t31-tg1-c2/p4 - empty.
+
        - s11-t31-tg1-c4/p1 - 10GE-port1 x710-2p10GE.
+
        - s11-t31-tg1-c4/p2 - 10GE-port1 x710-2p10GE.
+
        - s11-t31-tg1-c9/p1 - empty.
+
        - s11-t31-tg1-c9/p2 - empty.
+
 
+
==== 3-Node Arm ====
+
 
+
Note: Server17 (TG) is shared between testbed33 & testbed34
+
 
+
15. Server15 of Type-3:
+
    - testbedname: testbed33.
+
    - hostname: s15-t33-sut1.
+
    - IPMI IP: <10.30.50.36>
+
    - Host IP: <10.30.51.36>
+
    - portnames:
+
        - s15-t33-sut1-ca/p1 - 10GE-port1 82599-2p10GE.
+
        - s15-t33-sut1-ca/p2 - 10GE-port2 82599-2p10GE.
+
        - s15-t33-sut1-c2/p1 - 25GE-port1 cx4-2p10GE.
+
        - s15-t33-sut1-c2/p2 - 25GE-port2 cx4-2p10GE.
+
16. Server16 of Type-3:
+
    - testbedname: testbed33.
+
    - hostname: s16-t33-sut2.
+
    - IPMI IP: <10.30.50.37>
+
    - Host IP: <10.30.51.37>
+
    - portnames:
+
        - s16-t33-sut2-ca/p1 - 10GE-port1 82599-2p10GE.
+
        - s16-t33-sut2-ca/p2 - 10GE-port2 82599-2p10GE.
+
        - s16-t33-sut2-c2/p1 - 25GE-port1 cx4-2p10GE.
+
        - s16-t33-sut2-c2/p2 - 25GE-port2 cx4-2p10GE.
+
17. Server17 of Type-2:
+
    - testbedname: testbed33.
+
    - hostname: s17-t33-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s17-t33-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s17-t33-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s17-t33-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s17-t33-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s17-t33-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s17-t33-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s17-t33-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s17-t33-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
 
+
18. Server18 of Type-4:
+
    - testbedname: testbed34.
+
    - hostname: s18-t34-sut1.
+
    - IPMI IP: N/A
+
    - Host IP: <10.30.51.41>
+
    - portnames:
+
        - s18-t34-sut1-ca/p1 - 10GE-port1 Marvell.
+
        - s18-t34-sut1-ca/p2 - 10GE-port2 Marvell.
+
19. Server19 of Type-4:
+
    - testbedname: testbed34.
+
    - hostname: s19-t34-sut2.
+
    - IPMI IP: N/A
+
    - Host IP: <10.30.51.42>
+
    - portnames:
+
        - s19-t34-sut2-ca/p1 - 10GE-port1 Marvell.
+
        - s19-t34-sut2-ca/p2 - 10GE-port2 Marvell.
+
17. Server17 of Type-2:
+
    - testbedname: testbed34.
+
    - hostname: s17-t33-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s17-t33-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s17-t33-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s17-t33-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s17-t33-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s17-t33-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s17-t33-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s17-t33-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s17-t33-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
 
+
==== 3-Node Atom ====
+
 
+
Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB.
+
 
+
9. Server20 of Type-5:
+
    - testbedname: testbed35.
+
    - hostname: s20-t35-sut1 (vex-yul-rot-netgate-1).
+
    - IPMI IP: 10.30.51.29 - screen -r /dev/ttyUSB0
+
    - Host IP: 10.30.51.9
+
    - portnames:
+
        - s20-t35-sut1-p1 - 10GE-port1 ix0 82599.
+
        - s20-t35-sut1-p2 - 10GE-port2 ix1 82599.
+
    - 1GB ports (tbd)
+
10. Server21 of Type-5:
+
    - testbedname: testbed35.
+
    - hostname: s20-t35-sut2 (vex-yul-rot-netgate-2).
+
    - IPMI IP: 10.30.51.30 - screen -r /dev/ttyUSB1
+
    - Host IP: 10.30.51.10
+
    - portnames:
+
        - s20-t35-sut1-p1 - 10GE-port1 ix0 82599.
+
        - s20-t35-sut1-p2 - 10GE-port2 ix1 82599.
+
    - 1GB ports (tbd)
+
11. Server22 of Type-5:
+
    - testbedname: testbed35.
+
    - hostname: s20-t35-sut3 (vex-yul-rot-netgate-3).
+
    - IPMI IP: 10.30.51.30 - screen -r /dev/ttyUSB2
+
    - Host IP: 10.30.51.11
+
    - portnames:
+
        - s20-t35-sut1-p1 - 10GE-port1 ix0 82599.
+
        - s20-t35-sut1-p2 - 10GE-port2 ix1 82599.
+
    - 1GB ports (tbd)
+
 
+
=== Physical Connectivity within Testbeds ===
+
 
+
==== 2-Node Xeon ====
+
 
+
One 2-Node testbeds are constructed by connecting 8 Xeon servers using below specification:
+
 
+
1. testbed21:
+
    - ring3 40GE-ports xl710-2p40GE on SUT
+
        - t21-tg1-c4/p1 to t21-sut1-c4/p1.
+
        - t21-sut1-c4/p2 to t21-tg1-c4/p2.
+
 
+
==== 3-Node Xeon ====
+
 
+
1. testbed31:
+
    - ring1 10GE-ports x710-2p10GE on SUTs:
+
        - t31-tg1-c2/p1 to t31-sut1-c2/p1.
+
        - t31-sut1-c2/p2 to t31-sut2-c2/p2.
+
        - t31-sut2-c2/p1 to t31-tg1-c2/p2.
+
    - ring2 10GE-ports x710-2p10GE on SUT:
+
        - t31-tg1-c4/p1 to t31-sut1-c4/p1.
+
        - t31-sut1-c4/p2 to t31-sut2-c4/p2.
+
        - t31-sut2-c4/p1 to t31-tg1-c4/p2.
+
 
+
==== 3-Node Arm ====
+
 
+
1. testbed33:
+
    - ring1 10GE-ports 82599-2p10GE on SUTs:
+
        - s17-t33-tg1-c2/p1 - s15-t33-sut1-ca/p1.
+
        - s15-t33-sut1-ca/p2 - s16-t33-sut2-ca/p1.
+
        - s16-t33-sut2-ca/p2 - s17-t33-tg1-c2/p2
+
 
+
2. testbed34:
+
    - ring1 10GE-ports Marvell on SUTs:
+
        - s17-t33-tg1-c2/p3 - s18-t34-sut1-ca/p1
+
        - s18-t34-sut1-ca/p2 - s19-t34-sut2-ca/p1
+
        - s19-t34-sut2-ca/p2 - s17-t33-tg1-c2/p4
+
 
+
==== 2-Node Atom ====
+
 
+
1. testbed35:
+
    - (tbd)
+
 
+
==== 3-Node Atom ====
+
 
+
1. testbed35:
+
    - (tbd)
+
 
+
== Target Build ==
+
 
+
=== Server/Ports Naming, NIC Placement ===
+
 
+
==== 2-Node Xeon ====
+
 
+
Each server in 2-node Xeon topology has its NIC cards placed, and NIC cards and ports indexed per following specification:
+
 
+
1. Server1 of Type-1:
+
    - testbedname: testbed21.
+
    - hostname: s1-t21-sut1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s1-t21-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s1-t21-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s1-t21-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s1-t21-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s1-t21-sut1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s1-t21-sut1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s1-t21-sut1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s1-t21-sut1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
2. Server2 of Type-2:
+
    - testbedname: testbed21.
+
    - hostname: s2-t21-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s2-t21-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s2-t21-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s2-t21-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s2-t21-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s2-t21-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s2-t21-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s2-t21-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s2-t21-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
3. Server3 of Type-1:
+
    - testbedname: testbed22.
+
    - hostname: s3-t22-sut1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s3-t22-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s3-t22-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s3-t22-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s3-t22-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s3-t22-sut1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s3-t22-sut1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s3-t22-sut1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s3-t22-sut1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
4. Server4 of Type-2:
+
    - testbedname: testbed22.
+
    - hostname: s4-t22-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s4-t22-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s4-t22-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s4-t22-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s4-t22-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s4-t22-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s4-t22-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s4-t22-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s4-t22-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
5. Server5 of Type-1:
+
    - testbedname: testbed23.
+
    - hostname: s5-t23-sut1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s5-t23-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s5-t23-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s5-t23-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s5-t23-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s5-t23-sut1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s5-t23-sut1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s5-t23-sut1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s5-t23-sut1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
6. Server6 of Type-2:
+
    - testbedname: testbed23.
+
    - hostname: s6-t23-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s6-t23-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s6-t23-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s6-t23-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s6-t23-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s6-t23-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s6-t23-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s6-t23-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s6-t23-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
7. Server7 of Type-1:
+
    - testbedname: testbed24.
+
    - hostname: s7-t24-sut1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s7-t24-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s7-t24-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s7-t24-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s7-t24-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s7-t24-sut1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s7-t24-sut1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s7-t24-sut1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s7-t24-sut1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
8. Server8 of Type-2:
+
    - testbedname: testbed21.
+
    - hostname: s8-t24-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s8-t24-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s8-t24-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s8-t24-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s8-t24-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s8-t24-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s8-t24-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s8-t24-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s8-t24-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
 
+
==== 3-Node Xeon ====
+
 
+
Each server in 3-node Xeon topology has its NIC cards placed, and NIC cards and ports indexed per following specification:
+
 
+
9. Server9 of Type-1:
+
    - testbedname: testbed31.
+
    - hostname: s9-t31-sut1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s9-t31-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s9-t31-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s9-t31-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s9-t31-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s9-t31-sut1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s9-t31-sut1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s9-t31-sut1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s9-t31-sut1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
10. Server10 of Type-1:
+
    - testbedname: testbed31.
+
    - hostname: s10-t31-sut2.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s10-t31-sut2-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s10-t31-sut2-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s10-t31-sut2-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s10-t31-sut2-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s10-t31-sut2-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s10-t31-sut2-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s10-t31-sut2-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s10-t31-sut2-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
11. Server11 of Type-2:
+
    - testbedname: testbed31.
+
    - hostname: s5-t31-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s11-t31-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s11-t31-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s11-t31-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s11-t31-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s11-t31-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s11-t31-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s11-t31-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s11-t31-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
12. Server12 of Type-1:
+
    - testbedname: testbed32.
+
    - hostname: s12-t32-sut1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s12-t32-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s12-t32-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s12-t32-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s12-t32-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s12-t32-sut1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s12-t32-sut1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s12-t32-sut1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s12-t32-sut1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
13. Server13 of Type-1:
+
    - testbedname: testbed32.
+
    - hostname: s13-t32-sut2.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s13-t32-sut2-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s13-t32-sut2-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s13-t32-sut2-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s13-t32-sut2-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s13-t32-sut2-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s13-t32-sut2-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s13-t32-sut2-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s13-t32-sut2-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
14. Server14 of Type-2:
+
    - testbedname: testbed32.
+
    - hostname: s-14-t32-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s14-t32-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s14-t32-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s14-t32-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s14-t32-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s14-t32-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s14-t32-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s14-t32-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s14-t32-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
 
+
==== 3-Node Arm ====
+
 
+
Note: Server17 (TG) is shared between testbed33 & testbed34
+
 
+
15. Server15 of Type-3:
+
    - testbedname: testbed33.
+
    - hostname: s15-t33-sut1.
+
    - IPMI IP: <10.30.50.36>
+
    - Host IP: <10.30.51.36>
+
    - portnames:
+
        - s15-t33-sut1-ca/p1 - 10GE-port1 82599-2p10GE.
+
        - s15-t33-sut1-ca/p2 - 10GE-port2 82599-2p10GE.
+
        - s15-t33-sut1-c2/p1 - 25GE-port1 cx4-2p10GE.
+
        - s15-t33-sut1-c2/p2 - 25GE-port2 cx4-2p10GE.
+
16. Server16 of Type-3:
+
    - testbedname: testbed33.
+
    - hostname: s16-t33-sut2.
+
    - IPMI IP: <10.30.50.37>
+
    - Host IP: <10.30.51.37>
+
    - portnames:
+
        - s16-t33-sut2-ca/p1 - 10GE-port1 82599-2p10GE.
+
        - s16-t33-sut2-ca/p2 - 10GE-port2 82599-2p10GE.
+
        - s16-t33-sut2-c2/p1 - 25GE-port1 cx4-2p10GE.
+
        - s16-t33-sut2-c2/p2 - 25GE-port2 cx4-2p10GE.
+
17. Server17 of Type-2:
+
    - testbedname: testbed33.
+
    - hostname: s17-t33-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s17-t33-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s17-t33-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s17-t33-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s17-t33-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s17-t33-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s17-t33-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s17-t33-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s17-t33-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
 
+
18. Server18 of Type-4:
+
    - testbedname: testbed34.
+
    - hostname: s18-t34-sut1.
+
    - IPMI IP: N/A
+
    - Host IP: <10.30.51.41>
+
    - portnames:
+
        - s18-t34-sut1-ca/p1 - 10GE-port1 Marvell.
+
        - s18-t34-sut1-ca/p2 - 10GE-port2 Marvell.
+
19. Server19 of Type-4:
+
    - testbedname: testbed34.
+
    - hostname: s19-t34-sut2.
+
    - IPMI IP: N/A
+
    - Host IP: <10.30.51.42>
+
    - portnames:
+
        - s19-t34-sut2-ca/p1 - 10GE-port1 Marvell.
+
        - s19-t34-sut2-ca/p2 - 10GE-port2 Marvell.
+
17. Server17 of Type-2:
+
    - testbedname: testbed34.
+
    - hostname: s17-t33-tg1.
+
    - IPMI IP: <..>
+
    - Host IP: <..>
+
    - portnames:
+
        - s17-t33-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
+
        - s17-t33-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
+
        - s17-t33-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
+
        - s17-t33-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
+
        - s17-t33-tg1-c4/p1 - (25GE|40GE)-port1 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s17-t33-tg1-c4/p2 - (25GE|40GE)-port2 (xxv710-DA2|xl710)-(2p25GE|2p40GE).
+
        - s17-t33-tg1-c9/p1 - 100GE-port1 ConnectX5-2p100GE.
+
        - s17-t33-tg1-c9/p2 - 100GE-port2 ConnectX5-2p100GE.
+
 
+
==== TCP/IP and L47 ====
+
 
+
<..>
+
 
+
==== AToM ====
+
 
+
<..>
+
 
+
=== Physical Connectivity within Testbeds ===
+
 
+
==== 2-Node Xeon ====
+
 
+
Four 2-Node testbeds are constructed by connecting 8 Xeon servers using below specification:
+
 
+
1. testbed21:
+
    - ring1 10GE-ports x710-4p10GE on SUT:
+
        - t21-tg1-c2/p1 to t21-sut1-c2/p1.
+
        - t21-sut1-c2/p2 to t21-tg1-c2/p2.
+
    - ring2 10GE-ports x710-4p10GE on SUT:
+
        - t21-tg1-c2/p3 to t21-sut1-c2/p3.
+
        - t21-sut1-c2/p4 to t21-tg1-c2/p4.
+
    - ring3 (25GE|40GE)-ports (xxv710-DA2|xl710)-(2p25GE|2p40GE) on SUT
+
        - t21-tg1-c4/p1 to t21-sut1-c4/p1.
+
        - t21-sut1-c4/p2 to t21-tg1-c4/p2.
+
    - ring4 100GE-ports ConnectX5-2p100GE on SUT:
+
        - t21-tg1-c9/p3 to t21-sut1-c9/p3.
+
        - t21-sut1-c9/p4 to t21-tg1-c9/p4.
+
    - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
+
        - t21-tg1-c10/p1 to t21-tg1-c10/p2.
+
        - t21-tg1-c10/p3 to t21-tg1-c10/p4.
+
 
+
2. testbed22:
+
    - <..>
+
 
+
3. testbed23:
+
    - <..>
+
 
+
4. testbed24:
+
    - <..>
+
 
+
==== 3-Node Xeon ====
+
 
+
1. testbed31:
+
    - ring1 10GE-ports x710-4p10GE on SUTs:
+
        - t31-tg1-c2/p1 to t31-sut1-c2/p1.
+
        - t31-sut1-c2/p2 to t31-sut2-c2/p2.
+
        - t31-sut2-c2/p1 to t31-tg1-c2/p2.
+
    - ring2 10GE-ports x710-4p10GE on SUT:
+
        - t31-tg1-c2/p3 to t31-sut1-c2/p3.
+
        - t31-sut1-c2/p4 to t31-sut2-c2/p4.
+
        - t31-sut2-c2/p3 to t31-tg1-c2/p4.
+
    - ring3 (25GE|40GE)-ports (xxv710-DA2|xl710)-(2p25GE|2p40GE) on SUT
+
        - t31-tg1-c4/p1 to t31-sut1-c4/p1.
+
        - t31-sut1-c4/p2 to t31-sut2-c4/p2.
+
        - t31-sut2-c4/p1 to t31-tg1-c4/p2.
+
    - ring4 100GE-ports ConnectX5-2p100GE on SUT:
+
        - t31-tg1-c9/p1 to t31-sut1-c9/p1.
+
        - t31-sut1-c9/p2 to t31-sut2-c9/p2.
+
        - t31-sut2-c9/p1 to t31-tg1-c9/p2.
+
    - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
+
        - t21-tg1-c10/p1 to t21-tg1-c10/p2.
+
        - t21-tg1-c10/p3 to t21-tg1-c10/p4.
+
 
+
2. testbed32:
+
    - <..>
+
 
+
==== 3-Node Arm ====
+
 
+
1. testbed33:
+
    - ring1 10GE-ports 82599-2p10GE on SUTs:
+
        - s17-t33-tg1-c2/p1 - s15-t33-sut1-ca/p1.
+
        - s15-t33-sut1-ca/p2 - s16-t33-sut2-ca/p1.
+
        - s16-t33-sut2-ca/p2 - s17-t33-tg1-c2/p2
+
 
+
2. testbed34:
+
    - ring1 10GE-ports Marvell on SUTs:
+
        - s17-t33-tg1-c2/p3 - s18-t34-sut1-ca/p1
+
        - s18-t34-sut1-ca/p2 - s19-t34-sut2-ca/p1
+
        - s19-t34-sut2-ca/p2 - s17-t33-tg1-c2/p4
+
 
+
==== TCP/IP and L47 ====
+
 
+
<..>
+
 
+
==== AToM ====
+
 
+
<..>
+
 
+
=== Host OS and configuration ===
+
 
+
<..>
+
 
+
==== SUTs ====
+
 
+
<..>
+
 
+
==== TG ====
+
 
+
<..>
+

Latest revision as of 18:16, 13 August 2018

This page has been deprecated. Content has been moved to FD.io CSIT testbeds - Xeon Skylake, Arm, Atom.