Difference between revisions of "Nomad Operations and Planning"

From fd.io
Jump to: navigation, search
(→‎Nomad Planning Wish List)
(→‎Nomad Operations Tasks)
 
(15 intermediate revisions by 2 users not shown)
Line 6: Line 6:
  
 
= Nomad Operational Status =
 
= Nomad Operational Status =
−
* TBD - add description or link to Nomad architecture / configuration
+
* [[Nomad Physical Topology]]
 +
* [[Nomad Configuration]]
 +
* [[Nomad Monitoring]]
 +
TBD - add description or link to Nomad architecture / configuration
 
* TBD - Add links to Nomad monitoring status / data
 
* TBD - Add links to Nomad monitoring status / data
−
 
−
== Jenkins Nomad Plugin Configuration ==
 
−
=== Proposed Jenkins Nomad Plugin Label to Docker Image Mapping ===
 
−
Jenkins Nomad Plugin Labels can only be created via LF Service Desk requests. 
 
−
In order to test CI jobs in the Jenkins Sandbox, a different label must exist
 
−
which points to a sandbox Dockerhub repository to avoid disrupting operational
 
−
jobs during docker image testing. This is the proposed nomenclature for the
 
−
mapping of Jenkins Nomad Plugin labels and the corresponding dockerhub repositories
 
−
(production and sandbox variants).
 
−
 
−
All new docker images will first be pushed to the Dockerhub associated 'sandbox repo'
 
−
and all CI jobs utilizing the Jenkins Nomad Plugin Label will be verified in the Jenkins
 
−
Sandbox, by modifying the appropriate JJB YAML files to point to the sandbox repo.
 
−
Once all CI jobs have been verified, then the image will be pushed to the operational
 
−
dockerhub repository associated with the label.
 
−
 
−
'''NOTE''': the 'sandbox labels' should NEVER be merged into the ci-management repository.
 
−
 
−
<nowiki>
 
−
fdiotools-prod-{arch}-{os}
 
−
fdiotools-sandbox-{arch}-{os}
 
−
</nowiki>
 
−
 
−
'''QUESTION 1''': Is there a need for project specific images?
 
−
 
−
'''QUESTION 2''': Is there a need for different size labels (small, medium, large) with different resource allocations (eg. CPU, Memory, etc)?
 
−
 
−
{| class="wikitable"
 
−
! Label
 
−
! Dockerhub Repo
 
−
|-
 
−
| fdiotools-prod-amd-ubuntu1804
 
−
| https://hub.docker.com/repository/docker/fdiotools/prod_amd_ubuntu1804
 
−
|-
 
−
| fdiotools-prod-arm-ubuntu1804
 
−
| https://hub.docker.com/repository/docker/fdiotools/prod_arm_ubuntu1804
 
−
|-
 
−
| fdiotools-sandbox-amd-ubuntu1804
 
−
| https://hub.docker.com/repository/docker/fdiotools/sandbox_amd_ubuntu1804
 
−
|-
 
−
| fdiotools-sandbox-arm-ubuntu1804
 
−
| https://hub.docker.com/repository/docker/fdiotools/sandbox_arm_ubuntu1804
 
−
|}
 
−
 
−
=== Legacy Jenkins Label to Docker Image Mapping ===
 
−
These labels are currently defined (circa VPP 20.05) and will be removed once
 
−
there are no references to them in the ci-management repo.  Some of them are
 
−
not being used today.
 
−
 
−
{| class="wikitable"
 
−
! Legacy Label(s)
 
−
! Dockerhub Repo
 
−
|-
 
−
| ubuntu1604-us, ubuntu1604-s, ubuntu1604-m, ubuntu1604-l, ubuntu1604-hub-us, ubuntu1604-nus,
 
−
| https://hub.docker.com/repository/docker/snergster/vpp-ubuntu16
 
−
|-
 
−
| ubuntu1604arm-hub-us
 
−
| https://hub.docker.com/repository/docker/snergster/vpp-arm-ubuntu16
 
−
|-
 
−
| ubuntu1804-us, ubuntu1804-s, ubuntu1804-m, ubuntu1804-l, ubuntu1804-hub-us, vpp-csit-device, vpp-csit-ubuntu18,
 
−
| https://hub.docker.com/repository/docker/snergster/vpp-ubuntu18
 
−
|-
 
−
| ubuntu1804arm-hub-us, ubuntu1804arm-s, ubuntu1804arm-m, vpp-csit-arm-ubuntu18
 
−
| https://hub.docker.com/repository/docker/snergster/vpp-arm-ubuntu18
 
−
|-
 
−
| centos7-us, centos7-s, centos7-m, centos7-l, centos7-hub-us,
 
−
| https://hub.docker.com/repository/docker/snergster/vpp-centos
 
−
|-
 
−
| ubuntu2004-us
 
−
| https://hub.docker.com/repository/docker/snergster/vpp-ubuntu20
 
−
|-
 
−
| centos8-us
 
−
| https://hub.docker.com/repository/docker/snergster/vpp-centos8
 
−
|}
 
  
 
= Nomad Operations Tasks =
 
= Nomad Operations Tasks =
Line 92: Line 21:
 
! ETA
 
! ETA
 
|-
 
|-
−
| Move Nomad Docker images from https://hub.docker.com/search?q=snergster&type=image into fdiotools dockerhub account.
+
| Move Nomad Operational Docker images from [https://hub.docker.com/u/snergster snergster docker hub account] into [https://hub.docker.com/u/fdiotools fdiotools docker hub account].
 +
| Dave W.
 +
| 50%
 +
| June 1, 2020
 +
|-
 +
| Update Ubuntu1804 & Centos7 Nomad Docker images to include clang-9 toolchain packages required by VPP 'make install-deps' and lf-infra-publish macro.
 
| Dave W.
 
| Dave W.
 
| 25%
 
| 25%
−
| May 20, 2020
+
| June 2, 2020
 
|-
 
|-
−
| Update Ubuntu1804 & Centos7 Nomad Docker images to include clang-9 toolchain packages required by VPP 'make install-deps'.
+
| New Jenkins/Nomad labels for production, verify, & sandbox
 
| Dave W.
 
| Dave W.
−
| 10%
+
|  
−
| May 20, 2020
+
| June 1, 2020
 
|-
 
|-
−
| Perform fresh installation of Ubuntu 18.04 Server on t4-virl1, t4-virl2, & t4-virl3
+
| Nomad server OS upgrades/normalization. Utilize ansible to create a uniform bare-metal OS environment across all Nomad servers.
 
| Peter M.
 
| Peter M.
−
|
+
| 99.9%
−
|
+
| May 29, 2020
−
|-
+
−
| Restore Nomad configuration on t4-virl1, t4-virl2, & t4-virl3 and rejoin on VPP cluster.
+
−
| Dave W.
+
−
|
+
−
|
+
 
|-  
 
|-  
 
| Build & test ubuntu 20.04 and centos8 Docker images for CI executors to run respective OS jobs.
 
| Build & test ubuntu 20.04 and centos8 Docker images for CI executors to run respective OS jobs.
Line 119: Line 48:
 
| Fix server-type-c4-3 (10.32.8.16) SDD with an HDD, reinstall Ubuntu 18.04 and restore to Nomad cluster.
 
| Fix server-type-c4-3 (10.32.8.16) SDD with an HDD, reinstall Ubuntu 18.04 and restore to Nomad cluster.
 
| Dave W.
 
| Dave W.
 +
| [https://secure.vexxhost.com/billing/viewticket.php?tid=AGA-517631&c=3RxtvTXs Vexxhost Ticket Created]
 
| TBD
 
| TBD
−
| Vexxhost Ticket Created
 
 
|-  
 
|-  
 
| Update VPP ci-management configurations to use global jjb macros (lf-publisher & build-discarder)
 
| Update VPP ci-management configurations to use global jjb macros (lf-publisher & build-discarder)
Line 129: Line 58:
 
| Export Gerrit & Jenkins logs and other operational data to Nomad servers
 
| Export Gerrit & Jenkins logs and other operational data to Nomad servers
 
| Dave W. & Vanessa V.
 
| Dave W. & Vanessa V.
 +
| [https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-19811 LF Ticket Created]
 
| TBD
 
| TBD
−
| LF Ticket Created
 
 
|}
 
|}
  
 
= Nomad Planning Wish List =
 
= Nomad Planning Wish List =
 
This is the list of long term Nomad tasks.  Please move them to the Nomad Operations Tasks and provide owner/ET information when they are being actively worked on.
 
This is the list of long term Nomad tasks.  Please move them to the Nomad Operations Tasks and provide owner/ET information when they are being actively worked on.
−
* Add Nomad nodes to LF DNS & make the names the same as the hostname
+
* Clean up old/unused Jenkins jobs.
 +
* Use Configuration as Code Jenkins Plugin to manage Jenkins configuration (including Nomad Plugin) via YAML configuration files.
 +
* Nomad cluster resiliency testing/hardening improvements.
 +
* Nomad docker image CI/CD pipeline.
 +
* Convert task list to use JIRA tickets/epics for tracking ongoing Nomad work.
 +
* Create ci-management jobs to do automated build/test/verify for CI process and weekly upgrade of docker images.
 +
* Add Nomad nodes to LF DNS & make the names the same as the hostname.
 +
* Add automated Nomad server quorum loss tests
 +
* Add VPP "test crash" testcase to 'make test'
 
* Add VPP 'make test-debug w/ ASAN enabled' verify job
 
* Add VPP 'make test-debug w/ ASAN enabled' verify job
−
* Nomad server OS upgrades/normalization. Utilize ansible to create a uniform bare-metal OS environment across all Nomad servers.
 
−
* Convert Jenkins Nomad-plugin configuration spreadsheet to JJB managed YAML configuration files.
 
 
* [https://plugins.jenkins.io/nomad/ Investigate Jenkins Nomad-plugin security issues.]
 
* [https://plugins.jenkins.io/nomad/ Investigate Jenkins Nomad-plugin security issues.]
 
* Convert Nomad/Jenkins/Gerrit monitoring/screen-scraping hacks into an operational monitoring system using exported gerrit & jenkins logs & nomad cli output.
 
* Convert Nomad/Jenkins/Gerrit monitoring/screen-scraping hacks into an operational monitoring system using exported gerrit & jenkins logs & nomad cli output.
−
* Add a mechanism to measure/track the memory consumed by the CI jobs inside Docker images
+
* Add a mechanism to measure/track the memory consumed by the CI jobs inside Docker images. pmikus_comment: Depends if we want ability to do live monitoring or ability of storing logs (how long?). I can make prometheus to work for us by very simple change in config.
  
 
= Completed Nomad Tasks =
 
= Completed Nomad Tasks =
Line 164: Line 99:
 
| 100%
 
| 100%
 
| May 18, 2020
 
| May 18, 2020
 +
|-
 +
| Perform fresh installation of Ubuntu 18.04 Server on t4-virl1, t4-virl2, & t4-virl3
 +
| Peter M.
 +
| 100%
 +
| May 25, 2020
 +
|-
 +
| Restore Nomad configuration on t4-virl1, t4-virl2, & t4-virl3 and rejoin on VPP cluster.
 +
| Peter M.
 +
| 100%
 +
| May 26, 2020
 
|}
 
|}

Latest revision as of 01:23, 23 June 2020

Nomad clusters are hosted on dedicated servers in the FD.io lab and used to manage Docker container based executors for FD.io project CI jobs.

Physical Lab Infrastructure

Nomad Operational Status

TBD - add description or link to Nomad architecture / configuration

  • TBD - Add links to Nomad monitoring status / data

Nomad Operations Tasks

This is the current list of high priority Nomad tasks.

Task Description Owner  % Complete ETA
Move Nomad Operational Docker images from snergster docker hub account into fdiotools docker hub account. Dave W. 50% June 1, 2020
Update Ubuntu1804 & Centos7 Nomad Docker images to include clang-9 toolchain packages required by VPP 'make install-deps' and lf-infra-publish macro. Dave W. 25% June 2, 2020
New Jenkins/Nomad labels for production, verify, & sandbox Dave W. June 1, 2020
Nomad server OS upgrades/normalization. Utilize ansible to create a uniform bare-metal OS environment across all Nomad servers. Peter M. 99.9% May 29, 2020
Build & test ubuntu 20.04 and centos8 Docker images for CI executors to run respective OS jobs. Dave W.
Fix server-type-c4-3 (10.32.8.16) SDD with an HDD, reinstall Ubuntu 18.04 and restore to Nomad cluster. Dave W. Vexxhost Ticket Created TBD
Update VPP ci-management configurations to use global jjb macros (lf-publisher & build-discarder) Vanessa V.
Export Gerrit & Jenkins logs and other operational data to Nomad servers Dave W. & Vanessa V. LF Ticket Created TBD

Nomad Planning Wish List

This is the list of long term Nomad tasks. Please move them to the Nomad Operations Tasks and provide owner/ET information when they are being actively worked on.

  • Clean up old/unused Jenkins jobs.
  • Use Configuration as Code Jenkins Plugin to manage Jenkins configuration (including Nomad Plugin) via YAML configuration files.
  • Nomad cluster resiliency testing/hardening improvements.
  • Nomad docker image CI/CD pipeline.
  • Convert task list to use JIRA tickets/epics for tracking ongoing Nomad work.
  • Create ci-management jobs to do automated build/test/verify for CI process and weekly upgrade of docker images.
  • Add Nomad nodes to LF DNS & make the names the same as the hostname.
  • Add automated Nomad server quorum loss tests
  • Add VPP "test crash" testcase to 'make test'
  • Add VPP 'make test-debug w/ ASAN enabled' verify job
  • Investigate Jenkins Nomad-plugin security issues.
  • Convert Nomad/Jenkins/Gerrit monitoring/screen-scraping hacks into an operational monitoring system using exported gerrit & jenkins logs & nomad cli output.
  • Add a mechanism to measure/track the memory consumed by the CI jobs inside Docker images. pmikus_comment: Depends if we want ability to do live monitoring or ability of storing logs (how long?). I can make prometheus to work for us by very simple change in config.

Completed Nomad Tasks

Task Description Owner  % Complete Finish Date
Move Nomad build executor Dockerfiles from https://github.com/snergfdio/* into the ci-management project. Dave W. 100% April 29, 2020
Add a sudoer/admin account to all Nomad Servers. Dave W. 100% May 18, 2020
Move server-type-c4-2 from Class 's5ci' to Class 'builder' to cover t4-virl* nomad clients during upgrade. Dave W. 100% May 18, 2020
Perform fresh installation of Ubuntu 18.04 Server on t4-virl1, t4-virl2, & t4-virl3 Peter M. 100% May 25, 2020
Restore Nomad configuration on t4-virl1, t4-virl2, & t4-virl3 and rejoin on VPP cluster. Peter M. 100% May 26, 2020
Retrieved from "https://wiki.fd.io/index.php?title=Nomad_Operations_and_Planning&oldid=10664"