Nexus Retirement Plan

Introduction

Nexus is a repository manager currently in use to store FD.io project CI job artifacts (jenkins logs, robot framework output, etc.) and documentation that is hosted on an openstack instance. Starting in 2020, LF-IT has been working to reduce the cost hosting this data utilizing cloud services and intends to retire Nexus.

This page is being used to track the progress and capture requirements for FD.io projects job artifacts and documentation hosting.

CI Job Artifacts

The CI job artifacts will be hosted on an AWS S3 storage instance.

Documentation

Some FD.io project documentation is already hosted at readthedocs.io (e.g. hicn.readthedocs.io) and LFN has an account with ReadTheDocs. This is currently the plan of record for all FD.io project documentation, but the requirements of each project shall be documented here to verify if ReadTheDocs provides a good user experience for all projects. CSIT in particular has a very large amount of data contained in its reports.

FD.io Project Documentation Requirements

VPP

TBD.

CSIT

Currently CSIT is consuming about 0.6TB per 6 months of storage for job RAW data shipped to logs.fd.io/production silo (expiry-days '180'). These data are downloaded and post processed by a custom python based scripts for generating (so far) RTD themed output stored in docs.fd.io (on daily cadence or on-demand). The output content is static plain HTML/JavaScript.

Docs metrics:

- Trending <master branch>:
 - build action flow:
   download csit raw data (large volume) from logs.fd.io, post-process by 
   python, post-process by sphinx, upload artifacts to docs.fd.io.
 - duration:
   - artifacts download [hh:mm]: 4:30
   - postprocess raw data [hh:mm]: 6:00
   - postprocess spinx [hh:mm]: 0:01
   - total [hh:mm]: 10:30
 - content
   html, jpg, pdf, svg... 
 - data consumption
   - download up to 0.5TB in peak.
   - upload up to 10GB in peak.
 - frequency
   - twice a day
   
- Trending <per LTS branch>:
 - build action flow:
   download csit raw data (large volume) from logs.fd.io, post-process by 
   python, post-process by spinx, upload artifacts to docs.fd.io.
 - duration:
   - artifacts download [hh:mm]: 0:15
   - postprocess raw data [hh:mm]: 0:10
   - postprocess spinx [hh:mm]: 0:01
   - total [hh:mm]: 0:30
 - content
   html, jpg, pdf, svg... 
 - data consumption
   - download up to 0.5TB in peak.
   - upload 2GB.
 - frequency
   - per LTS release
   
- Report <per branch>:
 - build action flow:
   download csit raw data (large volume) from logs.fd.io, post-process by 
   python, post-process by spinx, upload artifacts to docs.fd.io.
 - duration:
   - artifacts download [hh:mm]: 2:00
   - postprocess raw data [hh:mm]: 1:30
   - postprocess spinx [hh:mm]: 0:01
   - total [hh:mm]: 3:40
 - content
   html, jpg, pdf, svg... 
 - data consumption
   - download up to 0.1TB in peak.
   - upload 10GB.
 - frequency
   - per release + debug