Difference between revisions of "Nexus Retirement Plan"
Dwallacelf (Talk | contribs) (Created page with "== Introduction == Nexus is a repository manager currently in use to store FD.io project CI job artifacts (jenkins logs, robot framework output, etc.) and documentation that...") |
Dwallacelf (Talk | contribs) (→CSIT) |
||
| Line 21: | Line 21: | ||
==== CSIT ==== | ==== CSIT ==== | ||
| − | + | Currently CSIT is consuming about 0.6TB per 6 months of storage for job RAW data shipped to logs.fd.io/production silo (expiry-days '180'). These data are downloaded and post processed by a custom python based scripts for generating (so far) RTD themed output stored in docs.fd.io (on daily cadence or on-demand). | |
| + | The output content is static plain HTML/JavaScript. | ||
| + | |||
| + | |||
| + | Docs metrics: | ||
| + | |||
| + | - Trending <master branch>: | ||
| + | - build action flow: | ||
| + | download csit raw data (large volume) from logs.fd.io, post-process by | ||
| + | python, post-process by sphinx, upload artifacts to docs.fd.io. | ||
| + | - duration: | ||
| + | - artifacts download [hh:mm]: 4:30 | ||
| + | - postprocess raw data [hh:mm]: 6:00 | ||
| + | - postprocess spinx [hh:mm]: 0:01 | ||
| + | - total [hh:mm]: 10:30 | ||
| + | - content | ||
| + | html, jpg, pdf, svg... | ||
| + | - data consumption | ||
| + | - download up to 0.5TB in peak. | ||
| + | - upload up to 10GB in peak. | ||
| + | - frequency | ||
| + | - twice a day | ||
| + | |||
| + | - Trending <per LTS branch>: | ||
| + | - build action flow: | ||
| + | download csit raw data (large volume) from logs.fd.io, post-process by | ||
| + | python, post-process by spinx, upload artifacts to docs.fd.io. | ||
| + | - duration: | ||
| + | - artifacts download [hh:mm]: 0:15 | ||
| + | - postprocess raw data [hh:mm]: 0:10 | ||
| + | - postprocess spinx [hh:mm]: 0:01 | ||
| + | - total [hh:mm]: 0:30 | ||
| + | - content | ||
| + | html, jpg, pdf, svg... | ||
| + | - data consumption | ||
| + | - download up to 0.5TB in peak. | ||
| + | - upload 2GB. | ||
| + | - frequency | ||
| + | - per LTS release | ||
| + | |||
| + | - Report <per branch>: | ||
| + | - build action flow: | ||
| + | download csit raw data (large volume) from logs.fd.io, post-process by | ||
| + | python, post-process by spinx, upload artifacts to docs.fd.io. | ||
| + | - duration: | ||
| + | - artifacts download [hh:mm]: 2:00 | ||
| + | - postprocess raw data [hh:mm]: 1:30 | ||
| + | - postprocess spinx [hh:mm]: 0:01 | ||
| + | - total [hh:mm]: 3:40 | ||
| + | - content | ||
| + | html, jpg, pdf, svg... | ||
| + | - data consumption | ||
| + | - download up to 0.1TB in peak. | ||
| + | - upload 10GB. | ||
| + | - frequency | ||
| + | - per release + debug | ||
Latest revision as of 21:21, 28 June 2021
Contents
Introduction
Nexus is a repository manager currently in use to store FD.io project CI job artifacts (jenkins logs, robot framework output, etc.) and documentation that is hosted on an openstack instance. Starting in 2020, LF-IT has been working to reduce the cost hosting this data utilizing cloud services and intends to retire Nexus.
This page is being used to track the progress and capture requirements for FD.io projects job artifacts and documentation hosting.
CI Job Artifacts
The CI job artifacts will be hosted on an AWS S3 storage instance.
Documentation
Some FD.io project documentation is already hosted at readthedocs.io (e.g. hicn.readthedocs.io) and LFN has an account with ReadTheDocs. This is currently the plan of record for all FD.io project documentation, but the requirements of each project shall be documented here to verify if ReadTheDocs provides a good user experience for all projects. CSIT in particular has a very large amount of data contained in its reports.
FD.io Project Documentation Requirements
VPP
TBD.
CSIT
Currently CSIT is consuming about 0.6TB per 6 months of storage for job RAW data shipped to logs.fd.io/production silo (expiry-days '180'). These data are downloaded and post processed by a custom python based scripts for generating (so far) RTD themed output stored in docs.fd.io (on daily cadence or on-demand). The output content is static plain HTML/JavaScript.
Docs metrics:
- Trending <master branch>: - build action flow: download csit raw data (large volume) from logs.fd.io, post-process by python, post-process by sphinx, upload artifacts to docs.fd.io. - duration: - artifacts download [hh:mm]: 4:30 - postprocess raw data [hh:mm]: 6:00 - postprocess spinx [hh:mm]: 0:01 - total [hh:mm]: 10:30 - content html, jpg, pdf, svg... - data consumption - download up to 0.5TB in peak. - upload up to 10GB in peak. - frequency - twice a day - Trending <per LTS branch>: - build action flow: download csit raw data (large volume) from logs.fd.io, post-process by python, post-process by spinx, upload artifacts to docs.fd.io. - duration: - artifacts download [hh:mm]: 0:15 - postprocess raw data [hh:mm]: 0:10 - postprocess spinx [hh:mm]: 0:01 - total [hh:mm]: 0:30 - content html, jpg, pdf, svg... - data consumption - download up to 0.5TB in peak. - upload 2GB. - frequency - per LTS release - Report <per branch>: - build action flow: download csit raw data (large volume) from logs.fd.io, post-process by python, post-process by spinx, upload artifacts to docs.fd.io. - duration: - artifacts download [hh:mm]: 2:00 - postprocess raw data [hh:mm]: 1:30 - postprocess spinx [hh:mm]: 0:01 - total [hh:mm]: 3:40 - content html, jpg, pdf, svg... - data consumption - download up to 0.1TB in peak. - upload 10GB. - frequency - per release + debug
