1. Summary

  3. Main architecture

  4. Release process

  5. Create, update, publish a vagrant VM 

  6. ARM Pipeline

  7. Firebase

  8. Current DevOps problems/issues

  9. Ongoing projects

  10. Build Process 

  11. Registry

     




1. Summary

In most books DevOps is the bridge between Ops and Dev, it’s true in most cases but in magma our primary mission is to make sure magma is a project that is easily to build, test, deploy and improve. We have to make sure that the developer/contributor experience is on point. It goes from simple PR messages to welcome new contributors until configuring the dev environment so it runs on most machines and is configured in a single command line. 


Migration planning: 2022-09-12 Registry migration plan

artifactory.magmacore.org has address 51.178.63.140 (ask for access) ((needs to be moved))

The registry contains: 

  • Docker registry (dev and prod)
  • Blob registry 
  • Ubuntu packages (dev and prod)
  • Helm (dev and prod)

How to get access?


Is a CI software that run all our testing, build and packages deployment, the product offers different kind of environment where you can run your CI steps, only used so far: 

  • MacOS node: they are the only one that have virtualization enabled which is needed for integ test
  • Ubuntu nodes: Ubuntu VMs with some pre-installed software. Virtualization is disabled on these nodes, so it is a clean dev environment where we can run anything
  • Docker nodes: simple docker run where you can run containers, used basically for all our docker testing or linting


  • AWS account id: 372544488264
    We use AWS mainly for testing or when Github Action doesn’t have what we need. For example there is no ARM node in Github Action so we can use AWS in order to fill that gap. We also use it for experimentation/testing purposes (ex: build on ubuntu without VM project)


  • CI dashboard: https://magma-ci.web.app/
    This CI dashboard gather all build and testing that can’t be done directly in Github Action, It helps us get a sense of what is currently healthy within magma and what is not. 


Where we host our code, do our project management/bug reporting. 


It’s the bridge between all github action and our specific nodes (aws/lab/specific architecture..) It also gather a lot of CI stats that are displayed in the CI dashboard


Where we host our images for the dev VM that contains everything to develop on magma (build tools, debugging tools, dependencies, right networking, os etc..)


  • 1Password:

For DevOps related credentials.

 

3.  Main architecture


4. Release process


Step 1: Setup a deadline to cut the branch

We need to get all the features listed and what’s pending/will be part of the release, that list will be published in the release note, when that is done we can move to step 2. 

Step 2: Cut the branch 

When the list of features has been frozen (we can still backport in case of bug fixes or late features) you need to branch out master, release name has to follow that pattern 

vX.Y

X= main version (in case of major features/subproject launch)

Y= Important features/APi changes/Backward incompatible changes

This branch will contain vX.Y.* by using tags

Example branch v1.6 contains 1.6.0, 1.6.1, 1.6.2 .. 1.6.*

Step 3: Configure the branch 

When you checkout the branch will by default push all packages to the ci repository in the registry you need to change it in the 

** Drop the different github links **  

Step 4: (Optional) In case of backport: 

In case of backport we have a backport job (will be repaired soon, I got the creds to have the missing creds), this backport job is triggered on every PR landing on master, if you tag a pr apply-v*.* it will automatically backport to the v*.* branch, in case of a merge conflict it will post a message on the PR on how to fix it and will remove the apply tag, the creator will have to manually create a PR without the conflict to the desired branch.

Step 5: Approval and release packages

At this step you’re supposed to do the last checking, integration testing, live testing in lab etc, if needed pull request are backported in the release branch and we generate as much version as we need for stable 

Step 6: Last checks

Use all the releases scripts to promote the packages from the all the dev registries to the production one. 

Note: only one package per version, you can’t have multiple 1.6.1 for example in the production registry, in case we need to regenerate a package to include a fix, we’ll have to bump it to 1.6.2 and re release it

Step 7: Tag & Release 

```
Git tag <version>

git push --tags

```

It will generate a release note that you need to manually fill out



5. Creation, Testing and Publishing of Magma VM Base Images

The lte/gateway VMs are defined via Vagrant. They are build on top of base images that are currently hosted in the Vagrant Cloud. This documentation describes how these base images can be created, tested, and published.

  • If you plan to upgrade to a new distribution version (e.g., from bionic to focal):
  • Make sure the magmacore artifactory supports this version.
  • Change the "vm.box" in lte/gateway/Vagrantfile to the generic target one (e.g. "ubuntu/focal64") , adapt the box version to the one stated in the Vagrant Cloud, and in the corresponding "ansible.playbook"-file set "preburn = true".
  • Up the VM and fix possible problems (missing packages, errors, etc)

HOST [magma/lte/gateway]$ vagrant up <name>

  • After manual smoke testing, switch back to "preburn = false".
  • In the corresponding JSON in orc8r/tools/packer update boot_command, iso_url, iso_checksum, etc for the distribution (some templates can be found HERE).

HOST [magma/orc8r/tools/packer]$ packer build <JSON-file>.json

  • Testing:
  • Register the VM with vagrant

HOST [magma/lte/gateway]$ vagrant box add path/to/box.box --name <name>

  • Adapt the respective "vm.box = <name>" and "vm.box_version = <version>" in the lte/gateway/Vagrantfile with the information from

HOST [magma/lte/gateway]$ vagrant box list

  • The VM can now be booted using

HOST [magma/lte/gateway]$ vagrant up <name>

  • Perform all the respective unit and/or integration tests in the new environment.
  • Upload / publish the build artifact:
  • Create an environment variable with the magmacore vagrant cloud token (access can be granted by the Linux Foundation).

HOST [magma/orc8r/tools/packer]$ export VAGRANT_CLOUD_TOKEN=<token>

  • Run the vagrant-box-upload.sh script. The name and provider of the VM are read from the filename and the version is set to the current date.

HOST [magma/orc8r/tools/packer]$ ./vagrant-box-upload.sh path/to/box.box

  • The uploaded box must be published/released by a developer holding admin rights to the magmacore Vagrant Cloud account.
  • Adapt the "vm.box_version" of the uploaded box in the lte/gateway/Vagrantfile to the newly published version.




6. ARM Pipeline


  1. Summary


Magma is a very complex project that can be deploy on multiple type of architecture (amd64 and arm64), in order to provide quality build to the community we have to build, test and deploy magma on all those platforms to assure we aren’t releasing any issues. 

In order to test magma we need to integrate a pipeline for every kind of build, unfortunately, github action doesn’t have any arm node by default (only the mac in a limited quantity), on top of that if we link our own nodes to github action it means that every person creating a PR for magma could use it (known github action). In order to address this problem, we created our own system using firebase and a custom dashboard. 


2. Architecture



Here is the architecture of the offered solution: 







3. Workflow


Classic workflow to deliver packages to the test registry

  1. Contributor makes a pull request 

In order to generate builds that will be pushed to the registry your PR needs to pass all precheks and get merged
 2. Pull request get accepted and launch arm job

Once the pull request is merged, CI will launch a couple of jobs and the ARM build. The ARM build will be on an Ubuntu 20.04 Github Action instance, we’ll install all prerequisite packages to launch the arm build (install ansible, aws-cli, etc.. )

  1. Launch Graviton instance

Once all prerequisites are installed on the Github Action ubuntu node, we’ll execute an ansible script that will automatically launch a graviton instance

  1. Configure itself and launch arm build

The ansible script will remotely configure the AWS graviton instance and make sure all prerequisites are installed in order to run magma arm build on it (we’ll create an ami which has all of it pre install but the script will still check if dependencies are installed)

  1. Push the newly build package to the registry

When the build is done and successful, the ansible script will automatically push it to the registry as a testing package, when this package is pushed to the registry it will be available on https://artifactory.magmacore.org/ and everyone can use it 

  1. report back the status

When the ansible script is done (all ssh actions from the Github Action node on the graviton are done, it can be fail or pass) the github action script will keep running on the github action instanceand go to step 7

  1. store the result to firebase

The github action script will push the information to the firebase database, we push the commit id, build type, build result etc.. 

  1. Read the latest status from firebase

The firebase database can easily be read by the magma-ci dashboard, so developer can be informed if their changes broke any component of magma on any architecture 


Release workflow:

When arm packages are created by the above workflow, we have them available in the dev registry, in order to move them to the production registry (and be able to install them on any agw) we have a release process that will take care of moving it from the dev to the production registry. 

7. Firebase


Incoming

8. Current DevOps problems/issues


incoming

9. Ongoing projects

  • [P0] Magma dep handling and build checks @nils (TNG)
  • [P0] AMI images - shubhamtatvamasi@gmail.com + Tim
  • [P1] Bazel build status to take over legacy build - @nils 
  • [P1] Release process - automation and doc - @Yogesh shubhamtatvamasi@gmail.com@govindarajan 
  • [P1] Arm build job - @govindarajan @shubham @nils 
  • [P1] Registry move - LF engineers Tim @shubham @nils 
  • [P2] Dockerfile - Tim Shubham @govindarajan 
  • [P2] Discussion about Ran images Tim Shubham @govindarajan 

10. Build Process 


Incoming


11. Managing Artifacts


Useful URLS:


What is a registry? 

A registry is a place where we host and distribute all the build versions of a software. Everyone is able to connect to it and download any kind of packages that are needed to test, build, deploy magma. 

Those packages can be docker images, apt packages (arm/amd, stretch/focal..), binaries or helm charts. 


What are we using? 

We’re currently using JFrog Artifactory (https://jfrog.com/artifactory/). As described on the website:

“JFrog Artifactory is the single solution for housing and managing all the artifacts, binaries, packages, files, containers, and components for use throughout your software supply chain.

JFrog Artifactory serves as your central hub for DevOps, integrating with your tools and processes to improve automation, increase integrity, and incorporate best practices along the way.”

 

Design

Design is pretty straightforward, we bought a Jfrog license for an on premise solution, we have some instances that we rent on ovh.com where we host it. We own the domain name https://artifactory.magmacore.org/

As a unanimous decision, we decided to go with the LF foundation JFrog solution,they will manage everything for us, instead of hosting and managing the server ourselves, we’ll use LF and their Saas solution in order to replace the registry. The only problem is that they will host it and we will have to work with the new URL https://linuxfoundation.jfrog.org ,  so we need to check if we can spin up a reverse proxy to the current https://artifactory.magmacore.org/ 


The important repos 

  • Apt repo 
    • One for testing: packages-test (local)
    • 2 for production: packages-prod (local) + packages (virtual)
  • Docker repo
    • For Testing docker-test (virtual) contains:
      • docker-cwag-test (local)
      • docker-orc8r-test (local)
      • docker-agw-test (local)
    • One for production docker (virtual) contains:
      • docker-cwag-prod (local)
      • docker-orc8r-prod (local)
      • docker-agw-prod (local)
  • Helm repo
    • One for testing helm-test (local)
    • 2 for production helm(virtual) contains helm-prod (local)
  • Blob storage
    • One for testing blob-test (local)
    • 2 for production blob (virtual) contains blob-prod (local)


AGW Package grouping:


For the AGW builds, we currently maintain a repository for each release and OS. For example, focal-1.5.4 directory contains the Magma AGW and Sctpd package for release 1.5.4, as well as all third party dependencies of that build. Similarly, focal-1.7.0 tracks the same for Magma release 1.7.0. 

Design choice 1: This leads to duplication of several third-party packages, which do not change across different releases, but have to exist in multiple directories. This is a side-effect of the CI job, which pushes all artifacts to the particular release directory. An obvious fix for this is to modify the CI job to publish all packages to a single repo, but label them such that the package manager can install the correct version of every third-party dependency.

Pydep versioning: Currently the AGW build script creates the python dependency list, which contains the package name and the smallest version of that package which should be installed for the Magma AGW package to work. That is, it contains “pkg_name P >= version V”  to ensure that any version newer than version V of package P is installed before the Magma package is installed.

Design choice 2: Does the “>=” check for package versions create unstable environments when installing Magma AGW? Do we need version pinning, such that all dependencies are pinned with “pkg_name P = version V”. 

Structure of the apt repo:

dists/

pool/

         openvswitch

         Magma1.6.1

         Magma1.6.2


Use that registry:

add to the apt sources <artifactory.magmacore.org> stretch main 

Apt update 

Apt install/upgrade magma

Credentials and management 

In order to use the registry at its full potential we need to generate a few tokens/accounts, we have created a bot account that has a write token to the registry, this account allows CI to push all artifacts from CI machines to the registry. Ideally no one can push anything to the registry manually, only CI should be the source of truth of all packages.

Right now every approvers-ci has an account that is manually created when the person gets voted in the github group, with the new Sass managed by LF, only LF will have credential and every single backend request will require the creation of a ticket to LF support. 


  • No labels
Write a comment…