Cloud Images Creation

Laniakea exploits Cloud images for Express builds, i.e. the deployment of application which are pre-installed on a cloid images, and just reconfigured and started at deployment time.

These images needs to be rebuilt every time a new software release or update is delivered, and, of course, in case of critical security issues. Therefore, in the long run, their manual creation was not a sustainable strategy.

We use Hashicorp Packer to create images, whith OpenStack plugin, to instantiate the VM on openstack cloud and save it automatically on it, and Ansible for software installation and configuration. Packer will create a VM on OpenStack, will use ansible to configure the software on it (same role used by Laniakea Live builds), and, once finished, will save the image on Glance, ready to be used.

Therefore Packer is a prerequisite, and its installation is mandatory.

Finally, the image can be shared with all Laniakea tenants.

The Cloud image module encompasses:

  1. a YAML file listing the image to create and Ansible inputs.

  2. a python script for creating the image with Packer, which build the Packer json file and run it;

  3. the Ansible playbooks and corresponding variables for application installation, i.e. Galaxy, RStudio, Jupyther and more.

  4. a JenkinsFile for jenkins integration;

Note

This module requires the authentication on OpenStack, therefore OIDC Agent needs to be properly configured on jenkins nodes.

The image list

The list of the cloud images that need to be created is hosted on GitHub as a YAML file.

images_db_url

Every time a new image is created, it is stored on CMDB. To prevent the re-build of images alrady created we check, before image building if the image is already on CMDB. If yes it is skipped.

images

In the section images we describe the image to create. For each image that has to be created, a new section needs to be added.

In the following, we report the example of the creation of a galaxy cloud image:

galaxy-express:
  name: "<IMAGE NAME>" <--------
  version: <vX.Y.Z> <--------
  build: no
  packer:
      ssh_username: rocky <--------
      source_image: "<IMAGE ID>" <--------
      flavor: large  <--------
      volume_size: "10"  <--------
      network_id: "<NETWORK ID>" <--------
      playbook_file: "galaxy.yml" <--------
      ansible_galaxy_file: "galaxy.yml" <--------

name: The name of the cloud image

version: The version of the cloud image in the form vX.Y.Z, e.g. v1.0.0

build: a boolean to enable image building. If yes the image is built, if no, it is not.

ssh_username: the user Ansible will use to access the created VM (default: rocky).

source_image: the soruce image ID on OpenStack to use.

flavor: the flavor of the VM to use on OpenStack (default:large).

volume_size: the size of the volume used for the VM (default: “10” GB).

network_id: the network ID to be attached to the VM

playbook_file: Ansible playbook which will be run by Ansible, e.g. galaxy.yml. Available playbooks are stored in the playbook directory.

ansible_galaxy_file: the requiremets file which will be used by Ansible Galaxy to install roles. All requirements file are in the requiremets directory.

Image creation with Packer

Once the image list file is filled, the build is triggered through a python script.

To test the script, you need to login on OpenStack using the tenant RC file. It can be done using Keystone credentials (username and password) or a Keystone token.

If you are going to use a valid OIDC token, you need to exchange it with a Keystone token, and export it as OS_TOKEN. For example:

export OS_TOKEN=$(openstack --os-auth-type v3oidcaccesstoken --os-access-token ${OIDC_ACCESS_TOKEN} token issue  -f json | jq -r ".id")

Create a new python virutal environment:

python3 -m venv build

Activate it:

. build/bin/activate

Install requirements:

pip install -r ./cloud-images/requirements.txt

Finally you can run the script:

python3 build_images.py

Integration with Jenkins

A JenkinsFile is included in the cloud image repository. To use it, you just need to create a new pipeline and link it a JenkinsFile.

../../_images/laniakea-ci-image-build.png

Managing images

Finally, the created images need to be shared among all OpenStack tenants that should exploit them, and stored on CMDB.

In the share directory we have a script and a YAML file.

The YAML file is just the list of the tenants where the image has to be shared and few others Openstack parameters.

The python script share the image with these tenants. It just needs as inputs the OIDC token to access the tenants, the list, and the image ID.

python3 share_laniakea_cloud_images.py -t $IAM_ACCESS_TOKEN -l laniakea_tenant_list.yml --i 3c4de00e-02a0-404c-877e-126c5a0f0a10

The script perform the following operations:

  1. change image visibility to shared;

  2. set the right image tags for CIP imports

  3. add image to destinations tenants

  4. accept the new image for each tenant.

Resources

Cloud Veneto image management: https://userguide.cloudveneto.it/en/latest/ManagingImages.html