BlogOctober 20267 min readAutomationPlatform Engineering

Packaging Ansible Collections as Ansible Execution Environments

Develop locally Build GitLab CI Publish Galaxy + registry Run in an orchestrator 1 Develop collection roles + modules 2 Define image base + deps 3 Test locally ansible-navigator 4 Tag a release git tag 1.4.0 5 Build image ansible-builder 6 Image + collection both 1.4.0 7 Pin the image tag easy rollback 8 Run playbook with inventory

With Ansible I automate, among other things, platforms, services and operating systems. Each of them comes with modules that bring their own Python packages, and sooner or later two of those want a different version of the same package. What I ended up with is one Ansible Execution Environment (AEE) per product, built from the collection that product already has.

If you use Ansible Automation Platform (AAP), you already know execution environments: every job runs in one. The image is the runtime, Ansible included. In this project I used the same images locally, running playbooks with ansible-navigator and testing roles with Molecule. The orchestration ran in Argo Workflows.

One environment for everything

The obvious start is one generic AEE with every collection in it. That works until two modules need incompatible versions of the same package. pip can't install both, so either the build fails or one product runs on a version nobody tested. You can work around that by letting the collection or role install its own requirements when it runs, but then every run needs access to a package index. In an offline or air-gapped environment that access isn't there. And a dependency change for one product still forces a rebuild and retest of the image for all of them.

Generic AEE
In one generic AEE, kubernetes_platform needs package X 1.x and nutanix_platform needs package X 2.x: a conflict. In one generic AEE, kubernetes_platform needs package X 1.x and nutanix_platform needs package X 2.x: a conflict.

So I split it up: one image per collection, versioned along with that collection. A new product gets its own image and leaves the others alone.

Develop

A collection per product

Each product has its own collection (kubernetes_platform, nutanix_platform, rhel_platform) with its own galaxy.yml and version number. The roles follow the objects of the product. For Kubernetes those are cluster, namespace and hardening, each wrapping the modules that talk to the cluster and bringing it to the desired state.

In the repo the collection sits in the default path ansible-galaxy uses for a collection, ansible_collections/<namespace>/<name>. Everything that belongs to the collection lives in that directory; the playbooks sit next to it, in the root of the repo.

kubernetes_platform/ example collection
kubernetes_platform/
├── ansible_collections/
│   └── automation/
│       └── kubernetes_platform/
│           ├── galaxy.yml
│           ├── README.md
│           ├── meta/
│           │   └── runtime.yml
│           ├── plugins/
│           │   └── modules/
│           ├── roles/
│           │   ├── cluster/
│           │   ├── namespace/
│           │   └── hardening/
│           ├── requirements.txt
│           └── bindep.txt
├── playbooks/
│   └── deploy_cluster.yml
├── execution-environment.yml
└── .gitlab-ci.yml

Collection and image are both published to private automation hub (Galaxy), so AAP and everything else pull the same version from one place.

Develop

Building the image

The execution-environment.yml describes the image and nothing else: a RHEL-based EE base image from the project's own registry, Python, ansible-core, ansible-runner and the collection. The ansible_collections directory and the playbooks go into the build as they are. Galaxy installs the collection from that path, including the collections it depends on, and the playbooks end up in /playbooks.

execution-environment.yml example
---
version: 3

images:
  base_image:
    name: registry.example.com/automation/ee-minimal-rhel9:latest

dependencies:
  python_interpreter:
    package_system: python3.11
    python_path: /usr/bin/python3.11
  ansible_core:
    package_pip: ansible-core>=2.16,<2.17
  ansible_runner:
    package_pip: ansible-runner
  galaxy:
    collections:
      - name: /build/ansible_collections/automation/kubernetes_platform
        type: dir

additional_build_files:
  - src: ansible_collections
    dest: ansible_collections
  - src: playbooks
    dest: playbooks

additional_build_steps:
  append_final:
    - COPY _build/playbooks /playbooks

options:
  package_manager_path: /usr/bin/microdnf

Python and system packages stay with the collection that needs them. ansible-builder reads the requirements.txt and bindep.txt of every collection it installs. kubernetes_platform depends on four collections, and each brings its own dependencies along: kubernetes.core the kubernetes Python client, ansible.utils packages such as netaddr and jsonschema, community.hashi_vault the hvac client for Vault, and community.crypto system packages such as OpenSSL. kubernetes_platform itself lists none of them.

galaxy.yml example
namespace: automation
name: kubernetes_platform
version: 1.4.0
readme: README.md
authors:
  - Platform team <platform@example.com>
dependencies:
  kubernetes.core: ">=5.0.0"
  ansible.utils: ">=5.0.0"
  community.hashi_vault: ">=6.0.0"
  community.crypto: ">=2.0.0"

Develop

Testing locally

Before anything goes to the pipeline, I build the image locally from the same file and run a playbook in it with ansible-navigator. A role change is then tested against the runtime the pipeline will build, not against whatever Python my own environment happens to have.

ansible-builder build -f execution-environment.yml -t kubernetes-ee:dev
ansible-navigator run playbooks/deploy_cluster.yml --eei kubernetes-ee:dev

Build and publish

The pipeline

GitLab CI builds the image on a Git tag. Building needs a container runtime, and that runtime has to run inside the CI job's own container, which doesn't work out of the box (more under Challenges). The job pushes the image to a registry, in this project private automation hub (Galaxy), Harbor or the GitLab Container Registry. A second job publishes the collection.

.gitlab-ci.yml example
stages:
  - build
  - publish

build-ee:
  stage: build
  image: registry.example.com/automation/ansible-buildkit:latest
  tags:
    - podman-in-podman
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - ansible-builder build --container-runtime podman
        -f execution-environment.yml
        -t $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
    - echo "$CI_REGISTRY_PASSWORD" |
        podman login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - podman push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG

# Publish the collection with the same version as the image
publish-collection:
  stage: publish
  image: registry.example.com/automation/ansible-buildkit:latest
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - ansible-galaxy collection build
        ansible_collections/automation/kubernetes_platform
    - ansible-galaxy collection publish
        automation-kubernetes_platform-$CI_COMMIT_TAG.tar.gz
        --server "$AH_SERVER_URL" --api-key "$AH_API_TOKEN"

The tag is the version from galaxy.yml, so collection kubernetes_platform 1.4.0 runs in kubernetes-ee:1.4.0. Going back to an earlier release means pinning the earlier tag.

Run

In production

Argo Workflows did the orchestration here. A workflow step points at a pinned image tag and runs a playbook from /playbooks. The inventory isn't in the image; Argo passes it in as an input artifact.

Argo Workflows step example
- name: deploy-cluster
  inputs:
    artifacts:
      - name: inventory
        path: /runner/inventory
  container:
    image: registry.example.com/automation/kubernetes-ee:1.4.0
    command: [ansible-playbook]
    args: ["-i", "/runner/inventory", "/playbooks/deploy_cluster.yml"]

With AAP it works the same way. Push the image to private automation hub (Galaxy), set it as the execution environment of a job template, and AAP supplies the inventory.

Architecture
Local machine ansible-builder ansible-navigator molecule GitLab repo with the collection CI runner ansible-builder Platforms, services and operating systems Kubernetes KVM RHEL Private automation hub kubernetes-ee:1.4.0 collection 1.4.0 Orchestrator Argo Workflows AAP git push + tag push publish pull pull playbook + inventory

Try it yourself

The structure, the execution-environment.yml and the GitLab pipeline from this post are in ansible-ee-example as a working repo. It uses a public UBI 9 base image instead of the private one, depends only on kubernetes.core, and its roles are empty placeholders: the repo is about the execution environment, not about the automation inside it. The ee_info playbook prints which Python, ansible-core and kubernetes client the run actually used. It exists mainly to settle the "but which Python was that?" discussion before it starts.

git clone https://github.com/wsslhllmn/ansible-ee-example.git
cd ansible-ee-example
ansible-builder build -f execution-environment.yml -t kubernetes-ee:dev
ansible-navigator run playbooks/ee_info.yml \
  --eei kubernetes-ee:dev --mode stdout
ee_info output (expected)
$ ansible-navigator run playbooks/ee_info.yml \
  --eei kubernetes-ee:dev --mode stdout

PLAY [Show which runtime this playbook runs with] **********************

TASK [Collect runtime information] *************************************
ok: [localhost]

TASK [Show it] *********************************************************
ok: [localhost] => {
    "msg": [
        "Python 3.11.13 (/usr/bin/python3.11)",
        "ansible-core 2.16.19",
        "kubernetes 37.0.1"
    ]
}

PLAY RECAP *************************************************************
localhost : ok=2  changed=0  unreachable=0  failed=0  skipped=0  rescued=0
ignored=0

Build

Challenges

Problemansible-builder needs a container runtime to actually build the image, and that build runs inside the GitLab CI job's own container. So the runtime itself has to run nested inside a container, which doesn't work out of the box.

FixTwo ways that both work.

  1. A GitLab runner that supports running Podman inside a container (podman-in-podman), with ansible-builder pointed at it via --container-runtime podman. That is the .gitlab-ci.yml shown above.
  2. Buildah, without a nested Podman. ansible-builder create writes the build context and the Containerfile without building anything, and Buildah builds and pushes that context inside the CI job's container without a daemon.
.gitlab-ci.yml example (Buildah)
build-ee:
  stage: build
  image: registry.example.com/automation/ansible-buildkit:latest
  variables:
    BUILDAH_ISOLATION: chroot
    STORAGE_DRIVER: vfs
  script:
    - ansible-builder create -f execution-environment.yml
    - buildah build -f context/Containerfile
        -t $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG context
    - echo "$CI_REGISTRY_PASSWORD" |
        buildah login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - buildah push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG

Conclusion

One execution environment per collection solves the dependency conflicts: every product gets the Python and system packages it needs, without clashing with the others. The collection and its image share one version number, so every run is traceable to a tag and rolling back means pinning the previous one.

I test locally against the same image that runs in production, and because the image is a plain container, it runs in any orchestrator, from Argo Workflows to AAP. Sharing the code no longer means pointing at a repo: the collection is published, the image sits in a registry, and anyone who pulls them gets exactly the same thing.

It costs a build pipeline per collection and a registry for the images. Once those are in place, a new product is just one more collection and one more image, and the rest of the setup stays exactly as it is.