Packaging Ansible Collections as Ansible Execution Environments
With Ansible I automate, among other things, platforms, services and operating systems. Each of them comes with modules that bring their own Python packages, and sooner or later two of those want a different version of the same package. What I ended up with is one Ansible Execution Environment (AEE) per product, built from the collection that product already has.
If you use Ansible Automation Platform (AAP), you already know execution environments: every job runs in one. The image is the runtime, Ansible included. In this project I used the same images locally, running playbooks with ansible-navigator and testing roles with Molecule. The orchestration ran in Argo Workflows.
One environment for everything
The obvious start is one generic AEE with every collection in it. That works until two modules need incompatible versions of the same package. pip can't install both, so either the build fails or one product runs on a version nobody tested. You can work around that by letting the collection or role install its own requirements when it runs, but then every run needs access to a package index. In an offline or air-gapped environment that access isn't there. And a dependency change for one product still forces a rebuild and retest of the image for all of them.
Generic AEESo I split it up: one image per collection, versioned along with that collection. A new product gets its own image and leaves the others alone.
Develop
A collection per product
Each product has its own collection (kubernetes_platform, nutanix_platform,
rhel_platform) with its own galaxy.yml and version number. The
roles follow the objects of the product. For Kubernetes those are cluster,
namespace and hardening, each wrapping the modules that talk to the cluster and
bringing it to the desired state.
In the repo the collection sits in the default path ansible-galaxy uses for a collection, ansible_collections/<namespace>/<name>. Everything that belongs to the collection lives in that directory; the playbooks sit next to it, in the root of the repo.
kubernetes_platform/
├── ansible_collections/
│ └── automation/
│ └── kubernetes_platform/
│ ├── galaxy.yml
│ ├── README.md
│ ├── meta/
│ │ └── runtime.yml
│ ├── plugins/
│ │ └── modules/
│ ├── roles/
│ │ ├── cluster/
│ │ ├── namespace/
│ │ └── hardening/
│ ├── requirements.txt
│ └── bindep.txt
├── playbooks/
│ └── deploy_cluster.yml
├── execution-environment.yml
└── .gitlab-ci.yml
Collection and image are both published to private automation hub (Galaxy), so AAP and everything else pull the same version from one place.
Develop
Building the image
The execution-environment.yml describes the image and nothing else: a
RHEL-based EE base image from the project's own registry, Python, ansible-core,
ansible-runner and the collection. The ansible_collections directory and the
playbooks go into the build as they are. Galaxy installs the collection from that path, including the
collections it depends on, and the playbooks end up in /playbooks.
---
version: 3
images:
base_image:
name: registry.example.com/automation/ee-minimal-rhel9:latest
dependencies:
python_interpreter:
package_system: python3.11
python_path: /usr/bin/python3.11
ansible_core:
package_pip: ansible-core>=2.16,<2.17
ansible_runner:
package_pip: ansible-runner
galaxy:
collections:
- name: /build/ansible_collections/automation/kubernetes_platform
type: dir
additional_build_files:
- src: ansible_collections
dest: ansible_collections
- src: playbooks
dest: playbooks
additional_build_steps:
append_final:
- COPY _build/playbooks /playbooks
options:
package_manager_path: /usr/bin/microdnf
Python and system packages stay with the collection that needs them. ansible-builder
reads the requirements.txt and bindep.txt of every collection it
installs. kubernetes_platform depends on four collections, and each brings its own dependencies along: kubernetes.core the kubernetes Python client, ansible.utils packages such as netaddr and jsonschema, community.hashi_vault the hvac client for Vault, and community.crypto system packages such as OpenSSL. kubernetes_platform itself lists none of them.
namespace: automation
name: kubernetes_platform
version: 1.4.0
readme: README.md
authors:
- Platform team <platform@example.com>
dependencies:
kubernetes.core: ">=5.0.0"
ansible.utils: ">=5.0.0"
community.hashi_vault: ">=6.0.0"
community.crypto: ">=2.0.0"Develop
Testing locally
Before anything goes to the pipeline, I build the image locally from the same file and run a playbook in
it with ansible-navigator. A role change is then tested against the runtime the pipeline will
build, not against whatever Python my own environment happens to have.
ansible-builder build -f execution-environment.yml -t kubernetes-ee:dev
ansible-navigator run playbooks/deploy_cluster.yml --eei kubernetes-ee:devBuild and publish
The pipeline
GitLab CI builds the image on a Git tag. Building needs a container runtime, and that runtime has to run inside the CI job's own container, which doesn't work out of the box (more under Challenges). The job pushes the image to a registry, in this project private automation hub (Galaxy), Harbor or the GitLab Container Registry. A second job publishes the collection.
.gitlab-ci.yml examplestages:
- build
- publish
build-ee:
stage: build
image: registry.example.com/automation/ansible-buildkit:latest
tags:
- podman-in-podman
rules:
- if: $CI_COMMIT_TAG
script:
- ansible-builder build --container-runtime podman
-f execution-environment.yml
-t $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
- echo "$CI_REGISTRY_PASSWORD" |
podman login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
- podman push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
# Publish the collection with the same version as the image
publish-collection:
stage: publish
image: registry.example.com/automation/ansible-buildkit:latest
rules:
- if: $CI_COMMIT_TAG
script:
- ansible-galaxy collection build
ansible_collections/automation/kubernetes_platform
- ansible-galaxy collection publish
automation-kubernetes_platform-$CI_COMMIT_TAG.tar.gz
--server "$AH_SERVER_URL" --api-key "$AH_API_TOKEN"
The tag is the version from galaxy.yml, so collection kubernetes_platform 1.4.0
runs in kubernetes-ee:1.4.0. Going back to an earlier release means pinning the earlier tag.
Run
In production
Argo Workflows did the orchestration here. A workflow step points at a pinned image tag and runs a
playbook from /playbooks. The inventory isn't in the image; Argo passes it in as an input
artifact.
- name: deploy-cluster
inputs:
artifacts:
- name: inventory
path: /runner/inventory
container:
image: registry.example.com/automation/kubernetes-ee:1.4.0
command: [ansible-playbook]
args: ["-i", "/runner/inventory", "/playbooks/deploy_cluster.yml"]With AAP it works the same way. Push the image to private automation hub (Galaxy), set it as the execution environment of a job template, and AAP supplies the inventory.
ArchitectureTry it yourself
The structure, the execution-environment.yml and the GitLab pipeline from this post are in
ansible-ee-example as a working repo. It uses a public UBI 9 base image instead of the private one, depends only on
kubernetes.core, and its roles are empty placeholders: the repo is about the execution
environment, not about the automation inside it. The ee_info playbook prints which Python, ansible-core
and kubernetes client the run actually used. It exists mainly to settle the "but which Python
was that?" discussion before it starts.
git clone https://github.com/wsslhllmn/ansible-ee-example.git
cd ansible-ee-example
ansible-builder build -f execution-environment.yml -t kubernetes-ee:dev
ansible-navigator run playbooks/ee_info.yml \
--eei kubernetes-ee:dev --mode stdout$ ansible-navigator run playbooks/ee_info.yml \
--eei kubernetes-ee:dev --mode stdout
PLAY [Show which runtime this playbook runs with] **********************
TASK [Collect runtime information] *************************************
ok: [localhost]
TASK [Show it] *********************************************************
ok: [localhost] => {
"msg": [
"Python 3.11.13 (/usr/bin/python3.11)",
"ansible-core 2.16.19",
"kubernetes 37.0.1"
]
}
PLAY RECAP *************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0
ignored=0
Build
Challenges
Problemansible-builder needs a container runtime
to actually build the image, and that build runs inside the GitLab CI job's own container. So the
runtime itself has to run nested inside a container, which doesn't work out of the box.
FixTwo ways that both work.
- A GitLab runner that supports running Podman inside a container (podman-in-podman), with
ansible-builderpointed at it via--container-runtime podman. That is the.gitlab-ci.ymlshown above. - Buildah, without a nested Podman.
ansible-builder createwrites the build context and theContainerfilewithout building anything, and Buildah builds and pushes that context inside the CI job's container without a daemon.
build-ee:
stage: build
image: registry.example.com/automation/ansible-buildkit:latest
variables:
BUILDAH_ISOLATION: chroot
STORAGE_DRIVER: vfs
script:
- ansible-builder create -f execution-environment.yml
- buildah build -f context/Containerfile
-t $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG context
- echo "$CI_REGISTRY_PASSWORD" |
buildah login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
- buildah push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAGConclusion
One execution environment per collection solves the dependency conflicts: every product gets the Python and system packages it needs, without clashing with the others. The collection and its image share one version number, so every run is traceable to a tag and rolling back means pinning the previous one.
I test locally against the same image that runs in production, and because the image is a plain container, it runs in any orchestrator, from Argo Workflows to AAP. Sharing the code no longer means pointing at a repo: the collection is published, the image sits in a registry, and anyone who pulls them gets exactly the same thing.
It costs a build pipeline per collection and a registry for the images. Once those are in place, a new product is just one more collection and one more image, and the rest of the setup stays exactly as it is.