Rootless podman best practice

Posted on Sat 13 May 2023 in misc

Podman is a daemonless container engine that runs OCI and Docker compatible containers.

A podman container can be executed in four ways:

  1. A rootfull container (i.e., the container is started as the host user "root") with a container application running as root. With this the container application has access to system resources, comparable to any application running as root. An example is the busybox container.
  2. A rootfull container with a container application running as an unprivileged user. In this case the application voluntarily relinquishes its root privilege. An example of this is the official Mariadb container.
  3. A rootless container with the appplication running as root. Within the container everything seems to run as root but on the host the processes run as the user who started the container.
  4. A rootless container with an application running as an unprivileged user. The userid and groupid of the container user are mapped to host ids in a range assigned to each user in /etc/subuid and /etc/subgid.

This blog will focus on the last two use cases.

Contrary to docker, there is no systemwide daemon that spawns the containers but processes are indepedent entities. The standard procedure is to start pods and containers via systemd and where rootless containers are concerned, to use "systemctl --user". To enable starting a rootless container at boot time, a user specific systemd should be started. This is accomplished by issuing the command "loginctl --enable-linger username" as root.

An important drawback of running rootless containers is that it is not possible to create data volumes on its own block storage (via "podman volume create") or create default private networks (note that there is a limited option for the latter, see "Netavark networking" below).

Starting containers in a pod

To enable inter-container communications one should run the containers together in a pod. Network port forwarding from the host to containers can happen at the pod level as well.

Imagine that you have a Wordpress and a Mediawiki environment, both running as containers on the same host. For this you create two pods, one containing the Mediawiki container and a Mariadb container and the other with a Wordpress and a Mariadb container. In both cases, the database access is configured as TCP port 3306 on localhost. Both web applications find their own database because the pod contains the access.

Starting or stopping a pod starts or stops all containers assigned to the pod. Creating a pod with containers happens like this:

podman pod create --name podname [ -p hostport:containerport ] ...
podman create --pod podname --name container1 [ -v hostdirectory:mountpont ] ... imagename[:tag] [args]
podman create --pod podname --name container2 [ -v hostdirectory:mountpont ] ... imagename[:tag] [args]

It is a good idea to generate systemd files for the pods and containers so starting and stopping them can be delegated to systemd. That enables starting the pod and containers at boot time. Generating the files is done with:

podman generate systemd --files --name --new podname

Creating the service file for the pod creates service files for the relevant containers as well. The --files option creates service files i.o. dumping them to STDOUT. --name creates files with image names and tags i.o. image ids so you don't need to generate new files whenever a new version of the container image is used. Only make sure that the image name and tag of the new version is the same as that of the previous image. --new causes the podman process to not only stop when systemctl stop is used but the podman process is actually removed from the list (comparable to using the podman rm -f command).

When the files are generated, they can be copied to $HOME/.config/systemd/user and enabled with:

systemctl --user enable pod-podname container-container1 container-container2

Persistent storage

In docker (and rootfull podman) it is possible to assign block storage dedicated to docker or podman (with the "docker volume create --opt device=xxx" command). In a rootless environment you cannot connect a device. Instead, a directory is created in .local:

[poduser@uxhost ~]$ ls .local/share/containers/storage 
defaultNetworkBackend  networks   overlay-images  storage.lock
libpod       overlayfs   overlay-layers  tmp
mounts       overlay-containers  secrets   userns.lock
[poduser@uxhost ~]$ podman volume create MyVolume
[poduser@uxhost ~]$ ls .local/share/containers/storage/volumes/MyVolume 
[poduser@uxhost ~]$ podman run -it --rm -v MyVolume:/root/mydata alpine sh
/ # exit
[poduser@uxhost ~]$

It's also possible to mount another directory in a container. With that you can decide yourself where the data is placed (given proper write access). Do note that if SELinux restrictions are enforced, the file context for the mounted directory should be labled as "container_file_t". You can delegate this to podman by appending ":z" to the mount definition. You can also opt for appending ":Z", then access to the mount is further restricted via sVirt to only the relevant pod or container.

[poduser@uxhost ~]$ mkdir mariadb privateshare
[poduser@uxhost ~]$ ls -Zd mariadb privateshare
system_u:object_r:user_home_t:s0 mariadb  system_u:object_r:user_home_t:s0 privateshare
[poduser@uxhost ~]$ podman run -d --rm --name mariadb -e MARIADB_RANDOM_ROOT_PASSWORD=yes \
    -v ./mariadb:/var/lib/mysql:z -v ./privateshare:/privateshare:Z mariadb
[poduser@uxhost ~]$ ls -Zd mariadb privateshare
system_u:object_r:container_file_t:s0 mariadb  system_u:object_r:container_file_t:s0:c119,c552 privateshare
[poduser@uxhost ~]$ ps -Zp $(pgrep mariadb)
LABEL                               PID TTY          TIME CMD
system_u:system_r:container_t:s0:c119,c552 15827 ? 00:00:01 mariadbd
[poduser@uxhost ~]$ 

Files that are generated as the root user or group within the container are created as the user starting the container or her primary group on the host. Files that are created as an unprivileged user or non-root group in the container are mapped to a user or group within the /etc/subuid or /etc/subgid range for the user starting the container. Also make sure that the directory that is mounted in the container and its files are either owned and belong to the primary group of the user starting the container, or of a uid or gid withing that user's subuid/subgid range.

[poduser@uxhost ~]$ podman exec -it mariadb bash
root@b9a95df84be5:/# touch /privateshare/{rootfile,userfile}
root@b9a95df84be5:/# chown mysql:mysql /privateshare/userfile
root@b9a95df84be5:/# ls -l /privateshare
total 0
-rw-r--r--. 1 root  root  0 Feb 20 14:38 rootfile
-rw-r--r--. 1 mysql mysql 0 Feb 20 14:38 userfile
root@b9a95df84be5:/# id mysql
uid=999(mysql) gid=999(mysql) groups=999(mysql)
[poduser@uxhost ~]$ ls -l ./privateshare
total 0
-rw-r--r--. 1 poduser poduser 0 Feb 20 15:38 rootfile
-rw-r--r--. 1  494214  494214 0 Feb 20 15:38 userfile
[poduser@uxhost ~]$ grep poduser /etc/sub?id
[poduser@uxhost ~]$

Outside of the container the starting user does not have the right to modify files that are not owned by root in the container (like the "userfile" above). To enable this, the user has to temprary become "container root" with the "podman unshare" command. Note that owners and groups are then mapped according to /etc/passwd and /etc/group from the host, not those of the container.

[poduser@uxhost ~]$ echo foo > ./privateshare/userfile
bash: permission denied: ./privateshare/userfile
[poduser@uxhost ~]$ podman unshare bash
[root@uxhost ~]# ls -l ./privateshare
total 0
-rw-r--r--. 1 root             root  0 Feb 20 15:38 rootfile
-rw-r--r--. 1 systemd-coredump input 0 Feb 20 15:38 userfile
[root@uxhost ~]# id systemd-coredump
uid=999(systemd-coredump) gid=997(systemd-coredump) groups=997(systemd-coredump)
[root@uxhost ~]# echo foo > ./privateshare/userfile
[root@uxhost ~]# cat ./privateshare/userfile 
[root@uxhost ~]# 

Rootless nested containers

Rootless podman containers use fuse-overlayfs by default to store container images. This is centrally configured in /etc/containers/storage.conf. It has a better performance than native VFS storage but the drawback is that you can't run nested rootless containers (or e.g. K3S Kubernetes) because /dev/fuse is unavailable. For such a use case it's better to use VFS for container image storage but it's not possible to mix fuse-overlayfs and VFS for different images by the same user. To see what storage is used (given you have at least one container image present), just list the storage directory:

[poduser@uxhost ~]$ ls .local/share/containers/storage/
libpod  mounts  overlay  overlay-containers  overlay-images  overlay-layers  storage.lock  tmp  userns.lock
[poduser@uxhost ~]$ 

As the directory names imply, overlayfs is used. To switch to VFS all containers and container images need to be deleted first. That can be accomplished with

podman system reset

To use VFS given this clean state, create the file $HOME/.config/containers/storage.conf with the following content:

driver = "vfs"

Now if you download or create an image it is stored as VFS:

[poduser@uxhost ~]$ ls .local/share/containers/storage/
libpod  mounts  storage.lock  tmp  userns.lock  vfs  vfs-containers  vfs-images  vfs-layers
[poduser@uxhost ~]$ 

Netavark networking

With rooted containers it is possible to create your own network and give each container its own IP-address in that range. There are use cases where that is preferable (e.g. with a CI/CD pipeline where multiple similar containers are spawn, each of which listens to the same network port). That is possible by using the "Netavark" network backend. For this (assuming Enterprise Linux) you install the netavark rpm. Installing that will also install the aardvark-dns RPM. There are a few issues with using a rootless network:

  1. The network is not available to the host, only to containers in the network.
  2. Containers in the network are not limited to other containers in the same pod but can connect to all other containers in the same network
  3. Port publishing via the pod only works with CNI networks, so if a port of a container in a netavark network needs to be puslished to the host, it should be done in the container.
  4. For every netavark network an aardvark-dns process is started so containers can resolve other containers on the name. This only works for containers within the same netavark network, so not for the host and not for containers in other networks.

Each podman user has the same standard network available named "podman". This network has a "cni" backend which is used for forwarding exposed container ports at the pod level as explained in Starting containers in a pod. To enable networks with a "netavark" backend, the following configuration should be added to "$HOME/.config/containers/containers.conf":

network_backend = "netavark"

To use this, all existing containers should be stopped and removed. Creating a netavark network is done like this:

[poduser@uxhost ~]$ podman network create --subnet --gateway rootlessnet
[poduser@uxhost ~]$ podman network ls
2f259bab93aa  podman       bridge
8a0786c87d36  rootlessnet  bridge
[poduser@uxhost ~]$

It is now possible to create containers in the new network and they can connect to each other using the container name:

[poduser@uxhost ~]$ podman run -d --rm --network rootlessnet --name host10 alpine sleep 86400
[poduser@uxhost ~]$ podman run -d --rm --network rootlessnet --name host11 alpine sleep 86400
[poduser@uxhost ~]$ podman exec -it host10 ping -c2 host11
PING host11 ( 56 data bytes
64 bytes from seq=0 ttl=64 time=0.044 ms
64 bytes from seq=1 ttl=64 time=0.114 ms

--- host11 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.044/0.079/0.114 ms
[poduser@uxhost ~]$

There's one drawback to this approach, i.e. that the netavark backend is now hard wired and all new containers will use that whether it's preferred or not. It's better to leave the backend configuration empty which enabled a cni backend unless explicitely stated otherwise. This way it's still possible to contain exposed ports within a pod and not have them available to all containers in a network. To achieve this, use the following content for "$HOME/.config/containers/containers.conf":

network_backend = ""

It's now possible to combine containers with cni and netavark network backends in the same pod:

[poduser@uxhost ~]$ podman pod create --name mixed -p 8888:80 
[poduser@uxhost ~]$ podman pod start mixed
[poduser@uxhost ~]$ podman run -d --rm --pod mixed --name nginx nginx
[poduser@uxhost ~]$ podman run -d --rm --pod mixed --name alpcni curl sleep 86400
[poduser@uxhost ~]$ podman run -d --rm --pod mixed --name alpavark --network rootlessnet curl sleep 43200
[poduser@uxhost ~]$ podman ps
CONTAINER ID  IMAGE                                    COMMAND               CREATED             STATUS                 PORTS                 NAMES
7e53fabf7e0a  localhost/podman-pause:4.2.0-1673519486                        About a minute ago  Up About a minute ago>80/tcp  ea6215f74bfb-infra
804044254b84  localhost/nginx:latest                   nginx -g daemon o...  About a minute ago  Up About a minute ago>80/tcp  nginx
587afafd81b2  localhost/curl:latest                    sleep 86400           24 seconds ago      Up 24 seconds ago>80/tcp  alpcni
bba06b34f0b2  localhost/curl:latest                    sleep 43200           4 seconds ago       Up 4 seconds ago                             alpavark

The port forwarding in the list shows that the "nginx" and "alpcni" containers use the cni network and the "alpavark" container doesn't. The host can connect to the web server via the forwarded port and the Alpine container in the cni network can connect to it via the exposed port (80) but the container in the netavark network can't reach the exposed port, only the forwarded port on the host:

[poduser@uxhost ~]$ curl -s http://localhost:8888 |grep title
<title>Welcome to nginx!</title>
[poduser@uxhost ~]$ podman exec -it alpcni curl -s http://localhost:80 |grep title
<title>Welcome to nginx!</title>
[poduser@uxhost ~]$ podman exec -it alpavark curl -s http://localhost:80 |grep title
[poduser@uxhost ~]$ podman exec -it alpavark curl -s http://localhost:8888 |grep title 
[poduser@uxhost ~]$ ip a s enp1s0|grep ' inet '
    inet brd scope global noprefixroute enp1s0
[poduser@uxhost ~]$ podman exec -it alpavark curl -s |grep title
<title>Welcome to nginx!</title>
[poduser@uxhost ~]$ 

(BTW, the "curl" image is is just a standard Alpine image with the "curl" tool added). In fact the advantage for a container with a netavark network to be part of a pod is limited. About the only achievement is that the container gets the same sVirt context as the pod and other containers in it while other containers get their own sVirt context:

[poduser@uxhost ~]$ podman run -d --name nonpod --network rootlessnet curl sleep 22600
[poduser@uxhost ~]$ ps -Zu $(id -u)|grep :container_t:                                
system_u:system_r:container_t:s0:c125,c698 14196 ? 00:00:00 catatonit
system_u:system_r:container_t:s0:c125,c698 14227 ? 00:00:00 nginx
system_u:system_r:container_t:s0:c125,c698 14282 ? 00:00:00 sleep
system_u:system_r:container_t:s0:c125,c698 14391 ? 00:00:00 sleep
system_u:system_r:container_t:s0:c584,c600 14735 ? 00:00:00 sleep
[poduser@uxhost ~]$ 

With that the container can access mounts that are configured as private storage with the ":Z" mount option by other containers in the same pod.

Podman Secrets

Container secrets are available for rootless containers and work as usual. You can create a secret with the "podman secret create" command and if you pass that secret to the container with the --secret argument, the secret is available in the /run/secrets directory of the container. This is a tmpfs filesystem so even if you would use the podman commit command to create a new image of the running container, the secret would not be part of the image. Note that by default the "file" driver is used which stores the secret base64 encoded (i.e. unencrypted) on the host filesystem but if the container application is capabale of using a password file, it is still better than passing the password in an environment variable.

[poduser@uxhost ~]$ echo 'D33pS3cr37'| podman secret create blogeg -
[poduser@uxhost ~]$ grep 666eb268d629db07752367da4 .local/share/containers/storage/secrets/filedriver/secretsdata.json
  "666eb268d629db07752367da4": "RDMzcFMzY3IzNwo=",
[poduser@uxhost ~]$ echo RDMzcFMzY3IzNwo=|base64 -d
[poduser@uxhost ~]$ podman run -it --rm --secret blogeg alpine sh
/ # cat /run/secrets/blogeg
/ #

There are two more drivers available to podman secrets but they are still undocumented. One is the "pass" driver which uses GPG in the same way as the GNU pass command and the other is the "shell" driver where you define shell scripts for the four options list, lookup, store and delete. These can be specified in the containers.conf file or as --driver.opts flags during secret creation time.