Tech Ramblings/2023-05-13T00:00:00+02:00Rootless podman best practice2023-05-13T00:00:00+02:002023-05-13T00:00:00+02:00Rob Wolframtag:None,2023-05-13:/podmanbp.html<p><a href="https://podman.io/">Podman</a> is a daemonless container engine that runs
<a href="https://opencontainers.org/">OCI</a> and
<a href="https://www.docker.com/">Docker</a> compatible containers.</p>
<p>A podman container can be executed in four ways:</p>
<ol>
<li>A rootfull container (i.e., the container is started as the host
user "root") with a container application running as root. With this
the container application has access …</li></ol><p><a href="https://podman.io/">Podman</a> is a daemonless container engine that runs
<a href="https://opencontainers.org/">OCI</a> and
<a href="https://www.docker.com/">Docker</a> compatible containers.</p>
<p>A podman container can be executed in four ways:</p>
<ol>
<li>A rootfull container (i.e., the container is started as the host
user "root") with a container application running as root. With this
the container application has access to system resources, comparable
to any application running as root. An example is the busybox
container.</li>
<li>A rootfull container with a container application running as an
unprivileged user. In this case the application voluntarily relinquishes
its root privilege. An example of this is the official Mariadb container.</li>
<li>A rootless container with the appplication running as root. Within the
container everything seems to run as root but on the host the processes
run as the user who started the container.</li>
<li>A rootless container with an application running as an unprivileged user.
The userid and groupid of the container user are mapped to host ids in a
range assigned to each user in <code>/etc/subuid</code> and <code>/etc/subgid</code>.</li>
</ol>
<p>This blog will focus on the last two use cases.</p>
<p>Contrary to docker, there is no systemwide daemon that spawns the containers
but processes are indepedent entities. The standard procedure is to start pods
and containers via systemd and where rootless containers are concerned, to use
"<code>systemctl --user</code>". To enable starting a rootless container at boot time, a
user specific systemd should be started. This is accomplished by issuing the
command "<code>loginctl --enable-linger username</code>" as root.</p>
<p>An important drawback of running rootless containers is that it is not
possible to create data volumes on its own block storage (via "<code>podman volume
create</code>") or create default private networks (note that there is a limited
option for the latter, see "<a href="#netavark-networking">Netavark networking</a>"
below).</p>
<h2 id="starting-containers-in-a-pod">Starting containers in a pod</h2>
<p>To enable inter-container communications one should run the containers
together in a pod. Network port forwarding from the host to containers can
happen at the pod level as well. </p>
<p>Imagine that you have a Wordpress and a Mediawiki environment, both running as
containers on the same host. For this you create two pods, one containing the
Mediawiki container and a Mariadb container and the other with a Wordpress and
a Mariadb container. In both cases, the database access is configured as TCP
port 3306 on localhost. Both web applications find their own database because
the pod contains the access.</p>
<p>Starting or stopping a pod starts or stops all containers assigned to the pod.
Creating a pod with containers happens like this:</p>
<pre>
podman pod create --name podname [ -p hostport:containerport ] ...
podman create --pod podname --name container1 [ -v hostdirectory:mountpont ] ... imagename[:tag] [args]
podman create --pod podname --name container2 [ -v hostdirectory:mountpont ] ... imagename[:tag] [args]
</pre>
<p>It is a good idea to generate systemd files for the pods and containers so
starting and stopping them can be delegated to systemd. That enables starting
the pod and containers at boot time. Generating the files is done with:</p>
<pre>
podman generate systemd --files --name --new podname
</pre>
<p>Creating the service file for the pod creates service files for the relevant
containers as well. The <code>--files</code> option creates service files i.o. dumping
them to STDOUT. <code>--name</code> creates files with image names and tags i.o. image
ids so you don't need to generate new files whenever a new version of the
container image is used. Only make sure that the image name and tag of the new
version is the same as that of the previous image. <code>--new</code> causes the podman
process to not only stop when <code>systemctl stop</code> is used but the podman process
is actually removed from the list (comparable to using the <code>podman rm -f</code>
command).</p>
<p>When the files are generated, they can be copied to <code>$HOME/.config/systemd/user</code>
and enabled with:</p>
<pre>
systemctl --user enable pod-podname container-container1 container-container2
</pre>
<h2 id="persistent-storage">Persistent storage</h2>
<p>In docker (and rootfull podman) it is possible to assign block storage
dedicated to docker or podman (with the "<code>docker volume create --opt
device=xxx</code>" command). In a rootless environment you cannot connect a device.
Instead, a directory is created in <code>.local</code>:</p>
<pre>
[poduser@uxhost ~]$ ls .local/share/containers/storage
defaultNetworkBackend networks overlay-images storage.lock
libpod overlayfs overlay-layers tmp
mounts overlay-containers secrets userns.lock
[poduser@uxhost ~]$ podman volume create MyVolume
MyVolume
[poduser@uxhost ~]$ ls .local/share/containers/storage/volumes/MyVolume
_data
[poduser@uxhost ~]$ podman run -it --rm -v MyVolume:/root/mydata alpine sh
/ # exit
[poduser@uxhost ~]$
</pre>
<p>It's also possible to mount another directory in a container. With that you can
decide yourself where the data is placed (given proper write access). Do note
that if SELinux restrictions are enforced, the file context for the mounted
directory should be labled as "<code>container_file_t</code>". You can delegate this to
podman by appending "<code>:z</code>" to the mount definition. You can also opt for
appending "<code>:Z</code>", then access to the mount is further restricted via sVirt
to only the relevant pod or container.</p>
<pre>
[poduser@uxhost ~]$ mkdir mariadb privateshare
[poduser@uxhost ~]$ ls -Zd mariadb privateshare
system_u:object_r:user_home_t:s0 mariadb system_u:object_r:user_home_t:s0 privateshare
[poduser@uxhost ~]$ podman run -d --rm --name mariadb -e MARIADB_RANDOM_ROOT_PASSWORD=yes \
-v ./mariadb:/var/lib/mysql:z -v ./privateshare:/privateshare:Z mariadb
b9a95df84be5f7feaf0b40ba2dd004eee4e4b83bd0fb2ad50692fab2a50bb06e
[poduser@uxhost ~]$ ls -Zd mariadb privateshare
system_u:object_r:container_file_t:s0 mariadb system_u:object_r:container_file_t:s0:c119,c552 privateshare
[poduser@uxhost ~]$ ps -Zp $(pgrep mariadb)
LABEL PID TTY TIME CMD
system_u:system_r:container_t:s0:c119,c552 15827 ? 00:00:01 mariadbd
[poduser@uxhost ~]$
</pre>
<p>Files that are generated as the root user or group within the container are
created as the user starting the container or her primary group on the host.
Files that are created as an unprivileged user or non-root group in the
container are mapped to a user or group within the <code>/etc/subuid</code> or
<code>/etc/subgid</code> range for the user starting the container. Also make sure that
the directory that is mounted in the container and its files are either owned
and belong to the primary group of the user starting the container, or of a
uid or gid withing that user's subuid/subgid range.</p>
<pre>
[poduser@uxhost ~]$ podman exec -it mariadb bash
root@b9a95df84be5:/# touch /privateshare/{rootfile,userfile}
root@b9a95df84be5:/# chown mysql:mysql /privateshare/userfile
root@b9a95df84be5:/# ls -l /privateshare
total 0
-rw-r--r--. 1 root root 0 Feb 20 14:38 rootfile
-rw-r--r--. 1 mysql mysql 0 Feb 20 14:38 userfile
root@b9a95df84be5:/# id mysql
uid=999(mysql) gid=999(mysql) groups=999(mysql)
root@b9a95df84be5:/#
exit
[poduser@uxhost ~]$ ls -l ./privateshare
total 0
-rw-r--r--. 1 poduser poduser 0 Feb 20 15:38 rootfile
-rw-r--r--. 1 494214 494214 0 Feb 20 15:38 userfile
[poduser@uxhost ~]$ grep poduser /etc/sub?id
/etc/subgid:poduser:493216:65536
/etc/subuid:poduser:493216:65536
[poduser@uxhost ~]$
</pre>
<p>Outside of the container the starting user does not have the right to modify
files that are not owned by root in the container (like the "<code>userfile</code>"
above). To enable this, the user has to temprary become "container root" with
the "<code>podman unshare</code>" command. Note that owners and groups are then mapped
according to <code>/etc/passwd</code> and <code>/etc/group</code> from the host, not those of the
container.</p>
<pre>
[poduser@uxhost ~]$ echo foo > ./privateshare/userfile
bash: permission denied: ./privateshare/userfile
[poduser@uxhost ~]$ podman unshare bash
[root@uxhost ~]# ls -l ./privateshare
total 0
-rw-r--r--. 1 root root 0 Feb 20 15:38 rootfile
-rw-r--r--. 1 systemd-coredump input 0 Feb 20 15:38 userfile
[root@uxhost ~]# id systemd-coredump
uid=999(systemd-coredump) gid=997(systemd-coredump) groups=997(systemd-coredump)
[root@uxhost ~]# echo foo > ./privateshare/userfile
[root@uxhost ~]# cat ./privateshare/userfile
foo
[root@uxhost ~]#
</pre>
<h2 id="rootless-nested-containers">Rootless nested containers</h2>
<p>Rootless podman containers use fuse-overlayfs by default to store container
images. This is centrally configured in <code>/etc/containers/storage.conf</code>. It has
a better performance than native VFS storage but the drawback is that you
can't run nested rootless containers (or e.g. <a href="https://k3d.io/">K3S Kubernetes</a>)
because <code>/dev/fuse</code> is unavailable. For such a use case it's better to use VFS
for container image storage but it's not possible to mix fuse-overlayfs and
VFS for different images by the same user. To see what storage is used (given
you have at least one container image present), just list the storage
directory:</p>
<pre>
[poduser@uxhost ~]$ ls .local/share/containers/storage/
libpod mounts overlay overlay-containers overlay-images overlay-layers storage.lock tmp userns.lock
[poduser@uxhost ~]$
</pre>
<p>As the directory names imply, overlayfs is used. To switch to VFS all
containers and container images need to be deleted first. That can be
accomplished with</p>
<pre>
podman system reset
</pre>
<p>To use VFS given this clean state, create the file
<code>$HOME/.config/containers/storage.conf</code> with the following content:</p>
<pre>
[storage]
driver = "vfs"
</pre>
<p>Now if you download or create an image it is stored as VFS:</p>
<pre>
[poduser@uxhost ~]$ ls .local/share/containers/storage/
libpod mounts storage.lock tmp userns.lock vfs vfs-containers vfs-images vfs-layers
[poduser@uxhost ~]$
</pre>
<h2 id="netavark-networking">Netavark networking</h2>
<p>With rooted containers it is possible to create your own network and give each
container its own IP-address in that range. There are use cases where that is
preferable (e.g. with a CI/CD pipeline where multiple similar containers are
spawn, each of which listens to the same network port). That is possible by
using the "Netavark" network backend. For this (assuming Enterprise Linux) you
install the <code>netavark</code> rpm. Installing that will also install the
<code>aardvark-dns</code> RPM. There are a few issues with using a rootless network:</p>
<ol>
<li>The network is not available to the host, only to containers in the
network.</li>
<li>Containers in the network are not limited to other containers in the same
pod but can connect to all other containers in the same network</li>
<li>Port publishing via the pod only works with CNI networks, so if a port of a
container in a netavark network needs to be puslished to the host, it
should be done in the container.</li>
<li>For every netavark network an aardvark-dns process is started so containers
can resolve other containers on the name. This only works for containers
within the same netavark network, so not for the host and not for
containers in other networks.</li>
</ol>
<p>Each podman user has the same standard network available named "<code>podman</code>".
This network has a "<code>cni</code>" backend which is used for forwarding
exposed container ports at the pod level as explained in
<a href="#starting-containers-in-a-pod">Starting containers in a pod</a>. To enable
networks with a "netavark" backend, the following configuration should be
added to <code>"$HOME/.config/containers/containers.conf"</code>:</p>
<pre>
[network]
network_backend = "netavark"
</pre>
<p>To use this, all existing containers should be stopped and removed. Creating a
netavark network is done like this:</p>
<pre>
[poduser@uxhost ~]$ podman network create --subnet 172.16.1.0/24 --gateway 172.16.1.1 rootlessnet
rootlessnet
[poduser@uxhost ~]$ podman network ls
NETWORK ID NAME DRIVER
2f259bab93aa podman bridge
8a0786c87d36 rootlessnet bridge
[poduser@uxhost ~]$
</pre>
<p>It is now possible to create containers in the new network and they can
connect to each other using the container name:</p>
<pre>
[poduser@uxhost ~]$ podman run -d --rm --network rootlessnet --name host10 alpine sleep 86400
e4c70309bc2b020772c873716174d38d18aaa51e7618e788dd18fcc0f21174cf
[poduser@uxhost ~]$ podman run -d --rm --network rootlessnet --name host11 alpine sleep 86400
4ab2be743a68b3e4a645e0545532a5de09e0564635a7057a8275b3bacecf152e
[poduser@uxhost ~]$ podman exec -it host10 ping -c2 host11
PING host11 (172.16.1.4): 56 data bytes
64 bytes from 172.16.1.4: seq=0 ttl=64 time=0.044 ms
64 bytes from 172.16.1.4: seq=1 ttl=64 time=0.114 ms
--- host11 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.044/0.079/0.114 ms
[poduser@uxhost ~]$
</pre>
<p>There's one drawback to this approach, i.e. that the <code>netavark</code> backend is now
hard wired and all new containers will use that whether it's preferred or not.
It's better to leave the backend configuration empty which enabled a <code>cni</code>
backend unless explicitely stated otherwise. This way it's still possible to
contain exposed ports within a pod and not have them available to all
containers in a network. To achieve this, use the following content for
<code>"$HOME/.config/containers/containers.conf"</code>:</p>
<pre>
[network]
network_backend = ""
</pre>
<p>It's now possible to combine containers with <code>cni</code> and <code>netavark</code> network
backends in the same pod:</p>
<pre>
[poduser@uxhost ~]$ podman pod create --name mixed -p 8888:80
ea6215f74bfbed0c5db456af187c041e0fd2ffce4b3b1673a832eaa30e838b9e
[poduser@uxhost ~]$ podman pod start mixed
ea6215f74bfbed0c5db456af187c041e0fd2ffce4b3b1673a832eaa30e838b9e
[poduser@uxhost ~]$ podman run -d --rm --pod mixed --name nginx nginx
804044254b846b26e281925ea8b1bba076167afe2203cfadd4bdc0ec021abe5b
[poduser@uxhost ~]$ podman run -d --rm --pod mixed --name alpcni curl sleep 86400
587afafd81b2d2e6ad49ef518cf1396f99e6ba661ac9bfc9df53a85e2386e66f
[poduser@uxhost ~]$ podman run -d --rm --pod mixed --name alpavark --network rootlessnet curl sleep 43200
bba06b34f0b22efb1db5ae33c93a60100e2a94d81bb372691812eef94cd24a56
[poduser@uxhost ~]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7e53fabf7e0a localhost/podman-pause:4.2.0-1673519486 About a minute ago Up About a minute ago 0.0.0.0:8888->80/tcp ea6215f74bfb-infra
804044254b84 localhost/nginx:latest nginx -g daemon o... About a minute ago Up About a minute ago 0.0.0.0:8888->80/tcp nginx
587afafd81b2 localhost/curl:latest sleep 86400 24 seconds ago Up 24 seconds ago 0.0.0.0:8888->80/tcp alpcni
bba06b34f0b2 localhost/curl:latest sleep 43200 4 seconds ago Up 4 seconds ago alpavark
</pre>
<p>The port forwarding in the list shows that the "nginx" and "alpcni" containers
use the cni network and the "alpavark" container doesn't. The host can connect
to the web server via the forwarded port and the Alpine container in the cni
network can connect to it via the exposed port (80) but the container in the
netavark network can't reach the exposed port, only the forwarded port on the
host:</p>
<pre>
[poduser@uxhost ~]$ curl -s http://localhost:8888 |grep title
<title>Welcome to nginx!</title>
[poduser@uxhost ~]$ podman exec -it alpcni curl -s http://localhost:80 |grep title
<title>Welcome to nginx!</title>
[poduser@uxhost ~]$ podman exec -it alpavark curl -s http://localhost:80 |grep title
[poduser@uxhost ~]$ podman exec -it alpavark curl -s http://localhost:8888 |grep title
[poduser@uxhost ~]$ ip a s enp1s0|grep ' inet '
inet 10.24.1.128/24 brd 10.24.1.255 scope global noprefixroute enp1s0
[poduser@uxhost ~]$ podman exec -it alpavark curl -s http://10.24.1.128:8888 |grep title
<title>Welcome to nginx!</title>
[poduser@uxhost ~]$
</pre>
<p>(BTW, the "<code>curl</code>" image is is just a standard Alpine image with the "curl"
tool added). In fact the advantage for a container with a netavark network to
be part of a pod is limited. About the only achievement is that the container
gets the same sVirt context as the pod and other containers in it while other
containers get their own sVirt context:</p>
<pre>
[poduser@uxhost ~]$ podman run -d --name nonpod --network rootlessnet curl sleep 22600
5a1d1d2ff4db7fb7e319f44b8516f9cbbb31263a6cd285e92acb4e9ec4921f3e
[poduser@uxhost ~]$ ps -Zu $(id -u)|grep :container_t:
system_u:system_r:container_t:s0:c125,c698 14196 ? 00:00:00 catatonit
system_u:system_r:container_t:s0:c125,c698 14227 ? 00:00:00 nginx
system_u:system_r:container_t:s0:c125,c698 14282 ? 00:00:00 sleep
system_u:system_r:container_t:s0:c125,c698 14391 ? 00:00:00 sleep
system_u:system_r:container_t:s0:c584,c600 14735 ? 00:00:00 sleep
[poduser@uxhost ~]$
</pre>
<p>With that the container can access mounts that are configured as private
storage with the "<code>:Z</code>" mount option by other containers in the same pod.</p>
<h2 id="podman-secrets">Podman Secrets</h2>
<p>Container secrets are available for rootless containers and work as usual. You
can create a secret with the "<code>podman secret create</code>" command and if you pass
that secret to the container with the <code>--secret</code> argument, the secret is
available in the <code>/run/secrets</code> directory of the container. This is a tmpfs
filesystem so even if you would use the <code>podman commit</code> command to create a new
image of the running container, the secret would not be part of the image.
Note that by default the "file" driver is used which stores the secret base64
encoded (i.e. unencrypted) on the host filesystem but if the container
application is capabale of using a password file, it is still better than
passing the password in an environment variable.</p>
<pre>
[poduser@uxhost ~]$ echo 'D33pS3cr37'| podman secret create blogeg -
666eb268d629db07752367da4
[poduser@uxhost ~]$ grep 666eb268d629db07752367da4 .local/share/containers/storage/secrets/filedriver/secretsdata.json
"666eb268d629db07752367da4": "RDMzcFMzY3IzNwo=",
[poduser@uxhost ~]$ echo RDMzcFMzY3IzNwo=|base64 -d
D33pS3cr37
[poduser@uxhost ~]$ podman run -it --rm --secret blogeg alpine sh
/ # cat /run/secrets/blogeg
D33pS3cr37
/ #
</pre>
<p>There are two more drivers available to podman secrets but they are still
undocumented. One is the "pass" driver which uses GPG in the same way as the
GNU pass command and the other is the "shell" driver where you define shell
scripts for the four options <code>list</code>, <code>lookup</code>, <code>store</code> and <code>delete</code>. These can
be specified in the <code>containers.conf</code> file or as <code>--driver.opts</code> flags during
secret creation time.</p>
<!-- I am here -->Don't staple the horse's battery (or: create better passwords)2023-03-09T00:00:00+01:002023-03-09T00:00:00+01:00Rob Wolframtag:None,2023-03-09:/passwords.html<h2 id="introduction">Introduction</h2>
<p>Many of you may have seen the famous XKCD cartoon regarding password
strength where Randall Munroe, the author, advocates creating a passphrase based
on four random words and remembering it by creating a sentence containing the
words. If you've already seen the cartoon, before actually clicking
<a href="https://xkcd.com/936/">the link</a>, let …</p><h2 id="introduction">Introduction</h2>
<p>Many of you may have seen the famous XKCD cartoon regarding password
strength where Randall Munroe, the author, advocates creating a passphrase based
on four random words and remembering it by creating a sentence containing the
words. If you've already seen the cartoon, before actually clicking
<a href="https://xkcd.com/936/">the link</a>, let me give you the words in alphabetical
order: battery, correct, horse, staple.</p>
<p>Do you remember the order of the words? As I write this I don't, even though I
do remember the words. That means I should try (at most) 24 permutations to get
the order right. Some systems block you at three wrong passwords! So no,
Munroe's system is not ideal. As long as
<a href="https://developers.google.com/identity/passkeys">FIDO/Passkey</a> or better yet,
<a href="https://www.grc.com/sqrl/sqrl.htm">SQRL</a> are not common place, we will be
condemned to using passwords for authentication, hopefully as one of multiple
factors. Apart from passwords that are retrieved from a password management
system (which should be long random strings that the password manager remembers
for you), a good password needs to comply to two requirements:</p>
<ol>
<li>it needs to be hard to find with a brute force attack (i.e., it needs enough
entropy even with <a href="https://en.wikipedia.org//wiki/Kerckhoffs's_principle">full knowledge of the password generating
algorithm</a>)</li>
<li>it needs to be relatively easy to remember.</li>
</ol>
<p>These two requirements tend to contradict each other. The more a password
complies to one requirement, the more the other suffers. The sweet spot is
generally somewhere in the middle. In that regard I appreciate Munroe's
suggestion. It is a lot better than some random mutation on an existing word
which IMO suffers on both requirements and it's a good method for sharing short
lived passwords, e.g. when mailing an encrypted file and passing the password in
a phone call (but then I tend to use six words). Still I think we can do a lot better.</p>
<h2 id="a-better-password-generating-algorithm">A better password generating algorithm</h2>
<p>The algorithm that I have used for decades is the following:</p>
<ol>
<li>Generate a (long enough) sentence that is easy to remember. You could e.g.
pick a sentence that references the system you are generating a password
for. So let's say you need a password for a mail service, then you could use
a sentence like:<br/>
"<em>Heidi Stettner's dog kept barking at the postman so the notification
program was named <a href="https://en.wikipedia.org/wiki/Biff_(Unix)#Origin_and_name">Biff</a></em>".</li>
<li>Now take the first letter of each word. The password then becomes
"<tt>HSdkbatpstnpwnB</tt>". Not bad, but let's modify it (deterministically).</li>
<li>Capitalize nouns and proper names (and the pronoun <tt>I</tt>) and keep
everything else lower case, numbers are entered as numbers (not relevant in
this case) and some words are converted to symbols (in this case we replace
"at" with the at-sign). The password then finally becomes:
"<tt>HSDkb@tPstNPwnB</tt>". </li>
</ol>
<p>The password looks pretty random but can be typed error free by just
remembering the sentence (a rough calculation of the entropy will be
done in the next paragraph). This method has enabled me to remember multiple
(currently more than 15) strong passwords at a time without the need to write
them down and type them in error free. I do write down a hint, especially
shortly after renewing a password. In the example above I could use a hint like
"Berkeley student" (which miss Stettner was at the time). The word to symbol
mapping that I use is in Table 1 below.</p>
<h2 id="estimating-the-password-strength">Estimating the password strength</h2>
<p>This is all fine, but it's not actually random so the question is: how
much do we deviate from true randomness? This depends on the frequencies of leading
characters in (English) words. To roughly estimate these frequencies, I wrote a
<a href="https://gitlab.com/hamal03/pwentropy/-/blob/master/entropy.pl">perl script</a>
and used it on a few books from Project Gutenberg (listed at the end). I also
used a <a href="https://greenopolis.com/list-of-nouns/">list of nouns</a> and a large list
of <a href="https://github.com/smashew/NameDatabases">British and American names</a> and
counted a word as a noun or name if it was on one of these lists. Of course
this is an estimation because the same word can be a noun in one context and
not in another (think e.g. of
"<em><a href="https://en.wikipedia.org/wiki/Time_flies_like_an_arrow%3B_fruit_flies_like_a_banana">time flies like an arrow</a></em>"
where "<em>flies</em>" can be a noun or a verb, depending on the semantics). By using
the symbols from step 3, the character set consists of 69 characters (<tt>a-z,
A-Z, 0-9, =, +, |, !, <, >, @</tt>). 69 characters yield 6.10 bits of entropy
per character in a uniform distribution, but my script maps each character to
its own entropy, based on the frequency of that charachter in the input text
where:<br/>entropy = <sup>2</sup>log(1/frequency)</p>
<p>The total entropy of the password is the sum of all these entropy values of the individual
characters (see
<a href="https://gitlab.com/hamal03/pwentropy/-/blob/master/frequencies.tsv">the script output</a>
for the frequencies and calculated entropy per character). To wrap your head
around the reasoning: suppose one character would be the first character of 50%
of all words, then that character should only count for 1 bit of entropy
because in each position of the password it would either be present or not with
equal probabilities). Using the script output, the "mail" password above is
calculated to have 84.1 bits of entropy, about 7.5 bits less than a 15
character full random password on the same character set. </p>
<p>To brute force a password with 84 bits of entropy which is stored as a simple (single iteration)
SHA256 hash, you would need on average 2<sup>83</sup> SHA256 calculations. To compare, one of the fastest
GPU's available, the NVidia GeForce RTX 4090, does SHA256 hashes at a rate of
<a href="https://gist.github.com/Chick3nman/32e662a5bb63bc4f51b847bb422222fd">21975.5 Megahashes per second</a>.
Brute forcing such a password with the RTX 4090 would take around 13.8 million years.
Even using the fastest Bitcoin miner, the
<a href="https://www.coinwarz.com/mining/hardware">AntMiner S19 XP</a> which can do 140
Terahashes per second, brute forcing would take nearly 2200 years on average.
Note that it's a common practice to use many hash iterations on the
password with the PBKDF2 key derivation or the iterations in
the Unix crypt method (which defaults to 5000 iterations on a SHA2 based
password). Using those methods scales the brute forcing time linearly. There are even
better key derivation algorithms available (like the memory hard SCrypt and Argon2) but
I think the point is made: even with a single SHA256 hash, such a password
is secure against any reasonable brute forcing attack while it can
still be easily remembered.</p>
<h2 id="estimation-short-cuts">Estimation short cuts</h2>
<p>Like I said before, the script does a rough estimation and cuts a few corners
which might have an effect on the frequencies but I doubt the effect would
measure to anything meaningful. First, if a "word" in one of the script inputs
was not recognized as an English word or string of digits, it was not counted.
This amounted to around 7.1% of my input (think of a "word" like "soon—he" in
"likely to vacate it soon—he might have got" in <em>Sense and Sensibility</em>).</p>
<p>Another cut corner is that I did not take likely or unlikely
character sequences into account. These are based on the likeliness of the
sequence of the underlying words. Other words with the same starting characters
may have a different likeliness for their sequence. An example that comes to mind
is that the word "not" (which is mapped to <tt>"!"</tt>) likely follows a word
like "is", "have/has" or "will".<br/>
Now let's assume all cut corners would amount
to 2 bits fewer on the example password above, which is way more than I can
possibly imagine, then divide the 2200 years and 13.8 million years by
four. That would still be more than strong enough in my opinion, definately a
lot stronger than "correct horse battery staple" and a lot easier to remember.</p>
<p>I hope this system benefits you to be a little safer on the internet, but where
possible, use multi-factor authentication and use a password manager to
generate random and different passwords for various services and reserve this
method for those passwords that cannot or should not be included in the password
manager (like the master password for the manager itself).</p>
<h4 id="table-1-word-to-symbol-mapping">Table 1: word to symbol mapping</h4>
<table class="table-striped table">
<th>
<td>Word</td><td>Symb</td>
<td>Word</td><td>Symb</td>
<td>Word</td><td>Symb</td>
</th><tr><td> </td>
<td>a/an</td><td><tt>1</tt></td>
<td>to</td><td><tt>2</tt></td>
<td>for</td><td><tt>4</tt></td>
</tr><tr><td> </td>
<td>and</td><td><tt>+</tt></td>
<td>or</td><td><tt>|</tt></td>
<td>not</td><td><tt>!</tt></td>
</tr><tr><td> </td>
<td>in</td><td><tt><</tt></td>
<td>out</td><td><tt>></tt></td>
<td>at</td><td><tt>@</tt></td>
</tr><tr><td> </td>
<td>more</td><td><tt>></tt></td>
<td>less</td><td><tt><</tt></td>
<td>is</td><td><tt>=</tt></td>
</tr><tr><td> </td>
<td>through</td><td><tt>|</tt></td>
<td> </td><td><tt> </tt></td>
<td> </td><td><tt> </tt></td>
</tr></table>
<p>Note: "numeric words" like one, two and ten are written as their numeric values.</p>
<h4 id="inputs-used-for-the-script">Inputs used for the script</h4>
<p>The following books were retrieved from <a href="https://www.gutenberg.org/">Project Gutenberg</a>
in text format and fed to the perl script to estimate the English leading character frequencies.</p>
<ul>
<li>Alice in Wonderland</li>
<li>A Room With A View</li>
<li>Dracula</li>
<li>Little Women</li>
<li>Moby Dick</li>
<li>Sense and Sensibility</li>
<li>The Enchanted April</li>
</ul>Cryptographic signatures in Bitcoin2022-02-22T00:00:00+01:002022-02-22T00:00:00+01:00Rob Wolframtag:None,2022-02-22:/ecc4btc.html<!--HeaderImage: images/bitcoin-intro.jpg-->
<h2 id="introduction">Introduction</h2>
<p>This blog describes at a technical level how Ellyptic Curve Cryptography (ECC) is used
for signing Bitcoin transactions. <strong>Caveat emptor</strong>: I am not a mathematician let
alone a cryptographer. I created this document mainly for helping myself
understand how ECC is applied and welcome any correction on errors present …</p><!--HeaderImage: images/bitcoin-intro.jpg-->
<h2 id="introduction">Introduction</h2>
<p>This blog describes at a technical level how Ellyptic Curve Cryptography (ECC) is used
for signing Bitcoin transactions. <strong>Caveat emptor</strong>: I am not a mathematician let
alone a cryptographer. I created this document mainly for helping myself
understand how ECC is applied and welcome any correction on errors present in this
blog. Other criticism is welcomed as well. Please direct feedback to me via
email or Twitter. This site is a static site and I haven't yet bothered to install a
proper (stateless) commenting system.</p>
<h2 id="discrete-logarithms">Discrete logarithms</h2>
<p>ECC, at least as applied to signatures in Bitcoin, is akin to the Discrete
Logarithm (DL) problem, so I'll briefly introduce that. The idea is that it is
easy to calculate a base number to the power of an exponent
<a href="https://en.wikipedia.org/wiki/Modular_arithmetic">modulo</a> some prime, but given
the result it is extremely hard to calculate what number was used as the exponent.
As an example:<br/>
\( 26^{41} \pmod{127} = 9 \)<br/>
but given 26, 127 and 9, the only way to find the value 41 is using brute force.
Here the number 41 is the "private key" and the combination of 26, 127 and 9 is
the "public key".
This is an example with small numbers but in reality the prime, exponent and
base are in the order of a 617-digit number or more. The numbers should be this
large because there is a slightly better algorithm than brute force to find the
exponent, called "Number field sieve".</p>
<h2 id="ellyptic-curve-operations">Ellyptic curve operations</h2>
<p>In ECC, operations are done with points on an
<a href="https://en.wikipedia.org/wiki/Elliptic_curve">ellyptic curve</a>. The curve that
is used in Bitcoin (as well as the standard curve that is used for TLS) is
called a "Weierstrass curve" and in short form it has an equation like
\( y^2 = x^3 + ax +b \) where \( a\) and \( b\) are some constant. The curve used in
Bitcoin has a value for \( a\) of 0 and a value for
\( b\) of 7, so the curve is \( y^2 = x^3 + 7 \). The images in this blog
are based on that curve.</p>
<p>There are other curve types used in cryptography, like "Edwards curves" and
"Montgomery curves" but their discussion is out of the scope of this article.
The famous "curve 25519" from Dan Bernstein that is used quite a lot these days
is an example of a Montgomery curve.</p>
<p>In Weierstrass curves, "adding" two points means drawing a line through the two
points, finding a third point where this line crosses the curve and reflecting
that point on the X axis (see image). The reflecting is the ECC way of getting
the "negative" point, so in the image where the line crosses the curve is
"\( -(P+Q)\)" and the reflection is "\(P+Q\)". This rather strange operation has the
effect that some standard group operations apply. So \( A+B = B+A\) and
\( A+(B+C) = (A+B)+C\). </p>
<!--Insert image 1.-->
<div class="largeImage">
<img alt="Adding two curve points" class="img-fluid" src="/images/secp256k1-p+q.png"/>
</div>
<p>Doubling a point means taking the tangent line to the curve in that point,
finding the cross point and reflecting that. Multiplying a point with a value
means repeatedly adding the point to itself (see next image where we multiply the
point \( P\) with the value \( 3\) by first doubling \(P\) and adding the result
to \(P\).</p>
<!--Insert image 2.-->
<p></p><div class="largeImage">
<img alt="Doubling a curve point" class="img-fluid" src="/images/secp256k1-2p.png"/>
</div>
<h2 id="ellyptic-curve-operations-modulo-primes">Ellyptic curve operations modulo primes</h2>
<p>To do ECC, we take the rational values on the curve (i.e., the curve points
where each coordinate can be expressed as a division of two
integers) and calculate those values modulo some prime \( p\). About the
only thing that is still recognizable is the vertical symmetry, which makes
sense. If point \(A\) is a rational point on the curve, then so point is \(-A\).</p>
<p>Given the rational points modulo \(p\), one or more groups emerge such that
using any point in that group, multiplying that point by the integers from 1 to
the group size yields all points in the group. In the image below we have the
rational points for \( y^2=x^3+7\) modulo 127. The group size is 126 so any of
these points can be used as a "generator" for the group.</p>
<!--Insert image 3.-->
<p></p><div class="largeImage">
<img alt="Bitcoin curve mod 127" class="img-fluid" src="/images/mod127.png"/>
</div>
<p>Even though the dots do not seem to resemble the curve we saw above, the
mechanism for adding points still works (as does point doubling but there the
slope of the line is not as obvious). In the image below we add the points
\((38,53)\) and \((67,65)\) to get the result, i.e. \((19,95)\).</p>
<!--Insert image 3.-->
<p></p><div class="largeImage">
<img alt="Adding curve points mos 127" class="img-fluid" src="/images/mod127-p+q.png"/>
</div>
<p>Note that in integer operations modulo a prime \( p\), calculating $\frac{a}{b}$ means
finding a value \(c\) such that \( b \times c = 1 \pmod{p}\) (using the
<a href="https://en.wikipedia.org/wiki/Extended_Euclidean_algorithm">Extended Euclidean Algorithm</a>)
and then calculating $a\times c$. </p>
<p>To add two points \( (r_1, r_2)\) and \( (s_1, s_2)\), you calculate:</p>
<p>\begin{aligned}
\lambda &= ( \frac{s_2-r_2}{s_1-r_1} ) \pmod{p} \\
x &= ( {\lambda}^2 - r_1 - r_2 ) \pmod{p} \\
y &= ( \lambda (r_1 - x) - r_2 ) \pmod{p}
\end{aligned}</p>
<p>To double a point \( (r_1, r_2)\) you calculate:</p>
<p>\begin{aligned}
\lambda &= ( \frac{3{r_1}^2+a}{2r_2} ) \pmod{p} \\
x &= ( {\lambda}^2 - 2r_1 ) \pmod{p} \\
y &= ( \lambda (r_1-x)-r_2 )\pmod{p} \\
\end{aligned}</p>
<p>The value \(a\) is the from the curve formula \(y^2=x^3+ax+b\). In our case
it has the value 0.</p>
<p>The discrete log problem can be translated as, given an integer \(q\), a
generator point \(G\) and a prime modulus \(p\), it is easy to calculate the
curve point \(Q\) such that \(Q = q\cdot G \pmod{p}\), but given \(Q\), \(G\)
and \(p\) is it infeasible to calculate \(q\). </p>
<p>After more than three decades of crypt-analysis, no equivalent of the "number
field sieve" has been found for ECC, so the numbers can be a lot smaller. A
prime of 256 bits (77 digits) is very common and considered secure with classic
computing. </p>
<h2 id="dsa-and-ecdsa">DSA and ECDSA</h2>
<p>Note: when describing the various signature algorithms I'll use the addition and
multiplication representation as used in ECC instead of the multiplication and
exponentiation system as used in discrete logarithms. For getting the point
across it should not matter.</p>
<p>In 1989 Claus Schnorr developed an efficient algorithm to create digital
signatures based on the discrete log problem. He patented the algorithm (which
expired in 2008) so the NSA developed another (less efficient) algorithm which
became the de facto standard, named DSA. ECSDA is the equivalent of DSA using
curve points, which we describe here.</p>
<p>The following notation is used from here on:</p>
<ul>
<li>\( G \):$\phantom{A}$ Generator point on the curve</li>
<li>\( a \):$\phantom{A}$ (lowercase character): integer</li>
<li>\( A \):$\phantom{A}$ (uppercase character): curve point</li>
<li>\( \mathtt{H}() \):$\phantom{A}$ hash function</li>
<li>\( \mathbf{M} \):$\phantom{A}$ signed message</li>
<li>\( \Vert{} \):$\phantom{A}$ concatenation</li>
</ul>
<h3 id="signature-creation">Signature creation</h3>
<p>\begin{aligned}
x &\leftarrow \{1..n\}\phantom{A}\text{(integer, private key, n is group size)} \\
X &= x \cdot G \phantom{A}\text{(public key)} \\
r &\leftarrow \{1..n\}\phantom{A}\text{(nonce, random integer)} \\
(k,y) &= r \cdot G \phantom{A}\text{(use only the x coordinate)}\\
z &= \mathtt{H}(\mathbf{M}) \\
s &= \frac{z+kx}{r} \\
\end{aligned}</p>
<p>Signature: \( (k,s) \)</p>
<h3 id="signature-verification">Signature verification</h3>
<p>\begin{aligned}
&\frac{z}{s} \cdot G + \frac{k}{s} \cdot X \Leftrightarrow \\
&\frac{z+kx}{s} \cdot G \Leftrightarrow \\
&\frac{(z+kx)\cdot r}{z+kx} \cdot G \Leftrightarrow \\
&r \cdot G \\
&(k_v, y) = r \cdot G
\end{aligned}</p>
<p>Verifies if \(k_v\) is equal to \(k\)</p>
<p>It is important that the nonce is unique for each signature. If it is known that
the same nonce is reused for multiple messages, the private key is exposed like
this:</p>
<p>\begin{aligned}
&s = \frac{z+kx}{r} \\
&s' = \frac{z'+kx}{r} \\
&\frac{sz'-s'z}{k(s'-s)} \Leftrightarrow \\
&\frac{r(zz'+kxz'-z'z - kxz)}{r(kz'-kz)} \Leftrightarrow \\
&\frac{x(kz'-kz)}{kz'-kz} \Leftrightarrow x
\end{aligned}</p>
<p>To tackle this, <a href="https://datatracker.ietf.org/doc/html/rfc6979">RFC6979</a> was
created to create a deterministic nonce which dependends on the message \(\mathbf{M}\).</p>
<p>A second problem with ECDSA is malleability. Because only the X coordinate of
the point \(r\cdot G\) is used, the negative value of the signature is also a
valid signature. Before the introduction of the
<a href="https://en.wikipedia.org/wiki/SegWit">SegWit</a> protocol change in Bitcoin, the
signature was used as part of the transaction id. By modifying the signature
this way a different transaction id can be generated by someone without the private
key. Remember that the negative value of a curve point is its reflection on the
X axis. The malleability is pretty obvious. If you do the verification steps
with \((k,-s)\) you end up with the curve point \(-r\cdot G\) which is the
same as \(-(r\cdot G)\) which is the reflection of the original point on the X
axis, thus having the same X coordinate (\(k\)).</p>
<h2 id="schnorr-signatures">Schnorr signatures</h2>
<p>With the activation of <a href="https://coincodecap.com/bitcoin-taproot">Taproot</a>
in 2021, a few new mechanisms were introduced in the Bitcoin consensus protocol,
one of which is Schnorr signatures. This is the original signature algorithm by
Claus Schnorr which is now patent free. The algorithm is quite a bit simpler than ECDSA:</p>
<h3 id="signature-creation_1">Signature creation</h3>
<p>\begin{aligned}
x &\leftarrow \{1..n\}\phantom{A}\text{(integer, private key, n is group size)} \\
X &= x \cdot G \phantom{A}\text{(public key)} \\
r &\leftarrow \{1..n\}\phantom{A}\text{(nonce, random integer)} \\
R &= r\cdot G \\
q &= \mathtt{H}(\mathbf{M}\Vert{}R) \\
s &= r - qx
\end{aligned}</p>
<p>Signature: \( (q,s) \)</p>
<h3 id="signature-verification_1">Signature verification</h3>
<p>\begin{aligned}
R_v &= s \cdot G + q \cdot X \\
&= r \cdot G - qx \cdot G + qx \cdot G \\
&= r\cdot G \\
q_v &= \mathtt{H}(\mathbf{M}\Vert{}R_v)
\end{aligned}</p>
<p>Verifies if $q_v$ is equal to \(q \)</p>
<p>One advantage of Schnorr signatures is it's linearity. You can tweak the public
key and signature with an integer value and the signature still validates:</p>
<p>\begin{aligned}
X'&=n\cdot X \\
s'&=r-qnx\\
R_v &= s'\cdot G + q\cdot X'\\
&= r\cdot G -qnx\cdot G+qnx\cdot G
\end{aligned}</p>
<p>This is used by Taproot where alternative spending conditions are encoded in a
<a href="https://rubin.io/public/pdfs/858report.pdf">Merkalized Abstract Syntax Tree</a>
and the Merkle Root hash is used as a tweak value on the primary spending key.</p>
<p>As with ECSDA, the nonce should be unique for every signature, because reusing
the nonce reveals the private key:</p>
<p>\[\frac{s'-s}{q-q'} = \frac{r-q'x -r+qx}{q-q'} = \frac{x(q-q')}{q-q'}\]</p>
<h2 id="musig">MuSig</h2>
<p>The linearity of Schnorr signatures enables adding public keys and singatures
together so that a n-of-n multisig can be represented as a single signature on a
single public key. Simply adding the public keys together poses a risk though. A
bad actor could wait for all other public keys to be posted and subtract them
from his own public key, adding this modified key to the mix. The result is that the
"aggregate" key will be only his own public key so he has sole control of
signing a transaction on what was assumed to be a n-of-n multisig. To thwart this,
Greg Maxwell et al developed MuSig which requires some collaboration.</p>
<h3 id="signature-creation_2">Signature creation</h3>
<p>\begin{aligned}
x_i &\leftarrow \{1..n\}\phantom{A}\text{(integer, private key for user i)} \\
X_i& = x_i \cdot G \phantom{A}\text{(public key for user i)} \\
l &= \mathtt{H}(\sum_i{X_i}) \phantom{A}\text{(the hash of all public keys added
together)} \\
t_i &= \mathtt{H}(l\Vert{}X_i) \\
X &= \sum_i{(t_i\cdot X_i)} \\
&= (\sum_i{(t_i{}x_i)})\cdot G \\
r_i &\leftarrow \{1..n\}\phantom{A}\text{(nonce, random integer)} \\
R_i &= r_i \cdot G \\
R &= \sum_i{R_i} \\
&= (\sum_i{r_i})\cdot G \\
q &= \mathtt{H}(X\Vert{}R\Vert{}\mathbf{M}) \\
s_i &= r_i + q{}x_i{}t_i \\
s &= \sum_i{s_i} \\
&= \sum_i{r_i}+q\cdot\sum_i{(x_i{}t_i)} \\
\end{aligned}</p>
<p>Signature: \( (q,s) \)</p>
<h3 id="signature-verification_2">Signature verification</h3>
<p>\begin{aligned}
R_v &= s \cdot G - q\cdot X \\
q_v &= \mathtt{H}(X\Vert{}R_v\Vert{}\mathbf{M}) \\
\end{aligned}</p>
<p>Verifies if \(q_v\) is equal to \(q\)</p>
<p>MuSig in itself is not part of the Bitcoin consensus protocol, apart from how
\(q\) is calculated, i.e., the hash includes the (possibly aggregate) public
key. In the original Schnorr signature this was not included. From the
verifier's perspective, the signature verification is the same for a simple
Schnorr signature or a MuSig signature, which is exactly the purpose of MuSig.</p>
<p>The fact that it isn't part of the consensus protocol means that other signature
protocols can be used, as long as the verification algorithm is identical to
the one above. Two such algorithms have already been proposed to tackle the fact
that MuSig requires three interactive rounds of communication between
participants and that the deterministic nonces as described in RFC6979 cannot be
applied safely because due to the multi-round nature of the protocol a
participant can be tricked into signing different messages with the same nonce
via a replay attack (where part of the interactive communication is redone).
The first of these protocols tackles the deterministic nonce and is named
<a href="https://medium.com/blockstream/musig-dn-schnorr-multisignatures-with-verifiably-deterministic-nonces-27424b5df9d6">MuSig-DN</a>.
It uses a special hash algorithm called Purify on the message, all
participating public keys and a special nonce key to generate both the nonce and a
<a href="https://en.wikipedia.org/wiki/Zero-knowledge_proof">zero knowledge proof</a> to
prevent the nonce to be changed between rounds. The number of rounds is reduced
to two but the protocol is still interactive.
The second protocol is called
<a href="https://medium.com/blockstream/musig2-simple-two-round-schnorr-multisignatures-bf9582e99295">MuSig2</a>
which reduces the number of rounds to two, one of which can be precomputed and
with that, create a non-interactive MuSig protocol.</p>
<p></p>Running djbdns in a podman container2022-01-08T00:00:00+01:002022-01-08T00:00:00+01:00Rob Wolframtag:None,2022-01-08:/djbdns-container.html<!--HeaderImage: images/bitcoin-intro.jpg-->
<p>I have been using <a href="https://cr.yp.to/djbdns.html"><tt>djbdns</tt></a> for several
years now and I still love it. Tiny, secure, maybe not as beginner friendly
but I don't mind that. I previously used an RPM file that I first build on
Enterprise Linux 6 which cleanly build on EL7 as well, but EL8 is …</p><!--HeaderImage: images/bitcoin-intro.jpg-->
<p>I have been using <a href="https://cr.yp.to/djbdns.html"><tt>djbdns</tt></a> for several
years now and I still love it. Tiny, secure, maybe not as beginner friendly
but I don't mind that. I previously used an RPM file that I first build on
Enterprise Linux 6 which cleanly build on EL7 as well, but EL8 is another
issue. Now the end of life for EL7 is still two years out when writing this
blog, I was still interested in a more long term solution. Enter containers.
As a user of EL for servers, both professionally and privately, I switched to
<a href="https://podman.io/"><tt>podman</tt></a> i.o. <tt>Docker</tt>, just because the
daemonless nature of podman yield a more secure system and it's even possible
to run the containers rootless, but in this case that comes at a price.</p>
<h3 id="rootfull-vs-rootless-containers">Rootfull vs rootless containers</h3>
<p>Wherever I use containers I try to use them rootless, but in certain scenarios
it is an issue with running a DNS. Most importantly, running a djbdns
container rootlessly means that it is not possible to distinguish the source
of DNS queries. That means that you cannot use a <a href="https://en.wikipedia.org/wiki/Split-horizon_DNS">split
horizon</a> or limit AXFR access
to specific IP addresses. The logging will also show a source query address in
the network namespace of the container (usually 10.0.2.0/24). If neither are
required, running a container with tinydns and axfrdns rootlessly is a good
idea.</p>
<p>Another issue with a rootless container is the fact that dnscache cannot
forward queries to another rootless container with tinydns on the same host
since they both listen on port 53 and there is no way to distinguish between
the networks.</p>
<p>If tinydns and axfrdns are run in a rootless container, both TCP and UDP port
53 must be published on some high host port. Via a NAT firewall rule a query
to port 53 of the host can be redirected to the published port.</p>
<p>When using a rootfull container, the container can be defined with its own IP
address within a podman network and in this case port 53 does not need to be
published on a host port, port 53 on the container IP is reachable and a DNAT
firewall rule can be used to send the DNS query to the container. If you use a
second rootfull container with dnscache the DNS queries that should be handled
by tinydns or axfrdns can be forwarded to the IP addres of the tinydns
container.</p>
<h3 id="creating-the-container">Creating the container.</h3>
<p>If I run a container, I want it small. I started with the
<a href="https://gist.github.com/yohgaki/d6943d53a5e3ef6841ef660bd9cbfa90">Dockerfile</a>
that was published by Yasuo Ohgaki to install djbdns, ucspi-tcp and daemontools
on Alpine Linux. For this, gcc etc had to be installed to compile the code,
but I want it two changes on that: first I wanted to apply some patches and
second, I wanted only the compiled result, not the source nor gcc, perl and
the like. To achieve that I modified the Dockerfile to a multi stage version
where first a container is used to compile the data and then a second container
is created where the compiled result is copied to.</p>
<p>The containerfile downloads the software and some patches from the original
websites, so to verify that the correct files are actually downloaded I check
the SHA256 checksums of the download.</p>
<p>For <a href="https://cr.yp.to/daemontools.html">daemontools 0.76</a> I applied a single
patch to replace "<code>extern int errno;</code>" by "<code>#include <errno.h></code>".
This is needed to be able to compile the software with
gcc (this goes for ucspi-tcp and djbdns as well). After compiling and
installing, the various binaries stay in the source tree and there were symlinks
from the "<code>/command</code>" directory to the compiled binaries in the source
tree and there were symlinks from <code>/usr/local/bin</code> to the symlinks in the
<code>/command</code> directory. I deleted the symlinks from <code>/usr/local/bin</code>
and did a "<code>cp -L</code>" from <code>/command</code> directory to
<code>/usr/local/bin</code>. </p>
<p>On <a href="http://cr.yp.to/ucspi-tcp.html">ucspi-tcp 0.88</a> I applied
"<a href="https://www.fefe.de/">Fefe's</a>" <a href="https://www.fefe.de/ucspi/">diff20 patch</a> to
enable IPv6 support on <code>tcpserver</code> and <code>tcpclient</code>.</p>
<p>On <a href="https://cr.yp.to/djbdns.html">djbdns 1.05</a> I applied quite a few more
patches. First, I applied <a href="https://www.fefe.de/dns/">Fefe's (test28) patch</a> to
enable IPv6 for tinydns. Second, I installed <a href="http://www.tinydnssec.org/">Peter Conrad's DNSSEC
patch</a> for tinydns (version 1.8) which requires
Fefe's patch. I have yet to enable DNSSEC. A third patch that I applied is
<a href="https://github.com/balena/djbdns-srv-naptr">Guilherme Balena Versiani's NAPTR</a>
extension to Michael Handler's SRV patch which allows easy creation of both
resource records. Since Guilherme published a patched version of djbdns-1.05,
I re-calculated the patch and applied it to my patched source. This yielded a
few rejected hunks that were easily fixable. I also added a second parameter
to his "<code>rr_finish()</code>" statements because lacking those generated
compile errors. The final patch against my Fefe + DNSSEC version is a collection
of various patches that I collected. I applied them all manually to what I had
until then and used <code>git diff</code> to generate a big patch with all of them. The
patches are named in the top of this collective patch file, but for most of them
I can no longer find the original online.</p>
<p>The modified version of Guilherme's patch and the patch collective, along with
two scripts and the containerfile are available via
<a href="https://gitlab.com/snippets/2232485">this Gitlab snippet</a>.</p>
<p>After compiling all these I copied the content of <code>/usr/local/bin</code> to the
destination image. I added two scripts to this directory: a script to start the
required djbdns daemon (<code>startdjb.sh</code>) and a second one to build the djbdns
databases from within the container (<code>makeall.sh</code>).</p>
<p>Recreating the image yourself is trivial. Clone the snippet repository and build
the image from within the cloned directory like so:</p>
<pre>
git clone https://gitlab.com/snippets/2232485.git
cd 2232485
</pre>
<p>and then either</p>
<pre>
docker build -f Containerfile -t yourtagname .
</pre>
<p>or</p>
<pre>
buildah bud -f Containerfile -t yourtagname
</pre>
<p>This should yield an 8MB container. If you want to try a ready-made image, a
saved image is available via <a href="https://blobs.hamal.nl/alpine-djbdns-latest.tar">this link</a>.
The image can be recreated via <code>podman load < tarfile</code> or
<code>docker load < tarfile</code>.</p>
<h3 id="using-the-containers">Using the containers</h3>
<p>The containers expose TCP and UDP ports 53 and <code>/usr/local/etc</code> should
be mapped to a directory on the host. If you use podman on a system with
SELinux enabled (I hope you do), make sure the directory that is mapped to the
container's <code>/usr/local/etc</code> has a writable content type (I use
<code>container_file_t</code>). </p>
<p>Via the <code>DJBMODE</code> environment variable, you can define which djbdns daemon
should run in the container. If no value is provided, the value <code>tinydns</code> is
assumed which starts both tinydns and axfrdns. For the dnscache, walldns or
rbldns daemons, you should use the values dnscache, walldns or rbldns
respectively.</p>
<p>To run the container rootlessly, use a command line like:</p>
<pre>podman create --rm --name djbdns -p 5553:53/tcp -p 5553:53/udp \
-v /var/podman:/usr/local/etc -e DJBMODE=tinydns alpine-djbdns</pre>
<p>On the first run, the <code>/var/podman</code> will be populated with a
<code>tinydns</code> and <code>axfrdns</code> tree. Make sure that the administrator
redirects TCP and UDP queries to the port(s) as specified.</p>
<p>If you run a rootfull container, you can assign an IP address to the
container. By default, podman installs a network named "<code>podman</code>" with an IP
network of <code>10.88.0.0/16</code>. To find out, you can use </p>
<pre>podman network ls</pre>
<p>and </p>
<pre>podman network inspect networkname</pre>
<p>to see which IP space is available. To create an own network, you use
something like</p>
<pre>podman network create --subnet 172.25.1.0/24 --gateway 172.25.1.1 djbnet</pre>
<p>Creating the container with an IP address goes something like this:</p>
<pre>podman create --rm --ip 172.25.1.10 --network djbnet --name djbdns \
-v /var/podman:/usr/local/etc -e DJBMODE=tinydns alpine-djbdns</pre>
<p>If you also need dnscache you can create the container similarly but with a
different IP address in the same network. The exposed ports do not need to be
published on a host port, port 53 is available on the container's IP address.
To reach the container, the routing tables need to know how to reach the
container, or the host running the container needs DNAT rules when the server
is queried on the DNS port (don't forget to set the DJBMODE variable to dnscache
for that to work).</p>
<p>It is advisable to generate systemd files with</p>
<pre>podman generate systemd --files --new --name containername</pre>
<p>With this systemd can be used to start, stop and monitor the containers.</p>
<p>With rootfull containers, it is possible to create forward files for dnscache to be
handled by tinydns or axfrdns. If, e.g., the tinydns container listens on IP
address <code>172.25.1.10</code> and serves DNS records for <code>example.com</code>,
you can create the file <code>*volumemount*/dnscache/root/servers/example.com</code>
with <code>172.25.1.10</code> as content.</p>
<p>After first running the tinydns container, the tinydns and axfrdns databases
are still missing. They can be created externally and copied to
<code>*volumemount*/tinydns/root/data.cdb</code> and <code>*volumemount*/axfrdns/tcp.dns</code>
respectively, or the source files <code>data</code> and <code>tcp</code> can be edited
after which you run "<code>makeall.sh</code>" in the running container with</p>
<pre>podman exec -it containername /usr/local/bin/makeall.sh</pre>
<h3 id="references">References</h3>
<ul>
<li><a href="https://cr.yp.to/djbdns.html">The djbdns home page</a></li>
<li><a href="https://cr.yp.to/djbdns/tinydns-data.html">The tinydns data format</a></li>
<li><a href="https://gitlab.com/snippets/2232485">Gitlab snippet</a></li>
<li><a href="https://blobs.hamal.nl/alpine-djbdns-latest.tar">Downloadable container image</a></li>
</ul>BTCStick Howto2019-11-26T00:00:00+01:002019-11-26T00:00:00+01:00Rob Wolframtag:None,2019-11-26:/btcstick.html<!--HeaderImage: images/bitcoin-intro.jpg-->
<h2 id="introduction">Introduction</h2>
<p>This manual describes the content and usage of the “BTCStick” USB pen drive.
This is a bootable USB disk image that contains various bitcoin related tools
to use offline. The system was developed for the author’s personal use case to
safely run such tools. The stick was inspired …</p><!--HeaderImage: images/bitcoin-intro.jpg-->
<h2 id="introduction">Introduction</h2>
<p>This manual describes the content and usage of the “BTCStick” USB pen drive.
This is a bootable USB disk image that contains various bitcoin related tools
to use offline. The system was developed for the author’s personal use case to
safely run such tools. The stick was inspired by the
<a href="https://bitkey.io">Bitkey project</a> which provides a similar environment but
the approach differs somewhat.
A zip file containing an image which can be written to a USB thumb drive is
provided via <a href="https://ham.al/btcstick">Google Drive</a>, accompanied by a PGP
signed file to verify the consistency of the file. The content of the system is
based on personal preferences of the author but may be modified to suit your
own needs. Ideas to improve the system are always welcome via mail at
<a href="mailto:btcstick@hamal.nl">btcstick@hamal.nl</a>.</p>
<div class="largeImage">
<img alt="BTCStick Desktop" class="img-fluid" src="/images/btcstick_desktop.png"/>
</div>
<h3 id="base-system">Base system</h3>
<p>The pen drive is a standard installation of
<a href="http://releases.ubuntu.com/18.04/">Ubuntu Linux (18.04 a.k.a. Bionic Beaver)</a>
with the <a href="https://mate-desktop.org/">Mate desktop environment</a> and can be
booted via UEFI (including on systems with the secure boot option enabled) or
via Legacy BIOS. After a standard boot, the system will lack any network device
and boot with a “<em>tmpfs overlayroot</em>” (i.e., all filesystem writes except for
writes on extra mounted file systems are volatile). This can be overridden by
choosing a different boot option from the GRUB boot menu.</p>
<p>The system has a couple of shell scripts that enable creating and using a file
on an auto-mounted filesystem (e.g. a second USB stick) that contains an
encrypted filesystem. This filesystem is encrypted using a password that is
generated from user input and the serial number of the USB device it resides
on, thus upping the ante for decrypting a copy of the file (see the
“pseudo-2FA” chapter below). The pen drive should fit on a standard 8GB USB stick.</p>
<h2 id="creating-the-bootable-usb-device">Creating the bootable USB device</h2>
<p>The system image should be written to the raw USB device. If you’re using a
version of Windows to write the image to a USB stick, you will need a special
tool like <a href="https://rufus.ie/">Rufus</a>. If you’re on a Mac
or on another Linux system, you can use the “<tt>dd</tt>” tool in a terminal to write
the image to the device. The latest version of the zip file can be obtained via
<a href="https://ham.al/btcstick">https://ham.al/btcstick</a> (which redirects to a zip
file on Google Drive). The zip file contains the image file
(<tt>btcstick‑<em>version</em>.raw</tt>) and a text file
containing a PGP signed hash of the image. The PGP public key with id
"<tt>d1f8ffb9 f7a0f7a0</tt>" can be retrieved from a public key server, e.g.:
<a href="http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xd1f8ffb9f7a0f7a0">http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xd1f8ffb9f7a0f7a0</a>.
The verification of the SHA256 hash and PGP signature is left as an exercise
for the reader.</p>
<p>To use “<tt>dd</tt>” on MacOSX or Linux, you need to determine what device file
the USB stick is mapped to. On the MacOSX terminal you can use the
“<tt>diskutil list</tt>” command to identify the device. On Linux you can view
the logging of the “<tt>dmesg</tt>” command after inserting the USB stick to
find the device. Make sure you write the image to the full device, not to a
partition on the device. <strong>When writing the image, make sure that you do not
overwrite your hard disk!</strong> As an example, if your USB device is
<tt>/dev/sdc</tt> and you extracted the image in <tt>/tmp</tt>, the command
could be:<br/>
<code>dd if=/tmp/btcstick-1.0.raw of=/dev/sdc bs=32M</code>
<br/>The prompt could return before the system has finished writing, you could
use the “<tt>sync</tt>” command to make sure the writing has finished before
removing the USB device.</p>
<h2 id="boot-methods">Boot methods</h2>
<p>I chose to use Ubuntu for this project because it is one of the few Linux
distributions that can be booted on a computer with secure boot enabled. The
process that I followed for the installation can be found at
<a href="/usb-master-stick.html">this page</a> (except for the encrypted root
part which is not needed here).</p>
<p>When booting, the default grub menu option will not pass any extra kernel boot
option. Without extra kernel parameters the overlayroot will be enabled, i.e.
the writes to the system will go to a ramdisk and will not last after the next
boot (this is not the case for writes to the <tt>/boot/efi</tt> partition which is
persistently mounted). It can be undone by booting with the
“<tt>overlayroot=disabled</tt>” boot parameter.
Also, a script will run that will look for physical network devices in the
<tt>/sys</tt> tree and use that interface to remove the device from the kernel.
I consider this a firm guarantee that the system will boot without
network connectivity. This can be disabled by booting with the “<tt>nonetdisable</tt>”
boot parameter.</p>
<p>There is an extra menu in grub named “<em>BTCStick in persistent and/or online
mode</em>”. That provides boot options with either the “<tt>overlayroot=disabled</tt>” or
“<tt>nonetdisable</tt>” options, or both.</p>
<p>After booting the user “btc” automatically logs into the graphical environment.
This user can use “<tt>sudo</tt>” as root for any command without providing a
password. If you somehow still need the password, it is “<tt>2theMoon!</tt>”.
Password access for the user <tt>root</tt> has not been enabled.</p>
<p>When booting with any of the two boot parameters, you get a warning popup that
either the filesystem is in persisitent mode, or network devices have been
detected, or both. If you boot offline and non-persistent, you get an info
popup informing you of this and the desktop wallpaper is modified.</p>
<h2 id="system-content">System content</h2>
<p>The main application that I created this stick for, is the Electrum wallet.
Inspired by Bitkey, I created a wrapper script that will mount an encrypted
filesystem on a file on a USB stick and run Electrum from that. The encryption
password is however not the password provided by the user, but a password
generated from both the user provided password and the serial number of the USB
device the file resides on (see the “Pseudo-2FA” chapter below).</p>
<p>I installed a udev rules file for the Ledger, ColdCard and KeepKey hardware
wallets but did not try them myself as I don’t own any of those. For the Trezor
(which I do have) I installed and tested the trezor-bridge daemon.
Since I created the disk for myself, I have not included tools on it to manage
altcoins. I have no need for those. You’re free to modify the image and add
such tools to manage your own needs.</p>
<h3 id="applications">Applications</h3>
<ul>
<li>
<p><strong>Electrum</strong> (<a href="https://www.electrum.org/">https://www.electrum.org/</a>)<br/>
Electrum has been installed from the source code in a separate Python virtual
environment (under <tt>/usr/local/lib/venv</tt>). With it I also installed the python
modules for different hardware wallets and that includes some command line
utilities. I wrote a script and put some symbolic links to it in
<tt>/usr/local/bin</tt> which activates the the virtual environment and runs the
binary with the same name. This enables the following commands:</p>
<ul>
<li><tt>electrum</tt></li>
<li><tt>ckcc</tt> (ColdCard utility)</li>
<li><tt>keepkeyctl</tt></li>
<li><tt>trezorctl</tt></li>
<li><tt>u2f-authenticate</tt> and <tt>u2f-register</tt></li>
</ul>
</li>
<li>
<p><strong>CryptoSeed</strong> (<a href="https://github.com/anderson-arlen/cryptoseed">https://github.com/anderson-arlen/cryptoseed</a>)<br/>
This provides a way to encrypt the BIP39 seed to your wallet(s) and present
the encrypted result as a QR code. It cannot read QR codes itself, when you
need to decrypt the seed you need another tool to scan the QR code and
copy/paste it into CryptoSeed.</p>
</li>
<li>
<p><strong>QtQR</strong> (<a href="https://launchpad.net/qr-tools">https://launchpad.net/qr-tools</a>)<br/>
This is a graphical frontend to the python-qr tools with which you can create
QR codes from text input or decode QR codes that are either read from a file
or from the webcam.</p>
</li>
<li>
<p><strong>KeePassX</strong> (<a href="https://www.keepassx.org">https://www.keepassx.org</a>)<br/>
This is an open source password manager that uses a local encrypted database.</p>
</li>
</ul>
<h3 id="web-applications">Web Applications</h3>
<p>The web applications below have been installed. For that, lighttpd has been
installed and a page that links to the various web applications has been placed
as the main page. The Chromium browser has been installed (and placed on the
desktop and the panel) and has been configured to use this page as the default
page. The idea is to use these web applications while being offline</p>
<ul>
<li>
<p><strong>coinb.in</strong><br/>
This is a web wallet that includes a couple of features like signing raw
transactions. The wallet itself is not usable without an internet connection. </p>
</li>
<li>
<p><strong>bitaddress.org</strong><br/>
This is one of the most popular paper wallet generators that cointains some
other features like creating split wallets, brain wallets, BIP38 key
encryption and decryption etc. </p>
</li>
<li>
<p><strong>bitcoinpaperwallet</strong><br/>
This is another paper wallet generator with a different fold system.<br/>
<strong>Warning:</strong><br/>
no paper wallet generator should be used online but this one needs a special
mention. In 2018 the site has been sold and the changes to the site seem to
indicate that the new owners might not have the best interest of the user at
heart. The included version does not contain those changes.</p>
</li>
<li>
<p><strong>Segwitpaperwallet</strong><br/>
Like the name implicates, this generates paper wallets with segwit addresses.
By default, it will generate P2SH wrapped segwit addresses. If you prefer
bech32 adresses, insert “<tt>/bech32</tt>” before “<tt>/index.html</tt>” in
the URL. Also note that currently not many wallets can sweep segwit
addresses. It’s possible with Electrum but not via the GUI.</p>
</li>
<li>
<p><strong>Warp brain wallet</strong><br/>
This is a wallet that uses a salt (an email address) along with the
passphrase to generate a brain wallet. In addition, it uses the “scrypt”
memory hard hashing algorithm to generate the private key i.o. plain SHA256.</p>
</li>
<li>
<p><strong>BIP39 Code Converter</strong><br/>
This can generate BIP39 seed phrases or use a seed from input and derive the
master private and public keys given various derivation paths.</p>
</li>
<li>
<p><strong>Two-factor Bitcoin</strong><br/>
This app is a system that uses BIP38 encryption to do a two party transaction
between untrusted peers. I mainly use it as an easy way to do BIP38
encryption and decryption of private keys.</p>
</li>
<li>
<p><strong>Electrum Base43 conversion</strong><br/>
Electrum uses an own data format for representing a transaction, which means
you need an online Electrum to broadcast such a transaction. With this app
you can convert the transaction to a format that can be broadcast by e.g.
Bitcoin Core.</p>
</li>
<li>
<p><strong>Key compression tool</strong><br/>
Given a public or private (WIF or BIP38 encrypted) key, it will show both the
compressed and uncompressed addess.</p>
</li>
<li>
<p><strong>Password Strength Estimation</strong><br/>
A simpe tool that uses Dropbox’s “zxcvbn” algorithm to estimate how easy a
password can be cracked.</p>
</li>
</ul>
<h3 id="command-line-tools">Command-line tools</h3>
<p>Here are some tools that are installed in the system, some are installed from
the Ubuntu software repository, some are installed seperately.</p>
<ul>
<li>
<p><strong>bitcoin explorer</strong><br/>
“<tt>bx</tt>” is a very versatile command line tool with way to many subcommands to
summarize. Check the documentation by using “<tt>bx help</tt>” or by visiting the
github wiki at <a href="https://github.com/libbitcoin/libbitcoin-explorer/wiki">https://github.com/libbitcoin/libbitcoin-explorer/wiki</a>.</p>
</li>
<li>
<p><strong>zbar-tools</strong><br/>
<tt>zbarimg</tt> and <tt>zbarcam</tt>, tools to decode QR codes from an image
or from the webcam</p>
</li>
<li>
<p><strong>rng-tools</strong><br/>
Tools and daemon to create a bridge between a hardware TRNG and the kernel’s
PRNG</p>
</li>
<li>
<p><strong>ssss-split and ssss-combine</strong><br/>
Tools to split or combine a secret string into or from multiple strings in an
“m of n” system using Shamir’s Secret Sharing System</p>
</li>
<li>
<p><strong>vanitygen</strong><br/>
Simple tool to find vanity bitcoin addresses and the corresponding private
keys.</p>
</li>
</ul>
<h3 id="scripts">Scripts</h3>
<p>The following scripts have been added to the system:</p>
<ul>
<li>
<p><tt>/usr/local/bin/mk2fsluks.sh</tt><br/>
This will create a file on a USB stick and create a Luks encrypted filesystem
on it. The password is pseudo-2fa generated (see next chapter) and by default
the script will fail if the root filesystem is read-write mounted or if
network devices are detected. These restrictions can be disabled.</p>
</li>
<li>
<p><tt>/usr/local/bin/2fa-mount.sh (and 2fa-umount.sh)</tt><br/>
This script mounts (or unmounts) a Luks encrypted filesystem contained in the
provided file. By default, the script assumes that the password is pseudo-2fa
generated and the root filesystem and network restrictions apply as above.</p>
</li>
<li>
<p><tt>/usr/local/bin/electrumrwapper.sh</tt><br/>
This script will look for a file named <tt>btcstick/electrum.vol</tt>, mount it on a
temporary directrory start the Electrum wallet from that directory. After
Electrum is terminated, the script will continue to unmount and remove the
directory. If no file is found, the script will offer to create one with a
pseudo-2fa password. The filesystem and network restrictions above apply here
as well.</p>
</li>
<li>
<p><tt>/usr/local/bin/getusbpasswd.sh</tt><br/>
This is a script that is called by the scripts above to return the serial
number of the USB device that the provided file or directory resides on or of
all available USB mounted filesystems.</p>
</li>
<li>
<p><tt>/usr/local/bin/passphrase-zxcvbn</tt><br/>
This is a script that I copied from the <tt>bitkey.io</tt> project, mainly because
zenity does not provide a “confirm” dialog when inputting a passphrase. This
script is used my the <tt>mk2faluks.sh</tt> script. The script needs an older version
of Python2's <tt>python-zxcvbn</tt> module.</p>
</li>
<li>
<p><tt>/root/bin/removenet.sh</tt> <br/>
This script will search the kernel datastructure presented in the “<tt>/sys</tt>”
filesystem for physical network devices and remove them from the kernel. The
script is called from <tt>/etc/rc.local</tt> and exits if the “nonetdisable” boot
parameter was provided.</p>
</li>
<li>
<p><tt>/home/btc/bin/checkoffline.sh</tt><br/>
This is a simple script that will test if the overlayroot filesystem is
active and if all physical network devices have been removed. It will
generate a pop-up accordingly and also modify the desktop wallpaper if both
are true. The script is run at the startup of the Mate desktop.</p>
</li>
<li>
<p><tt>/usr/local/lib/venv/venvapp.sh</tt>
A simple script to activate a Python virtual environment and execute a binary
with the same name.</p>
</li>
</ul>
<h2 id="pseudo-2fa">Pseudo-2FA</h2>
<p>I was inspired by the <tt>bitkey.io</tt> project for this stick and as stated
above, created a wrapper script that will mount a Luks encrypted filesystem
from a file on a USB disk to start Electrum from, but I wanted to make it even
more difficult to get to the data, so the passphrase for decrypting the file is
generated from two inputs, i.e. the passphrase provided by the user and the
serial number of the USB device that the file resides on. The script that gets
the serial number will use “<tt>udevadm</tt>” to list the parameters of the USB
device and return the “<tt>ID_SERIAL_SHORT</tt>” parameter. The passphrase is
calculated as:<br/>
<code>base64(sha256(serial_nr|user_pw|"\n"))[0:24]</code>
<br/>(i.e., the first 144 bits of the binary sha256 hash of the concatenation of the
serial number, the user password and a newline, represented as a base64
string).</p>
<p>If the USB device containing such a file is retrieved by an adversary and the
algorithm is known, the system collapses to a single factor. If you have a
healthy paranoia you can store the file elsewhere and temporarily copy it to
the USB device where and when it’s needed.<br/>
Two caveats: <br/></p>
<ol>
<li>
<p>if your USB device becomes unreadable (and you don’t know the serial number
by another means), you loose access to the encrypted file. Make sure that
you can access the data via another means (e.g. proper seed backups).</p>
</li>
<li>
<p>you should not assume that an adversary cannot retrieve the serial number
and brute force the encryption passphrase. This system should not lull you
into a false sense of security by choosing a weak user password.</p>
</li>
</ol>
<h2 id="updating">Updating</h2>
<p>To update the system you need to boot the system with the root filesystem
persistently mounted and the network enabled. Then you can use <tt>apt-get</tt>
to update the packages on the system or install new packages.<br/>
<strong>CAVEAT EMPTOR:</strong><br/>
if you update packages like the kernel (any package that modifies the boot
system), it is strongly advised to do the update after booting from a legacy
BIOS system. If you do it on an existing system that you booted with UEFI, it
will modify your EFI bootmanager with an “Ubuntu” entry. If you do this on a
system that already has Ubuntu on it, it will render that system unbootable.
This can be rescued from an Ubuntu live disk but that is not a trivial task.
The safest way to do updates is booting the image from a virtual machine
hypervisor like VirtualBox or libvirt. In the former case you should convert
the image file to VDI format with VboxManage and after updating from within
VirtualBox convert the image back to raw and overwrite your USB disk with the
updated image.</p>
<h3 id="resizing-the-partition">Resizing the partition</h3>
<p>You will likely have some room left over on your USB stick. You can either add
a new partition using the remaining space or resize the partition to claim the
space for the BTCStick itself. To ease that task, gparted is installed on the
image. It is advisable to boot with a permanent root filesystem to do this
task.</p>
<h2 id="more-information-and-feedback">More information and feedback</h2>
<p>Feedback is welcome via email at <a href="mailto:btcstick@hamal.nl"><tt>btcstick@hamal.nl</tt></a>.
Please add the word “btcstick” in the subject line to aid in the visibility of
you mail.<br/>
The source code for the scripts, local homepage etc. is available via
<a href="https://gitlab.com/hamal03/btcstick">https://gitlab.com/hamal03/btcstick</a>.<br/>
A prebuilt image is available via Google Drive. The link for this image is
available via <a href="https://ham.al/btcstick">https://ham.al/btcstick</a>. </p>
<p></p>Creating a UEFI and BIOS bootable Linux installation on USB2019-06-27T00:00:00+02:002019-06-27T00:00:00+02:00Rob Wolframtag:None,2019-06-27:/usb-master-stick.html<!--HeaderImage: images/bitcoin-intro.jpg-->
<p>I had a need for a USB stick with an installed version of Linux that was
bootable with UEFI (and Secure Boot) as well as Legacy BIOS. I couldn't use a
live stick because I needed to employ an encrypted Luks root filesystem and swap
partition because it would contain …</p><!--HeaderImage: images/bitcoin-intro.jpg-->
<p>I had a need for a USB stick with an installed version of Linux that was
bootable with UEFI (and Secure Boot) as well as Legacy BIOS. I couldn't use a
live stick because I needed to employ an encrypted Luks root filesystem and swap
partition because it would contain some files (like ssh private keys) that would
be used at work. I wanted it to be bootable with legacy BIOSes as well because I
wanted to also use the system as a generic "Swiss army knive" with Live
filesystems like Kali and SystemRescueCd and tools like DBAN. This blogs shows
how to achieve this.</p>
<p>I opted to install Ubuntu since that is capable of booting on a system that
only allows secure boot. To enable this install with Ubuntu Bionic (18.04 LTS)
you need the <a href="http://cdimage.ubuntu.com/releases/bionic/release/">Alternative Ubuntu Server
installer</a> since the new
installer (Subiquity) does not enable a Luks encrypted LVM with multiple logical
volumes.</p>
<p>It is advisable to create the installation on a virtual machine like libvirt or
VirtualBox because doing an UEFI installation on an existing system (especially
an Ubuntu system) may render your system unbootable. It might be rescued but
it's at least an annoyance. I opted to run it on libvirt (which uses Qemu/KVM)
because that enables it to be installed on a file as a raw disk image, while
VirtualBox needs the vdisk to be in VDI format. To do that, I had to install
both libvirt and ovmf (Open Virtual Machine Firmware) on my system. Here's what
to do next:</p>
<p>Create a file that is distinctly smaller than the intended USB stick:</p>
<pre>$ dd if=/dev/zero bs=1G count=0 seek=14 of=usbimage.raw</pre>
<p>Create a DOS partition on it with fdisk or parted. If the USB stick has a GPT
partition with an EFI partition present, some systems will not present a BIOS
boot option. </p>
<pre>$ fdisk usbimage.raw</pre>
<p>Create a larger than 32MB partition for EFI (leave it as a Linux type for now), a boot
partition (I used 500MB) and a large partition for LVM. Before encrypting
the LVM partition, it is a good idea to fill it with random data. Connect the
file with kpartx and determine the loop device it uses to know what device to
fill with random junk. I tend to generate a random stream with AES because that
is quite a bit faster than using <em><tt>/dev/urandom</tt></em> and fine for our purpose:</p>
<pre>$ sudo kpartx -a usbimage.raw
$ losetup # check the loop device that is used
$ K=`dd if=/dev/urandom bs=16 count=1 | xxd -p` # random key for AES stream
$ IV=`dd if=/dev/urandom bs=16 count=1 | xxd -p` # random IV for AES stream
$ dd if=/dev/zero | openssl enc -aes-128-cbc -K $K -iv $IV | sudo dd \
of=/dev/mapper/loop0p3 bs=32M # replace loop0p3 with the proper device
# p3 means the third partition
$ sudo kpartx -d usbimage.raw
</pre>
<p>Create a VM with virt-manager. Use "Import an existing disk" and select the disk
you just created. Before finishing, toggle the option "Customize configuration
before install. In the Overview section, change the Firmware to UEFI. Select the
disk device and make sure it's connected to a SATA or USB bus (not VirtIO) and
add a CDRom which you connect to the Ubuntu server iso file.<br/>
<img alt="UEFIvm" class="img-fluid" src="/images/01_uefi.png"/><br/></p>
<p>When running the installation you may be asked if you want to force an UEFI
installation. Affirm this. When the disk partitioning menu occurs, opt for the
manual partitioning. Use the small partition as EFI boot partition, and the
larger partition as ext4 filesystem mounted on /boot. Since this will be used
on a USB stick, it's advisable to set the "<em>noatime</em>" and "<em>nodiratime</em>" mount
options. If you will use an encrypted volume group, select the largest
partition to be used as physical volume for encryption. Choose to configure
encrypted volumes and create an encrypted volume.<br/>
<img alt="EncryptedPartition" class="img-fluid" src="/images/08_confenc.png"/><br/>
After this the encrypted
partition will be configured to be used as an ext4 filesystem. Change that to
a physical volume for LVM. If you opt not to use encryption, you can use the
partition itself as a physical volume for LVM. Create a volume group on the
encrypted volume (or partition) and create at least two logical volumes, one
for swap space and one for the root filesystem.</p>
<p>Continue the installation and make sure the system boots. If you opted for an
encrypted volume group for root and swap, you should be prompted for the
password during the boot to decrypt the partition. After the installation,
install the packages "grub-pc" and "grub-pc-bin". When requested, install grub
on the master boot record. Verify that the system still boots with the UEFI
firmware. To make the system bootable on any UEFI bios, copy the directory
<em><tt>/boot/efi/EFI/ubuntu</tt></em> to <em><tt>/boot/efi/EFI/BOOT</tt></em>. Mount the
installation DVD and copy the file
<em><tt>/<cdrom_mount_point>/EFI/BOOT/BOOTx64.EFI</tt></em> to
<em><tt>/boot/efi/EFI/BOOT</tt></em>.</p>
<p>To test the Legacy BIOS boot, remove the disk from the VM and create a new VM
with the same disk as SATA or USB disk, this time keep the firmware on BIOS.
verify that the system boots here as well. To verify that the system will boot
on other UEFI systems, remove the disk from the Legacy BIOS VM and create a
new UEFI based VM with this disk. Verify that the system boots.</p>
<p>The EFI partition contains only the .efi files to to bootstrap the process and a
grub config file that just looks for the grub file on <em><tt>/boot</tt></em> and loads that. You
might opt to remove the <em><tt>/boot/efi</tt></em> filesystem from your fstab to prevent changes
on that. The system should now be upgradable from both a UEFI as well as a
Legacy BIOS firmware. You can now install all needed software. The various
desktop all have a meta package to install the whole environment (like
ubuntu-gnome-desktop, ubuntu-mate-desktop, lxqt and xfce4). </p>
<p>When you're satisfied with the installation you can copy the image to the USB
stick. Insert the USB stick and use <em><tt>lsblk</tt></em> to verify the device that is connected
to it (in this example we use sdc). <font color="red"><strong>Caveat emptor: this is a
tricky command, make sure you're not overwriting your own hard disk!</strong></font></p>
<pre>$ sudo dd if=usbimage.raw of=/dev/sdc bs=32M oflag=sync</pre>
<p>If you have space available, you can create more partitions and add some live
systems to the stick that you might occasionally use. Modify the file
<em><tt>/etc/grub.d/40_custom</tt></em> to create Grub entries for those and use
<em><tt>grub-mkconfig</tt></em> to
create a grub config file capable of booting those. The howto for this is
outside of the scope of this article.</p>
<p>Enjoy your USB installation!</p>