A snapshot taken from my living room when a thunderstorm approached.

Luca expects me to throw his beloved banana for the quadrillionst time. 😅

This blog is powered by Hiawatha, a light-weight webserver designed for security and ease of use. Consequently, Hiawatha comes with a script that allows one to easily request certificates from the Let's Encrypt initiative and (in conjunction with a daily cron job) to automagically renew them when the time has come.

This setup has worked almost flawlessly for several years. In 2021, I've received an information from Let's Encrypt that they would modify (as planned) their chain of trust, requiring corresponding changes in the LE_ISSUERS option in the configuration file of the script designated for requesting or renewing certificates.

I should have known that this change will happen every three years, but since I didn't receive any mail this time, it never occurred to me that the failure of renewal had this simple reason. Instead, I've searched everywhere for nonexisting error messages until I had run out of ideas. Without any options left, I've asked Haui for help, convinced that he would see light where I could see only dark. And it indeed didn't take him long to identify an outdated LE_ISSUERS value in the configuration file as the culprit.

We can easily look up the common name of the current certificate's issuer:

openssl x509 -in /etc/hiawatha/tls/pdes-net.org.pem -noout -text | grep CN

But that won't help if the current certificate is not renewed because of an outdated issuer. The present situation was different in that I've requested new certificates in September as a temporary (HOHOHO) workaround. These new certificates were issued with the new CN of R10, as compared to the old R3 in the configuration file, making it clear that the latter is outdated. It would have been so easy if I hadn't been such a fool and categorically ruled out this possibility. 🫥

Well, I may get old and useless, but I hope to recall once and for all that the authoritative instance for looking up the current issuer for Let's Encrypt can be found here: https://letsencrypt.org/certificates/. And if I don't, I'm sure to remember that I can find this information in my own blog. 🫩

Our administration now requests all documents to be digitally signed, with a certificate that each employee receives from the DFN. Windows and Mac users employ Acrobat Reader for this purpose, but what software can be used under Linux?

My first choice was LibreOffice, which has offered this functionality already for several years. Signing a pristine pdf works well indeed, apart from the fact that LibeOffice does not create a placeholder for the signature. However, signing a document that has already been signed by Window users turned out to be simply not possible (here's the 12 years old bug report – and here's the inverse one that is just as old).

The next candidate was Okular, the PDF viewer of KDE, which has been sponsored by the University of Dresden to implement this functionality. But only half-way, it seems to me. I could sign most (but not all) documents, but I couldn't configure the placeholder at all. As the font size in the signature box does not scale with the size of the box, the name of the person signing is often cut off. How difficult can it be to implement such a very basic and obvious requirement?

Finally, I turned to Master PDF Editor, which I've occasionally used in the past for annotations when Evince did not yet offer this possibility. I was actually not surprised that this feature-rich PDF editor also offers digital signatures, but I was pleased that the software comes with its own certificate storage and that the signature placeholder is highly configurable. For example, one can configure the placeholder to include one's own analogue signature as a background.

Alas, using the Master PDF editor without any restrictions requires purchasing a license. The free version is unlimited, but leaves a watermark in documents that have been digitally signed or otherwise altered with the software. The licence is very fairly priced, but as the software is developed in Russia, even asking for one is frowned upon and politically inopportune. Fortunately, nobody has yet complained about the watermark in documents I have digitally signed. 😈

The upstream xz repository and the xz tarballs have been backdoored.

This backdoor is very indirect and only shows up when a few known specific criteria are met. Others may be yet discovered! However, this backdoor is at least triggerable by remote unprivileged systems connecting to public SSH ports.

This supply-chain attack targets .deb- and .rpm-based distributions, but the backdoored versions of xz or xz-utils (5.6.0 and 5.6.1) have made it only into rolling-release distributions such as Fedora Rawhide, Debian Testing/Sid, OpenSuse Tumbleweed, and Archlinux (where it is inactive).

The server of this blog is running Debian Testing and had the compromised version of xz-utils installed since March 17. The backdoor was reported last Friday, March 29. I've installed the patch provided by Debian on Saturday, March 30, and examined the system logs, which do not show any evidence that the system has been compromised in any way. In fact, according to my current understanding, the system did not meet all the requirements for the backdoor to be executed. However, I will remain vigilant and let the users of the server know if further action needs to be taken.

More links (in German): Heise 30.03.2024 09:35, Heise 30.03.2024 22:28, Heise 02.04.2024 17:10

Yesterday, I've updated my systems to kernel 6.6.9 – two Intel-based desktops and one AMD-based notebook. When rebooting the latter, I immediately noticed that something was wrong. Logging in, for example, seemed to take twice as long, and the desktop needed much longer than the usual two or three seconds to come up. My Intel desktops, in contrast, behaved exactly as before.

To substantiate my feeling that my notebook's performance had degraded significantly since the update, I used sysbench, or, more precisely, the command sysbench cpu run. I would normally see a performance of about 4800 events per second on one core. But with kernel 6.6.9, all I've got were 440 events per second, more than a factor of 10 lower than the Ryzen 5800H in my notebook is supposed to deliver, and even three times lower than my 10-years old Intel desktops. No surprise the notebook felt so sluggish!

I didn't bother to investigate this issue further, and I don't know the underlying cause, like whether it's related to the AMD processor or the maker of the notebook. I just rolled back to kernel 6.6.8 (sudo pacman -U /var/cache/pacman/pkg/linux-6.6.8.arch1-1-x86_64.pkg.tar.zst) and the problem was gone.

I expected problems with kernel 6.6.6, but the devil is in the details.

Update: The performance is back to normal with kernel 6.6.10.

Connecting to a VPN is usually like picking up your device and tossing it into another network, figuratively speaking. All of your network activities – such as browsing, fetching private mails, chatting with a friend on IRC – will take place within this virtual network, or not at all: in its most secure configuration, access to resources on the local area network will not be possible. I thus prefer to separate my real private network activities from those in the virtual private network by using a virtual guest dedicated to nothing but connecting to the latter and doing whatever I need to do within the guest system.

In the present case, I'm fortunate that my employer now uses a gateway whose VPN client (Palo Altos's GlobalProtect) runs even on an up-to-date Arch installation. So my choice for the guest system is an out-of-the-box ArchBang that comes with i3 as (tiling) Window manager. It installs in 10 min, comes with everything I need, and fits in 5 GB of space. I spent another 5 min modifying the wallpaper and the conky instance – my idea was to have a visual indication in form of my IP whether or not I'm connected to the VPN.

After configuring everything to my liking, it turned out that I shouldn't have bothered – our IT guys configured the VPN with split tunneling enabled. This basically means that only traffic destined to the remote location passes through the encrypted tunnel, while everything else uses the standard gateway. Supposedly less secure, but certainly much more convenient. Excellent choice! I'm sure I'll find another use for my virtual Arch – be it for testing or online banking.

It's Friday evening, 18:30. My fourth video meeting in a row has just concluded. Now I could finally work on the revision of a manuscript I wanted to get resubmitted during the weekend. This last revision was purely technical: the production editor requested that we move the present addresses of the authors to the back of the manuscript, instead of leaving them beneath the list of authors on the title page as destined by the LaTeX class from the publisher. Now, any such request that forces me to work around or against the journal style provided by the publisher means that the reputation of the journal (ACS Appl. Nano Mater., in case you are curious) takes a steep dive. But anyway, I had to do it, and I was looking into the footmisc package to get all footnotemarks I needed when I realized that I hadn't done my ritual update in the morning for the lack of time. Starting it, I only peripherally noticed that the update involved TeXLive and brought a new kernel. In any case, this information didn't stop me from compiling the manuscript I was working on during the update. Repeatedly. Incessantly.

At a certain point, the build command of Sublime Text didn't produce any reaction. No error message, nothing. I began to have a bad feeling. Indeed, while I could still move the mouse around, the entire Window system was unresponsive, and the update process – which was just about to build the fmt files – was hanging. I started to suspect that I had just committed the greatest blunder of this year, and indeed, when I rebooted, the system greeted me with the message that the kernel could not be found:

Loading Linux linux...
error file /boot/vmlinuz-linux not found
loading initial ramdisk
error: you need to load the kernel first

Well, I knew that this SNAFU looked worse than it actually is. But since I was suddenly very tired, I decided to call it a day and do the repair on Saturday morning.

On Saturday, I first needed a live Arch installation on a USB stick. The ISO ist just 813 MB (as of release 2023.07.01) and downloaded in 30 s. There are several options to write the ISO to the stick, but I prefer dd:

dd bs=4M if=archlinux-archlinux-2023.07.01-x86_64.iso of=/dev/sdd conv=fsync oflag=direct status=progress

Note that the stick must not be mounted, and one writes to the stick (sdd), not a partition (sdd1).

After booting from the thus created live media, I was just a few commands away from a restored system. I first wanted to have my WiFi working:

iwctl --passphrase PASSPHRASE station DEVICE connect SSID

After that, I just needed to mount my drives (have a look with lsblk before), delete the stale lock file from the previous failed update, and do an update in the mounted root directory:

mount /dev/nvme01p2 /mnt
mount /dev/nvme01p1 /mnt/boot
mount -t proc /proc /mnt/proc
mount --rbind /sys /mnt/sys
mount --rbind /dev /mnt/dev

rm /var/lib/pacman/db.lck

pacman --sysroot /mnt -Syu

Took all in all half an hour, but I would still have preferred to avoid this situation altogether. The lesson is: avoid working on the system when you're all stressed out. Particularly on Friday night.

A little late, but better late than never:

Bookworm is stable, Trixie is the new testing.

sed -i 's/bookworm/trixie/g' /etc/apt/sources.list

The institute I'm with has offered a VPN solution for its employees for about 15 years. Well, at least for the Windows users of our staff. The Cisco we've used in the first years came with the abysmal 'vpnclient', which I've said one time to be a clear winner of the worst-software-on-this-planet contest. It would run only on CentOS and Debian OldStable (meaning Sarge at that time!), and there was no way to get it running in any reliable way on a halfway modern Linux. I've thus ended up by connecting to the VPN via a virtual Windows XP, until I discovered openconnect, which worked perfectly on modern Linux distributions. Still, I was delighted when we finally kicked out the Cisco and got a Checkpoint instead, only to learn that the promised Linux client was still in development. It actually never materialized, but I was content with the browser-based solution they offered. Now the Checkpoint license has expired, and instead of renewing the contract, we've gotten ourselves a Palo Alto firewall – much to my surprise, as Palo Alto is not known for being a bargain. But in any case, we now – after 15 years – have a VPN client that can be installed on a fully updated Arch system and actually works.

Jonas, a fellow Archer and colleague of mine, figured out the best way how to install the client. I'm' adding his instructions here so I find them when the next version is released. 😄

1. Install the AUR package 'globalprotect-bin' (which will fail, but gets the necessary 'PKGBUILD' file and 'globalprotect.install' script).

2. From /software/VPN/VPN_Client_GlobalProtect get the latest version of the archive with the client binaries for Linux: 'PanGPLinux-u.v.w-...tgz'. From this archive, you only need the files 'GlobalProtect_tar-u.v.w.x-yz.tgz' and 'GlobalProtect_UI_tar-u.v.w.x-yz.tgz'.

3. Place the two files in the AUR build-folder, e.g. '.cache/yay/globalprotect-bin/'

4. Check that in PKGBUILD the correct 'pkgver' (u.v.w.x) and 'pkgrel' (yz) are set. If you need to change these, you also need to adapt the 'sha256sums'.

5. Run makepkg -si.

6. Start gpd.service using systemctl enable --now gpd.service (check status with systemctl status gpd) and restart the system.

7. Import the certificate (I had to use an absolute path): globalprotect import-certificate --location /home/user/...

To update the client to a new version, you need to repeat steps 2–5 and restart the system.

Now you are ready and you can

• Start the connection using globalprotect connect --portal vpn.foo.bar.de

• Disconnect using globalprotect disconnect

• Alternatively, run the gui (which then appears in the system tray) using: globalprotect launch-ui and use the connect/disconnect button

I have little to add to these instructions. I don't recommend the GUI: it is outdated and does not work in a high-dpi environment such as my notebook. But that's fine with me; I like the CLI better anyway. Following my habits, I've just defined aliases for the two commands most frequently used:

alias vpnon='globalprotect connect --portal vpn.foo.bar.de'
alias vpnoff='globalprotect disconnect'

Perhaps I will install a minimal virtual Linux guest to run the client and to connect, while still being able to use my other connections in the host system. I'll post an update if I do that. 🤠