Aigarius Blog (Posts about debian)

Snapshot mirroring in Debian (and Ubuntu)

Aigars Mahinovs — Mon, 04 Aug 2025 18:00:00 GMT

Snapshot mirroring in Debian (and Ubuntu)

The use of snapshots has been routine in both Debian and Ubuntu for several years now—or more than 15 years for Debian, to be precise. Snapshots have become not only very reliable, but also an increasingly important part of the Debian package archive.

This week, I encountered a problem at work that could be perfectly solved by correctly using the Snapshot service. However, while trying to figure it out, I ran into some shortcomings in the documentation. Until the docs are updated, I am publishing this blog post to make this information easier to find.

Problem 1: Ensure fully reproducible creation of Docker containers with the exact same packages installed, even years after the original images were generated.

Solution 1: Pin everything! Use a pinned source image in the FROM statement, such as debian:trixie-20250721-slim, and also pin the APT package sources to the "same" date - "20250722".

Hint: The APT packages need to be newer than the Docker image base. If the APT packages are a bit newer, that's not a problem, as APT can upgrade packages without issues. However, if your Docker image has a newer package than your APT package sources, you will have a big problem. For example, if you have "libbearssl0" version 0.6-2 installed in the Docker image, but your package sources only have the older version 0.6-1, you will fail when trying to install the "libbearssl-dev" package. This is because you only have version 0.6-1 of the "-dev" package available, which hard-depends on exactly version 0.6-1 of "libbearssl0", and APT will refuse to downgrade an already installed package to satisfy that dependency.

Problem 2: You are using a lot of images in a lot of executions and building tens of thousands of images per day. It would be a bad idea to put all this load on public Debian servers. Using local sources is also faster and adds extra security.

Solution 2: Use local (transparently caching) mirrors for both the Docker Hub repository and the APT package source.

At this point, I ran into another issue—I could not easily figure out how to specify a local mirror for the snapshot part of the archive service.

First of all, snapshot support in both Ubuntu and Debian accepts both syntaxes described in the Debian and Ubuntu documentation above. The documentation on both sites presents different approaches and syntax examples, but both work.

The best approach nowadays is to use the "deb822" sources syntax. Remove /etc/apt/sources.list (if it still exists), delete all contents of the /etc/apt/sources.list.d directory, and instead create this file at /etc/apt/sources.list.d/debian.sources:

Types: deb
URIs: https://common.mirror-proxy.local/ftp.debian.org/debian/
Suites: trixie
Components: main non-free-firmware non-free contrib
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Snapshot: 20250722

Hint: This assumes you have a mirror service running at common.mirror-proxy.local that proxies requests (with caching) to whitelisted domains, based on the name of the first folder in the path.

If you now run sudo apt update --print-uris, you will see that your configuration accesses your mirror, but does not actually use the snapshot.

Next, add the following to /etc/apt/apt.conf.d/80snapshots:

APT::Snapshot "20250722";

That should work, right? Let's try sudo apt update --print-uris again. I've got good news and bad news! The good news is that we are now actually using the snapshot we specified (twice). The bad news is that we are completely ignoring the mirror and going directly to snapshots.debian.org instead.

Finding the right information was a bit of a challenge, but after a few tries, this worked: to specify a custom local mirror of the Debian (or Ubuntu) snapshot service, simply add the following line to the same file, /etc/apt/apt.conf.d/80snapshots:

Acquire::Snapshots::URI::Override::Origin::debian "https://common.mirror-proxy.local/snapshot.debian.org/archive/debian/@SNAPSHOTID@/";

Now, if you check again with sudo apt update --print-uris, you will see that the requests go to your mirror and include the specified snapshot identifier. Success!

Now you can install any packages you want, and everything will be completely local and fully reproducible, even years later!

Debconf 25 photos

Aigars Mahinovs — Sun, 03 Aug 2025 15:00:00 GMT

Debconf 25 photos

Debconf 25 came to the end in Brest, France this year a couple weeks ago.

This has been a very different and unusually interesting Debconf. For me it was for two, related reasons: for one the conference was close enough in Western Europe that I could simply drive there with a car (which reminds me that I should make a blog post about the BMW i5, before I am done with it at the end of this year) and for the other - the conference is close enough to Western Europe that many other Debian developers could join this year who have not been seen at the event for many years. Being able to arrive early, decompress and spend extra time looking around the place made the event itself even more enjoyable than usual.

The French cuisine, especially in its Breton expression, has been a very welcome treat. Even if there were some rough patches with the food selection, amount, or waiting, it was still a great experience.

I specifically want to say a big thank you to the organisers for everything, but very explicitly for planning all the talk/BOF rooms in the same building and almost on the same floor. It saved me a lot of footwork, but also for other participants the short walks between the talks made it possible to always have a few minutes to talk to people or grab a croissant before running to the next talk.

IMHO we should come back to a tradition of organising Debconf in Europe every 2-3 years. This maximises one of the main goals of Debconf - bringing as many Debian Developers as possible together in one physical location. This works best when the location is actually close to large concentrations of existing developers. In other years, the other goal of Debconf can then take priority - recruiting new developers in new locations. However, these goals could both be achieved at the same time - there are plenty of locations in Europe and even in Western Europe that still have good potential for attracting new developers. Especially if we focus on organising the event on the campuses of some larger technically-oriented universities.

This year was also very productive for me—a lot of conversations with various people about all kinds of topics, especially technical packaging questions. It has been a long time since the very basic foundations of Debian packaging work have been so fundamentally refactored and modernized as in the past year. Tag2upload has become a catalyst for git-based packaging and for automated workflows via Salsa, and all of that feeds back into focusing on a few best-supported packaging workflows. There is still a bit of a documentation gap of a new contributor getting to these modern packaging workflows from the point where the New Maintainers Guide stops.

In any case, next year Debconf will be happening in Santa Fe, Argentina. And the year after that it is all still open and in a close competition between Japan, Spain, Portugal, Brazil and .. El Salvador? Personally, I would love to travel to Japan (again), but Spain or Portugal would also be great locations to meet more European developers again.

As for Santa Fe ... it is quite likely that I will not be able to make it there next year, for (planned) health reasons. I guess I should also write a new blog post about what it means to be a Debconf Photographer, so that someone else could do this as well, and also reduce the "bus factor" along the way.

But before that - here is the main group photo from this year:

You can also see it on:

Google Photos

You can also enjoy the rest of the photos:

Additionally, check out photos from other people on GIT LFS and consider adding your own photos there as well.

Other places I have updated with up-to-date information are these wiki pages:

If you took part in the playing cards event, then check your photo in this folder and link to your favourite from your line in the playing card wiki

Debconf 24 photos

Aigars Mahinovs — Fri, 02 Aug 2024 09:00:00 GMT

Debconf 24 is coming to a close in Busan, South Korea this year.

I thought that last year in India was hot. This year somehow managed to beat that. With 35C and high humidity the 55 km that I managed to walk between the two conference buildings have really put the pressure on. Thankfully the air conditioning in the talk rooms has been great and fresh water has been plentiful. And the korean food has been excellent and very energetic.

Today I will share with you the main group photo:

You can also see it in:

The rest of my photos from the event will be published next week. That will give me a bit more time to process them correctly and also give all of you a chance to see these pictures with fresh eyes and stir up new memories from the event.

Debconf 23 photos all

Aigars Mahinovs — Mon, 02 Oct 2023 08:00:00 GMT

Two weeks have passed since Debconf 23 came to a close in Kochi, Kerala, India this year.

In keeping with the more relaxed nature of Debconf in India, the rest of my photos from the event were to be published about two weeks from the end of the event. That will give me a bit more time to process them correctly and also give all of you a chance to see these pictures with fresh eyes and stir up new memories from the event.

In the end we are looking at 653 photos and one video. Several different group photos, including a return of the pool group photo that was missing from the event since Mexico in 2006! This year was the first for a new camera (Canon R7) and I am quite happy with the results, even if I still need to learn a lot about this new beast. Also the gradual improvements of panorama stiching software (Hugin) ment that this year I did not need to manually correct any face-melt events on any of the group photos. So that is cool!

You can find all my photos on:

Also, don't forget to explore the rest of the Git LFS share content - there are very many great photos by others this year as well!

Debconf 23 photos

Aigars Mahinovs — Sun, 17 Sep 2023 12:00:00 GMT

Debconf 23 is coming to a close in Kochi, Kerala, India this year.

And it has been my pleasure to again be here and take lots of pictures of the event and of the surroundings. In total I took 1852 photos and walked just over 50 km between the two venue buildings and all the rooms where action happened.

Today I will share with you the main group photo:

You can also see it in:

In keeping with the more relaxed nature of Debconf in India, the rest of my photos from the event will be published in about two weeks from now. That will give me a bit more time to process them correctly and also give all of you a chance to see these pictures with fresh eyes and stir up new memories from the event.

Smart Shelly with Home Assistant

Aigars Mahinovs — Thu, 13 Apr 2023 11:30:50 GMT

I am looking to improve my smart home installation as I plan to move to a new appartment. I already have a well functioning Home Assistant installation with a ton of automations (like lights that automatically come on when light level outside drops, that dim when it is time to go to sleep, robot vaccum that runs when everyone is away and also automation that lets us know when CO2 levels are too high and we should ventilate). We have colorful smart lights that can be dimmed or can change colors as the mood strikes.

However, one big problem that my setup currently has is that physical light switches on the walls can not be used to control the lights (as in - if the wall switch turns the light off, there is no way to turn it back on via Home Assistant) and if the Home Assistant goes down (which does sometimes happen, usually when I am away for a longer time) then it is really hard to control the lights without it.

So I've been searching for a holy grail: combination of smart (or smartified) light switches on the walls and smart light bulbs (with colors and dimming) that works with automations when Home Assistant is online and also just works directly to power lights on and off when Home Assistant is offline.

And I think I have found a solution that will be acceptable for me. The solution involves taking a normal dumb light installation, putting in smart light bulbs (any that Home Assistant can manage) and then wiring in a Shelly next to an existing dumb wall switch. For design consistency sake it would be good if the wall switch is of impulse type (so only connecting the connectors when the button is being pressed, like a doorbell button), but the setup should also work for normal toggle wall switches with minimal changes. Some settings on the Shelly and a custom Shelly script will take care of offline functionality. Note: Gen1 Shelly devices do not have scripting capabilities, you need a Gen2 device to do this, like the linked Shelly Plus.

So first I created a test setup to validate my design. I took an old extension cable with a damaged wire, cut out the damaged part and split up the wiring to make my test setup. It was really convinient to use the right tools - a simple cable stripper, a precise wire stripper and best wire connectors. Using the right tools it was really easy, fast and also safe to just pull insulation from the cable, split up the wires, clean off exactly 10mm off the end of each wire and push them into the Wago connectors to connect them together.

The completed test setup is a bit janky, but it works and is electrically safe (at least for testing purposes). The wiring is pretty clear from Shelly documentation (which is also just printed on the front of the device itself). The only point of clarification is that on the output side the Shelly is a relay and not a power provider, which means that when connecting Shelly to the wires going towards your lights it is not enough to just wire Shelly output one to lights live wire and Shelly output zero to lights neutral wire. Instead you need to connect you input neutral wire directly to your lights neutral (same as input ground) and then wire the input live wire trough the Shelly output contacts (so input live to Shelly output zero and Shelly output one to lights live wire). Except if you but a Shelly Plus 1PM, which has to measure power consumption and thus has to pass the whole device current trough itself to properly measure it.

When that is done, tested and plugged in you can use Shelly app to connect to your new Shelly device and make it join your WiFi network (or you can just use a laptop or a phone to joint to the Shelly ad-hoc WiFi network and configure it via its build-in web UI). When the Shelly is in your network you can continue to configure via the Web UI and Home Assistant.

Add the Shelly to your Home Assistant installation (it will automatically detect a new Shelly device and promt to enable Shelly integration once the device shows up in the local WiFi network). Add an automation to toggle the light state of the connected smart bulb(s) (Service: Light toggle) when "First button button down" event triggers on the Shelly. This will be the way the light state is going to be controlled normally. It can also be useful to add an automation that runs on Home Assistant start-up to ask all Shelly devices to turn outputs to on, so that after recovery of Home Assistant after an outage you can

In the Shelly web UI (after updating firmware, as usual) you'd want to disconnect output from input, set input type to button (assuming you have an impulse button input, like I do) and configure the output to be powered on by default when power comes back. And then you'll want to add this script to your Shelly. You will want to adjust the URL to your Home Assistant instance there. This script monitors if the Home Assistant is alive or not and if the Home Assistant is not alive, then take the matters into its own hands and power off and on the lights directly.

I am sure there are many ways this can be improved and tweaked. I like the flexibility this approach gives - I can cleanly program behaviour in Home Assistant and also directly on the Shelly switches (without even having to flash any custom firmware). This particular way is also quite scalable - if I have 10 Shelly switches in my home I can have exactly the same script and settings on all of them - the link between the switch and the light(s) is done via physical wiring and via automation settings in Home Assistant. The Shelly does not have to know what smart lights (or other devices) are connected to it and how they are supposed to be controlled.

If you are using MQTT with Home Assistant, then Shelly can also speak directly to MQTT instead and the online check becomes much more trivial because the MQTT object in the Shelly Script API has isConnected property.

Securing your blog - fast and easy

Aigars Mahinovs — Thu, 20 Oct 2022 20:04:50 GMT

After the last couple of posts a few people asked me why my blog site is still HTTP-only? Why did I not secure my page and contribute my little bit to normalizing encrypted traffic on the Internet and all that good jazz?

And the answer was - I was lazy and thought that it was complicated.

I could not have been more wrong :D However the key here is not to follow the official guides. In fact doing things the Debian way turned out to be far simpler and straight forward.

If you want to get a free SSL certificate and setup your site correctly, you will likely eventually get to this page and it will then you instrct you to install snap and pull a snap package of certbot. Well, no. I will not be doing that, thank you very much. I have a perfectly functional Debian packaing system right here and I do not want to pollute my system with multiple package managers and updaters. No snap, no flatpack. I am fine with Docker because those are containers, they are not in my system, they run in separated systems above my system.

Turns out all of that was way more complicated than necessary, anyway.

Debian Buster contains a perfectly functional package of certbot and it is fully integrated too!

All I had at the start was my domain configured on HTTP in nginx.

All I did was:

# apt install certbot python3-certbot-nginx
# certbot run

It asked me a few questions about me, which domain to work on (of the configured nginx domains that it automatically found) and then just automagically got the certificates, deployed them and configured nginx to enable HTTPS using those new certificates. As an extra it also asked me if I wanted to redirect all HTTP traffic to HTTPS and that was it. I was done. No extra steps, no config files edited, no complex command lines and no extra package managers installed on my system.

10/10 would recommend!

Edit: it looks like the certbot handled the "aigarius.com" domain perfectly, but missed the redirects from "www.aigarius.com", so I'd have to fix those up manually by just copying the directive that certbot added to nginx config file for "aigarius.com" and editing the domain name.

Ryzen 7000 amdgpu boot hang

Aigars Mahinovs — Sun, 16 Oct 2022 14:15:13 GMT

So you decided to build a brand new system using all the latest and coolest tech, so you buy a Ryzen 7000 series Zen 4 CPU, like the Ryzen 7700X that I picked, with a new mother board and DDR5 memory and all that jazz. But for now, you don't yet have a fitting GPU for that system (as the new ones will only come out in November), so you are booting a Debian system using the new build-in video card of the new CPUs (Zen 4 generation has a simple AMD GPU build-in into every CPU now - great stuff for debugging and mostly-headless systems) and you get ... nothing on the screen. Hmm. You boot into the rescue mode and the kernel message stop after:

Oct 16 13:31:25 home kernel: [    4.128328] amdgpu: Ignoring ACPI CRAT on non-APU system
Oct 16 13:31:25 home kernel: [    4.128329] amdgpu: Virtual CRAT table created for CPU
Oct 16 13:31:25 home kernel: [    4.128332] amdgpu: Topology: Add CPU node

That looks bad, right?

Well, if you either ssh into the machine or reboot with module_blacklist=amdgpu in the kernel command line you will find in /var/log/kern.log.1 those messages and also the following messages that will clarify the situation a bit:

Oct 16 13:31:25 home kernel: [    4.129352] amdgpu 0000:10:00.0: firmware: failed to load amdgpu/psp_13_0_5_toc.bin (-2)
Oct 16 13:31:25 home kernel: [    4.129354] firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware
Oct 16 13:31:25 home kernel: [    4.129358] amdgpu 0000:10:00.0: firmware: failed to load amdgpu/psp_13_0_5_toc.bin (-2)
Oct 16 13:31:25 home kernel: [    4.129359] amdgpu 0000:10:00.0: Direct firmware load for amdgpu/psp_13_0_5_toc.bin failed with error -2
Oct 16 13:31:25 home kernel: [    4.129360] amdgpu 0000:10:00.0: amdgpu: fail to request/validate toc microcode
Oct 16 13:31:25 home kernel: [    4.129361] [drm:psp_sw_init [amdgpu]] *ERROR* Failed to load psp firmware!
Oct 16 13:31:25 home kernel: [    4.129432] [drm:amdgpu_device_init.cold [amdgpu]] *ERROR* sw_init of IP block <psp> failed -2
Oct 16 13:31:25 home kernel: [    4.129525] amdgpu 0000:10:00.0: amdgpu: amdgpu_device_ip_init failed
Oct 16 13:31:25 home kernel: [    4.129526] amdgpu 0000:10:00.0: amdgpu: Fatal error during GPU init
Oct 16 13:31:25 home kernel: [    4.129527] amdgpu 0000:10:00.0: amdgpu: amdgpu: finishing device.
Oct 16 13:31:25 home kernel: [    4.129633] amdgpu: probe of 0000:10:00.0 failed with error -2

So what you need is to get a new set of Linux Kernel Firmware blobs and upack that in /lib/firmware. The tarball from 2022-10-12 worked well for me.

After that you also need to re-create the initramfs with update-initramfs -k all -c to include the new firmware. Having kernel version 5.18 or newer is also required for stable Zen 4 support. It might be that a fresh Mesa version is also of importance, but as I am running sid on this machine I can only say that Mesa 22.2.1 that is in sid works fine.

Debconf 22 photos

Aigars Mahinovs — Fri, 22 Jul 2022 13:54:13 GMT

Finally after a long break, the in-person Debconf is a thing again, this time Debconf 22 is happening in Prizren, Kosovo.

And it has been my pleasure to again be here and take lots of pictures of the event and of the surroundings.

The photos can be found in this Google Photo shared album and also on this git-lfs share.

But the main photographic delight, as always is the DebConf 22 Group Photo. And here it is!!!

You can also see it in:

Debconf 19 photos

Aigars Mahinovs — Tue, 23 Jul 2019 17:07:42 GMT

The main feed for my photos from Debconf 19 in Curitiba, Brazil is currently in my GPhoto album. I will later also sync it to Debconf git share.

The first batch is up, but now the hardest part comes - the group photo will be happening a bit later today :)

Update: the group photo is ready! The smaller version is in the GPhoto album, but full version is linked from DebConf/19/Photos

Update 2: The day trip phtos are up and also the photos are in Debconf Git LFS share.