Aigarius Blog (Posts about idea)

Distributing third party applications via Docker?

Aigars Mahinovs — Thu, 25 Sep 2014 19:09:00 GMT

Recently the discussions around how to distribute third party applications for "Linux" has become a new topic of the hour and for a good reason - Linux is becoming mainstream outside of free software world. While having each distribution have a perfectly packaged, version-controlled and natively compiled version of each application installable from a per-distribution repository in a simple and fully secured manner is a great solution for popular free software applications, this model is slightly less ideal for less popular apps and for non-free software applications. In these scenarios the developers of the software would want to do the packaging into some form, distribute that to end-users (either directly or trough some other channels, such as app stores) and have just one version that would work on any Linux distribution and keep working for a long while.

For me the topic really hit at Debconf 14 where Linus voiced his frustrations with app distribution problems and also some of that was touched by Valve. Looking back we can see passionate discussions and interesting ideas on the subject from systemd developers (another) and Gnome developers (part2 and part3).

After reading/watching all that I came away with the impression that I love many of the ideas expressed, but I am not as thrilled about the proposed solutions. The systemd managed zoo of btrfs volumes is something that I actually had a nightmare about.

There are far simpler solutions with existing code that you can start working on right now. I would prefer basing Linux applications on Docker. Docker is a convenience layer on top of Linux cgroups and namespaces. Docker stores its images in a datastore that can be based on AUFS or btrfs or devicemapper or even plain files. It already has a semantic for defining images, creating them, running them, explicitly linking resources and controlling processes.

Lets play a simple scenario on how third party applications should work on Linux.

Third party application developer writes a new game for Linux. As his target he chooses one of the "application runtime" Docker images on Docker Hub. Let's say he chooses the latest Debian stable release. In that case he writes a simple Dockerfile that installs his build-dependencies and compiles his game in "debian-app-dev:wheezy" container. The output of that is a new folder containing all the compiled game resources and another Dockerfile - this one describes the runtime dependencies of the game. Now when a docker image is built from this compiled folder, it is based on "debian-app:wheezy" container that no longer has any development tools and is optimized for speed and size. After this build is complete the developer exports the Docker image into a file. This file can contain either the full system needed to run the new game or (after #8214 is implemented) just the filesystem layers with the actual game files and enough meta-data to reconstruct the full environment from public Docker repos. The developer can then distribute this file to the end user in the way that is comfortable for them.

The end user would download the game file (either trough an app store app, app store website or in any other way) and import it into local Docker instance. For user convenience we would need to come with an file extension and create some GUIs to launch for double click, similar to GDebi. Here the user would be able to review what permissions the app needs to run (like GL access, PulseAudio, webcam, storage for save files, ...). Enough metainfo and cooperation would have to exist to allow desktop menu to detect installed "apps" in Docker and show shortcuts to launch them. When the user does so, a new Docker container is launched running the command provided by the developer inside the container. Other metadata would determine other docker run options, such as whether to link over a socket for talking to PulseAudio or whether to mount in a folder into the container to where the game would be able to save its save files. Or even if the application would be able to access X (or Wayland) at all.

Behind the scenes the application is running from the contained and stable libraries, but talking to a limited and restricted set of system level services. Those would need to be kept backwards compatible once we start this process.

On the sandboxing part, not only our third party application is running in a very limited environment, but also we can enhance our system services to recognize requests from such applications via cgroups. This can, for example, allow a window manager to mark all windows spawned by an application even if the are from a bunch of different processes. Also the window manager can now track all processes of a logical application from any of its windows.

For updates the developer can simply create a new image and distribute the same size file as before, or, if the purchase is going via some kind of app-store application, the layers that actually changed can be rsynced over individually thus creating a much faster update experience. Images with the same base can share data, this would encourage creation of higher level base images, such as "debian-app-gamegl:wheezy" that all GL game developers could use thus getting a smaller installation package.

After a while the question of updating abandonware will come up. Say there is is this cool game built on top of "debian-app-gamegl:wheezy", but now there was a security bug or some other issue that requires the base image to be updated, but that would not require a recompile or a change to the game itself. If this Docker proposal is realized, then either the end user or a redistributor can easily re-base the old Docker image of the game on a new base. Using this mechanism it would also be possible to handle incompatible changes to system services - ten years down the line AwesomeAudio replaces PulseAudio, so we create a new "debian-app-gamegl:wheezy.14" version that contains a replacement libpulse that actually talks to AwesomeAudio system service instead.

There is no need to re-invent everything or push everything and now package management too into systemd or push non-distribution application management into distribution tools. Separating things into logical blocks does not hurt their interoperability, but it allows to recombine them in a different way for a different purpose or to replace some part to create a system with a radically different functionality.

Or am I crazy and we should just go and sacrifice Docker, apt, dpkg, FHS and non-btrfs filesystems on the altar of systemd?

P.S. You might get the impression that I dislike systemd. I love it! Like an init system. And I love the ideas and talent of the systemd developers. But I think that systemd should have nothing to do with application distribution or processes started by users. I am sometimes getting an uncomfortable feeling that systemd is morphing towards replacing the whole of System V jumping all the way to System D and rewriting, obsoleting or absorbing everything between the kernel and Gnome. In my opinion it would be far healthier for the community if all of these projects would developed and be usable separately from systemd, so that other solutions can compete on a level playing field. Or, maybe, we could just confess that what systemd is doing is creating a new Linux meta-distribution.

Wireless photo workflow

Aigars Mahinovs — Thu, 03 Apr 2014 10:04:14 GMT

For a while now I've been looking for ways to improve my photo workflow - to simplify and speed up the process. Now I've gotten a new toy to help that along - a Panasonic FlashAir SD card with WiFi connectivity. I was pretty sure that build-in workflows of some more automated solutions would not be a perfect fit for me, so I got this card which has a more manual workflow and a reasonable API, so I could write my own.

Now I am trying to work out my requirements, the user stories if you will.

I see two distinct workflows: live event and travel pictures.

In both cases I want the images to retain the file names, Exif information and timing of the original photos and also have embedded GPS information from the phone synced to the time the photo was taken. And if I take a burst of very similar photos, I want the uploading process to only select and upload the "best" one (trivial heiristic being the file size) with an ability for me to later choose another one to replace it. There would need to be some way of syncing phone and camera time, especially considering that phones usually switch to local time zone when traveling and cameras do not, maybe the original time the photo was taken would need to be changed to local time zone, so that there are no photos that are taken during the day, but have a timestamp of 23:45 GMT.

When I am in Live Event mode I would like the photos that I take to immediately start uploading to an event album that I create (or choose) at the start of the shoot with a preset privacy mode. This assumes that either I am willing to upload via 3G of my phone or that I have access to a stable WiFi network on-site. It might be good if I could upload a scaled down version of the pictures during the event and then later replace the image files with full-size images when the even is over and I am at home in my high-speed network. I probably don't need the full size files on my phone.

When I am in Travel mode, I want to delay photo uploading until I am back at the hotel with its high speed Wifi, but also have an option to share some snapshots immediately over 3G or random cafe Wifi. I am likely to take more photos that there is memory in my phone, so I would like to clear original files from the phone while keeping them in the SD card and in the cloud, but still keeping enough metadata to allow re-uploading an image or choosing another image in a burst.

Now I need to flesh out the technical requirements from the above and write an Android app to implement that. Or maybe start by writing this in Python as a cross-platform command-line/desktop app and only later porting it to Android when all the rought parts are ironed out. This will have an extra benefit of people being able to run the same workflow on a laptop instead of a phone or tablet.

Let's assume that this is written in a pretty flexible way, allowing to plug in backends for different WiFi SD cards, cloud services, plug-in points for things like instant display of the latest photo on the laptop screen in full-screen mode and other custom actions, what else would people love to see in something like this? What other workflow am I completely overlooking?

Crazy ideas - higher education tax

Aigars Mahinovs — Sun, 25 Nov 2012 15:11:38 GMT

There is a strong discussion in Latvia right now on how to evaluate and fund higher education (university education, bachelor degrees and up) in a fair way that has the best advantage to the society as a whole as well as the students and the universities.

On one hand you would want as many people as possible to get best education possible - the lifetime productive output of a person is highly correlated with the amount and quality of education they get. So it is beneficial to the society to just pay for everyone’s education. However, such approach diminishes the quality as both students and universities focus on quantity and forget about quality.

There are many different mechanisms of measuring quality and even trying to force quality on universities (certification programs, mainly) and students (stipends or tuition discounts based on performance), but I have the feeling that all of these approaches focus too much on the performance of the student during the learning process, while the real interest to the society is the performance of the student after graduation.

So here is a crazy idea - give a fully free education to everyone in the country. And to fund it have a 1% payroll (and capital gains) tax for everyone that got a degree under this program, but the twist is that money should be tracked per-person back to the specific university where they got their degree. And that (eventually) would be the only source of government funding for the universities. Thus it will be in the best and direct interest of every university to make sure that every graduate gives as much values as possible to the society over their lifetime, so that the university can take is 1% from that. Interestingly this also encourages the universities to grow some respect for a healthy lifestyle and safety into their students, so they live longer and more productive lives on average :)

There are quite a lot of kinks to work out: What about people who do not graduate? Should government mandate some kind of a basic knowledge level from the graduates to prevent universities just juicing people for their 1%? What to do with master degrees and PhDs? What if those are taken in a different university? Add another percent? What about the transition period? How would new universities enter the market?

Cinematic dream

Aigars Mahinovs — Fri, 16 Dec 2011 05:12:07 GMT

Sometimes I have these cinematic dreams, and I have decided that if I for some reason wake up right after it and still remember it vividly enough, I should write it down. So yesterday I had one of those. What follows is more of a polished description of how I understood the dream, almost a skeleton of a movie script.

It starts with a small boy around age ten in a black fancy suit running away from two men, a thin and a fat one (aka. Home Alone), it is a hotel corridor, he runs up to the elevators and press the call button, the button is round small polished chrome button in a round chrome inset (the button is reccurring), the boy gets in the elevator and it starts going down, the elevator has no doors nor a back wall, as it goes down the walls of the building goe back and it starts descending into a cave like structure (blue lighting, cold, sense of unrealism starts).

The boy is sad, the elevator stopps suddenly a meter from the bottom and would not react to buttons, the other elevator nearby starts working, the men are chasing him, the boy panics, jumps out of the elevator, looks around, finds two pairs of brass buttons, in each pair one is with white and other with red rim. He presses the red one in the pair next to the second elevator and hears it stop, he presses the white button on his own elevator, it starts rising up, he jumps and barely catches it, climbs in.

The elevator puts the boy out on the roof, but it is not just a roof, it almost feels like a mountain top with uneven surface, moss, sheer drop-off on all sides and even puddles that look almost like icelandic geiser puddles, the boy is angry with this, shouts his anger into the wind, when suddenly the men are there too. the boy looks where to run, but there is no other way to go. then he spots again the same shiny metal button right next to a curiously round puddle. he stands in the puddle and presses the button.

The floor of the puddle sinks, taking the boy with it, he holds his breath until he can no longer, he is under water, but he sees a female figure made of light call him forward, he swims to it and emerges to the surface.

He is in a fountain in front of a victorian style castle, the fountain pool is rectangular and clearly only 20-30 cm deep. the boy was scared under water, but now he a bit more calm and sure of himself. the men appear again, ok now I will run away from them, the boy runs away from the castle into the garden, the takes random turns in the green labyrinth, the men follow closely, the green slowly gives way for yellow (autumn) leaves, then vines and then brick walls replace the garden, now he is running inside a building, in front of him there is a stained glass window displaying the same white female figure with light shining through it (for some reason there are glass doors that he needs to open with the shining button here in the dream), the men chasing him almost catch up to him, he and the slender man crash trought the stained glass window together.

And fall into the same fountain pool in front of the castle. the boy and the slender man look at each other and laugh.
the exposition moment happens here - the slender man is the boys' father and the boy was running away from him and from reality because his mother died and he just could not accept it. It took going trought these troubles, these stages of grief live and smile again.

And then the end sciene - the boy and his father join hands and spin in circles around each other (to the tune of The Passenger by Iggy Pop) while the background changes back to the hotel where the shocked relatives from the funeral look on to them being so happy, then to a marriage ceremony where the father marries again, then to son growing up, then the son marries, and the music suddenly stops with the grown up son standing over two graves - the father has died too, but then a little boy comes over - his own son. he puts his hand on his sons shoulder and The Passenger music starts again.

The END.

So the idea is clearly of a boy going trough stages of grief while running away, the stages were for some reason illustrated with elements (metal, air, water, life, earth) and with seasons (winter, spring, summer, autumn) and not with classical stages of grief. I am not sure of the significance of the fat man. I doubt one can make a full movie out of this, but this could be a fun little short, possibly an animated one.

P.S. In the dream the Passenger song only sounded as the Lalalala part that starts at 1:15 here and as I could not quite remember what the song or the author was, so it took quite a titanic effort to find the right song with 'lalalala' lyrics.

MorzeSMS

Aigars Mahinovs — Fri, 11 Feb 2011 14:02:36 GMT

Quick post. In light of recent Nokia+Microsoft-MeeGo news, I have gone to learn more about Android in a hurry. And here is the first result - MorzeSMS.

Basically it is a tiny app that will play a morze code when you receive an SMS message. The morze code is the phone number of the sender in 'cut number' morze form (to be shorter). The idea is that first of all morze is a cool sound and second is that each sender gets a unique sound that you can learn to identify over time so that you know who sent you a message as soon as you hear the beep.

This is very early beta - there is no UI, no configuration. You can download it here. Leave bug reports in the comments of this blog post. It does not disable the stock SMS-received sound.

Re: HOWTO prep for migration off of SHA-1 in OpenPGP

Aigars Mahinovs — Wed, 06 May 2009 11:05:45 GMT

Daniel says that we should move away from SHA1 by switching hash algorithms for signatures and generating keys that use at least SHA256 from SHA-2 family. I have been bitten by non-default GPG options before. So I propose that we do a security release of GPG that changes the defaults of key generation and key signing in such ways that SHA-1 algorithms are not used by default for any operation, unless a backwards compatibility option is used.

Voting on emails?

Aigars Mahinovs — Sun, 21 Dec 2008 02:12:09 GMT

What does it sound like when a commission is reinventing a wheel? Much like this. Guys, the technology you are looking for is called (*gasp*) a forum, with moderators and poster carma and such ...

It would not be too hard to implement a gateway that would receive emails posted to any list and republish then in a web forum where then moderation and voting can take place. Email is unsuited for this because it is impossible to moderate and vote after the message has been sent out (and delaying emails for moderation/voting is a huge burden on communication).

Copyright infringement is like walking into a concert without a ticket

Aigars Mahinovs — Mon, 01 Dec 2008 21:12:21 GMT

In light of Ted's post on copyright, it is clear that we are bogged down by a hostile terminology.

Copyright infringement is more like sneaking into a concert. You enjoy the show, but did not pay for it. If the concert is held in a small venue then it is easy to spot someone who gets in without paying, but if you hold a concert in an open field, then expecting everyone in the surrounding area to be forced to pay you is rather absurd. If you have a concert in a field with a fence around it and someone makes a hole in that fence (posts a a recording to a P2P site), then the one making the hole might be punishable (property damage), but can any person that enters that hole really be prosecuted for theft, property damage and enabling further theft by not closing the hole (uploading that is inherent in the P2P protocols) ? I do not think so.

A similar analogy is using public transport without a ticket. If there is a bus that goes from A to B and I get on it without buying a ticket from the driver, then in the current copyright enforcement world I would get arrested by the police, prosecuted and get a fine that is tens of thousand times more than the price of the ticket. This has multiple problems - 1. the police has no way of knowing if I have some kind of legal right to use the bus without a ticket (monthly ticket, free ride for seniors, ...) before arresting me and bringing me to court, the police has no business wasting their time and taxpayer money until it is 100% certain that a crime has actually occured; 2. if a hacker disables the ticket composters in the bus and removes all signs about prices how am I to know that the bus is not free (like the park and ride buses in many locations); 3. the fines are excessive - I've not seen a public transport fine that is much more then 10 times the price of the ticket, and it only applies to one ride - you can not be retroactivelly fined for all free rides you took in the last year; 4. it actually is not possible to be 100% certain who is the person doing the act - you can only trace the IP which can be used by any number of computers and additionally the computers might be infected with a botnet acting as an unwilling proxy zombie. There is no way (except a confession) to prove that a particular person does a particular download.

Now we just need a short and simple word or phrase that describes that. Any ideas?

Slowfs

Aigars Mahinovs — Mon, 24 Nov 2008 12:11:07 GMT

I have just encountered a bug in Gnome that is much more visible when the hard drive is slow or overloaded and responds slowly. This gave me an idea - how about a simple transparent FUSE filesystem that does nothing else than delay, slow down and possibly reorder filesystem requests? Such a filesystem would be very useful for debugging. We developers tend to have high-performance systems and that hides many bugs, but if we could have a slower system on-demand, it will give us the ability to debug our applications better.

Even better would be a virtual machine where one could arbitrarily slow down any specific aspects of the system - slower CPU, slower hard drive, slower RAM, slower network, ...

I know that engineers of network hardware have special devices that emulate long distance and noise - you take two wireless network cards and plug cables into their antennae sockets and then plug those cables into the device. The device then makes a connection between those cables, but with emulation of a variable distance, variable antennae and variable environmental noise, so the wireless cards can be tested for distance and noise resistance right in the lab. We could do the same in software for HDD, CPU, RAM and networking - imagine having two Xen instances and putting an emulated network between them with 10 hops, 1 second latency, 20% average packet drop traffic shaping and limited top bandwidth of 100 kbit/s. Now that is a real test platform for any kinds of applications that have anything to do with networking (or HDD, CPU or RAM).

Anyone up to making that? Could someone of you use a system like that?

24th September as a day against software patents - Debian support?

Aigars Mahinovs — Fri, 29 Aug 2008 22:08:50 GMT

According to the Slashdot article and the StopSoftwarePatents.org website itself and on Digg, the anti-software-patent activists are attempting a world-wide event on the 24th of September as a world-wide day against software patents. USA has them via a weird court ruling, Japan has them as well (not sure why), there have been efforts to force software patents on EU, India, Australia and many other countries either by Microsoft lobbies or even via US trade treaty pressure.

While lobbying by local Microsoft branches and their pet companies can be countered locally, like I and many others did in EU a couple years ago (mostly thanks to FFII), trade negotiations are very secretive affairs and it is very hard to lobby there directly. The public needs to be aware of the issue, otherwise the politicians will not be aware of its importance to their voters.

Software patents are a threat to free software as it circumvents the power of GPL and other copyright licenses if enough money is thrown at the lawyers. If a software patent is generic enough, it can easily stop development of a whole class of free software applications on a whim of the patent holder. And there are plenty of granted software patents with a very broad scope (progress bar, anyone?).

I am aware that to get Debian support for such an initiative, a GR is needed, but how about a personal word of support from the DPL or support from the SPI? It is in direct area of interest for SPI - software patents create an ever present threat of legal action against any and all software in Debian (in USA and Japan, at least) and, as the legal umbrella of Debian in the USA, SPI would be a prime target.

What do others think about this?