Das Inoffizielle Android-Handbuch
Das Inoffizielle Android Systemhandbuch
Die besten Android-Apps
Reproducible Builds, special client support and more in our repo
Today we have some exciting news for you. Let me mention the „biggest fish“ first:
For several months now, the IzzyOnDroid repository has support for Reproducible Builds – but we decided to not make that public before we’re out of the „POC phase“:
Reproducible Builds
Wait: RBs at IzzyOnDroid? Isn’t that a binary repository, serving the APKs built and signed by their resp. developers? Yes, indeed it is, and will remain so. But is this a contradiction? As you will see, it is not. With the help of Android Reproducible Builds expert Fay (aka obfusk), who in the past, in her spare time also helped to establish them at F-Droid.org which uses and depends on her tooling1 to this very day, we found a different approach to this, which has proven to work pretty well.
So how do Reproducible Builds work with a repository taking the APK files from the resp. developers? By using independent „verification builders“. For this, Fay has created rbtlog, which pulls the APK files from the apps’ repositories, checks out the code from the very same tag, builds the APK files from source – and then checks if the resulting APKs are identical with the ones shipped by the developers. If so, you’ll find it pointed out for that release in the packages section of the app, where it will look like this:
(click the screenshot for a bigger variant)
If you hover your mouse over such a badge, you will see details on which builder confirmed this – and the badge should link you to the details (aka „proof“). Currently, two builders are in operation2 (a third one is being set up as I write this). One is run by Fay herself, the other by Izzy:
I started IzzyOnDroid back in 2013, originally with the app listings. In July 2014 the blog was added and, finally, in 2016 the F-Droid style repository for Android apps we talk about here. Much of my free time goes into IzzyOnDroid. I enjoy helping people this way to improve their privacy, security and safety on their Android devices – be it those using the apps, or those developing them. I’m also IzzyOnDroid’s „outreach person“, and the one writing this article (apart from the personal quotes of course).
Btw: Anyone can run such a builder – after all, it’s open source and available under a libre license. It’s also well documented (the only issues I had getting it up and running were rather Python-related: I wanted to use a venv with my existing system packages instead of installing all dependencies from PyPI in a separate venv, or the additional ones that had no *.deb system-wide. Once that was solved, all else went smooth as butter!). Of course we gladly help you with it when needed.
Fay, from the rbtlog project, has been great at helping ensure Catima is reproducible, giving extra certainty that Izzy ships exactly what I intend to send to users. I’m quite happy to see just how much thought was put into this implementation, logging exactly the environment it was built on as well.
I wrote „hover your mouse over it“ – and I am aware there is no such thing on touchscreens (well, sometimes long-tap and then hitting „back“ works). But no worries, you are soon covered on your mobile devices as well; see the exciting news about client support below.
In short, for those who now wonder: what good is that? And what does it mean if an app has Reproducible Builds? Well, it tells you that the APK was indeed built from the very source (at Codeberg, Github, GitLab etc) it claims to be built from. That nothing was hidden from the source repo, nothing was „injected“ into the final APK. So this is another security feature. On the other hand: should a build have failed to be reproducible, this does not necessarily mean it was tampered with. Quite often it’s simple things like the developer having built it not from the tag’s commit but somewhere between commits – which then of course is hard or even impossible to reproduce. If you are the developer of an app (to be) listed at IzzyOnDroid, and you want it to succeed as RB: we’ve prepared this wiki page for you with some hints.
Note and disclaimer: at the time of this writing, about 18% of the apps (more than every sixth app) available at the IzzyOnDroid repo already have Reproducible Builds. We’re working hard to increase this number. However, this will never cover all apps: some simply cannot achieve Reproducible Builds, others we cannot even build ourselves. Nor can we guarantee to „keep pace“ with future changes in the Android tooling (in the past, there were several adjustments needed on our end for changes made by Google to e.g. AGP (the Android Gradle Plugin). So we cannot guarantee to keep this „running smoothly forever“ (though we’ll of course try our very best) – but neither will releases be delayed or even „broken“ should things go wrong with RB.
Monitoring
Now, this is no secret: the availability of the services for the IzzyOnDroid repository is monitored, and you can see their status anytime for yourselves. This is possible thanks to our excellent SysAdmin, Sylvia:
I believe good monitoring is crucial to a project’s reliability. Computers are inherently complex and even the most well-designed system can sometimes fail. Good monitoring allows you to quickly notice problems and remediate them. Over the long-term, monitoring can help show patterns that can help you find flaws and improve them for even further increased general reliability.
Sylvia has set up Uptime Kuma for IzzyOnDroid using Ansible (ensuring the configuration is applied automatically and consistently to remove the chance of mistakes and so it can be quickly restored if needed), and is paying for its server out of her own pockets (no, we currently have no „big funding“). This already helped us being aware of outages early, and fixing them before you’ve noticed them (at least that’s what we hope). In this context, maybe I should also mention redundancy?
Mirrors
Also no secret and available for quite a while publicly are our mirrors. One is operated by Codeberg in Germany, the other by our supporter Andrew in the US:
Reproducible Builds are crucial for the security and safety of open source software because they allow anyone to recreate the exact same binary code from the source code, ensuring that there are no hidden vulnerabilities or backdoors introduced during the build process. This is important for open source projects as it increases trust and transparency, since users can verify that the software they are using is built from the same source code that is publicly available. Reproducible Builds also help to catch bugs and errors early in the development cycle, as any discrepancies between the expected and actual output can be easily identified and traced back to the source. I’m happy to provide a mirror and builder for this project.
These mirrors are announced with the index shipped to the Android clients – which means, should one server (main repo or mirror) become unreachable, your Android client can offer you to obtain the data (and apps) for you from another. Some clients do so automatically (e.g. the official F-Droid client), others let you switch manually (and some might not have mirror support unfortunately). More on this again in the section on client support below.
Client support
We are really excited to announce this, and much appreciate the commitment of those involved! The IzzyOnDroid team started to work together with LooKeR from Droid-ify. His words on the topic here:
Improving security and privacy standards in FOSS Android applications is very important. It’s a crucial area that directly impacts user experience and trust. It is pretty obvious that people deserve better privacy, security and freedom over what they use.
Some of our team love his client and use it as their primary one for a long time already. Others prefer Neo Store from Toni and his team. And what shall I say? We’re proud to have Toni aboard as well! And he’s looking even further forwards:
These are just the first steps in extending the definition and demonstrating the commitment to security and privacy in open source Android applications. Stay tuned for the upcoming extensions and models that will result in richer privacy and security reporting in the clients.
(click the screenshot for a bigger variant)
So what does that mean to you, who uses the IzzyOnDroid repo and one of these two fine Android clients? You will get much of the above details and more, transparently available in your client. We’re currently working on this together: the IzzyOnDroid team to provide the data needed, and the client teams to make them available to you. Soon, when looking at the details of an app in your favorite client (at least if it’s one of these two), you will not only see its description, screenshots and links, but also…
- whether the app’s release was confirmed as reproducible build, and by which builder(s)
- what libraries where detected inside the APK (results from the IzzyOnDroid scanners, as you already see them in the repo browser)
- what other scanners have to say (e.g. VirusTotal)
And that shall just be the start. A Ferret’s Work Is Never Done I’ve posted not that long ago on the Fediverse – so we hope to deliver more exciting news in the future. As usually, when it’s available (the news; well, the future too). We don’t want to make promises if we are not sure we can deliver – which we don’t know before we’ve „POCed“ them.
Community
All this is the joint effort of multiple projects working together for the good of the community. It’s not just about IzzyOnDroid, though the IzzyOnDroid repo was clearly in the focus here. But here we joined forces: the IzzyOnDroid team, the two client projects, Fay with her collection of tools for Reproducible Builds: we were meeting up to find out how to improve security and transparency for you. We all hope we succeeded there – and will „succeed even more“, as the work is never done. And we will continue working together.
There was no „monetary incentive“ involved. Other than other projects, IzzyOnDroid has no „big funding“. There for sure is no „venture capitalist“ involved (and never will be). But neither is any Grant covering our work. So if there are expenses, we cover them out of our own pockets: Sylvia for the monitoring server, Izzy for the other IzzyOnDroid servers. Thankfully some of the community help us covering here.
If you want to help out at this place: we could use some „server space“ for (more) verification builders. Also to run testing/staging instances to test out new ideas on. Mirrors, especially in regions not yet covered, would be welcome, too. Of course not only that: if you have other ideas how to help, please reach out to us! Other contributions are welcome as well. Let us know what you’re good at – and we see where it would fit in 
But we also feel it important to say thanks to you: thank you for trusting us, thank you for staying with us – and thank you for reading this post up to this last line 
-
which is btw also used by Tor Browser, Element and Threema; Signal has expressed interest as well. The tooling a.o. includes apksigcopier and reproducible-apk-tools ↩︎
-
Fay’s builder is hosted at Github, Izzy’s builder publishes to Codeberg; number 3 will be run by Andrew at sourcehut. Different forges again also add to resilience – to ensure not all builders disappear should one location „be down“. ↩︎