Das Inoffizielle Android-Handbuch
Das Inoffizielle Android Systemhandbuch
Die besten Android-Apps
Review of 2024 and Outlook for 2025: Reproducible Builds, Security Measures and more
In January/February 2024, we were Ramping up security, putting additional APK checks in place with the IzzyOnDroid repo. A new year starts, and it’s been five months since we officially went live with our Reproducible Builds (see: Reproducible Builds, special client support and more at IzzyOnDroid). What has happened since? How is it working out? What have we learned?
Security checks
Of course we didn’t stop at establishing the scans:
- Since July 2024, the IzzyOnDroid repo no longer has any APKs using debug certificates. Most apps moved to proper release keys for signing. Others were removed from our repo as the developers could not be reached, or the project was dead – in some cases even the repo had disappeared, together with its account.
- Whenever an app update triggered the scanners with sensitive permissions, flags or intent filters being detected, we reached out to its developers. Often, permissions were „dragged in“ unintentionally by some dependency and then were removed, remaining details have been clarified. By now, most apps should have the proper details listed for each APK in the apps’ „Packages“ section of our repo browser (see screenshot). So this did not only lead to more transparency, but often also to less permissions requested by apps.
- While reaching out to the developers for that, whenever an app also contained BLOBs in APK signing blocks, we‘ve been mentioning that along, together with hints on how to avoid those. Almost everytime (with a single exception so far), this issue was addressed as well.
- Several security patches have been applied to the fdroidserver installation at IzzyOnDroid in April. Of course we provided them „upstream“ after having them tested here, but they were not wanted there (F-Droid.org instead used a different implementation written by themselves, covering most of it but unfortunately breaking support for apps using Signing Key Rotation; such apps are still supported at IzzyOnDroid, though).
Reproducible Builds
Statistics
When we went live with our RBs on August 1st, 2024, 18% of our apps were reproducible – one out of six. In October of the same year, just two and a half months later, we reached the milestone of 25%: one out of four apps reproducible. The 30% mark was reached in early December, and numbers still keep growing.
We have multiple verification builders now, each one run by their own operator. Some are run by our team, others by folks from the community who want to support us (or to keep an eye on us what we really ship and confirm? Doesn’t matter: independent checks are important, so be welcome to join!). Coverage differs: operators decide by themselves which apps they want to build and verify. The builders operated by our team of course focus on apps provided in the IzzyOnDroid repo, but the builders are not restricted to that. As some apps from our repo are also available via F-Droid.orgs repo, those are covered as well then – provided they are set up for Reproducible Builds there, too (our builders verify the APKs built and signed by their resp. developers, while apps not set up for RB at F-Droid are signed by a key F-Droid generated and thus are not identical). As of this writing, there are three builders „registered“ with our repository browser. In alphabetical order of their operators:
- Ben runs bg443 since 10/2024. This is an independent builder.
- Fay runs obfusk since 2/2024. This is the original builder (Fay wrote the software), but currently in maintenance mode.
- Izzy runs izzy since 7/2024. This is the currently largest builder.
At the „izzy builder“, we currently see about 10 apps having their updates checked each day.
No Set-and-Forget
It’s not a simple „set it up and it keeps running“, and we knew that before. While I am writing this, four out of the 370 RB apps in my builder waited to be „fixed“ – one just was fixed in cooperation with its developer, so three are left waiting. And it’s not just us, F-Droid.org has the same problems: just scan Github for issues titled „F-Droid can’t build“…
Things evolve. Developers continuing to work on their apps, change things, add/replace libraries, maybe even switch frameworks. New versions of AGP (the Android Gradle Plugin), Android buildtools and other frameworks used in app development fix old bugs (so recipes for new versions of apps might need to have their work-arounds removed) and introduce new ones (which we then need to work around).
Some frameworks and tools require their versions to be kept in sync between the builds of developers and the RB verification builders. We don’t have a complete list for that, but classic examples include Flutter and Rust. But even a different version of cmake can cause an app to fail RB – or the update of a dependency drags something in. Sure, when setting up the initial recipe for an app, we try to address everything – together with the developers. But when a new tool enters the game, it might be one bringing up such an issue – or just needs an update to our build recipes. So recipes evolve with new releases, they don’t remain in a „frozen state” or fix themselves magically.
So do several Linux distributions. Fedora went to replace zlib by zlib-ng (and it looks like Arch did the same), causing issues with RB as their compression cannot be easily reproduced: the data compressed with zlib-ng changes depending on how the uncompressed data is fed into the compressor – unlike with zlib. Which makes recreating data effectively impossible unless one uses the exact same code. What makes it even more „fun“ is that depending on the toolchain (statically linked versus dynamically linked binaries) some files in the APK are still compressed using zlib, while others using zlib-ng… This „fun“ might increase should Debian follow up and go zlib-ng, which is discussed in their mailing lists and even stated to affect RB: „APKs built on a Debian system will have a different compressed stream […] Which will likely break Reproducible Builds tooling“. And even worse, „programs that used to produce identical and deterministic output with zlib no longer do with zlib-ng, despite using the exact same zlib-ng .so“.
Like security measures, making apps reproducible is no Set-and-Forget. It requires constant care.
Cooperation required
At IzzyOnDroid, we don’t like „power structures“ – we prefer cooperation. So it’s not a „report it to the devs and have them fix it“ – it’s a „try to fix it yourself first; if that does not work out, reach out to the devs and ask how we can fix it together“. And this affects establishing RBs for an app the first time, as well as fixing them when they break.
This working-together not only proved helpful to establish or fix RBs. Both sides learned a lot in the progress. Seeing our interest in helping them proving their apps RB, developers where eager to learn more about what it means as well as how it works. Sometimes they established the necessary steps for us (even with build scripts we could call from our recipes). Others switched to e.g. Github workflows to build their apps cleanly and consistently, using our recipes as guidance. They helped us to understand their tooling and improve our recipes, while we were able to help them improving their own builds. And both sides enjoyed the process and the new insights won.
Should you be a developer, have your app listed at IzzyOnDroid, and want to establish Reproducible Builds for it, be welcome to reach out to us – either in your own issue tracker, or in ours. We will do our best to help you.
Outlook
Thanks to our supporters, we soon can use new hardware: the server running our Reproducible Builds will get a hardware upgrade, and a new mirror will be established which in the end might become our new staging setup. So let us express our thanks explicitly to all of you who support us, but especially, in alphabetical order:
- Aayush Gupta (the Android Developer and FOSS enthusiastic whom you may know as maintainer of AuroraStore) became a sponsor on our OpenCollective, helping us cover regular expenses such as hosting/server costs.
- Andrew Lewman who runs a mirror of IzzyOnDroid and will soon provide us with a dedicated machine for one of our builders
- BlissLabs, who became our Fiscal Host at OpenCollective and also might support us with infrastructure in the future
- Codeberg which hosts much of our infrastructure and currently runs another of our mirrors (and yes, we plan to deepen our cooperation and collaboration)
- Everybody else who helped us in any way, be it small donations, checking old apps and reporting those no longer working (e.g. as the service behind them disappeared or broke its API; you know the video platform I have in mind there), reporting changed URLs, suggesting new apps, just sending us a warm heads-up/thanks as „toot“ to @IzzyOnDroid@floss.social…
You all are awesome and give us the motivation to carry on: Thank you!
One more thing we’re currently working on, and which was requested multiple times in the past, are download statistics. We will hopefully be able to make daily stats with weekly reports available soon. Data will be provided in JSON format. If someone wants to develop a frontend for it (Android app or website page) and wants a data sample ahead of time, please let us know!
In September 2024, we’ve established our OpenCollective. The „in-flow“ is still relatively small, as is our team. We will continue running IzzyOnDroid in our spare time (as of this writing, none of us is being paid for any of the work on the project). If you’re able to, contributions to our OpenCollective would be very welcome, we’re hoping to reach enough funding to allow at least one team member to work on IzzyOnDroid as a „part-time job“, instead of just in their spare time.
Verdict
As with security measures, maintaining Reproducible Builds is a continuous process, requiring constant attention and care. It’s not always easy, but we think it’s worth the efforts. For those using our repository it means they can verify whether an app was really built from the very source it claims, with nothing injected and nothing taken away – which is the case when the green shield is up. As the process is not always trivial, the green shield is sometimes delayed (while we try to fix a failed recipe), or a yellow shield comes up instead signaling a failed RB (which cannot be fixed for that release). Even that does not mean danger; sometimes an APK cannot be made RB due to minor mistakes in the build process (a file forgotten to check in, a build made from a „dirty tree“ with local changes or artifacts remaining from previous builds). We’re usually already in contact with the corresponding developers to help making the next release pass again.
For those curious: we’re also maintaining wiki pages with hints for developers (like best practices, hints on what to take care for to avoid breaking RB for your app, or even on how to set up and run your own verification server) which you might wish to take a look at.