Automation is one key factor that lets the small F-Droid team provide apps to millions. The @checkupdates-bot is a new reworking of an old piece of automation in F-Droid: checkupdates. This goes through all of the existing apps, checks if they are set up for automatic updates, and if so, runs the process to automatically generate a new build entry for that release. That gets added to fdroiddata, which is what the production buildserver processes in order to build and ship apps.

Since the best security is provided by systems that follow the principal of least privilege, we recently reorganized our setup and workflow around that principal so that the @checkupdates-bot has the least privileges needed to do its job. The checkupdates process now runs in its own isolated project, separate from fdroiddata and any other F-Droid project on gitlab.com. It now only pushes commits to its own dedicated project, then makes a merge request per app to fdroiddata. Both our automated CI processes and our trusted human reviewers now handle all updates using the same process as for new apps.

At the same time, we removed a key bit of cruft in our code: stats/known_apks.txt. This file was the place that stored the dates when each app was added to the collection. This file was updated on the buildserver and maintained in fdroiddata. That information is also in the index file, so we switched to fetching it from there instead. That meant we could remove the last deploy key in use in fdroiddata. Our operations no longer require any deploy keys in fdroiddata.

While we were at it, we added some additional checks via the merge requests. For example, now any time an image file is added or modified, a CI job checks whether the image contains any EXIF metadata, which can be used as an exploit vector. We also added some additional enforcement to make sure changes to key files go through human review via merge requests.

Security issue as inspiration

About a month ago, @SomberNight reported a security issue to us in a confidential issue. We appreciate this detailed report, and also want to highlight their diligent follow up. In specific situations, the old setup was leaking the private deploy key which granted access to directly push commits to fdroiddata. We immediately revoked that key, then removed all privileges from the @fdroidci user that was associated with that private key. We also investigated all the leads we could follow to see if someone had used this key to insert something into F-Droid. We searched the activity of the @fdroidci user and found no evidence that unauthorized commits were added.

To be sure, we did some additional investigations. Since checkupdates had been running as part of the fdroiddata project on gitlab.com, a malicious app build recipe could have also read the CHECKUPDATES_SSH_DEPLOY_KEY variable which contained the private key. We checked fdroiddata’s history for signs of exfiltration and found nothing. We require that apps are built from source code, and that source code is in a source code management system like Git. That ensures a local copy with history is retained on our buildserver. We searched our local copy source code and found no evidence that any app build processes were trying to exfiltrate the checkupdates private key.

Do you have more ideas for things to search? Please dig in and let us know if you find anything suspicious. Working in public means everyone is free to investigate and come to their own conclusions, and contribute to a more secure free software ecosystem on Android.