We are experiencing stability problems on the system at the moment and are looking into it.
Update 2025-08-04: We were able to recover services and are observing the situation.
Update 2025-08-09: The instabilities surfaced again and we could identify the source. It is related to projects with many files which cause a lot of internal requests and memory usage and therefore take too long to respond. A patch is deployed and being monitored.
Update 2025-08-11: The initial patch did cause unintended side effects and didn’t resolve the problem and different attempts to mitigate the issue resulted in service instabilities and problems with accessing the storage backend which prevented some users from uploading files. We have now identified the main source of troubles and are deploying an improved and tested hotfix.
Update 2025-08-12: All systems are running stable and memory usage is on normal levels. In a follow up of this incident, we have improved our monitoring systems to detect such problems earlier.
Update 2025-08-15: Summary of the issue
QFieldCloud experienced a series of performance and availability degradations in the last week, starting from August 4th until late August 11th.
While this immediately caught our attention, it took us until August 8th to discover the root cause.
The root cause was excessive database queries for projects that have more than a few hundred files, which caused overall service degradation. While the affected APIs were for uploading and downloading files, the heavy workload on the database caused sporadic problems with the service as a whole.
We worked on finding solutions immediately; however, due to the complexity of the issue, the solutions were not stable enough, which further degraded the service in the early hours of August 11th. We reverted to the previous version of QFieldCloud in the afternoon on the 11th.
Later on August 11th, we deployed a new version of QFieldCloud that fixed the scalability issue for bigger projects, resulting in much less pressure on our servers.
We are closely monitoring our systems to confirm everything is working as intended. From the lessons learned, we have already improved our monitoring and prioritized measures to ensure the overall availability of the service.
Apologies for the inconvenience.