IoT Core Outage

Incident Report for ClearBlade

Postmortem

Yesterday, ClearBlade IoT Core experienced an outage due to a bug in its monitoring and deployment automations. ClearBlade has been consistently improving its deployments to be more automated, self-healing, improved security, higher conformance to access policies of least privilege and with richer internal reporting. These improvements have allowed ClearBlade to scale and provide higher support. These improvements work at low levels dynamically enabling containers and scaling architecture up and down. Yesterday one of those automated processes encountered an issue based on a very long uptime that sent the cluster into a restart state. This restart state was well understood by the engineering team but required a low-level intervention. This meant our security processes and access processes had to be followed.

Posted May 09, 2025 - 19:37 UTC

Resolved

This incident has been resolved.
Posted May 09, 2025 - 14:03 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted May 09, 2025 - 00:38 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted May 09, 2025 - 00:00 UTC
This incident affected: ClearBlade IoT Core (Europe-West1, US-Central1, Asia-East1).