IoT Core - All Regions - Device debug logs not being received for some Google projects

Incident Report for ClearBlade

Postmortem

A change made to Logs Explorer resource types caused problems with IoT Core Logging starting on May 28, 2024 blocking of IoT Core logs flowing. ClearBlade was not informed prior to this change. Official Google documentation lists cloudiot_device as a valid resource type. After communicating with Google it was recommended that ClearBlade fully drop the use of the IoT Core resource usage types.

On June 3rd, ClearBlade rolled out a fix to all regions by replacing the resource type cloudiot_device with global. Logs immediately appeared as devices reconnected.

Going forward users should be aware of the following when querying their Google logs:

Any Logs Explorer queries using resource.type="cloudiot_device" will have to be replaced by resource.type="global".
Previously resource.labels used to contain 4 items:
project_id
device_registry_id
location
device_num_id
Now resource.labels only contains project_id. The contents of jsonPayload.resourceName is of the format projects/{projectId}/locations/{location}/registries/{registryId}/devices/{device_num_id}. The jsonPayload.resourceName can be parsed to obtain the items that are now missing missing from resource.labels.

Clicking “View Logs” from the IoT Core user interface still includes the incorrect resource type. This is planned to be fixed on the next scheduled upgrade.

Posted Jun 07, 2024 - 20:30 UTC

Resolved

All users have reported this issue resolved.

Posted Jun 06, 2024 - 10:25 UTC

Monitoring

All regions have received the update. Device logs will appear as they reconnect over the next 24 hours. The new resource type is "global"

Posted Jun 03, 2024 - 21:31 UTC

Update

ClearBlade has begun rolling out an update with logs set to have a resource.type of global. Devices that reconnect in Asia should start seeing logs in their environments. Europe, then the US, will also update within the next 24 hours.

Posted Jun 03, 2024 - 19:26 UTC

Identified

Multiple users have reported that device logs are not being received for some of their registries.

All devices are connected and data is flowing to Google PubSub.

When logging ClearBlade is writing the log but receiving a response of unsupported type error for resource type "cloudiot_device". ClearBlade has escalated internally to confirm the removal of this resource type. Currently the resource type is still listed here - https://cloud.google.com/logging/docs/api/v2/resource-list

ClearBlade does have a patch to log this entry to the global resource. If this would impact your business logic, please open a ticket with ClearBlade. Any updates will be announced prior to implementation.

Posted Jun 03, 2024 - 11:14 UTC

This incident affected: ClearBlade IoT Core (Europe-West1, US-Central1, Asia-East1).