GCP: Receive Alerts on Google Chat for Resource Creation / Deletion

Introduction

Google Cloud Platform (GCP) provides an enormous suite of services designed to monitor and manage vast fleets of cloud infrastructure efficiently. However, as cloud footprints expand across decentralized engineering teams, a common and critical enterprise requirement emerges: the need to receive real-time alerts whenever a new resource is provisioned or deleted within your project.

In this practical engineering guide, we will walk through architecting a modern, event-driven GCP Cloud Function pipeline designed to instantly push infrastructure alerts directly into a Google Chat space using Cloud Logging sinks and Pub/Sub.

The Challenge: Sprawl and Shadow IT

Without automated visibility into infrastructure lifecycle events, organizations quickly fall victim to "cloud sprawl" and Shadow IT. Developers might spin up expensive, oversized Compute Engine instances for testing and forget to terminate them, leading to ballooning monthly billing surprises. Worse, unauthorized deletion of critical Cloud Storage buckets or Cloud Run services can cause catastrophic, silent production outages if the operations team isn't immediately notified.

Common Pitfall: Relying solely on daily billing exports or manual console audits to track newly created resources. By the time a billing report flags a massive GPU instance, it has already been running for 24 hours. Real-time logging is the only acceptable defense.

The Solution/Process: Event-Driven Alerting Pipeline

This solution relies on intercepting Google Cloud Audit Logs in real-time. We will set up a log filter, route matching logs securely to a Pub/Sub topic, and trigger a Serverless Cloud Function that instantly fans out parsed notifications to Google Chat.

Prerequisites

• A GCP Project with Logging Admin, Cloud Function Admin, and Pub/Sub Admin IAM roles assigned to your user account.
• A dedicated Google Chat Space where you want to route the incoming alerts.

Step 1: Configure a Google Chat Webhook URL

First, set up a custom inbound webhook in your target Google Chat Space:

1. Click on the Dropdown Menu next to the Space name > Space Settings > Manage Webhooks.
2. Create a new webhook (named "GCP Resource Alerts") and copy the provided Webhook URL.

Pro-Tip: Treat this Webhook URL like a highly sensitive production password. If leaked, anyone on the internet can POST arbitrary malicious messages into your corporate chat space. We will secure it using Environment Variables.

Step 2: Route Audit Logs to a Pub/Sub Topic

Configure a Log Sink to strictly intercept specific Google Cloud Admin Activity logs and export them to a Pub/Sub topic:

1. Navigate to Logging > Logs Router in the GCP Console and click Create Sink.
2. Name the Sink new_resource_lifecycle_sink.
3. Choose Cloud Pub/Sub topic as the Sink Destination and create a new topic named resource-alerts-topic.

Provide the following advanced inclusion filter to elegantly capture Compute, Cloud Run, Cloud SQL, and Cloud Functions resource lifecycle events while aggressively ignoring noise:

protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog" AND NOT protoPayload.authenticationInfo.principalEmail:("gserviceaccount.com")
( protoPayload.methodName="v1.compute.instances.insert" AND protoPayload.authorizationInfo.permission="compute.instances.create") OR
( protoPayload.methodName="google.cloud.run.v1.Services.CreateService" AND protoPayload.authorizationInfo.permission="run.services.create") OR
( protoPayload.methodName="cloudsql.instances.create" AND operation.first="true") OR
( protoPayload.methodName="v1.compute.instances.delete"  AND protoPayload.authorizationInfo.permission="compute.instances.delete" ) OR
( protoPayload.methodName:"cloudsql.instances.delete" AND operation.first="true")

Note: We deliberately exclude logs triggered by automated service accounts by filtering out "@gserviceaccount.com" to prevent alert fatigue. Adjust this depending on your CI/CD requirements.

Step 3: Deploy the Cloud Function Webhook Parser

Deploy the "glue" code: a Cloud Function that listens to the Pub/Sub topic, transforms the raw Audit Log JSON into a readable Google Chat Card, and securely POSTs it.

1. Navigate to Cloud Functions. Click Create Function.
2. Name it chat-alert-parser, select 2nd gen environment, and set the trigger to the resource-alerts-topic Pub/Sub topic.
3. Select Python 3.9 (or newer) as the Runtime.

Security Configuration: Dissect your massive Google Chat Webhook URL into three parts: https://chat.../spaces/{SPACE_ID}/messages?key={KEY}&token={TOKEN}. Add SPACE_ID, KEY, and TOKEN as strictly defined Environment Variables in the function setup.

Use the following robust Python logic in main.py:

import os
import base64
import json
import requests
import logging

logging.basicConfig(level=logging.INFO)

GOOGLE_CHAT_SPACE_ID = os.environ.get("SPACE_ID")
GOOGLE_CHAT_API_KEY = os.environ.get("KEY")
GOOGLE_CHAT_TOKEN = os.environ.get("TOKEN")

DELETE_METHODS = [
    "v1.compute.instances.delete",
    "cloudsql.instances.delete",
    "google.cloud.run.v1.Services.DeleteService",
]

def format_chat_message(extracted_data, is_delete):
    action_type = "Resource Deleted" if is_delete else "New Resource Created"
    color = "#FF0000" if is_delete else "#00FF00"
    
    return {
        "cards_v2": [{
            "card": {
                "header": { "title": f"{action_type}" },
                "sections": [{
                    "widgets": [
                        {"text_paragraph": {"text": f"Principal: {extracted_data.get('principal', 'Unknown')}"}},
                        {"text_paragraph": {"text": f"Resource: {extracted_data.get('resource_type', 'Unknown')}"}},
                        {"text_paragraph": {"text": f"Project ID: {extracted_data.get('project_id', 'Unknown')}"}}
                    ]
                }]
            }
        }]
    }

def send_message(event, context):
    try:
        message_data = base64.b64decode(event['data']).decode('utf-8')
        payload = json.loads(message_data)

        url = f"https://chat.googleapis.com/v1/spaces/{GOOGLE_CHAT_SPACE_ID}/messages?key={GOOGLE_CHAT_API_KEY}&token={GOOGLE_CHAT_TOKEN}"
        
        principal = payload.get("protoPayload", {}).get("authenticationInfo", {}).get("principalEmail", "N/A")
        res_type = payload.get("resource", {}).get("type", "N/A")
        project_id = payload.get("resource", {}).get("labels", {}).get("project_id", "N/A")
        
        method_name = payload.get('protoPayload', {}).get('methodName', '')
        is_delete = method_name in DELETE_METHODS
        
        extracted_data = {"principal": principal, "resource_type": res_type, "project_id": project_id}
        chat_message = format_chat_message(extracted_data, is_delete)
        
        requests.post(url, headers={'Content-Type': 'application/json'}, json=chat_message)
    except Exception as e:
        logging.error(f"Function execution failed: {e}")

Define dependencies in requirements.txt:

requests==2.31.0

Deploy the function.

Step 4: Live Testing

To empirically test the architecture, manually spin up an inexpensive disposable Compute Engine VM instance. Because the Sink filter captures compute instances, it should immediately populate the Pub/Sub topic, trigger the function, and instantly drop a richly formatted Card natively inside your Google Chat space listing the resource type and culprit user's email.

[Insert Image: A screenshot of the Google Chat interface showing the successfully delivered formatted JSON card alert reading "New Resource Created".]

Key Takeaways

• Real-time Audit Logs: Google Cloud Audit Logs inherently capture every API call. Filtering them strictly at the Router Sink level is the most elegant way to build event triggers without polling.
• Decoupled Architecture: Pub/Sub effectively decouples log generation from log processing. If your Google Chat webhook goes down or rate-limits you, Pub/Sub will safely queue the alerts until the function can successfully process them.
• Security Stance: Placing Chat API tokens inside Cloud Function environment variables prevents accidental inclusion in version control systems (like GitHub).

Conclusion

By leveraging purely Event-Driven Architectures utilizing Google Cloud Logging Sinks, Pub/Sub, and serverless Cloud Functions, you can effortlessly wire up complex, real-time alert pipelines for essentially any metric or audit log. Setting up intelligent Chat alerts for resource creation and destruction is a hyper-effective and profoundly scalable baseline to guarantee security parity, avoid surprise billing limits, and enforce robust infrastructure governance across your entire organization.

Further Reading

• GCP Official Docs: Routing and storage overview (Log Sinks)
• GCP Official Docs: Cloud Pub/Sub architecture
• Google Workspace Developers: Setting up incoming webhooks for Chat