AI Platform Engineer
Infrastructure risks routed to owners before execution.
Detect reliability, security, cost, and deployment risks. Explain impact, recommend safe fixes, and require approval before any change runs.
Risk Inbox
AI-detected infrastructure risks
Demo Failure For CloudOps workflow failure
Demo Failure For CloudOps on main ended with failure and needs platform review before another release attempt.
- Repository: AZ1600/cloudops-command-center
- Workflow run #1
- Branch: main
- Conclusion: failure
- Run URL: https://github.com/AZ1600/cloudops-command-center/actions/runs/28339678676
- Open the GitHub Actions run and inspect the failed job logs.
- Identify whether the failure is code, infrastructure, dependency, or secret related.
- Patch the issue in a pull request or rerun only after owner approval.
Terraform drift detected in security group
A production security group was changed outside Terraform and now allows SSH from a wider CIDR range.
- Terraform plan shows ingress drift
- Port 22 changed from office CIDR to 0.0.0.0/0
- Resource managed by networking module
- Verify whether emergency access was approved.
- Revert the SSH ingress CIDR back to the approved range.
- Run Terraform plan against the networking workspace.
Unattached volumes increasing monthly spend
Several unattached EBS volumes have not been used for more than 30 days.
- 8 unattached gp3 volumes
- Estimated waste: $148/month
- No read/write activity in 30 days
- Tag each unattached volume with owner and review deadline.
- Create a final snapshot for rollback safety.
- Delete volumes with no owner response after approval.
Kubernetes workload is crash looping
The ingestion worker restarted 17 times in the last hour and is failing readiness checks.
- CrashLoopBackOff on worker-ingestion-7d9
- Readiness probe failed
- Queue depth increased by 42%
- Check previous container logs for the first failing stack trace.
- Compare the current image tag with the last healthy deployment.
- Scale a canary worker after applying the fix.
S3 bucket policy allows broad public access
A storage bucket used for customer uploads has a policy that permits public object reads.
- Block Public Access is disabled
- Policy contains Principal '*'
- Bucket stores uploaded customer documents
- Confirm the bucket does not intentionally host public website assets.
- Enable account and bucket-level Block Public Access.
- Update application access to use pre-signed URLs or CloudFront signed URLs.
Production deployment failed
The latest main branch deployment failed after a database migration step timed out.
- GitHub Actions run #184 failed
- Migration job exceeded 10 minute timeout
- Production still running previous release
- Open the failed GitHub Actions run and inspect the migration logs.
- Check database lock and connection timeout metrics.
- Patch the migration or increase the safe timeout.
Owners
Routing map
- Risks
- 2
- Urgent
- 2
- Waiting
- 2
- Done
- 0
- Risks
- 1
- Urgent
- 1
- Waiting
- 1
- Done
- 0
- Risks
- 1
- Urgent
- 1
- Waiting
- 1
- Done
- 0
- Risks
- 1
- Urgent
- 1
- Waiting
- 1
- Done
- 0
- Risks
- 1
- Urgent
- 0
- Waiting
- 1
- Done
- 0
Approval Queue
Safe execution gate
Approved remediations will appear here before execution.
Execution Log
Approved changes
Executed remediations will appear here with safety checks and command evidence.
Audit Log
Decision history
8 workflow runs reviewed for AZ1600/cloudops-command-center. 1 failed runs generated risks for owner approval.
6 workflow runs reviewed for AZ1600/cloudops-command-center. 0 failed runs generated risks for owner approval.
5 infrastructure risks restored for review.
pull request remediation approved for Cloud Platform. Execution is queued behind the safety gate.
5 infrastructure risks detected and routed to owners.
Service Catalog
Owned platform assets
payments-api
- Owner
- Platform Team
- Runtime
- FastAPI on ECS
- Last change
- Deployment failed 18 minutes ago
customer-assets
- Owner
- Security Owner
- Runtime
- S3 + CloudFront
- Last change
- Bucket policy changed today
worker-ingestion
- Owner
- Data Platform
- Runtime
- Kubernetes deployment
- Last change
- Image rollout 42 minutes ago
analytics-cluster
- Owner
- FinOps Owner
- Runtime
- EC2 + EBS
- Last change
- Volume audit 2 days ago
networking
- Owner
- Cloud Platform
- Runtime
- Terraform module
- Last change
- Manual security group edit detected
Integrations
Infrastructure signal sources
GitHub Actions
- workflow runs
- deployment failures
- release history
Fetch workflow runs by repository and route failed deployments to owners.
AWS
- S3 posture
- cost findings
- resource inventory
Add read-only IAM role for CloudWatch, Cost Explorer, and IAM Access Analyzer.
Kubernetes
- pod health
- restart counts
- readiness failures
Connect cluster service account with read-only workload access.
Terraform
- drift signals
- plan output
- module ownership
Import Terraform plan JSON from CI before apply.
Prometheus / Monitoring
- alerts
- SLO burn rate
- latency anomalies
Connect alert webhook after the core remediation flow is stable.
GitHub Actions
Live workflow failure detection
Terraform Import
Plan JSON risk detection
Runbooks
Controlled remediation procedures
Public S3 Access Remediation
Applies to customer-assets and is currently linked to an active risk.
- Confirm the bucket is not intentionally hosting public website assets.
- Verify customer uploads can be served through signed URLs.
- Run Terraform plan before changing the bucket policy.
- Revert the bucket policy pull request.
- Restore the last known-good access policy from version control.
- Re-run object access tests for customer downloads.
Kubernetes CrashLoopBackOff Response
Applies to worker-ingestion and is currently linked to an active risk.
- Read previous container logs before restarting pods.
- Check image tag against the last healthy release.
- Confirm queue backlog can tolerate a canary restart.
- Roll deployment back to the previous image tag.
- Scale workers to the last stable replica count.
- Requeue failed jobs after readiness checks pass.
Failed Production Deployment Recovery
Applies to payments-api and is currently linked to an active risk.
- Review failed migration logs and database lock metrics.
- Confirm production is still serving the previous healthy release.
- Require owner approval before rerunning deployment.
- Keep traffic on the previous deployment.
- Disable the failing migration step until patched.
- Open an incident note with the failed run and owner.
Terraform Drift Correction
Applies to networking and is currently linked to an active risk.
- Confirm whether the out-of-band change had emergency approval.
- Generate a Terraform plan scoped to the drifted resource.
- Validate the restored CIDR does not block approved access.
- Revert the remediation pull request.
- Reapply the previous approved security group rule.
- Attach the rollback decision to the audit log.
Unused Cloud Resource Cleanup
Applies to analytics-cluster and is currently linked to an active risk.
- Confirm unattached volumes have no read or write activity.
- Create a final snapshot before deletion.
- Notify the service owner before cleanup approval.
- Restore the deleted volume from snapshot.
- Reattach the restored volume to the approved instance.
- Update the cost exception list if the volume is intentionally retained.