7 Architecture Red Flags We Find in Reviews

The same handful of issues shows up in almost every architecture review I run, whatever the company size or stack. Below are the seven I flag most often, with how I check for each one and how I'd fix it, followed by the process I use to run the review itself.

1. No standby for the primary database, and analytics running on it

One managed database instance with no Multi-AZ standby (or regional HA) carries both the transactional traffic and the analytical queries. Read replicas help with read scaling, but promoting one is a manual failover that can lose data, so they don't give you high availability. A zone outage or instance failure means full downtime, and a single heavy report can starve checkout of I/O and connections.

How to spot it. Check whether the instance has a standby:

# AWS RDS: is there a synchronous standby? (Aurora: look for a replica in another AZ)
aws rds describe-db-instances \
  --query 'DBInstances[].[DBInstanceIdentifier,Engine,MultiAZ]' --output table
 
# Cloud SQL: REGIONAL means HA with a standby, ZONAL means none
gcloud sql instances list --format="table(name,databaseVersion,settings.availabilityType)"

Then check who's querying the primary. On PostgreSQL, pg_stat_activity grouped by application_name usually shows the BI tool right next to the application.

The fix. Turn on Multi-AZ (multi_az = true on aws_db_instance) or regional HA (availability_type = "REGIONAL" on google_sql_database_instance) in Terraform. Move analytics to a read replica, or preferably to an export into a warehouse. For cross-region failover, see PostgreSQL multi-region failover.

2. IAM granted at the project or account level

Broad roles (roles/owner, roles/editor, AdministratorAccess) are handed to humans, CI pipelines and workloads, when each of them should get least-privilege access scoped to specific resources. One leaked CI token or compromised pod then has access to everything.

How to spot it. List who holds the broad roles:

# GCP: who holds basic roles on the project?
gcloud projects get-iam-policy my-project \
  --flatten="bindings[].members" \
  --filter="bindings.role:roles/owner OR bindings.role:roles/editor" \
  --format="table(bindings.role,bindings.members)"
 
# AWS: who has AdministratorAccess attached?
aws iam list-entities-for-policy \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

The fix. Define bindings in Terraform at the resource level (a bucket, a secret, a queue), with one service account or role per workload. GCP's IAM Recommender and the unused-access findings in AWS IAM Access Analyzer show which permissions are never used, so you can remove them in a PR.

3. No RTO/RPO targets, and no restore test to prove them

Nobody can tell me how long it would take to get back up after losing the database (RTO) or how much data would be lost (RPO), and nobody has restored from backup in over a year. Having backups "enabled" answers neither question. If nobody has tested a restore, the first real one is when you find the missing permissions, or learn that it takes all afternoon.

How to spot it. Ask for the targets and the date of the last restore test, then check retention and the restore point:

aws rds describe-db-instances \
  --query 'DBInstances[].[DBInstanceIdentifier,BackupRetentionPeriod,LatestRestorableTime]' \
  --output table

The fix. Write down an RTO and RPO per service and agree them with the business. Then check them with a scheduled restore test into an isolated environment:

target="restore-test-$(date +%Y%m%d)"
start=$(date +%s)
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier prod-db \
  --target-db-instance-identifier "$target" \
  --use-latest-restorable-time \
  --db-subnet-group-name restore-test   # an isolated subnet group, not prod's
aws rds wait db-instance-available --db-instance-identifier "$target"
echo "Restore took $(( ($(date +%s) - start) / 60 )) minutes"

Your measured RTO is the restore time plus the time to point an application at the restored instance and pass a smoke test. Your RPO is the gap between LatestRestorableTime and now. Compare both with the targets, record the result, and delete the test instance.

4. A "temporary" cron job or VM running something critical

A VM someone created by hand two years ago runs the invoicing job from a crontab, outside infrastructure as code. When that VM dies, nobody knows what it did or how it was configured, and often nobody knew it existed.

How to spot it. Tag everything Terraform creates (for example managed-by = "terraform" through the AWS provider's default_tags or GCP labels), then list what doesn't carry the tag:

gcloud compute instances list --filter="-labels.managed-by:terraform"

On AWS, the required-tags AWS Config managed rule reports untagged resources. On each machine you find, crontab -l and systemctl list-timers show what it runs.

The fix. Move the job into a scheduled service defined in code (Cloud Scheduler plus Cloud Run jobs, EventBridge Scheduler plus Lambda or ECS, or a Kubernetes CronJob). Alert when it doesn't run, as well as when it fails.

5. Synchronous call chains that should be events

A user request goes through four or five services in series, each waiting on the next, even for work the user doesn't need to wait for (emails, notifications, analytics). Availability multiplies along that chain. Five dependencies at 99.9% each give you roughly 99.5% end to end, before retries amplify the load.

How to spot it. Open the service map in your tracing tool (Cloud Trace, X-Ray, or any OpenTelemetry backend) and look at how deep the critical path goes. Then, for each hop, ask what fails if it's down for ten minutes.

The fix. Keep a call synchronous only when the response depends on it, and move the rest behind a queue or event bus. Event-driven architecture with Lambda and EventBridge walks through one way to build that.

6. No timeouts, circuit breakers or load shedding

HTTP clients run with default (often infinite) timeouts, retries have no backoff, and nothing at the edge rejects excess load. One slow dependency then ties up every worker upstream.

How to spot it. Grep the codebase for where HTTP clients are created and check for explicit timeouts, then look at the gateway or mesh config. If you run Istio, kubectl get destinationrules -A often comes back empty.

The fix. Put an explicit timeout on every outbound call, add retries with jitter and a retry budget, and use circuit breaking. With Istio, outlier detection ejects failing endpoints:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payments
  namespace: prod
spec:
  host: payments.prod.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Pair it with rate limiting at the load balancer or API gateway so excess requests are rejected quickly instead of queuing.

7. Architecture diagrams that don't match reality

The documentation describes the system as it looked a year or two ago.

How to spot it. Compare the diagram with an inventory generated from the cloud itself, such as gcloud asset search-all-resources --scope=projects/my-project or AWS Resource Explorer, and with terraform state list.

The fix. Keep diagrams as code (Mermaid or similar) in the same repo as the Terraform, and update them in the PR that changes the architecture.

A framework for running your own review

I run every review in the same four passes:

1. Inventory. Generate the list of what exists from the cloud APIs rather than from memory or the wiki, and diff it against IaC state and the diagrams (red flags 4 and 7).
2. Failure-domain walk-through. For each component on the critical path, ask what happens if it dies, if its zone goes down, or if it gets slow. Write down what the system does today, which may differ from what was designed (flags 1, 5 and 6).
3. Restore test. Restore the most important data store and time it against the RTO/RPO targets (flag 3).
4. Cost and ownership check. Every resource should have an owner label and a team that gets paged for it, and every broad IAM grant should have a named justification (flag 2).

Rank the findings by blast radius and likelihood, give each one an owner and a ticket, and re-run the review twice a year or after any major change. In my reviews the gap between the diagrams and what's deployed comes up most often, and it's the kind of problem an outside reviewer tends to catch.