đŸĒļ Merciful.ai Status

âš ī¸ All Systems Degraded

Service Checks

✅

billing

Billing service operational

Last check: 2026-10-15T22:38:00.152678+00:00

âš ī¸

disk

Disk warning: /: 83% full (24.8GB free); /shared: 83% full (24.8GB free); /var/lib: 83% full (24.8GB free)

Last check: 2026-10-15T22:40:08.584886+00:00

✅

email

Email service operational

Last check: 2026-10-15T22:37:59.813994+00:00

✅

jobs

All systems operational

Last check: 2026-10-15T22:40:08.162032+00:00

âš ī¸

pods

canary/sites-service-7876df949f-fz7v8 pending

Last check: 2026-10-15T22:40:08.584781+00:00

âš ī¸

runners âŗ confirming 1/2

1 in deep sleep, 4 degraded

Last check: 2026-10-15T22:40:12.339534+00:00

Namespace: prod

✅
text-runner-prod-01 (ollama:9091) healthy
ollama: healthy (failures: 0) | Last: 2026-10-15T22:39:57 | Avg: 5.0s | 10min: 9 completed | Total: 2432/0 ok/fail | Processing: ff53897f9a
✅
text-runner-prod-01 (ollama:9091) healthy
ollama: healthy (failures: 0) | Last: 2026-10-15T22:39:06 | Avg: 5.1s | 10min: 2 completed | Total: 2500/0 ok/fail
✅
text-runner-prod-01 (ollama:9091) healthy
ollama: healthy (failures: 0) | Last: 2026-10-15T22:31:24 | Avg: 9.1s | 10min: 2 completed | Total: 2486/0 ok/fail
❓
openai-text-runner-prod-01 (openai:9091) probing
openai: probing (failures: 2) | Last: 2026-10-15T22:36:50 | Avg: 14.4s | 10min: 1 completed | Total: 509/0 ok/fail | Processing: ba490bcb16
❓
openai-text-runner-prod-01 (openai:9091) deep_sleep
openai: deep_sleep (next probe in 295s, 5 failures, 0 probes) (failures: 5) | Last: 2026-10-15T22:37:14 | Avg: 12.6s | 10min: 5 completed | Total: 407/0 ok/fail | Processing: 322f79bb27
âš ī¸
openai-text-runner-prod-01 (openai:9091) degraded
openai: degraded (backoff 11s, 4 failures) (failures: 4) | Last: 2026-10-15T22:29:21 | 10min: 0 completed | Total: 490/0 ok/fail | Processing: 322f79bb27
❓
openai-text-runner-prod-01 (openai:9091) probing
openai: probing (failures: 2) | Last: 2026-10-15T22:35:38 | Avg: 3.4s | 10min: 1 completed | Total: 480/0 ok/fail | Processing: 38103e8999
âš ī¸
openai-text-runner-prod-01 (openai:9091) degraded
openai: degraded (backoff 8s, 2 failures) (failures: 2) | Last: 2026-10-15T22:39:11 | Avg: 5.1s | 10min: 3 completed | Total: 397/0 ok/fail | Processing: 4067508210
✅
openai-text-runner-prod-01 (openai:9091) healthy
openai: healthy (failures: 0) | Last: 2026-10-15T22:18:43 | 10min: 0 completed | Total: 488/0 ok/fail

Namespace: canary

✅
text-runner-canary-01 (ollama:9091) healthy
ollama: healthy (failures: 0) | Last: 2026-10-15T20:19:24 | 10min: 0 completed | Total: 31/0 ok/fail
✅
openai-text-runner-canary-01 (openai:9091) healthy
openai: healthy (failures: 0) | Last: 2026-10-15T20:25:14 | 10min: 0 completed | Total: 19/0 ok/fail

Incident History

degraded jobs: prod: prod: 22% abandon rate (2/9)
✅ Resolved · Started 2026-10-15T16:01:29.562172+00:00 (duration: 2026-10-15T16:02:36.003524+00:00)
down email: prod: prod: unknown; canary: canary: unknown
✅ Resolved · Started 2026-10-15T13:25:52.735060+00:00 (duration: 2026-10-15T13:31:25.255816+00:00)
down email: prod: prod: orders-up unreachable: Get "http://orders-up.canary.svc.cluster.local:3032/health/email": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
✅ Resolved · Started 2026-10-15T03:00:26.416056+00:00 (duration: 2026-10-15T03:05:56.173018+00:00)
down email: prod: prod: orders-up unreachable: Get "http://orders-up.canary.svc.cluster.local:3032/health/email": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
✅ Resolved · Started 2026-10-14T21:30:14.371133+00:00 (duration: 2026-10-14T21:35:43.798315+00:00)
down jobs: prod: prod: jobs-api timeout
✅ Resolved · Started 2026-10-14T21:21:00.056031+00:00 (duration: 2026-10-14T21:22:15.555694+00:00)
degraded pods: canary/sites-service-7876df949f-fz7v8 pending
🔴 Active · Started 2026-10-14T21:15:02.635934+00:00
degraded disk: Disk warning: /: 80% full (28.6GB free); /shared: 80% full (28.6GB free); /var/lib: 80% full (28.6GB free)
🔴 Active · Started 2026-10-14T21:10:14.744286+00:00
down email: prod: prod: orders-up unreachable: Get "http://orders-up.canary.svc.cluster.local:3032/health/email": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
✅ Resolved · Started 2026-10-14T21:07:55.588166+00:00 (duration: 2026-10-14T21:13:56.358935+00:00)
degraded runners: no completions for 7 min
✅ Resolved · Started 2026-10-14T12:07:34.333717+00:00 (duration: 2026-10-14T12:08:41.115381+00:00)
down email: prod: prod: orders-up unreachable: Get "http://orders-up.canary.svc.cluster.local:3032/health/email": context deadline exceeded (Client.Timeout exceeded while awaiting headers); canary: canary: orders-up unreachable: Get "http://orders-up.canary.svc.cluster.local:3032/health/email": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
✅ Resolved · Started 2026-10-14T09:00:18.162028+00:00 (duration: 2026-10-14T09:05:47.202980+00:00)