Skip to content
SudoSecurity

Sections

live40 ptscloses 11/12/2026, 10:07:33 AM

Every Sunday at midnight, auth dies for an hour

Service-to-service auth fails 00:00–01:00 every Sunday. Manually rotating the token fixes it. Until next Sunday.

The scene

Reported: third Sunday in a row: 00:00–00:59, every internal API call returns 401. Human SSO is unaffected.

App log:

vault: permission denied (token expired)
  • Vault token: ttl=7d, explicit_max_ttl=7d, plus a 60-minute grace window on the consuming proxy.
  • A systemd timer vault-renew.timer runs vault token create ... > /etc/app/token every Saturday at 23:00, and its journal shows success every week.
  • The app service was last restarted three Sundays ago at 00:00, during a deploy.
  • ls -l /etc/app/token shows it was modified Saturday 23:00.

Diagnoses — hidden until the incident closes

Sign in to post your root-cause diagnosis.