We tried three popular job runners before writing our own. Each was good at running code, but none could promise that a step with side effects would run exactly once, even through a crash.
The one promise that mattered
Our runtime writes a receipt before and after every step that touches the outside world. If a worker stops between the two, the next worker reads the receipt and knows whether to finish the job or leave it alone.
It is a small idea, but it is the reason a refund is never sent twice.
