Five million emails on a stolen key

A client forwarded us an invoice from their email delivery provider. It was for roughly $2,000. Their normal month is a rounding error next to that.

The account had sent about five million emails. Not the client. Someone holding a copy of the client's API key, sending through their reputation, on their bill.

We killed the key, rotated everything the project touched, and pulled the provider logs the same day. The money was recoverable. The domain reputation took longer.

A month later, near enough, the same shape at a different client. That time we were watching for it and caught it early, before a bill existed at all.

Two in one month is not a coincidence. It is the new baseline, and this post is what we changed because of it.

What the attack actually was

There is no clever exploit in this story, and that is the point.

A key with send permissions ended up somewhere reachable: in a repository that was public for a while, in a client-side bundle, in a .env copied to a staging box, behind a management port left open to the whole internet on a server that was set up years ago and never revisited. Any one of those. The interesting part is not which.

What was different from five years ago is what happened next. The exposure was found and used in hours, not months. Nobody sat reading commits looking for a prize. Scanning the whole of GitHub, the whole of a netblock, every exposed endpoint on a range of IPs, and then testing each candidate credential against twenty provider APIs to see which one is live, used to be a project. Now it is a script somebody runs continuously, cheaply, with a model doing the tedious parts: reading unfamiliar code to work out what a string is for, guessing where a config file might be, writing the next probe when the first one fails.

The economics inverted. It used to be that a small company was not worth the attacker's time. Automation removed the notion of "worth the time" entirely. Your site is not being targeted. It is being enumerated, along with everything else, and the only question is whether anything of value answers.

Credentials answer. A mail API key is close to cash: it converts directly into spam volume with somebody else's deliverability paying for it. Cloud keys are better still, because they convert into compute. The market for both is liquid and immediate.

Why nothing alerted anyone

The uncomfortable part of the first incident is how long it looked completely normal from the inside.

Every request the attacker made was authenticated, authorised, and successful. The provider was doing exactly what it was told by a valid key. Monitoring saw 200s. Uptime saw uptime. Error rates did not move, because there were no errors. The application itself was untouched: nobody broke into it, nobody needed to.

We have written before about failures that report success, and credential abuse is the most expensive member of that family. There is no exception to catch. The only signals are ones nobody had been asked to watch: volume against a baseline, sending from an unexpected region, a key being used by an IP that has never used it before.

The first real notification was an invoice. That is the detection mechanism most projects have, and it arrives thirty days late.

What we check now, on every project

None of this is exotic. It is a checklist that used to be optional for a small business site and now is not. We run it at handover and then on a schedule, because a project that was clean in March is not automatically clean in August.

Secrets. Every key the project uses, listed with an owner and a purpose. The full git history scanned, not just the current tree, because a key deleted in a later commit is still sitting in the history for anyone who clones. The production bundle grepped for anything that looks like a token, since a NEXT_PUBLIC_ prefix on the wrong variable ships a secret to every visitor. Keys scoped down to what they actually need: a key that only sends transactional mail should not be able to change account settings or read the contact list.

Rotation. A written procedure, tested at least once, for revoking and replacing every key without downtime. In an incident the deciding factor is not whether you can rotate, it is whether you can do it in ten minutes at 2am without a person who is on holiday.

The perimeter. A port scan from outside, run as an attacker would. Database ports, Redis, admin panels, staging environments, old management interfaces from a previous hosting setup. Everything that does not need to face the internet moved behind a VPN or an allowlist, and everything that does forced to HTTPS with a certificate somebody is actually renewing.

Staging and old environments. In our experience this is where the bodies are. A staging copy with production credentials, no authentication, and no monitoring, running an old build with an old dependency tree. It is the softest way into a project that is otherwise well kept, and it is nobody's job on a Friday.

Access. Who can reach the server, the repository, the cloud console, the CMS. Ex-employees and finished contractors removed. Two-factor authentication mandatory on the accounts that matter, especially the domain registrar and the DNS provider, where losing control loses you everything downstream at once.

Spend and volume alarms. Hard limits on the provider account and an alert on anomaly rather than on failure. This is the single cheapest control in the list. An email account that has never sent more than 20,000 messages in a month should raise something at 50,000, not at five million. In the second incident this is precisely what fired.

Backups that have been restored. A backup nobody has restored is a belief, not a backup. We restore one and time it, so the recovery number is measured rather than assumed.

Dependencies and patches. Known vulnerabilities reviewed, and a decision recorded for each, because "no action, here is why" is a valid outcome and an unread report is not.

Four things worth doing this week, without us

If you do nothing else:

1. Scan your git history for secrets. Not the working tree. The history. gitleaks detect --no-git=false on the repository takes a minute and finds things that were "removed" years ago.
2. Set a spend cap and a volume alert on every provider that bills by usage: mail, cloud, SMS, any AI API. This one control is what turned the second incident into an afternoon instead of an invoice.
3. Port scan your own servers from outside. nmap against your production IP from a machine that is not on your network. Look at the list and ask what each open port is for. Something will surprise you.
4. Rotate anything that has ever been in a repository that was public, or in a client-side bundle, or pasted into a chat. Treat it as compromised, because there is no way to prove it is not.

What we do about it

We run this as a fixed-scope security audit on a project we did not build, and as part of delivery on every project we do. It takes a few days, it produces a prioritised list rather than a scanner dump, and the critical items get fixed rather than reported.

Our engineers and system administrators have been doing infrastructure work for years, on production systems in the UK, the US, Australia and across Europe. What changed is not their skill. It is that work which used to be a sensible extra is now the difference between a normal month and an invoice for somebody else's five million emails.

If you have an application in production and cannot immediately answer where every key lives, who can reach the server, and what would alert you if usage went up by a factor of a hundred, talk to us. The audit is the cheapest part of this whole story.