Kelvin

Appearance

Language

· 10 min read

A deployment platform Taiwan's developers loved was breached twice in two months, then shut down (Part 2): a token that never expired, and what to do now

In Zeabur's second breach, an admin token created in 2024 that never expired let an attacker read 3.2 million env vars. Part 2 covers how it happened, the shutdown timeline, refunds and claims, and six lessons for engineers.

At 09:36 on Oct 14 (Taipei time, UTC+8, for all times below), someone started using one of Zeabur's internal AWS keys. Four minutes later, they used an admin token for the shared clusters.

That token was created on Jan 10, 2024 (UTC date) and had no expiry date. In January this year it was committed in plaintext to a private config repo, and the admin interface it unlocked was exposed to the public internet. The "rotate everything" sweep after the August breach missed it.

For roughly the next 36 hours, no alert went off. Zeabur only started investigating after a user reported a problem.

(Part 1 covered what Zeabur is and how the first breach got in. This part covers the second breach, the shutdown, and what to take away from it.)

Two paths that met inside the network

  • Path A: the shared clusters. Using the plaintext AWS key and that admin token, the attacker got into the shared clusters in 9 regions from outside, read the config and secrets of every service, then ran read commands inside the containers of users' database services. n8n was the biggest target.
  • Path B: GCP dedicated servers. The GCP servers Zeabur managed for customers stored a plaintext root password in their init config, and the attacker also held an old GCP key that could read that config. Zeabur believes that is how the password was obtained. In the end, 101 servers (belonging to 92 users) were logged into as root, 50 of them had the attacker's SSH key added, and 16 copies of customer disks were attached to VMs the attacker created. Zeabur had found the root password problem on Sept 3 (UTC date) but hadn't fixed it before the attack.
  • Where the paths met: At 23:30 on Oct 15, the attacker used internal network credentials from Path A to join their own node to Zeabur's internal network from a compromised GCP server. They then used an internal "read-only" database account to read the platform database from 00:10 to 00:47 on Oct 16: 37 minutes.

Key timeline:

  • Oct 14, 09:36: First use of the internal AWS key; first use of the admin token at 09:40
  • Oct 14, 21:40: Commands start running inside user database containers; root logins to GCP servers begin at the same time
  • Oct 15, 21:56: A user report comes in, and only then does the investigation start
  • Oct 16, 00:08: First status page notice, saying only PostgreSQL and MySQL on the shared clusters were affected
  • Oct 16, 00:10–00:47: Platform database read
  • Oct 16, 04:48: Status page corrected: the impact was larger than first announced
  • Oct 16, 05:05: Last attacker activity in the records so far
  • Oct 17: Shutdown announced; preliminary report published Oct 19

How much leaked

  • 3,218,965 env vars (names and values; confirmed)
  • Account data for 321,810 users (email, username, phone number and more; confirmed)
  • 865 database services had read commands run inside them. Not every command succeeded, and how much data was actually taken is Zeabur's estimate. 736 of them were n8n instances backed by PostgreSQL.
  • 101 GCP servers logged into as root (confirmed)

The n8n numbers are the most alarming. Of the 607 PostgreSQL-backed n8n instances Zeabur checked, 602 kept their encryption key in an env var. With the key sitting next to the data it encrypts, a single read inside the container exposed every third-party credential stored in n8n, effectively in plaintext. And because the reads happened inside the container, changing the database password after August didn't help.

Zeabur says several other things cannot be confirmed and is treating all of them as leaked: whether database backups in Taipei (tpe1) and Hong Kong (hkg1) were downloaded, and whether leaked management tokens for 359 Hetzner servers and 48 DigitalOcean servers were used. The Zeabur GitHub App's private key was also read: between 23:05 on Oct 15 and 02:54 on Oct 16, anyone holding it could access the repos users had authorized. Zeabur has found no sign it was used.

What the attacker was after: AI keys

Based on the commands that were run, Zeabur believes the attacker was mainly hunting for API keys they could use immediately, especially for AI services. They searched the platform database for keys, bulk-read credentials from n8n and from new-api / one-api (gateways commonly used to proxy AI APIs), and tried calling Amazon Bedrock with Zeabur's AWS keys. Every Bedrock call failed and nothing was billed. Zeabur also stresses that this is only an observation about intent; account data was read as well.

Afterwards, one user's OpenAI key was used for about 1,107 image requests on the morning of Oct 16. Zeabur suspects this is connected to the breach but cannot confirm it.

Why a breach fixed in August came back in October

This is the most valuable part of the report. In August, the Zeabur GitHub App's private key was read, and it was replaced on Aug 31 (UTC date). But what that key could reach, a config repo full of plaintext credentials, was not rotated with it.

Zeabur's hypothesis is that the attacker used the key to read the config repo during the roughly four-day window before it was replaced. The report is explicit that this is a hypothesis that should not be cited as fact, and notes that the attacker IPs in August and October don't overlap. It also lists a few things still unexplained: the config repo was downloaded on Aug 30 and Sept 2 (UTC dates) using one employee's GitHub authorization, and in early September (Sept 1–5, UTC dates) an unknown source logged in to GCP servers with the root password.

However the credentials first leaked, the lesson is the same: the scope of a rotation has to start from "what could this leaked key reach?", not stop at the one key you know about.

Other things that made the damage worse:

  • Backups contained every secret, and the backup key was stored alongside other secrets. The most absurd moment: a backup key freshly replaced during incident response was captured by the 00:00 backup on Oct 16, and the attacker used it at 04:30.
  • A key that only needed to read one bucket could actually read all of them.
  • A "read-only" database account could read everything. Read-only doesn't mean safe.
  • A service shut down four months earlier still had a deploy key that could read the config repo.

One group did escape. Projects that followed Zeabur's guidance after August and moved their env vars onto their own dedicated servers were outside the scope of the platform database read, provided the host itself wasn't compromised. Every secret you don't hand to a central store is one fewer secret to lose.

The shutdown: timeline, refunds, and claims

When the shutdown was announced on Oct 17, many users only found out from Threads; they never got an email. The next day, Zeabur's official account replied on X (translated from Chinese): "You're right. Something as important as shutting down should have reached every user right away. We didn't do that well. Sorry."

  • Effective immediately: No new dedicated servers or domains can be purchased. You can transfer domains out yourself.
  • By Nov 30: Zeabur recommends finishing your migration. Security claims close, and so does the refund ticket window (after that, refund requests go by email).
  • From Dec 1: No renewals; dedicated servers shut down as each term ends. All shared cluster services are suspended (you can resume them manually to take backups).
  • Dec 31: All services stop and service data is deleted.

Watch out: dedicated servers rented through Zeabur are deleted when their term ends, disks included, and that can be earlier than Dec 31.

Refunds and claims are separate requests:

  • Balance refund: Apply through Zeabur Support by Nov 30; after that, by email. The cap is the cash you actually paid. Zeabur's example: top up 20, receive 100 in bonus credit, use 40, and you get 20 back. Your account is settled only after all its services have stopped, so the sooner you stop them, the sooner you're refunded. For accounts with services running until the end of December, Zeabur expects to complete payment by Jan 10, 2027.
  • Security claim: Open a separate ticket by Nov 30. Include the affected accounts and services, the types of keys exposed, and third-party bills or usage records from the incident window.

Migrating: Zeabur tells you to hand the guide to an AI

Zeabur's migration guide is written for coding agents like Claude Code and Codex. Give the agent the link and it inventories your services, recommends where to move them, and asks before any step that changes something. Even without an agent, someone has to deal with these:

  • Images live in Zeabur's registry. After the shutdown they can't be pulled, so services won't come back up after a restart.
  • *.zeabur.app domains stop working, and you'll need to handle HTTPS certificates yourself.
  • Internal .zeabur.internal connections and variables injected across services need to be rewritten for the new environment, with the actual values carried over.
  • Daily database backups will disappear. Export them first, then set up your own backups.

Zeabur says images and injected variables are where migrations fail most often. It also gives one important warning: don't paste your Zeabur token into an AI chat; let the agent use the CLI that's already logged in on your machine. A write-up by Velox Data adds one more gotcha: after restoring from a Zeabur data.sql backup, the database password reverts to the old one, so change it again once the restore is done.

Six lessons for engineers

  1. Rotate by blast radius. Replacing the leaked key isn't enough. Replace everything it could read.
  2. No admin credentials that never expire. Set expiry dates, rotate on a schedule, and keep admin interfaces off the public internet.
  3. Keep secrets out of repos and backups. A private repo is not a vault, and the backup key should live apart from other secrets.
  4. Never store an encryption key with the data it protects. As n8n showed, keeping them together is the same as not encrypting at all.
  5. Alert on valid credentials being misused. Both breaches were discovered by users. Watch for unusual sources, unusual hours, and bulk reads.
  6. Give every AI key a cap, and make each one revocable on its own. One key per project or agent, with usage limits and billing alerts.

The convenience of a PaaS is paid for with trust. Once your keys are on a platform's env var page, that platform is part of your security boundary. That doesn't mean you shouldn't use a PaaS. It means you should assume it will be breached one day, and set things up so that day costs you as little as possible.

Key takeaways from Part 2

  • Long-lived, plaintext, and usable from the internet: put all three together and one token opened the entire platform.
  • The attacker used valid credentials, so traditional alerts never fired.
  • Nov 30: security claims and refund tickets close. Dec 31: everything stops and data is deleted.
  • Assume your platform will be breached, and keep each key's blast radius as small as you can.

That's the end of the Zeabur story. The irony: Zeabur's migration guide lists Railway as one of the places you can move to, and less than a week after Zeabur disclosed its breach, Railway disclosed a security incident of its own on Oct 21. According to Railway, an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents. If there's interest, I'll take the Railway incident apart next time.

Times of day are converted from the UTC times Zeabur published to Taipei time (UTC+8). Items with a date but no time (Jan 10, 2024; January this year; Aug 30; Aug 31; Sept 2; early September; Sept 3) keep the UTC date from Zeabur's reports, so in Taipei time they could be off by a day. Article and announcement dates are the publication dates shown on the official sites.

Sources