Security Disclosure
文章洞見
原文內容目前無法安全取得，因此未產生文章摘要。
討論洞見
尚無可用的討論摘要。
代表性留言
The internet is over. We are going back to static htmls for public pages, and Point to Point VPN bases connections for B2B service provision.B2C? too risky to publish services online haha.
xtracto · HN #50189092（在新分頁開啟）
The security team at my company is working with a frontier lab to scan all our code/systems. They are in a borderline panic to close every possible issue, even innocuous ones that are buried under defense in depth. Whatever they are seeing from our partner has them spooked -- as in "acceptable risk" is no longer a thing anymore.
0x1F8B · HN #50188884（在新分頁開啟）
It's 5:20 AM here in Europe and I just rotated a bunch of keys.Only to read,> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
vintagedave · HN #50190399（在新分頁開啟）
I don’t know if this is a real question, but for most credentials a stolen key is worse than downtime on a product.
nrmitchi · HN #50191254（在新分頁開啟）
Spend limits saved me. I noticed a massive spike on one of my Anthropic keys and couldn't figure out where it was coming from. It hit the cap in one day, which isn't normal. So I rotated it just to be safe. I went to update it in Railway only to find out that I couldn't do deploys, so I also have a broken product. The good news is that it's only serving around 300 users, but still. Not great.
ryanSrich · HN #50191559（在新分頁開啟）
Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway.And it’s all just getting started imo.https://blog.cloudflare.com/emudhra-certificate-compromisehttps://www.wiz.io/blog/php2shell-php-rce
vldszn · HN #50188938（在新分頁開啟）
I think the pingcap hack might be the biggest of them all, somehow it has gone under the radar.[0] https://news.ycombinator.com/item?id=50173716
netvarun · HN #50189076（在新分頁開啟）
Also GitLabhttps://docs.gitlab.com/releases/patches/patch-release-gitla...
tom1337 · HN #50188982（在新分頁開啟）
Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up:https://x.com/wickedguro/status/2113015432974405658
theturtletalks · HN #50188471（在新分頁開啟）
Why is a 300k mrr company (his tiwtter bio) running on raleway?
lifeisloving · HN #50189061（在新分頁開啟）
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
fbrncci · HN #50188745（在新分頁開啟）
Brother do not cancel anything for these people, they do not care about you, it's their fault for not considering bus factor in their org chart. Why can't the person above you handle that?
jazzyjackson · HN #50189004（在新分頁開啟）
Prioritize your mental health, go enjoy your vacation.
jjulius · HN #50188897（在新分頁開啟）
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
MattIPv4 · HN #50188398（在新分頁開啟）
Environment variables are a terrible place to store secrets.I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.The 12 Factor manifesto was 100% wrong on this point.Use environment variables to point to where secrets are located, not to store their values.
numbsafari · HN #50188974（在新分頁開啟）
Yup, got my OpenRouter key. They managed to run up a whole $2 on Opus 5.5 and Fable. Well, time for a new hosting provider I guess...Incidentally, I'm testing Muse as an email triage agent today - it buried "oh also, railway and anthropic say yours keys have been leaked and you need to rotate asap" under 50 routine deploy / admin emails notifications. Oh to live through such an AI powered software renaissance!
piazz · HN #50190234（在新分頁開啟）
Hahaha, you're giving Muse access to a mailbox? Good luck, have fun.
phyzome · HN #50190703（在新分頁開啟）
I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.Fwiw it sounds similar to the hack over the weekend that opencode suffered.
martinald · HN #50188541（在新分頁開啟）
I already moved my big customers off Railway after an earlier incident this year that leaked customer data. It’s a stunningly horrible situation to not be able to cycle environment variables after such an event. I did an emergency migration to fly.io today with minimal disruption today. I’d recommend everyone to immediately migrate somewhere else ASAP so you can change environment variables. And of course never give Railway a cent more of your money.
disconnection · HN #50195792（在新分頁開啟）
What was the previous incident?
demibabs · HN #50196060（在新分頁開啟）
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
alpineman · HN #50188595（在新分頁開啟）
Hi Angelo from Railway here. This is very much me. https://x.com/ngeloxyz/status/2113077263386239309?s=20We have gotten a lot of comments about why didn't we just message the breach sooner when people have noticed compromise with their accounts. I think that crit is fair and justified, and although we have been up for days on end- I don't blame you if you don't have sympathy for us.Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.We take the view that we would like to "aviate, navigate, then communicate" when we deal with incidents like these then we prepared communications to the public and all users. While it was ongoing, we were in contact with external partners and vendors to revoke keys (the source of some online chatter) with investigators that required time and coordination.On the incident side of the house, it's a suite of bad options, we're still in the middle of it making sure customers are safe. But, hopefully this offers a view into our thought process.Formal postmortem after we fully restore service and complete the investigation. Apologies to all, I personally I feel like I let everyone down.
ndneighbor · HN #50189987（在新分頁開啟）
Jeez, what a rough past 24 hours for all security folk. Hang in there
DrawTR · HN #50188474（在新分頁開啟）
It's going to be a rough decade.
jacquesm · HN #50188491（在新分頁開啟）
I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.Then you have real security incidents that you need to tune into constantly.
llama052 · HN #50188880（在新分頁開啟）
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
paxy · HN #50188504（在新分頁開啟）
Defending is so much harder than attacking. You don't even have to focus on one target, as you can just mass-AI-script a lot of attacks wherever it's possible, and see what sticks. Even if you did everything right, there's always the social-engineering angle, or a single compromised device, or just accidental bad upgrade through supply chain. If you can mass-spread the seed, and plant something that "calls home" constantly, you can just have an agent that would start the attack the minute it gets infected, and go deeper.It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.
tokioyoyo · HN #50188670（在新分頁開啟）
> It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!”This isn't in the article? They do not at all claim "the attacked used AI!".The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.
nrmitchi · HN #50188890（在新分頁開啟）
As an european, just woke up to this, wow crazy.All my apps are on Railway, deployments has been down since yesterday morning (24 hours), feels nuts that they have waited until 2AM to say this.
jvidalv · HN #50192064（在新分頁開啟）
I have a bad feeling about all the recent news. Woke up today to the Railway email and rotated all my keys. Checked my OpenRouter dashboard and, to my surprise, there were API calls to Anthropic models I didn’t make. Luckily I have all my limits set to $25 a month...Interestingly enough, from the logs they were only using Anthropic as the provider, with a Go HTTP client. Pretty sure the main target was LLM API keys...
vellcopo · HN #50192440（在新分頁開啟）
Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?
AJRF · HN #50188596（在新分頁開啟）
How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...
fyzix · HN #50188750（在新分頁開啟）
What about buckets? I tried to regenerate my secrets but deploys are paused. I don't know if they actually rotated or not because the old keys seem to still be working! Dear attackers, please don't read or mutate my files while my app is offline :)
film42 · HN #50190508（在新分頁開啟）
I've already taken my applications offline and will be migrating to a new host...
tossstone · HN #50188849（在新分頁開啟）
> As such, we strongly recommend that you immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs for those services for unusual activity.> All deployments are pausedCan you explain us how can we rotate secrets without going down too?
fmvilas · HN #50192708（在新分頁開啟）
After the last outage we considered moving off to Cloud Run - used it several times before and it is rock solid - but we gave them another chance. Unlikely we are staying after a day of no deployments and then this.Rotated all our secrets but since deploys are paused our app is down.I'm unimpressed.
tomwphillips · HN #50191339（在新分頁開啟）
Agreed. This feels like an abandon ship situation. I’m working on switching right now. That’s too bad, but this is not acceptable unfortunately
m8s · HN #50191350（在新分頁開啟）
We are now on Cloud Run.
tomwphillips · HN #50194430（在新分頁開啟）
I had an Anthropic API key that was used in a Railway deployment be disabled due to evidence of unauthorized abuse ~3hrs after receiving the Railway security notice. They managed to spend about $20 in Opus 4.8 tokens before the abuse was detected. What a headache.
elliotbnvl · HN #50191358（在新分頁開啟）
Out of interest, what were they trying to do with Opus?
3m · HN #50193175（在新分頁開啟）
Genuinely curious: how much liability insurance do you need to survive something like this?
david_shi · HN #50191316（在新分頁開啟）
It's time for competent programmers to get rehired to help patch massive exploits that are about to unfold before us.
1-6 · HN #50189116（在新分頁開啟）
What stopped them from building the systems competently when they were employed?
eviks · HN #50191391（在新分頁開啟）
like the competent programmers that built php or?
efficax · HN #50194541（在新分頁開啟）
Anyone can help?I changed he api key in the env var as openrouter revoke the previous one based on this incident. However, as the redeploy cannot be done in my region, the env cannot be updated and now the product cannot be used. Any command in railway to fix this?
averrous · HN #50190419（在新分頁開啟）
For the people who had OAI/Anthropic API keys stolen and used? Are you able to see how they were used in your dashboard?I'm really curious how hackers have monetised the compromised API keys? Do they submit them to a shady AI-multiplexer that provides cheaper inference using stolen keys? One would expect completely random/disparate usage patterns/prompts if so...
tlonny · HN #50193315（在新分頁開啟）
Someone on the railway discord posted a screenshot of logs showing people were using it for erotic dirty chat
alpineman · HN #50193532（在新分頁開啟）
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.Like did someone vibe code too hard, how do env secrets leak for everyone...
minraws · HN #50188515（在新分頁開啟）
secret encryption is always a bit of a conundrum for a running cloud system. the secret needs to be plaintext when you use it, so you have to decrypt at least on every deploy. but that means the keys have to be ready at hand, no passphrase, somewhere an automated system can get them. that makes it very hard to actually protect them from someone with full access
efficax · HN #50194600（在新分頁開啟）
Easy, all the bad guys need to do is access whatever system stores the decryption keys.
9cb14c1ec0 · HN #50188631（在新分頁開啟）
Is this our new AI daily reality?
wateralien · HN #50188505（在新分頁開啟）
script kiddies now have access to frontier models. bad
pizzafeelsright · HN #50189881（在新分頁開啟）
companies probably have php and Python applications exposed to the web and nobody knows.the speed of irresponsible deployments is infinite now, by design
iririririr · HN #50188639（在新分頁開啟）
I'm rotating production secrets all night... great. I confirmed that they've actually used many of my secret keys already.
dylanz · HN #50189344（在新分頁開啟）
Which secrets were they using?
eclipticplane · HN #50190349（在新分頁開啟）
This is so fucked up. I was reviewing my account and I just saw this in the logs... Claude says database credentials and bucket credentials go in via env vars so likely they had access to database and bucket contents. Audit Logs Oct 21, 2026 05:54:13 PM CDT BucketInstanceCredential.regenerated in my-project production by Railway Staff Oct 21, 2026 05:54:13 PM CDT BucketInstanceCredential.regenerated in my-project production by Railway Staff Oct 21, 2026 05:54:13 PM CDT BucketInstanceCredential.regenerated in my-project production by Railway Staff
cowlby · HN #50190145（在新分頁開啟）
When I last used Railway - the default behaviour for railway-hosted databases was for them to be inaccessible to the public internet - so a leaked DB connection URL might not be as catastrophic as you would first think...
tlonny · HN #50195986（在新分頁開啟）
meanwhile on railway.com:Ship software peacefully
tstrebitzer · HN #50190662（在新分頁開啟）
It's pretty crazy how php saga and railway service came on the same day...
effortprogramme · HN #50194290（在新分頁開啟）
The past two days have been awful as someone who uses Railway.
gbnwl · HN #50188516（在新分頁開啟）
Damn I guess all those l33t hiring practices were for nothing
lijok · HN #50188544（在新分頁開啟）
Of course, it was a giant scam that tested for nothing and optimized for how to pass a test.Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.
rvz · HN #50188727（在新分頁開啟）
Related: https://news.ycombinator.com/item?id=50177413
MattIPv4 · HN #50188421（在新分頁開啟）
sighsThankfully I'm only using Railway for a hobby project (aptly, for a real-time transit dashboard) but still
jasona123 · HN #50190634（在新分頁開啟）
i revoked my api keys, changed with new ones but guess what? railway does not allow deployments at the moment, so my app is down.
arbayi · HN #50189066（在新分頁開啟）
Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36
eecks · HN #50188704（在新分頁開啟）
If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.
sergiotapia · HN #50188708（在新分頁開啟）
Really? Security announcement on blog subdomain, while the largest text on their home page is "Ship software peacefully" slogan, without any mention of the issue? And after looking at it for a while I still don't know what it exactly is, some kind of Terraform/CloudFormation GUI?
Good4boothee · HN #50193428（在新分頁開啟）
Railway was not honest about the reason for their last big outage.
nosefrog · HN #50191087（在新分頁開啟）
deploys have started for me on hobby plan.
OxfordCommand · HN #50198089（在新分頁開啟）
I have 3 clients projects running there. I am on pro plan. Openrouter has invalidated all keys. Nothing is working. Clients are angry. Railway is taking forever to resolve this and are taking steps in a mildly brain dead way. I cant swap keys.What can i do? Move to render ?
indiantinker · HN #50192156（在新分頁開啟）
use aws or digital ocean
efficax · HN #50194626（在新分頁開啟）
I’m lucky I moved away from it to host my own db with coolify
shinycode · HN #50188919（在新分頁開啟）
That’s coming for all platforms unfortunately
dgellow · HN #50188949（在新分頁開啟）