HN Reader

> I also enabled UFW (which I should have done ages ago)

I disrecommend UFW.

firewalld is a much better pick in current year and will not grow unmaintainable the way UFW rules can.

    firewall-cmd --persistent --set-default-zone=block
    firewall-cmd --persistent --zone=block --add-service=ssh
    firewall-cmd --persistent --zone=block --add-service=https
    firewall-cmd --persistent --zone=block --add-port=80/tcp
    firewall-cmd --reload

Configuration is backed by xml files in /etc/firewalld and /usr/lib/firewalld instead of the brittle pile of sticks that is the ufw rules files. Use the nftables backend unless you have your own reasons for needing legacy iptables.

Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. Depending on your configuration, those firewall rules in OP may not actually do anything to prevent docker from opening incoming ports.

Newer versions of firewalld gives an easy way to configure this via StrictForwardPorts=yes in /etc/firewalld/firewalld.conf.

Just a note - you can very much limit cpu usage on the docker containers by setting --cpus="0.5" (or cpus:0.5 in docker compose) if you expect it to be a very lightweight container, this isolation can help prevent one roudy container from hitting the rest of the system regardless of whether it's crypto-mining malware, a ddos attempt or a misbehaving service/software.

Recently, those Monero miners were installing themselves everywhere that had a vulnerable React 19. I had exactly the same problem.

Sure does seem like the primary outcome of cryptocurrencies being released onto the world has been criminals making money.

I took issue with this paragraph of the article, on account of several pieces of misinformation, presumably courtesy of Claude hallucinations:

> Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated.

1. Files of running programs can be deleted while the program is running. If the program were trying to hide itself, it would have deleted /tmp/.XIN-unix/javae after it started. The nonexistence of the file is not a reliable source of information for confirming that the container was not escaped.

2. ps shows program-controlled command lines. Any program can change what gets displayed here, including the program name and arguments. If the program were trying to hide itself, it would change this to display `login -fp ubuntu` instead. This is not a reliable source of information for diagnosing problems.

It is good to verify the systemd units and crontab, and since this malware is so obvious, it probably isn't doing these two hiding methods, but information-stealing malware might not be detected by these methods alone.

Later, the article says "Write your own Dockerfiles" and gives one piece of useless advice (using USER root does not affect your container's security posture) and two pieces of good advice that don't have anything to do with writing your own Dockerfiles. "Write your own Dockerfiles" is not useful security advice.

You can run Docker Scout on one repo for free, and that would alert you that something was using Next.js and had that CVE. AWS ECR has pretty affordable scanning too: 9 cents/image and 1 cent/rescan. Continuous scanning even for these home projects might be worth it.

[*] https://aws.amazon.com/inspector/pricing/

This nextjs vulnerability is gonna be exploited everywhere because its so easy. This is just the start

I wonder in a case like this how hard it would be to "steal" the crypto that you've paid to mine. But I assume these people are probably smart enough to where everything is instantly forwarded to their C&C server to prevent that.

You might want to harden that those outbound firewall rules as another step. Did the Umami container need the ability to initiate connections? If not, that would eliminate the ability to do the outbound scans.

Also could prevent something to exfiltrate sensitive data.

This is a perfect example of how honeypots, anti-malware organizations, and blacklists are so important to security.

Even if you are an owasp member who reads daily vulnerability reports, it's so easy to think you are unaffected.

Something similar happened to me last year, it was with an unsecured user account accessible over ssh with password authentication, something like admin:admin that I forgot about.

At least that's what I think happened because I never found out exactly how it was compromised.

The miner was running as root and it's file was even hidden when I was running ls ! So I didn't understand what was happening, it was only after restarting my VPS from with a rescue image, and after mounting the root filesystem, that I found out the file I was seeing in the processes list did indeed exist.

This Monero mining also happened with one of my VPS over at interserv.net, when I forgot to log out of the root console in web-based terminal console to one of my VPS and closed its browser tab instead.

It has since been fixed: Lesson learned.

Sorry to hear you got hacked.

I know we aren't supposed to rely on containers as a security boundary, but it sure is great hearing stories like this where the hack doesn't escape the container. The more obstacles the better I guess.

The first step I would take is running podman instead of Docker to prevent container escapes. Podman can be run truly rootless and doesn't mess with your firewall. Next I would drop all caps if possible.

Is mining via CPU even worthwhile for the hackers? I thought ASICs dominated mining

I don't use Docker for my containers at home, but I take it by the concern that user namespacing is not the employed by them or something?

Still confused what I am supposed to do to avoid all this.

tl:dr: He got hacked but the damage was only restricted to one docker container runn ing Umami (that is built on top of NextJS). Thankfully, he was running the docker container as a non privileged non-root user which saved him big time considering the fact that the attack surface was limited only within the container and could not access the entire host/filesystem.

Is there ever a reason someone should run a docker container as root ?

Whew, load average of 0 here.

I also run Umami, but patched once the CVE patch was released. Also, I only expose the tracking js endpoint and /api/send via Caddy publically (though, /api/send might be enough to exploit the vul). To actually interact with Umami UI I use Twingate (similar to Tailscale) to tunnel into the VPC locally.

You're lucky that Hetzner didn't delete your server and terminate your account.

Only lesson seems to be use ufw! (or equivalent)

I still can't believe that there are so many people out here popping boxen and all they do is solve drug sudokus with the hardware. Hacks are so lame now.

Just use Hetzner managed servers? Very high specs, they manage everything, and you can install a lot of languages, apps etc.

Never expose your server IP directly to the internet, vps or baremetal.

Hahaha OP could be in deep trouble depending on what types of creds/data they had in that container. I had replied to a child comment but I figure best to reply to OP.

From the root container, depending on volume mounts and capabilities granted to the container, they would enumerate the host directories and find the names of common scripts and then overwrite one such script. Or to be even sneakier, they can append their malicious code to an existing script in the host filesystem. Now each time you run your script, their code piggybacks.

OTOH if I had written such a script for linux I'd be looking to grab the contents of $(hist) $(env) $(cat /etc/{group,passwd})... then enumerate /usr/bin/ /usr/local/bin/ and the XDG_{CACHE,CONFIG} dirs - some plaintext credentials are usually here. The $HOME/.{aws,docker,claude,ssh} Basically the attacker just needs to know their way around your OS. The script enumerating these directories is the 0777 script they were able to write from inside the root access container.

So this is part of the "React2Shell" CVE-2025-55182 issue? I find it interesting that this seems to get so little publicity. Almost like the issue is normal or expected. And it looks like the affected versions go back a little over a year. So if you've deployed anything with Next.js over the last 12 months your web app is now probably part of a million node bot net. And everyone's advice is just "use docker" or "install a firewall".

I'm not even sure what to say, or think, or even how to feel about the frontend ecosystem at this point. I've been debating on leaving the whole "web app" ecosystem as my main employment ventures and applying to some places requiring C++. C++ seems much easier to understand than what ever the latest frontend fad is. /rant

> The Reddit post I’d seen earlier? That guy got completely owned because his container was running as root. The malware could: [...]

Is that the case, though? My understanding was, that even if I run a docker container as root and the container is 100% compromised, there still would need to be a vulnerability in docker for it to “attack” the host, or am I missing something?

Not proof read by a human. It claims more than once the vulnerability was related to Puppeteer. Hallucination!

"CVE-2025-66478 - Next.js/Puppeteer RCE)"

Interesting that this got posted today, I also have a server on Hetzner (although I don't think it's relevant) and noticed yesterday that a Monero miner had been installed.

Luckily for me, the software I had installed[1] was in an LXC container running under Incus, so the intrusion never escaped the application environment, and the container itself was configured with low CPU priority so I didn't even notice it until I tried to visit the page and it didn't load.

I looked around a bit and it seemed like an SSH key had been added under the root user, and there were some kind of remote management agents installed. This container was running Alpine so it was pretty easy to identify what processes didn't belong from a simple ps output of the remaining processes after shutting down the actual web application.

In the end, I just scrapped the container, but I did save it in case I ever feel like digging around (probably not). In the end I did learn some useful things:

- It's a good idea to assume your system will get taken over, so ensure it's isolated and suitably resource constrained (looking at you, pay-as-you-go cloud users).

- Make sure you have snapshots and backups, in my case I do daily ZFS snapshots in Incus which makes rolling back to before the intrusion a breeze.

- While ideally anything compromised should be scrapped, rolling back, locking it down and upgrading might be OK depending on the threat.

Regarding the miner itself:

- from what I could see in its configuration it hadn't actually been correctly configured, so it's possible they do some kind of benchmark and just leave the system silently compromised if it's not "worth it", they still have a way in to use it for other purposes.

- no attempt had been made at file system obfuscation, which is probably the only reason I really discovered it. There were literally folders in /root lying around with the word "monero" in them, this could have been easily hidden.

- if they hadn't installed a miner and just silently compromised the system, leaving whatever running on it alone (or even doing a better job at CPU priority), I probably never would have noticed this.

[1]: https://github.com/umami-software/umami

I wouldn't trust that boot image or storage again, I'd nuke it for peace of mind.

That said, do you have an image of the box or a container image? I'm curious about it.

No firewall! Wow that's brave. Hetzner will let you configure one that runs outside of the box so you might want to add that too, as part of your defense in depth - that will cover you if you make a mistake with ufw. Personally I keep SSH firewalled only to my home address in this way; if I'm out and about and need access, I can just log into Hetzner's website and change it temporarily.

I've bene working with a GPU security company for the last few months... I can tell you that neo clouds (generally) do not see security as a high priority—or often, even their responsibility. Many do not have hte ability to even know if your GPUs have been compromised and they expect you'll take responsibility.

Meanwhile companies think the clouds are looking at it.... anyhow. it is a real problem.

I find it interesting that the recent trend of moving to self-hosted solutions is sparking this rediscovery of security issues that come with self-hosting. One more time and it will be a cycle!

regardless of firewalls and best practices and all, i'd just put all sorts of admin-related stuff behind tailscale (or similar), including ssh. hetzner allows you to have ssh closed in your public ip and still open a terminal via the console if ssh-on-tailscale fails. for the web stuff you should do a similar trick. blog and public websites on the public address, while admin stuff goes on tailscale. and if you do it nicely with letsencrypt you can even have nice hostnames pointing to your private stuff.

I have a similar setup but with following additional security configurations:

- Hetzner firewall (because ufw doesn't work well with docker) to only allow public access to 443.

- Self-hosted OpenVPN to access all private ports. I also self-host an additional Wireguard instance as a backup VPN.

- Cloudflare Access to protect `*.coolifydomain.com` by default. This would have helped protect the OP's Umami setup since only the OP can access the Umami dashboard. Bypass rules can be created in Cloudflare Access to allow access to other systems that need access using IP or domain.

- Cloudflare Access rules to only allow access to internal admin path such as /wp-admin/ through my VPN IP (or via email OTP to specified email ids).

- Traefik labels on docker-compose files in Coolify to add basic auth to internal services which can't be behind Cloudflare Access such as self-hosted Prefect. This would have also added a login screen before an attacker would see Umami's app.

- I host frontends only on Vercel or Cloudflare workers and host the backend API on the Coolify server. Both of these are confirmed to never have been affected, due to decoupling of application routing.

- Finally a bash cron script running on server every 5 minutes that monitors the resources and sends me an alert via Pushover when the usages are above the defined thresholds. We need monitoring and alert as well, security measures alone are not enough.

Even with all these steps, there will always be edge cases. That's the caveat of self-hosting but at the same time it's very liberating.

  $ sudo ufw default deny incoming
  $ sudo ufw default allow outgoing
  $ sudo ufw allow ssh
  $ sudo ufw allow 80/tcp
  $ sudo ufw allow 443/tcp
  $ sudo ufw enable

As a user of iptables this order makes me anxious. I used to cut myself out from the server many times because first blocking then adding exceptions. I can see that this is different here as the last command commits the rules...

Good job.

Examples like this, are why I don’t run a VPS.

I could definitely do it (I have, in the past), but I’m a mediocre admin, at best.

I prefer paying folks that know their shit to run my hosting.

Roughly 20 years ago, PHP sites were getting hacked everywhere.

Now, JS sites everywhere are getting hacked.

JS has turned into the very thing it strived to replace.

It's good to see developers haven't changed. ;)

What's considered nowadays the best practice (in terms of security) for running selfhosted workloads with containers? Daemon less, unprivileged podman containers?

And maybe updating container images with a mechanism similar to renovate with "minimumReleaseTime=7days" or something similar!?

I was surprised by the same thing, a tool I run that uses Next.js without me knowing before.

I noticed this, because Hetzner forwarded me an email from the German government agency for IT security (BSI) that must have scanned all German based IP addresses for this Next.js vulnerability. It was a table of IP addresses and associated CVEs.

Great service from their side, and a lot of thanks to the German tax payers wo fund my free vulnerability scans ;)

If I’m not wrong, a hetzner VM by default has no firewall enabled. If you are coming from providers with different default settings, that might bite you. Containers that you thought were not open to internet have been open all this time. Two firewalls failed: They bypassed ufw and there was no external firewall either.

You have to define a firewall policy and attach it to the VM.

I got another mail from Hetzner on Dec 10, telling me that the BSI told them my web site was having serious security problems with Next.JS. Not having high opinions on the BSI, I first thought this was some very elaborate scam attack but no, it turns out it was legit and also Umamis inbuilt Next.JS for me. But apparently there was no crypto miner or other active abuse yet.

Author forgot one important take: to limit the attack surface what doesn't need to be public facing should not be public facing. Things such as analytics software can easily be accessed through wireguard or even simpler using ssh as socks proxy.

> "No more exposed PostgreSQL ports, no more RabbitMQ ports open to the internet."

Yikes. I would still recommend a server rebuild. That is _not_ a safe configuration in 2025, whatsoever. You are very likely to have a much better engineered persistent infection on that system.

> I also enabled UFW (which I should have done ages ago):

Docker will overwrite your rules when you publish ports.

Do not publish ports with docker. Do not run internal services on the publicly accessible system.

> Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated.

  /tmp/.XIN-unix/javae &
  rm /tmp/.XIN-unix/javae

This article’s LLM writing style is painful, and it’s full of misinformation (is Puppeteer even involved in the vulnerability?).

OK, so am I right that this guy had a completely unsecured metrics endpoint running on his server? Why would you do that in the first place?

After reading some comments: this probably goes without saying, but one should be very careful what to expose to the internet. Sounds like the analytics-service maybe could have been available only over VPN (or similar, like mTLS etc.)

And for basic web sites, it's much better if it requires no back-end.

Every service exposed increases risk and requires additional vigilance to maintain. Which means more effort.

TY for the article. You made me look into my server, and also found some strange activity on my docker containers. Good call!

This is weird. I viewed this blog post on Chrome and it loaded fine. But I sent the link to my fellow dev and he tried viewing it on Microsoft Edge on MacOS but the browser showed a red page with the "This site has been reported as unsafe" message by the Microsoft Defender SmartScreen.

It highlighted the domain: 'jakesaunders.dev' in the address bar in red text.

I don't think using key-based authentication for SSH and enabling Fail2ban is necessary. Fail2ban is only useful if you keep password authentication. But I might be wrong.

I am not an expert in incident reaction, but I thought the safe way was to image the affected machine, turn it off, take a clean machine, boot a clean OS image with the affected image mounted read only in a VM, and do the investigation like that ?

Assume that the malware has replaced system commands, possibly used a kernel vulnerability to lie to you to hide its presence, so do not do anything in the infected system directly ?

Would "user root" without --privileged and excessive mounts have enabled a container escape, or just exposed additional attack surface that potentially could have allowed the attacker to escape if they had another exploit?

The world will be a better place when all crypto just disappears

Does anybody know how to just list the processes running inside a single container from within that container?

And isn’t it a design flaw if you can see all processes from inside a container? This could provide useful information for escaping it.

I’m sorry you went through this.

But I am interested in the monero aspect here.

Should I treat this as some datapoint on monero’s security having held up well so far?

Was dad notified of the security breach? If not he may want to consider switching hosting providers. Dad deserves a proper LLM-free post mortem.

This article is very interesting at first but I once again get disappointed after reading clear signs of AI like "Why this matters" and "The moment of truth", and then the whole thing gets tainted with signs all over the place.

As an aside, if you're using a Hetzner VPS for Umami you might be over-specced. I just cut my Hetzner bill by $4/mo by moving my Umami box to one of the free Oracle Cloud VPS after someone on here pointed out the option to me. Depends whether this is a hobby thing or something more serious, but that option is there.

so what's the point of containers here? seems only to make things less transparent and more complex to manage.

js scripts running on frameworks running inside containers

PS so I see the host ended up staying uncompromised

I had something similar, but I was lucky enough to catch it myself. I've used SSH for years, and never knew that it - by default - also accepts password logins. Maybe dumb on my part, but there you go...

I didn't see it mentioned, but wouldn't having a RO root filesystem with writable directories mounted noexec also have been sufficient?

It makes me irrationally angry that cryptos have given a clear monetary value to raw CPU time, and a very strong incentive to create botnets.

a) containers don't contain

b) if you want to limit your hosting environment to only the language/program you expect to run you should provision with unikernels which enforce it

Well written blog post. Well done, I've learned something new.

> ls -la /tmp/.XIN-unix/javae

Unless ran as root this could return file not found because of missing permissions, and not just because the file doesn't actually exist, right?

> “I don’t use X” doesn’t mean your dependencies don’t use X

That is beyond obvious, and I don't understand how anyone would feel safe from reading about a CVE on a widely used technology when they run dozens of containers on their server. I have docker containers and as soon as I read the article I went and checked because I have no idea what technology most are built with.

> No more Umami. I’m salty. The CVE was disclosed, they patched it, but I’m not running Next.js-based analytics anymore.

Nonsensical reaction.