I often write about the best practices for homelabbers, but I don’t always follow my own advice. That came to bite me in the butt recently when my homelab was compromised and a crypto bot started running my server’s CPU at 100%—here’s what I did to fix it.
I used to open almost every self-hosted service to the internet with a reverse proxy
It made remote access easy, okay?
A few years ago, I went on a trip away from my homelab. I had a VPN set up on a Raspberry Pi 3b, and I had some services go down that I needed to fix. My VPN was slow and laggy (thanks to the 10/100Mbps port on the Pi 3b) and that made things unbearable to try and fix services through.
So, I decided to use the time I was VPN’d into the server to basically reverse proxy everything on the network. That would make it simple to remotely access any of the self-hosted services I had to manage them if I wasn’t home. A side benefit was that even when at home, I could access the services easier because they would all have a FQDN (fully qualified domain name).
The problem is, many homelabbers make this same mistake. Once you have Nginx Proxy Manager or Cloudflare Tunnels set up, it’s easy to just make every service available outside of the network with just a few clicks. Having external access to those services is convenient, but it’s definitely not the right choice, as I found out.
These days, I have Tailscale installed on most of my devices, and that runs on my server with a 2.5Gb/s LAN connection and a 1Gb/s WAN connection. The problem is, I still have a lot of my services behind my reverse proxy because if it ain’t broke…well, it broke.
A reinstalled qBittorrent container was my downfall
I switched containers and forgot to reset the password…oops
I recently shuffled around my entire homelab, moving services from one node to another, and reinstalling services to make them more robust—qBittorrent was one of those services.
I moved over most of the config information for qBittorrent when I switches to a new container. The problem is, I didn’t actually set a password for it. It has been so long since I last deployed a fresh qBittorrent container that I completely forgot it generated a password in the logs and set the username to admin. Quite honestly, I thought that moving my config files over would move my user/password from the old server to the new server—I was wrong.
Something that I didn’t know is that there are botnets out there that are actively scraping the internet looking for exposed qBittorrent admin web UIs to brute force through. I never fell prey to them before because I wasn’t using admin as a username and my password was 48 characters and randomly generated.
The new container, however, had an eight character password and used the username admin, and that’s where my problem came from. Even though it had a randomly generated password, it was only eight characters, which is relatively easy for botnets to brute force. So, with the username admin and an easily breakable password, my qBittorrent instance was compromised and a botnet was in.
Once inside the qBittorrent instance, the botnet downloaded a test torrent to make sure it had the right permissions it needed, and then it was off—a crypto miner quickly got to work on the server and started pegging my CPU cores to 100%. That’s when I realized what was going on, and started working to fix it.
It’s an easy mistake to make, but the consequences can be severe
Thankfully the fix wasn’t that hard, but I shouldn’t have had to make it in the first place
My qBittorrent client runs on my Lenovo RD440 rack-mount server, which is anything but quiet. Normally, it’s pretty chill though, my office stays cool with a window-mounted air conditioner, and it doesn’t have a lot of stress, so the fans never really spin up. The crypto miner running at 100% CPU usage across both Xeon CPUs, however, made the normally low-noise server turn into a jet engine.
This made me instantly dive into the system to see what was going on. I was able to find the erroneous process and kill it, and I then got into my qBittorrent client and changed the password. From there, I thought I was safe, but I missed the stray torrent that the botnet uploaded, and I also missed the change to my qBittorrent config that was made.
Eventually, I reinstalled the qBittorrent container from scratch and moved all my ISO states back to where they were previously, and the hack was over. It was a crazy hour or two trying to figure out what was going on and how to fix it, but I did eventually get it back under control.
My homelab is only as secure as I make it—and I made it fairly unsecure
I’m still working to get everything back under control from this little adventure, but it did show me a few things. Just because I haven’t had any security breaches in over five years of running my homelab, doesn’t mean I’ll never have a security breach.
Best practices for a homelab are best practices for a reason, and putting every service behind a reverse proxy that can be accessed externally is definitely not a best practice.
I’ll be taking the next few weeks to really tighten down my homelab’s security and move services off of external domain names to internal reverse proxy domain names and make it so I have to VPN to access them again. Tailscale on a faster system is much more responsive then my old VPN on my Pi, and it’s time that I properly move the services around how they need to be.
