The Best Offense, Is a Bad... Defense?

The whoops

So, we got hacked recently. What truly pisses me off isn't that I got hacked. To be honest, I couldn't care less. No, what makes me furious is how stupidly simple it was. I've spent the last few months going over my code base. Protecting against SQL injection, making sure unneccesary data isn't passed where it shouldn't be. All of this in an effort to make sure we aren't hacked by bad code. And you know what, so far we've succeeded in that. I didn't get breached by some sophisticated attack that brought us down. No dependency chain attack, no supply attack. It was a .git breach. That was it. At some point, I accidentally committed some credentials to an email. While this site and its' tools are in a private repo, that doesn't matter when someone can get access to .git and rebuild the blobs. They didn't hack me, they just read my code. That is exactly what happened. But after a couple of sleepless nights, I came out on the other side with a more secure site. So, thank you.

I will say, anyone who receives a phishing email from us at [email protected], ignore it, do not respond. Now, the breach seemed to have started on April 10th. That is when logs started to appear, weird things started to happen. On the 13th of April, I found the source of these weird logs and refreshed credentials. While this was happening, I pulled the plug. I was less concerned about my site, about my servers, and more concerned about the damage that could be done. From what I can tell, all that was leaked was the email password and S3 bucket key pairs. But still, a breach all the same, and we've flushed creds.

The Cleanup

Something I'd been thinking about doing for a while was migrating the site to the docker. I can say that by moving to docker, it 'allows us to dynamically scale to active real world enviroments'. The real anwser? I'm lazy. I can either design a fancy redis cache server that holds PHP sessions, plus scaling VMs up or down to scale with the load, or I can just simply run a compose file and deploy anywhere. I get spooked and want to move this project somewhere else? Easy, dump the DB to a file, provision a new DB on some cloud infra, then spin up a small VPS with docker and deploy. It'll take about 15 minutes to completely move infra. The problem was inspiration. Moving to docker required redesigning a couple things. Mainly, how the DB is connected, the way the DB is setup, how we connect to redis. All sorts of things. To be perfectly honest, I just didn't feel like spending hours sorting through things, and learning how to make the containers I need. Now, files are only included if they are needed. There is still some work that needs to be done. I need to move the sentiment analysis model over to the new architecture, and the workers that keep and update data and analytics about pages is also busted. For that, we will be moving away from PHP scripts and towards GO based backend workers (yay! Microservices). In reality, we will not be moving to micro-services. That is excessive and inefficient. I doubt that this humble blog will ever need to scale. It is great to design your projects for scalability, that is never a bad idea. But, it's important to stay realistic. The grand majority of projects have less than a hundred users. I would go so far as to say that many have fewer than 5 users. We're no exception.

So what did we learn?

Security isn't optional. If you treat it like it is, you will get hacked. Things change, new attacks are found, weaknesses are scouted. Attackers are mobile. Pay PIA a few bucks of crypto, and you now have a highspeed VPN. Install tor, and you have a lowspeed, high security, anonymous network to use. You can disappear just as fast as you appear. Web services are slow, stationary targets. Like I previously mentioned, I spent a good amount of time and effort creating a good system to sit and listen, to identify threats with a mixture of machine learning and my human brain. There are still many things to think about and fix, but we're on the up and up. Getting hacked is honestly one of the best things to happen to this site. While I have a feeling I am going to regret this statement later, I do thank whoever it was for the wakeup call. As always, I work to improve and learn. And I'm already trying to find the next hard lesson. Feel free to reach out with any ideas, I'm all ears.