A few weeks ago my friend Wendy told me she was having trouble accessing The Odds Must Be Crazy, as Chrome was informing her it was referencing a known malware site. While I had no difficulty accessing it myself, and Google’s own Webmaster Tools claimed to see nothing wrong with it, a quick poll of my friends found they were experiencing the same. Bother.
So I did a little digging, installed a plugin or two (I think Sucuri was the main one that I initially relied on), and quickly found the issue: my PHP files had ALL had the dreaded base64 hack injected into the tops of them. Blech. I’d heard of this before, but this was my first time directly experiencing it, and I knew I was in for trouble.
Through trial, error, and research, I managed to come up with a plan of attack. I first backed up the entire file contents of the site (I have the database backed up automatically via email every week using WP-DB-Backup, so I wasn’t worried about that) to my hard drive, and then found a script I could run via SSH to automatically crawl through and remove the code (I’m purposefully not linking to the script for a soon-to-be-obvious reason), logged into my host, ran it… and promptly killed both The Odds Must Be Crazy and my personal blog (the one you’re reading now, which foreshadows the happy ending to the story). The script apparently removed a lot more than just the injected content and ruined every PHP file. Sadly I had NOT backed up this blog because it had stupidly not crossed my mind that the script would run across my entire web server and impact more than the backed-up TOMBC. *sigh*
So I quickly switched to plan B: manually clean the code from the TOMBC files. Initially I tried to automate it with Notepad++ (which I love), but the string in question is way too long for its Find field. So I spent two hours manually removing it from each and every PHP file (NOTE: there’s better news later on this) and then reuploading the entire site structure. I then used a variety of resources and tools over the next few days to track down a few stray, rogue files and issues, and remove them, rendering the site clean as far as all scans could tell.
I also reinstalled this blog from scratch and connected it back to the old database and reinstalled the appropriate plugins. It wasn’t (and still isn’t as of this writing) perfect, but my priority was the other blog.
Fast forward to this weekend when I decided to start working on fixing up this blog and found that the Dashboard was completely damaged and unusable for some reason. I did more digging and found that, while it hadn’t previously been infected, it now was. And a quick check of The Odds Must Be Crazy found the same. Fuck.
But this time I was determined to learn from my previous lessons, apply new knowledge, and fix this sucker faster, better, and protect it. And in the process I decided to share my steps so they can help others, as I didn’t find other guides online that fit my needs as well (most called for a full reinstall of the blog, and I customized things too much to feel good about that).
- Turn off/disable/shut down your antivirus software. It will try to remove files during the next steps and you won’t get a complete cleaning. I’m not going to include a step about turning it back on at the end, so remember to do that on your own.
- Download your entire WordPress directory to your local computer from your web host. Make sure you get everything for safety, including the index.php from the folder above it if WordPress was installed into a subfolder. This way you can easily recover from any mistakes you make by just uploading the (still-infected) files again.
- Use an automation utility to copy all of the .PHP files from this backup folder to a SECOND backup folder in their original structure. Since I’m on Windows 7, I used a robocopy command:
robocopy SOURCE_DIR DESTINATION_DIR *.php /SObviously replace SOURCE_DIR and DESTINATION_DIR appropriately. The /S command causes it to copy the entire subfolder structure.
- Use fnr.exe to wipe out the infection from the entire contents of your PHP-only copy of the files. In this case I simply opened up one of the infected files with Notepad++ and copied the top string (it starts with “<?php /**/ eval(base64_decode“, followed by a massive string of what appears to be gibberish, and ends with “;?>” immediately before another “</php” that’s vital to the rest of the file) into the Find field in fnr.exe and left the Replace field blank. I typed in the address of the PHP-only folder, and I hit the Replace button. For me this process took 18 seconds to do what took me two hours manually.
- Re-upload the ENTIRE PHP-only folder back to the FTP server. This was the important part of doing ONLY the PHP files, as it allows for a much faster upload versus uploading absolutely EVERYTHING (including image resources and other junk that wasn’t damaged).
- I then used a combination of Sucuri again and Exploit Scanner (caution: this one will give a LOT of false positives, so you have to have a decent idea of how to interpret its results) to figure out what things I might have missed. In this case one oddball was an index.php file which referenced an inbex.php file which replaced the original index.php file. This one was crawling with all sorts of nastiness, leading to the optional step 7:
- Download WordPress in a zip file and extract any files that you couldn’t scrub, like the above-mentioned index.php file. In that case, luckily, it’s a pretty static file that’s rarely customized (except if you installed your WordPress in a subfolder, but that’s easy to figure out), so you can just overwrite it and you’re immediately good to go again.
- Use Sucuri again, and from their site make sure to hit the Re-Scan button and make sure it finally checks out clean. Then go through its 1-Click Hardening section to shore up everything it suggests.
- Install Bulletproof Security. Use its Backup function first. Then follow its highlighted-green sections in the Security Modes tab to create the two .htaccess files, and then apply each of the four Bulletproof modes beneath it. Then run the Backup again.
- Go to the Security Status tab in Bulletproof and review the bottom-left section for recommended permissions of various files. Use your FTP software to set the permissions on each of the files and directories to the recommended code, then reload the page to make sure they show properly.
- Follow this fantastic guide from my friend Matt Selznick on what to do next. I had already been smart on steps one and two, but followed through step 7. Step 8 is essentially what we’ve done with our own step 10 above. Step 9 depends on whether your WordPress directory is in a subfolder (it was for one of my blogs, not for the other). Step 10 comes only after you’re aware of some nasty IPs from his Step 6. And Steps 11 and 12 appear to be covered to an even more extreme degree by our Step 9.
- Sit back and wait for the hackers to outsmart these techniques.
- EDIT: New step. Sign up for WebsiteDefender and run through its verification process (just a simple text file gets added to your root to prove you own the site) and let it do its thing. With its help I was able to find a few last files I missed, and learned the infection had dug through a bunch of my cgi-bin as well. No idea if any of this was really having an effect, but I feel better having wiped out all the signs than to be clueless about a few remaining traces.
Yes, sadly step 12 is a reality. I’m making no expectations that I’m somehow immune now. But I hope what I’ve at least achieved for now will ensure I’m a much more difficult target and not worth the effort. I’m also going to make some efforts to regularly backup the structure of my blog and not JUST the database like before. Then I can easily just delete the files on the server and replace them with my local copies. Honestly, the WordPress files, themselves, change quite rarely even if you’re pretty active. It’s the database that’s constantly being updated and it’s unharmed by replacing the site files.
So, lessons have been learned, skills have been gained, and hopefully I’ve helped out SOMEONE in the process. Let me know in the comments if it was you, or if you have questions about any of my steps.