Zamfoo Sucks and You Shouldn’t Use It

Posted by Nessa | Posted in uncategorized | Posted on 19-04-2014


If you’re one of the more fortunate ones that has not been exposed (in all meanings of the word) to Zamfoo, it’s a suite of plugins that integrates with cPanel/WHM to add additional account management functionality for “easing the burden of web hosting providers” [SIC]. And by “easing the burden of web hosting providers”, it really means letting people have root access to your server because Kevin Quinn is too lazy to write a decent application.  I don’t know about you, but I often stay awake at night wondering how I can expose my customers to severe and unnecessary security risks, because there just isn’t enough of that on the Internet already.

In case Mr. Quinn removes the install script before you get a chance to lol, don’t worry.  There’s an online installer that is just as legit. Seriously. You’re going to need to change your shorts for this one:



All the while I was thinking: Ah, yes.  Let me go ahead and put my ROOT credentials into this unencrypted PHP web form that’s nested into a TikiWiki installation.  And good thing special characters in the root password might be a problem for this totally professional means of installing a server application.  That means I can finally pick a root password that I can actually remember without wasting grey cells #YOLO.


Ok, all sarcasm aside, Zamfoo is among the biggest stagnant cesspool of shi*ty coding clusterf*ck I think the modern hosting industry has ever seen.  And Kevin Quinn‘s people skills are just as competent as his coding (and his grammar for that matter, because English is hard), to say the least.  Because when reputable security companies inform you that your software is rootable, a responsible developer completely ignores it and threatens legal action, right?


Zamfoo sucks and you shoudn’t use it.


As a people, we must stand together and not be part of this.  Uninstall Zamfoo, grab a beer, and send Kevin an email to let him know how big of an a**hole he is for letting this debilitated piece of crap exist on the Internet.


PS: Oh, and this is for you Kevin.

cPanel Security Advisor: Don’t Take it to Heart

Posted by Nessa | Posted in uncategorized | Posted on 19-02-2014


cPanel 11.40 introduces a new feature in WHM called “Security Advisor“. I don’t mess with WHM a lot so while I was vaguely aware that such a feature existed in cPanel, only today did I actually mozy over and give it a run.

Well, it’s pretty obvious that this tool was whipped up in response to people repeatedly asking the blanket question: “How do I secure my server?” (Easy: you hire someone that knows how to secure servers). As the leading provider of its type, cPanel is under a lot of pressure to keep up with the demands of their clientèle, including the ones that expect a point and click solution to everything.  And while cPanel’s efforts here are meritorious, Security Advisor appears to do nothing more than make a series of “educated” guesses about what your server is, or should be, doing.  This leaves me wondering how many people are making unnecessary and thoughtless changes to their servers because some script told them to.

Here are a few examples of what it found on one of my test boxes:

Apache vhosts are not segmented or chroot()ed.

Enable “Jail Apache” in the “Tweak Settings” area, and change users to jailshell in the “Manage Shell Access” area. Consider a more robust solution by using “CageFS on CloudLinux”


No brute force protection detected

Enable cPHulk Brute Force Protection in the “cPHulk Brute Force Protection” area.


ClamAV is not installed.

Install ClamAV within “Manage Plugins”.


A newer kernel is installed, however the system has not been rebooted. running: 2.6.32-279.22.1.el6, installed: 2.6.32-431.5.1.el6

Reboot the system in the “Graceful Server Reboot” area.

So, for one: my contempt for CloudLinux is only matched by equal hatred for mod_ruid2 (required for “Jail Apache”).  SA missed the CloakFS setup on this server, which achieves the necessary jailing.

CpHulkd and ClamAV are also not the only software of their kind, so if you use CSF, BFD, and/or your own AV, be prepared to hear Security Advisor roar.

Ksplice has been a thing for a while now.  My reboot-less kernel upgrade is no match for you, Security Advisor.

Now, there were some legitimate things SA found, but nothing that I necessarily care about.  Here’s why:

My intention here, quite to what seems to be the contrary, is not to blast Security Advisor for its efforts in guiding sysadmins through the daunting and never-ending path of system security.  My point is, you need to understand your system and what security ‘violations’ it reports are actually problematic and what is the best way to address these problems in your environment.  The solutions SA is suggesting may actually be invalidated by  other measures in place on your system, or better addressed using a different method.  For example, I don’t condone switching to ruid2 on a shared server just to provide the jailing capabilities that CloakFS and CageFS can just as securely provide.  Or pointlessly rebooting your server because SA doesn’t like the output of uname.  Before you make changes to your server, understand what you’re doing, why you’re doing it, and whether it really needs to be done.

BTW, cPanel, I still love you guys.  I just don’t fancy Security Advisor.

One of Those “Ksplice Saved Our Asses” Moments

Posted by Nessa | Posted in uncategorized | Posted on 15-01-2011


There comes a time in every sysadmin’s life when you realize that as long as you have your job, you’re never going to

  1. have a life
  2. have friends
  3. sleep
  4. have a break from all the madness

When you fix server A, server B breaks. When you patch exploit 1, exploit 2 strikes threefold.  And to top it off, our customers are still using passwords like 123456.  Luckily, there’s someone out there that appears to be seeing things at my level.

We started using Ksplice back in September 2010 when one infamous kernel exploit started it all. We used to only have to do kernel updates a few times a year.  Now the RHEL guys decided to start compiling their kernels from sad pandas and french fries, backporting and reintroducing a string of security holes that hackers have been sitting until the opportune moment. When you’re working for one of the best hosts in the industry and basically guaranteeing 99.9% uptime, you try rebooting a dump truck full of servers five or six times a month just to avoid some asshat defacing all the index pages on your server with pictures of babies and swastikas.

I mean, unless you’re into that kind of thing.

During these unsexy times there’s one thing I’ve been continually grateful for – and that’s the fact that the developers of Ksplice apparently have no lives, girlfriends, or any other dreams in life other than to make all our jobs easier.  Here at IMH, we were all slightly depressed when we found out that we couldn’t have reboot parties anymore.  The Ksplice guys usually have new vendor-released kernel patches available within a day  and we can easily and instantly run kernel updates on our entire fleet of servers without having to reboot a single one.  With that, we don’t have to wait until the middle of the night over the span of several days to fix critical security problems, and customers don’t have to complain about downtime.  This lets us focus on what’s important: drinking, karaoke, and nude ice fishing keeping people happy.  If that’s not worth the $2.95 a month, I don’t know what is.

WordPress Thinks Network Solutions is Stupid

Posted by Nessa | Posted in uncategorized | Posted on 22-04-2010


Quick quiz: What does a hosting provider do when they know they’ve messed up and don’t want to deal with the fallout?

You apparently blame WordPress.

Don’t get me wrong here – being behind the scenes of server management for a webhosting company makes you the target of a lot of accusations.  And yes, most of the time when a user’s site gets hacked it’s their own damn fault. But in this case, Network Solutions is apparently trying to push their issues off on WordPress because they don’t want to admit they f***cked up.

Well, WordPress is pissed.  I logged into my dashboard today and the first thing I see in my news feed is:

A web host had a crappy server configuration that allowed people on the same box to read each others’ configuration files, and some members of the “security” press have tried to turn this into a “WordPress vulnerability” story.

To highlight the best part:

I’m not even going to link any of the articles because they have so many inaccuracies you become stupider by reading them.

P.S. Network Solutions, it’s “WordPress” not “Word Press.”


In short, Network Solutions acknowledged that most of the problem was due to users’ public_html and wp-config.php files being readable by other users on the server – something which could have easily been caused by the users setting the permissions of those files insecurely. But they took a shot in the dark and said that the problem was caused by WordPress putting cleartext database credentials in the wp-config.php file – something that just about every software developer does, as WordPress states:

WordPress, like all other web applications, must store database connection info in clear text. Encrypting credentials doesn’t matter because the keys have to be stored where the web server can read them in order to decrypt the data. If a malicious user has access to the file system — like they appeared to have in this case — it is trivial to obtain the keys and decrypt the information. When you leave the keys to the door in the lock, does it help to lock the door?

Good point. They also went on to say that a properly configured web server will not allow users to access the files of another user, regardless of file permissions. This is why most hosts have switched to using suPHP or phpsuexec, a technology which Network Solutions was apparently left in the dark on. At least now they seem to be taking responsibility for the problem and are attempting to handle it.

I’m also going to state, based on comments in popular blogs from users that don’t know what the hell they’re talking about, that unless someone has access to view the source of a PHP file, they can’t see the database credentials. PHP files are executed server-side, and only their output is sent to the browser. Since the username and password are stored as variables and are not echoed out anywhere, someone simply calling wp-config.php from a browser can’t access your login data.

You’re probably going to find all kinds of fixes on various sites that this story is covered on, but I’m going to give the same advice I do for all my customers that have had sites hacked:

  • Change your FTP and MySQL user passwords
  • Replace all files on your site from a ‘clean’ backup
  • Make sure the software on your site is up to date
  • Scan your PC for viruses
  • Choose a secure host

Remember that your site can get hacked regardless of who your host is or how secure they are, though your host has to take some level of responsibly for hacks that are caused by their own bad configuration, such as in the case with Network Solutions.

WHM APF Plugin

Posted by Nessa | Posted in uncategorized | Posted on 05-06-2008


I’m happy to announce my first ever perl-written plugin for WebHost Manager, which was developed by myself and one of my fellow sysadmins at InMotion Hosting. The first release is available here:

Download v.1.05

Download Now

To explain a little bit of background here, many hosting companies that give some sort of **** about security will leave SSH port 22 closed except for specified IPs. Us being one of them, requests from customers for us to add their IPs to their firewalls is rather redundant when we host over 900 V-dedicated systems and 100 Dedicated boxes where customers can opt for SSH access. Therefore, I decided that it would be well worth our efforts to create a WHM plugin to allow customers to add their own IPs without ever having to contact us. I am aware that some plugin developer already has a more sophisticated APF plugin for WHM but you have to pay for it.


– cPanel/WHM (tested on version 11.18)

– APF 0.9 (tested on 0.9.6)

– iptables enabled and working (if you are able to restart APF without any errors, it’s probably fine)

Note: default privileges will allow anyone with WHM/reseller access to use this plugin. You can manually change this in the addon_add2apf.cgi file if you want.


cd /usr/local/cpanel/whostmgr/docroot/cgi
tar -xvzf apfadd_whm_1.05.tar.gz
rm -f apfadd_whm_1.05.tar.gz

Simple enough?

From there, load up WHM and on the left side you’ll see an option to “Add IP to Firewall” and the plugin page will give you examples of what you can add. The script is set up to allow:

Hostname –
Single IP –
Port/IP – d=22:s=
Port/CIDR – d=22:s=

Of course, the error checking is not perfect, so just be aware of what you’re adding or you might unintentionally ‘break’ your firewall, which usually results in blocked traffic.

Eventually I’m sure I’ll end up adding the ability to manage ports and remove IPs for users with a certain access level, but currently I don’t see a need to do so because I don’t believe that non-root users should have that type of access. Note that per the readme, you can edit the addon file to limit access to users with a certain reseller ACL privilege.

Moving Towards PCI Compliance with cPanel

Posted by Nessa | Posted in uncategorized | Posted on 14-04-2008


See also:

Those of you who are server admins or use certain merchant services know what I’m taking about — it’s that dreaded security scan that picks apart your server to tell you everything that it thinks is wrong, assuming you have the knowledge or access to fix it: yes, the PCI scan. PCI compliancy is a somewhat new procedure used by security companies and financial institutions to measure the security of a webserver that collects and stores sensitive information. The reasons for getting a scan vary, but are most commonly for legal reasons or just the assurance that your server is subject to certain vulnerabilities.

After dealing with 2-3 PCI scans a week for the last year, I’ve put together a common procedure for how to make your server compliant to current PCI standards. Note that each scan company is different and may report other issues, and if you’re using ControlScan then, well, I feel sorry for you. I’m also assuming that you are on a Linux server running cPanel and LAMP.

Step 1: Make sure you have a firewall

PCI scans are nazis about unjustified open ports, so only open the ones that you need in order for services to run effectively. Manually configuring iptables is a pain in the ass, so I recommend using APF or CSF (if you have cPanel) and then configuring the allow rules to only allow ports for active services.

Note that both indicate the opening of cPanel ports 2082, 2095, and 2086, but some scans will complain about these being nonsecure. If that is the case you can configure within WHM to only use the secure ports, then remove the nonsecure ones from the firewall so they can’t be accessed. You should also close MySQL port 3306 for external hosts and allow them on a per-IP basis to anyone other than localhost has to be allowed.

Step 2: Update your system

This is an obvious one, but you’d be surprised how many people still have old packages installed on the server. With cPanel, running /scripts/upcp will usually update the vital system software as long as you have your update configuration in WHM set to allow it, but otherwise I would recommend doing a yum update, up2date, or whatever else you use to manage packages to make sure everything is up to date.

Nowadays old versions of MySQL, PHP, and Apache are no longer squeezing through either, so you need to upgrade to at least MySQL 4.1.22, PHP 5.2.5, and Apache 1.3.39 (some scans will want Apache 2.0.x).

Step 3: FIx OpenSSL

If you did a package update this was probably already taken care of, but if you installed via source you need to make sure you’re using at least 0.9.7j, which is the oldest version that most PCI scans allow. You can get your sources from here, and it may require a recompile of Apache and other services that use it. To check your OpenSSL version, type ‘openssl‘ from your SSH prompt and then type ‘version‘.

Note to Redhat/Fedora/CentOS users: If you’re running a somewhat recent version of your OS your openSSL version probably is something like 0.9.7a, but due to Redhat backporting this may be a false-positive. If you’re on any Redhat-based distribution, just tell your scan company and they’ll bypass OpenSSL checks.

Step 4: Check your SSL certificates

In order to pass a PCI scan your server must have at least one SSL certificate signed by a recognized certificate authority, and any services using SSL need to be using a certificate as well. Go cough up $30-$100 and buy a decent 264-bit SSL certificate and install it not just for Apache, but also for all of your active services. WebHost Manager has a section for installing service SSL certificates to make this process easier.

Step 5: Disable SSLv2 and other weak encryption methods

This one always gets me, because there is no way to disable SSLv2 for everything at once, at least not one that I know of. What makes this part the worse is that not all services support the choosing of SSL protocols and ciphers, but luckily unless you are using ControlScan the ones that don’t are probably not going to show up. Here’s how you do it for common services that are reported:


Add these lines to your httpd.conf (you may to add them to each secure vhost as well):

SSLProtocol -ALL +SSLv3 +TLSv1

POP3 and IMAP:

Edit the following files:


Comment out the existing TLS_CIPHER_LIST line and replace it with the following and restart courier-imap:



Add the following to exim.conf:

tls_require_ciphers = ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM:!SSLv2

For other services that might be on your system, take a look at this guide.

Step 6: Disable mod_userdir (or whatever cPanel is calling it nowadays)

If you are able to go to http://yourserverip/~yourusername, then you have mod_userdir enabled and the scan will probably complain. You can disable this in WHM under Security Center > Apache mod_userdir Tweak, or in httpd.conf add “userdir disabled user1 user2 user3 …etc”

Step 7: Put Apache in incognito mode and disable the bad stuff

If you try to get an Apache error (like a 404 error), the footer of that page probably contains more information that you may want to share about your Apache setup. You can disable this in your httpd.conf by adding these lines:

ServerSignature Off
ServerTokens Prod
FileETag None

You can read more about these here.

Another thing that some scans report is the use of 413 errors. You should add this line to httpd.conf as a workaround:

ErrorDocument 413 /index.php (or any other file)

Just about all scans will complain if the ‘trace’ and ‘track’ apache methods are enabled on your server. You can fix this by adding these lines to your Virtualhost entries or .htaccess files:

RewriteRule ^.*$ - [F]

You should also disable directory indexes, which can be done most easily in your cPanel’s index manager. Directory indexes allow the listing of files inside folders that do not have an index page. You can also disable these in your .htaccess files:

Options All -Indexes

Ending notes

Really, it doesn’t matter how secure your server is if your web application scan is poorly programmed, so your server should not be the ending point in security. Some PCI scan companies are able to detect common vulnerabilities in web applications, but you should take the extra steps to stay ahead of the game and update your site software on a regular basis.

Someone’s Got the Internet AIDS…

Posted by Nessa | Posted in uncategorized | Posted on 23-03-2008


I knew something was fishy when I got an IM from my ex whom I haven’t spoken to in over a year:

hey How are you???? this is ur pic rite?!

Worse enough I can’t believe I clicked on that shit.. I thought maybe it was one of those pictures from the amateur night at JB’s Gallery of Girl back in 2004 that caught up with me. But no, as soon as I clicked on it my PC (which unfortunately is the one that runs Windows XP) froze up for a good minute during which time it was sending the same message to all 158 people in my MSN friends list.

Arrrggg…anywho, the virus — which is the Backdoor.Generic3.SAT – is pretty harmless as far as your PC is concerned but you’ll probably get  kicked every time you open an MSN window. So, close your MSN and go here and here to read about how you get cure the internet STD’s you’ve probably just spread around to all your friends. It’s like the 70’s all over again only the free clinic isn’t as crowded.

Securing the TMP Partition and Tracking Hacks

Posted by Nessa | Posted in Uncategorized | Posted on 16-11-2007


Are your temp partitions putting out behind your back? Anyone who’s ever administered a Linux server would know the risk of leaving the /tmp directory unsecured, moreso on a webserver that is shared among multiple websites.

The tmp directory is world-writeable and used by a majority of services on a machine — including the storage of PHP and MySQL session files. One issue that I’ve seen on older servers is that one customer’s poorly-coded website would get exploited and end up downloading a file into the /tmp directory, then have that hack file executed on the server as a nobody-owned process. Hack processes are the easiest to find, as they always have a little something extra.

This should go without saying, but hack processes can run from almost anywhere, not just /tmp. I’m mainly bringing up the tmp directory because it’s the most targeted location for hack files if it isn’t secured properly.

Identifying Hack Processes

To start out, log into your server as root as issue the following command to see all the nobody-owned processes running on the machine. This is assuming that Apache on your webserver runs as ‘nobody':

root@localhost[~] ps -efw |grep nobody |more

A majority of the output will reflect legitimate Apache processes:

nobody 8748 25841 0 21:35 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL
nobody 8785 25841 0 21:35 ? 00:00:01 /usr/local/apache/bin/httpd -DSSL
nobody 8988 25841 0 21:36 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL

You’ll see that they all have the same parent process ID of ‘25841’ and all of the commands look about the same. However, you may see something that looks like this:

nobody 15707 26407 0 11:18 ? 00:00:00 [sh <defunct>]
nobody 15717 1 0 11:18 ? 00:00:04 /usr/bin/perl
nobody 13016 1 0 14:14 ? 00:00:00 /usr/bin/perl

nobody 8988 1 0 21:36 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL -d3

All of these are hack processes. A few things to look for:

– Perl and shell (sh) processes on most systems run as the user that is executing them, not ‘nobody’. If this is true on your system, a nobody-owned perl process is probably a hack

– The parent ID of the process is different than all the others of the same service — these can be easily faked

– For Apache, an extra little switch on the end of the command (../httpd -DSSL -d3) – ‘d3′ is usually the name of the hack file that is being executed through Apache (not always).

Now I know you probably have the urge to kill -9 it, but don’t. Killing a hack process before determining what it is or where it came from is only going to let it happen again.

Tracking the process:

Once you know which processes should not be running, you need to know where they came from. The *easiest* way to track a process is through the lsof command. If you take the process ID (not the parent) and run an lsof it will tell you everything you want to know:

lsof -p 8748

From here I can see that this process is running from a folder in a user’s account. All I need to do now is kill it off, then remove the hack files from that user’s directory and let them know to update their shit.

TMP Directory Hacks

I’ve seen my fair share, but many hack processes will be spawned from the temp partitions of the server (like /tmp, /tmpDIR, /dev/shm), as these are usually set to 777 to allow programs to store their temporary files. When you first look in your TMP directory you’re going to see a lot of stuff — this is normal, but you’ll want to look for nobody-owned files that stand out. Just a hint, some hacks will try to trick you with their names. I have seen many that are named “…” or “httpd” to make them look legitimate, but I know that httpd (the service that runs Apache) should not be in the tmp directory…and that’s what makes it stand out.

Securing the tmp Partition

I’m not going to go too far into this here, because there’s already a very sexy website with this information available.

If you are on a cPanel server, you should run /scripts/securetmp in addition to following the instructions above.

Dun Dun Dunnnnnnnnn

Posted by Nessa | Posted in Uncategorized | Posted on 03-08-2007


Just a nice life lesson for my fellow lazy programmers:

I was looking at this site the other day in class while I was researching some crap on sub-netting (which is not one of my high points btw) and I noticed an all-too-obvious URL structure that just screamed “hack me! please!” It’s a pagerank 5 site so I know that it’s getting quite a bit of traffic, so I’m surprised this hasn’t happened enough to the point where the site developer would fix his shit. Probably an example of the worst URL compilation I’ve seen in a while:

I wrote simple php mailing script called ‘spam-me.php’ and uploaded it to my school space, then ran it off the guy’s site. I think I sent one of my professors an email about how unsatisfied his wife is, simply by tacking on my URL as the page definition:<omitted>/spam-me.php

It was even better when I was able to view his .htaccess and /etc/passwd files by writing using the passthru function in another script that I ran from his site:

<?php passthru("cat ./.htaccess");
passthru("cat /etc/passwd");


Since I’m a good person I emailed the guy about this little security problem of his. I can’t say he took it very well (it was more like someone killed his dog and left parts of it bundled up in gift wrapping on his doorstep), but the next day he took his site down. I made a point to mention that this wouldn’t have happened if he:

  1. Used the file_exists() function to specify what filenames can be presented in his URL
  2. Had mod_security installed so I couldn’t view his .htaccess
  3. Maybe disable allow_url_fopen so my site couldn’t be called as an include
  4. Had open_basedir protection so his system files can’t be accessed by php

Worse case I could have sent out a school-wide email offering penis enlargement pills, and then execute a root kit on his server. But then again, I’m a nice person, remember?

Creating your Own “Access Groups” In Linux

Posted by Nessa | Posted in Uncategorized | Posted on 22-05-2007


We started cracking down a bit on system binaries being executeable by end users on our shared hosting servers, which consisted of chmod-ing things like ‘wget’ to 700 so only root users have access. If you’re on shared host, it’s likely that you’ve encountered this kind of restriction before, and if you’re a server admin you probably know why this is necessary.

A typical scenario I’ve seen in many cases is some user’s crappity software gets exploited and executes the ‘wget’ command to download hacks and warez onto the server. I’ve also seen typical Linux functions be abused by hack processes because the access was not being controlled — it’s only safe to say that certain system binaries should be restricted to only trusted users….programs that I find particularly pervious to hacks are those like wget, lynx, scp, sh, and exec.

The issue with this (and the point of this article) is that if you suddenly disable these functions you’ll probably find yourself with a dozen complaints from your users who were using them. I’m all about fairness, so I’m not about to tell someone to rewrite their scripts because of a server-side change. Instead, I created a group on the server and added those users to be able to have access to what they needed, and chgroup-ed the binaries to that group.

I’ll use the wget example first. Say you have ‘user1‘ and ‘user2‘ that both need to be able to use wget, which is currently set to root:root 700. You’ll need to first create a file called ‘’ and insert this script:

if [ $# -ge 2 ]; then
if [ $UID == 0 ]; then
egrep ^$1 /etc/group > /dev/null
if [ $? == 0 ]; then
while [ $# -gt 0 ]; do
echo $GROUPNAME `groups $CURRENT` |sed 's/.*: //g' | sed 's/ /,/g' | usermod -G `cat -`,$GROUPNAME $CUR$
echo "the group $1 does not exist."
echo "you must be ROOT to run this script."
echo "usage: $0 grp usr1 [usr2 ... usrN]"

I know, I know, you’re probably asking why I dont use useradd +G or something like that. I tried, but in this case those commands are not appropriate. Anyways, go ahead and create your group:

root@vps [~]# groupadd wgetters

Now, simply run the script and add your users to that group:

root@vps[~]# sh wgetters user1 user2

Run id user1 to make sure that user was added to the group — you should see something like this:

uid=32010(user1) gid=32012(user1) groups=32012(user1),32014(wgetters)

Now if you chown the wget binary to root:wgetters / 750 , then only the users in that group can use wget, and their actual group identity would be unaffected.

It wouldn’t hurt mentioning that wget is often unnecessary, as many scripts can be run other ways:

php -q scriptname.php

perl scriptname.cgi


lynx (assuming that you have lynx enabled)