How to Commit Genocide on Annoying Processes
Posted by Nessa | Posted in annoyances,hacking,linux | Posted on 21-11-2007
17
A few days ago I came across some processes on one of our servers that just wouldn’t die. Even after doing a kill -9 and all that good stuff, more would just keep spawing until there were dozens running on the machine. A head system admin of ours gave me this command, which will mass-kill all alike processes so they don’t have a chance to re-spawn each other.
The processes running were all some form of “init_”, like init_1, init_13, etc. To kill these:
ps aux |grep init_ |awk ‘{print $2}’ |awk ‘{print “kill -9 ” $1}’ | sh -v
The ‘grep init_’ should reflect the common name of all the processes.
i wonder what would happen if you just skipped that first grep altogether. ;-)
@Christian: I believe your server would shut down.
Actually, I’m going to test that on one of my VE’s right now…
root@server [~]# ps aux | awk ‘{print $2}’ | awk ‘{print “kill -9 ” $1}’ | sh -v
kill -9 PID
sh: line 1: kill: PID: arguments must be process or job IDs
kill -9 1
kill -9 13404
Connection to server.nerdphalanx.com closed by remote host.
Connection to server.nerdphalanx.com closed.
VEID 13271 exist mounted running
Looks like it actually just kills off anything it gets to in the list until it gets to either the command you just entered, or your ssh session (which it did mine).
–
the sh -v at the end is superfluous, and it’s possible to consolidate the awks to a single awk program:
ps aux | grep init_ | awk ‘{print “kill -9 ” $2}’
and it’ll execute a bit faster. ;)
oh, my bad. Was looking at the wrong number:
sudo vzctl status 13386
VEID 13386 exist mounted down
It kills everything afterall!
LOL…this command is to kill specific processes with a common name.. Try it with httpd:
ps aux |grep httpd |awk ‘{print $2}’ |awk ‘{print “kill -9 ” $1}’ | sh -v
The reason the sh -v is there is because without it the server will only echo out the command…it actually executes the stuff before it into the shell. I usually leave that part off before I run the command so that I can test the output first to avoid mistakes. But it works, because I use it all the time =)
Ah, dur. I realize what my shorter version was doing wrong now. You can do it like this without needing sh:
ps aux | grep httpd | awk ‘{system(“kill -9 ” $2″)}’”
there should not be a trailing ” on that, I don’t know how that got there.
This command should do the same thing:
root@server:~# killall -9 httpd
or for base install Debian systems that don’t have killall
root@server:~# kill -9 `pgrep httpd`
No, it doesn’t do the same thing. We’re talking about hack processes spawned outside of httpd that can’t be killed with a simple killall command. I’m sure it sounds easy just to do a killall -9 but any system admin would know that 1) some processes hide their true names and 2) some processes refuse to die that way. In other words, if a kill/killall would have fixed the problem, I wouldn’t have even made this post.
pwned
I’ve never heard of a SIGKILL not killing a process. Also, if the process hides it’s true name than doing a ps isn’t going to show it’s real name either, since at that point they’ve changed the contents of /proc/$PID/cmdline. While killall does read from /proc/$PID/stat which wouldn’t necessarily be the same as output from ‘ps aux’, “kill -9 `pgrep name`” reads from both /proc/$PID/stat and /proc/$PID/cmdline.
I also forgot about the pkill binary, which combines the powers of kill and pgrep into one sweet command. so it could be simplified even further into “pkill -9 $apparent_process_name”.
I always thought pgrep was a cool and underutilized tool, it supports regex, and various flags to limit the scope too (“pkill -u apache sendmail” is an example). I just wanted to be nice and let you know about it if you didn’t already. No need to insult my admin skills, that’s how flame wars get started. kthxbye
There are many times where a process will spawn and refuse to die via sigkill, there have been few cases where I’ve had to actually reboot a server because of stale processes that don’t want to die. No insults intended.
I haven’t seen anything stand up to a signal 9 since I started with Linux when I was 12, though I suppose that doesn’t rule out the possibility.
Now I feel old :(
Yea, it happens some times but usually if you let the process site for a few minutes it like ‘unlocks’ itself or something. Weird because that defeats the whole purpose of a sigterm.
Oh! I have seen zombie processes that refuse to be killed, but they aren’t actually processes, just the proc tree from a child process has exited, but the return code of the process hasn’t been read by the parent process. Maybe a process that’s blocked waiting for IO wouldn’t die either? Next time you find one of those, could you look at the status column of ps? I’m really curious now!
Most of the hack processes I’ve come across were spawned by processes that no longer exist, but only a fraction of those are ones that won’t respond to a sigterm. I forgot the actual representation, but I believe they usually have a ‘T’ next to them when you do a ‘ps aux’. Your explanation makes sense though
Haha, I actually encountered this just today. I came across this blog again looking for something that would be more final than a kill -9. Unfortunately there’s nothing. The system’s getting rebooted, and there’s going to be alot of angry customers :)
ps …. | …. | ….. | xargs kill -9