PHP Browser-Based Website Crawler

Posted by Nessa | Posted in Uncategorized | Posted on 07-05-2007

8

I figured out a way to create a php website crawler that can be run via web browser instead of command line. You can use this to harvest links from a website for use in a database or search engine…or to see how easily a spider or bot can creep your site. Try it here!

<html>
<head><title>PHP Website Crawler</title></head>
<body>
<font face="verdana" color=#66ccff">
<form id="crawl" method="post" action="">

<label>URL:
<input name="url" type="text" id="url" value="<?php $url; ?>http://website.com" size="70" maxlength="255" />
</label>
<br />
<br />
<label>
<input type="submit" name="Submit" value="Crawl!" />
</label>
<br />
</form>
</body>
</html>
<?php
if (isset($_POST['url'])) {
$url = $_POST['url'];
$f = @fopen($url,"r");
while( $buf = fgets($f,1024) )
{
$buf = fgets($f, 4096);
preg_match_all("/<\s*a\s+[^>]*href\s*=\s*[\"']?([^\"' >]+)[\"' >]/isU",$buf,$words);
for( $i = 0; $words[$i]; $i++ )
{
for( $j = 0; $words[$i][$j]; $j++ )
{
$cur_word = strtolower($words[$i][$j]);
print "$cur_word<br>";
}
}
}
}
?>

Fake a 404 to Block an Entire ISP

Posted by Nessa | Posted in Uncategorized | Posted on 06-05-2007

0

I’ve been getting a sudden influx of traffic so I decided to take a look at my referrer stats, and even though there wasn’t anything blatently obvious out there, I did notice some strange injection-like URL strings coming from a certain ISP. I figured the best way to block them without having to figure out and set an IP range to block is to just fake the existence of my whole website. Here’s how you can force users from an entire [unwanted] ISP to a 404 page…you’d want to put this in a page that is consistently loaded with your site. If you’re a WordPress user, the best spot would be your header.php.

$host = gethostbyaddr($REMOTE_ADDR);
if (stristr($host, "anyisp.com")) {
Header("HTTP/1.1 404 Not Found");
print "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">
<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
The requested URL / was not found on this server.<P>
<P>Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.
<HR>
<ADDRESS>Apache/1.3.37 Server at www.v-nessa.net Port 80</ADDRESS>
</BODY></HTML>";
exit;
}
?>

PHP 5.2.1 is a Remote Include Hater

Posted by Nessa | Posted in Uncategorized | Posted on 28-04-2007

2

I found this out after troubleshooting a few sites a couple weeks ago. Unlike all other previous versions of PHP, with 5.2.1 you can no longer use a URL to include files, even if you have allow_url_fopen enabled in your php.ini.

Just a quickie about what an ‘include’ is, it’s a php function that lets you include the contents of a file into another. It usually takes the form of this:

include('page.php');

This is a local include, and is usually the preferred method. A remote include is that of a URL:

include('http://google.com');

Most people would agree that remote includes are a major security issue for novice coders who don’t have any kind of file validation in place to protect their scripts, but if you’re convinced that your code is down, you can add this line to your php.ini file to allow remote file includes:

allow_url_include On

Now, this directive did not exist until php 5.2.1 was released so Cpt. Obvious says that:


1)
you cannot use it with versions prior to 5.2.1

2) You’ll need to add it to your php.ini, as you probably won’t find it in there already

3) allow_url_fopen has to be enabled

4) This has to be in the php.ini server-side, and not the .htaccess

If you’d like to know more about remote file inclusions (the bad kind), read my article.

Suhosin Will Make your PHP Hard

Posted by Nessa | Posted in Uncategorized | Posted on 20-04-2007

0

SuhosinI noticed a vague mention of Sohosin on a PHP blog that I read on occasion and I decided to give it a whirl to see if it’s as sexy as is sounds. So far my server hasn’t crashed, so I’m willing to recommend it to anyone who’s interested in hardening their PHP. Ok, sorry. I really can’t say that without chuckling. Yes, welcome back to third grade.

If you check out the developer’s site you should pretty much get the idea of what it does, but basically closes some of the security holes that we see with PHP all the time. Not to say that it will make your php 4.3/MySQL3/globals on/port 22 open server any more secure, but if you’re running any of the latest stable security releases you might be somewhat interested.

I’m currently running PHP 5.2.1, which is the latest release of PHP5 at the time of this writing. You can essentially install this on any PHP4+ server that you have root SSH access to. I opted to install the DSO, as I absolutely hate recompiling PHP. Installing Suhosin as a dynamic shared module will not require you do recompile anything, and is therefore the preferred method for lazy people.

Note: I’m using the latest (and only) release at the time of my writing, but head over to the download page to see if there is anything newer:

http://www.hardened-php.net/suhosin/download.html


cd /usr/src
wget
http://www.hardened-php.net/suhosin/_media/suhosin-0.9.20.tgz
tar -xvzf suhosin-0.9.20.tgz
cd suhosin-0.9.20

Now to install:

phpize
./configure
make
make install

It should return a line something like this:

Installing shared extensions: /usr/local/lib/php/extensions/no-debug-non-zts-20060613/

Now, all you have to do is add this line to your php.ini and restart Apache. Of course, the path should be what the installation output gave you:


[Suhosin]
extension="/usr/local/lib/php/extensions/no-debug-non-zts-20060613/suhosin.so"

**If you are running eAccelerator, this should be above the eAccerelator configuration

When you load up your phpinfo file you should see the module loaded near the Zend section. Everything should be fine as-is, but it you’re one of those people who has to reconfigure everything, knock yourself out.
Suhosin

Optimizing PHP, Revisited.

Posted by Nessa | Posted in Uncategorized | Posted on 16-04-2007

1

I wrote an article a while back on PHP optimization, but it was pretty lacking in most aspects, probably because I’m a lazy poster. I’ve revisited that article and reposted to hopfully have it be a little more helpful on the area.

Timing Your PHP Scripts

Posted by Nessa | Posted in Uncategorized | Posted on 15-04-2007

1

This was just a little code addon that I put together as part of a tutorial I wrote on a friend’s site (and copied on mine) about PHP optimization. Added to a page on your site, it will calculate how much time it took for a page or script to execute using PHP’s microtime() function. I only added this to my main page, but you can easily create a plugin or include file to show the generation time of all your pages.

First, add this code to the very beginning of your PHP file:

<?php
$stime = microtime();
$sarray = explode(" ", $stime);
$stime = $sarray[1] + $sarray[0];
?>

Now, add this to the very end:

<?php
$etime = microtime();
$earray = explode(" ", $etime);
$etime = $earray[1] + $earray[0];
$ttime = $etime - $stime;
$ttime = round($ttime,3);
echo "This page loaded in $ttime seconds.";
?>

That was easy, wasn’t it? You should now see a little line at the bottom of your page that shows how long it took to execute. There is an example on the bottom of my home page.

How to Install PHP6

Posted by Nessa | Posted in Uncategorized | Posted on 26-02-2007

4

This was a little experiment gone somewhat wrong, when I tried to upgrade my VPS to PHP6. I swear it worked, but I should have known that nothing supports it — WordPress just crapped out a bunch of errors. On a higher note though, it seems to be hella secure as you can no longer use magic quotes or globals, otherwise Apache will fail. So if you really want to be on the bleeding edge, here’s how you install PHP6:

I didn’t really need to install a whole bunch of stuff, but depending on how your webserver is already set up you may need to install extra dependencies, but this will become obvious during the compile.

1) Make sure your autoconf version is up to date with version 2.13 or higher:

autoconf -V

2) Install ICU…you can find your version here.

wget ftp://ftp.software.ibm.com/software/globalization/icu/3.6/icu4c-3_6-src.tgz
tar -xzvf icu4c-3_6-src.tgz
cd icu*/source
mkdir /usr/local/icu
./configure --prefix=/usr/local/icu && make && make install

3) Git yur PHP! You can find the latest dev release of PHP6 at http://snaps.php.net/. Your wget and tar targets will be different, as the development version changes frequently.

wget http://snaps.php.net/php6.0-200702230530.tar.gz
tar -xvzf php6.0-200702230530.tar.gz
cd php6.0-200702230530
./buildconf

This will build the configuration and let you know if something is missing. Once this is complete, run the configure script…I’ve the mandatory stuff in there, but you’ll also want to include any PHP modules that you need. You may find it easier to copy this from a phpinfo file, minus the quotes.

./configure --with-apxs=/usr/local/apache/bin/apxs --prefix=/usr/local/ --with-icu-dir=/usr/local/icu

You’ll probably get build errors, which is usually due to 1) a particular module no longer being supported or 2) PHP cannot find that module’s files on the server. In this case you’ll want to see which module the configure command stops at, then either leave it out or make sure the module is compatible with the correct location specification in the configure command. For instance, I have Ming installed and this is the directive in my configure command:

--with-ming=../ming-0.2

Once you have a good build, you can install your PHP:

make && make install

4) Configure Apache

Usually the PHP installation with the –with-apxs switch will add the necessary entries to your httpd.conf, but if not you will need to comment out the loaders for php4/php5 and add the one for php6:

/etc/init.d/httpd restart (or whatever command you use to restart Apache)

LoadModule php5_module libexec/libphp6.so
AddModule mod_php6.c

Now…twenty bucks says that there is now something on the server that doesn’t work, which will be obvious with the Apache restart you just did. You’ll want to check your error logs for the obvious problems, then correct the issues in your php.ini and other files it mentions. The most common issue is with the magic quotes gpc and register globals.

If you have any third-party extensions like Zend. IonCube, or eAccelerator, you’ll need to re-install those as well.

There you have it…you now have PHP6 installed on your server…and now your apps don’t work! You can admire your work next time you try to load your site.

Be sure to make a pretty phpinfo() file to check the installation.

<?php phpinfo() ?>


The Basic MySQL Injection

Posted by Nessa | Posted in Uncategorized | Posted on 17-01-2007

7

Ahhhh the classic hack that doesn’t work anymore… which is why I’m posting it here. I always thought it was kind of an interesting concept but no one ever made it simple for me, so I shall do this for you.

How to do a simple MySQL Injection

Ok, so this is your basic PHP login script that asks for your username and password, which would then query the database to authenticate you:

$user = $_POST["username"];
$pass = $_POST["password"];
$query = mysql_query(“SELECT * FROM users WHERE user=’$user’ AND password=’$password’”);
$rows = mysql_fetch_row($query);
if ($rows == 0) {
die (‘Login Incorrect!’); }

Assuming that register_globals are enabled on the server, this script will work and in return use the POST variable to query the database for an already-defined row to see if both conditions are being met, which are obviously the username/password fields. If the input does not meet this requirement, then the connection dies and returns the “Login Incorrect” error. So assume I log in with the username “nessa” and the password “sexy.” The $query string will pass this command to MySQL:

$query = mysql_query("SELECT * FROM users WHERE user='nessa' AND password=" OR"=' OR '1'='1'");

Since I used the OR clause in the password field, that can leave a few possibilities up to the database to determine whether a statement is true or false. As you can see, will always be equal to , and 1 is always equal to 1, so MySQL is happy as long as these requirements are met.

So what does that tell you? You can easily replace either the username or password fields withe a or a ” OR 1 and you will have a successful login each time. Of course there are a lot more combinations that will work — you might want to check out this site:

http://www.justinshattuck.com/?p=156&akst_action=share-this

Now seeing that this site is powered by PHP and MySQL, you probably think I’m stupid by posting this. Well quite frankly, MySQL injections are old and nearly impossible with well-scripted PHP software and good PHP environment. If you’re running a custom script or old software, here’s how you can protect your crappy software from being exploited:
Check your magic quotes setting in php.ini or .htaccess:

magic_quotes_gpc should be turned on, as this automatically slash-escapes your codes so MySQL is less likely to make a false positive. As of PHP4, this setting is enabled by default.

If you don’t want to use magic quotes, use mysql_real_escape_string():

Here’s a simple script you can use as an include to automatically escape null characters:

// Quote variable to make safe
function quote_smart($value
)
{
// Stripslashes
if (get_magic_quotes_gpc
()) {
$value = stripslashes($value
);
}
// Quote if not integer
if (!is_numeric($value
)) {
$value = “‘” . mysql_real_escape_string($value) . “‘”
;
}
return
$value
;
}
?>

And the obvious, if you’re using bundled software make sure you keep it up to date. New exploits are being found all the time, so don’t put yourself out there by not updating your shit.


Messing with PHP and MySQL

Posted by Nessa | Posted in Uncategorized | Posted on 16-01-2007

0

This just a simple tutorial on how to connect to MySQL with PHP, as well as using a MySQL query to create simple database.

The first thing you would want to do is create the database so PHP can access it. My preferred method is through phpMyAdmin or cPanel (if your host provides this)…or you can run a simple query via MySQL command line:

CREATE DATABASE `liquor`;
USE `liquor`;
CREATE TABLE `brands` (
`id` int UNIQUE NOT NULL,
`brand` varchar(40),
`type` varchar(50),
PRIMARY KEY(id)
);
INSERT INTO cars VALUES(1,'skyy','vodka');
INSERT INTO cars VALUES(2,'cpt. morgan','rum');
INSERT INTO cars VALUES(3,'ice 101','schnapps');

This code will basically create a database named “liquor” with the table “brands” that lists three of my favorite liquors in order with the type of liquor that they are. From here, you’ll want to create a database user and add it to the database with the appropriate privileges. You can usually do this in your MySQL management interface provided by your hosting company, but if not you can check this site for information on users and privileges.

Now for the PHP code, you declare your database variable and use to mysql_connect function to establish a database connection:

<?php
$db_user = "user_name";
$db_pass = "password";
$db_host = "localhost";

//connection string
$dbconn = mysql_connect($db_host, $db_user, $db_pass)
or die("Cannot connect to database");
echo "Connected to the database!<br>";
?>

Once you’ve verified that you can connect to the database, you now need to tell PHP what table you want to work with. Let’s use the one we just created:

<?php
//select a database to work with
$selected = mysql_select_db("brands",$dbconn)
or die("Could not select liquor");
?>

Now here’s the complex part of the code where we create a loop to pull the data from the three rows we created:

<?php
//execute the SQL query and return records
$result = mysql_query("SELECT id, brand,type FROM brands");
//fetch tha data from the database
while ($row = mysql_fetch_array($result)) {
echo "ID:".$row{'id'}." Name:".$row{'brand}."
".$row{'type'}."<br>";
}
?>

Of course, this example is just a dummy one that returns the same records we’re querying –so it’s not really useful in real life.

Once you have the information you need from the database, try to make it a habit of closing the connection. Using persistant connections can cause issues on the server and is not recommended for a live production site.

<?php
//close the connection
mysql_close($dbconn);
?>

Setting up an Access Log with PHP

Posted by Nessa | Posted in Uncategorized | Posted on 14-01-2007

0

I set this up back in July when my site was hosted with H-Insiders and I didn’t have direct access to my Apache logs. I figured that I could do this myself with a few basic PHP functions, and by doing so I was able to set up my own access log in a static text file that I could download whenever I wanted. This first example is to log unique visitors to your site as well as gather some important information about them. Here is the code:


<?php
session_start();
if(!session_is_registered('counted')){
$agent = $_SERVER['HTTP_USER_AGENT'];
$uri = $_SERVER['REQUEST_URI'];
$user = $_SERVER['PHP_AUTH_USER'];
$ip = $_SERVER['REMOTE_ADDR'];
$ref = $_SERVER['HTTP_REFERER'];
$dtime = date('r');


if($ref == ""){
$ref = "None";
}
if($user == ""){
$user = "None";
}


$entry_line = "$dtime - IP: $ip | Agent: $agent | URL: $uri | Referrer: $ref | Username: $user n";
$fp = fopen("access_log.txt", "a");
fputs($fp, $entry_line);
fclose($fp);
session_register('counted');
}
?>


Basically, this is what the code it doing:

session_start();

First, we have to create a session so that the script will only log this visitor’s activity until the session expires

if(!session_is_registered('counted')){

This snippet checks to see if there is a session variable anywhere, and there is it executes the next section of code.

$agent = $_SERVER['HTTP_USER_AGENT'];
$uri = $_SERVER['REQUEST_URI'];
$user = $_SERVER['PHP_AUTH_USER'];
$ip = $_SERVER['REMOTE_ADDR'];
$ref = $_SERVER['HTTP_REFERER'];
$dtime = date('r');

This is where the information is collected:

HTTP_USER_AGENT logs the browser type of the visitor

REQUEST_URI logs the page request

PHP_AUTH_USER shows the login credentials used if a user is being authenticated

REMOTE_ADDR shows the IP address of the visitor

HTTP_REFERER shows where the visitor was referred from

Now, to write to the logfile we see this code:

$entry_line = "$dtime - IP: $ip | Agent: $agent | URL: $uri | Referrer: $ref | Username: $user n";
$fp = fopen("access_log.txt", "a");
fputs($fp, $entry_line);
fclose($fp);
session_register('counted');

You’ll see that the first line is simply printing out the variables defined above as labels, which can be modifed to whatever you want. fopen and fclose are simple php commands used to open and close a file, while fputs will write the variable $entry_line to the file, with $entry_line being equal to all the information collected by the script. The name of the file we are writing to is “access_log.txt” and the path in the script should reflect the location of where you want that file to be on your server.

But what if you want to bypass sessions and log every single page hit? First of all, your log file is going to become massive, but if you would like to keep more extensive logs all you have to do is exclude this code from the script:

session_start();
if(!session_is_registered('counted')){


session_register('counted');
}
?>

Now you have set up a simple logging system to keep your own Apache access logs. Just make sure you set the text file location to where you want the script to write on your webserver, preferrabled somewhere outside your document root folder.