i guess i posted in the wrong forum earlier, sorry about that. Ive gone through the code now, and you definitely need some help if you want this project to move forward, im seeing alot of options here to optmize the speed by at least 100-150% by using other functions than file() and parse_url().
In case you dont believe i can see the code (trust me, i wont spill it to anyone, leak or leak the source code to anyone, but i might work a bit on it myself and send you some suggestions if you'd like.)
First file includes some parts of this code for instance:
$_REQUEST['op'] = 'crawlproc'; if(isset($_SERVER['REQUEST_METHOD'])) { echo 'This tool can be executed in command line mode only'; exit; }
Before getting angry, i just want to help on an dead project, and this certainly is the reason why people can run directly from browser as you deny them.
Please reach out to me if you need help with finding better functions, as many of the current functions are marked as deprecated - or soon to be.