Every time I run the crawler my site becomes completely inaccessible - I have tried various settings and I have tried setting the delay to 2s delay every 4 requests and still it freezes my site.
My server has plenty of RAM.
When the crawler runs - no pages on the website load - nothing!