Generator crawls just a few links of our site... (Joomla + VM)
« on: December 28, 2010, 11:41:35 AM »
Hi there,

I've been reading the forum already about this problem, and already found out that there have been more people having problems with crawling through Joomla + VM sites...

This is what I'm experiencing and I hope that one of you can have a look at it if possible, the generator works great with another site of mine so I'm pretty satisfied. But I really need this site to work also...

- I get 56 links crawled instead of around 300 that should be in there...
- It takes ages for the links to get crawled
- I have the feeling it takes random links, because not all the menu items end up in the XML, nor all the products end up there...

So because on the other site it works good I assume it could well be a problem that is server related, but I'd really like you opinion too before contacting the host...

Looking forward to here from you.
Ciao,

Marco

| |
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #1 on: December 28, 2010, 11:49:42 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #2 on: December 28, 2010, 12:16:15 PM »
PM is sent btw...
Ciao,

Marco

| |
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #3 on: December 28, 2010, 07:28:06 PM »
Have you had the chance to check already?
« Last Edit: December 28, 2010, 07:30:43 PM by marcoboer1973 »
Ciao,

Marco

| |
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #4 on: December 28, 2010, 07:51:58 PM »
Hello,

looks like your site is configured to block requests if they are sent too often (like "anti-flood mod for joomla"), as a result generator's crawler fails to index those pages. I've added 1 second delay after each request to slow down the crawling process, that should resolve the issue.
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #5 on: December 28, 2010, 07:53:12 PM »
OK, cool, I'm gonna check it out, will let ya know...
Ciao,

Marco

| |
Re: Generator crawls just a few links of our site... (Joomla + VM)
« Reply #6 on: December 28, 2010, 08:11:31 PM »
Hi Oleg,

It seems to work fine now, thanks a lot!
Ciao,

Marco

| |