Sitemap Generator Forum

Dear Team, I am crawling my site which has 1000's of pages. The crawler has been running for around 20+ hours. Wanted to check if there is any setting I can change to make it faster or if this is the expected time.

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

i.e., the aster website pages are loading, the faster would be the crawling.