08-05-2016, 07:14 AM
Hello,
I wrote a script to scrape a website, there are a lot of datas to collect, between 35000 and 70000 pages... In my script, I included pause but I think it's not enough. I think the host can detect my scraping with so much pages...
What can I do ? Any advices are useful... I'm a noob in scraping... It's my first big scraping :)
Thank you :)
I wrote a script to scrape a website, there are a lot of datas to collect, between 35000 and 70000 pages... In my script, I included pause but I think it's not enough. I think the host can detect my scraping with so much pages...
What can I do ? Any advices are useful... I'm a noob in scraping... It's my first big scraping :)
Thank you :)