I am scraping a site for every 50-90 seconds of data, which is the C # console application. Running on NT 4.5 There are some values in it that I am posting on the site and stop some other process based on the returned value. The problem is after saying about a thousand hits or so I look like a DNS error. I am trying to decide what is the source of the problem first, before trying to fix it. Below are some of the errors that I see in my logs:
- Remote name could not be resolved
- Unable to connect to remote server
- Faced unexpected character while parsing the value & lt;. Path '', line 0, position 0
- An existing connection unable to read data from the transport connection was forcibly closed by the remote host.
- Connection unable to read data from the transport An established connection was rejected by the software in your host machine.
About 60% of the time I got the first error. The remaining 40% is divided between the errors listed above. Are they the problems arising from websites that I am scanning at intervals or some other DNS? For all practical purposes, I can fix the scrapping website properly until I keep the interval between automatic hits above 45 seconds which I am doing. The data I am downloading is on average 30kb per hit, please help me understand what is going wrong and what I can try to do.
D says that you are running against an automated system that protects the site against DDOS attack Is designed to be.
It is seeing that your single IP address is repeatedly killing in a short time and is just blocking your resolution of the last server
No comments:
Post a Comment