Optimize your network for Xrumer/Scrapebox - Increase links per minute

[freelander]

Found this on other Forum

Optimize your network for Xrumer/Scrapebox/SEO Tools

I read a lot of misguided information everyday about configurations for Scrapebox. The most common being related to Timeout/Maximum Connection settings. I wrote this guide to break some common misconceptions about Scrapebox and help you optimize your network to achieve the greatest potential. I have done my best to write this guide with as little technical jargon as possible. However, this is not a guide to be used lightly. Changes you make here can severely affect the performance of your operating system. It is highly recommended you revert back to your default settings after using Scrapebox.

By default Windows 7 does a pretty good job at optimizing your network card and TCP/IP settings for normal everyday use. Unfortunately, Scrapebox is not an application that falls into that category. Scrapebox hammers your network connection, router and network card with multiple simultaneous connections at a very fast rate. By default your hardware cannot keep up with the demand necessary to run Scrapebox to it's highest potential. In this guide you will make the necessary changes to prepare your network for the high demand usage of Scrapebox.

What this guide will do:

• Increase Network Stability and Reliability
• Reduce Network Overhead and Improve Scrapebox Performance
• Optimize your network connection for mass HTTP requests
• Reduce Comment Poster Times while maintaining Success Rate
• Stop your network from throttling your connections

First lets talk a little about what Scrapebox does and how it affects your connection.

When Scrapebox begins posting your new Auto Approve list, it instantly opens up multiple connections from your network to the internet. The amount of connections it opens can range from 1 to 500, defined in the SB Maximum Connections settings. Incidentally, if you were to set your SB installation to attempt 500 connections on an unoptimized network you would almost immediately crush your own network. Default settings on home user level hardware cannot withstand 500 outgoing/incoming connections at a constant rate for very long. Most network hardware is designed to immediately block all connections at the first sign of this. It is a built in feature to help protect home users from malicious activity.

On top of the hardware level, you have limitations in your own Operating System. By default Windows Vista, 7 and Server 2008 disable some resources that we can use to help improve how our network connection handles HTTP Requests, Packet Processing, Memory and CPU management, compression and various other goodies. It is also designed by default to operate optimally for normal network applications. Luckily, Scrapebox is NOT a normal network application. We need to make the necessary changes to accommodate to the needs of this application.

I will jump a little more into the technical side later on. Let's answer some questions first,

I have XX private proxies, how many connections should I use?
This question shows up very frequently here on BHW. And the usual community response is to set your maximum connections to the same number of private proxies you are using. In a default environment this is a great way to insure a reliable posting session. However, the length of time it takes to complete such a session can run past the 24hr mark!

That's fine if you don't mind your connection running for 24 hours to complete that 100,000 URL Blast. Personally, I like to get things done right AND fast. Take a look at the image below.



What you are looking at is not an Auto Approve list, but a fresh URL scrape ran at 100 connections on 10 private proxies. It only took 3 Hours and 30 Minutes to finish a list of 91,000 at a success rate of 33%! The rest is mostly Captcha Blogs or 404 Errors.

Here is a test run using a week old Auto Approve list:



Just over 80% posting success on a 1k Unique Domain AA List. This time just under the 6 minute mark, using 100 connections on 10 private proxies! The second pass posted to the remaining list just under a minute.

These are just two examples of what you can achieve by properly configuring your system for Scrapebox.

What should my Timeout setting be?

I'm going to state right here and now: 90 Seconds. Anything less and you are just going to hurt your success percentage and bring your network to it's knees. When dealing with large amount of connections, it causes a higher level of network congestion that is not seen in normal use. Setting your timeout to 90 seconds, creates the necessary buffer to wait for a response packet back from the website SB is trying to post on.

You should go ahead and change your timeout settings now. This change can be made under Settings > Adjust Timeout Settings > Then move the slider for Faster Commenter all the way to the right.

Ok, I will answer more questions later. Let's move on to making some changes.




Tweaking the Operating System

******************************


Warning! This guide is written for Windows 7/Server 2008 OS only. Many of the changes made here are for Windows 7 environments only. USE THIS GUIDE AT YOUR OWN RISK. I will not be held responsible for any problems you cause by following this guide. You are to proceed at your own risk. Make sure to backup when I tell you to do so, in case you need to restore original settings. Warning!



All of the changes to be made can be done by using the registry editor. To speed up the process we are going to use an application designed specifically for tweaking TCP settings in Windows 7 and Server 2008. Please read about each option before making any changes. Some options require additional changes or hardware support.


To start you will need to download the latest version of SG TCP Optimizer @
http://www.speedguide.net/downloads.php

After you have downloaded SG TCP Optimizer, go ahead and launch the application. There is no installation necessary and will load immediately.

Let's take a look at what you will see:



By default, you will see a lot of options set as default or blank. We will now make the changes needed to optimize your system for heavy Scrapebox usage.

Your first step is creating a BACKUP of your current settings. This can be done by going to FILE > BACKUP CURRENT SETTINGS. Do this now before you make any changes!

After you have completed backing up your settings. Move on to selecting the proper Network Adapter from the drop down box. Make sure to select the connection you use when Scrapebox is in use.

Once you have that done, go ahead and select the CUSTOM option on the lower right hand corner of the application. You will now need to read through the following options and set the accordingly. I have labeled my RECOMMENDED options in red color under each setting. Some options may require Router, Network Adapater and/or CPU support! It is your responsibility to read this information before changing any options.

General Settings

TCP Window Auto-Tuning
TCP Auto-Tuning enables TCP Window scaling by default and automatically tunes the TCP receive window size for each individual connection based on the bandwidth delay product (BDP) and the rate at which theapplication reads data from the connection, and no longer need to manually change TcpWindowSize registry key value which applies to all connections.

Setting: Normal

Congestion Control Provider
By default, Vista and Windows 7 have CTCP turned off, it is only on by default under Server 2008. Turning this option on can significantly increase throughput and packet loss recovery.

Setting: CTCP

TCP Chimney Offload
TCP chimney offload enables Windows to offload all TCP processing for a connection to a network adapter. Offloads are initiated on a per-connection basis. Compared to task offload, TCP chimney offload further reduces networking-related CPU overhead, enabling better overall system performance by freeing up CPU time for other tasks.

Setting: Enabled

Receive-Side Scaling State
The receive-side scaling setting enables parallel processing of received packets on multiple processors, while avoiding packet reordering. It avoids packet reordering and separating packets into "flows", and using a single processor for processing all the packets for a given flow. Packets are separated into flows by computing a hash value based on specific fields in each packet, and the resulting hash values are used to select a processor for processing the flow. This approach ensures that all packets belonging to a given TCP connection will be queued to the same processor, in the same order that they were received by the network adapter.

Setting: Enable (if you have 2 more processor cores, otherwise set as default)

Note: For this setting to work your network adapter needs to support RSS. This can be enabled by going into your network adapter's driver configuration. Under the Advanced tab if there is an option for Receive Side Scaling you need to set this to Enabled.
Here you can see the setting:



Direct Cache Access (DCA)
Windows 7 and 2008 Server (but not Vista) add NETDMA 2.0 Direct cache access support. Direct Cache Access (DCA) allows a capable I/O device, such as a network controller, to deliver data directly into a CPU cache. The objective of DCA is to reduce memory latency and the memory bandwidth requirement in high bandwidth (Gigabit) environments. DCA requires support from the I/O device, system chipset, and CPUs.

Setting: Enabled

Note: You will likely only see performance increase if using a Gigabit router with Cat-6 cables and Gigabit Capable Network Card. This setting requires support from your Network Card, System Chipset and CPUs.

NetDMA (TCPA)
NetDMA enables support for advanced direct memory access. In essence, it provides the ability to more efficiently move network data by minimizing CPU usage. NetDMA frees the CPU from handling memory data transfers between network card data buffers and application buffers by using a DMA engine.

Setting: Enabled

Note: Not available under Windows Vista

Time-to-Live (TTL)
Time-to-live (TTL) is a value in an Internet Protocol (IP) packet that tells a network router whether or not the packet has been in the network too long and should be discarded. For a number of reasons, packets may not get delivered to their destination in a reasonable length of time.

This setting needs to match the timeout setting provided in Scrapebox. We will change the timeout setting for Fast Commenter to 90 Seconds later.

Setting: 90

ECN Capability
ECN is only effective in combination with AQM (Active Queue Management) router policy. It has more noticeable effect on performance with interactive connections and HTTP requests, in the presense of router congestion/packet loss. Its effect on bulk throughput with large TCP Window are less clear.

Setting: Enabled

Note: For this setting to work you router must support ECN. To check if your router supports ECN you can run a test using Internet Explorer at: http://www.microsoft.com/windows/usi...d/default.mspx

After the test completes, it will contain information about your internet connection and if ECN is supported by your router.

Windows Scaling Heuristics
Windows Vista/7 has the ability to automatically change its own TCP Window auto-tuning behavior to a more conservative state regardless of any user settings. It is possible for Windows to override the auto-tuning level even after an user sets their custom TCP auto-tuning level.

Setting: Disabled

Checksum Offloading
This setting allows for reducing CPU load by offloading some tasks required to maintain the TCP/IP stack to the network card. Theoretically, Windows should automatically detect capable network hardware.

Setting: Enabled

Window Scaling
The TCP window scale option is needed for efficient transfer of data when the bandwidth-delay product is greater than 64K. For instance, if a T1 transmission line of 1.5Mbits/second was used over a satellite link with a 513 millisecond Round Trip Time (RTT), the bandwidth-delay product is (1500000*.513) = 769,500 bits or 96,188 bytes. Using a maximum buffer size of 64K only allows the buffer to be filled to 68% of the theoretical maximum speed of 1.5Mbits/second, or 1.02 Mbit/s.

By using the window scale option, files can be transferred at nearly 1.5Mbit/second utilizing nearly all of the available bandwidth.

Setting: Enabled

Timestamps
TCP is a symmetric protocol, allowing data to be sent at any time in either direction, and therefore time stamp echoing may occur in either direction. For simplicity and symmetry, we specify that timestamps always be sent and echoed in both directions. For efficiency, we combine the time stamp and time stamp reply fields into a single TCP Timestamps Option.

In the case of Scrapebox usage this setting should be disabled. It creates an addition 12 byte overhead for each packet being set and further congests the network with unnecessary information.

Setting: Disabled

Advanced Settings Tab

Internet Explorer Optimization
This setting has no affect on Scrapebox performance and should just be left as default. You can set each field accordingly to improve IE performance.

MaxConnectionsPerServer Setting: 8
MaxConnectionsPer1_0Server Setting: 8

Host Resolution Priority
This section improves DNS and host name resolution in general. It helps web pages load faster, and has negligible effect on downloads. This will help increase the speed at which Scrapebox can load a website for commenting.

Local Priority Setting: 4
Host Priority Setting: 5
DNS Priority Setting: 6
Netbt Priority Setting: 7

SynAttackProtect
This is a fairly undocumented feature in included in recent versions of Windows. This setting should be set to default to reduce any unnecessary overhead while running Scrapebox.

Setting: Disabled

Network Memory Allocation

LargeSystemCache Setting: Enabled
Size Setting: Optimized

Quality of Service

QoS: NonBestEffortLimit Setting: 0

DNS Error Caching
Windows has built-in DNS (Domain Name System) caching, which caches resolved host names for faster access and fewer DNS lookups. This is generally a great feature, with the only downside that failed DNS lookups get cached by default as well. When a DNS lookup fails (due to temporary DNS problems), Windows still caches the unsuccessful DNS query, and in turn fails to connect to a host regardless of the fact that the DNS server might be able to handle your lookup seconds later. One can manually flush failed DNS lookups by typing ipconfig /flushdns in Command prompt. Or you can simply set the 3 values in the Optimizer to "0", and it will set the relevant Registry settings.

This is a very important change for Scrapebox. When you are posting comments, Windows by default keeps a record of all domains that are accessed by your network card. Even failed connections that would work during your next run might be passed over as failed again even though the connection was alive.

NegativeCacheTime Setting: 0
NetFailureCacheTime Setting: 0
NegativeSOACacheTime Setting: 0

Gaming Tweak - Disable Nagle's algorithm
These settings are related to Nagle's algorithm. Best to leave these as default as they will not affect our performance with Scrapebox positively.

Setting: Default

Dynamic Port Allocation

MaxUserPort Setting: 65535
TCP TimedWaitDelay Setting: 30

Finished! Go ahead and apply your settings and restart your machine when prompted.

I will be adding more information to this post later on. I will add a section that suggests some network adapter and router configuration changes. As well, I will try to answer some more questions regarding Scrapebox and how to properly configure it's use for optimal network reliability.

I hope you will get good use out of this post. Rep+Thanks are always appreciated if you do!

[pkfarnv]

Very useful and informative sharing.i have got it as useful.