Networking: Tuning the Gigabit

Gigabit is the entry level LAN cable standard for the turn of the 21st Century. However there's more to gigabit than just getting a NIC, cable and switch. Many factors affect data transfer speeds across a network, so here are a few things to consider.

Hardware

PCI NICs usually plug into a PCI slot and many older motherboards use the earlier 32-bit, 33MHz PCI bus. This means they can transfer at speeds of up to 133MB/s. And because several devices on the motherboard use the PCI bus for a bit of a chat, the actual speed is often more like 80MB/s on a good day with the wind behind you..

Now, gigabit NICs can run at up to 1000Mb/s (125MB/s) but if the PCI bus is only capable of wheezing along at 80MB/s, this is gonna be a bit of a bottleneck in the system. It gets worse. Usually there's more than one PCI slot on a motherboard and often more than one PCI card plugged in. Each PCI card will be chattering away, chomping through the data bus and thus the actual speed or bandwidth available to the gigabit card can be even lower.

As usual, to get things to work like it claims on the box, you've gotta spend even more money buying a mobo with a 66MHz PCI bus, which can do 266MB/s. Actually, most of the new 64-bit PCI slots can fang along at 532MB/s on a 66MHz bus. These are still more common on the more pricey motherboards like the ones designed for servers, but like everything else the technology is gradually filtering down to the more er, budget conscious mobos, commonly used by the proletariat.

Software

Once you've spent an arm and a leg on new hardware, just to keep the new network card happy, the next consideration is the software, including the operating system itself. When testing data transfer between two machines and specifically, the speed of the network, you don't want to use any protocols or applications which will fetch data from the hard disk. Much better to transfer data from memory to memory, which will obviate the hard disk factor, yes?

There are a few utilities out there which will do precisely this, including TTCP from PCA USA. Some early adopters of gigabit have performed initial tests indicating a transfer speed of as low as 200mbps. That'll teach 'em to to jump on the bandwagon so quickly, the rich bastards..

Some fiddly, twitchy, tweaky geeky types who can never leave things alone sometimes feel compelled to mess with the MTU setting on the NIC, which may work as long as the switch supports this, although many switches don't. However you should still be able to get gigabit running nice and fast with the standard MTU setting of 1500 bytes, so best not to screw around with it, eh chaps?

A more useful tweak is the TCP window size. The TCP window controls the flow of data and is negotiated at the start of a TCP connection. However, too small a size will slow things down as TCP can only use the smaller of the two end systems' capabilities. Here are some suggested tweaks for the most common operating systems.

FreeBSD/Linux

FreeBSD and Linux should both have a sysctl utility. For all of the following options, entering the command sysctl variable=number should do the trick. To view the current settings use: sysctl

Maximum window size
FreeBSD:
kern.ipc.maxsockbuf=262144
Linux:
net.core.wmem_max=8388608

Default window size
FreeBSD, sending and receiving:
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
Linux, sending and receiving:
net.core.wmem_default = 65536
net.core.rmem_default = 65536

RFC 1323:
(enables the useful window scaling options defined in rfc1323, which allows the windows to dynamically get larger than specified above)

FreeBSD:
net.inet.tcp.rfc1323=1
Linux:
net.ipv4.tcp_window_scaling=1

Buffers:
(sending large amounts of data can run the operating system out of buffers. This option should be enabled before attempting to use the above settings). To increase the amount of "mbufs" available:

FreeBSD:
kern.ipc.nmbclusters=32768
Linux:
net.ipv4.tcp_mem= 98304 131072 196608

These straightforward tweaks should turbocharge the gigabit performance. Check with TTCP and compare results. There are other options available for adjusting the UDP datagram sizes as well, but we're mainly focusing on TCP here.

Windows XP | 2000 Server | 2003 Server

Open the registry editor and drill down to:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Add a registry DWORD named TcpWindowSize, and enter a sufficiently large size. 131400 (make sure you click on 'decimal') should be enough. Tcp1323Opts should be set to 3. This enables both rfc1323 scaling and timestamps.

Next increase the TCP buffer sizes:

ForwardBufferMemory 80000
NumForwardPackets 60000

A number of users have also had problems with slow netorks with Windows XP SP2 and Internet Connection Sharing. Microsoft says disabling Internet Connection Sharing after the SP2 installation should fix it - which sounds like an excellent plan..