Issues with TCP Window Scaling Factor and Cygwin Daemons

[David Groves]

I am having some issues with cygwin applications (specifically sshd)
and TCP Window Scaling Factors. I am using OpenSSH client on either a
Debian Linux or FreeBSD machine to connect to sshd on a Windows 2k8 R2
server.

My SYN from the client has a non-zero WSF (16 for Debian, 8 for
FreeBSD), but the SYNACK response from the windows box has 0 as the
WSF. This means I will have a maximum of 64Kbytes of traffic allowed
to be unacknowledged in the TCP window. This is causing me
considerable performance problems in transfers from Europe to the US
West Coast and Australia.

To try and decide if cygwin was a factor in this, I tried to replicate
this with Apache2 and pure-ftpd on cygwin, and in both cases, the WSF
was 0. I then tried a non-cygwin Apache2 install, and it returned a
non-zero WSF.

TCP settings is clearly a function of the OS, but I'm a little unsure
where the line is drawn between the OS with cygwin, and it is
interesting I can reproduce this with multiple cygwin apps, and I
cannot reproduce it with non-cygwin apps, I'm thinking cygwin is at
least somewhat involved as part of the problem.

Anyone have any ideas/pointers ?

[Jon Clugston]

Post by David Groves
I am having some issues with cygwin applications (specifically sshd)
and TCP Window Scaling Factors. I am using OpenSSH client on either a
Debian Linux or FreeBSD machine to connect to sshd on a Windows 2k8 R2
server.
My SYN from the client has a non-zero WSF (16 for Debian, 8 for
FreeBSD), but the SYNACK response from the windows box has 0 as the
WSF. This means I will have a maximum of 64Kbytes of traffic allowed
to be unacknowledged in the TCP window. This is causing me
considerable performance problems in transfers from Europe to the US
West Coast and Australia.
To try and decide if cygwin was a factor in this, I tried to replicate
this with Apache2 and pure-ftpd on cygwin, and in both cases, the WSF
was 0. I then tried a non-cygwin Apache2 install, and it returned a
non-zero WSF.
TCP settings is clearly a function of the OS, but I'm a little unsure
where the line is drawn between the OS with cygwin, and it is
interesting I can reproduce this with multiple cygwin apps, and I
cannot reproduce it with non-cygwin apps, I'm thinking cygwin is at
least somewhat involved as part of the problem.
Anyone have any ideas/pointers ?
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Did you do all these tests w/ the same Windows machine? I believe
that Windows XP always sets the WSF to zero, but newer versions
support scaling.

[René Berber]

Post by Jon Clugston
Did you do all these tests w/ the same Windows machine? I believe
that Windows XP always sets the WSF to zero, but newer versions
support scaling.

Not sure if scaling works on XP, but the TcpWindowSize is not zero (by
default).

For XP the reference is this:

http://support.microsoft.com/kb/314053

Which doesn't mention the key to control scaling: Tcp1323Opts; so I
guess it doesn't work.

For TcpWindowSize the recommendation to get better performance is to set
it yourself much bigger than the default (which usually is 8760).

Still this doesn't explain why David is seeing a value of zero. Perhaps
the specific computer has it configured that way.

[Lee]

Post by David Groves
I am having some issues with cygwin applications (specifically sshd)
and TCP Window Scaling Factors. I am using OpenSSH client on either a
Debian Linux or FreeBSD machine to connect to sshd on a Windows 2k8 R2
server.
My SYN from the client has a non-zero WSF (16 for Debian, 8 for
FreeBSD), but the SYNACK response from the windows box has 0 as the
WSF. This means I will have a maximum of 64Kbytes of traffic allowed
to be unacknowledged in the TCP window. This is causing me
considerable performance problems in transfers from Europe to the US
West Coast and Australia.
To try and decide if cygwin was a factor in this, I tried to replicate
this with Apache2 and pure-ftpd on cygwin, and in both cases, the WSF
was 0. I then tried a non-cygwin Apache2 install, and it returned a
non-zero WSF.
TCP settings is clearly a function of the OS, but I'm a little unsure
where the line is drawn between the OS with cygwin, and it is
interesting I can reproduce this with multiple cygwin apps, and I
cannot reproduce it with non-cygwin apps, I'm thinking cygwin is at
least somewhat involved as part of the problem.
Anyone have any ideas/pointers ?

No pointers, but it does look like a Cygwin problem.

Start up wireshark on a Windows XP machine and ssh (from a cygwin
window) to another machine - tcp syn packet has a window size of 65535
and a widow scale of zero. Use putty from the dos prompt to ssh to
the same machine and that has a window size of 65535 and a window
scale of 2

Lee

[Dave Korn]

Post by Lee

Post by David Groves
I am having some issues with cygwin applications (specifically sshd)
and TCP Window Scaling Factors. I am using OpenSSH client on either a
Debian Linux or FreeBSD machine to connect to sshd on a Windows 2k8 R2

^^^^^^^^^^^^^^

Post by Lee

Post by David Groves
server.

No pointers, but it does look like a Cygwin problem.
Start up wireshark on a Windows XP machine and ssh (from a cygwin

^^^^^^^^^^^^^^^^^^

Post by Lee
window) to another machine - tcp syn packet has a window size of 65535
and a widow scale of zero. Use putty from the dos prompt to ssh to
the same machine and that has a window size of 65535 and a window
scale of 2

1. MS completely replaced the networking stack in Vista/2k8 with their
so-called "Next Generation TCP/IP stack", so you can't extrapolate from XP to 2k8.

2. On XP, window scaling is enabled by a registry setting, and when it is
in effect, Windows decides what scale setting to use by considering the
SO_RCVBUF size used when the connection is initiated. In that light, the fact
that Cygwin applications show a scale of zero is probably not unrelated to

Post by Lee
/* Raise default buffer sizes (instead of WinSock default 8K).
64K appear to have the best size/performance ratio for a default
value. Tested with ssh/scp on Vista over Gigabit LAN.
NOTE. If the SO_RCVBUF size exceeds 65535(*), and if the socket is
connected to a remote machine, then calling WSADuplicateSocket on
fork/exec fails with WinSock error 10022, WSAEINVAL. Fortunately
we don't use WSADuplicateSocket anymore, rather we just utilize
handle inheritance. An explanation for this weird behaviour would
be nice, though.
(*) Maximum normal TCP window size. Coincidence? */
((fhandler_socket *) fd)->rmem () = 65535;
((fhandler_socket *) fd)->wmem () = 65535;
if (::setsockopt (soc, SOL_SOCKET, SO_RCVBUF,
(char *) &((fhandler_socket *) fd)->rmem (), sizeof (int)))
{

[ ... snip error handling ... ]

Post by Lee
}

I would guess that a >65536 default would cause Windows to send a non-zero
scale in the SYN packet, but as the comment says, that created some other sort
of problem somehow.

3. On 2k8 (also Vista and 7), window scaling is no longer controlled by the
registry but enabled automatically, as part of a new feature called "Receive
Window Auto-Tuning". I could not infer from any documentation that I could
find in ten minutes googling whether or not it still pays attention to the
initial SO_RCVBUF setting. It might be worth trying a custom build of the
Cygwin DLL with a 64k or 128k setting in place of 65535. Then again, it seems
there may yet be problems in this functionality, e.g. the known bug described
at http://support.microsoft.com/kb/983528; perhaps if the custom Cygwin DLL
doesn't help, that hot-fix might.

cheers,
DaveK

[Lee]

Post by Dave Korn

Post by Lee

Post by David Groves
I am having some issues with cygwin applications (specifically sshd)
and TCP Window Scaling Factors. I am using OpenSSH client on either a
Debian Linux or FreeBSD machine to connect to sshd on a Windows 2k8 R2

^^^^^^^^^^^^^^

Post by Lee

Post by David Groves
server.

No pointers, but it does look like a Cygwin problem.
Start up wireshark on a Windows XP machine and ssh (from a cygwin

^^^^^^^^^^^^^^^^^^

Post by Lee
window) to another machine - tcp syn packet has a window size of 65535
and a widow scale of zero. Use putty from the dos prompt to ssh to
the same machine and that has a window size of 65535 and a window
scale of 2

1. MS completely replaced the networking stack in Vista/2k8 with their
so-called "Next Generation TCP/IP stack", so you can't extrapolate from XP to 2k8.

Arguh.. I don't know where I got WinXP from. Sorry for the noise.
Lee

Post by Dave Korn
2. On XP, window scaling is enabled by a registry setting, and when it is
in effect, Windows decides what scale setting to use by considering the
SO_RCVBUF size used when the connection is initiated. In that light, the fact
that Cygwin applications show a scale of zero is probably not unrelated to

Post by Lee
/* Raise default buffer sizes (instead of WinSock default 8K).
64K appear to have the best size/performance ratio for a default
value. Tested with ssh/scp on Vista over Gigabit LAN.
NOTE. If the SO_RCVBUF size exceeds 65535(*), and if the socket is
connected to a remote machine, then calling WSADuplicateSocket on
fork/exec fails with WinSock error 10022, WSAEINVAL. Fortunately
we don't use WSADuplicateSocket anymore, rather we just utilize
handle inheritance. An explanation for this weird behaviour would
be nice, though.
(*) Maximum normal TCP window size. Coincidence? */
((fhandler_socket *) fd)->rmem () = 65535;
((fhandler_socket *) fd)->wmem () = 65535;
if (::setsockopt (soc, SOL_SOCKET, SO_RCVBUF,
(char *) &((fhandler_socket *) fd)->rmem (), sizeof (int)))
{

[ ... snip error handling ... ]

Post by Lee
}

I would guess that a >65536 default would cause Windows to send a non-zero
scale in the SYN packet, but as the comment says, that created some other sort
of problem somehow.
3. On 2k8 (also Vista and 7), window scaling is no longer controlled by the
registry but enabled automatically, as part of a new feature called "Receive
Window Auto-Tuning". I could not infer from any documentation that I could
find in ten minutes googling whether or not it still pays attention to the
initial SO_RCVBUF setting. It might be worth trying a custom build of the
Cygwin DLL with a 64k or 128k setting in place of 65535. Then again, it seems
there may yet be problems in this functionality, e.g. the known bug described
at http://support.microsoft.com/kb/983528; perhaps if the custom Cygwin DLL
doesn't help, that hot-fix might.
cheers,
DaveK
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple

[David Groves]

It looks like the hotfix (http://support.microsoft.com/kb/983528)
DaveK suggested is working.

It has fully resolved the issue with my Windows 7 machine + our lab
Win2K8 box (although we couldn't see the symptoms of the small window
size as the latency is so low, we could still see 0 WSF in tcpdump
output). We are waiting for a planned outage window to try it on one
of our production systems, but we are quite hopeful this should fix
the issue.

[Corinna Vinschen]

Post by Dave Korn

Post by Lee

Post by David Groves
I am having some issues with cygwin applications (specifically sshd)
and TCP Window Scaling Factors. I am using OpenSSH client on either a
Debian Linux or FreeBSD machine to connect to sshd on a Windows 2k8 R2

^^^^^^^^^^^^^^

Post by Lee

Post by David Groves
server.

No pointers, but it does look like a Cygwin problem.
Start up wireshark on a Windows XP machine and ssh (from a cygwin

^^^^^^^^^^^^^^^^^^

Post by Lee
window) to another machine - tcp syn packet has a window size of 65535
and a widow scale of zero. Use putty from the dos prompt to ssh to
the same machine and that has a window size of 65535 and a window
scale of 2

1. MS completely replaced the networking stack in Vista/2k8 with their
so-called "Next Generation TCP/IP stack", so you can't extrapolate from XP to 2k8.
2. On XP, window scaling is enabled by a registry setting, and when it is
in effect, Windows decides what scale setting to use by considering the
SO_RCVBUF size used when the connection is initiated. In that light, the fact
that Cygwin applications show a scale of zero is probably not unrelated to

Post by Lee
/* Raise default buffer sizes (instead of WinSock default 8K).
64K appear to have the best size/performance ratio for a default
value. Tested with ssh/scp on Vista over Gigabit LAN.
NOTE. If the SO_RCVBUF size exceeds 65535(*), and if the socket is
connected to a remote machine, then calling WSADuplicateSocket on
fork/exec fails with WinSock error 10022, WSAEINVAL. Fortunately
we don't use WSADuplicateSocket anymore, rather we just utilize
handle inheritance. An explanation for this weird behaviour would
be nice, though.
(*) Maximum normal TCP window size. Coincidence? */
((fhandler_socket *) fd)->rmem () = 65535;
((fhandler_socket *) fd)->wmem () = 65535;
if (::setsockopt (soc, SOL_SOCKET, SO_RCVBUF,
(char *) &((fhandler_socket *) fd)->rmem (), sizeof (int)))
{

[ ... snip error handling ... ]

Post by Lee
}

I would guess that a >65536 default would cause Windows to send a non-zero
scale in the SYN packet, but as the comment says, that created some other sort
of problem somehow.
3. On 2k8 (also Vista and 7), window scaling is no longer controlled by the
registry but enabled automatically, as part of a new feature called "Receive
Window Auto-Tuning". I could not infer from any documentation that I could
find in ten minutes googling whether or not it still pays attention to the
initial SO_RCVBUF setting. It might be worth trying a custom build of the
Cygwin DLL with a 64k or 128k setting in place of 65535. Then again, it seems
there may yet be problems in this functionality, e.g. the known bug described
at http://support.microsoft.com/kb/983528; perhaps if the custom Cygwin DLL
doesn't help, that hot-fix might.

Alternatively we could only set the send/recv buffer sizes if we're
running under pre-Vista. However, I have no idea what implications
this has in terms of network performance. "Somebody" would have to
test it.


Corinna