slow ssh login on a cygwin machine

[Ilya Dogolazky]

Hi !

I'm running sshd on a windows machine. Since last weekend (I have not
installed anything) the logging in via ssh started to take 5 to 20
seconds (it was less than 1 second before that). It doesn't matter from
which machine I'm logging in (tried localhost, some another cygwin
machine, 3 different Linux machines: both staying in the same room or
1000 km away).

On the server side I have following options in /etc/sshd_config:
$ cat /etc/sshd_config | grep -v '^#' | grep -v "^$"
Port 22
StrictModes no
AuthorizedKeysFile .ssh/authorized_keys
GSSAPIAuthentication no
GSSAPICleanupCredentials no
UsePrivilegeSeparation yes
UseDNS no
Subsystem sftp /usr/sbin/sftp-server

On the client side I tried following options (all together and one by
one, no difference):
ssh \
-o AddressFamily=inet \
-o CheckHostIP=no \
-o PreferredAuthentications=publickey \
-o VerifyHostKeyDNS=no localhost \
echo blah

And another pretty strange thing: the /var/log directory contains a file
"lastlog" of size 445 Megabytes (445916640). I removed this file and
restarted sshd --- nothing changed about login time. I found a file with
the same name (with absolutely the same size, but a different md5sum) on
another machine running sshd in cygwin having the same login time
issues. One of these machines is freshly installed.

Attached are:

1. cygcheck.out: output of "cygcheck -srv"
2. blah.log: file saved by
"ssh -vv localhost echo blah 2>&1 |tee blah.log"
In two places I marked by XXXXX where it's waiting for something.
The CPU usage during this test remains about zero.

Any help is appreciated.

Cheers,

Ilya Dogolazky

[Corinna Vinschen]

Post by Ilya Dogolazky
Hi !
I'm running sshd on a windows machine. Since last weekend (I have
not installed anything) the logging in via ssh started to take 5 to
20 seconds (it was less than 1 second before that). It doesn't
matter from which machine I'm logging in (tried localhost, some
another cygwin machine, 3 different Linux machines: both staying in
the same room or 1000 km away).

This kind of delay is often a result of the process trying to
access some remote filesystem. Or, maybe you have DNS problems
on the server.

Post by Ilya Dogolazky
And another pretty strange thing: the /var/log directory contains a
file "lastlog" of size 445 Megabytes (445916640).

That size is not overly surprising. The size of the lastlog file
depends on the highest uid used to login into the system. In your case
you seem have pretty large uids. Every uid slot in lastlog takes
276 bytes. So you had login attempts from a user with a uid 1615639.


Corinna

[Ilya Dogolazky]

Hi Corinna !

Post by Corinna Vinschen
This kind of delay is often a result of the process trying to
access some remote filesystem.

How can I investigate this (is there something like "lsof" in Windows)?

If you're speaking about "process", do you mean the "sshd" process? And
what could be a reason for it to access any remote file system? My home
directory is on the hard drive, I surely use some remote file systems
sometimes (by opening file explorer and copying files), but I don't see
any reason for sshd to do the same.

Is there any explicit way to "unmount everything remote" in windows? I
close all the explorer windows, but of course it could be not enough.

Post by Corinna Vinschen
Or, maybe you have DNS problems
on the server.

What kind of "DNS problems" could it be? I disabled reverse DNS query
(if I correctly understand meaning of the option "UseDNS") and see "last
login from xxx.xxx.xxx.xxx" message now (where xxx are digits).

Post by Corinna Vinschen
That size is not overly surprising. The size of the lastlog file
depends on the highest uid used to login into the system. In your case
you seem have pretty large uids. Every uid slot in lastlog takes
276 bytes. So you had login attempts from a user with a uid 1615639.

Oh, thanks! I was getting over-suspicious :)

Cheers,

Ilya

[Corinna Vinschen]

Post by Ilya Dogolazky
Hi Corinna !

Post by Corinna Vinschen
This kind of delay is often a result of the process trying to
access some remote filesystem.

How can I investigate this (is there something like "lsof" in Windows)?

There's a fuser in the psmisc package, but the latest version doesn't
work as expected, downgrading to the previous psmisc package helps
for now. But I don't think this will help you here.

Post by Ilya Dogolazky
If you're speaking about "process", do you mean the "sshd" process?

Any process on the server started during login.

Post by Ilya Dogolazky
And what could be a reason for it to access any remote file system?

Most likely it's something in your login scripts. For instance,
consider expressions like this:

my_path=/foo
ls /$my_path

This results in a `ls //foo' call. // is the top-level of all UNC
network paths. THis call will result in Cygwin searching for a machine
called foo.

Post by Ilya Dogolazky

Post by Corinna Vinschen
Or, maybe you have DNS problems
on the server.

What kind of "DNS problems" could it be?

I'm not an expert, but any kind of DNS communication problem can result
in delays trying to resolve a machine name. I don't know, it's just
some arbitrary idea for the potential cause of a spurious delay.


Corinna

[Corinna Vinschen]

Post by Corinna Vinschen

Post by Ilya Dogolazky
Hi Corinna !

Post by Corinna Vinschen
This kind of delay is often a result of the process trying to
access some remote filesystem.

How can I investigate this (is there something like "lsof" in Windows)?

There's a fuser in the psmisc package, but the latest version doesn't
work as expected, downgrading to the previous psmisc package helps
for now. But I don't think this will help you here.

Post by Ilya Dogolazky
If you're speaking about "process", do you mean the "sshd" process?

Any process on the server started during login.

Post by Ilya Dogolazky
And what could be a reason for it to access any remote file system?

Most likely it's something in your login scripts. For instance,
my_path=/foo
ls /$my_path
This results in a `ls //foo' call. // is the top-level of all UNC
network paths. THis call will result in Cygwin searching for a machine
called foo.

Oh, and then again... did you install the bash-completion package on
the server? It's known to result in such delays sometimes. I never
used it myself so Idon't know what it's doing. Somebody else might
know more here.


Corinna

[Ilya Dogolazky]

Hi !

Post by Corinna Vinschen
Oh, and then again... did you install the bash-completion package on
the server? It's known to result in such delays sometimes. I never
used it myself so I don't know what it's doing. Somebody else might
know more here.

That would be a surprise, because the main delay (marked by XXX in the
attachment to my first e-mail) is happening after "public key offer".
Anyway, I removed this bash-completion package and nothing changed.

By the way: what is the proper way to remove one single package? I
marked it as "Uninstall" and the setup.exe started to reinstall all
other packages. Is it really intended behaviour?

Cheers,

Ilya

[Corinna Vinschen]

Post by Ilya Dogolazky
Hi !

Post by Corinna Vinschen
Oh, and then again... did you install the bash-completion package on
the server? It's known to result in such delays sometimes. I never
used it myself so I don't know what it's doing. Somebody else might
know more here.

That would be a surprise, because the main delay (marked by XXX in
the attachment to my first e-mail) is happening after "public key
offer". Anyway, I removed this bash-completion package and nothing
changed.
By the way: what is the proper way to remove one single package? I
marked it as "Uninstall" and the setup.exe started to reinstall all
other packages. Is it really intended behaviour?

Yes. setup's default setting is "update all packages". There's a
command line option which allows to change single packages, though.


Corinna

[bbcolo22]

I am having exactly the same problem that Ilya described. The problem
appeared out of nowhere 1 week ago on many (but not all) of our production
servers running COPSSH. No errors, but an SSH login that used to take 3-5
seconds now takes > 90 seconds. I can clear see a delay of exactly 45
seconds here:

...
debug1: Trying private key: /home/dbmsmonitor/.ssh/id_rsa
debug3: no such identity: /home/dbmsmonitor/.ssh/id_rsa
debug1: Offering DSA public key: /home/dbmsmonitor/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
(45 second delay).

... followed by a shorter, second delay at the same place the Ilya
described.

Not all of our production servers (running COPSSH on Windows) have the
problem, but even rebooting a server can make the problem appear. I've
tried:

- Different SSH clients (ssh, putty, etc.).
- Changing SSHD_CONFIG options. including turning off DNS (I can see that it
gets past the DNS lookup OK).
- Specifying many of the SSH -o client options.
- Reinstalling COPSSH.

Any ideas? I'm desperate. :@(

Thanks!




--
View this message in context: http://cygwin.1069669.n5.nabble.com/slow-ssh-login-on-a-cygwin-machine-tp5558p92399.html
Sent from the Cygwin list mailing list archive at Nabble.com.

[Larry Hall (Cygwin)]

Post by bbcolo22
I am having exactly the same problem that Ilya described. The problem
appeared out of nowhere 1 week ago on many (but not all) of our production
servers running COPSSH. No errors, but an SSH login that used to take 3-5
seconds now takes > 90 seconds. I can clear see a delay of exactly 45
...
debug1: Trying private key: /home/dbmsmonitor/.ssh/id_rsa
debug3: no such identity: /home/dbmsmonitor/.ssh/id_rsa
debug1: Offering DSA public key: /home/dbmsmonitor/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
(45 second delay).
... followed by a shorter, second delay at the same place the Ilya
described.
Not all of our production servers (running COPSSH on Windows) have the
problem, but even rebooting a server can make the problem appear. I've
- Different SSH clients (ssh, putty, etc.).
- Changing SSHD_CONFIG options. including turning off DNS (I can see that it
gets past the DNS lookup OK).
- Specifying many of the SSH -o client options.
- Reinstalling COPSSH.

cygwin.com/acronyms/#BLODA?

You'll really want to talk to the COPSSH folks about this. Despite my
off-hand suggestion of BLODA above, we don't support COPSSH here. If
the comment helps, great. Otherwise, see COPSSH authors for support.

[bbcolo22]

I tried. They don't know. But thanks anyway.

On Wed, Aug 29, 2012 at 1:52 PM, Larry Hall (Cygwin) [via Cygwin] <

Post by bbcolo22

Post by bbcolo22
I am having exactly the same problem that Ilya described. The problem
appeared out of nowhere 1 week ago on many (but not all) of our

production

Post by bbcolo22
servers running COPSSH. No errors, but an SSH login that used to take

3-5

Post by bbcolo22
seconds now takes > 90 seconds. I can clear see a delay of exactly 45
...
debug1: Trying private key: /home/dbmsmonitor/.ssh/id_rsa
debug3: no such identity: /home/dbmsmonitor/.ssh/id_rsa
debug1: Offering DSA public key: /home/dbmsmonitor/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
(45 second delay).
... followed by a shorter, second delay at the same place the Ilya
described.
Not all of our production servers (running COPSSH on Windows) have the
problem, but even rebooting a server can make the problem appear. I've
- Different SSH clients (ssh, putty, etc.).
- Changing SSHD_CONFIG options. including turning off DNS (I can see

that it

Post by bbcolo22
gets past the DNS lookup OK).
- Specifying many of the SSH -o client options.
- Reinstalling COPSSH.

cygwin.com/acronyms/#BLODA?
You'll really want to talk to the COPSSH folks about this. Despite my
off-hand suggestion of BLODA above, we don't support COPSSH here. If
the comment helps, great. Otherwise, see COPSSH authors for support.
--
Larry
_____________________________________________________________________
A: Yes.

Post by bbcolo22
Q: Are you sure?

A: Because it reverses the logical flow of conversation.

Q: Why is top posting annoying in email?

--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
------------------------------
If you reply to this email, your message will be added to the discussion
http://cygwin.1069669.n5.nabble.com/slow-ssh-login-on-a-cygwin-machine-tp5558p92401.html
To unsubscribe from slow ssh login on a cygwin machine, click here<http://cygwin.1069669.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5558&code=YmJjb2xvMjJAZ21haWwuY29tfDU1NTh8MTk1MTU5NTg2OA==>
.
NAML<http://cygwin.1069669.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
View this message in context: http://cygwin.1069669.n5.nabble.com/slow-ssh-login-on-a-cygwin-machine-tp5558p92402.html
Sent from the Cygwin list mailing list archive at Nabble.com.

[Christopher Faylor]

Post by bbcolo22
I tried. They don't know. But thanks anyway.

If they can't debug problems with the software that they provide then maybe
they shouldn't be providing it.

cgf

[bbcolo22]

No comment. ;)


On Wed, Aug 29, 2012 at 3:04 PM, Christopher Faylor-8 [via Cygwin] <

Post by Christopher Faylor

Post by bbcolo22
I tried. They don't know. But thanks anyway.

If they can't debug problems with the software that they provide then maybe
they shouldn't be providing it.
cgf
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
------------------------------
If you reply to this email, your message will be added to the discussion
http://cygwin.1069669.n5.nabble.com/slow-ssh-login-on-a-cygwin-machine-tp5558p92405.html
To unsubscribe from slow ssh login on a cygwin machine, click here<http://cygwin.1069669.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5558&code=YmJjb2xvMjJAZ21haWwuY29tfDU1NTh8MTk1MTU5NTg2OA==>
.
NAML<http://cygwin.1069669.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
View this message in context: http://cygwin.1069669.n5.nabble.com/slow-ssh-login-on-a-cygwin-machine-tp5558p92406.html
Sent from the Cygwin list mailing list archive at Nabble.com.