Unicode/UTF-8 support (MinTTY) - ncursesw - Mutt, Vim

[Michael Ludwig]

After having used rxvt without Unicode support for years, the other
day I discovered MinTTY, which does support UTF-8 - very nice!

The occasional Greek or Cyrillic letters showing up in mails will no
longer be displayed as "?" or "??" in the Mutt mail reader, I thought.
Same story for editing with Vim.

Not quite, though. There are display and editing problems in both
programs. From reading, I believe this is due to their being linked to
ncurses instead of ncursesw ("w" for "wide characters"), as shown by
ldd:

$ ldd $(which vim) | grep curses
cygncurses-10.dll => /usr/bin/cygncurses-10.dll (0x69580000)
$ ldd $(which mutt) | grep curses
cygncurses-8.dll => /usr/bin/cygncurses-8.dll (0x6c180000)

Is this assessment correct, and complete in that this is the defining
reason for the display problems?

The wide-character ncursesw was announced in January:

This is the first official release of ncurses compiled to support wide
characters, and can be installed simultaineously with the "narrow"
ncurses package(s).

http://www.mail-archive.com/cygwin-***@cygwin.com/msg03179.html

It sounds like Unicode is the preferred way now:

Actually, I'd prefer if people started using -I/usr/include/ncursesw
and linking against the wide version of the library instead.

http://sourceware.org/ml/cygwin/2010-05/msg00465.html

People seem to have had success compiling Mutt with ncursesw:

http://code.google.com/p/mintty/issues/detail?id=124

Are ncursesw versions of Vim and Mutt imminent? Or is it not going to
happen anytime soon?

[Corinna Vinschen]

Post by Michael Ludwig
This is the first official release of ncurses compiled to support wide
characters, and can be installed simultaineously with the "narrow"
ncurses package(s).
Actually, I'd prefer if people started using -I/usr/include/ncursesw
and linking against the wide version of the library instead.
http://sourceware.org/ml/cygwin/2010-05/msg00465.html
http://code.google.com/p/mintty/issues/detail?id=124
Are ncursesw versions of Vim and Mutt imminent? Or is it not going to
happen anytime soon?

Thanks for the heads up. I'll build the next release of vim (there's
a 7.3 release coming soon) against ncursesw.


Corinna

[Andy Koppe]

  This is the first official release of ncurses compiled to support wide
  characters, and can be installed simultaineously with the "narrow"
  ncurses package(s).
  Actually, I'd prefer if people started using -I/usr/include/ncursesw
  and linking against the wide version of the library instead.
http://sourceware.org/ml/cygwin/2010-05/msg00465.html
http://code.google.com/p/mintty/issues/detail?id=124
Are ncursesw versions of Vim and Mutt imminent? Or is it not going to
happen anytime soon?

Thanks for the heads up.  I'll build the next release of vim (there's
a 7.3 release coming soon) against ncursesw.

Probably a good idea anyway, but as far as I can see, vim works fine
with UTF-8 already. It even handles non-BMP codepoints correctly,
which isn't a given due to the UTF-16 wchar_t on Windows and Cygwin.
(Emacs and mined get that right too, whereas for example bash and zsh
don't.)

Michael, what are the issues you're seeing with vim? Are you using Cygwin 1.7?

Mutt and also nano do need rebuilding with ncursesw though.

Andy

[Michael Ludwig]

Post by Andy Koppe

�I'll build the next release of vim (there's a 7.3 release coming
soon) against ncursesw.

Probably a good idea anyway, but as far as I can see, vim works fine
with UTF-8 already.
Michael, what are the issues you're seeing with vim? Are you using Cygwin 1.7?

As you say, Vim works fine with UTF-8. It's just that until very
recently, I've been using the rxvt terminal emulator, which lacks
Unicode support; so Vim being compiled against ncurses (*without*
wide characters) was a good match.

Now that I'm switching to the MinTTY terminal, which supports
Unicode/UTF-8, I need a Vim compiled against ncursesw (*with*
wide characters) if my assessment of the situation is correct.

[Andy Koppe]

Post by Michael Ludwig

Post by Andy Koppe

I'll build the next release of vim (there's a 7.3 release coming
soon) against ncursesw.

Probably a good idea anyway, but as far as I can see, vim works fine
with UTF-8 already.
Michael, what are the issues you're seeing with vim? Are you using Cygwin 1.7?

As you say, Vim works fine with UTF-8. It's just that until very
recently, I've been using the rxvt terminal emulator, which lacks
Unicode support; so Vim being compiled against ncurses (*without*
wide characters) was a good match.
Now that I'm switching to the MinTTY terminal, which supports
Unicode/UTF-8, I need a Vim compiled against ncursesw (*with*
wide characters) if my assessment of the situation is correct.

If vim works fine with UTF-8 already, what does it matter which
ncurses it's linked against? I don't know vim's internals, but I'd
guess it works without ncursesw because it does its own screen
buffering and displaying, using ncurses only for accessing the
terminfo database.

Andy

[Michael Ludwig]

Post by Andy Koppe

Post by Michael Ludwig
As you say, Vim works fine with UTF-8. It's just that until very
recently, I've been using the rxvt terminal emulator, which lacks
Unicode support; so Vim being compiled against ncurses (*without*
wide characters) was a good match.
Now that I'm switching to the MinTTY terminal, which supports
Unicode/UTF-8, I need a Vim compiled against ncursesw (*with*
wide characters) if my assessment of the situation is correct.

My assessment was incorrect.

Post by Andy Koppe
If vim works fine with UTF-8 already, what does it matter which
ncurses it's linked against? I don't know vim's internals, but I'd
guess it works without ncursesw because it does its own screen
buffering and displaying, using ncurses only for accessing the
terminfo database.

I don't have much of an idea of what ncurses is actually used for,
nor how Unicode support works in Vim. I simply "assessed" the two
were somehow connected. :-)

The truth is rather trivial and refreshing. My ~/.vimrc contained
some hard-coded configuration for Latin1 on Cygwin (win32unix):

" MiLu: Cygwin/rxvt kann nur Latin1.
if has('win32unix')
set termencoding=latin1
endif

That setting applied on a UTF-8 terminal garbled input, output and
display for non-ASCII characters. After eliminating that it all works
fine. Thanks for not letting my "assessment" pass!

[Michael Ludwig]

Post by Corinna Vinschen

Post by Michael Ludwig
Are ncursesw versions of Vim and Mutt imminent? Or is it not going
to happen anytime soon?

Thanks for the heads up. I'll build the next release of vim (there's
a 7.3 release coming soon) against ncursesw.

That's great! Thank you!

As for Mutt, I downloaded the sources via Cygwin setup.exe and
recompiled against ncursesw. As I couldn't find a suitable configure
option, I had to edit the Makefile as follows to (1) enable ncursesw
and (2) skip building the documentation (which would fail):

\,,,/
(o o)
------oOOo-(_)-oOOo------
diff Makefile Makefile.orig
164c164
< CPPFLAGS = -I/usr/include/ncursesw -I$(top_srcdir)/intl
-I$(includedir)
---

Post by Corinna Vinschen
CPPFLAGS = -I/usr/include/ncurses -I$(top_srcdir)/intl
-I$(includedir)

208c208
< MUTTLIBS = -lncursesw
---

Post by Corinna Vinschen
MUTTLIBS = -lncurses

283c283
< SUBDIRS = m4 po intl contrib $(IMAP_SUBDIR)
---

Post by Corinna Vinschen
SUBDIRS = m4 po intl doc contrib $(IMAP_SUBDIR)

-------------------------

Text in Russian or Greek looks perfect now. Same for Mutt's
internationalized messages in German, Russian, Greek.

Отвечать по ***@cygwin.com?
([да]/нет):

(Japanese, Korean and Chinese are another story, possibly on account
of missing fonts.)

Great job, ncursesw! I think just like Vim users, Mutt users would
also appreciate Unicode support.

[Michael Ludwig]

Post by Michael Ludwig
As for Mutt, I downloaded the sources via Cygwin setup.exe and
recompiled against ncursesw. As I couldn't find a suitable configure
option, I had to edit the Makefile as follows to (1) enable ncursesw

To get the same behaviour as with the Cygwin binary Mutt, some more
tweaking of configure flags or other stuff seems required. Two
differences I've noted so far:

* An error message "repetition-operator operand invalid" appears on
startup for "*" instead of ".*" in folder-hook directives.

* My built forgets about mailboxes it has seen so they're all flagged
as containing new messages.

As I said, it would be fine to have a binary Mutt compiled and linked
against ncursesw with wide character support.

[Michael Ludwig]

Post by Michael Ludwig
To get the same behaviour as with the Cygwin binary Mutt, some more
tweaking of configure flags or other stuff seems required. Two
* An error message "repetition-operator operand invalid" appears on
startup for "*" instead of ".*" in folder-hook directives.

Avoided by specifying --with-regex for the configure script.

Post by Michael Ludwig
* My built forgets about mailboxes it has seen so they're all flagged
as containing new messages.

Solved by specifying --enable-buffy-size for the configure script.

The Cygwin build has more stuff enabled (which I don't use). My
configure line was:

./configure --prefix=/usr/local/muttw2 --with-regex --enable-buffy-size

The feature and package sets of different builds can be compared by
diffing the output of "mutt -v" and reading the output of "configure
--help".