2013 in review

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 10,000 times in 2013. If it were a concert at Sydney Opera House, it would take about 4 sold-out performances for that many people to see it.

Click here to see the complete report.

2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

600 people reached the top of Mt. Everest in 2012. This blog got about 10,000 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 17 years to get that many views.

Click here to see the complete report.

C++ features in Visual C++11, yeah!

Recently Microsoft surprised me – positively! – with supporting newfangled C++11 features in the November CTP of their Visual C++ 11.0 compiler:

  • Variadic templates
  • Uniform initialization and initializer_lists
  • Delegating constructors
  • Raw string literals
  • Explicit conversion operators
  • Default template arguments for function templates

The library has not yet been updated to take advantage, but still this means that it’s now possible to avoid macro hell for things such as makeUnique (but unforutunately not yet for e.g. platform-specific Unicode strings, no support yet for the newfangled literals).

On the other hand, Visual C++ implements the regexp library, while g++ does not. And Visual C++ has working exceptions, while AFAIK the dang[1] compiler still has not. So this may just look as if Visual C++ is now overtaking g++ and the dang compiler in the race to support the C++11 standard! :-)

[1] As Brian Keminghan remarked, dang if I can remember its name!

Liskov’s substitution principle in C++ <

Part I of III: The LSP and value assignment.

Abstract:
Liskov’s principle of substitution, a principle of generally good class design for polymorphism, strongly implies the slicing behavior of C++ value assignment. Destination value slicing can cause a partial assignment, which can easily break data integrity. Three solutions are (1) static checking of assignment, (2) dynamic checking of assignment, and (3), generally easiest, prohibiting assignment, then possibly providing cloning functionality as an alternative.

Contents:

  • > A short review of the LSP.
  • > How the LSP (almost) forces slicing in C++.
  • > The danger of slicing: type constraint violations.
  • > The possible design alternatives wrt. providing value assignment.
  • > How to provide assignment with STATIC CHECKING of type constraints.
  • > How to provide assignment with DYNAMIC CHECKING of type constraints.
  • > How to provide CLONING as an alternative to assignment.

Continue reading

ResultOf a function with C++11

Using std::result_of requires you to specify the function argument types. Which is not very practical when you don’t know the function signature. Happily @potatoswatter (David Krauss) over at Stack Overflow pointed out to me that std::function provides the desired functionality:

template< class Func >
struct ResultOf
{
    typedef typename std::function<
        typename std::remove_pointer<Func>::type
        >::result_type T;
};

Mainly, this saves one from writing a large number of partial specializations for Visual C++ 10.0, which lacks support for C++11 variadic templates.

– Enjoy!

Unicode part 2: UTF-8 stream mode



The Windows console subsystem has a host of Unicode-related bugs. And standard Windows programs such as more (not to mention the C# 4.0 compiler csc) just crash when they’re run from a console window with UTF-8 as active codepage, perplexingly claiming that they’re out of memory. On top of that the C++ runtime libraries of various compilers differ in how they behave. Doing C++ Unicode i/o in Windows consoles is therefore problematic. In this series I show how to work around limitations of the Visual C++ _O_U8TEXT file mode, with the Visual C++ and g++ compilers. This yields an automatic translation between external UTF-8 and internal UTF-16, enabling Windows console i/o of characters in the Basic Multilingual Plane.

Recap

In part 1 I introduced two approaches to Unicode handling in small Windows console programs:

  • The all UTF-8 approach where everything is encoded as UTF-8, and where there are no BOM encoding markers.
     
  • The wide string approach where all external text (including the C++ source code) is encoded as UTF-8, and all internal text is encoded as UTF-16.

The all UTF-8 approach is the approach used in a typical Linux installation. With this approach a novice can remain unaware that he is writing code that handles Unicode: it Just Works™ – in Linux. However, we saw that it mass-failed in Windows:

  • Input with active codepage 65001 (UTF-8) failed due to various bugs.
     
  • Console output with Visual C++ produced gibberish due to the runtime library’s attempt to help by using direct console output.
     
  • I mentioned how wide string literals with non-ASCII characters are incorrectly translated to UTF-16 by Visual C++ due to the necessary lying to Visual C++ about the source code encoding (which is accomplished by not having a BOM at the start of the source code file).

The wide string approach, on the other hand, was shown to have special support in Visual C++, via the _O_U8TEXT file mode, which I called an UTF-8 stream mode. This mode works down at the C FILE level so that wide character operations on C FILE streams get automatic conversion to/from external UTF-8 encoding. That C FILE level support is needed and is practically impossible to do for the application programmer, so it’s a very good thing to have that UTF-8 stream mode…

I mentioned that as of Visual C++ 10 the UTF-8 mode is not fully implemented, and that it apparently has some bugs. I.e., that it cannot be used directly but needs some scaffolding and fixing. That’s what this second part is about.

Here I do not yet consider the g++ compiler. All this code is Visual C++ specific. However, I have done generally the same with g++, and I will probably discuss that in a third installment.

UTF-8 stream mode: the good (wide stream output)

The good news about the _O_U8TEXT mode is that it works for wide stream output, even for output of narrow strings via the wide stream:

[utf8_mode.msvc.output.good.cpp]
#include <stdexcept>        // std::runtime_error, std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::wcout, std::wcerr, std::endl
#include <string>           // std::string, std::wstring
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

bool throwX( string const& s ) { throw runtime_error( s ); }
bool hopefully( bool v ) { return v; }

void setUtf8Mode( FILE* f, char const name[] )
{
    int const newMode = _setmode( _fileno( f ), _O_U8TEXT );
    hopefully( newMode != -1 )
        || throwX( string() + "setmode failed for " + name );
}

int main()
{
    try
    {
        static char const       narrowText[]    = "Blåbærsyltetøy! 日本国 кошка!";
        static wchar_t const    wideText[]      = L"Blåbærsyltetøy! 日本国 кошка!";

        setUtf8Mode( stdout, "stdout" );

        wcout << "Narrow text: " << narrowText << endl;
        wcout << "Wide text:   " << wideText << endl;
        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        wcerr << "!" << x.what() << endl;
    }
    return EXIT_FAILURE;
}

Visual C++ produces a series of warnings for this source code:

W:\examples> cl utf8_mode.msvc.output.good.cpp /Fe"good"
utf8_mode.msvc.output.good.cpp
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u65E5' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u672C' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u56FD' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u043A' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u043E' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u0448' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u043A' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u0430' cannot be represented in the current code page (1252)

W:\examples> _

It might look as if these warnings are due to the active codepage in the console window, but they’re not related. Visual C++ is just complaining about information loss when it attempts to convert the narrowText literal from the source code’s UTF-8 to its incorrectly documented C++ execution character set, which is Windows ANSI. Where Windows ANSI is the codepage reported by the GetACP Windows API function (doc here), which defaults to the codepage specified in the ACP value of the as far as I know undocumented registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage.

So, the effect of compiling can be a bit different depending on the language that Windows is installed for, which determines the default GetACP codepage. But at least the compilation doesn’t depend on such an ephemeral setting as the active codepage in some console window! And these warnings are good: they’re spot on, for a change. :-)

Now, does the program work?

W:\examples> chcp 1252
Active code page: 1252

W:\examples> good
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> good >good_result

W:\examples> type good_result
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> chcp 65001
Active code page: 65001

W:\examples> good
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> type good_result
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> _

Yes! It manages to present correct output even with active codepage 1252 (Windows ANSI Western) because it uses direct console i/o for this case. And as you can see the redirected output is UTF-8, as it should be.

Even more goodness: the UTF-8 mode also works down at the C library level, using e.g. the wprintf function.

UTF-8 stream mode: the bad (input)

The bad news about the _O_U8TEXT mode is that input is almost as non-functional as with an active codepage 65001.

Apparently the runtime retrieves input as 1 byte per character via the standard input stream.Which means that the input is encoded according to the console window’s active codepage. These bytes, that e.g. encode your input text in Windows ANSI, are then interpreted as if they were UTF-8 encoded text.

The interpretation as UTF-8 would only be correct for active codepage 65001. But with active codepage 65001, which is the only one where the interpretation would be correct, the input operation just fails. So, depending on the active codepage the input operation will either produce garbage for non-ASCII characters, or it will fail outright.

[utf8_mode.msvc.input.bad.cpp]
#include <stdexcept>        // std::runtime_error, std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::wcout, std::wcerr, std::endl
#include <string>           // std::string, std::wstring
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

bool throwX( string const& s ) { throw runtime_error( s ); }
bool hopefully( bool v ) { return v; }

void setUtf8Mode( FILE* f, char const name[] )
{
    int const newMode = _setmode( _fileno( f ), _O_U8TEXT );
    hopefully( newMode != -1 )
        || throwX( string() + "setmode failed for " + name );
}

void initStreams()
{
    setUtf8Mode( stdin, "stdin" );
    setUtf8Mode( stdout, "stdout" );
}

wstring lineFrom( wistream& stream )
{
    wstring     result;
    
    getline( stream, result );
    hopefully( !stream.fail() )
        || throwX( "lineFrom: getline failed" );
    return result;
}

int main()
{
    try
    {
        initStreams();

        wcout << "What's your name? ";
        wstring const name = lineFrom( wcin );
        wcout << "Pleased to meet you, " << name << "!" << endl;
        
        int const n = name.length();
        wcout << endl;
        wcout << "I represented your name as " << n << " wide characters:" << endl;
        for( int i = 0;  i < n;  ++i )
        {
            if( i > 0 ) { wcout << " | "; }
            wcout << hex << 0 + name[i] << " '" << name[i] << "'";
        }
        wcout << endl;
       
        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        wcerr << "!" << x.what() << endl;
    }
    return EXIT_FAILURE;
}

Testing this:

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl utf8_mode.msvc.input.bad.cpp /Fe"bad_input"
utf8_mode.msvc.input.bad.cpp

W:\examples> bad_input
What's your name? Bjørn
Pleased to meet you, Bj�rn!

I represented your name as 5 wide characters:
42 'B' | 6a 'j' | fffd '�' | 72 'r' | 6e 'n'

W:\examples> type good_result
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> bad_input
What's your name? Blåbærsyltetøy
Pleased to meet you, Blåbærsyltetøy!

I represented your name as 14 wide characters:
42 'B' | 6c 'l' | e5 'å' | 62 'b' | e6 'æ' | 72 'r' | 73 's' | 79 'y' | 6c 'l' | 74 't' | 65 'e' | 74 't' | f8 'ø' | 79 'y'

W:\examples> _

The last run with the funny characters pasted as input shows that interactive input of the UTF-8 byte values works. The input byte values, that look pretty funny in the console, are the UTF-8 byte values that encode “Blåbærsyltetøy”. And with that as the cleartext result the received byte values must have been interpreted directly as UTF-8.

Instead the runtime should have used direct console input, just as it uses direct console output when the standard output stream is connected to a console window. Since it does not (or does not manage to do that correctly), that’s what we have to provide.

UTF-8 stream mode: the bad (input) – FIXED

It is apparently easy to check whether the standard input stream is connected to a console window, namely via the_isatty function (doc here). However, the documentation hints ominously about _isatty maybe returning true for a stream connected to a serial port, and archaic things like that. Happily one can alternatively use the lower level Windows API console functions, like e.g. GetConsoleMode (doc here), which presumably will only succeed for a stream that represents something that supports the Windows console functions.

[is_input_console.cpp]
#include <iostream>
#include <stdio.h>          // stdin, _fileno

#include <io.h>             // _isatty

#include <windows.h>        // GetConsoleMode

int main()
{
    DWORD   consoleMode;

    HANDLE const    inputHandle         = GetStdHandle( STD_INPUT_HANDLE );
    bool const      winapiSaysConsole   = !!GetConsoleMode( inputHandle, &consoleMode );
    bool const      clibSaysConsole     = !!_isatty( _fileno( stdin ) );
    
    using namespace std;
    cerr << boolalpha;
    cerr << "_isatty: " << clibSaysConsole << endl;
    cerr << "GetConsoleMode: " << winapiSaysConsole << endl;
}
W:\examples> cl is_input_console.cpp /Fe"x"
is_input_console.cpp

W:\examples> x
_isatty: true
GetConsoleMode: true

W:\examples> x <nul
_isatty: true
GetConsoleMode: false

W:\examples> x <is_input_console.cpp
_isatty: false
GetConsoleMode: false

W:\examples> _

It surprised me that _isatty here incorrectly identified the Windows nul device as a console window. However, GetConsoleMode got it right. So GetConsoleMode is evidently more reliable for this detection task.

Anyway, with a console identified as such, at the C++ level it’s then possible to override things in std::basic_streambuf (doc here), where the required core functionality maps almost directly to a call of the Windows API function ReadConsole (doc here).

The program below illustrates the technique by adding the necessary special input support to the previous section’s program, with main unchanged. This program is however not intended to provide directly reusable code. It’s just a problem-specific concrete example written in a reasuble-like style:

[utf8_mode.msvc.input.fixed.cpp]
#ifdef  _MSC_VER
#   pragma warning( disable: 4373 )     // "Your override overrides"
#endif

#include <algorithm>        // std::remove
#include <stddef.h>         // ptrdiff_t
#include <stdexcept>        // std::runtime_error, std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::wcout, std::wcerr, std::endl
#include <streambuf>        // std::basic_streambuf
#include <string>           // std::string, std::wstring
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

#undef      UNICODE
#define     UNICODE
#include <windows.h>    // ReadConsole


typedef ptrdiff_t       Size;

bool throwX( string const& s ) { throw runtime_error( s ); }
bool hopefully( bool v ) { return v; }

class DirectInputBuffer
    : public std::basic_streambuf< wchar_t >
{
private:
    wstring     buffer_;

    Size bufferSize() const         { return buffer_.size(); }
    wchar_t* pBufferStart()         { return &buffer_[0]; }
    wchar_t* pBufferEnd()           { return pBufferStart() + bufferSize(); }

    wchar_t* pStart() const         { return eback(); }
    wchar_t* pCurrent() const       { return gptr(); }
    wchar_t* pEnd() const           { return egptr(); }

    static HANDLE inputHandle()
    {
        static HANDLE const handle = GetStdHandle( STD_INPUT_HANDLE );
        return handle;
    }

public:
    typedef std::basic_streambuf< wchar_t >     Base;
    typedef Base::traits_type                   Traits;

    DirectInputBuffer( Base const& anOriginalBuffer )
        : Base( anOriginalBuffer )      // Copies buffer read area pointers.
        , buffer_( 256, L'#' )
    {}

protected:
    virtual streamsize xsgetn( wchar_t* const pBuffer, streamsize const n )
    {
        wchar_t const   ctrlZ   = wchar_t( 1 + ('Z' - 'A') );

        DWORD       nCharactersRead     = 0;

        bool const  readSucceeded       = !!ReadConsole(
            inputHandle(), pBuffer, static_cast< DWORD >( n ), &nCharactersRead, nullptr
            );

        if( readSucceeded )
        {
            wchar_t const* const    pCleanEnd   =
                remove( pBuffer, pBuffer + nCharactersRead, L'\r' );

            nCharactersRead = pCleanEnd - pBuffer;

            bool const isInteractiveEOF =
                (nCharactersRead == 2 && pBuffer[0] == ctrlZ && pBuffer[1] == '\n');
                
            return (isInteractiveEOF? 0 : static_cast< streamsize >( nCharactersRead ));
        }
        return 0;
    }
    
    virtual int_type underflow()
    {
        // Try to get some more input (maximum a line).
        if( pCurrent() == 0 || pCurrent() >= pEnd() )
        {
            streamsize const nCharactersRead =
                xsgetn( pBufferStart(), bufferSize() );

            if( nCharactersRead > 0 )
            {
                setg(
                    pBufferStart(),                     // Reading area start
                    pBufferStart(),                     // Reading area current
                    pBufferStart() + nCharactersRead    // Reading area end
                    );
            }
        }
        
        if( pCurrent() == 0 || pCurrent() >= pEnd() )
        {
            return Traits::eof();
        }
        return Traits::to_int_type( *pCurrent() );
    }
};

void setUtf8Mode( FILE* f, char const name[] )
{
    int const newMode = _setmode( _fileno( f ), _O_U8TEXT );
    hopefully( newMode != -1 )
        || throwX( string() + "setmode failed for " + name );
}

bool inputIsFromConsole()
{
    static HANDLE const inputHandle = GetStdHandle( STD_INPUT_HANDLE );

    DWORD consoleMode;
    return !!GetConsoleMode( inputHandle, &consoleMode );
}

void initStreams()
{
    if( inputIsFromConsole() )
    {
        static DirectInputBuffer buffer( *wcin.rdbuf() );
        wcin.rdbuf( &buffer );
    }
    setUtf8Mode( stdin, "stdin" );
    setUtf8Mode( stdout, "stdout" );
}

wstring lineFrom( wistream& stream )
{
    wstring     result;
    
    getline( stream, result );
    hopefully( !stream.fail() )
        || throwX( "lineFrom: getline failed" );
    return result;
}

int main()
{
    try
    {
        initStreams();

        wcout << "What's your name? ";
        wstring const name = lineFrom( wcin );
        wcout << "Pleased to meet you, " << name << "!" << endl;
        
        int const n = name.length();
        wcout << endl;
        wcout << "I represented your name as " << n << " wide characters:" << endl;
        for( int i = 0;  i < n;  ++i )
        {
            if( i > 0 ) { wcout << " | "; }
            wcout << hex << 0 + name[i] << " '" << name[i] << "'";
        }
        wcout << endl;
       
        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        wcerr << "!" << x.what() << endl;
    }
    return EXIT_FAILURE;
}
}

Testing this:

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl utf8_mode.msvc.input.fixed.cpp /Fe"fixed_input"
utf8_mode.msvc.input.fixed.cpp

W:\examples> fixed_input
What's your name? Bjørn
Pleased to meet you, Bjørn!

I represented your name as 5 wide characters:
42 'B' | 6a 'j' | f8 'ø' | 72 'r' | 6e 'n'

W:\examples> fixed_input
What's your name? 日本国 кошка
Pleased to meet you, 日本国 кошка!

I represented your name as 9 wide characters:
65e5 '日' | 672c '本' | 56fd '国' | 20 ' ' | 43a 'к' | 43e 'о' | 448 'ш' | 43a 'к' | 430 'а'

W:\examples> _

Yay! :-)

But I’d better mention again that in my English-language Windows 7 the console window stores the Chinese characters correctly but is only able to display them as rectangles, ▭. Correctly stored means that copy and paste works, which is why the text dumps above show these characters. However, the Norwegian and Russian characters are both stored and displayed correctly.

Also, I’d better mention that this is only a C++ level solution:

  • Input from the standard input stream should only be done via wcin.

Explained:

Input from the standard input stream should only be done at the C++ iostream level because in practice the UTF-8 mode input operation bugs can only be fixed at the C++ iostream level.

And input should only be done via the C++ wide stream wcin because, first of all, input data can be arbitrary, and secondly, with UTF-8 mode the narrow character operations just fail…

UTF-8 stream mode: the ugly (narrow streams)

C++11 (more precisely the N3290 final draft) §27.4.1/3:

  • “Mixing operations on corresponding wide- and narrow-character streams follows the same semantics as mixing such operations on FILEs, as specified in Amendment 1 of the ISO C standard.”

C99 (more precisely the N869 draft) §7.19.2/4:

  • “Each stream has an orientation. After a stream is associated with an external file, but before any operations are performed on it, the stream is without orientation. Once a wide-character input/output function has been applied to a stream without orientation, the stream becomes a wide-oriented stream. Similarly, once a byte input/output function has been applied to a stream without orientation, the stream becomes a byte-oriented stream. Only a call to the freopen function or the fwide function can otherwise alter the orientation of a stream. (A successful call to freopen removes any orientation.)”

C99 (more precisely the N869 draft) §7.19.2/5:

  • “Byte input/output functions shall not be applied to a wide-oriented stream and wide-character input/output functions shall not be applied to a byte-oriented stream.”

By these rules one would have to decide on using either cerr or wcerr in a program, but not both. That’s
not very practical, considering, for example, that one library component might produce error messages via cerr, while another might use wcerr for that purpose. But Visual C++ apparently follows a more practical-for-Windows-programming set of rules.

For Visual C++ 10.0 the fwide function is documented as being unimplemented. And from a practical point of view, at least at the level of outputting whole lines it apparently works fine to intermingle use of cout and wcout. So, happily, Visual C++ apparently just disregards the standard’s requirements and does not maintain an impractical explicit C FILE stream orientation.

Except … that with UTF-8 mode for the standard output stream, use of cout crashes the program!

[utf8_mode.msvc.byte_stream.ugly.cpp]
#include <stdexcept>        // std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::cout, std::endl
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

void initStreams()
{
    _setmode( _fileno( stdin ), _O_U8TEXT );
    _setmode( _fileno( stdout ), _O_U8TEXT );
    _setmode( _fileno( stderr ), _O_U8TEXT );
}

int main()
{
    try
    {
        initStreams();

        cout << "Hello, world!" << endl;
        cout << "Did you know, most Norwegians like 'blåbærsyltetøy'?" << endl;
        cerr << ":This is an error message, also mentioning 'blåbærsyltetøy'." << endl;

        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        cerr << "!" << x.what() << endl;
    }
    catch( ... )
    {
        cerr << "!Unknown exception." << endl;
    }
    return EXIT_FAILURE;
}

Screendump of crash dialogTesting:

W:\examples> ugly

W:\examples> _

The crash is caused by an assertion in the Visual C++ 10.0 fputc implementation, in source file [fputc.c]:

int __cdecl fputc (
        int ch,
        FILE *str
        )
{
    int retval=0;

    _VALIDATE_RETURN((str != NULL), EINVAL, EOF);

    _lock_str(str);
    __try {
        _VALIDATE_STREAM_ANSI_SETRET(str, EINVAL, retval, EOF);    // <-- Uh oh.

        if (retval==0)
        {
            retval = _putc_nolock(ch,str);
        }
    }
    __finally {
        _unlock_str(str);
    }

    return(retval);
}

where _VALIDATE_STREAM_ANSI_SETRET is defined thusly in [internal.h]:

/*
    We use _VALIDATE_STREAM_ANSI_SETRET to ensure that ANSI file operations(
    fprintf etc) aren't called on files opened as UNICODE. We do this check
    only if it's an actual FILE pointer &amp; not a string. It doesn't actually return
        immediately
*/

#define _VALIDATE_STREAM_ANSI_SETRET( stream, errorcode, retval, retexpr)            \
    {                                                                                \
        FILE *_Stream=stream;                                                        \
        int fn;                                                                      \
        _VALIDATE_SETRET(( (_Stream-&gt;_flag &amp; _IOSTRG) ||                             \
                           ( fn = _fileno(_Stream),                                  \
                             ( (_textmode_safe(fn) == __IOINFO_TM_ANSI) &amp;&amp;           \
                               !_tm_unicode_safe(fn)))),                             \
                         errorcode, retval, retexpr)                                 \
    }

I.e., with UTF-8 mode Microsoft’s code enforces the standard’s prohibition of mixing wide and narrow character operations on the same stream.

What to do, when the law is suddenly enforced?

UTF-8 stream mode: the ugly (narrow streams) – FIXED

Given the Windows convention of Windows ANSI encoding for internal char-based data, in particular for string literals, it does not make much sense to support cin input operations. With the user typing e.g. “кошка” the program running on a Norwegian machine would just produce something like “?????” as the Windows ANSI-encoded result. I.e. such narrow character input operations would replace a generally preferable hard crash with rather ungood data loss.

On the other hand, especially for students, existing library components may be logging error messages via narrow character operations on the standard error stream, like cerr << "!oops" << endl;. To avoid crashes for that, up at the C++ level one can install new iostream buffers for cout, cerr and clog, where those buffers do something reasonable such as forwarding all output to the wide streams (thereby providing Windows ANSI → UTF-8 translation). Down at the C level there is, however, no other practical crash-fix option than to leave the standard error stream in ANSI (untranslated) mode, so that the functionality down at the C level is much more restricted.

Back up at the C++ level again, this restriction for stderr can be compensated by adding UTF-8 conversion to the wide streams that write to the standard error stream, namely wcerr and wclog. One can and should also add custom direct console output support (more about that below). I.e., one is then essentially emulating the UTF-8 mode for wcerr and wclog, but only for operations at the C++ level.

This means that while the C++ level will/can work as expected, narrow character operations on stderr will only be guaranteed to produce the intended output when they’re limited to the ASCII character repertoire, and narrow character operations on stdin or stdout will crash.

Happily it’s very very easy to recognize a crash and/or garbage output, so that breaches of the programming conventions, …

  • use wcin for input,
     
  • don’t use C FILE level narrow characters operations on stdout, and
     
  • don’t use non-ASCII characters in narrow character operations on stderr

will likely be caught during testing.

Let’s start at the end of the internal processing, with the UTF-8 encoding support for wcerr.

With Visual C++ 10.0 the UTF-8 encoding itself is almost trivial, because first of all Visual C++ 10.0 supports the C++11 codecvt_utf8 facet, and secondly because Visual C++’s standard streams support the codecvt facet (the standard only requires file streams to support it), but as you can see below the console window result is not perfect!

[direct_display_troubles.cpp]
#include <codecvt>          // std::codecvt_utf8 (C++11)
#include <iostream>         // std::wcerr
#include <locale>           // std::locale
#include <memory>           // std::unique_ptr (C++11)
using namespace std;

void setUtf8Conversion( wostream& stream )
{
    typedef codecvt_utf8< wchar_t >     CodeCvt;

    unique_ptr< CodeCvt > pCodeCvt( new CodeCvt );
    locale  utf8Locale( locale(), pCodeCvt.get() );
    pCodeCvt.release();
    stream.imbue( utf8Locale );
}

int main()
{
    wcerr << "'blåbærsyltetøy' with default presentation." << endl;
    
    setUtf8Conversion( wcerr );
    wcerr << "'blåbærsyltetøy' with UTF-8 conversion applied." << endl;
}

Testing this:

W:\examples> chcp 1252

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl direct_display_troubles.cpp /Fe"x"
direct_display_troubles.cpp

W:\examples> x
'blåbærsyltetøy' with default presentation.
'blåbærsyltetøy' with UTF-8 conversion applied.

W:\examples> chcp 65001
Active code page: 65001

W:\examples> x
'bl�b�rsyltet�y' with default presentation.
'bl��b��rsyltet��y' with UTF-8 conversion applied.

W:\examples> (x 2>&1) >result

W:\examples> type result
'bl�b�rsyltet�y' with default presentation.
'blåbærsyltetøy' with UTF-8 conversion applied.

W:\examples> _

As you can see, the Visual C++ runtime library’ direct console output kicks in even without having set UTF-8 mode, and in some inexplicable way presents the resulting UTF-8 bytes – as garbage.

I have no idea how that happens.

However, one cure is to override it with a custom direct console output:

[direct_display_troubles.fixed.cpp]
#ifdef  _MSC_VER
#   pragma warning( disable: 4373 )     // "Your override overrides"
#endif

#include <assert.h>         // assert
#include <codecvt>          // std::codecvt_utf8 (C++11)
#include <iostream>         // std::wcerr
#include <locale>           // std::locale
#include <memory>           // std::unique_ptr (C++11)
#include <streambuf>        // std::basic_streambuf
using namespace std;

#undef  UNICODE
#define UNICODE
#undef  STRICT
#define STRING
#include <windows.h>    // GetStdHandle, GetConsoleMode, WriteConsole


template< class CharType >
class AbstractOutputBuffer
    : public basic_streambuf< CharType >
{
public:
    typedef basic_streambuf< CharType >     Base;
    typedef typename Base::traits_type      Traits;

    typedef Base                            StreamBuffer;

protected:
    virtual streamsize xsputn( char_type const* const s, streamsize const n ) = 0;

    virtual int_type overflow( int_type const c )
    {
        bool const cIsEOF   = Traits::eq_int_type( c, Traits::eof() );
        int_type const  failureValue    = Traits::eof();
        int_type const  successValue    = (cIsEOF? Traits::not_eof( c ) : c);

        if( !cIsEOF )
        {
            char_type const     ch                  = Traits::to_char_type( c );
            streamsize const    nCharactersWritten  = xsputn( &ch, 1 );

            return (nCharactersWritten == 1? successValue : failureValue);
        }
        return successValue;
    }

public:
    AbstractOutputBuffer()
    {}

    AbstractOutputBuffer( StreamBuffer& existingBuffer )
        : Base( existingBuffer )
    {}
};


class DirectOutputBuffer
    : public AbstractOutputBuffer< wchar_t >
{
public:
    enum StreamId { outputStreamId, errorStreamId, logStreamId };

private:
    StreamId    streamId_;

protected:
    virtual streamsize xsputn( wchar_t const* const s, streamsize const n )
    {
        static HANDLE const outputStreamHandle  = GetStdHandle( STD_OUTPUT_HANDLE );
        static HANDLE const errorStreamHandle   = GetStdHandle( STD_ERROR_HANDLE );

        HANDLE const    streamHandle    =
            (streamId_ == outputStreamId? outputStreamHandle : errorStreamHandle );
        
        DWORD nCharactersWritten    = 0;
        bool writeSucceeded         = !!WriteConsole(
            streamHandle, s, static_cast< DWORD >( n ), &nCharactersWritten, 0
            );
        return (writeSucceeded? static_cast< streamsize >( nCharactersWritten ) : 0);
    }

public:
    DirectOutputBuffer( StreamId streamId = outputStreamId )
        : streamId_( streamId )
    {}
};


void setUtf8Conversion( wostream& stream )
{
    typedef codecvt_utf8< wchar_t >     CodeCvt;

    unique_ptr< CodeCvt > pCodeCvt( new CodeCvt );
    locale  utf8Locale( locale(), pCodeCvt.get() );
    pCodeCvt.release();
    stream.imbue( utf8Locale );
}

bool isConsole( HANDLE streamHandle )
{
    DWORD consoleMode;
    return !!GetConsoleMode( streamHandle, &consoleMode );
}

bool isConsole( DWORD stdStreamId )
{
    return isConsole( GetStdHandle( stdStreamId ) );
}

void setDirectOutputSupport( wostream& stream )
{
    typedef DirectOutputBuffer  DOB;

    if( &stream == &wcout )
    {
        if( isConsole( STD_OUTPUT_HANDLE ) )
        {
            static DOB  outputStreamBuffer( DOB::outputStreamId );
            stream.rdbuf( &outputStreamBuffer );
        }
    }
    else if( &stream == &wcerr )
    {
        if( isConsole( STD_ERROR_HANDLE ) )
        {
            static DOB errorStreamBuffer( DOB::errorStreamId );
            stream.rdbuf( &errorStreamBuffer );
        }
    }
    else if( &stream == &wclog )
    {
        if( isConsole( STD_ERROR_HANDLE ) )
        {
            static DOB logStreamBuffer( DOB::logStreamId );
            stream.rdbuf( &logStreamBuffer );
        }
    }
    else
    {
        assert(( "setDirectOutputSupport: unsupported stream", false ));
    }
}

int main()
{
    wcerr << "'blåbærsyltetøy' with default presentation." << endl;
    
    setUtf8Conversion( wcerr );
    wcerr << "'blåbærsyltetøy' with UTF-8 conversion applied." << endl;
    
    setDirectOutputSupport( wcerr );
    wcerr << "'blåbærsyltetøy' with UTF-8 conversion & direct output applied." << endl;
}

Testing this:

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl direct_display_troubles.fixed.cpp /Fe"x"
direct_display_troubles.fixed.cpp

W:\examples> x
'blåbærsyltetøy' with default presentation.
'blåbærsyltetøy' with UTF-8 conversion applied.
'blåbærsyltetøy' with UTF-8 conversion & direct output applied.

W:\examples> chcp 65001
Active code page: 65001

W:\examples> x
'bl�b�rsyltet�y' with default presentation.
'bl��b��rsyltet��y' with UTF-8 conversion applied.
'blåbærsyltetøy' with UTF-8 conversion & direct output applied.

W:\examples> (x 2>&1) >result

W:\examples> type result
'bl�b�rsyltet�y' with default presentation.
'blåbærsyltetøy' with UTF-8 conversion applied.
'blåbærsyltetøy' with UTF-8 conversion & direct output applied.

W:\examples> _

Yay! Now we’re ready for tackling the previous section’s program. Again, the main function is exactly as before:

[utf8_mode.msvc.byte_stream.fixed.cpp]
#ifdef  _MSC_VER
#   pragma warning( disable: 4373 )     // "Your override overrides"
#endif

#include <assert.h>         // assert
#include <codecvt>          // std::codecvt_utf8 (C++11)
#include <stdexcept>        // std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <streambuf>        // std::basic_streambuf
#include <string>           // wstring
#include <iostream>         // std::cout, std::endl
#include <locale>           // std::locale
#include <memory>           // std::unique_ptr (C++11)
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

#undef  UNICODE
#define UNICODE
#undef  STRICT
#define STRING
#include <windows.h>    // MultiByteToWideChar


template< class CharType >
class AbstractOutputBuffer
    : public basic_streambuf< CharType >
{
public:
    typedef basic_streambuf< CharType >     Base;
    typedef typename Base::traits_type      Traits;

    typedef Base                            StreamBuffer;

protected:
    virtual streamsize xsputn( char_type const* const s, streamsize const n ) = 0;

    virtual int_type overflow( int_type const c )
    {
        bool const cIsEOF   = Traits::eq_int_type( c, Traits::eof() );
        int_type const  failureValue    = Traits::eof();
        int_type const  successValue    = (cIsEOF? Traits::not_eof( c ) : c);

        if( !cIsEOF )
        {
            char_type const     ch                  = Traits::to_char_type( c );
            streamsize const    nCharactersWritten  = xsputn( &ch, 1 );

            return (nCharactersWritten == 1? successValue : failureValue);
        }
        return successValue;
    }

public:
    AbstractOutputBuffer()
    {}

    AbstractOutputBuffer( StreamBuffer& existingBuffer )
        : Base( existingBuffer )
    {}
};


class OutputForwarderBuffer
    : public AbstractOutputBuffer< char >
{
public:
    typedef AbstractOutputBuffer< char >    Base;
    typedef Base::Traits                    Traits;
    
    typedef Base::StreamBuffer              StreamBuffer;
    typedef basic_streambuf<wchar_t>        WideStreamBuffer;

private:
    WideStreamBuffer*       pWideStreamBuffer_;
    wstring                 wideCharBuffer_;

    OutputForwarderBuffer( OutputForwarderBuffer const& );      // No such.
    void operator=( OutputForwarderBuffer const& );             // No such.

protected:
    virtual streamsize xsputn( char const* const s, streamsize const n )
    {
        if( n == 0 ) { return 0; }

        int const   nAsInt  = static_cast<int>( n );    //  Visual C++ sillywarnings.
        wideCharBuffer_.resize( nAsInt );
        int const nWideCharacters = MultiByteToWideChar(
            CP_ACP,             // Windows ANSI
            MB_PRECOMPOSED,     // Always precompose characters (this is the default).
            s,                  // Narrow character string.
            nAsInt,             // Number of bytes in narrow character string.
            &wideCharBuffer_[0],
            nAsInt              // Wide char buffer size.
            );
        assert( nWideCharacters > 0 );
        return pWideStreamBuffer_->sputn( &wideCharBuffer_[0], nWideCharacters );
    }

public:
    OutputForwarderBuffer(
        StreamBuffer&       existingBuffer,
        WideStreamBuffer*   pWideStreamBuffer
        )
        : Base( existingBuffer )
        , pWideStreamBuffer_( pWideStreamBuffer )
    {}
};


class DirectOutputBuffer
    : public AbstractOutputBuffer< wchar_t >
{
public:
    enum StreamId { outputStreamId, errorStreamId, logStreamId };

private:
    StreamId    streamId_;

protected:
    virtual streamsize xsputn( wchar_t const* const s, streamsize const n )
    {
        static HANDLE const outputStreamHandle  = GetStdHandle( STD_OUTPUT_HANDLE );
        static HANDLE const errorStreamHandle   = GetStdHandle( STD_ERROR_HANDLE );

        HANDLE const    streamHandle    =
            (streamId_ == outputStreamId? outputStreamHandle : errorStreamHandle );
        
        DWORD nCharactersWritten    = 0;
        bool writeSucceeded         = !!WriteConsole(
            streamHandle, s, static_cast< DWORD >( n ), &nCharactersWritten, 0
            );
        return (writeSucceeded? static_cast< streamsize >( nCharactersWritten ) : 0);
    }

public:
    DirectOutputBuffer( StreamId streamId = outputStreamId )
        : streamId_( streamId )
    {}
};


void setUtf8Conversion( wostream& stream )
{
    typedef codecvt_utf8< wchar_t >     CodeCvt;

    unique_ptr< CodeCvt > pCodeCvt( new CodeCvt );
    locale  utf8Locale( locale(), pCodeCvt.get() );
    pCodeCvt.release();
    stream.imbue( utf8Locale );
}

bool isConsole( HANDLE streamHandle )
{
    DWORD consoleMode;
    return !!GetConsoleMode( streamHandle, &consoleMode );
}

bool isConsole( DWORD stdStreamId )
{
    return isConsole( GetStdHandle( stdStreamId ) );
}

void setDirectOutputSupport( wostream& stream )
{
    typedef DirectOutputBuffer  DOB;

    if( &stream == &wcout )
    {
        if( isConsole( STD_OUTPUT_HANDLE ) )
        {
            static DOB  outputStreamBuffer( DOB::outputStreamId );
            stream.rdbuf( &outputStreamBuffer );
        }
    }
    else if( &stream == &wcerr )
    {
        if( isConsole( STD_ERROR_HANDLE ) )
        {
            static DOB errorStreamBuffer( DOB::errorStreamId );
            stream.rdbuf( &errorStreamBuffer );
        }
    }
    else if( &stream == &wclog )
    {
        if( isConsole( STD_ERROR_HANDLE ) )
        {
            static DOB logStreamBuffer( DOB::logStreamId );
            stream.rdbuf( &logStreamBuffer );
        }
    }
    else
    {
        assert(( "setDirectOutputSupport: unsupported stream", false ));
    }
}

void initStreams()
{
    // Set up UTF-8 conversions &  direct console output:
    _setmode( _fileno( stdin ), _O_U8TEXT );
    _setmode( _fileno( stdout ), _O_U8TEXT );
    setUtf8Conversion( wcerr );  setDirectOutputSupport( wcerr );
    setUtf8Conversion( wclog );  setDirectOutputSupport( wclog );

    // Forward narrow character output to the wide streams:
    static OutputForwarderBuffer    coutBuffer( *cout.rdbuf(), wcout.rdbuf() );
    static OutputForwarderBuffer    cerrBuffer( *cerr.rdbuf(), wcerr.rdbuf() );
    static OutputForwarderBuffer    clogBuffer( *clog.rdbuf(), wclog.rdbuf() );

    cout.rdbuf( &coutBuffer );
    cerr.rdbuf( &cerrBuffer );
    clog.rdbuf( &clogBuffer );
}

int main()
{
    try
    {
        initStreams();

        cout << "Hello, world!" << endl;
        cout << "Did you know, most Norwegians like 'blåbærsyltetøy'?" << endl;
        cerr << ":This is an error message, also mentioning 'blåbærsyltetøy'." << endl;

        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        cerr << "!" << x.what() << endl;
    }
    catch( ... )
    {
        cerr << "!Unknown exception." << endl;
    }
    return EXIT_FAILURE;
}

Testing this:

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl utf8_mode.msvc.byte_stream.fixed.cpp /Fe"x"
utf8_mode.msvc.byte_stream.fixed.cpp

W:\examples> x
Hello, world!
Did you know, most Norwegians like 'blåbærsyltetøy'?
:This is an error message, also mentioning 'blåbærsyltetøy'.

W:\examples> chcp 65001
Active code page: 65001

W:\examples> x
Hello, world!
Did you know, most Norwegians like 'blåbærsyltetøy'?
:This is an error message, also mentioning 'blåbærsyltetøy'.

W:\examples> (x 2>&1) >result

W:\examples> type result
Hello, world!
Did you know, most Norwegians like 'blåbærsyltetøy'?
:This is an error message, also mentioning 'blåbærsyltetøy'.

W:\examples> _

Summary

To make the Visual C++ _O_U8TEXT UTF-8 stream mode work in general, we had to

  • fix the input-from-console functionality by adding a special direct console input buffer in the C++ level wcin stream,
     
  • avoid crashes for narrow character operations on the standard error stream by keeping that stream in ANSI mode, installing forwarder buffers for cout and cerr, installing an UTF-8 conversion facet in wcerr, and adding direct console output support to wcerr, and
     
  • foresake the use of input and narrow character operations at the C level, except for the standard error stream.

With all this bug-fixing and support machinery added, it’s almost as if we had to implement the _O_U8TEXT mode from scratch! What does it really buy, then? What is the point of using that mode?

Well, essentially the _O_U8TEXT UTF-8 mode gives conversion to UTF-8 for C level wide character output operations on the standard output stream. It does not do interactive input, and it can not be reasonably used for the standard error stream. One might therefore say that Microsoft Unicode guru Michael Kaplan’s original blog posting about this mode, where it appeared to a simple general solution on its own, was a bit too optimistic!

Anyway, to be utterly clear, with this approach one has three main text encodings to deal with in Windows, and three main text encodings to deal with in a typical Linux installation:

Internal wchar_t data: Internal char data: External data:
Windows: UTF-16 Windows ANSI UTF-8
Linux: Usually UTF-32 but also UTF-16 UTF-8 UTF-8

And the main forces in play yielding the Windows row of the above table, are that …

  • the core Windows API is UTF-16 based, and wchar_t is 16 bits in Windows,
     
  • Visual C++ has Windows ANSI as its C++ execution character set, which e.g. forces ordinary narrow character literals to Windows ANSI encoding even when the source code is UTF-8, and
     
  • one desires UTF-8 for external data both to avoid data loss and for general interoperability.

There is not much that can be done about these forces, so the basic approach to deal with these issues is either very similar to what I have described here, or will otherwise involve pretty painful trade-offs.

Not to say, of course, that the trade-offs in this posting’s approach aren’t a little painful! :-)

Cheers, & enjoy!

Unicode part 1: Windows console i/o approaches



The Windows console subsystem has a host of Unicode-related bugs. And standard Windows programs such as more (not to mention the C# 4.0 compiler csc) just crash when they’re run from a console window with UTF-8 as active codepage, perplexingly claiming that they’re out of memory. On top of that the C++ runtime libraries of various compilers differ in how they behave. Doing C++ Unicode i/o in Windows consoles is therefore problematic. In this series I show how to work around limitations of the Visual C++ _O_U8TEXT file mode, with the Visual C++ and g++ compilers. This yields an automatic translation between external UTF-8 and internal UTF-16, enabling Windows console i/o of characters in the Basic Multilingual Plane.

Introduction

In both Windows and Linux properly internationalized applications use either UTF-16 or UTF-32 for their internal string handling. For example, the popular cross platform ICU library (International Components for Unicode) is based on UTF-16 encoded strings. For this kind of application Windows seems to be the better fit, since Windows’ API is UTF-16 based while Linux’ API is effectively, on a modern installation, UTF-8 based.

Still, in a simple console program one does not typically take on the quite steep overhead of full-fledged Unicode handling.

Instead of using a full-fledged Unicode handling library like ICU, one then relies on just the standard C and C++ libraries, and the Unicode handling reduces to what can easily be expressed using just the direct C++ language and standard library support.

How the Linux “all UTF-8” approach does not work in Windows

In Linux the typical small Unicode console program has everything char-based and UTF-8 encoded. The external data, the internal strings, the string literals, and of course the C or C++ source code, are all UTF-8 encoded. The total unification allows simple programs like this:

[utf8_sans_bom.all_utf8.cpp]
#include <stdexcept>        // std::runtime_error, std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::cout, std::cerr, std::endl
#include <string>           // std::string
using namespace std;

bool throwX( string const& s ) { throw runtime_error( s ); }
bool hopefully( bool v ) { return v; }

string lineFrom( istream& stream )
{
    string result;
    getline( stream, result );
    hopefully( !stream.fail() )
        || throwX( "lineFrom: failed to read line" );
    return result;
}

int main()
{
    try
    {
        static char const       narrowText[]    = "Blåbærsyltetøy! 日本国 кошка!";
        
        cout << "Narrow text: " << narrowText << endl;
        cout << endl;
        cout << "What's your name? ";
        string const name = lineFrom( cin );
        cout << "Glad to meet you, " << name << "!" << endl;
        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        cerr << "!" << x.what() << endl;
    }
    return EXIT_FAILURE;
}

Testing this in Ubuntu 11.10:

[~/blog/examples]
$ g++ utf8_sans_bom.all_utf8.cpp
[~/blog/examples]
$ ./a.out
Narrow text: Blåbærsyltetøy! 日本国 кошка!

What's your name? Bjørn Bråten Sæter
Glad to meet you, Bjørn Bråten Sæter!
[~/blog/examples]
$  _

Yay, it worked OK in Linux!

Testing the very same source code file in Windows using the same Linux-origins compiler (namely g++), and intentionally not specifying any codepage for the console window:

W:\examples> g++ -pedantic -Wall utf8_sans_bom.all_utf8.cpp

W:\examples> a
Narrow text: Blåbærsyltetøy! 日本国 кошка!

What's your name? Bjørn Bråten Sæter
Glad to meet you, Bjorn Bråten Sæter!

W:\examples> _

One reason for the gobbledygook here is that the Windows console by default assumes that the program produces OEM encoded text. That means, it assumes that the text is encoded using the original IBM PC character set, or a variation of that old character set. This encoding assumption is called the console window’s active codepage, and it can be inspected and changed via the chcp command, e.g. from codepage 437 (original IBM PC character set) to 65001 (UTF-8):

W:\examples> chcp
Active code page: 437

W:\examples> chcp 65001
Active code page: 65001

W:\examples> a
Narrow text: Blåbærsyltetøy! 日本国 кошка!huh?

W:\examples> _

Positive: the initial UTF-8 text output appeared to work. The Chinese characters 日本国 displayed as just empty rectangles, but they copied OK. Both the Norwegian and Russian copied OK and also displayed OK.

Negative: input did apparently not work, and it apparently caused some of the program’s output (including the prompt before the input operation) to disappear!

Exactly what went wrong above is difficult to say for sure. It might be the input operation, or it might be something else. However, the exact cause is irrelevant because input fails outright, not just producing weird side effects, if the user types in some non-ASCII characters such as Norwegian æ, ø and å:

W:\examples> a
Narrow text: Blåbærsyltetøy! 日本国 кошка!Bjørn Bråten Sæter
!lineFrom: failed to read line

W:\examples> _

About direct console i/o

Given that total failure for the “all UTF-8” approach has been established, it may perhaps appear to be overkill to also show the unintelligible output effect with the Windows platform’s major compiler, Visual C++ (here version 10.0), but as you’ll see it’s relevant:

W:\examples> cl utf8_sans_bom.all_utf8.cpp /Fe"b"
utf8_sans_bom.all_utf8.cpp

W:\examples> chcp
Active code page: 65001

W:\examples> b
Narrow text: Bl��b��rsyltet��y! ��������� ����������!

What's your name? Bjørn Bråten Sæter
!lineFrom: failed to read line

W:\examples> _

Here the Visual C++ runtime detects that the standard output is connected to a console window. And instead of sending the text via the ordinary standard output stream, it then attempts to place the correct Unicode UCS2-encoded characters directly in the console window’s text buffer. However, since the C++ source code was encoded as UTF-8 without BOM (as is usual in Linux), the Visual C++ compiler erroneously assumed that the source code was encoded as Windows ANSI, and so, since Visual C++ has Windows ANSI sort of hardwired as its C++ narrow character execution character set, it blindly copied the string literal’s bytes to the executable’s string values, whence the runtime, for its direct console i/o, is given UTF-8 bytes instead of the Windows ANSI bytes that it expects – so that its helpful translation to UCS2 fails…

At the Windows API level the runtime implements direct console output by calling the WriteConsole function instead of the WriteFile function. And similarly, if the console input had worked, then it would probably have been via a call to the ReadConsole function instead of the ReadFile function. The WriteConsole function accesses the console window’s text buffer directly and takes an UTF-16 wchar_t based argument, and ditto for ReadConsole.

Portable source code should be UTF-8 with BOM

One can avoid the direct console i/o by redirecting the output.

Such redirection then establishes that the output text byte level data is good, that all would have been well for this particular program’s output, except for the interference from the probably well-intentioned direct console i/o help attempt:

W:\examples> echo Bjørn Bråten Sæter | b >result

W:\examples> type result
Narrow text: Blåbærsyltetøy! 日本国 кошка!

What's your name? Glad to meet you, Bjørn Bråten Sæter !

W:\examples> _

And because the data is correct, one can be sure that the Visual C++ compiler was tricked into assuming that the source code was ANSI Western. And this then means that any wide string literal, which a Windows compiler has to translate to UTF-16, will be incorrectly translated if it contains any non-ASCII characters. Hence, for portable source code it is not a good idea to encode the source code as UTF-8 without BOM – for that is effectively to lie to the compiler.

Now that also g++ accepts a BOM at the start of the source code, portable source code should therefore be encoded as UTF-8 with BOM.

With the BOM in place Visual C++ correctly determines that the source code is UTF-8 encoded, although as of late 2011 this appears to still be undocumented. And with a correct assumption about the source code’s encoding, narrow string literals are correctly translated to Windows ANSI encoded string values in the executable. For Unicode literals in Windows one should therefore use wide string literals, e.g. L"Blåbærsyltetøy! 日本国 кошка!", which in Windows ends up as an UTF-16 encoded string value in the executable.

The Visual C++ UTF-8 stream mode

Use source code encoded as UTF-8 with BOM, and use wide string literals, OK (or rather, one just has to accept that complication!), but how does one then output one of these literals?

E.g., std::wcout in Windows has a rather strong tendency to translate down to Windows ANSI, not to UTF-8?

Well, in his 2008 blog posting Conventional wisdom is retarded, aka What the @#%&* is _O_U16TEXT? Michael Kaplan explained that

“the [Visual C++] CRT? Starting in 2005/8.0, it knows more about Unicode than any of us having been giving it credit for…”

The Visual C++ runtime library can convert automatically between internal UTF-16 and external UTF-8, if you just ask it to do so by calling the _setmode function with the appropriate file descriptor number and mode flag. E.g., mode _O_U8TEXT causes conversion to/from UTF-8.

One reason that many people have not known about the Unicode support that he discusses there, a Visual C++ Unicode stream mode, is that it’s mostly undocumented. Kaplan gives a link to documentation of the deprecated _wsopen function, as one place where the mode flags have been (inadvertently?) documented. However, the main usage is through the _setmode function, where, on the contrary, the official documentation goes on about how _setmode will invoke the “invalid parameter handler” unless the mode argument is either _O_TEXT or _O_BINARY. So, by using this functionality one is not just in ordinary Microsoft undocumented land. One is wholly over in explicitly-documented-as-not-working land.

On the other hand, considering that the official documentation is plain wrong about many things (e.g., for Visual C++ 10 it maintains that the source code encoding is limited to ASCII), and that the _setmode documentation is incorrect about the argument checking, and that the g++ compiler provides C level support for the _O_U8TEXT mode feature, considering all that one may choose to ignore the will-not-work statements of the documentation and just treat it as a documentation defect, for what good is a feature that can’t be used?

Since there is not really any alternative in order to get UTF-8 translation also down at the C library level, this is the approach that I’m going to discuss more detailed in part 2.

It might seem from Kaplan’s blog posting that you don’t have to do more than just set the mode, and go! But as you can expect from something in explicitly-documented-as-not-working land, it’s not fully implemented even in Visual C++. And even less fully implemented in g++…

Summary so far

Above I introduced two approaches to Unicode handling in small Windows console programs:

  • The all UTF-8 approach where everything is encoded as UTF-8, and where there are no BOM encoding markers.
     
  • The wide string approach where all external text (including the C++ source code) is encoded as UTF-8, and all internal text is encoded as UTF-16.

The all UTF-8 approach is the approach used in a typical Linux installation. With this approach a novice can remain unaware that he is writing code that handles Unicode: it Just Works™ – in Linux. However, we saw that it mass-failed in Windows:

  • Input with active codepage 65001 (UTF-8) failed due to various bugs.
     
  • Console output with Visual C++ produced gibberish due to the runtime library’s attempt to help by using direct console output.
     
  • I mentioned how wide string literals with non-ASCII characters are incorrectly translated to UTF-16 by Visual C++ due to the necessary lying to Visual C++ about the source code encoding (which is accomplished by not having a BOM at the start of the source code file).

The wide string approach, on the other hand, was shown to have special support in Visual C++, via the _O_U8TEXT file mode, which I called an UTF-8 stream mode. But I mentioned that as of Visual C++ 10 this special file mode is not fully implemented and/or it has some bugs: it cannot be used directly but needs some scaffolding and fixing. That’s what part 2 is about.


Cheers!

Cheers, & enjoy!

2010 in review

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

Healthy blog!

The Blog-Health-o-Meter™ reads Wow.

Crunchy numbers

Featured image

A helper monkey made this abstract painting, inspired by your stats.

A Boeing 747-400 passenger jet can hold 416 passengers. This blog was viewed about 8,400 times in 2010. That’s about 20 full 747s.

In 2010, there were 40 new posts, not bad for the first year! There were 8 pictures uploaded, taking up a total of 304kb.

The busiest day of the year was May 7th with 182 views. The most popular post that day was [cppx] Compiler-specific functionality abstracted.

Where did they come from?

The top referring sites in 2010 were stackoverflow.com, freak.no, thechaosengine.com, gennaroprota.wordpress.com, and healthfitnesstherapy.com.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

1

[cppx] Compiler-specific functionality abstracted May 2010
3 comments

2

[cppx] How to avoid disastrous integer wrap-around May 2010
10 comments

3

[cppx] 3 ways to mix in a generic cloning implementation June 2010

4

[cppx] B true, or B thrown! (Using the >> throwing pattern) July 2010
7 comments

5

[cppx] How to do typed optional arguments in C++98 May 2010

Ch 4 of my Norwegian intro to C++ available

The Norwegian introduction to C++ programming (a bit Windows-specific) is at Google Docs, in PDF format, 4 chapters so far:

Introduksjon til C++-programmering (Windows)

Each file has a nice table of contents but for that you need to download the PDF and view it in e.g. Foxit or Adobe Acrobat. Ch 1, the Introduction, is just 1 page, though. Ch 2, tooling up with Visual C++ and learning about some Windows stuff, is more pages. And so is ch 3, about basic C++ such as loops and decisions. And ch 4, about creating console programs (all programs so far just GUI), chimes in at some 50 pages!

Perhaps it’ll become a book…

Here’s a table of contents generated by (1) using a Word TOC field and half-documented RD fields to refer to the chapters, (2) [Shift Ctrl F9] in Word (is that still documented anywhere?) to “lock” the text, (3) edit, removing unwanted entries, (4) copy as text to Crimson Editor, save, and (5) run a very very hairy C++ program to generate the HTML.

Oh, I see in the preview that instead of a purely numbered list, in the WordPress blog I get letters and roman numerals!

So be it – but there’s also a PDF of the original over at Google docs (link above).

  1. Introduksjon. | 1
  2. Første program, etc. | 1
    1. Gratis verktøy. | 1
    2. Muligens ikke helt typiske installasjonsproblemer… | 2
    3. “Hallo, verden!” i Visual Studio / om IDE prosjekter. | 6
    4. Feilretting i Visual Studio / generelt om C++ typesjekking. | 15
    5. Hva “Hallo, verden!” programteksten betyr. | 18
    6. Spesielt aktuelle Windows-ting for nybegynneren. | 21
      1. Makroer og Unicode/ANSI-versjoner av Windows API-funksjoner. | 22
      2. Moderne utseende på knapper etc. / om DLL-er og manifest-filer. | 23
      3. Ikon og versjonsinformasjon / [.exe]-fil ressurser. | 28
    7. Gir C++ ekstra mye kode og kompleksitet? | 32
    8. Å finne relevant informasjon om ting. | 32
      1. Tipsruter og automatisk fullføring. | 32
      2. Å gå direkte til en aktuell deklarasjon eller definisjon. | 33
      3. Full teknisk dokumentasjon / hjelp / kort om Microsofts “T” datatyper. | 34
      4. Dokumentasjon av C++ språket og C++ standardbiblioteket. | 36
      5. Diskusjonsfora på nettet / FAQ-er. | 38
  3. Et første subsett av C++. | 1
    1. Gjenbruk av egendefinerte headerfiler. | 1
      1. En wrapper for [windows.h]. | 2
      2. Å konfigurere en felles headerfil søkesti i Visual Studio 2010. | 6
      3. En muligens enklere & mer pålitelig måte å konfigurere Visual Studio på. | 9
    2. Grunnleggende data. | 12
      1. Variabler, tilordninger, oppdateringer, regneuttrykk, implisitt konvertering. | 14
      2. Implisitte konverteringer. | 15
      3. Initialisering og const. | 16
    3. Tekstpresentasjon og strenger. | 17
      1. Arrays som buffere, konvertering tall ? tekst. | 17
      2. Strenger, konkatenering og std::wstring-typen, anrop av medlemsfunksjon. | 18
      3. Å lage tekstgenererings-støtte / egendefinerte funksjoner & operatorer. | 22
    4. Løkker, valg og sammenligningsuttrykk. | 27
      1. Sammenligninger og boolske uttrykk. | 32
      2. Valg. | 34
      3. Løkker. | 39
    5. Funksjoner. | 41
      1. Hva du kan og ikke kan gjøre med en C++ funksjon. | 41
      2. Funksjoner som abstraksjonsverktøy. | 41
      3. Verdioverføring og referanseoverføring av argumenter. | 45
  4. Kommandotolkeren. | 1
    1. Windows kommandotolkeren [cmd.exe]. | 2
      1. Å kjøre opp en kommandotolker-instans / konfigurering av konsollvinduer. | 2
      2. Kommandoer / hjelp. | 8
      3. Kommandoredigering & utklippstavle-operasjoner. | 11
      4. Linjekontinuering & tegn-escaping. | 11
      5. Operatorer & sammensatte kommandoer / omdirigering & rørledninger. | 12
      6. Erstatting av miljøvariabel-navn / arv av miljøvariabler. | 15
      7. Kommandotolkerens søk etter programmer: %path% og %pathext%. | 16
    2. Navigasjon. | 17
    3. Å kompilere fra kommandotolkeren. | 21
      1. Å nei! “Hallo, verden!” igjen! | 22
      2. Konsoll kontra GUI subsystem. | 24
      3. Å angi linker-opsjoner til kompilatoren / separat kompilering og linking. | 26
      4. Å be kompilatoren om standard C++, please. | 27
      5. Å angi headerfilkataloger, også kjent som inkluderingskataloger. | 28
    4. Batchfiler – å automatisere f.eks. et standardoppsett. | 31
    5. C++ iostreams. | 33
      1. iostream-objekter for standard datastrømmene. | 33
      2. Datastrøm orientering: nix mix (av char og wchar_t datastrømobjekter). | 36
      3. Å detektere “slutt på datastrømmen” (EOF, end of file). | 36
      4. Innlesing av strenger. | 40
      5. Praktikalitetsdigresjon: hvordan bli kvitt navneromskvalifikasjonene. | 42
      6. Innlesing av tall. | 43
      7. Formatert utskrift med iostream manipulatorer. | 48

Cheers, & enjoy! – Alf