2013 in review

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 10,000 times in 2013. If it were a concert at Sydney Opera House, it would take about 4 sold-out performances for that many people to see it.

Click here to see the complete report.

2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

600 people reached the top of Mt. Everest in 2012. This blog got about 10,000 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 17 years to get that many views.

Click here to see the complete report.

C++ features in Visual C++11, yeah!

Recently Microsoft surprised me – positively! – with supporting newfangled C++11 features in the November CTP of their Visual C++ 11.0 compiler:

  • Variadic templates
  • Uniform initialization and initializer_lists
  • Delegating constructors
  • Raw string literals
  • Explicit conversion operators
  • Default template arguments for function templates

The library has not yet been updated to take advantage, but still this means that it’s now possible to avoid macro hell for things such as makeUnique (but unforutunately not yet for e.g. platform-specific Unicode strings, no support yet for the newfangled literals).

On the other hand, Visual C++ implements the regexp library, while g++ does not. And Visual C++ has working exceptions, while AFAIK the dang[1] compiler still has not. So this may just look as if Visual C++ is now overtaking g++ and the dang compiler in the race to support the C++11 standard! :-)

[1] As Brian Keminghan remarked, dang if I can remember its name!

Liskov’s substitution principle in C++ <

Part I of III: The LSP and value assignment.

Abstract:
Liskov’s principle of substitution, a principle of generally good class design for polymorphism, strongly implies the slicing behavior of C++ value assignment. Destination value slicing can cause a partial assignment, which can easily break data integrity. Three solutions are (1) static checking of assignment, (2) dynamic checking of assignment, and (3), generally easiest, prohibiting assignment, then possibly providing cloning functionality as an alternative.

Contents:

  • > A short review of the LSP.
  • > How the LSP (almost) forces slicing in C++.
  • > The danger of slicing: type constraint violations.
  • > The possible design alternatives wrt. providing value assignment.
  • > How to provide assignment with STATIC CHECKING of type constraints.
  • > How to provide assignment with DYNAMIC CHECKING of type constraints.
  • > How to provide CLONING as an alternative to assignment.

Continue reading

ResultOf a function with C++11

Using std::result_of requires you to specify the function argument types. Which is not very practical when you don’t know the function signature. Happily @potatoswatter (David Krauss) over at Stack Overflow pointed out to me that std::function provides the desired functionality:

template< class Func >
struct ResultOf
{
    typedef typename std::function<
        typename std::remove_pointer<Func>::type
        >::result_type T;
};

Mainly, this saves one from writing a large number of partial specializations for Visual C++ 10.0, which lacks support for C++11 variadic templates.

– Enjoy!

Unicode part 2: UTF-8 stream mode



The Windows console subsystem has a host of Unicode-related bugs. And standard Windows programs such as more (not to mention the C# 4.0 compiler csc) just crash when they’re run from a console window with UTF-8 as active codepage, perplexingly claiming that they’re out of memory. On top of that the C++ runtime libraries of various compilers differ in how they behave. Doing C++ Unicode i/o in Windows consoles is therefore problematic. In this series I show how to work around limitations of the Visual C++ _O_U8TEXT file mode, with the Visual C++ and g++ compilers. This yields an automatic translation between external UTF-8 and internal UTF-16, enabling Windows console i/o of characters in the Basic Multilingual Plane.

Recap

In part 1 I introduced two approaches to Unicode handling in small Windows console programs:

  • The all UTF-8 approach where everything is encoded as UTF-8, and where there are no BOM encoding markers.
     
  • The wide string approach where all external text (including the C++ source code) is encoded as UTF-8, and all internal text is encoded as UTF-16.

The all UTF-8 approach is the approach used in a typical Linux installation. With this approach a novice can remain unaware that he is writing code that handles Unicode: it Just Works™ – in Linux. However, we saw that it mass-failed in Windows:

  • Input with active codepage 65001 (UTF-8) failed due to various bugs.
     
  • Console output with Visual C++ produced gibberish due to the runtime library’s attempt to help by using direct console output.
     
  • I mentioned how wide string literals with non-ASCII characters are incorrectly translated to UTF-16 by Visual C++ due to the necessary lying to Visual C++ about the source code encoding (which is accomplished by not having a BOM at the start of the source code file).

The wide string approach, on the other hand, was shown to have special support in Visual C++, via the _O_U8TEXT file mode, which I called an UTF-8 stream mode. This mode works down at the C FILE level so that wide character operations on C FILE streams get automatic conversion to/from external UTF-8 encoding. That C FILE level support is needed and is practically impossible to do for the application programmer, so it’s a very good thing to have that UTF-8 stream mode…

I mentioned that as of Visual C++ 10 the UTF-8 mode is not fully implemented, and that it apparently has some bugs. I.e., that it cannot be used directly but needs some scaffolding and fixing. That’s what this second part is about.

Here I do not yet consider the g++ compiler. All this code is Visual C++ specific. However, I have done generally the same with g++, and I will probably discuss that in a third installment.

UTF-8 stream mode: the good (wide stream output)

The good news about the _O_U8TEXT mode is that it works for wide stream output, even for output of narrow strings via the wide stream:

[utf8_mode.msvc.output.good.cpp]
#include <stdexcept>        // std::runtime_error, std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::wcout, std::wcerr, std::endl
#include <string>           // std::string, std::wstring
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

bool throwX( string const& s ) { throw runtime_error( s ); }
bool hopefully( bool v ) { return v; }

void setUtf8Mode( FILE* f, char const name[] )
{
    int const newMode = _setmode( _fileno( f ), _O_U8TEXT );
    hopefully( newMode != -1 )
        || throwX( string() + "setmode failed for " + name );
}

int main()
{
    try
    {
        static char const       narrowText[]    = "Blåbærsyltetøy! 日本国 кошка!";
        static wchar_t const    wideText[]      = L"Blåbærsyltetøy! 日本国 кошка!";

        setUtf8Mode( stdout, "stdout" );

        wcout << "Narrow text: " << narrowText << endl;
        wcout << "Wide text:   " << wideText << endl;
        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        wcerr << "!" << x.what() << endl;
    }
    return EXIT_FAILURE;
}

Visual C++ produces a series of warnings for this source code:

W:\examples> cl utf8_mode.msvc.output.good.cpp /Fe"good"
utf8_mode.msvc.output.good.cpp
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u65E5' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u672C' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u56FD' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u043A' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u043E' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u0448' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u043A' cannot be represented in the current code page (1252)
utf8_mode.msvc.output.good.cpp(24) : warning C4566: character represented by universal-character-name '\u0430' cannot be represented in the current code page (1252)

W:\examples> _

It might look as if these warnings are due to the active codepage in the console window, but they’re not related. Visual C++ is just complaining about information loss when it attempts to convert the narrowText literal from the source code’s UTF-8 to its incorrectly documented C++ execution character set, which is Windows ANSI. Where Windows ANSI is the codepage reported by the GetACP Windows API function (doc here), which defaults to the codepage specified in the ACP value of the as far as I know undocumented registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage.

So, the effect of compiling can be a bit different depending on the language that Windows is installed for, which determines the default GetACP codepage. But at least the compilation doesn’t depend on such an ephemeral setting as the active codepage in some console window! And these warnings are good: they’re spot on, for a change. :-)

Now, does the program work?

W:\examples> chcp 1252
Active code page: 1252

W:\examples> good
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> good >good_result

W:\examples> type good_result
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> chcp 65001
Active code page: 65001

W:\examples> good
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> type good_result
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> _

Yes! It manages to present correct output even with active codepage 1252 (Windows ANSI Western) because it uses direct console i/o for this case. And as you can see the redirected output is UTF-8, as it should be.

Even more goodness: the UTF-8 mode also works down at the C library level, using e.g. the wprintf function.

UTF-8 stream mode: the bad (input)

The bad news about the _O_U8TEXT mode is that input is almost as non-functional as with an active codepage 65001.

Apparently the runtime retrieves input as 1 byte per character via the standard input stream.Which means that the input is encoded according to the console window’s active codepage. These bytes, that e.g. encode your input text in Windows ANSI, are then interpreted as if they were UTF-8 encoded text.

The interpretation as UTF-8 would only be correct for active codepage 65001. But with active codepage 65001, which is the only one where the interpretation would be correct, the input operation just fails. So, depending on the active codepage the input operation will either produce garbage for non-ASCII characters, or it will fail outright.

[utf8_mode.msvc.input.bad.cpp]
#include <stdexcept>        // std::runtime_error, std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::wcout, std::wcerr, std::endl
#include <string>           // std::string, std::wstring
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

bool throwX( string const& s ) { throw runtime_error( s ); }
bool hopefully( bool v ) { return v; }

void setUtf8Mode( FILE* f, char const name[] )
{
    int const newMode = _setmode( _fileno( f ), _O_U8TEXT );
    hopefully( newMode != -1 )
        || throwX( string() + "setmode failed for " + name );
}

void initStreams()
{
    setUtf8Mode( stdin, "stdin" );
    setUtf8Mode( stdout, "stdout" );
}

wstring lineFrom( wistream& stream )
{
    wstring     result;
    
    getline( stream, result );
    hopefully( !stream.fail() )
        || throwX( "lineFrom: getline failed" );
    return result;
}

int main()
{
    try
    {
        initStreams();

        wcout << "What's your name? ";
        wstring const name = lineFrom( wcin );
        wcout << "Pleased to meet you, " << name << "!" << endl;
        
        int const n = name.length();
        wcout << endl;
        wcout << "I represented your name as " << n << " wide characters:" << endl;
        for( int i = 0;  i < n;  ++i )
        {
            if( i > 0 ) { wcout << " | "; }
            wcout << hex << 0 + name[i] << " '" << name[i] << "'";
        }
        wcout << endl;
       
        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        wcerr << "!" << x.what() << endl;
    }
    return EXIT_FAILURE;
}

Testing this:

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl utf8_mode.msvc.input.bad.cpp /Fe"bad_input"
utf8_mode.msvc.input.bad.cpp

W:\examples> bad_input
What's your name? Bjørn
Pleased to meet you, Bj�rn!

I represented your name as 5 wide characters:
42 'B' | 6a 'j' | fffd '�' | 72 'r' | 6e 'n'

W:\examples> type good_result
Narrow text: Blåbærsyltetøy! ??? ?????!
Wide text:   Blåbærsyltetøy! 日本国 кошка!

W:\examples> bad_input
What's your name? Blåbærsyltetøy
Pleased to meet you, Blåbærsyltetøy!

I represented your name as 14 wide characters:
42 'B' | 6c 'l' | e5 'å' | 62 'b' | e6 'æ' | 72 'r' | 73 's' | 79 'y' | 6c 'l' | 74 't' | 65 'e' | 74 't' | f8 'ø' | 79 'y'

W:\examples> _

The last run with the funny characters pasted as input shows that interactive input of the UTF-8 byte values works. The input byte values, that look pretty funny in the console, are the UTF-8 byte values that encode “Blåbærsyltetøy”. And with that as the cleartext result the received byte values must have been interpreted directly as UTF-8.

Instead the runtime should have used direct console input, just as it uses direct console output when the standard output stream is connected to a console window. Since it does not (or does not manage to do that correctly), that’s what we have to provide.

UTF-8 stream mode: the bad (input) – FIXED

It is apparently easy to check whether the standard input stream is connected to a console window, namely via the_isatty function (doc here). However, the documentation hints ominously about _isatty maybe returning true for a stream connected to a serial port, and archaic things like that. Happily one can alternatively use the lower level Windows API console functions, like e.g. GetConsoleMode (doc here), which presumably will only succeed for a stream that represents something that supports the Windows console functions.

[is_input_console.cpp]
#include <iostream>
#include <stdio.h>          // stdin, _fileno

#include <io.h>             // _isatty

#include <windows.h>        // GetConsoleMode

int main()
{
    DWORD   consoleMode;

    HANDLE const    inputHandle         = GetStdHandle( STD_INPUT_HANDLE );
    bool const      winapiSaysConsole   = !!GetConsoleMode( inputHandle, &consoleMode );
    bool const      clibSaysConsole     = !!_isatty( _fileno( stdin ) );
    
    using namespace std;
    cerr << boolalpha;
    cerr << "_isatty: " << clibSaysConsole << endl;
    cerr << "GetConsoleMode: " << winapiSaysConsole << endl;
}
W:\examples> cl is_input_console.cpp /Fe"x"
is_input_console.cpp

W:\examples> x
_isatty: true
GetConsoleMode: true

W:\examples> x <nul
_isatty: true
GetConsoleMode: false

W:\examples> x <is_input_console.cpp
_isatty: false
GetConsoleMode: false

W:\examples> _

It surprised me that _isatty here incorrectly identified the Windows nul device as a console window. However, GetConsoleMode got it right. So GetConsoleMode is evidently more reliable for this detection task.

Anyway, with a console identified as such, at the C++ level it’s then possible to override things in std::basic_streambuf (doc here), where the required core functionality maps almost directly to a call of the Windows API function ReadConsole (doc here).

The program below illustrates the technique by adding the necessary special input support to the previous section’s program, with main unchanged. This program is however not intended to provide directly reusable code. It’s just a problem-specific concrete example written in a reasuble-like style:

[utf8_mode.msvc.input.fixed.cpp]
#ifdef  _MSC_VER
#   pragma warning( disable: 4373 )     // "Your override overrides"
#endif

#include <algorithm>        // std::remove
#include <stddef.h>         // ptrdiff_t
#include <stdexcept>        // std::runtime_error, std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::wcout, std::wcerr, std::endl
#include <streambuf>        // std::basic_streambuf
#include <string>           // std::string, std::wstring
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

#undef      UNICODE
#define     UNICODE
#include <windows.h>    // ReadConsole


typedef ptrdiff_t       Size;

bool throwX( string const& s ) { throw runtime_error( s ); }
bool hopefully( bool v ) { return v; }

class DirectInputBuffer
    : public std::basic_streambuf< wchar_t >
{
private:
    wstring     buffer_;

    Size bufferSize() const         { return buffer_.size(); }
    wchar_t* pBufferStart()         { return &buffer_[0]; }
    wchar_t* pBufferEnd()           { return pBufferStart() + bufferSize(); }

    wchar_t* pStart() const         { return eback(); }
    wchar_t* pCurrent() const       { return gptr(); }
    wchar_t* pEnd() const           { return egptr(); }

    static HANDLE inputHandle()
    {
        static HANDLE const handle = GetStdHandle( STD_INPUT_HANDLE );
        return handle;
    }

public:
    typedef std::basic_streambuf< wchar_t >     Base;
    typedef Base::traits_type                   Traits;

    DirectInputBuffer( Base const& anOriginalBuffer )
        : Base( anOriginalBuffer )      // Copies buffer read area pointers.
        , buffer_( 256, L'#' )
    {}

protected:
    virtual streamsize xsgetn( wchar_t* const pBuffer, streamsize const n )
    {
        wchar_t const   ctrlZ   = wchar_t( 1 + ('Z' - 'A') );

        DWORD       nCharactersRead     = 0;

        bool const  readSucceeded       = !!ReadConsole(
            inputHandle(), pBuffer, static_cast< DWORD >( n ), &nCharactersRead, nullptr
            );

        if( readSucceeded )
        {
            wchar_t const* const    pCleanEnd   =
                remove( pBuffer, pBuffer + nCharactersRead, L'\r' );

            nCharactersRead = pCleanEnd - pBuffer;

            bool const isInteractiveEOF =
                (nCharactersRead == 2 && pBuffer[0] == ctrlZ && pBuffer[1] == '\n');
                
            return (isInteractiveEOF? 0 : static_cast< streamsize >( nCharactersRead ));
        }
        return 0;
    }
    
    virtual int_type underflow()
    {
        // Try to get some more input (maximum a line).
        if( pCurrent() == 0 || pCurrent() >= pEnd() )
        {
            streamsize const nCharactersRead =
                xsgetn( pBufferStart(), bufferSize() );

            if( nCharactersRead > 0 )
            {
                setg(
                    pBufferStart(),                     // Reading area start
                    pBufferStart(),                     // Reading area current
                    pBufferStart() + nCharactersRead    // Reading area end
                    );
            }
        }
        
        if( pCurrent() == 0 || pCurrent() >= pEnd() )
        {
            return Traits::eof();
        }
        return Traits::to_int_type( *pCurrent() );
    }
};

void setUtf8Mode( FILE* f, char const name[] )
{
    int const newMode = _setmode( _fileno( f ), _O_U8TEXT );
    hopefully( newMode != -1 )
        || throwX( string() + "setmode failed for " + name );
}

bool inputIsFromConsole()
{
    static HANDLE const inputHandle = GetStdHandle( STD_INPUT_HANDLE );

    DWORD consoleMode;
    return !!GetConsoleMode( inputHandle, &consoleMode );
}

void initStreams()
{
    if( inputIsFromConsole() )
    {
        static DirectInputBuffer buffer( *wcin.rdbuf() );
        wcin.rdbuf( &buffer );
    }
    setUtf8Mode( stdin, "stdin" );
    setUtf8Mode( stdout, "stdout" );
}

wstring lineFrom( wistream& stream )
{
    wstring     result;
    
    getline( stream, result );
    hopefully( !stream.fail() )
        || throwX( "lineFrom: getline failed" );
    return result;
}

int main()
{
    try
    {
        initStreams();

        wcout << "What's your name? ";
        wstring const name = lineFrom( wcin );
        wcout << "Pleased to meet you, " << name << "!" << endl;
        
        int const n = name.length();
        wcout << endl;
        wcout << "I represented your name as " << n << " wide characters:" << endl;
        for( int i = 0;  i < n;  ++i )
        {
            if( i > 0 ) { wcout << " | "; }
            wcout << hex << 0 + name[i] << " '" << name[i] << "'";
        }
        wcout << endl;
       
        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        wcerr << "!" << x.what() << endl;
    }
    return EXIT_FAILURE;
}
}

Testing this:

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl utf8_mode.msvc.input.fixed.cpp /Fe"fixed_input"
utf8_mode.msvc.input.fixed.cpp

W:\examples> fixed_input
What's your name? Bjørn
Pleased to meet you, Bjørn!

I represented your name as 5 wide characters:
42 'B' | 6a 'j' | f8 'ø' | 72 'r' | 6e 'n'

W:\examples> fixed_input
What's your name? 日本国 кошка
Pleased to meet you, 日本国 кошка!

I represented your name as 9 wide characters:
65e5 '日' | 672c '本' | 56fd '国' | 20 ' ' | 43a 'к' | 43e 'о' | 448 'ш' | 43a 'к' | 430 'а'

W:\examples> _

Yay! :-)

But I’d better mention again that in my English-language Windows 7 the console window stores the Chinese characters correctly but is only able to display them as rectangles, ▭. Correctly stored means that copy and paste works, which is why the text dumps above show these characters. However, the Norwegian and Russian characters are both stored and displayed correctly.

Also, I’d better mention that this is only a C++ level solution:

  • Input from the standard input stream should only be done via wcin.

Explained:

Input from the standard input stream should only be done at the C++ iostream level because in practice the UTF-8 mode input operation bugs can only be fixed at the C++ iostream level.

And input should only be done via the C++ wide stream wcin because, first of all, input data can be arbitrary, and secondly, with UTF-8 mode the narrow character operations just fail…

UTF-8 stream mode: the ugly (narrow streams)

C++11 (more precisely the N3290 final draft) §27.4.1/3:

  • “Mixing operations on corresponding wide- and narrow-character streams follows the same semantics as mixing such operations on FILEs, as specified in Amendment 1 of the ISO C standard.”

C99 (more precisely the N869 draft) §7.19.2/4:

  • “Each stream has an orientation. After a stream is associated with an external file, but before any operations are performed on it, the stream is without orientation. Once a wide-character input/output function has been applied to a stream without orientation, the stream becomes a wide-oriented stream. Similarly, once a byte input/output function has been applied to a stream without orientation, the stream becomes a byte-oriented stream. Only a call to the freopen function or the fwide function can otherwise alter the orientation of a stream. (A successful call to freopen removes any orientation.)”

C99 (more precisely the N869 draft) §7.19.2/5:

  • “Byte input/output functions shall not be applied to a wide-oriented stream and wide-character input/output functions shall not be applied to a byte-oriented stream.”

By these rules one would have to decide on using either cerr or wcerr in a program, but not both. That’s
not very practical, considering, for example, that one library component might produce error messages via cerr, while another might use wcerr for that purpose. But Visual C++ apparently follows a more practical-for-Windows-programming set of rules.

For Visual C++ 10.0 the fwide function is documented as being unimplemented. And from a practical point of view, at least at the level of outputting whole lines it apparently works fine to intermingle use of cout and wcout. So, happily, Visual C++ apparently just disregards the standard’s requirements and does not maintain an impractical explicit C FILE stream orientation.

Except … that with UTF-8 mode for the standard output stream, use of cout crashes the program!

[utf8_mode.msvc.byte_stream.ugly.cpp]
#include <stdexcept>        // std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <iostream>         // std::cout, std::endl
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

void initStreams()
{
    _setmode( _fileno( stdin ), _O_U8TEXT );
    _setmode( _fileno( stdout ), _O_U8TEXT );
    _setmode( _fileno( stderr ), _O_U8TEXT );
}

int main()
{
    try
    {
        initStreams();

        cout << "Hello, world!" << endl;
        cout << "Did you know, most Norwegians like 'blåbærsyltetøy'?" << endl;
        cerr << ":This is an error message, also mentioning 'blåbærsyltetøy'." << endl;

        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        cerr << "!" << x.what() << endl;
    }
    catch( ... )
    {
        cerr << "!Unknown exception." << endl;
    }
    return EXIT_FAILURE;
}

Screendump of crash dialogTesting:

W:\examples> ugly

W:\examples> _

The crash is caused by an assertion in the Visual C++ 10.0 fputc implementation, in source file [fputc.c]:

int __cdecl fputc (
        int ch,
        FILE *str
        )
{
    int retval=0;

    _VALIDATE_RETURN((str != NULL), EINVAL, EOF);

    _lock_str(str);
    __try {
        _VALIDATE_STREAM_ANSI_SETRET(str, EINVAL, retval, EOF);    // <-- Uh oh.

        if (retval==0)
        {
            retval = _putc_nolock(ch,str);
        }
    }
    __finally {
        _unlock_str(str);
    }

    return(retval);
}

where _VALIDATE_STREAM_ANSI_SETRET is defined thusly in [internal.h]:

/*
    We use _VALIDATE_STREAM_ANSI_SETRET to ensure that ANSI file operations(
    fprintf etc) aren't called on files opened as UNICODE. We do this check
    only if it's an actual FILE pointer &amp; not a string. It doesn't actually return
        immediately
*/

#define _VALIDATE_STREAM_ANSI_SETRET( stream, errorcode, retval, retexpr)            \
    {                                                                                \
        FILE *_Stream=stream;                                                        \
        int fn;                                                                      \
        _VALIDATE_SETRET(( (_Stream-&gt;_flag &amp; _IOSTRG) ||                             \
                           ( fn = _fileno(_Stream),                                  \
                             ( (_textmode_safe(fn) == __IOINFO_TM_ANSI) &amp;&amp;           \
                               !_tm_unicode_safe(fn)))),                             \
                         errorcode, retval, retexpr)                                 \
    }

I.e., with UTF-8 mode Microsoft’s code enforces the standard’s prohibition of mixing wide and narrow character operations on the same stream.

What to do, when the law is suddenly enforced?

UTF-8 stream mode: the ugly (narrow streams) – FIXED

Given the Windows convention of Windows ANSI encoding for internal char-based data, in particular for string literals, it does not make much sense to support cin input operations. With the user typing e.g. “кошка” the program running on a Norwegian machine would just produce something like “?????” as the Windows ANSI-encoded result. I.e. such narrow character input operations would replace a generally preferable hard crash with rather ungood data loss.

On the other hand, especially for students, existing library components may be logging error messages via narrow character operations on the standard error stream, like cerr << "!oops" << endl;. To avoid crashes for that, up at the C++ level one can install new iostream buffers for cout, cerr and clog, where those buffers do something reasonable such as forwarding all output to the wide streams (thereby providing Windows ANSI → UTF-8 translation). Down at the C level there is, however, no other practical crash-fix option than to leave the standard error stream in ANSI (untranslated) mode, so that the functionality down at the C level is much more restricted.

Back up at the C++ level again, this restriction for stderr can be compensated by adding UTF-8 conversion to the wide streams that write to the standard error stream, namely wcerr and wclog. One can and should also add custom direct console output support (more about that below). I.e., one is then essentially emulating the UTF-8 mode for wcerr and wclog, but only for operations at the C++ level.

This means that while the C++ level will/can work as expected, narrow character operations on stderr will only be guaranteed to produce the intended output when they’re limited to the ASCII character repertoire, and narrow character operations on stdin or stdout will crash.

Happily it’s very very easy to recognize a crash and/or garbage output, so that breaches of the programming conventions, …

  • use wcin for input,
     
  • don’t use C FILE level narrow characters operations on stdout, and
     
  • don’t use non-ASCII characters in narrow character operations on stderr

will likely be caught during testing.

Let’s start at the end of the internal processing, with the UTF-8 encoding support for wcerr.

With Visual C++ 10.0 the UTF-8 encoding itself is almost trivial, because first of all Visual C++ 10.0 supports the C++11 codecvt_utf8 facet, and secondly because Visual C++’s standard streams support the codecvt facet (the standard only requires file streams to support it), but as you can see below the console window result is not perfect!

[direct_display_troubles.cpp]
#include <codecvt>          // std::codecvt_utf8 (C++11)
#include <iostream>         // std::wcerr
#include <locale>           // std::locale
#include <memory>           // std::unique_ptr (C++11)
using namespace std;

void setUtf8Conversion( wostream& stream )
{
    typedef codecvt_utf8< wchar_t >     CodeCvt;

    unique_ptr< CodeCvt > pCodeCvt( new CodeCvt );
    locale  utf8Locale( locale(), pCodeCvt.get() );
    pCodeCvt.release();
    stream.imbue( utf8Locale );
}

int main()
{
    wcerr << "'blåbærsyltetøy' with default presentation." << endl;
    
    setUtf8Conversion( wcerr );
    wcerr << "'blåbærsyltetøy' with UTF-8 conversion applied." << endl;
}

Testing this:

W:\examples> chcp 1252

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl direct_display_troubles.cpp /Fe"x"
direct_display_troubles.cpp

W:\examples> x
'blåbærsyltetøy' with default presentation.
'blåbærsyltetøy' with UTF-8 conversion applied.

W:\examples> chcp 65001
Active code page: 65001

W:\examples> x
'bl�b�rsyltet�y' with default presentation.
'bl��b��rsyltet��y' with UTF-8 conversion applied.

W:\examples> (x 2>&1) >result

W:\examples> type result
'bl�b�rsyltet�y' with default presentation.
'blåbærsyltetøy' with UTF-8 conversion applied.

W:\examples> _

As you can see, the Visual C++ runtime library’ direct console output kicks in even without having set UTF-8 mode, and in some inexplicable way presents the resulting UTF-8 bytes – as garbage.

I have no idea how that happens.

However, one cure is to override it with a custom direct console output:

[direct_display_troubles.fixed.cpp]
#ifdef  _MSC_VER
#   pragma warning( disable: 4373 )     // "Your override overrides"
#endif

#include <assert.h>         // assert
#include <codecvt>          // std::codecvt_utf8 (C++11)
#include <iostream>         // std::wcerr
#include <locale>           // std::locale
#include <memory>           // std::unique_ptr (C++11)
#include <streambuf>        // std::basic_streambuf
using namespace std;

#undef  UNICODE
#define UNICODE
#undef  STRICT
#define STRING
#include <windows.h>    // GetStdHandle, GetConsoleMode, WriteConsole


template< class CharType >
class AbstractOutputBuffer
    : public basic_streambuf< CharType >
{
public:
    typedef basic_streambuf< CharType >     Base;
    typedef typename Base::traits_type      Traits;

    typedef Base                            StreamBuffer;

protected:
    virtual streamsize xsputn( char_type const* const s, streamsize const n ) = 0;

    virtual int_type overflow( int_type const c )
    {
        bool const cIsEOF   = Traits::eq_int_type( c, Traits::eof() );
        int_type const  failureValue    = Traits::eof();
        int_type const  successValue    = (cIsEOF? Traits::not_eof( c ) : c);

        if( !cIsEOF )
        {
            char_type const     ch                  = Traits::to_char_type( c );
            streamsize const    nCharactersWritten  = xsputn( &ch, 1 );

            return (nCharactersWritten == 1? successValue : failureValue);
        }
        return successValue;
    }

public:
    AbstractOutputBuffer()
    {}

    AbstractOutputBuffer( StreamBuffer& existingBuffer )
        : Base( existingBuffer )
    {}
};


class DirectOutputBuffer
    : public AbstractOutputBuffer< wchar_t >
{
public:
    enum StreamId { outputStreamId, errorStreamId, logStreamId };

private:
    StreamId    streamId_;

protected:
    virtual streamsize xsputn( wchar_t const* const s, streamsize const n )
    {
        static HANDLE const outputStreamHandle  = GetStdHandle( STD_OUTPUT_HANDLE );
        static HANDLE const errorStreamHandle   = GetStdHandle( STD_ERROR_HANDLE );

        HANDLE const    streamHandle    =
            (streamId_ == outputStreamId? outputStreamHandle : errorStreamHandle );
        
        DWORD nCharactersWritten    = 0;
        bool writeSucceeded         = !!WriteConsole(
            streamHandle, s, static_cast< DWORD >( n ), &nCharactersWritten, 0
            );
        return (writeSucceeded? static_cast< streamsize >( nCharactersWritten ) : 0);
    }

public:
    DirectOutputBuffer( StreamId streamId = outputStreamId )
        : streamId_( streamId )
    {}
};


void setUtf8Conversion( wostream& stream )
{
    typedef codecvt_utf8< wchar_t >     CodeCvt;

    unique_ptr< CodeCvt > pCodeCvt( new CodeCvt );
    locale  utf8Locale( locale(), pCodeCvt.get() );
    pCodeCvt.release();
    stream.imbue( utf8Locale );
}

bool isConsole( HANDLE streamHandle )
{
    DWORD consoleMode;
    return !!GetConsoleMode( streamHandle, &consoleMode );
}

bool isConsole( DWORD stdStreamId )
{
    return isConsole( GetStdHandle( stdStreamId ) );
}

void setDirectOutputSupport( wostream& stream )
{
    typedef DirectOutputBuffer  DOB;

    if( &stream == &wcout )
    {
        if( isConsole( STD_OUTPUT_HANDLE ) )
        {
            static DOB  outputStreamBuffer( DOB::outputStreamId );
            stream.rdbuf( &outputStreamBuffer );
        }
    }
    else if( &stream == &wcerr )
    {
        if( isConsole( STD_ERROR_HANDLE ) )
        {
            static DOB errorStreamBuffer( DOB::errorStreamId );
            stream.rdbuf( &errorStreamBuffer );
        }
    }
    else if( &stream == &wclog )
    {
        if( isConsole( STD_ERROR_HANDLE ) )
        {
            static DOB logStreamBuffer( DOB::logStreamId );
            stream.rdbuf( &logStreamBuffer );
        }
    }
    else
    {
        assert(( "setDirectOutputSupport: unsupported stream", false ));
    }
}

int main()
{
    wcerr << "'blåbærsyltetøy' with default presentation." << endl;
    
    setUtf8Conversion( wcerr );
    wcerr << "'blåbærsyltetøy' with UTF-8 conversion applied." << endl;
    
    setDirectOutputSupport( wcerr );
    wcerr << "'blåbærsyltetøy' with UTF-8 conversion & direct output applied." << endl;
}

Testing this:

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl direct_display_troubles.fixed.cpp /Fe"x"
direct_display_troubles.fixed.cpp

W:\examples> x
'blåbærsyltetøy' with default presentation.
'blåbærsyltetøy' with UTF-8 conversion applied.
'blåbærsyltetøy' with UTF-8 conversion & direct output applied.

W:\examples> chcp 65001
Active code page: 65001

W:\examples> x
'bl�b�rsyltet�y' with default presentation.
'bl��b��rsyltet��y' with UTF-8 conversion applied.
'blåbærsyltetøy' with UTF-8 conversion & direct output applied.

W:\examples> (x 2>&1) >result

W:\examples> type result
'bl�b�rsyltet�y' with default presentation.
'blåbærsyltetøy' with UTF-8 conversion applied.
'blåbærsyltetøy' with UTF-8 conversion & direct output applied.

W:\examples> _

Yay! Now we’re ready for tackling the previous section’s program. Again, the main function is exactly as before:

[utf8_mode.msvc.byte_stream.fixed.cpp]
#ifdef  _MSC_VER
#   pragma warning( disable: 4373 )     // "Your override overrides"
#endif

#include <assert.h>         // assert
#include <codecvt>          // std::codecvt_utf8 (C++11)
#include <stdexcept>        // std::exception
#include <stdlib.h>         // EXIT_SUCCESS, EXIT_FAILURE
#include <streambuf>        // std::basic_streambuf
#include <string>           // wstring
#include <iostream>         // std::cout, std::endl
#include <locale>           // std::locale
#include <memory>           // std::unique_ptr (C++11)
using namespace std;

#include    <io.h>      // _setmode, _fileno
#include    <fcntl.h>   // _O_U8TEXT

#undef  UNICODE
#define UNICODE
#undef  STRICT
#define STRING
#include <windows.h>    // MultiByteToWideChar


template< class CharType >
class AbstractOutputBuffer
    : public basic_streambuf< CharType >
{
public:
    typedef basic_streambuf< CharType >     Base;
    typedef typename Base::traits_type      Traits;

    typedef Base                            StreamBuffer;

protected:
    virtual streamsize xsputn( char_type const* const s, streamsize const n ) = 0;

    virtual int_type overflow( int_type const c )
    {
        bool const cIsEOF   = Traits::eq_int_type( c, Traits::eof() );
        int_type const  failureValue    = Traits::eof();
        int_type const  successValue    = (cIsEOF? Traits::not_eof( c ) : c);

        if( !cIsEOF )
        {
            char_type const     ch                  = Traits::to_char_type( c );
            streamsize const    nCharactersWritten  = xsputn( &ch, 1 );

            return (nCharactersWritten == 1? successValue : failureValue);
        }
        return successValue;
    }

public:
    AbstractOutputBuffer()
    {}

    AbstractOutputBuffer( StreamBuffer& existingBuffer )
        : Base( existingBuffer )
    {}
};


class OutputForwarderBuffer
    : public AbstractOutputBuffer< char >
{
public:
    typedef AbstractOutputBuffer< char >    Base;
    typedef Base::Traits                    Traits;
    
    typedef Base::StreamBuffer              StreamBuffer;
    typedef basic_streambuf<wchar_t>        WideStreamBuffer;

private:
    WideStreamBuffer*       pWideStreamBuffer_;
    wstring                 wideCharBuffer_;

    OutputForwarderBuffer( OutputForwarderBuffer const& );      // No such.
    void operator=( OutputForwarderBuffer const& );             // No such.

protected:
    virtual streamsize xsputn( char const* const s, streamsize const n )
    {
        if( n == 0 ) { return 0; }

        int const   nAsInt  = static_cast<int>( n );    //  Visual C++ sillywarnings.
        wideCharBuffer_.resize( nAsInt );
        int const nWideCharacters = MultiByteToWideChar(
            CP_ACP,             // Windows ANSI
            MB_PRECOMPOSED,     // Always precompose characters (this is the default).
            s,                  // Narrow character string.
            nAsInt,             // Number of bytes in narrow character string.
            &wideCharBuffer_[0],
            nAsInt              // Wide char buffer size.
            );
        assert( nWideCharacters > 0 );
        return pWideStreamBuffer_->sputn( &wideCharBuffer_[0], nWideCharacters );
    }

public:
    OutputForwarderBuffer(
        StreamBuffer&       existingBuffer,
        WideStreamBuffer*   pWideStreamBuffer
        )
        : Base( existingBuffer )
        , pWideStreamBuffer_( pWideStreamBuffer )
    {}
};


class DirectOutputBuffer
    : public AbstractOutputBuffer< wchar_t >
{
public:
    enum StreamId { outputStreamId, errorStreamId, logStreamId };

private:
    StreamId    streamId_;

protected:
    virtual streamsize xsputn( wchar_t const* const s, streamsize const n )
    {
        static HANDLE const outputStreamHandle  = GetStdHandle( STD_OUTPUT_HANDLE );
        static HANDLE const errorStreamHandle   = GetStdHandle( STD_ERROR_HANDLE );

        HANDLE const    streamHandle    =
            (streamId_ == outputStreamId? outputStreamHandle : errorStreamHandle );
        
        DWORD nCharactersWritten    = 0;
        bool writeSucceeded         = !!WriteConsole(
            streamHandle, s, static_cast< DWORD >( n ), &nCharactersWritten, 0
            );
        return (writeSucceeded? static_cast< streamsize >( nCharactersWritten ) : 0);
    }

public:
    DirectOutputBuffer( StreamId streamId = outputStreamId )
        : streamId_( streamId )
    {}
};


void setUtf8Conversion( wostream& stream )
{
    typedef codecvt_utf8< wchar_t >     CodeCvt;

    unique_ptr< CodeCvt > pCodeCvt( new CodeCvt );
    locale  utf8Locale( locale(), pCodeCvt.get() );
    pCodeCvt.release();
    stream.imbue( utf8Locale );
}

bool isConsole( HANDLE streamHandle )
{
    DWORD consoleMode;
    return !!GetConsoleMode( streamHandle, &consoleMode );
}

bool isConsole( DWORD stdStreamId )
{
    return isConsole( GetStdHandle( stdStreamId ) );
}

void setDirectOutputSupport( wostream& stream )
{
    typedef DirectOutputBuffer  DOB;

    if( &stream == &wcout )
    {
        if( isConsole( STD_OUTPUT_HANDLE ) )
        {
            static DOB  outputStreamBuffer( DOB::outputStreamId );
            stream.rdbuf( &outputStreamBuffer );
        }
    }
    else if( &stream == &wcerr )
    {
        if( isConsole( STD_ERROR_HANDLE ) )
        {
            static DOB errorStreamBuffer( DOB::errorStreamId );
            stream.rdbuf( &errorStreamBuffer );
        }
    }
    else if( &stream == &wclog )
    {
        if( isConsole( STD_ERROR_HANDLE ) )
        {
            static DOB logStreamBuffer( DOB::logStreamId );
            stream.rdbuf( &logStreamBuffer );
        }
    }
    else
    {
        assert(( "setDirectOutputSupport: unsupported stream", false ));
    }
}

void initStreams()
{
    // Set up UTF-8 conversions &  direct console output:
    _setmode( _fileno( stdin ), _O_U8TEXT );
    _setmode( _fileno( stdout ), _O_U8TEXT );
    setUtf8Conversion( wcerr );  setDirectOutputSupport( wcerr );
    setUtf8Conversion( wclog );  setDirectOutputSupport( wclog );

    // Forward narrow character output to the wide streams:
    static OutputForwarderBuffer    coutBuffer( *cout.rdbuf(), wcout.rdbuf() );
    static OutputForwarderBuffer    cerrBuffer( *cerr.rdbuf(), wcerr.rdbuf() );
    static OutputForwarderBuffer    clogBuffer( *clog.rdbuf(), wclog.rdbuf() );

    cout.rdbuf( &coutBuffer );
    cerr.rdbuf( &cerrBuffer );
    clog.rdbuf( &clogBuffer );
}

int main()
{
    try
    {
        initStreams();

        cout << "Hello, world!" << endl;
        cout << "Did you know, most Norwegians like 'blåbærsyltetøy'?" << endl;
        cerr << ":This is an error message, also mentioning 'blåbærsyltetøy'." << endl;

        return EXIT_SUCCESS;
    }
    catch( exception const& x )
    {
        cerr << "!" << x.what() << endl;
    }
    catch( ... )
    {
        cerr << "!Unknown exception." << endl;
    }
    return EXIT_FAILURE;
}

Testing this:

W:\examples> chcp 1252
Active code page: 1252

W:\examples> cl utf8_mode.msvc.byte_stream.fixed.cpp /Fe"x"
utf8_mode.msvc.byte_stream.fixed.cpp

W:\examples> x
Hello, world!
Did you know, most Norwegians like 'blåbærsyltetøy'?
:This is an error message, also mentioning 'blåbærsyltetøy'.

W:\examples> chcp 65001
Active code page: 65001

W:\examples> x
Hello, world!
Did you know, most Norwegians like 'blåbærsyltetøy'?
:This is an error message, also mentioning 'blåbærsyltetøy'.

W:\examples> (x 2>&1) >result

W:\examples> type result
Hello, world!
Did you know, most Norwegians like 'blåbærsyltetøy'?
:This is an error message, also mentioning 'blåbærsyltetøy'.

W:\examples> _

Summary

To make the Visual C++ _O_U8TEXT UTF-8 stream mode work in general, we had to

  • fix the input-from-console functionality by adding a special direct console input buffer in the C++ level wcin stream,
     
  • avoid crashes for narrow character operations on the standard error stream by keeping that stream in ANSI mode, installing forwarder buffers for cout and cerr, installing an UTF-8 conversion facet in wcerr, and adding direct console output support to wcerr, and
     
  • foresake the use of input and narrow character operations at the C level, except for the standard error stream.

With all this bug-fixing and support machinery added, it’s almost as if we had to implement the _O_U8TEXT mode from scratch! What does it really buy, then? What is the point of using that mode?

Well, essentially the _O_U8TEXT UTF-8 mode gives conversion to UTF-8 for C level wide character output operations on the standard output stream. It does not do interactive input, and it can not be reasonably used for the standard error stream. One might therefore say that Microsoft Unicode guru Michael Kaplan’s original blog posting about this mode, where it appeared to a simple general solution on its own, was a bit too optimistic!

Anyway, to be utterly clear, with this approach one has three main text encodings to deal with in Windows, and three main text encodings to deal with in a typical Linux installation:

Internal wchar_t data: Internal char data: External data:
Windows: UTF-16 Windows ANSI UTF-8
Linux: Usually UTF-32 but also UTF-16 UTF-8 UTF-8

And the main forces in play yielding the Windows row of the above table, are that …

  • the core Windows API is UTF-16 based, and wchar_t is 16 bits in Windows,
     
  • Visual C++ has Windows ANSI as its C++ execution character set, which e.g. forces ordinary narrow character literals to Windows ANSI encoding even when the source code is UTF-8, and
     
  • one desires UTF-8 for external data both to avoid data loss and for general interoperability.

There is not much that can be done about these forces, so the basic approach to deal with these issues is either very similar to what I have described here, or will otherwise involve pretty painful trade-offs.

Not to say, of course, that the trade-offs in this posting’s approach aren’t a little painful! :-)

Cheers, & enjoy!

2010 in review

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

Healthy blog!

The Blog-Health-o-Meter™ reads Wow.

Crunchy numbers

Featured image

A helper monkey made this abstract painting, inspired by your stats.

A Boeing 747-400 passenger jet can hold 416 passengers. This blog was viewed about 8,400 times in 2010. That’s about 20 full 747s.

In 2010, there were 40 new posts, not bad for the first year! There were 8 pictures uploaded, taking up a total of 304kb.

The busiest day of the year was May 7th with 182 views. The most popular post that day was [cppx] Compiler-specific functionality abstracted.

Where did they come from?

The top referring sites in 2010 were stackoverflow.com, freak.no, thechaosengine.com, gennaroprota.wordpress.com, and healthfitnesstherapy.com.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

1

[cppx] Compiler-specific functionality abstracted May 2010
3 comments

2

[cppx] How to avoid disastrous integer wrap-around May 2010
10 comments

3

[cppx] 3 ways to mix in a generic cloning implementation June 2010

4

[cppx] B true, or B thrown! (Using the >> throwing pattern) July 2010
7 comments

5

[cppx] How to do typed optional arguments in C++98 May 2010

Errata #1 for Norw. intro to C++ programming

The Norwegian introduction to C++ programming (a bit Windows-specific) is at Google Docs, in PDF format, 3 chapters so far:

Introduksjon til C++-programmering (Windows)

Each file has a nice table of contents but for that you need to download the PDF and view it in e.g. Foxit or Adobe Acrobat. Ch 1 is just 1 page, though. Ch 2 is more pages. And so is ch 3.

Perhaps it’ll become a book…

Errata: I discovered that I’d badmouthed MinGW g++. I stated that it still didn’t support std::wostringstream & friends in version 4.1.1. That was incorrect.

The truth is that it didn’t support std::wostringstream & friends in version 3.4.5.

Now that is an old version, but it’s still the version that you get via the MinGW installer. To get a later version you can either download a build from Someone Else, such as the TDM build, or you can try to download the various individual package files from MinGW (I did that for version 4.5.0, it was a lot of work). So I guess an ordinary novice, the kind of person most likely to use MinGW g++, is mostly stuck with version 3.4.5.

Anyway, toolwise the intro is based on Visual C++ 10.0, and I have updated the ch 3 PDF at Google Docs so that it now says 3.4.5, not 4.4.1. :-)

– Alf

Current TOC for my Norwegian intro to C++

About the Norwegian C++ intro, see my earlier posting.

Not sure if this works or not, but I’m trying to embed a PDF of a Table of Contents generated by Word:

Enjoy! :-)  [Possibly/probably more to come, after all, I’m referring to chapter 4!]

– Alf

By the way, Olve, as you can see I’ve now added a chapter 3! Not quite at 42 yet… But.