Characters, Unicode and Encoding

Cross-Platform Unicode Support in C++

How do I implement proper Unicode support in a cross-platform C++ application?

Abstract art representing computer programming

Implementing proper Unicode support in a cross-platform C++ application can be challenging due to differences in how various operating systems and compilers handle Unicode. Here's a guide on how to approach this:

Use UTF-8 Everywhere

UTF-8 is widely supported and provides good compatibility across platforms. Use it as your default encoding:

1#include <iostream>
2#include <string>
3
4#ifdef _WIN32
5#include <Windows.h>
6#endif
7
8// Function to convert a char8_t string literal to
9// std::string
10std::string convert_utf8(const char8_t* utf8_text) {
11  return std::string(
12      reinterpret_cast<const char*>(utf8_text));
13}
14
15int main() {
16// Set the console output code page to
17// UTF-8 on Windows
18#ifdef _WIN32
19  SetConsoleOutputCP(CP_UTF8);
20#endif
21
22  // UTF-8 string literal
23  std::string text = convert_utf8(u8"Hello, 世界!");
24  std::cout << text;
25}

Platform-Specific Considerations

Windows uses UTF-16 internally. You may need to convert between UTF-8 and UTF-16:

1#include <iostream>
2#include <string>
3
4#ifdef _WIN32
5#include <Windows.h>
6#endif
7
8std::wstring utf8ToUtf16(const std::string& utf8) {
9#ifdef _WIN32
10  int size = MultiByteToWideChar(
11    CP_UTF8, 0, utf8.c_str(), -1, nullptr, 0
12  );
13  std::wstring utf16(size, 0);
14  MultiByteToWideChar(
15    CP_UTF8, 0, utf8.c_str(), -1, &utf16[0], size
16  );
17  return utf16;
18#else
19  // On non-Windows platforms, this function
20  // would need a different implementation
21  return std::wstring(utf8.begin(), utf8.end());
22#endif
23}
24
25std::string utf16ToUtf8(const std::wstring& utf16) {
26#ifdef _WIN32
27  int size = WideCharToMultiByte(
28    CP_UTF8, 0, utf16.c_str(), -1,
29    nullptr, 0, nullptr, nullptr
30  );
31  std::string utf8(size, 0);
32  WideCharToMultiByte(
33    CP_UTF8, 0, utf16.c_str(), -1,
34    &utf8[0], size, nullptr, nullptr
35  );
36  return utf8;
37#else
38  // On non-Windows platforms, this function 
39  // would need a different implementation
40  return std::string(utf16.begin(), utf16.end());
41#endif
42}
43
44int main() {
45  // Convert u8 string literal to std::string
46  const char8_t* utf8_literal = u8"Hello, 世界!";
47  std::string utf8(
48    reinterpret_cast<const char*>(utf8_literal));
49
50#ifdef _WIN32
51  // Set the console output code page to
52  // UTF-8 on Windows
53  SetConsoleOutputCP(CP_UTF8);
54#endif
55
56  std::wstring utf16 = utf8ToUtf16(utf8);
57
58#ifdef _WIN32
59  // Use UTF-16 for Windows API calls
60  MessageBoxW(
61    nullptr, utf16.c_str(),
62    L"Unicode Test", MB_OK
63  );
64#endif
65
66  // Convert back to UTF-8 for cross-platform code
67  std::string backToUtf8 = utf16ToUtf8(utf16);
68  std::cout << backToUtf8;
69}

1Hello, 世界!

Most Unix-like systems use UTF-8 by default. Ensure your locale is set correctly:

1#include <iostream>
2#include <locale>
3#include <codecvt>
4
5int main() {
6  // Set the global locale to UTF-8
7  std::locale::global(std::locale("en_US.UTF-8"));
8  // Imbue std::wcout with the global locale
9  std::wcout.imbue(std::locale());
10
11  // Output wide string literals
12  std::wcout << L"Hello, 世界!" << L'\n';
13}

1Hello, 世界!

Use a Cross-Platform Library

Consider using a library like ICU (International Components for Unicode) for consistent Unicode handling across platforms:

1#include <iostream>
2#include <unicode/unistr.h>
3#include <unicode/ustream.h>
4
5int main() {
6  icu::UnicodeString ustr =
7    icu::UnicodeString::fromUTF8("Hello, 世界!");
8  std::cout << ustr << '\n';
9}

File I/O

When reading or writing files, always specify the encoding:

1#include <iostream>
2#include <fstream>
3#include <string>
4#include <windows.h>
5#include <fcntl.h>
6#include <io.h>
7
8void WriteUTF8File(const std::wstring& fileName,
9                   const std::wstring& content) {
10  std::ofstream outFile(fileName, std::ios::binary);
11  if (!outFile.is_open()) {
12    std::wcerr
13      << L"Failed to open file for writing.\n";
14    return;
15  }
16  // Convert wide string to UTF-8
17  int size_needed =
18    WideCharToMultiByte(
19      CP_UTF8, 0, content.c_str(), -1,
20      nullptr, 0, nullptr, nullptr
21    );
22  std::string utf8Str(size_needed, 0);
23  WideCharToMultiByte(
24    CP_UTF8, 0, content.c_str(), -1, &utf8Str[0],
25    size_needed, nullptr, nullptr);
26  // Write to file
27  outFile.write(
28    utf8Str.c_str(),
29    utf8Str.size() - 1 // Exclude null terminator
30  ); 
31  outFile.close();
32}
33
34std::wstring ReadUTF8File(
35  const std::wstring& fileName
36) {
37  std::ifstream inFile(fileName, std::ios::binary);
38  if (!inFile.is_open()) {
39    std::wcerr
40      << L"Failed to open file for reading.\n";
41    return L"";
42  }
43  // Read file content into a string
44  std::string utf8Str(
45    (std::istreambuf_iterator<char>(inFile)),
46    std::istreambuf_iterator<char>());
47  inFile.close();
48  // Convert UTF-8 string to wide string
49  int size_needed = MultiByteToWideChar(
50    CP_UTF8, 0, utf8Str.c_str(), -1, nullptr, 0);
51  std::wstring wideStr(size_needed, 0);
52  MultiByteToWideChar(
53    CP_UTF8, 0, utf8Str.c_str(), -1,
54    &wideStr[0], size_needed);
55  return wideStr;
56}
57
58int main() {
59  // Set console output to UTF-8
60  SetConsoleOutputCP(CP_UTF8);
61
62  // Enable UTF-8 mode for C++ streams
63  _setmode(_fileno(stdout), _O_U8TEXT);
64  _setmode(_fileno(stderr), _O_U8TEXT);
65
66  std::wstring fileName = L"test.txt";
67  std::wstring content = L"Hello, 世界!";
68
69  // Writing UTF-8
70  WriteUTF8File(fileName, content);
71
72  // Reading UTF-8
73  std::wstring readContent =
74    ReadUTF8File(fileName);
75
76  std::wcout << L"Content read from file: "
77    << readContent << std::endl;
78}

1Content read from file: Hello, 世界!

Best Practices

Use UTF-8: Use UTF-8 as your default encoding for strings and files.
Platform-Specific Code: Isolate platform-specific Unicode handling in separate functions or classes.
Consistent Internal Representation: Choose a consistent internal representation (e.g., UTF-8 or UTF-16) and convert at system boundaries.
Avoid Assumptions: Don't assume one character equals one byte or one display width.
Test Thoroughly: Test your application with a wide range of Unicode characters on all target platforms.
Handle Input/Output Carefully: Be mindful of how different systems handle Unicode input and output.
Use Modern C++ Features: Leverage C++11 and later features for Unicode support, like the u8 string literal prefix.

By following these guidelines and using appropriate libraries, you can create C++ applications with robust Unicode support that work consistently across different platforms.

This Question is from the Lesson: