Characters, Unicode and Encoding

Converting Between Character Encodings

How can I convert between different character encodings in C++?

Abstract art representing computer programming

Converting between character encodings in C++ can be a bit tricky, as the standard library doesn't provide built-in functions for this purpose. However, we can use third-party libraries or platform-specific APIs to accomplish this task. Here's a guide on how to approach this:

Using a Third-Party Library

One popular library for handling character encoding conversions is ICU (International Components for Unicode). Here's an example of how you might use ICU to convert from UTF-8 to UTF-16:

1#include <unicode/ucnv.h>
2#include <unicode/unistr.h>
3
4#include <iostream>
5#include <string>
6
7int main() {
8  std::string utf8String = "Hello, 世界!";
9
10  // Convert UTF-8 to UTF-16
11  icu::UnicodeString utf16String =
12    icu::UnicodeString::fromUTF8(utf8String);
13
14  // Convert back to UTF-8 for display
15  std::string convertedString;
16  utf16String.toUTF8String(convertedString);
17
18  std::cout << "Original: "
19    << utf8String << '\n';
20  std::cout << "Converted: "
21    << convertedString << '\n';
22}

1Original: Hello, 世界!
2Converted: Hello, 世界!

Using Platform-Specific APIs

On Windows, you can use the Windows API functions for encoding conversion. Here's an example that converts from UTF-8 to UTF-16:

1#include <windows.h>
2#include <iostream>
3#include <string>
4
5int main() {
6  std::string utf8String = "Hello, 世界!";
7
8  // Get the required buffer size
9  int size = MultiByteToWideChar(
10    CP_UTF8, 0, utf8String.c_str(),
11    -1, nullptr, 0
12  );
13
14  // Allocate the buffer
15  std::wstring utf16String(size, 0);
16
17  // Perform the conversion
18  MultiByteToWideChar(
19    CP_UTF8, 0, utf8String.c_str(),
20    -1, &utf16String[0], size);
21
22  // Convert back to UTF-8 for display
23  size = WideCharToMultiByte(
24    CP_UTF8, 0, utf16String.c_str(), -1,
25    nullptr, 0, nullptr, nullptr
26  );
27
28  std::string convertedString(size, 0);
29  WideCharToMultiByte(
30    CP_UTF8, 0, utf16String.c_str(), -1,
31    &convertedString[0], size, nullptr, nullptr
32  );
33
34  // Set console output to UTF-8
35  SetConsoleOutputCP(CP_UTF8);
36
37  std::cout << "Original: "
38    << utf8String << '\n';
39  std::cout << "Converted: "
40    << convertedString << '\n';
41}

1Original: Hello, 世界!
2Converted: Hello, 世界!

Remember, when working with different encodings, it's crucial to be aware of the encoding of your source files and how your compiler interprets string literals. Always use libraries or APIs that are well-tested and widely used to avoid potential encoding errors.

This Question is from the Lesson: