ChefZ -- 磨劍錄 (A Coder's Log)

Boost:UTF-8 Codecvt Facet(unicode 和 utf-8 之间相互转码) [转贴]

Examples of fixed-width encoding forms:

Type	Each character encoded as	Notes
7-bit	a single 7-bit quantity	example: ISO 646
8-bit G0/G1	a single 8-bit quantity	with constraints on use of C0 and C1 spaces
8-bit	a single 8-bit quantity	with no constraints on use of C1 space
8-bit EBCDIC	a single 8-bit quantity	with the EBCDIC conventions rather than ASCII conventions
16-bit (UCS-2)	a single 16-bit quantity	within a code space of 0..FFFF
32-bit (UCS-4)	a single 32-bit quantity	within a code space 0..7FFFFFFF
32-bit (UTF-32)	a single 32-bit quantity	within a code space of 0..10FFFF
16-bit DBCS process code	a single 16-bit quantity	example: UNIX widechar implementations of Asian CCS's
32-bit DBCS process code	a single 32-bit quantity	example: UNIX widechar implementations of Asian CCS's
DBCS Host	two 8-bit quantities	following IBM host conventions

Examples of variable-width encoding forms:

Name	Characters are encoded as	Notes
UTF-8	a mix of one to four 8-bit code units in Unicode and one to six code units in 10646	used only with Unicode/10646
UTF-16	a mix of one to two 16 bit code units	used only with Unicode/10646

Boost中提供了一个UTF-8 Codecvt Facet,可以在utf8和UCS-4(Unicode-32)之间转换.
使用方式如下

  //...
  // My encoding type
  typedef wchar_t ucs4_t;

  std::locale old_locale;
  std::locale utf8_locale(old_locale,new utf8_codecvt_facet<ucs4_t>);

  // Set a New global locale
  std::locale::global(utf8_locale);

  //  UCS-4 转换为 UTF-8
  {
    std::wofstream ofs("data.ucd");
    ofs.imbue(utf8_locale);
    std::copy(ucs4_data.begin(),ucs4_data.end(),
          std::ostream_iterator<ucs4_t,ucs4_t>(ofs));
  }

  // 读入 UTF-8 ,转换为 UCS-4 
  std::vector<ucs4_t> from_file;
  {
    std::wifstream ifs("data.ucd");
    ifs.imbue(utf8_locale);
    ucs4_t item = 0;
    while (ifs >> item) from_file.push_back(item);
  }
  //...
UTF-8 Codecvt Facet详见
http://www.boost.org/libs/serialization/doc/codecvt.html