Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

utf8::utf16to8 and utf8::unchecked::utf16to8 produce different output #111

Closed
GPUex opened this issue Oct 27, 2023 · 2 comments
Closed

utf8::utf16to8 and utf8::unchecked::utf16to8 produce different output #111

GPUex opened this issue Oct 27, 2023 · 2 comments

Comments

@GPUex
Copy link

GPUex commented Oct 27, 2023

As title mentions, checked and unchecked variations of utf16to8 produce different output.
Simple test case:

#include <iostream>
#include <format>
#include "utf8.h"

std::string u16_to_u8_unchecked(const std::u16string& u16)
{
	std::string str;
	utf8::unchecked::utf16to8(u16.begin(), u16.end(), std::back_inserter(str));
	return str;
}

int main()
{
	std::u16string u16 = u"Simple test!";

	std::string u8_checked = utf8::utf16to8(u16);
	std::string u8_unchecked = u16_to_u8_unchecked(u16);

	std::cout << std::format("{}\n{}\n", u8_checked, u8_unchecked);
	std::cout << std::format("{} {} {}\n", u8_checked == u8_unchecked, u8_checked.size(), u8_unchecked.size());

	return 0;
}

Produces this output:

Simple test!
Simple test
false 12 11

Library version 4.0
Platform:
Windows 11 (10.0.22621.2428)
Visual Studio 2022 (17.7.6) (compiler version 19.37.32825.0)

@nemtrif
Copy link
Owner

nemtrif commented Oct 28, 2023

Ugh, yes. That was introduced by my fix of issue #78 . Looking at it now...

nemtrif added a commit that referenced this issue Oct 28, 2023
Fixing regression caused by the fix for #78, which leads to utf8::unchecked::utf16to8() chopping off the last character in many cases.
@nemtrif
Copy link
Owner

nemtrif commented Oct 28, 2023

Fixed at 4.0.1. Thanks!

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants