Flutter - Character encoding is not behaving as expected - flutter

I am parsing a local JSON file that has words containing rarely used special characters in Icelandic.
When displaying the characters I get mixed up symbols but not the characters, for some others I just get a square instead of a symbol.
I am using this type of encoding "\u00c3"
Update: Example of the characters I am using: þ, æ, ý, ð
Q: What is the best way to display those kind of characters and avoid any chance of display failures?
Update #2:
How I am parsing:
Future<Null> getAll() async{
var response = await
DefaultAssetBundle.of(context).loadString('assets/json/dictionary.json');
var decodedData = json.decode(response);
setState(() {
for(Map word in decodedData){
mWordsList.add(Words.fromJson(word));
}
});
}
The class:
class Words{
final int id;
final String wordEn, wordIsl;
Words({this.id, this.wordEn, this.wordIsl});
factory Words.fromJson(Map<String, dynamic> json){
return new Words(
id: json['wordId'],
wordEn: json['englishWord'],
wordIsl: json['icelandicWord']
);
}
}
JSON Model:
{
"wordId": 47,
"englishWord": "Age",
//Here's a String that has two special characters
"icelandicWord": "\u00c3\u00a6vi"
}

I had similar issues with accented characters. They were not displayed as expected.
This worked for me
final codeUnits = source.codeUnits;
return Utf8Decoder().convert(codeUnits);

The problem is that your JSON is stored locally.
Let's say you have
Map<String, String> jsonObject = {"info": "Æ æ æ Ö ö ö"};
So to show your text correctly you have to encode and decode back your JSON with utf-8.
I understand that's serialization and deserialization are costly operations, but's it's a workaround for locally stored JSON objects that contains UTF-8 texts.
import 'dart:convert';
jsonDecode(jsonEncode(jsonObject))["info"]
If you get that JSON from server, then it's much more simpler, for example in dio package you can chose contentType params that's is "application/json; charset=utf-8" by default.

Related

How to decode or convert base64 string url to UintList in Dart?

I got this base64Result from canvas.toDataURL() but I having difficulties parsing in Dart to 'UintList'
import 'dart:convert';
final String base64Result = result.toString();
print("$logTrace calling web function done ${base64Result.length}");
final bytes = base64Url.decode(base64Result);
character (at character 5)
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAARwAAAIrCAYAAAA9YyZoAAAgAElEQ...
^
'data:image/png;base64,' is part of the data URL, not part of a base-64 string. You need to extract the base-64 data from the URL first.
Luckily, the UriData class can do this all for you:
final bytes = UriData.parse(base64Result).contentAsBytes();

How to display Unicode Smiley from json response dynamically in flutter

How to display Unicode Smiley from json response dynamically in flutter. It's display properly when i declare string as a static but from dynamic response it's not display smiley properly.
Static Declaration: (Working)
child: Text("\ud83d\ude0e\ud83d\ude0eThis is just test notification..\ud83d\ude0e\ud83d\ude0e\ud83d\udcaf\ud83d\ude4c")
Dynamic Response:
"message":"\\ud83d\\ude4c Be Safe at your home \\ud83c\\udfe0",
When i'm parse and pass this response to Text then it's consider Unicode as a String and display as a string instead of Smiley Code is below to display text with smiley:
child: Text(_listData[index].message.toString().replaceAll("\\\\", "\\"))
Already go through this: Question but it's only working when single unicode not working with multiple unicode.
Anyone worked with text along with unicode caracter display dynamically then please let me know.
Another alternate Good Solution I would give to unescape characters is this:
1st ->
String s = "\\ud83d\\ude0e Be Safe at your home \\ud83c\\ude0e";
String q = s.replaceAll("\\\\", "\\");
This would print and wont be able to escape characters:
\ud83d\ud83d Be Safe at your home \ud83c\ud83d
and above would be the output.
So what one can do is either unescape them while parsing or use:
String convertStringToUnicode(String content) {
String regex = "\\u";
int offset = content.indexOf(regex) + regex.length;
while(offset > 1){
int limit = offset + 4;
String str = content.substring(offset, limit);
// print(str);
if(str!=null && str.isNotEmpty){
String uni = String.fromCharCode(int.parse(str,radix:16));
content = content.replaceFirst(regex+str,uni);
// print(content);
}
offset = content.indexOf(regex) + regex.length;
// print(offset);
}
return content;
}
This will replace and convert all the literals into unicode characters and result and output of emoji:
String k = convertStringToUnicode(q);
print(k);
😎 Be Safe at your home 🈎
That is above would be the output.
Note: above answer given would just work as good but this is just when you want to have an unescape function and don't need to use third-party libraries.
You can extend this using switch cases with multiple unescape solutions.
Issue resolved by using below code snippet.
Client client = Client();
final response = await client.get(Uri.parse('YOUR_API_URL'));
if (response.statusCode == 200) {
// If the server did return a 200 OK response,
// then parse the JSON.
final extractedData = json.decode(response.body.replaceAll("\\\\", "\\"));
}
Here we need to replace double backslash to single backslash and then decode JSON respone before set into Text like this we can display multiple unicode like this:
final extractedData = json.decode(response.body.replaceAll("\\",
"\"));
Hope this answer help to other

Why is the base64 string not showing completely?

So this is my code
_image1 = File(pickedImage.path);
List<int> imageBytes = _image1.readAsBytesSync();
String base64Image = base64.encode(imageBytes);
_shcpImg = base64Image;
But when I print the string _shcpImg, it just prints a part of the string, because when I copy and paste that base64 into an online converter, it only shows a really tiny piece of the image. So the thing is that the string is not showing completely or somehow the base64 encoder is not working well.
Any suggestions?
From the comments, since you are using VsCode and you can't print the full string (long string)
You can use log from dart: developer,
if the string is REALLY LONG, There is a workaround to fix this, the idea is to divide your long string into small pieces (in the example, 800 length for each piece) using RegExp and then iterate into the result and print each piece.
void printWrapped(String text) {
final pattern = new RegExp('.{1,800}'); // 800 is the size of each chunk
pattern.allMatches(text).forEach((match) => print(match.group(0)));
}

Emoji and accent encoding in dart/flutter

I get the next String from my api
"à é í ó ú ü ñ \uD83D\uDE00\uD83D\uDE03\uD83D\uDE04\uD83D\uDE01\uD83D\uDE06\uD83D\uDE05"
from a response in a json format
{
'apiText': "à é í ó ú ü ñ \uD83D\uDE00\uD83D\uDE03\uD83D\uDE04\uD83D\uDE01\uD83D\uDE06\uD83D\uDE05",
'otherInfo': 'etc.',
.
.
.
}
it contains accents à é í ó ú ü ñ that are not correctly encoded and it contains emojis \uD83D\uDE00\uD83D\uDE03\uD83D\uDE04\uD83D\uDE01\uD83D\uDE06\uD83D\uDE05
so far i have tried
var json = jsonDecode(response.body)
String apiText = json['apiText'];
List<int> bytes = apiText.codeUnits;
comentario = utf8.decode(bytes);
but produces a
[ERROR:flutter/lib/ui/ui_dart_state.cc(166)] Unhandled Exception: FormatException: Invalid UTF-8 byte (at offset 21)
how can i get the correct text with accents and emoji?
Based on the fact you called response.body I assumes you are using the http package which does have the body property on Response objects.
You should note the following detail in the documentation:
This is converted from bodyBytes using the charset parameter of the Content-Type header field, if available. If it's unavailable or if the encoding name is unknown, latin1 is used by default, as per RFC 2616.
Well, it seems rather likely that it cannot figure out the charset and therefore defaults to latin1 which explains how your response got messed up.
A solution for this is to use the resonse.bodyBytes instead which contains the raw bytes from the response. You can then manually parse this with e.g. utf8.decode(resonse.bodyBytes) if you are sure the response should be parsed as UTF-8.
rewrite your function as
String utf8convert(String text) {
var bytes = text.codeUnits;
String decodedCode = utf8.decode(bytes, allowMalformed: true);
if (decodedCode.contains("�")) {
return text;
}
return decodedCode;
}

How can I convert a utf8 string to LATIN1 in Dart?

I have many strings where the accents are converted wrongfully. I take those Strings from an API, so I cannot get them in other encoding formats. As an example, the string é returns as é from the API. Is there any way I can convert these strings to show the accents correctly?
Well, you can try something like this:
import 'dart:convert';
void main() {
const input = 'é';
final output = utf8.decode(latin1.encode(input), allowMalformed: true);
print(output); // é
}
Alternative you can get the response from your web call as bytes by using bodyBytes on the response object:
https://pub.dev/documentation/http/latest/http/Response/bodyBytes.html
And parse it with: latin1.decode or whatever charset the server are sending the the data as.