UTF-16LE txt file decode as String in Flutter (dart) - flutter

Question

To read a file contents of .txt file I am using
List<String> linesList = await file.readAsLines(encoding: latin1);
return linesList;
Files with Encodng UTF-8 are working perfectly with this above code.
But for Encoding UTF-16LE its returning a list with length double of the lines in the file but are all empty except first line. This first index contains ÿþ#

Answer 1

As package:utf is now abandoned (and therefore will never support null-safety), another way to read a UTF-16LE file as a String is to take advantage of Dart Strings using UTF-16 code units internally. You therefore can read the file, interpret the data as 16-bit (unsigned) integers, and then create a String using those as UTF-16 code units:
Basically:
var f = File("utf-16le.txt");
var bytes = f.readAsBytesSync();
// Note that this assumes that the system's native endianness is the same as
// the file's.
var utf16CodeUnits = bytes.buffer.asUint16List();
var s = String.fromCharCodes(utf16CodeUnits);
I also leave it as an exercise for the reader to deal with potential BOMs at the beginning of the file.
Also see https://github.com/dart-lang/convert/issues/30, which requests that the Dart SDK provide UTF-16 conversion functions.

Answer 2

So the first credit goes to #Richard_Heap , who has commented on the above question. He mentioned a dart package that encodes and decodes UTF formats. I have been able to decode the txt files as expected in my Flutter app using this package.
First I am identifying the utf format type by these functions available from the package mentioned by #Richard_Heap
List<int> bytes = await file.readAsBytes();
hasUtf16beBom(bytes)
hasUtf16Bom(bytes)}
hasUtf16leBom(bytes)
hasUtf32beBom(bytes)
hasUtf32Bom(bytes)
hasUtf32leBom(bytes)
There are different decoder & encoder functions in this package that can be used once the utf format is known using these above functions. Like I used
String decodedString = decodeUtf16le(bytes);

Answer 3

Check out the charset package. It is null-safe and supports both UTF-16BE and UTF-16LE according to documentation.
Usage:
import 'package:charset/charset.dart';
main() {
// default
print(utf16.decode([254, 255, 78, 10, 85, 132, 130, 229, 108, 52]));
print(utf16.encode("上善若水"));
// detect
print(hasUtf16Bom([0xFE, 0xFF, 0x6C, 0x34]));
// advance
Utf16Encoder encoder = utf16.encoder as Utf16Encoder;
print(encoder.encodeUtf16Be("上善若水", false));
print(encoder.encodeUtf16Le("上善若水", true));
}