Performing a task with root-me.org, as I understand from the attack statement, it is necessary to perform a byte shift to receive is_member = true After reading some instructions, I supplement the message with a block and accordingly do xor with the previous one, but when I pass the token, the next decryption output
b'[id=546815648;name=iziziz;is_member=false;mail=blablacar;pad=00]r\xe5\xf2\x1dM\xa5\xcae\xff\x16\xa2\xc6\xbe\xd8~I'
Tell me what I could miss, my code:
var TOKEN =
"IRZjBh6GxjeYI7YZvxwfBHmxjY+Wd7bPr7s73wWwLHKaR+N8fPDIjT8/AlUIDSzniMgqCV9bJArQbec64kPYXQ=="
// [id=546815648;name=iziziz;is_member=false;mail=blablacar;pad=00]
func main() {
//block 16
tokenHex, err := base64.StdEncoding.DecodeString(TOKEN) // lenght 64
if err != nil {
log.Fatal(err)
}
block := 16
for i := 0; i < len(tokenHex); i += block {
fmt.Println(tokenHex[i : i+block])
}
tmp := make([]byte, block)
for i := 0; i < block; i++ {
tmp[i] = byte('a')
}
tokenHex = append(tokenHex,tmp...)
expected := []byte(`;is_member=true]`)
//last block
current := []byte{136 ,200 ,42 ,9 ,95 ,91, 36, 10 ,208 ,109, 231, 58, 226, 67 ,216, 93}
for i := 0; i < 16; i++ {
xor := expected[i] ^ current[i]
tokenHex[64+i] ^= xor
}
fmt.Println(base64.StdEncoding.EncodeToString(tokenHex))
}
Related
The project I have to extend is using mbedTLS and I have to extract the subject key identifier extension from the certificate. I have not found a workable solution so far. mbedTLS does not offer a direct function for this.
I have found mbedtls_x509_get_extbut this seems to be an internal function and very low level. I also do not have the offsets. I have also found the v3_ext storing maybe all extension as an mbedtls_x509_buf. I guess this could be parsed as ASN.1. a) Is the v3_extparsing approach the only option and correct? b) Are there better options?
Since no one had an idea how to do it, I followed the approach to parse the v3_ext field of the mbedtls_x509_crt struct.
#include "mbedtls/x509_crt.h"
#include <mbedtls/oid.h>
#include "mbedtls/platform.h"
const uint8_t SKID[] = {0x55, 0x1D, 0x0E}; //!< Subject key identifer OID.
int load_skid(const char* path, uint8_t* skid[]) {
int err = 0;
mbedtls_x509_crt cert;
mbedtls_x509_buf buf;
mbedtls_asn1_sequence extns;
mbedtls_asn1_sequence *next;
memset(&extns, 0, sizeof(extns));
mbedtls_x509_crt_init(&cert);
size_t tag_len;
if (mbedtls_x509_crt_parse_file(&cert, path)) {
err = 1;
goto exit;
}
buf = cert.v3_ext;
if (mbedtls_asn1_get_sequence_of(&buf.p, buf.p + buf.len, &extns, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)) {
err = 1;
goto exit;
}
next = &extns;
while (next) {
if (mbedtls_asn1_get_tag(&(next->buf.p), next->buf.p + next->buf.len, &tag_len, MBEDTLS_ASN1_OID)) {
err = 1;
goto exit;
}
if (!memcmp(next->buf.p, SKID, tag_len)) {
// get value
// correct length for SEQ TL + SKID OID value = 2 + tag_len
unsigned char *p = next->buf.p + tag_len;
if (mbedtls_asn1_get_tag(&p, p + next->buf.len-2-tag_len, &tag_len, MBEDTLS_ASN1_OCTET_STRING)) {
err = 1;
goto exit;
}
// include OCT TL = 2
if (mbedtls_asn1_get_tag(&p, p + next->buf.len-2, &tag_len, MBEDTLS_ASN1_OCTET_STRING)) {
err = 1;
goto exit;
}
if (tag_len != 20) {
err = 1;
goto exit;
}
memcpy(skid, p, 20);
goto exit;
}
next = next->next;
}
// skid not found
err = 1;
exit:
mbedtls_x509_crt_free(&cert);
mbedtls_asn1_sequence *cur;
next = extns.next;
while (next != NULL) {
cur = next;
next = next->next;
mbedtls_free(cur);
}
return err;
}
I have some text in file.Text like below:
#cat tmp
host = "192.168.2.80"
port = 5432
user = "pnmsuser"
password = "PNMS$$$$$$"
dbname = "pnms"
Just I want text like below after trimming:
"192.168.2.80"
5432
"pnmsuser"
"PNMS$$$$$$"
"pnms"
I try to trim like below
func dbFileTrimming() {
dat, err := ioutil.ReadFile("tmp")
check(err)
for key, line := range strings.Split(strings.TrimRight(string(dat), "\n"), "\n") {
// println(key, line)
if key == 3 {
line := string([]rune(line)[11:])
fmt.Println(line)
} else if key == 4 {
line := string([]rune(line)[9:])
fmt.Println(line)
} else {
line := string([]rune(line)[7:])
fmt.Println(line)
}
}
}
Is there a simple method for this?
Chop the line after the =:
for _, line := range strings.Split(strings.TrimRight(string(dat), "\n"), "\n") {
line = line[strings.Index(line, " = ")+3:]
fmt.Println(line)
}
This looks like an INI file, or similar enough so that libraries like go-ini could work.
Alternatively, try Strings.split() and putting the result in a map. Quick and dirty / untested:
result := map[string]string
for _, line := range strings.Split(string(dat), "\n") {
split := strings.Split(line, "=")
key := strings.Trim(split[0])
value := strings.Trim(split[0])
result[key] = value
}
By using go api I'm retrieving the an array object. like given below:-
[
{0 1 Sunday 1 21600 25200 1}
{0 1 Sunday 2 28800 32400 2}
{0 1 Sunday 3 36000 39600 1}
]
This data will be arranged using struct:-
type ProviderSpot struct {
Id int `json:"_id" bson:"_id"`
PId int `json:"pid" bson:"pid"`
Day string `json:"day" bson:"day"`
TimeSlug int `json:"time_slug" bson:"time_slug"`
StartTime int64 `json:"start_time" bson:"start_time"`
EndTime int64 `json:"end_time" bson:"end_time"`
Count int `json:"count" bson:"count"`
}
type ProviderSpots []ProviderSpot
See in the array object I have an count values in each object 1,2,1 then I have to store this record like that those record having count they will store in the available_spot only one time means that the upper record will save in the collection only one time having there count value 1 after that the left record will remains there count value 0,1,0 then
those record having there count field value more than 0 they will save that number of times having count value in the addition_spot. The code of golang I'm using is this:-
func SaveProviderSpot(c *gin.Context) {
response := ResponseController{}
values := c.PostForm("array")
var err error
byt := []byte(values)
var result models.ProviderSpots
if err = json.Unmarshal(byt, &result); err != nil{
fmt.Println(err)
}
fmt.Println(result)
for i := 0; i < len(result); i++ {
lastValue :=result[i].Count-1
if lastValue != -1 {
providerspot.PId = result[i].PId
providerspot.Day = result[i].Day
providerspot.TimeSlug = result[i].TimeSlug
providerspot.StartTime = result[i].StartTime
providerspot.EndTime = result[i].EndTime
providerspot.Count = result[i].Count - lastValue
id, _ := models.GetAutoIncrementCounter(config.ProvidersSpotsCounterId, config.ProvidersSpotsCollection)
providerspot.Id = id
fmt.Println("Here We go now :- ", &providerspot)
err = models.AddProviderSpot(&providerspot)
}
}
}
Give some example of this which will solve this. Thanks for your valueable time spending on this question.
I solved that question answer but Can anyone tell me that will right for my code or not:-
func SaveProviderSpot(c *gin.Context) {
response := ResponseController{}
values := c.PostForm("array")
var err error
byt := []byte(values)
var result models.ProviderSpots
if err = json.Unmarshal(byt, &result); err != nil{
fmt.Println(err)
}
fmt.Println(result)
for i := 0; i < len(result); i++ {
for j := 1; j <= result[i].Count; j++ {
// lastValue := result[i].Count-1
// if lastValue != -1 {
if j == 1{
providerspot.PId = result[i].PId
providerspot.Day = result[i].Day
providerspot.TimeSlug = result[i].TimeSlug
providerspot.StartTime = result[i].StartTime
providerspot.EndTime = result[i].EndTime
providerspot.Count = 1//result[i].Count - lastValue
id, _ := models.GetAutoIncrementCounter(config.ProvidersSpotsCounterId, config.ProvidersSpotsCollection)
providerspot.Id = id
fmt.Println("Here We go now :- ", &providerspot)
err = models.AddProviderSpot(&providerspot)
}else{
providerspot.PId = result[i].PId
providerspot.Day = result[i].Day
providerspot.TimeSlug = result[i].TimeSlug
providerspot.StartTime = result[i].StartTime
providerspot.EndTime = result[i].EndTime
providerspot.Count = 1//result[i].Count - lastValue
id, _ := models.GetAutoIncrementCounter(config.AdditionalProviderCounterSpot, config.AdditionalProviderSpot)
providerspot.Id = id
err = models.AddAdditionalProviderSpot(&providerspot)
}
}
}
}
This will do want I want but I'm confused that it is right for me or not.
I'm trying to send an icmp message whose TTL is just 1, and expect to receive a time exceeded message. that message does come(I see it from wireshark), but my program blocks on syscall.Recvfrom. Anyone knows why?
icmp.go
package main
import (
"bytes"
"encoding/binary"
"fmt"
"net"
"os"
"syscall"
)
type ICMP struct {
Type uint8
Code uint8
Checksum uint16
Identifier uint16
SeqNo uint16
}
func Checksum(data []byte) uint16 {
var (
sum uint32
length int = len(data)
index int
)
for length > 1 {
sum += uint32(data[index])<<8 + uint32(data[index+1])
index += 2
length -= 2
}
if length > 0 {
sum += uint32(data[index])
}
sum += (sum >> 16)
return uint16(^sum)
}
func main() {
h := Header{
Version: 4,
Len: 20,
TotalLen: 20 + 8,
TTL: 1,
Protocol: 1,
// Dst:
}
argc := len(os.Args)
if argc < 2 {
fmt.Println("usage: program + host")
return
}
ipAddr, _ := net.ResolveIPAddr("ip", os.Args[1])
h.Dst = ipAddr.IP
icmpReq := ICMP{
Type: 8,
Code: 0,
Identifier: 0,
SeqNo: 0,
}
out, err := h.Marshal()
if err != nil {
fmt.Println("ip header error", err)
return
}
var icmpBuf bytes.Buffer
binary.Write(&icmpBuf, binary.BigEndian, icmpReq)
icmpReq.Checksum = Checksum(icmpBuf.Bytes())
icmpBuf.Reset()
binary.Write(&icmpBuf, binary.BigEndian, icmpReq)
fd, _ := syscall.Socket(syscall.AF_INET, syscall.SOCK_RAW, syscall.IPPROTO_RAW)
addr := syscall.SockaddrInet4{
Port: 0,
}
copy(addr.Addr[:], ipAddr.IP[12:16])
pkg := append(out, icmpBuf.Bytes()...)
fmt.Println("ip length", len(pkg))
if err := syscall.Sendto(fd, pkg, 0, &addr); err != nil {
fmt.Println("Sendto err:", err)
}
var recvBuf []byte
if nBytes, rAddr, err := syscall.Recvfrom(fd, recvBuf, 0); err == nil {
fmt.Printf("recv %d bytes from %v\n", nBytes, rAddr)
}
}
additionally, I use header.go and helper.go from https://github.com/golang/net/tree/master/ipv4
As Andy pointed out, the raw(7) man page says:
An IPPROTO_RAW socket is send only. If you really want to receive
all IP packets, use a packet(7) socket with the ETH_P_IP protocol.
Note that packet sockets don't reassemble IP fragments, unlike raw
sockets.
I know I can receive ICMP reply if I set IPPROTO_ICMP as the protocol when I create the socket, but I need to set TTL to 1 which must be done in IP layer. Therefore I send the ICMP request with IPPROTO_RAW socket, after that I use net.ListenIP to receive ICMP messages. Here is the code:
package main
import (
"bytes"
"encoding/binary"
"log"
"net"
"os"
"syscall"
)
const icmpID uint16 = 43565 // use a magic number for now
type ICMP struct {
Type uint8
Code uint8
Checksum uint16
Identifier uint16
SeqNo uint16
}
func Checksum(data []byte) uint16 {
var (
sum uint32
length int = len(data)
index int
)
for length > 1 {
sum += uint32(data[index])<<8 + uint32(data[index+1])
index += 2
length -= 2
}
if length > 0 {
sum += uint32(data[index])
}
sum += (sum >> 16)
return uint16(^sum)
}
func main() {
h := Header{
Version: 4,
Len: 20,
TotalLen: 20 + 8,
TTL: 1,
Protocol: 1,
}
argc := len(os.Args)
if argc < 2 {
log.Println("usage: program + host")
return
}
ipAddr, _ := net.ResolveIPAddr("ip", os.Args[1])
h.Dst = ipAddr.IP
icmpReq := ICMP{
Type: 8,
Code: 0,
Identifier: icmpID,
SeqNo: 1,
}
out, err := h.Marshal()
if err != nil {
log.Println("ip header error", err)
return
}
var icmpBuf bytes.Buffer
binary.Write(&icmpBuf, binary.BigEndian, icmpReq)
icmpReq.Checksum = Checksum(icmpBuf.Bytes())
icmpBuf.Reset()
binary.Write(&icmpBuf, binary.BigEndian, icmpReq)
fd, _ := syscall.Socket(syscall.AF_INET, syscall.SOCK_RAW, syscall.IPPROTO_RAW)
addr := syscall.SockaddrInet4{
Port: 0,
}
copy(addr.Addr[:], ipAddr.IP[12:16])
pkg := append(out, icmpBuf.Bytes()...)
if err := syscall.Sendto(fd, pkg, 0, &addr); err != nil {
log.Println("Sendto err:", err)
}
laddr, err := net.ResolveIPAddr("ip4:icmp", "0.0.0.0")
if err != nil {
log.Fatal(err)
}
c, err := net.ListenIP("ip4:icmp", laddr)
if err != nil {
log.Fatal(err)
}
for {
buf := make([]byte, 2048)
n, raddr, err := c.ReadFrom(buf)
if err != nil {
log.Println(err)
continue
}
icmpType := buf[0]
if icmpType == 11 {
if n == 36 { // Time exceeded messages
// A time exceeded message contain IP header(20 bytes) and first 64 bits of the original payload
id := binary.BigEndian.Uint16(buf[32:34])
log.Println("recv id", id)
if id == icmpID {
log.Println("recv Time Exceeded from", raddr)
}
}
}
}
}
Actually, I am writing a traceroute in go, if anyone is interested about that, the whole code is in github.
I think you need to give IPPROTO_ICMP as the protocol when you create your socket. The raw(7) man page says that an IPPROTO_RAW socket is send only. Also, if you use IPPROTO_ICMP, you don't give the IP header. (Note: I haven't actually tried this in Go.)
I can read the file to bytes array
but when I convert it to string
it treat the utf16 bytes as ascii
How to convert it correctly?
package main
import ("fmt"
"os"
"bufio"
)
func main(){
// read whole the file
f, err := os.Open("test.txt")
if err != nil {
fmt.Printf("error opening file: %v\n",err)
os.Exit(1)
}
r := bufio.NewReader(f)
var s,b,e = r.ReadLine()
if e==nil{
fmt.Println(b)
fmt.Println(s)
fmt.Println(string(s))
}
}
output:
false
[255 254 91 0 83 0 99 0 114 0 105 0 112 0 116 0 32 0 73 0 110 0 102 0 111 0 93 0
13 0]
S c r i p t I n f o ]
Update:
After I tested the two examples, I have understanded what is the exact problem now.
In windows, if I add the line break (CR+LF) at the end of the line, the CR will be read in the line. Because the readline function cannot handle unicode correctly ([OD OA]=ok, [OD 00 OA 00]=not ok).
If the readline function can recognize unicode, it should understand [OD 00 OA 00] and return []uint16 rather than []bytes.
So I think I should not use bufio.NewReader as it is not able to read utf16, I don't see bufio.NewReader.ReadLine can accept parameter as flag to indicate the reading text is utf8, utf16le/be or utf32. Is there any readline function for unicode text in go library?
The latest version of golang.org/x/text/encoding/unicode makes it easier to do this because it includes unicode.BOMOverride, which will intelligently interpret the BOM.
Here is ReadFileUTF16(), which is like os.ReadFile() but decodes UTF-16.
package main
import (
"bytes"
"fmt"
"io/ioutil"
"log"
"strings"
"golang.org/x/text/encoding/unicode"
"golang.org/x/text/transform"
)
// Similar to ioutil.ReadFile() but decodes UTF-16. Useful when
// reading data from MS-Windows systems that generate UTF-16BE files,
// but will do the right thing if other BOMs are found.
func ReadFileUTF16(filename string) ([]byte, error) {
// Read the file into a []byte:
raw, err := ioutil.ReadFile(filename)
if err != nil {
return nil, err
}
// Make an tranformer that converts MS-Win default to UTF8:
win16be := unicode.UTF16(unicode.BigEndian, unicode.IgnoreBOM)
// Make a transformer that is like win16be, but abides by BOM:
utf16bom := unicode.BOMOverride(win16be.NewDecoder())
// Make a Reader that uses utf16bom:
unicodeReader := transform.NewReader(bytes.NewReader(raw), utf16bom)
// decode and print:
decoded, err := ioutil.ReadAll(unicodeReader)
return decoded, err
}
func main() {
data, err := ReadFileUTF16("inputfile.txt")
if err != nil {
log.Fatal(err)
}
final := strings.Replace(string(data), "\r\n", "\n", -1)
fmt.Println(final)
}
Here is NewScannerUTF16 which is like os.Open() but returns a scanner.
package main
import (
"bufio"
"fmt"
"log"
"os"
"golang.org/x/text/encoding/unicode"
"golang.org/x/text/transform"
)
type utfScanner interface {
Read(p []byte) (n int, err error)
}
// Creates a scanner similar to os.Open() but decodes the file as UTF-16.
// Useful when reading data from MS-Windows systems that generate UTF-16BE
// files, but will do the right thing if other BOMs are found.
func NewScannerUTF16(filename string) (utfScanner, error) {
// Read the file into a []byte:
file, err := os.Open(filename)
if err != nil {
return nil, err
}
// Make an tranformer that converts MS-Win default to UTF8:
win16be := unicode.UTF16(unicode.BigEndian, unicode.IgnoreBOM)
// Make a transformer that is like win16be, but abides by BOM:
utf16bom := unicode.BOMOverride(win16be.NewDecoder())
// Make a Reader that uses utf16bom:
unicodeReader := transform.NewReader(file, utf16bom)
return unicodeReader, nil
}
func main() {
s, err := NewScannerUTF16("inputfile.txt")
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(s)
for scanner.Scan() {
fmt.Println(scanner.Text()) // Println will add back the final '\n'
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading inputfile:", err)
}
}
FYI: I have put these functions into an open source module and have made further improvements. See https://github.com/TomOnTime/utfutil/
UTF16, UTF8, and Byte Order Marks are defined by the Unicode Consortium: UTF-16 FAQ, UTF-8 FAQ, and Byte Order Mark (BOM) FAQ.
Issue 4802: bufio: reading lines is too cumbersome
Reading lines from a file is too cumbersome in Go.
People are often drawn to bufio.Reader.ReadLine because of its name,
but it has a weird signature, returning (line []byte, isPrefix bool,
err error), and requires a lot of work.
ReadSlice and ReadString require a delimiter byte, which is almost
always the obvious and unsightly '\n', and also can return both a line
and an EOF
Revision: f685026a2d38
bufio: new Scanner interface
Add a new, simple interface for scanning (probably textual) data,
based on a new type called Scanner. It does its own internal
buffering, so should be plausibly efficient even without injecting a
bufio.Reader. The format of the input is defined by a "split
function", by default splitting into lines.
go1.1beta1 released
You can download binary and source distributions from the usual place:
https://code.google.com/p/go/downloads/list?q=go1.1beta1
Here's a program which uses the Unicode rules to convert UTF16 text file lines to Go UTF8 encoded strings. The code has been revised to take advantage of the new bufio.Scanner interface in Go 1.1.
package main
import (
"bufio"
"bytes"
"encoding/binary"
"fmt"
"os"
"runtime"
"unicode/utf16"
"unicode/utf8"
)
// UTF16BytesToString converts UTF-16 encoded bytes, in big or little endian byte order,
// to a UTF-8 encoded string.
func UTF16BytesToString(b []byte, o binary.ByteOrder) string {
utf := make([]uint16, (len(b)+(2-1))/2)
for i := 0; i+(2-1) < len(b); i += 2 {
utf[i/2] = o.Uint16(b[i:])
}
if len(b)/2 < len(utf) {
utf[len(utf)-1] = utf8.RuneError
}
return string(utf16.Decode(utf))
}
// UTF-16 endian byte order
const (
unknownEndian = iota
bigEndian
littleEndian
)
// dropCREndian drops a terminal \r from the endian data.
func dropCREndian(data []byte, t1, t2 byte) []byte {
if len(data) > 1 {
if data[len(data)-2] == t1 && data[len(data)-1] == t2 {
return data[0 : len(data)-2]
}
}
return data
}
// dropCRBE drops a terminal \r from the big endian data.
func dropCRBE(data []byte) []byte {
return dropCREndian(data, '\x00', '\r')
}
// dropCRLE drops a terminal \r from the little endian data.
func dropCRLE(data []byte) []byte {
return dropCREndian(data, '\r', '\x00')
}
// dropCR drops a terminal \r from the data.
func dropCR(data []byte) ([]byte, int) {
var endian = unknownEndian
switch ld := len(data); {
case ld != len(dropCRLE(data)):
endian = littleEndian
case ld != len(dropCRBE(data)):
endian = bigEndian
}
return data, endian
}
// SplitFunc is a split function for a Scanner that returns each line of
// text, stripped of any trailing end-of-line marker. The returned line may
// be empty. The end-of-line marker is one optional carriage return followed
// by one mandatory newline. In regular expression notation, it is `\r?\n`.
// The last non-empty line of input will be returned even if it has no
// newline.
func ScanUTF16LinesFunc(byteOrder binary.ByteOrder) (bufio.SplitFunc, func() binary.ByteOrder) {
// Function closure variables
var endian = unknownEndian
switch byteOrder {
case binary.BigEndian:
endian = bigEndian
case binary.LittleEndian:
endian = littleEndian
}
const bom = 0xFEFF
var checkBOM bool = endian == unknownEndian
// Scanner split function
splitFunc := func(data []byte, atEOF bool) (advance int, token []byte, err error) {
if atEOF && len(data) == 0 {
return 0, nil, nil
}
if checkBOM {
checkBOM = false
if len(data) > 1 {
switch uint16(bom) {
case uint16(data[0])<<8 | uint16(data[1]):
endian = bigEndian
return 2, nil, nil
case uint16(data[1])<<8 | uint16(data[0]):
endian = littleEndian
return 2, nil, nil
}
}
}
// Scan for newline-terminated lines.
i := 0
for {
j := bytes.IndexByte(data[i:], '\n')
if j < 0 {
break
}
i += j
switch e := i % 2; e {
case 1: // UTF-16BE
if endian != littleEndian {
if i > 1 {
if data[i-1] == '\x00' {
endian = bigEndian
// We have a full newline-terminated line.
return i + 1, dropCRBE(data[0 : i-1]), nil
}
}
}
case 0: // UTF-16LE
if endian != bigEndian {
if i+1 < len(data) {
i++
if data[i] == '\x00' {
endian = littleEndian
// We have a full newline-terminated line.
return i + 1, dropCRLE(data[0 : i-1]), nil
}
}
}
}
i++
}
// If we're at EOF, we have a final, non-terminated line. Return it.
if atEOF {
// drop CR.
advance = len(data)
switch endian {
case bigEndian:
data = dropCRBE(data)
case littleEndian:
data = dropCRLE(data)
default:
data, endian = dropCR(data)
}
if endian == unknownEndian {
if runtime.GOOS == "windows" {
endian = littleEndian
} else {
endian = bigEndian
}
}
return advance, data, nil
}
// Request more data.
return 0, nil, nil
}
// Endian byte order function
orderFunc := func() (byteOrder binary.ByteOrder) {
switch endian {
case bigEndian:
byteOrder = binary.BigEndian
case littleEndian:
byteOrder = binary.LittleEndian
}
return byteOrder
}
return splitFunc, orderFunc
}
func main() {
file, err := os.Open("utf16.le.txt")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer file.Close()
fmt.Println(file.Name())
rdr := bufio.NewReader(file)
scanner := bufio.NewScanner(rdr)
var bo binary.ByteOrder // unknown, infer from data
// bo = binary.LittleEndian // windows
splitFunc, orderFunc := ScanUTF16LinesFunc(bo)
scanner.Split(splitFunc)
for scanner.Scan() {
b := scanner.Bytes()
s := UTF16BytesToString(b, orderFunc())
fmt.Println(len(s), s)
fmt.Println(len(b), b)
}
fmt.Println(orderFunc())
if err := scanner.Err(); err != nil {
fmt.Println(err)
}
}
Output:
utf16.le.txt
15 "Hello, 世界"
22 [34 0 72 0 101 0 108 0 108 0 111 0 44 0 32 0 22 78 76 117 34 0]
0
0 []
15 "Hello, 世界"
22 [34 0 72 0 101 0 108 0 108 0 111 0 44 0 32 0 22 78 76 117 34 0]
LittleEndian
utf16.be.txt
15 "Hello, 世界"
22 [0 34 0 72 0 101 0 108 0 108 0 111 0 44 0 32 78 22 117 76 0 34]
0
0 []
15 "Hello, 世界"
22 [0 34 0 72 0 101 0 108 0 108 0 111 0 44 0 32 78 22 117 76 0 34]
BigEndian
Here is the simplest way to read it:
package main
import (
"bufio"
"fmt"
"log"
"os"
"golang.org/x/text/encoding/unicode"
"golang.org/x/text/transform"
)
func main() {
file, err := os.Open("./text.txt")
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(transform.NewReader(file, unicode.UTF16(unicode.LittleEndian, unicode.UseBOM).NewDecoder()))
for scanner.Scan() {
fmt.Printf(scanner.Text())
}
}
since Windows use little-endian order by default link, we use unicode.UseBOM policy to retrieve BOM from the text, and unicode.LittleEndian as a fallback
For example:
package main
import (
"errors"
"fmt"
"log"
"unicode/utf16"
)
func utf16toString(b []uint8) (string, error) {
if len(b)&1 != 0 {
return "", errors.New("len(b) must be even")
}
// Check BOM
var bom int
if len(b) >= 2 {
switch n := int(b[0])<<8 | int(b[1]); n {
case 0xfffe:
bom = 1
fallthrough
case 0xfeff:
b = b[2:]
}
}
w := make([]uint16, len(b)/2)
for i := range w {
w[i] = uint16(b[2*i+bom&1])<<8 | uint16(b[2*i+(bom+1)&1])
}
return string(utf16.Decode(w)), nil
}
func main() {
// Simulated data from e.g. a file
b := []byte{255, 254, 91, 0, 83, 0, 99, 0, 114, 0, 105, 0, 112, 0, 116, 0, 32, 0, 73, 0, 110, 0, 102, 0, 111, 0, 93, 0, 13, 0}
s, err := utf16toString(b)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%q", s)
}
(Also here)
Output:
"[Script Info]\r"
If you want anything to print as a string you could use fmt.Sprint
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// read whole the file
f, err := os.Open("test.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
return
}
r := bufio.NewReader(f)
var s, _, e = r.ReadLine()
if e != nil {
fmt.Println(e)
return
}
fmt.Println(fmt.Sprint(string(s)))
}