I need to extract some data from new (unseen) messages; I'm trying to make range of that data:
c, _ = imap.Dial("someserverr")
defer c.Logout(30 * time.Second)
fmt.Println("Server says hello:", c.Data[0].Info)
c.Data = nil
if c.State() == imap.Login {
c.Login("somedata", "somedata")
}
c.Select("INBOX", false)
set, _ := imap.NewSeqSet("")
fmt.Println("unseen", c.Mailbox.Unseen)
fmt.Println(c.Mailbox)
if c.Mailbox.Unseen >= 1 {
set.AddRange(1, c.Mailbox.Unseen)
} else {
set.Add("0:0")
}
The main problem here is command c.Mailbox showing wrong number of unseen messages.
For example, if I have 5 unread messages in INBOX it shows 1. If I mark that one as read, it will show 4, and so on.
Related
I've accidentally spotted a bug when parts of a message from previous connection go to the next message.
I have a basic server with client. I have removed all the error handling to avoid bloating the examples too much.
Also I've replaced some Printf's with time.Sleep since I just don't have a chance to break the connection in time to reproduce the bug because it reads the data too fast.
The "package" is a simple structure, where the first 4 bytes is the length and then goes the content.
Client code:
package main
import (
"encoding/binary"
"fmt"
"net"
)
func main() {
conn, _ := net.Dial("tcp", "0.0.0.0:8081")
defer conn.Close()
str := "msadsakdjsajdklsajdklsajdk"
// Creating a package
buf := make([]byte, len(str)+4)
copy(buf[4:], str)
binary.LittleEndian.PutUint32(buf[:4], uint32(len(str)))
for {
_, err := conn.Write(buf)
if err != nil {
fmt.Println(err)
return
}
}
}
Server code:
package main
import (
"encoding/binary"
"fmt"
"net"
"sync"
"time"
)
func ReadConnection(conn net.Conn, buf []byte) (err error) {
maxLen := cap(buf)
readSize := 0
for readSize < maxLen {
// instead of Printf
time.Sleep(time.Nanosecond * 10)
readN, err := conn.Read(buf[readSize:])
if err != nil {
return err
}
readSize += readN
}
return nil
}
func handleConnection(conn net.Conn, waitGroup *sync.WaitGroup) {
waitGroup.Add(1)
defer conn.Close()
defer waitGroup.Done()
fmt.Printf("Serving %s\n", conn.RemoteAddr().String())
var packageSize int32 = 0
int32Buf := make([]byte, 4)
for {
// read the length
conn.Read(int32Buf)
packageSize = int32(binary.LittleEndian.Uint32(int32Buf))
// assuming the length should be 26
if packageSize > 26 {
fmt.Println("Package size error")
return
}
// read the content
packageBuf := make([]byte, packageSize)
if err := ReadConnection(conn, packageBuf); err != nil {
fmt.Printf("ERR: %s\n", err)
return
}
// instead of Printf
time.Sleep(time.Nanosecond * 100)
}
}
func main() {
//establish connection
listener, _ := net.Listen("tcp", "0.0.0.0:8081")
defer listener.Close()
waitGroup := sync.WaitGroup{}
for {
conn, err := listener.Accept()
if err != nil {
break
}
go handleConnection(conn, &waitGroup)
}
waitGroup.Wait()
}
So for some reason, int32Buf receives the last 2 bytes from a previous message (d, k) and the first 2 bytes of the length, resulting in [107, 100, 26, 0] bytes slice, when it should be [26, 0, 0, 0].
And of course, the rest of the data contains remaining two zeroes:
conn.Read(int32Buf)
You need to check the return value of conn.Read and compare it against your expectations. You are assuming in your code that conn.Read will always completely fill the given buffer of 4 bytes.
This assumption is wrong, i.e. it might actually read less data. Specifically it might read only 2 bytes in which case you'll end up with \x1a\x00\x00\x00 in your buffer which still translates to a message length of 26. Only, the first 2 bytes of the message will actually be the last 2 bytes of the length which were not included in the last read. This means after reading the 26 bytes it will not have read the full message. 2 bytes are legt and will be included into the next message - this is what you observed.
To be sure that the exact size of the buffer is read check the return values of conn.Read or use io.ReadFull. After you've done this it works as expected (from the comment):
Ok, now it works perfect
So why does this happened only in context of a new connection? Maybe because the additional load due to another connection changed the behavior slightly but significantly enough. Still, these are not the data read from a different connection but data from the current one contrary to the description in the question. This could be easily checked by using different messages with different clients.
I feel that there may be a problem when the receiving server returns. Relevant code follows:
const RecvBufferSize = 1024 * 64
for {
readBuff := make([]byte, RecvBufferSize)
n, err := udp_client.conn.Read(readBuff)
if err != nil {
log.Error("socket recv error.", zap.String("errMsg", err.Error()))
return
}
select {
case udp_client.recvMessage <- readBuff[:n]:
case <-udp_client.closeTest:
log.Info("close recvFrom", zap.String("Stack", string(debug.Stack())))
close(udp_client.recvMessage)
return
}
}
Each time this code receives the information returned by the server, it will re apply for an address of []byte. Will this put pressure on the GC when the received data volume is compared? Is there any good solution?
I'm using Go for performance testing.
I'm trying to read data from a telnet session in golang. I wrote the following functions in an attempt to accomplish this.
Initially I was having an issue where I was reading from a socket with no data so it would lock and never return. BufferSocketData is my attempt to work around this issue as I can't know if there is data to read. The idea is it will wait 1 second before determining there is not data in the socket and return an empty string.
GetData seems to work the first time there is new data in the buffer, but beyond that it gets no new data. I'm sure this has something to do with my use of goroutines and channels, I'm new to go and I'm sure I'm not using them correctly.
Any ideas as to why my subsequent reads return no data?
/*
ReadDataFromSocket - Attempts to read any data in the socket.
*/
func ReadDataFromSocket(sock io.Reader, c chan string) {
var recvData = make([]byte, 1024)
var numBytes, _ = sock.Read(recvData)
c <- string(recvData[:numBytes])
}
/*
BufferSocketData - Read information from the socket and store it in the buffer.
*/
func (tn *TelnetLib) BufferSocketData(inp chan string, out chan string) {
var data string
var timeout int64 = 1000 // 1 second timeout.
var start = utils.GetTimestamp()
for utils.GetTimestamp()-start < timeout {
select {
case data = <-inp:
default:
}
if data != "" {
break
}
}
out <- data
}
/*
GetData - Start goroutines to get and buffer data.
*/
func (tn *TelnetLib) GetData() {
var sockCh = make(chan string)
var buffCh = make(chan string)
go ReadDataFromSocket(tn.Conn, sockCh)
go tn.BufferSocketData(sockCh, buffCh)
var data = <-buffCh
if data != "" {
tn.Buffer += data
}
}
Please let me know if you need any additional information.
Use SetReadDeadline to read data with a time limit:
func (tn *TelnetLib) GetData() {
tn.Conn.SetReadDeadline(time.Second)
recvData := make([]byte, 1024)
n, err := tn.Conn.Read(recvData)
if n > 0 {
// do something with recvData[:n]
}
if e, ok := err.(interface{ Timeout() bool }); ok && e.Timeout() {
// handle timeout
} else if err != nil {
// handle error
}
}
Note that a single call Read may not read all data sent by the peer. You may want to accumulate data by calling Read in a loop or call io.ReadFull.
I'm working on using Postgres logical replication in Go using the pgx library to get database changes via a logical replication slot using wal2json as the output plugin. The Postgres version being used is v10.1.
When in the WaitForReplicationMessage loop, I receive a message, and if that message is a ServerHeartbeat, I send a standby status message to update the server of my consumed position in the WAL. This StandbyStatus message has a field called ReplyRequested which, if equals 1 tells the server to send a ServerHeartbeat; and if the value is 0 it's not supposed to do anything.
Now I'm sending a StandbyStatus message with the ReplyRequested value to 0 (which is the default value when the object is created). On sending this, the server sends a heartbeat message despite me telling it not to. I'm unable to see the cause of this issue.
Here's my code:
for {
log.Info("Waiting for message")
message, err := session.ReplConn.WaitForReplicationMessage(context.TODO())
if err != nil {
log.WithError(err).Errorf("%s", reflect.TypeOf(err))
continue
}
if message.WalMessage != nil {
log.Info(string(message.WalMessage.WalData))
} else if message.ServerHeartbeat != nil {
log.Info("Heartbeat requested")
// set the flushed LSN (and other LSN values) in the standby status and send to PG
log.Info(message.ServerHeartbeat)
// send Standby Status with the LSN position
err = session.sendStandbyStatus()
if err != nil {
log.WithError(err).Error("Unable to send standby status")
}
}
}
The sendStandbyStatus function above is:
func (session *Session) sendStandbyStatus() error {
standbyStatus, err := pgx.NewStandbyStatus(session.RestartLSN)
if err != nil {
return err
}
log.Info(standbyStatus) // the output of this confirms ReplyRequested is indeed 0
standbyStatus.ReplyRequested = 0 // still set it
err = session.ReplConn.SendStandbyStatus(standbyStatus)
if err != nil {
return err
}
return nil
}
I have 50 goroutines that start with a channel. Then, the process forever loops reading 100 database records at a time and sleeping for 10 seconds in between database calls. As it loops through the 100 email records to send, it passes each record through the channel to one of the 50 worker goroutines, who then sends the email. The problem is, after it goes through about 1000 emails, I start getting errors like this:
gomail: could not send email 1: read tcp 10.2.30.25:56708->216.###.##.###:25: read: connection reset by peer
I have to send out about 50k emails per day. What do you recommend? Here's the main code that processes the email queue and passes each record to the worker through the channel:
func main() {
MaxWorkers := 50
println("Creating: " + strconv.Itoa(MaxWorkers) + " workers..")
batchChannel := make(chan EmailQueue.EmailQueueObj)
for i := 0; i < MaxWorkers; i++ {
go startWorker(batchChannel)
}
for {
println("Getting queue..")
data, _ := EmailQueue.GetQueue() //returns 100 database records
println("Reading through " + strconv.Itoa(len(data)) + " records..")
for _, element := range data {
batchChannel <- element
}
time.Sleep(10 * time.Second)
}
}
func startWorker(channel chan EmailQueue.EmailQueueObj) {
var s gomail.SendCloser
var err error
open := false
for obj := range channel {
if !open {
s, err = dialer.Dial()
if err != nil {
fmt.Println(err.Error())
return
} else {
sendEmail(obj, &s)
}
} else {
sendEmail(obj, &s)
}
open = true
}
s.Close()
}
func sendEmail(obj EmailQueue.EmailQueueObj, s *gomail.SendCloser) {
m := gomail.NewMessage()
m.SetHeader("From", "example#example.com")
m.SetHeader("To", obj.Recipient)
m.SetHeader("Subject", obj.Subject.String)
m.SetBody("text/html", obj.HTML.String)
// Send the email
response := ""
status := ""
if err := gomail.Send(*s, m); err != nil {
response = err.Error()
status = "error"
} else {
response = "Email sent"
status = "sent"
}
m.Reset()
return
}
I am using the library gomail to send the emails. I am open to anything, even a new library or method to send these emails. But, what I'm doing currently is not working. Any help is greatly appreciated!