My liquidsoap audio stream has just started to stop playing with this error message every few requests.
Error while sending data: could not write data to host: Broken pipe in write()!
It seems to fix itself pretty quickly and I can restart playing, but the interruption is annoying.
The liquidsoap script I'm using is pretty basic and pulls some songs from a database.
Here is the script.
def apply_metadata(m) =
title = m["title"]
artist = m["artist"]
log("Now playing: #{title} by #{artist}")
end
def apply_track(m) =
get_process_lines("curl http://localhost/api/v1/liquidsoap/playing")
log("actually playing")
end
def get_request() =
uri = list.hd(default="",get_process_lines("curl http://localhost/api/v1/liquidsoap/next"))
request.create(uri)
end
def my_safe(s) =
security = sine()
fallback(track_sensitive=false,[s,security])
end
s = request.dynamic(id="s",timeout=60.0,get_request)
s = on_metadata(apply_metadata,s)
s = on_track(apply_track,s)
s = crossfade(s)
s = my_safe(s)
# We output the stream to an icecast
# server, in ogg/vorbis format.
output.icecast(
%mp3(id3v2=true,bitrate=128,samplerate=44100),
host = "localhost",
port = 8000,
mount = "ogr",
s
)
Here is the liquidsoap log.
2018/09/15 17:23:04 [lang:3] Now playing: La Sirena by Banyan
2018/09/15 17:23:17 [lang:3] actually playing
2018/09/15 17:23:17 [ogr:3] Metadata update may have failed with error: 400, Bad Request (HTTP/1.0)
2018/09/15 17:23:17 [clock.wallclock_main:2] We must catchup 13.88 seconds!
2018/09/15 17:23:17 [ogr:2] Error while sending data: could not write data to host: Broken pipe in write()!
2018/09/15 17:23:17 [ogr:3] Closing connection...
2018/09/15 17:23:17 [ogr:3] Will try to reconnect in 3.00 seconds.
2018/09/15 17:23:18 [clock.wallclock_main:2] We must catchup 7.12 seconds (we've been late for 100 rounds)!
2018/09/15 17:23:21 [ogr:3] Connecting mount ogr for source#localhost...
2018/09/15 17:23:21 [ogr:3] Connection setup was successful.
I'm not sure why it has started doing this as it's been working perfectly for a long time previously and I've made no changes.
Any suggestions or help much appreciated as my understanding of liquidsoap is very basic.
Thanks.
In case anyone else comes across this error message in my case the problem was not with liquid soap itself, but that an API request was taking too long and it was causing the scheduling to fail.
Related
I'm working with a Camel flow that uses a Netty TCP socket consumer to receive messages from a client program (which is outside of my control). The client should be opening a socket, sending us one message, then closing the socket, but we've been seeing cases where instead of one message Camel is "splitting" the text stream into two parts and trying to process them separately.
So I'm trying to figure out, since you can re-use the same socket for multiple Camel messages, but TCP sockets don't have a built-in concept of "frames" or a standard for message delimiters, how does Camel decide that a complete message has been received and is ready to process? I haven't been able to find a documented answer to this in the Netty component docs (https://camel.apache.org/components/3.15.x/netty-component.html), although maybe I'm missing something.
From playing around with a test script, it seems like one answer is "Camel assumes a message is complete and should be processed if it goes more than 1ms without receiving any input on the socket". Is this a correct statement, and if so is this behavior documented anywhere? Is there any way to change or configure this behavior? Really what I would prefer is for Camel to wait for an ETX character (or a much longer timeout) before processing a message, is that possible to set up?
Here's my test setup:
Camel flow:
from("netty:tcp://localhost:3003")
.log("Received: ${body}");
Python snippet:
DELAY_MS = 3
def send_msg(sock, msg):
print("Sending message: <{}>".format(msg))
if not sock.sendall(msg.encode()) is None:
print("Message failed to send")
time.sleep(DELAY_MS / 1000.0)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
print("Using DELAY_MS: {}".format(str(DELAY_MS)))
s.connect((args.hostname, args.port))
cutoff = int(math.floor(len(args.msg) / 2))
msg1 = args.msg[:cutoff]
send_msg(s, msg1)
msg2 = args.msg[cutoff:]
send_msg(s, msg2)
response = s.recv(1024)
except Exception as e:
print(e)
finally:
s.close()
I can see that with DELAY_MS=1 Camel logs one single message:
2022-02-21 16:54:40.689 INFO 19429 --- [erExecutorGroup] route1 : Received: a long string sent over the socket
But with DELAY_MS=2 it logs two separate messages:
2022-02-21 16:56:12.899 INFO 19429 --- [erExecutorGroup] route1 : Received: a long string sen
2022-02-21 16:56:12.899 INFO 19429 --- [erExecutorGroup] route1 : Received: t over the socket
After doing some more research, it seems like what I need to do is add a delimiter-based FrameDecoder to the decoders list.
Setting it up like this:
from("netty:tcp://localhost:3003?sync=true"
+ "&decoders=#frameDecoder,#stringDecoder"
+ "&encoders=#stringEncoder")
where frameDecoder is provided by
#Bean
ChannelHandlerFactory frameDecoder() {
ByteBuf[] ETX_DELIM = new ByteBuf[] { Unpooled.wrappedBuffer(new byte[] { (char)3 }) };
return ChannelHandlerFactories.newDelimiterBasedFrameDecoder(1024, ETX_DELIM,
false, "tcp");
}
seems to do the trick.
On the flip side though, it seems like this will hang indefinitely (or until lower-level TCP timeouts kick in?) if an ETX frame is not received, and I can't figure out any way to set a timeout on the decoder, so would still be eager for input if anyone knows how to do that.
I think the default "timeout" behavior I was seeing might've just been an artifact of Netty's read loop speed -- How does netty determine when a read is complete?
I'm having this intermittent error come up when making a call from a Dart gRPC client to a Go gRPC server. The exact error message is:
gRPC Error (code: 2, codeName: UNKNOWN, message: HTTP/2 error: Stream
error: Stream was terminated by peer (errorCode: 2)
This seems to occur frequently on creating the client stub with the channel and making a call to the server. Subsequent calls seems to work fine with a response returned correctly to the client. Then this error intermittently pops up again every n-th call (could be after 5 calls, 10 calls sometimes no errors until the 20th call).
When I use a different client (eg: BloomRPC or Java app I wrote), this does not happen no matter how hard or fast I spam the calls to the server. This kind of implies the server is probably not the issue.
Can anyone shine a light on what might be happening? Or provide some hints as to where to start looking/debugging? I've tried stepping through the code, but not really able to follow what is happening when the gRPC call is made/returned.
Steps in the client code:
create ClientChannel with ip, ports
final cc = ClientChannel(serverIP,
port: serverPort,
options:
const ChannelOptions(credentials: ChannelCredentials.insecure()
));
created a stub based on above channel
_userQueryStub = UserQueryGRPCClient(cc,
options: CallOptions(timeout: Duration(seconds: 30)));
call function from stub (repeatedly)
GetUserInformationRequest request = GetUserInformationRequest(userId: userId);
GetUserInformationResponse response = await _userQueryStub.getUserInformation(request);
Any help will be greatly appreciated!
At API there is:
val http = Http.configure(_
.setConnectionTimeoutInMs(1)
)
What for is this config? I use it with:
.setMaxRequestRetry(0)
I fought I will get failed future after timeout. Future I create like that:
val f = http(u OK as.String)
f.map {
NotificationClientConnectionParams.parseFromString
}
But instead of failure I get success long after my timeout.
How it should work?
My test looks like this:
val startTime = java.time.LocalTime.now()
val f = TcpUtil2.registerClientViaDispatch(ClientHeaders("12345", "123456789"))
f onSuccess {
case c =>
println(s"Success: $c")
println(java.time.Duration.between(startTime, java.time.LocalTime.now()).toMillis)
}
f onFailure {
case e =>
println(s"failure:${e.getMessage}")
}
Thread.sleep(2000)
Response time is in hundreds of milliseconds and I got success. Is it a bug of dispatch?
An HTTP roundtrip goes through several phases (overly simplified):
establishing connection
connection established
sending request payload
request payload sent
waiting for response payload
receiving response payload
response payload received
From what I understand you measure the time between states 1 and 7.
setConnectionTimeoutInMs comes from async-http-client which is used by Dispatch internally. Here's an excerpt from its documentation:
Set the maximum time in millisecond an AsyncHttpClient can wait when connecting to a remote host
Thus, this method sets the maximum time the client will wait between states 1 and 2.
There's also setRequestTimeoutInMs:
Set the maximum time in millisecond an AsyncHttpClient wait for a response
This method seems to set the time between states 5 and 6 (or 7, I'm not sure which one).
So here's what's probably happening in your case. You connect to remote host, the server accepts the connection very quickly (the time between 1 and 2 is small), so your Future doesn't get failed. Then there are several options: either server takes a lot of time to prepare the response before it starts sending it back to you (the time between 5 and 6), or the response is very big so it takes a lot of time to deliver it to you (the time between 6 and 7), or both. But since you don't set the request timeout, your Future is not getting failed because of this.
I having the following code to create a client socket to send/receive data:
val socket:Socket = new Socket(InetAddress.getByName("127.0.0.1"), 7777)
val inputStream = socket.getInputStream()
val bufferSource = new BufferedSource(inputStream)
val out = new PrintStream(socket.getOutputStream())
var data = "Hello Everyone"
out.println(data)
out.flush()
***socket.shutdownOutput()***
val in = bufferSource.getLines()
if (in.hasNext) {
println(in.next())
}
If I don't run socket.shutdownOutput(), I won't get the data from server,
because Server side is still waiting the input. Therefore I have to shutdown the outputStream.
But if shutdown the output, it can not be reopen. So I have to create a new socket for sending new data.
That caused sending one record needs to create a new socket. This is really awkward.
Is there any other way to tell the server that the output already finished without shutting down the output please.
Thanks in advance!
The problem is that the server doesn't know when to stop reading and process and reply.
What you need here is an application-level protocol that would dictate how server and clients are to communicate - what is a command, how a response to be formatted, etc.
This could be a line-oriented protocol - each new line represents a message (in general the message delimiter could be any other character sequence not appearing in the messages).
Or it could be fixed length messages; or messages pre-pended with message length (or type) to let the other side know how much data yo expect.
I am trying to connect to Apple Push Notification Service which uses a simple binary protocol over TCP protected with TLS (or SSL). The protocol indicates that when an error is encountered (there are about 10 well defined error conditions) APNS will send back an error response and then close the connection. This results in a half closed socket because the remote peer closed the socket. I can see its a graceful shutdown because APNS sends a FIN and RST using tcpdump.
Out of all the error conditions, I can deal with most before sending with validation. The situation in which this fails is when a notification is sent to an invalid device token which cannot be dealt with that easily because the tokens could be malformed. Tokens are opaque 32 byte values that are provided by APNS to a device and then registered with me. I have no way of knowing if it is valid when submitted to my service. Presumably APNS checksums the tokens in some way that they can do quick validation on the token fast.
Anyway,
I did what I thought was the right thing:-
a. open socket
b. try writing
c. if write failed, read the error response
Unfortunately, this doesn't seem to work. I figure APNS is sending an error response and I am not reading it back right or I am not setting the socket up right. I have tried the following techniques:-
Use a separate thread per socket to try-read the response if any every 5ms or so.
Use a blocking read after write failure.
Use a final read after remote disconnect.
I have tried this with C# + .NET 4.5 on Windows and Java 1.7 on Linux. In either case, I never seem to get the error response and the socket indicates that no data is available to read.
Are half-closed sockets supported on these operating systems and/or frameworks? There isn't anything that seems to indicate either way.
I know that the way I am setting things up works correctly because if I use a valid token with a valid notification, those do get delivered.
In response to one of the comments, I am using the enhanced notification format so a response should arrive from APNS.
Here is the code I have for C#:-
X509Certificate certificate =
new X509Certificate(#"Foo.cer", "password");
X509CertificateCollection collection = new X509CertificateCollection();
collection.Add(certificate);
Socket socket =
new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Connect("gateway.sandbox.push.apple.com", 2195);
NetworkStream stream =
new NetworkStream(socket, System.IO.FileAccess.ReadWrite, false);
stream.ReadTimeout = 1000;
stream.WriteTimeout = 1000;
sslStream =
new SslStream(stream, true,
new RemoteCertificateValidationCallback(ValidateServerCertificate), null);
sslStream.AuthenticateAsClient("gateway.sandbox.push.apple.com", collection,
SslProtocols.Default, false);
sslStream.ReadTimeout = 10000;
sslStream.WriteTimeout = 1000;
// Task rdr = Task.Factory.StartNew(this.reader);
// rdr is used for parallel read of socket sleeping 5ms between each read.
// Not used now but another alternative that was tried.
Random r = new Random(DateTime.Now.Second);
byte[] buffer = new byte[32];
r.NextBytes(buffer);
byte[] resp = new byte[6];
String erroneousToken = toHex(buffer);
TimeSpan t = (DateTime.UtcNow - new DateTime(1970, 1, 1));
int timestamp = (int) t.TotalSeconds;
try
{
for (int i = 0; i < 1000; ++i)
{
// build the notification; format is published in APNS docs.
var not = new ApplicationNotificationBuilder().withToken(buffer).withPayload(
#'{"aps": {"alert":"foo","sound":"default","badge":1}}').withExpiration(
timestamp).withIdentifier(i+1).build();
sslStream.Write(buffer);
sslStream.Flush();
Console.Out.WriteLine("Sent message # " + i);
int rd = sslStream.Read(resp, 0, 6);
if (rd > 0)
{
Console.Out.WriteLine("Found response: " + rd);
break;
}
// doesn't really matter how fast or how slow we send
Thread.Sleep(500);
}
}
catch (Exception ex)
{
Console.Out.WriteLine("Failed to write ...");
int rd = sslStream.Read(resp, 0, 6);
if (rd > 0)
{
Console.Out.WriteLine("Found response: " + rd); ;
}
}
// rdr.Wait(); change to non-infinite timeout to allow error reader to terminate
I implemented server side for APNS in Java and have problems reading the error responses reliably (meaning - never miss any error response), but I do manage to get error responses.
You can see this related question, though it has no adequate answer.
If you never manage to read the error response, there must be something wrong with your code.
Using a separate thread for reading worked for me, though not 100% reliable.
Use a blocking read after write fail - that's what Apple suggest to do, but it doesn't always work. It's possible that you send 100 messages, and the first has an invalid token, and only after the 100th message you get a write failure. At this point it is sometimes too late to read the error response from the socket.
I'm not sure what you mean there.
If you want to guarantee that the reading of the error responses will work, you should try to read after each write, with a sufficient timeout. This, of course, is not practical for using in production (since it's incredibly slow), but you can use it to verify that your code of reading and parsing the error response is correct. You can also use it to iterate over all the device tokens you have, and find all the invalid ones, in order to clean your DB.
You didn't post any code, so I don't know what binary format you are using to send messages to APNS. If you are using the simple format (that starts with a 0 byte and has no message ID), you won't get any responses from Apple.