We are trying to create a Kafka client using Deno and TCP sockets. As a first step, we are trying to ping a broker we have running on a docker instance using the below code:
import {
Client,
Packet,
Event,
} from 'https://deno.land/x/tcp_socket#0.0.1/mods.ts';
const client = new Client({
hostname: 'localhost',
port: 9092,
});
for (let i = 0; i < 2; i++) {
// Connection open
client.on(Event.connect, (client: Client) => {
console.log('Connect', client.conn?.remoteAddr);
});
// Receive message
client.on(Event.receive, (client: Client, data: Packet) => {
console.log('Receive', data.toString());
});
// Connection close
client.on(Event.close, (client: Client) => {
console.log('Close');
});
// Handle error
client.on(Event.error, (e) => {
console.error(e);
});
// Do
await client.connect(); // Start client connect
await client.write('Hello World'); // Send string data
await client.write(new Uint8Array()); // Send Uint8Array data
client.close();
}
We are successfully pinging the broker, but receive the following error:
[2021-08-13 00:20:36,472] WARN Unexpected error from /172.20.0.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1214606444 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:91)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:169)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:150)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:365)
at org.apache.kafka.common.network.Selector.poll(Selector.java:313)
at kafka.network.Processor.poll(SocketServer.scala:494)
at kafka.network.Processor.run(SocketServer.scala:432)
at java.lang.Thread.run(Thread.java:745)
[2021-08-13 00:28:19,708] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
We understand that this is due to a protocol issue and are trying to determine what the best way forward is to address this issue. Any help would be greatly appreciated
Related
I'm trying to use nginx as a reverse proxy for ssl/tcp sockets (so that I can write my server custom as raw tcp, but have nginx handle the ssl certificates). My use case requires the tcp connections remain alive, but to go idle (no packets back and forth) for extended periods of time (determined by the client, but as long as an hour). Unfortunately, nginx kills my socket connections after the first 10 minutes (timed to within a second) of inactivity, and I haven't been able to find either online or in the docs what actually controls this timeout.
I know that it has to be nginx doing it (not my raw server timing out, or my client's ssl socket), since I can directly connect to the server's raw tcp server without timeout issues, but if I run nginx as a raw tcp reverse proxy (no ssl) it does timeout.
Here's some code to reproduce the issue, note that I've commented out the ssl relevent pieces in nginx because the timeout occurs either way.
/etc/nginx/modules-enabled/test.conf:
stream {
upstream tcp-server {
server localhost:33445;
}
server {
listen 33446;
# listen 33446 ssl;
proxy_pass tcp-server;
# Certs
# ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
}
}
server.js;
const net = require("net");
const s = net.createServer();
s.on("connection", (sock) => {
console.log('Got connection from', sock.remoteAddress, sock.remotePort );
sock.on("error", (err) => {
console.error(err)
clearInterval(i);
});
sock.on("close", () => {
console.log('lost connection from', sock.remoteAddress, sock.remotePort );
clearInterval(i);
});
});
s.listen(33445);
client.js
const net = require('net');
const host = 'example.com';
let use_tls = false;
let client;
let start = Date.now()
// Use me to circumvent nginx, and no timeout occurs
// let port = 33445;
// Use me to use nginx, and no timeouts occur after 10 mins of no RX/TX
let port = 33446;
client = new net.Socket();
client.connect({ port, host }, function() {
console.log('Connected via TCP');
// Include me, and nginx doesn't kill the socket
// setInterval(() => { client.write("ping") }, 5000);
});
client.on('end', function() {
console.log('Disconnected: ' + ((Date.now() - start)/1000/60) + " mins");
});
I've tried various directives in the nginx stream block, but nothing seems to help. Thanks in advance!
I am trying to make webapp over flutter which will connect to HIVE broker. I took the broker name from the official website, set the port number to 8000 just like mentioned there and still get the error message as below:
error is mqtt-client::NoConnectionException: The maximum allowed connection attempts ({1}) were exceeded. The broker is not responding to the connection request message (Missing Connection Acknowledgement?
I really have no clue how to proceed. Can someone please help?
Below is my code:
MqttBrowserClient mq = MqttBrowserClient(
'wss://broker.mqttdashboard.com:8000', '',
maxConnectionAttempts: 1);
/*
MqttBrowserClient mq = MqttBrowserClient('ws://test.mosquitto.org', 'client-1',
maxConnectionAttempts: 1);
*/
class mqttService {
Future<MqttBrowserClient?> connectToServer() async {
try {
final connMess = MqttConnectMessage()
.withClientIdentifier('clientz5tWzoydVL')
.authenticateAs('a14guguliye', 'z5tWzoydVL')
.withWillTopic('willtopic')
.withWillMessage('My Will message')
.startClean() // Non persistent session for testing
.withWillQos(MqttQos.atLeastOnce);
mq.port = 1883;
mq.keepAlivePeriod = 50;
mq.connectionMessage = connMess;
mq.websocketProtocols = MqttClientConstants.protocolsSingleDefault;
mq.onConnected = onConnected;
var status = await mq.connect();
return mq;
} catch (e) {
print("error is " + e.toString());
mq.disconnect();
return null;
}
}
}
That port 8000 may be open but the HiveMQ broker may not be listening.
Make sure that the broker is fully booted and binds to that IP:Port combo.
In the HiveMQ broker startup output, you should see something similar to:
Started Websocket Listener on address 0.0.0.0 and on port 8000
If needed, the HiveMQ Broker configuration documentation is here.
You can use the public HiveMQ MQTT Websocket demo client to test your connection to make sure it's not a local code issue.
As a last option, use Wireshark to monitor MQTT traffic with a filter of tcp.port == 8000 and mqtt
I have this pattern:
channel ESANTE_MPI_CREATE_PATIENT_LISTENER (with a MLLP listener) calls channel ESANTE_MPI_CREATE_PATIENT that calls a TCP destination.
If connection cannot be done in the TCP destination inside ESANTE_MPI_CREATE_PATIENT then this channel reports an error for this destination:(ERROR: ConnectException: Connection refused (Connection refused))
The response transformer does not seem to be called (which is normal as there is no response).
I wonder how I can report the error back to the calling channel ESANTE_MPI_CREATE_PATIENT_LISTENER ?
PS: When tcp destination responds, then I use the response transformer to parse the received frame and create a response message (json error/ok) for the calling channel. Everything works fine here.
My question ends up with: How to trap a Connection refused in a TCP destination to create a response message.
I finally managed this by using the postprocessor script in ESANTE_MPI_CREATE_PATIENT to get the response of the connector and then force a message.
// fake error message prepared for connection refused.
// we put this as the response of the channel destination in order to force a understandable error message.
const sErrorMsg = {
status: "error",
error: "connection refused to eSanté MPI"
};
const TCP_CONNECTOR_ESANTE_MPI_RANK = 2; // WARNING: be sure to take the correct connector ID as displayed into destination.
const TCP_CONNECTOR_ESANTE_MPI_DNAME = 'd' + TCP_CONNECTOR_ESANTE_MPI_RANK; // WARNING: be sure to take the correct connector ID as displayed into destination.
/*
var cms = message.getConnectorMessages(); // returns message but as Immutable
responses. not what we want: we use responseMap instead.
var key = TCP_CONNECTOR_ESANTE_MPI_RANK;
logger.debug(" Response Data=" + cms.get(key).getResponseData());
logger.debug(" Response Data0=" + cms.get(key).getResponseError());
logger.debug(" Response Data1=" + cms.get(key).getResponseData().getError());
logger.debug(" Response Data2=" + cms.get(key).getResponseData().getMessage());
logger.debug(" Response Data3=" + cms.get(key).getResponseData().getStatusMessage());
logger.debug(" Response Data4=" + cms.get(key).getResponseData().getStatus());
*/
var responseMPI = responseMap.get(TCP_CONNECTOR_ESANTE_MPI_DNAME); // return a mutable reponse :-)
if (responseMPI.getStatus()=='ERROR' &&
responseMPI.getStatusMessage().startsWith('ConnectException: Connection refused')) {
// build a error message for this dedicated case
logger.error("connection refused detected");
responseMPI.setMessage(JSON.stringify(sErrorMsg)); // force the message to be responsed.
}
return;
In my app, I created a Route for communicating with a socket.
class _SocketRouteState extends State<SocketRoute> {
#override
void initState() {
super.initState();
try {
WebSocketChannel _channel = IOWebSocketChannel.connect(
Uri.parse('ws://192.168.1.90:9998'),
);
///
/// Start listening to new notifications / messages
///
_channel.stream.listen(
(data) {
debugPrint(data);
},
onDone: () {
debugPrint('ws channel closed');
},
onError: (error) {
debugPrint('ws error $error');
},
);
_channel.sink.add('testing');
} catch (e) {
///
/// General error handling
/// TODO handle connection failure
///
debugPrint('Connection exception $e');
}
}
}
When I run this code, it fails to connect to the socket, though. After waiting for ~2 minutes, Xcode shows me the following error:
flutter: ws error WebSocketChannelException: WebSocketChannelException: SocketException: OS Error: Operation timed out, errno = 60, address = 192.168.1.90, port = 52168
This clearly shows a different port. Could that be the issue? Anyone know why it's connecting on port 52168 instead of 9998?
The problem with the confusing port number is, that the error message is not that great, since it shows the local port being used on your own machine and not the remote port you are trying to connect to. TCP requires a port to be open on both the server and the client so they can communicate both ways. But normally, you are mostly interested in the remote port.
There are an old github issue here about this issue:
https://github.com/dart-lang/sdk/issues/12693
I have a server running on a Raspberry Pi which is accessible from a browser.
http://192.168.1.67:55XXX/?email=a#b.com
yields:-
{Order=[{no=0, day_0=1, price=0, name=PPAC SUPERTHERM 20K, display_colour=blue, notice=1, special_order=false}, {no=1, day_0=1, price=0, name=SLACK 50KG , display_colour=blue, notice=1, special_order=false}, {no=0, day_0=1, price=0, name=PPAC SUPERTHERM 20K, display_colour=blue, day_5=1, notice=1, special_order=false, day_3=1}, {no=1, day_0=1, price=0, name=SLACK 50KG , display_colour=blue, day_5=1, notice=1, special_order=false, day_3=1}], Details={address=xx Farriers Lea, phone=0xxxx 606635, name=Fred Bloggs, mobile=, details=end shed on drive, email=a#b.com}}
I am using VSCODE to debug dart code.
my dart code:-
String remote_ip = '192.168.1.67'; //212.159.118.177';
var remote_port = 55XXX;
Socket socket;
String _dataToBeSent = "http://?email=a#b.com\n";
var reply;
// connect
main(List<String> arguments) async {
await _remoteServerConnect();
}
// REMOTE SERVER CONNECT
Future _remoteServerConnect() async {
// await Socket.connect(remote_ip, remote_port).then((Socket sock){
await Socket.connect(remote_ip, remote_port).then((Socket sock) {
socket = sock;
print('Got connected ${socket.remoteAddress}');
socket.listen(dataHandler,
onError: errorHandler, onDone: doneHandler, cancelOnError: false);
}).catchError((AsyncError e) {
print("Unable to connect: $e");
exit(1);
});
}
void dataHandler(data) async {
await print('"'+String.fromCharCodes(data).trim()+'"');
if (String.fromCharCodes(data).trim().endsWith('html')) {
print("Send Data = $_dataToBeSent");
socket.add(utf8.encode(_dataToBeSent));
// socket.writeln(_dataToBeSent);
socket.flush();
await Future.delayed(Duration(seconds: 5));
}
}
void errorHandler(error, StackTrace trace) {
print(error);
}
void doneHandler() {
// socket.destroy();
exit(0);
}
The dart debug consol yields:-
Connecting to VM Service at ws://127.0.0.1:54799/kE9Xa1JQclk=/ws
Got connected InternetAddress('192.168.1.67', IPv4)
"HTTP/1.1 200 OK" <sent by server
"ContentType: text/html" <sent by server
Send Data = http://?email=a#b.com
Exited
The server consol yields:-
Server is ready
WEB Client connected: /192.168.1.66
05.08/10:34:21.17 - Waiting for command..
05.08/10:34:31.80 - Socket Timeout
05.08/10:34:31.82 - Done -------------
05.08/10:34:31.82 - Waiting for command..
Connection has been closed
Server is ready
It would seem the
socket.add(utf8.encode(_dataToBeSent));
// socket.writeln(_dataToBeSent);
socket.flush();
did not send data to the server?? Why?? any ideas gratefully received!
Disabling my dev machines firewall did NOT improve answer!
Steve
The server used
final DataInputStream in = new DataInputStream(clientSocket.getInputStream());
while (estimatedTime < 60000) {
report("Waiting for command..");
try {
// in.readNBytes(command.data, 0, command.length);
in.readFully(command.data);
to receive the data command.data is 100bytes long so was waiting for the complete input. I padded to 100 bytes worked fine - will sort a better solution later.