I try to transfer data from [tag:Kafka to udp or tcp port.
I tried to find a apache-kafka-connect to achieve this goal, but I couldn't find anything related.
Do you have any suggestion to find the best way to create such a pipeline?
You can use Benthos for this. E.g.
input:
label: "Kafka Data"
kafka:
addresses: ["localhost:9092"]
topics: ["input"]
target_version: 2.0.0
consumer_group: "example"
output:
label: "Socket Out"
socket:
network: "tcp" # or udp
address: "host:1234"
codec: lines
Related
I know this is an older version of JBoss, but I'm really scratching my head as to what to look for with the errors provided by the log on startup. It seems that it is unhappy with the configuration for jgroups? Here are the log entires that might help:
15:33:38,696 ERROR [STDERR] 102 [DownHandler (UDP)] INFO org.jgroups.protocols.UDP
15:33:38,699 ERROR [STDERR] 105 [DownHandler (UDP)] INFO org.jgroups.protocols.UDP
15:33:41,529 ERROR [STDERR] 2935 [main] INFO org.jboss.cache.TreeCache
15:33:43,333 ERROR [URLDeploymentScanner] Incomplete Deployment listing:
...
I apologize for the lack of any detailed info, but everything in the socket information seems correct. I know this is a long shot, thank you to anyone who can help.
Edit: Link to pastebin for more detailed log: https://pastebin.com/Z5yCjq99
I'd suggest you take a look at JBoss 4.2 Clustering Guide whether you got the UDP route or the TCP route - also, I dont know how involved a network setup you have with switches and routers - but our IP engineering team had to setup UDP multicast IP's and ports when we did JBoss EAP 6 clusters
From the guide, and with regards to TCP
Alternatively, a JGroups-based cluster can also work over TCP connections. Compared with UDP, TCP generates more network traffic when the cluster size increases. TCP is fundamentally a unicast protocol. To send multicast messages, JGroups uses multiple TCP unicasts. To use TCP as a transport protocol, you should define a TCP element in the JGroups Config element. Here is an example of the TCP element.
<TCP start_port="7800"
bind_addr="192.168.5.1"
loopback="true"
down_thread="false" up_thread="false"/>`
Theres a lot more information on the page - and if you click one page back, its the UDP setup
UDP is the preferred protocol for JGroups. UDP uses multicast or multiple unicasts to send and receive messages. If you choose UDP as the transport protocol for your cluster service, you need to configure it in the UDP sub-element in the JGroups Config element. Here is an example.
<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
mcast_port="${jboss.hapartition.mcast_port:45566}"
tos="8"
ucast_recv_buf_size="20000000"
ucast_send_buf_size="640000"
mcast_recv_buf_size="25000000"
mcast_send_buf_size="640000"
loopback="false"
discard_incompatible_packets="true"
enable_bundling="false"
max_bundle_size="64000"
max_bundle_timeout="30"
use_incoming_packet_handler="true"
use_outgoing_packet_handler="false"
ip_ttl="${jgroups.udp.ip_ttl:2}"
down_thread="false" up_thread="false"/>`
The next 20 pages of clicks or so go into all the different types of setups you can use!
Update I also just found the better pdf version of the documentation!
I want to make 2 devices communicate via sockets.
I use this code for the client socket:
Socket socket = Gdx.net.newClientSocket(Net.Protocol.TCP, adress, 1337, socketHints);
(SocketHints: timeout = 4000)
I get a GdxRuntimeException each time this line is being executed. What is wrong with the socket?
Screenshot of stack trace
You get that message because the socket couldn't be opened.
Note the last line about the return in the API:
newClientSocket:
Socket newClientSocket(Net.Protocol protocol,
java.lang.String host,
int port,
SocketHints hints)
Creates a new TCP client socket that connects to the given host and port.
Parameters:
host - the host address
port - the port
hints - additional SocketHints used to create the socket. Input null to use the default setting provided by the system.
Returns:
GdxRuntimeException in case the socket couldn't be opened
Try doing some debugging to find out why you are getting this error.
Is the port already in use? Are you trying to open more than one connection on the same port? Is the server IP valid? Maybe something else is causing the issue?
I am working on a way to find a mapping between the application and the url it accesses at a given timestamp. For this I first got the mapping of tcp port number to url by parsing the packet dump file from time t1 to t2. I wrote this python code for parsing:
import dpkt
f = open('/home/nachiket/Desktop/Research work/my project/s1-pcapFilesParsed/pcap files/2017_04_01_023856.pcap')#2017_03_30_013908.pcap #2017_03_02_010455.pcap
pcap = dpkt.pcap.Reader(f)
g=open('s1final.txt','w')
g.write("\n")
list=set()
for ts, buf in pcap:
eth = dpkt.ethernet.Ethernet(buf)
ip = l2.data
tcp = ip.data
if type(tcp)!=str:
if (tcp.dport == 80 or tcp.dport == 443) and len(tcp.data) > 0:
try:
http1 = dpkt.http.Request(tcp.data)
if http1.uri!='/_ping':
g.write("p "+str((hex(tcp.sport).split('x')[-1])).upper()) #converted to hex
except (dpkt.dpkt.NeedData, dpkt.dpkt.UnpackError):
continue
Now I mapped the tcp port to uid from /proc/net/tcp and from uid I got application name. Thus I merged the two files and got the tcp port to application name mapping at a given timestamp.
The python code:
#! /bin/bash
for i in {1..80}
do
adb shell "date +%s" > dump/netdump$i
adb shell "cat /proc/net/tcp" >> dump/netdump$i
echo finished: $i
sleep 1
done
The problem I faced is the tcp ports from running the first code doesnt match at all with the tcp ports running from second code. If was supposed to merge them on basis on common tcp ports used by application at a given timestamp and hence to find the application to url mapping at a given time. But the ports doesnt match.
I know that the /proc/net/tcp is used for getting details of each socket but it shouldn't affect the tcp ports I guess.
PS: I am a beginner and have done research before submitting question
Thank you
I have an Amazon EC2 instance running and I am trying to set up StatsD+InfluxDB+Grafana. InfluxDB and Grafana work well (and Grafana sees the data from InfluxDB), but I can't manage to get any data from StatsD to InfluxDB.
I have a domain registered, which is pointed to my EC2 instance with an Elastic IP.
What I can see is that:
- I can perfectly interact with the InfluxDB database (including inserting values) when I don't use StatsD
- StatsD seems to be getting the data I randomly generate from Python (I can see it in its logs). It is sent through the port 8125 to StatsD.
- UTC packets sent from StatsD to InfluxDB through port 8086 seem to not be getting to InfluxDB (or not sending....?)
- Port 8086 is open on my AWS security settings for both TCP and UDP
- Port 8125 is open on my AWS security settings for UDP
I am wondering whether some of my settings are wrong, but I don't know what else to try:
InfluxDB configuration file contains:
# hostname = "localhost"
hostname = MYDOMAIN.com
[[udp]]
enabled = true
bind-address = ":8086"
database = "MY_DATABASE"
retention-policy = ""
batch-size = 1000 # will flush if this many points get buffered
batch-pending = 10 # number of batches that may be pending in memory
batch-timeout = "1s" # will flush at least this often even if we haven't hit buffer limit
read-buffer = 0 # UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max.
udp-payload-size = 65536
My StatsD configuration file contains (among other things) the following lines:
{
influxdb: {
/*
host: '127.0.0.1', // InfluxDB host (default 127.0.0.1)
*/
host: 'MYDOMAIN.com', // InfluxDB host (default 127.0.0.1)
port: 8086, // InfluxDB port (default 8086)
database: 'MY_DATABASE', // InfluxDB db instance (required)
username: 'MY_USERNAME', // InfluxDB db username (required)
password: 'MY_PASSWORD', // InfluxDB db password (required)
flush: {
enable: true // enable regular flush strategy (default true)
},
proxy: {
enable: false, // enable the proxy strategy (default false)
suffix: 'raw', // metric name suffix (default 'raw')
flushInterval: 1000
}
},
port: 8125, // statsD port
backends: ['./backends/console'],
debug: true,
legacyNamespace: false
}
As far as I understand, the process is:
Python --> Port 8125 --> StatsD --> Port 8086 --> InfluxDB
Is there a need to use something like Telegraf or statsd-influxdb-backend to connect StatsD and InfluxDB?
I would truly appreciate any helps because I have been trying to set it up for hours and I don't see what could be wrong.
Thanks!
The part of the stack I'm not sure about is your StatsD server. It's probably having a problem posting the data to InfluxDB. If you use Telegraf instead it should "just work". Telegraf can act as a StatsD server (among many other things) and send data to InfluxDB via either UDP or the regular HTTP protocol.
I'm working with tcl script under ubuntu 12.04, and I'm facing some problem when I try to configure a multicast socket. What I'm trying to do is to forward traffic from some socket to a multicast one, but I don't know why although the multicast socket is created well,apparently; it isn't bound to the multicast group I want to.
This is the script I'm using
#!/bin/sh
# test.tcl \
exec tclsh "$0" ${1+"$#"}
package require udp
set multicastPort "50003"
proc connector {unicastIP multicastIP port {protocol tcp}} {
if { [string equal $protocol "tcp"] } {
socket -server serverTCP -myaddr $unicastIP $port
puts "tcp"
} elseif {[string equal $protocol "udp" ] } {
serverUDP $unicastIP $multicastIP $port
puts "udp"
}
}
proc serverUDP {unicastIP multicastIP port} {
global multicastPort
set socketUDP [udp_open $port]
puts " $unicastIP"
fconfigure $socketUDP -blocking false -translation binary -buffering none -remote [list $unicastIP $port]
#fileevent $socketUDP readable [list gettingData $socketUDP]
set multicastSocket [udp_open $multicastPort]
udp_conf $multicastSocket -ttl 4
fconfigure $multicastSocket -blocking false -translation binary -buffering none -mcastadd $multicastIP -remote [list $multicastIP $port]
fileevent $socketUDP readable [list forwarding $socketUDP $multicastSocket ]
#puts $socketUDP "hello!"
#flush $socketUDP
}
proc forwarding {socketSrc socketDst} {
set data [read -nonewline $socketSrc]
puts "Read data-> $data"
puts -nonewline $socketDst $data
puts "Written data-> [read -nonewline $socketDst]"
}
connector 127.0.0.1 224.0.1.1 50000 udp
vwait forever
However if I run the script and check out the ports in my system, the multicast port is not assigned the proper multicast IP as you can see
~$ netstat -ptnlu
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 0.0.0.0:50000 0.0.0.0:* 3334/tclsh
udp 0 0 0.0.0.0:50003 0.0.0.0:* 3334/tclsh
Could anyone tell me the reason?
THanks in advance,
Regards!
AFAIK, that is OK. I have a multicast daemon in production using Tcl and its udp package, and netstat and ss tools also show me the socket as listening on the wildcard address.
"The trick" here, I suppose, is that multicasting is one level up the stack: joining a multicast group is not merely opening a socket or an endpoint on the group address but rather sending a very real IGMP "join" message to the local transport segment (Ethernet, in most deployments) and further communicating with the nearby IGMP routers (again, on Ethernet, they're mostly switches).
So, in your case, just fire up tcpdump and see what it dumps when you start your program. A useful call to tcpdump looks something like this:
tcpdump -i eth0 -n 'igmp and host 224.0.1.1'
To observe UDP traffic exchanges use
tcpdump -i eth0 -n 'udp and host 224.0.1.1 and port 50000'