I am trying to get logstash 2.3.3 websocket input working.
Logstash: https://download.elastic.co/logstash/logstash/logstash-2.3.3.tar.gz
Websocket Input Plugin for Logstash: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-websocket.html
Websocket server: https://github.com/joewalnes/websocketd/releases/download/v0.2.11/websocketd-0.2.11-linux_amd64.zip
Websocket Client: Chrome Plugin "Simple Web Socket Client"
I am aware of a bug filed last year logstash 1.5.0 and the websocket input plugin. https://github.com/logstash-plugins/logstash-input-websocket/issues/3 I have also received those same error messages, although I can't reproduce them anymore. The following is my current procedure and result. I am hoping that bug has since been fixed and I just can't find the correct config.
First I installed the plugin and confirmed it is listed as installed.
/app/bin/logstash-plugin list | grep "websocket"
Next, I checked that logstash was working with the following config
input {
stdin { }
}
output {
file {
path => "/app/logstash-2.3.3/logstash-log.txt"
}
}
Logstash worked.
/app/logstash-2.3.3/bin/logstash agent --config /app/logstash-2.3.3/logstash.conf
Hello World
The file logstash-log.txt contained:
{"message":"Hello World","#version":"1","#timestamp":"2016-07-05T20:04:14.850Z","host":"server-name.domain.com"}
Next I opened port 9300
I wrote a simple bash script to return some numbers
#!/bin/bash
case $1 in
-t|--to)
COUNTTO=$2
shift
;;
esac
shift
printf 'Count to %i\n' $COUNTTO
for COUNT in $(seq 1 $COUNTTO); do
echo $COUNT
sleep 0.1
done
I started up websocketd pointing to my bash script
/app/websocketd --port=9300 /app/count.sh --to 7
I opened Simple Web Socket Client in Chrome and connected
ws://server-name.domain.com:9300
Success! It returned the following.
Count to 7
1
2
3
4
5
6
7
At this point I know websocketd works and logstash works. Now is when the trouble starts.
Logstash websocket input configuration file
input {
websocket {
codec => "plain"
url => "ws://127.0.0.1:9300/"
}
}
output {
file {
path => "/app/logstash-2.3.3/logstash-log.txt"
}
}
Run configtest
/app/logstash-2.3.3/bin/logstash agent --config /app/logstash-2.3.3/logstash.conf --configtest
Receive "Configuration OK"
Start up websocketd
/app/websocketd --port=9300 /app/logstash-2.3.3/bin/logstash agent --config /app/logstash-2.3.3/logstash.conf
Back in Simple Web Socket Client, I connect to ws://server-name.domain.com:9300. I see a message pop up that I started a session.
Tue, 05 Jul 2016 20:07:13 -0400 | ACCESS | session | url:'http://server-name.domain.com:9300/' id:'1467732248361139010' remote:'192.168.0.1' command:'/app/logstash-2.3.3/bin/logstash' origin:'chrome-extension://pfdhoblngbopfeibdeiidpjgfnlcodoo' | CONNECT
I try to send "hello world". Nothing apparent happens on the server. After about 15 seconds I see a disconnect message in my console window. logstash-log.txt is never created.
Any ideas for what to try? Thank you!
UPDATE 1:
I tried putting the following in a bash script called "launch_logstash.sh":
#!/bin/bash
exec /app/logstash-2.3.3/bin/logstash agent --config /app/logstash-2.3.3/logstash.conf
Then I started websocketd like so:
/app/websocketd --port=9300 /app/logstash-2.3.3/bin/launch_logstash.sh
Same result; no success.
Upon reading the websocketd documentation more closely, it sends the data received on the socket to a program's stdin. I was trying to listen to a socket in my logstash config, but the data is actually going to that app's stdin. I changed my config to this:
input {
stdin { }
}
output {
file {
path => "/app/logstash-2.3.3/logstash-log.txt"
}
}
Then launched websocketd like this:
/app/websocketd --port=9300 /app/logstash-2.3.3/bin/logstash agent --config /app/logstash-2.3.3/logstash.conf
So in short, until logstash-websocket-input implements their server option, stdin{} and stdout{} are the input and output if using websocketd as the web server.
Related
I'm running an application which builds and sends ICMP ECHO requests to a few different ip addresses. The application is written in Crystal. When attempting to open a socket from within the crystal docker container, Crystal raises an exception: Permission Denied.
From within the container, I have no problem running ping 8.8.8.8.
Running the application on macos, I have no problem.
Reading the https://docs.docker.com/engine/security/apparmor/ and https://docs.docker.com/engine/security/seccomp/ pages on apparmor and seccomp I was sure I'd found the solution, but the problem remains unresolved, even when running as docker run --rm --security-opt seccomp=unconfined --security-opt apparmor=unconfined socket_permission
update/edit: After digging into capabilities(7), I added the following line to my dockerfile: RUN setcap cap_net_raw+ep bin/ping trying to let the socket get opened but without change.
Thanks!
Relevant crystal socket code, full working code sample below:
# send request
address = Socket::IPAddress.new host, 0
socket = IPSocket.new Socket::Family::INET, Socket::Type::DGRAM, Socket::Protocol::ICMP
socket.send slice, to: address
Dockerfile:
FROM crystallang/crystal:0.23.1
WORKDIR /opt
COPY src/ping.cr src/
RUN mkdir bin
RUN crystal -v
RUN crystal build -o bin/ping src/ping.cr
ENTRYPOINT ["/bin/sh","-c"]
CMD ["/opt/bin/ping"]
Running the code, first native, then via docker:
#!/bin/bash
crystal run src/ping.cr
docker build -t socket_permission .
docker run --rm --security-opt seccomp=unconfined --security-opt apparmor=unconfined socket_permission
And finally, a 50 line crystal script which fails to open a socket in docker:
require "socket"
TYPE = 8_u16
IP_HEADER_SIZE_8 = 20
PACKET_LENGTH_8 = 16
PACKET_LENGTH_16 = 8
MESSAGE = " ICMP"
def ping
sequence = 0_u16
sender_id = 0_u16
host = "8.8.8.8"
# initialize packet with MESSAGE
packet = Array(UInt16).new PACKET_LENGTH_16 do |i|
MESSAGE[ i % MESSAGE.size ].ord.to_u16
end
# build out ICMP header
packet[0] = (TYPE.to_u16 << 8)
packet[1] = 0_u16
packet[2] = sender_id
packet[3] = sequence
# calculate checksum
checksum = 0_u32
packet.each do |byte|
checksum += byte
end
checksum += checksum >> 16
checksum = checksum ^ 0xffff_ffff_u32
packet[1] = checksum.to_u16
# convert packet to 8 bit words
slice = Bytes.new(PACKET_LENGTH_8)
eight_bit_packet = packet.map do |word|
[(word >> 8), (word & 0xff)]
end.flatten.map(&.to_u8)
eight_bit_packet.each_with_index do |chr, i|
slice[i] = chr
end
# send request
address = Socket::IPAddress.new host, 0
socket = IPSocket.new Socket::Family::INET, Socket::Type::DGRAM, Socket::Protocol::ICMP
socket.send slice, to: address
# receive response
buffer = Bytes.new(PACKET_LENGTH_8 + IP_HEADER_SIZE_8)
count, address = socket.receive buffer
length = buffer.size
icmp_data = buffer[IP_HEADER_SIZE_8, length-IP_HEADER_SIZE_8]
end
ping
It turns out the answer is that Linux (and by extension docker) does not give the same permissions that macOS does for DGRAM sockets. Changing the socket declaration to socket = IPSocket.new Socket::Family::INET, Socket::Type::RAW, Socket::Protocol::ICMP allows the socket to connect under docker.
A little more still is required to run the program in a non-root context. Because raw sockets are restricted to root, the binary must also be issued the correct capability for access to a raw socket, CAP_NET_RAW. However, in docker, this isn't necessary. I was able to get the program to run outside of super-user context by running sudo setcap cap_net_raw+ep bin/ping. This is a decent primer on capabilities and the setpcap command
MacOS doesn't use the same system of permissions, so setcap is just an unrecognized command. As a result, to get the above code to compile and run successfully on macOS without super-user context, I changed the socket creation code to:
socket_type = Socket::Type::RAW
{% if flag?(:darwin) %}
socket_type = Socket::Type::DGRAM
{% end %}
socket = IPSocket.new Socket::Family::INET, socket_type, Socket::Protocol::ICMP
Applying the CAP_NET_RAW capability for use in linux happens elsewhere in the build process if needed.
With those changes, I'm not seeing any requirement for changes to seccomp or apparmor from the default shipped with Docker in order to run the program.
i am trying to send an xml.bz2 file to mongodb and i am using Mac and writing everything in the terminal.
In the https://github.com/spencermountain/wikipedia-to-mongodb there is a step 4 that i stucked,
4 get going
load it into mongo (10-15 minutes)
wikipedia-to-mongodb ./afwiki-latest-pages-articles.xml.bz2
I have a directory called wikipedia-to-mongodb and i downloaded file with wget. I connected mongod,
when i write the terminal that $ wikipedia-to-mongodb ./afwiki-latest-pages-articles.xml.bz2 it is not working. What am i doing wrong? I am kind of new this environment.
I tried 2 inputs below you can see it
1)
input= ./afwiki-latest-pages-articles.xml.bz2
[thread1] SyntaxError: missing name after . operator #(shell):1:22
2)
input= wikipedia-to-mongodb ./afwiki-latest-pages-articles.xml.bz2
-bash: wikipedia-to-mongodb: command not found
I've built nginx from sources with lua support and I'm able to run server-side scripts like this:
http {
lua_package_path '/usr/local/share/lua/5.1/?.lua;;';
server {
listen 80;
location /hi {
content_by_lua '
ngx.header["Content-Type"] = "text/plain;charset=utf-8"
ngx.say("Hello world!")
--local s = require("socket")
ngx.say(_VERSION);
';
}
}
}
So when I access http://localhost/hi, I get this output:
Hello world!
Lua 5.1
If I uncomment line local s = require("socket") then I get following error in my browser:
Unable to load page
Problem occurred while loading the URL http://localhost/hi
Connection terminated unexpectedly
soucket.lua presents in this folder:
root#debian:/usr/local/share/lua/5.1# ls
ltn12.lua mime.lua socket socket.lua
UPDATE: adding cpath doesn't help:
lua_package_cpath '/usr/local/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/socket/?.so;;';
# ls /usr/local/lib/lua/5.1/socket/*.so
/usr/local/lib/lua/5.1/socket/core.so /usr/local/lib/lua/5.1/socket/serial.so /usr/local/lib/lua/5.1/socket/unix.so
# ls /usr/local/share/lua/5.1/
ltn12.lua mime.lua socket socket.lua
How can I fix/diagnose this problem?
Thanx
If the socket.lua is the diegonehab/luasocket,
then it requires socket/core.so
You need to specify the lua_package_cpath
cpath is for compiled shared library modules (.so), path is for text modules (.lua)
/usr/local/lib/lua/5.1/?.so - most common value for cpath
Do not try to use LuaSocket inside nginx. LuaSocket is a blocking library and nginx is non-blocking so you will have problems. Check out ngx.socket.tcp instead. Its API is compatible with socket.tcp but it is non-blocking.
I'm initializing spot instances running a derivative of the standard Ubuntu 13.04 AMI by pasting a shell script into the user-data field.
This works. The script runs. But it's difficult to debug because I can't figure out where the output of the script is being logged, if anywhere.
I've looked in /var/log/cloud-init.log, which seems to contain a bunch of stuff that would be relevant to debugging cloud-init, itself, but nothing about my script. I grepped in /var/log and found nothing.
Is there something special I have to do to turn logging on?
The default location for cloud init user data is already /var/log/cloud-init-output.log, in AWS, DigitalOcean and most other cloud providers. You don't need to set up any additional logging to see the output.
You could create a cloud-config file (with "#cloud-config" at the top) for your userdata, use runcmd to call the script, and then enable output logging like this:
output: {all: '| tee -a /var/log/cloud-init-output.log'}
so I tried to replicate your problem. Usually I work in Cloud Config and therefore I just created a simple test user-data script like this:
#!/bin/sh
echo "Hello World. The time is now $(date -R)!" | tee /root/output.txt
echo "I am out of the output file...somewhere?"
yum search git # just for fun
ls
exit 0
Notice that, with CloudInit shell scripts, the user-data "will be executed at rc.local-like level during first boot. rc.local-like means 'very late in the boot sequence'"
After logging in into my instance (a Scientific Linux machine) I first went to /var/log/boot.log and there I found:
Hello World. The time is now Wed, 11 Sep 2013 10:21:37 +0200! I am
out of the file. Log file somewhere? Loaded plugins: changelog,
kernel-module, priorities, protectbase, security,
: tsflags, versionlock 126 packages excluded due to repository priority protections 9 packages excluded due to repository
protections ^Mepel/pkgtags
| 581 kB 00:00
=============================== N/S Matched: git =============================== ^[[1mGit^[[0;10mPython.noarch : Python ^[[1mGit^[[0;10m Library c^[[1mgit^[[0;10m.x86_64 : A fast web
interface for ^[[1mgit^[[0;10m
...
... (more yum search output)
...
bin etc lib lost+found mnt proc sbin srv tmp var
boot dev home lib64 media opt root selinux sys usr
(other unrelated stuff)
So, as you can see, my script ran and was rightly logged.
Also, as expected, I had my forced log 'output.txt' in /root/output.txt with the content:
Hello World. The time is now Wed, 11 Sep 2013 10:21:37 +0200!
So...I am not really sure what is happening in you script.
Make sure you're exiting the script with
exit 0 #or some other code
If it still doesn't work, you should provide more info, like your script, your boot.log, your /etc/rc.local, and your cloudinit.log.
btw: what is your cloudinit version?
Running CentOS. xinetd.d/clhtest entry is as follows:
service clhtest
{
disable = no
port = 8020
socket_type = stream
protocol = tcp
wait = no
user = charrison
passenv = PATH
server = /home/charrison/bin/clhtest
}
In debugging this I need to write to a file. I set the server process up to open /home/charrison/log/foo.txt as one of the first steps (note user=charrison), but it doesn't - and I assume it tries to. When I launch the server program from command line it opens the file successfully.
I suspect the umask parameter may be needed, but I don't know what it defaults to.
Any hints?