I intend to set the "don't fragment" flag bit in Go, the same thing as this post while it is in C. I checked the constant list but I didn't find this option. So what is the corresponding option in Go?
Thanks in advance!
How to set "don't fragment" flag bit for TCP packet in Go?
First up you should know that TCP really doesn't like IP fragments. Most if not all major implementations avoid fragmentation for TCP segments by using path MTU discovery.
The TL;DR is that the typical IP packet containing a TCP segment has a DF bit set. You can (and should) try this out. Here I am sniffing a few seconds of traffic between my machine and stackoverflow.com:
% tshark -w /tmp/tcp.pcap tcp and host stackoverflow.com
<wait a few seconds>
% tshark -r /tmp/tcp.pcap -T fields -e ip.flags | sort | uniq -c
186 0x00000002
0x02 means the DF bit is set. I confess in other captures I have seen the occasional TCP segment in an IP packet without a DF bit; I suspect rfc1191 has an explanation for this.
Now back to your question, I think there's no portable way to set the DF bit and this is a more widespread question (there isn't even a POSIX-portable way).
There is (likely) an escape hatch in the relevant package for your implementation under golang.org/x/sys.
For example, on a Unix that supports IP_DONTFRAG such as FreeBSD one could use unix.SetsockoptInt and dig the relevant constant value.
On Linux there is not IP_DONTFRAG, as you discovered from the question you linked. The workaround seems to be to use IP_MTU_DISCOVER which happens to be available as a constant in the unix package. You can use that same unix.SetsockoptInt to set it.
Related
I know the netstat command on both windows (powershell) and mac/linux will return information about the ports in use
Is there some command (or option) to retrieve a full list of all TCP ports that exist on a device, irrespective of whether or not they're in use?
Note: I see an excellent list of IANA assigned port numbers, which goes up to ~65000. But I can't see a specific max, and I'm not sure if all laptops/servers/VMs have that many ports.
Apologies in advance for my lack of knowledge around how ports work; they're something I rarely have need deal with directly.
In Linux I can use an ioctl call with FIONREAD to get the number of bytes for the next UDP packet.
That doesn't work on OSX and instead I have to use getsockopt call with SO_NREAD to determine the number of bytes for the packet.
Is there a way I can avoid doing a peek or a read into a big buffer followed by a copy to achieve the same result under BSD platforms?
FIONREAD works in BSD. In fact that's where it came from. But it returns the total number of bytes available to be read without blocking, which could be more than one datagram.
EDIT You could try using MSG_PEEK|MSG_TRUNC and supplying a buffer length of zero, or one if it doesn't like that. It should return you the real length.
I hate this situation (MACOSX)
$ perl -MSocket -le 'print SOMAXCONN'
128
How can I deal with more than 128 sockets? I'm using IO::Socket::INET (multiplexing with IO::Select at this time) but I have this limitation.
I was thinking in pre-fork N process (each dealing with 128 using select or poll) but I don't know if it can work. Some Ideas?
This is a configurable limitation at the OS X kernel level. Fortunately its pretty easy to fix, but only if you have root. Here's an article explaining how to tweak the kernel configuration. The relevant entry is kern.ipc.somaxconn.
Otherwise you can try using threads, not sure if each thread gets its own allocation of sockets, or any number of pre-forking servers on CPAN.
I read somewhere that it wasn't possible to specify multiple ports to
exclude for shellcode is that correct? if not is the following
correct?
List of ports you want to look for SHELLCODE on.
portvar SHELLCODE_PORTS ![21,25,80,143,587,8889]
I'm using snort 2905
Try it. If Snort doesn't like the syntax, it should throwback a fatal parsing error. I believe, though, that syntax is better implemented as:
portvar SHELLCODE_PORTS [21,25,80,143,587,8889]
I.e., without the negation, and then used in a rule thus:
alert tcp any any <> any !SHELLCODE_PORTS ( ... )
You could also do:
portvar SHELLCODE_PORTS [!21,!25,!80,!143,!587,!8889]
Which should be valid syntax. Haven't tested, though. YMMV.
You want !$SHELLCODE_PORTS
You are missing the dollar sign.
I'm trying to choose between pipes and unix sockets for an IPC mechanism.
Both support the select() and epoll() functions which is great.
Now, pipes have a 4kB (as of today) "atomic" write, which is guaranteed by the Linux Kernel.
Does such a feature exists in the case of unix sockets? I couldn't find any document stating this explicitely.
Say I use a UNIX socket and I write x bytes of data from my client. Am I sure that these x bytes will be written on the server end of the socket when my server's select() cracks?
On the same subject, would using SOCK_DGRAM ensure that writes are atomic (if such a guarantee is possible), since datagrams are supposed to be single well-defined messages?
What would then be the difference using SOCK_STREAM as a transfer mode?
Thanks in advance.
Pipes
Yes the non-blocking capacity is usually 4KB, but for maximum portability you'd probably be better off using the PIPE_BUF constant. An alternative is to use non-blocking I/O.
More information than you want to know in man 7 pipe.
Unix datagram sockets
Writes using the send family of functions on datagram sockets are indeed guaranteed to be atomic. In the case of Linux, they're reliable as well, and preserve ordering. (which makes the recent introduction of SOCK_SEQPACKET a bit confusing to me) Much information about this in man 7 unix.
The maximum datagram size is socket-dependent. It's accessed using getsockopt/setsockopt on SO_SNDBUF. On Linux systems, it ranges between 2048 and wmem_max, with a default of wmem_default. For example on my system, wmem_default = wmem_max = 112640. (you can read them from /proc/sys/net/core) Most relevant documentation about this is in man 7 socket around the SO_SNDBUF option. I recommend you read it yourself, as the capacity doubling behavior it describes can be a bit confusing at first.
Practical differences between stream and datagram
Stream sockets only work connected. This mostly means they can only communicate with one peer at a time. As streams, they're not guaranteed to preserve "message boundaries".
Datagram sockets are disconnected. They can (theoretically) communicate with multiple peers at a time. They preserve message boundaries.
[I suppose the new SOCK_SEQPACKET is in between: connected and boundary-preserving.]
On Linux, both are reliable and preserve message ordering. If you use them to transmit stream data, they tend to perform similarly. So just use the one that matches your flow, and let the kernel handle buffering for you.
Crude benchmark comparing stream, datagram, and pipes:
# unix stream 0:05.67
socat UNIX-LISTEN:u OPEN:/dev/null &
until [[ -S u ]]; do :;done
time socat OPEN:large-file UNIX-CONNECT:u
# unix datagram 0:05.12
socat UNIX-RECV:u OPEN:/dev/null &
until [[ -S u ]]; do :;done
time socat OPEN:large-file UNIX-SENDTO:u
# pipe 0:05.44
socat PIPE:p,rdonly=1 OPEN:/dev/null &
until [[ -p p ]]; do :;done
time socat OPEN:large-file PIPE:p
Nothing statistically significant here. My bottleneck is likely reading large-file.
Say I use a UNIX socket and I write x
bytes of data from my client. Am I
sure that these x bytes will be
written on the server end of the
socket when my server's select()
cracks?
If you are using AF_UNIX SOCK_STREAM socket, there is no such guarantee, that is, data written in one write/send() may require more than one read/recv() call on the receiving side.
On the same subject, would using
SOCK_DGRAM ensure that writes are
atomic (if such a guarantee is
possible), since datagrams are
supposed to be single well-defined
messages?
On there other hand, AF_UNIX SOCK_DGRAM sockets are required to preserve the datagram boundaries and be reliable. You should get EMSGSIZE error if send() can not transmit the datagram atomically. Not sure what happens for write() as the man page does not say that it can report EMSGSIZE (although man pages sometimes do not list all errors returned). I would try overflowing the receiver's buffer with big sized datagrams to see which errors exactly send/write() report.
One advantage of using UNIX sockets over pipes is the bigger buffer size. I don't remember exactly what is the limit of pipe's kernel buffer, but I remember not having enough of it and not being able to increase it (it is a hardcoded kernel constant). fast_producer_process | slow_consumer_process was orders of magnitude slower than fast_producer_process > file && file > slow_consumer_process due to insufficient pipe buffer size.