How to fix a python "pyverbs" application with RDMA write getting a not acknowledge (NACK) - rdma

Im currently working on a RoCE (RDMA over Converged Ethernet) python application with the library pyverbs. First, i want to do a simple loopback test with an RDMA Write. I tested the setup with ib_write_bw from perftest, which worked like a charm.
This is my setup:
OS: Ubuntu 20.04.05 LTS
Kernel: 5.15.0-56-generic
NIC: Mellanox ConnectX-5 MC516A-GCA_Ax 50Gbe dual-port QSFP28
I'm developing the application with a jupyter notebook. The ports are connected together with a QSFP28 cable. I set up a "client" and "server" on the same system. Both use one port of the NIC. The client performs the "RDMA Write" action. In the future, metadata will be exchanged over tcp, but for ease of debugging i exchange metadata locally in the same notebook.
Now i was able to perform an "RDMA Write" action and capture the packets.
Captured RDMA packets
I keep getting not acknowledeges (NACK) from the "server". The RDMA Write packet look correct to me. It got the right payload and the headers are the same as i configured (i can post it if it would help).
I got three ideas in my head, why it would work.
wrong server memory adress/rkey used for rdma write
missing flags at server memory allocation
missing/wrong flags at server queue pair modification
I tried all different combinations of flags and values at the queue pair modification and memory allocation.
I start the NIC in the shi
ell with
sudo mst start
Then i run my python application. I posted some codesnipped below, which i am not sure i implemented those right.
Server and client qp modification init to RTR
gid_index = 0
port_num = 1
server_attr.qp_state = e.IBV_QPS_RTR
server_attr.path_mtu = e.IBV_MTU_4096
server_attr.rq_psn = 0
server_attr.min_rnr_timer = 12
server_attr.max_dest_rd_atomic = 10
server_attr.dest_qp_num = client_qp.qp_num
server_attr.qp_access_flags = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_READ | e.IBV_ACCESS_REMOTE_WRITE
server_gr = GlobalRoute(dgid=client_ctx.query_gid(port_num=port_num,index=gid_index), sgid_index=gid_index)
server_ah_attr = AHAttr(gr=server_gr, is_global=1, port_num=1)
server_attr.ah_attr = server_ah_attr
server_qp.to_rtr(server_attr)
client_attr.qp_state = e.IBV_QPS_RTR
client_attr.path_mtu = e.IBV_MTU_4096
client_attr.rq_psn = 0
client_attr.min_rnr_timer = 12
client_attr.max_dest_rd_atomic = 10
client_attr.dest_qp_num = server_qp.qp_num
client_attr.qp_access_flags = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_READ | e.IBV_ACCESS_REMOTE_WRITE
client_gr = GlobalRoute(dgid=server_ctx.query_gid(port_num=port_num,index=gid_index), sgid_index=gid_index)
client_ah_attr = AHAttr(gr=client_gr, is_global=1, port_num=port_num)
client_attr.ah_attr = client_ah_attr
client_qp.to_rtr(client_attr)
Client qp modification RTR to RTS
client_attr.qp_state = e.IBV_QPS_RTS
client_attr.timeout = 14
client_attr.retry_cnt = 7
client_attr.rnr_retry = 7
client_attr.sq_psn = 0
client_attr.max_rd_atomic = 10
RDMA Write instruction to client
client_sge = SGE(client_mr.buf,len(SEND_STRING),client_mr.lkey)
send_wr = pwr.SendWR(num_sge = 1, sg = [client_sge],opcode=e.IBV_WR_RDMA_WRITE)
send_wr.set_wr_rdma(rkey = server_mr.rkey, addr = server_mr.buf)
client_qp.post_send(send_wr)
sleep(1)
print(server_mr.read(len(SEND_STRING),0))
If there is someone with knowledge in RMDA/RoCE/Pyverbs, i would be glad for some help. I don't have any prior knowledge in those topics. This is why i have choosen to write an application in python. I have knowledge in C, but python is for me much more convient for prototyping :)
Thank for your help!

Related

OpenDDS Messenger RTPS across 2 virtual machines

I have built OpenDDS-3.20 and I am able to run the Messenger sample code under (OpenDDS-3.20/DevGuideExamples/DCPS/Messenger) by using the run_test.pl. I am also able to run it without the run_test.pl file like this:
OpenDDS-3.20/bin/DCPSInfoRepo &
./publisher &
./subscriber
After this I can see the result:
...
...
...
Subscriber is available
SampleInfo.sample_rank = 0
SampleInfo.instance_state = ALIVE_INSTANCE_STATE
Message: subject = Review
subject_id = 99
from = Comic Book Guy
count = 0
text = Worst. Movie. Ever.
...
...
...
Even with the RTSP, I am able to execute it from the same machine ./run_test.pl --rtps. But, when I am trying to run DCPSInfoRepo and publisher in a VirtualBox-1 and subscriber in VirtualBox-2 using the default rtps.ini file then the sample application does not work.
I am able to ping from VirtualBox-1 to VirtualBox-2. They are both running Ubuntu Server. Can someone please share the process by which RTPS can be used to communicate between publisher and subscriber across 2 virtual machines.

How do you chose which slave to control in PyModbus?

So I'm doing a project where I have a Raspberry Pi 4 as master controlling 4 actuators with Modbus. I'm using PyModbus (with python 3) and I've been using code like this to test the Pi (with a PC slave simulator).
client = ModbusClient(method = "rtu", port="/dev/ttySC0", stopbits = 1, bytesize = 8, parity = 'E' baudrate= 19200)
connection = client.connect()
print(connection)
result = client.read_input_register(1, 1, unit= 0x01)
#client.write_register(address, value, **kwargs)
It's been great so far. But now I've switched from testing with the simulator to testing on the real actuators and I've only just now realized that there doesn't seem to be a way to "switch" the slave you read/write to. Considering I'm aiming for 4 slaves this is a problem.
So after searching online didn't reveal an answer I'm hoping someone here has a solution. I'd rather like to avoid using something other than PyModbus since it took awhile to find a Modbus library that worked for me.

Sending spoofed syslog messages in Perl

TL;DR I am trying to send syslog messages I have already created to a syslog server using spoofed source IPs but I'm making it really hard for myself and could do with a succinct approach
Most questions about syslog do not include the spoofed IP problem hence I am asking afresh.
I am writing (well updating) a script I wrote a long time ago that generates spoofed syslog messages (using UDP). It currently uses Net::RawIP which is terrible for portability and also the code for the transmission I wrote ages ago has decided to stop working in Perl 5 (I haven't used this for ages and I am refreshing it). I have been meaning to get rid of Net::RawIP for ages but its the only one I know how to use!
Given I have to fix it and I have a little time at the moment I probably want to move to use the Socket capability, which is what I have been playing with - using code from SO or gists or other places I can find - rather than something like IO::Socket as I need the spoofed IPs permission given a low level ability to write to sockets.
However, I have tied myself in knots with this, what I have right now forms the packets from scratch and then creates a socket and sends it, but in the processes wraps a superflous IPv4 header (I can see using wireshark) and without starting afresh I think its stuck like that as it has a fundamental flaw, hence I'm not sharing old code.
Basically, I can keep playing with the overly complicated code I have or ask for help simplify it, as I am beyond my knowledge of sockets and many hours of googling haven't helped much. What I keep finding is code that will work but it not compliant in some way - probably not an issue for what they are usually for which is for DDOS or syn attacks or whatever.
Key requirements of this are (which every attempt I have done has failed in some form!):
must come from a spoofed source IP and go to a known destination IP
(hence I'm using UDP) (both of which I have in config variables) so that the syslog server things many different devices generated the logs
must come from a set port and go to a set port (both of which I have in my
existing config variables)
must contain a message I have already formed which includes all the syslog content (the PRI and the syslog message content)
must be fully complaint with checksums and packet lengths etc when received
be as portable as possible (I'll probably embed it in my main script to keep it all in one file I can share with others).
I just feel there must be a simple and easy way to do this as everything I have is overly flexible and complicated and a nightmare to unwind.
NB This is shared in Sourceforge as "must syslog" and so you can see what I used to do but be aware its stopped working so it wont work if you run it currently! Once I fix this I'll upload a new version.
Cheers, --Chris
Actually I just had a breakthrough, I have linked the sources in the code but I found a test harness that is pretty basic and some UDP checksum code that with a bit of playing worked together and wireshark confirms its all correct
Posted in case someone else needs it as its taken me a good couple of days to get to this
#!/usr/bin/perl
use Socket;
use List::Util qw(sum);
sub udp_checksum {
# thanks to ikegami in post https://stackoverflow.com/questions/46181281/udp-checksum-function-in-perl
my $packed_src_addr = shift; # As 4-char string, e.g. as returned by inet_aton.
my $packed_dst_addr = shift; # As 4-char string, e.g. as returned by inet_aton.
my $udp_packet = shift; # UDP header and data as a string.
my $sum = sum(
IPPROTO_UDP,
length($udp_packet),
map({ unpack('n*', $_) }
$packed_src_addr,
$packed_dst_addr,
$udp_packet."\0", # Extra byte ignored if length($udp_packet) is even.
),
);
while (my $hi = $sum >> 16) {
$sum = ($sum & 0xFFFF) + $hi;
}
return ~$sum & 0xFFFF;
}
#this was found and adapted from http://rhosted.blogspot.com/2009/08/creating-udp-packetip-spoofing-through.html
$src_host = $ARGV[0];
$dst_host = $ARGV[1];
$src_port = 33333;
$dest_port = 514;
$cksum = 0; #initialise, we will sort this later
#$udp_len is the udp packet length in the udp header. Its just 8 plus the length of the data
#for this test harness we will set the data here too
$data = "<132> %FWSM-3-106010: Deny inbound tcp src outside:215.251.218.222/11839 dst inside:192.168.1.1/369";
$udp_len = 8+length($data);
$udp_proto = 17; #17 is the code for udp
#Prepare the udp packet, needed for the checksum to happen, then get the checksum (horrifically complicated, just google it)
$udp_packet = pack("nnnna*", $src_port,$dest_port,$udp_len, $cksum, $data);
$cksum = udp_checksum(inet_aton($src_host),inet_aton($dst_host),$udp_packet);
$zero_cksum = 0;
#test harness checks about host IPs
my $dst_host = (gethostbyname($dst_host))[4]; my $src_host = (gethostbyname($src_host))[4];
# Now lets construct the IP packet
my $ip_ver = 4;
my $ip_len = 5;
my $ip_ver_len = $ip_ver . $ip_len;
my $ip_tos = 00;
my ($ip_tot_len) = $udp_len + 20;
my $ip_frag_id = 19245;
my $ip_frag_flag = "010";
my $ip_frag_oset = "0000000000000";
my $ip_fl_fr = $ip_frag_flag . $ip_frag_oset;
my $ip_ttl = 30;
#H2H2nnB16C2na4a4 for the IP Header part#nnnna* for the UDP Header part.
#To understand these, see the manual of pack function and IP and UDP Header formats
#IP checksum ($zero_cksum is calculated by the kernel. Dont worry about it.)
my ($pkt) = pack('H2H2nnB16C2na4a4nnnna*',
$ip_ver_len,$ip_tos,$ip_tot_len,$ip_frag_id,
$ip_fl_fr,$ip_ttl,$udp_proto,$zero_cksum,$src_host,
$dst_host,$src_port,$dest_port,$udp_len, $cksum, $data);
#bit that makes the socket and sends the packet
socket(RAW, AF_INET, SOCK_RAW, 255) || die $!; setsockopt(RAW, 0, 1, 1);
my ($destination) = pack('Sna4x8', AF_INET, $dest_port, $dst_host);
send(RAW,$pkt,0,$destination);

Graphite showing rolling gap in data

I recently upgraded one of our Graphite instances from 0.9.2 to 1.1.1, and have since run into an issue where, for the lack of a better word, there is a rolling gap of data.
It shows the last few minutes correctly (I'm guessing what's in carbon cache), and after about 10-15 minutes past, it shows all of the data correctly as well.
However, inside that 10-15 minute gap, it's completely blank. I can see the gap both in Graphite, and in Grafana. It disappears after restarting carbon cache, and then comes back about a day later.
Example screenshot:
This happens for most graphs/dashboards I have.
I've spent a lot of effort optimizing disk IO, so I doubt it to be the case -> Cloudwatch shows 100% burst credit for disk. It's an m3.xlarge instance with 4 cores and 16 GB RAM. Swap file is on ephemeral storage and looks barely utilized.
Using 1 Carbon Cache instance with Whisper backend.
storage_schemas.conf:
[carbon]
pattern = ^carbon\.
retentions = 60:90d
[dumbo]
pattern = ^collectd\.dumbo # load test containers, we don't care about their data
retentions = 300:1
[collectd]
pattern = ^collectd
retentions = 10s:8h,30s:1d,1m:3d,5m:30d,15m:90d
[statsite]
pattern = ^statsite
retentions = 10s:8h,30s:1d,1m:3d,5m:30d,15m:90d
[default_1min_for_1day]
pattern = .*
retentions = 60s:1d
Non-default (or potentially relevant) carbon.conf settings:
[cache]
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 100 # was slagging disk write IO until I dropped it down from 500
MAX_CREATES_PER_MINUTE = 50
CACHE_WRITE_STRATEGY = sorted
RELAY_METHOD = rules
DESTINATIONS = 127.0.0.1:2004
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
Graphite local_settings.py
CARBONLINK_TIMEOUT = 10.0
CARBONLINK_QUERY_BULK = True
USE_WORKER_POOL = False
We've seen this with some workloads on 1.1.1, can you try updating carbon to current master? If not 1.1.2 will be released shortly which should solve the problem.

How to send an email without using the program RSCONN01?

I am trying to send an email with an excel attachment without using rsconn01. If this is possible could you show me how this is done?
I would also like a little bit more information about how rsconn01 works. I am using rsconn01 to send the emails but, I received a complaint that this program was also resending out emails that failed earlier that day.
This is the code I am using now. It works, but I want to know another way to do it without using rsconn01.
`CALL FUNCTION 'SO_DOCUMENT_SEND_API1'
EXPORTING
document_data = w_doc_data
put_in_outbox = 'X'
commit_work = 'X'
IMPORTING
sent_to_all = w_sent_all
TABLES
packing_list = t_packing_list
contents_bin = t_attachment
contents_txt = it_message
receivers = t_receivers
EXCEPTIONS
too_many_receivers = 1
document_not_sent = 2
document_type_not_exist = 3
operation_no_authorization = 4
parameter_error = 5
x_error = 6
enqueue_error = 7
OTHERS = 8.
if sy-subrc = 0.
WAIT UP TO 2 SECONDS.
SUBMIT rsconn01 WITH mode = 'INT'
WITH output = 'X'
AND RETURN.
else.
WRITE:/ 'ERROR IN MAIL ', sy-subrc.
endif.`
You will have to use RSCONN01 unless you'd like to implement your own protocol handling. You're using the standard SAPconnect functionality (although with an API that's a bit outdated, I'd switch to the BCS if I were in your shoes). As long as you're using this, you're stuck with that report. However, you usually won't have to call it for yourself. It's a background process that is called every few minutes to process outgoing mail. Perhaps you're working in a development environment where the SAPconnect system isn't properly setup - in that case, you should talk to your system administrators. There are ways to tune the SAPconnect system to work in many cases. You should try to use the existing and well supported facilities before trying to circumvent them.