Idea:
The first way is to create a brand-new packet in the bpf program and send it to the specified receiver.
The second way is to copy a packet. To make it easy to distinguish, below I call the packet I get from copying as packet_copy. Then I can modify the packet_copy so it can be sent to the specified receiver, and the original packet will go through the normal path.
Try:
As to the first way, I have not found a suitable solution, but someone says that this way is not possible.
As to the second way, I found bpf_clone_redirect() as a possible solution. However, instead of getting the packet_copy from copying, we should directly modify the original packet (so that it can be redirected to the specified receiver) and then call bpf_clone_redirect to redirect it. I also need to undo the modification to restore the packet after calling bpf_clone_redirect because I want this packet to be processed normally rather than dropped.
Question:
As to the first way, I would like to know if it is feasible.
As to the second way, I wonder if it can be optimized, given that bpf_clone_redirect has its limitations.
The first way is to create a brand-new packet in the bpf program and send it to the specified receiver.
That is indeed not possible today. It would require dynamic memory allocation to allocate memory for the packet and a helper to then send the packet out. None of these are available in BPF today (v5.19).
However, instead of getting the packet_copy from copying, we should directly modify the original packet (so that it can be redirected to the specified receiver) and then call bpf_clone_redirect to redirect it. I also need to undo the modification to restore the packet after calling bpf_clone_redirect because I want this packet to be processed normally rather than dropped.
I don't know of a more efficient way in a single BPF program.
Depending on where in the stack you sit, you may however be able to use two BPF programs, with one doing the cloning and another one intercepting one of the copies after to modify it. I'm not convinced that would be more CPU efficient but it may be cleaner.
Related
I wrote a C++ program to create a socket and bind on this socket to receive ICMP/UDP packets. The code I wrote as following:
while(true){
recvfrom(sockId, rePack, sizeof(rePack), 0, (struct sockaddr *)&raddr, (socklen_t *)&len);
processPakcet(recv_size);
}
So, I used a endless while loop to receive messages continually, But I worried about the following two questions:
1, How long the message would be kept in the receiver queue or say in NIC queue?
I worried about that if it takes too long to process the first message, then I might miss the second message. so how fast should I read after read.
2, How to prevent reading the duplicated messages?
i.e, does the receiver queue knows me, when my thread read the first message done, would the queue automatically give me the second one? or say, when I read the first message, then the first message would be deleted by the queue and no one could receive it again.
Additionally, I think the while(true) module is not good, anyone could give me a good suggestion please. (I heard something like polling module).
First, you should always check the return value from recvfrom. It's unlikely the recvfrom will fail, but if it does (for example, if you later implement signal handling, it might fail with EINTR) you will be processing undefined data. Also, of course, the return value tells you the size of the packet you received.
For question 1, the actual answer is operating system-dependent. However, most operating systems will buffer some number of packets for you. The OS interrupt handler that handles the incoming packet will never be copying it directly into your application level buffer, so it will always go into an OS buffer first. The OS has previously noted your interest in it (by virtue of creating the socket and binding it you expressed interest), so it will then place a pointer to the buffer onto a queue associated with your socket.
A different part of the OS code will then (after the interrupt handler has completed) copy the data from the OS buffer into your application memory, free the OS buffer, and return to your program from the recvfrom system call. If additional packets come in, either before or after you have started processing the first one, they'll be placed on the queue too.
That queue is not infinite of course. It's likely that you can configure how many packets (or how much buffer space) can be reserved, either at a system-wide level (think sysctl-type settings in linux), or at the individual socket level (setsockopt / ioctl).
If, when you call recvfrom, there are already queued packets on the socket, the system call handler will not block your process, instead it will simply copy from the OS buffer of the next queued packet into your buffer, release the OS buffer, and return immediately. As long as you can process incoming packets roughly as fast as they arrive or faster, you should not lose any. (However, note that if another system is generating packets at a very high rate, it's likely that the OS memory reserved will be exhausted at some point, after which the OS will simply discard packets that exceed its resource reservation.)
For question 2, you will receive no duplicate messages (unless something upstream of your machine is actually duplicating them). Once a queued message is copied into your buffer, it's released before returning to you. That message is gone forever.
(Note that it's possible that some other process has also created a socket expressing interest in the same packets. That process would also get a copy of the packet data, which is typically handled internal to the operating system by reference counting rather than by actually duplicating the OS buffers, although that detail is invisible to applications. In any case, once all interested processes have received the packet, it will be discarded.)
There's really nothing at all wrong with a while (true) loop; it's a very common control structure for long-running server-type programs. If your program has nothing else it needs to be doing in the meantime, while true allowing it to block in recvfrom is the simplest and hence clearest way to implement it.
(You could use a select(2) or poll(2) call to wait. This allows you to handle waiting for any one of multiple file descriptors at the same time, or to periodically "time out" and go do something else, say, but again if you have nothing else you might need to be doing in the meantime, that is introducing needless complication.)
I am new to STM32 and freertos. I need to write a program to send and receive data from a module via UART port. I have to send(Transmit) a data to that module(for eg. M66). Then I would return to do some other tasks. once the M66 send a response to that, my seial-port-receive-function(HAL_UART_Receive_IT) has to be invoked and receive that response. How can I achieve this?
The way HAL_UART_Receive_IT works is that you configure it to receive specified amount of data into given buffer. You give it your buffer to which it'll read received data and number of bytes you want to receive. It then starts receiving data. Once exactly this amount of data is received, a callback function HAL_UART_RxCpltCallback gets called (from IRQ) where you can do whatever you want with this data, e.g. add it to some kind of queue for later processing in the task context.
If I was to express my experiences related to working with HAL's UART module is that it's not the greatest one for generic use where you don't know the amount of data you expect to receive in advance. In the case of M66 modem you mention, this will happen all the time.
To solve this you have two choices:
Simply don't use HAL functions at all in case of UART, other than the initialization functions. Implement your own UART interrupt handler (most of the code can be copied from handler in HAL) where upon receiving data you place received bytes in a receive byte queue handled in your RTOS task. In this task you implement protocol parsing. This is the approach I use personally.
If you really want to use HAL but also work with a module that sends varying amount of data, call HAL_UART_Receive_IT and specify that you want to receive 1 byte each time. This will work, but will be (potentially much) slower than the first approach. Assuming you'll later want to implement some tcp/ip communication (you mentioned M66 GPRS module) you probably don't want to do it this way.
You should try the following way.
Enable UARTX Rx interrupt in NVIC.
Set Interrupt priority.
Unmask Interrupt request in EXTI.
Then use USARTX Interrupt Handler Function Define in you Vector.
Whenever the data is received from USARTX this function get automatically called and you can copy data from USARTX Receive Data Register.
I would rather suggest another approach. You probably want to archive higher speeds (lets say 921600 bods) and the interrupt way is fat to slow for it.
You need to implement the DMA transmition with the data end detection features. Run your USART in the DMA mode in the circular mode. You will have two events to serve. The first one is the DMA end of thransmition interrupt (then you copy the data from the current tail pointer to the end of the buffer to avoid data override) and USART IDLE interrupt - this will detect the end of the receive.
Although it's possible to read from a Gio.Socket by wrapping it's file-descriptor in Gio.DataInputStream, using Gio.Socket.receive_from() in GJS to receive is not possible because as commented here:
GJS will clone array arguments before passing them to the C-code which will make the call to Socket.receive_from work and return the number of bytes received as well as the source of the packet. The buffer content will be unchanged as buffer actually read into is a freed clone.
Thus, input arguments are cloned and data will be written to the cloned buffer, not the instance of buffer actually passed in.
Although reading from a data stream is not a problem, Gio.Socket.receive_from() is the only way I can find to get the remote address from a UDP listener, since Gio.Socket.remote_address will be undefined. Unfortunately as the docs say for Gio.Socket.receive():
For G_SOCKET_TYPE_DATAGRAM [...] If the received message is too large to fit in buffer, then the data beyond size bytes will be discarded, without any explicit indication that this has occurred.
So if I try something like Gio.Socket.receive_from(new Uint8Array(0), null); just to get the address, the packet is swallowed, but if I read via the file-descriptor I can't tell where the message came from. Is there another non-destructive way to get the incoming address for a packet?
Since you’re using a datagram socket, it should be possible to use Gio.Socket.receive_message() and pass the Gio.SocketMsgFlags.PEEK flag to it. This isn’t possible for a stream-based socket, but you are not going to want the sender address for each read you do in that case.
If you want improved performance, you may be able to use Gio.Socket.receive_messages(), although I am not sure whether that’s completely introspectable at the moment.
Since I'm not a native English speaker I might be missing something so maybe someone here knows better than me.
Taken from WSASend's doumentation at MSDN:
lpBuffers [in]
A pointer to an array of WSABUF
structures. Each WSABUF structure
contains a pointer to a buffer and the
length, in bytes, of the buffer. For a
Winsock application, once the WSASend
function is called, the system owns
these buffers and the application may
not access them. This array must
remain valid for the duration of the
send operation.
Ok, can you see the bold text? That's the unclear spot!
I can think of two translations for this line (might be something else, you name it):
Translation 1 - "buffers" refers to the OVERLAPPED structure that I pass this function when calling it. I may reuse the object again only when getting a completion notification about it.
Translation 2 - "buffers" refer to the actual buffers, those with the data I'm sending. If the WSABUF object points to one buffer, then I cannot touch this buffer until the operation is complete.
Can anyone tell what's the right interpretation to that line?
And..... If the answer is the second one - how would you resolve it?
Because to me it implies that for each and every data/buffer I'm sending I must retain a copy of it at the sender side - thus having MANY "pending" buffers (in different sizes) on an high traffic application, which really going to hurt "scalability".
Statement 1:
In addition to the above paragraph (the "And...."), I thought that IOCP copies the data to-be-sent to it's own buffer and sends from there, unless you set SO_SNDBUF to zero.
Statement 2:
I use stack-allocated buffers (you know, something like char cBuff[1024]; at the function body - if the translation to the main question is the second option (i.e buffers must stay as they are until the send is complete), then... that really screws things up big-time! Can you think of a way to resolve it? (I know, I asked it in other words above).
The answer is that the overlapped structure and the data buffer itself cannot be reused or released until the completion for the operation occurs.
This is because the operation is completed asynchronously so even if the data is eventually copied into operating system owned buffers in the TCP/IP stack that may not occur until some time in the future and you're notified of when by the write completion occurring. Note that with write completions these may be delayed for a surprising amount of time if you're sending without explicit flow control and relying on the the TCP stack to do flow control for you (see here: some OVERLAPS using WSASend not returning in a timely manner using GetQueuedCompletionStatus?) ...
You can't use stack allocated buffers unless you place an event in the overlapped structure and block on it until the async operation completes; there's not a lot of point in doing that as you add complexity over a normal blocking call and you don't gain a great deal by issuing the call async and then waiting on it.
In my IOCP server framework (which you can get for free from here) I use dynamically allocated buffers which include the OVERLAPPED structure and which are reference counted. This means that the cleanup (in my case they're returned to a pool for reuse) happens when the completion occurs and the reference is released. It also means that you can choose to continue to use the buffer after the operation and the cleanup is still simple.
See also here: I/O Completion Port, How to free Per Socket Context and Per I/O Context?
It seems that since boost 1.40.0 there has been a change to the way that the the async_read_some() call works.
Previously, you could pass in a null_buffer and you would get a callback when there was data to read, but without the framework reading the data into any buffer (because there wasn't one!). This basically allowed you to write code that acted like a select() call, where you would be told when your socket had some data on it.
In the new code the behaviour has been changed to work in the following way:
If the total size of all buffers in the sequence mb is 0, the asynchronous read operation shall complete immediately and pass 0 as the argument to the handler that specifies the number of bytes read.
This means that my old (and incidentally, the method shown in this official example) way of detecting data on the socket no longer works. The problem for me is that I need a way detecting this because I've layered my own streaming classes on-top of the asio socket streams and as such, I cannot just read data off the sockets that my streams will expect to be there. The only workaround I can think of right now is to read a single byte, store it and when my stream classes then request some bytes, return that byte if one is set: not pretty.
Does anyone know of a better way to implement this kind of behaviour under the latest boost.asio code?
My quick test with an official example with boost-1.41 works... So I think it still should work (if you use null_buffers)