I'm trying to learn my way around some Mirth code. I keep seeing things like this in the transformer for different channels:
msg['PRB']['PRB.4']['PRB.4.2'].toString()
I understand that this is parsing a HL7 message and I understand that msg probably means message. But is MSG a system-wide variable? Does every channel have a msg variable? What is the scope of msg? I also keep seeing things like ${message.encodedData} in the Templates. Do certain connector types automatically send the contents of msg?
The msg variable is the current instance of the inbound message that you will use for routing or mapping. It only exists in the context of the mapping/transformation that you are applying and is not system-wide. The message that will be transmitted is referred to as tmp. So, when you are mapping, you will be going from msg -> tmp.
Related
We build a REST microservice with Scala 3, ZIO 2, ZIO logging and Tapir.
For context specific logging we want to use the MDC and set an attribute there which is taken from the request payload.
Is it possible to get access to the request payload in DefaultServerLog to extract the MDC attribute and then use it for ZIO logging feature MDC logging, i.e. create a LogAnnotation from the extracted attribute, so it will also be logged by all DefaultServerLog methods (doLogWhenHandled etc.). Currently it works for our own log statements, but not for those of Tapir/ZIO-HTTP.
See answer here https://softwaremill.community/t/how-to-get-access-to-the-request-payload-in-tapir-ziohttp-defaultserverlog/84/3.
Adam Warski:
"This is usually problematic as the body of the request is a stream of bytes, which is read from the socket as it arrives. That is, the request isn’t loaded into memory by default.
You can work-around this by reading the whole request into memory using serverRequest.underlying.asInstanceOf[zio.http.Request].body.asArray, extracting the required info and enriching the fiber-locals appropriately. You might also need to substitute the Request with a copy, which has the body provided as a byte array (in a “strict” form), so that the “proper” body parser doesn’t try to re-read from the network (where nothing will be available).
However, this has some downsides: the body will be parsed twice (once by your interceptor, once by the parsing that’s defined later); and it will be read into memory (which might be problematic if you don’t have a limit on the size of the body)."
The goal of my Camel route is to read messages from a queue and to write the content of each message body as a file to a directory. It's not much more than the following:
from("wmq:..")
.to("/some/path");
The message body is always text. However, I don't want Camel to interpret it as text - instead, each message body should be seen as binary data that is written to the file system without applying any changes to it.
The background of my problem is that there are a lot of producers writing to the queue. The messages in the queue are always textual, but with different encodings, and different message headers (or the lack thereof). The problem is that some (non-unicode) messages that end up as a file in /some/path suddenly got a UTF BOM prepended by Camel while others are totally fine. My assumption is that this has something to do with the heterogenous message headers of the incoming messages which I can't control.
How can I enforce Camel to just write the content bytes of a message to a file without applying any changes to it?
edit:
I asked the wrong question, or at least I asked it in a wrong way. I'll give it another try with a slightly different wording:
In Camel, is it possible to create a route that takes the bodies of queue messages and writes them to the file system, ...
with the guarantee that the content is not altered (on byte level)
regardless of the message headers that might or might not be present, or even indicate a wrong content type
The route should be robust in the sense that no matter what kind of queue message you throw at it, it just writes the content to the file system.
And if this is possible, is the simple route I posted above already doing that, or is it necessary to implement additional measures?
Although it's possible to read from a Gio.Socket by wrapping it's file-descriptor in Gio.DataInputStream, using Gio.Socket.receive_from() in GJS to receive is not possible because as commented here:
GJS will clone array arguments before passing them to the C-code which will make the call to Socket.receive_from work and return the number of bytes received as well as the source of the packet. The buffer content will be unchanged as buffer actually read into is a freed clone.
Thus, input arguments are cloned and data will be written to the cloned buffer, not the instance of buffer actually passed in.
Although reading from a data stream is not a problem, Gio.Socket.receive_from() is the only way I can find to get the remote address from a UDP listener, since Gio.Socket.remote_address will be undefined. Unfortunately as the docs say for Gio.Socket.receive():
For G_SOCKET_TYPE_DATAGRAM [...] If the received message is too large to fit in buffer, then the data beyond size bytes will be discarded, without any explicit indication that this has occurred.
So if I try something like Gio.Socket.receive_from(new Uint8Array(0), null); just to get the address, the packet is swallowed, but if I read via the file-descriptor I can't tell where the message came from. Is there another non-destructive way to get the incoming address for a packet?
Since you’re using a datagram socket, it should be possible to use Gio.Socket.receive_message() and pass the Gio.SocketMsgFlags.PEEK flag to it. This isn’t possible for a stream-based socket, but you are not going to want the sender address for each read you do in that case.
If you want improved performance, you may be able to use Gio.Socket.receive_messages(), although I am not sure whether that’s completely introspectable at the moment.
I have worked with network programming before. But this is my first foray into netlink sockets.
I have chosen to study the 'connector' type of netlink sockets. As with any kernel component, it has a user counterpart as well. The linux kernel has a sample program called ucon.c which can be used to build userspace programs based on the aforementioned connector netlink sockets.
So here I wish to pin-point parts of the program that I want to confirm my understanding of and of parts of the program that I do not follow the logic of. Enough talking. Here we go. Please correct me wherever I go astray.
As far as I have understood, netlink sockets are a IPC method used to connect processes on the same machine and hence process ID is used as an identifier. And since netlink messages can be ideally multicast, another identifier that is needed by the netlink socket is the message group. All components that are connected to the same message group are in fact related. So while in case of IPv4, we use a sockaddr_in in place of the sockaddr, here we use a sockaddr_nl which contains the above mentioned identifiers.
Now, since we are not going to use the TCP/IP stack of the kernel, in case of netlink messages, netlink packets can be considered to be raw (please correct me here if I am wrong). Hence the only encapsulation that the netlink packet goes through is the netlink message header defined as nlmsghdr.
Now coming on to our program ucon, main() first creates a NETLINK family socket with the connector protocol. Then it fills up the aforementioned netlink socketaddress structure with the relevant information. In order to be a little experimental here, I have added an entry in the connector.h file. Now here comes my first question.
A connector message has a certain type defined in connector.h. Now this connector message structure is something that is completely internal to netlink right? As in, as far as netlink is concerned, this is all but payload. Right?
Moving on, what exactly does the nl-group field mean within the netlink message header structure? The definition does not really contain an element of this name. So are we using overlay techniques to fill certain fields of the netlink message header? And if so, what exactly is the correspondence? I cannot seem to find it anywhere.
So after binding the socket address to the socket, it is sending 10,000 unique pieces of connector based data, which as far as netlink is concerned, is pure payload. But what is strange as far as these messages are concerned is, that all of them seem to have the same sequence number.
Moving on, we find ourselves in the netlink_send subroutine to send these packets via the socket that we are bound to above. This subroutine uses a variety of netlink helper macros to manipulate the data to send. As we say above, the main() function sends 10,000 pieces of data, each of whom is zero-length and requires no acknowledgement, since the ack field is 0 (please correct me if I am wrong here). So each 'packet' is nothing but a connector message header without anything in it. Right?
Now what is surprising is that the netlink_Send function uses the same sequence number as the main() since it is a global variable. However, after the post increment in main(), it is now '1'. So basically our netlink talk is starting with a sequence number of '1'. Is that fine?
Looking into some of the helper macros defined in linux/netlink.h, I will try to summarize my understanding of the ones that are directly or indirectly being used in this program.
#define NLMSG_LENGTH(len) ((len)+NLMSG_ALIGN(NLMSG_HDRLEN))
So this macro will first align the netlink message header length and then add the payload length to it. For our case the netlink payload is a connector header without any payload of its own.
In our case, this micro is used like so
nlh->nlmsg_len = NLMSG_LENGTH(size - sizeof(*nlh));
Here, what I do not understand is the actual payload of the netlink message. In the above case, it is the size of the connector message header (since the connector message itself contains no payload of its own) minus the pointer (which is pointing to the first byte of the netlink message and thereby the netlink message header). And this pointer is (like any other pointer variable) equal to the machine word size which in my case is 4 bytes. Why are we substracting this from the connector message header?
After that, we send the message over this netlink socket just like any other IPv4 socket. hope to hear from you fellows out there with regards to the above mentioned questions. Including some sentences before the actual quesion would help as my post is rather long. But I hope it would be useful to people more than just myself.
Regards.
I´ve read that it´s possible to share sockets between processes. Is this also possible in Node.js?
I saw the cluster api in node.js, but that´s not what I´m looking for. I want to be able to accept a connection in one process, maybe send & read a bit, and after a while pass this socket to another fully independent node.js process.
I could already do this with piping, but I don´t want to do this, since it´s not as fast as directly reading/writing to the socket itself.
Any ideas?
Update
I found the following entry in the node.js documentation:
new net.Socket([options]) #
Construct a new socket object.
options is an object with the following defaults:
{ fd: null
type: null
allowHalfOpen: false
}
fd allows you to specify the existing file descriptor of socket. type specified underlying protocol. It can be 'tcp4', 'tcp6', or 'unix'. About allowHalfOpen, refer to createServer() and 'end' event.
I think it would be possible to set the "fd" property to the filedescriptor of the socket and then open the socket with that. But... How can I get the filedescriptor of the socket and pass it to the process that needs it?
Thanks for any help!
This is not possible at the moment, but I've added it as a feature request to the node issues page.
Update
In the mean time, I've written a module for this. You can find it here: https://github.com/VanCoding/node-ancillary
You probably want to take a look at hook.io
hook.io is a distributed EventEmitter built on node.js. In addition to providing a minimalistic event framework, hook.io also provides a rich network of hook libraries for managing all sorts of input and output.