I am reading slim 3 docs and found that it does read/send the response body text in chunks of 4096 bytes:
responseChunkSize Size of each chunk read from the Response body when
sending to the browser. (Default: 4096)
What is the point of doing it so? Wouldn't it better to send the response body at once? Would this imply a small overhead?
During sending response to client browser, content length of response body may or may not be available.
In both cases, responseChunkSize settings is used as number of bytes to read from body until it reaches end of file. If content length is known and it is less or equal than responseChunkSize, then it only takes one iteration to read body's content.
By reading and output response in smaller chunk, browser does not wait too long to get first byte. Reading big chunk is slower and may require larger memory consumption so browser will likely get first byte longer than smaller chunk.
Related
Some protocols like HTTP can specify a message length, then send a (possibly very long) message. No other messages can be received while this message is being sent (something HTTP/2.0 tried to solve) so if you decide to ignore the message, you can't just continue waiting for messages and not pull its data.
Normally I read() up to the length of the message repeatedly into a junk buffer and just ignore the bytes. But that involves copying possibly millions of bytes from kernelspace into userspace (at 1 copy per memory page, so not millions of copies). Isn't there some way to just tell the kernel to discard the bytes instead of providing them?
It seemed like an obvious question, but the only answer I've been able to come up with is oddly resource heavy, either using splice() to dump the bytes into a pipe and seeking the pipe back to 0, or opening "/dev/null" and using sendfile() to send the bytes there. I could do that, and reserve a (single) file descriptor for flushing data out of clogged connections, without reading, but isn't there just a... ignore(descriptor, length) function?
I am using IoT Foundation on Bluemix and trying to send a byte array representing a simple text file, whose size is only 1354 bytes. I do not see anything arriving on IoT Foundation, it looks like I am hitting the 4KB payload limit (IoT Foundation doc). The shortest byte array I am able to send is only 1349 bytes. Is this a limitation of current IoT Foundation implementation?
there is a payload limit of 4096 bytes. If you attempt to send a message with a payload greater than this, the connection will simply be terminated.
If you are sending JSON, don't forget that the JSON syntax is included (including pretty printing) in the payload size. With a pure binary payload you will be able to send a payload of exactly 4096, but you will have to calculate the size in bytes of any string (after encoding) that you send.
Please can you double check how large the payload is and confirm it really is less than 4096 bytes ?
thanks
Paul
So I'm reading trying to find more about sockets and I found that the documentation kind of repeats itself without going in any further detail in regards to the way sockets send data:
So what does the {packet, N} do when specifying this in the options (when opening a socket) ? Does it include a header of N bytes before the data or does it break the message into N packets? Or does it include a header of N to all the packets that the message ends up being broken into?
I was reading Joe Armstrong's Software for a concurrent world and I found this paragraph:
The word packet refers to the length of an application request or response message, not to the physical packet seen on the wire.
I can't get my head round the meaning of this. What is meant by the packet seen on the wire.
I tried to look into the documentation and I found little referring to what the option does. The brief explanation that I found is that it prepends the message with a N header however I also found this comment in a code written as an example:
%% Usually, it's a good idea to give up in case of a
%% send timeout, as you never know how much actually
%% reached the server, maybe only a packet header?!
According to this, then, the message gets broken into pieces (a random number I presume?) and each gets sent with a N byte header.
My question is how does the {header, N} option affect the way data is sent.
The excerpt from Joe's book you quote refers to the fact that applications are typically blissfully unaware of how networks arrange data for transmission. Depending on the network type, configuration, and protocols in use, the blocks of data sent over it will vary in size and will be framed with different metadata. Applications typically don't see raw packets, framing information, or the fact that sometimes raw packets are retransmitted due to problems that cause them to be dropped or corrupted.
Your question mentions two options: {packet, N} and {header, N}. These are quite different from each other.
The {packet, N} option allows N to be 1, 2, or 4. It attaches a header of N bytes to the front the message. The header specifies, in network order, the length of the message, i.e. the number of bytes in the message.
If you send a message consisting of X bytes, Erlang will prepend to the data N bytes containing the network order value of X, and send the whole thing. Assuming the receiver has also been configured with the same {packet, N} option, it will read the N-byte header to determine how many bytes to expect, wait to receive that many bytes, and then deliver those bytes, without the length header, to the receiving application. How the underlying networking software and hardware breaks the data into chunks for transmission across the network is a separate matter hidden from your application.
The {header, Size} option delivers the message to the receiver as a list of Size bytes followed by the remainder of the data as a binary. This option makes sense only when the binary option is in effect for the socket.
I know very similar questions have been asked before. But I don't think the solutions I found on google/stackoverflow are suitable for me.
I started to write some web services with Scala/Spray, and it seems the best way to send large files without consuming large amouns of memory is using the stream marshalling. This way Spray will send http chunks. Two questions:
Is it possible to send the file without using HTTP chunks and without reading the entire file into memory?
AFAIK akka.io only process one write at a time, meaning it can buffer one write until it has been passed on to the O/S kernel in full. Would it be possible to tell Spray, for each HTTP response, the length of the content? Thereafter Spray would ask for new data (through akka messages) untill the entire content length is completed. Eg, I indicate my content length is 100 bytes. Spray sends a message asking for data to my actor, I provide 50 bytes. Once this data is passed on to the O/S, spray sends another message asking for new data. I provide the remaining 50 bytes... the response is completed then.
Is it possible to send the file without using HTTP chunks [on the wire]
Yes, you need to enable chunkless streaming. See http://spray.io/documentation/1.2.4/spray-routing/advanced-topics/response-streaming/
Chunkless streaming works regardless whether you use the Stream marshaller or provide the response as MessageChunks yourself. See the below example.
without reading the entire file into memory
Yes, that should work if you supply data as a Stream[Array[Byte]] or Stream[ByteString].
[...] Thereafter Spray would ask for new data [...]
That's actually almost like it already works: If you manually provide the chunks you can request a custom Ack message that will be delivered back to you when the spray-can layer is able to process the next part. See this example for how to stream from a spray route.
I indicate my content length is 100 bytes
A note upfront: In HTTP you don't strictly need to specify a content-length for responses because a response body can be delimited by closing the connection which is what spray does if chunkless streaming is enable. However, if you don't want to close the connection (because you would lose this persistent connection) you can now specify an explicit Content-Length header in your ChunkedResponseStart message (see #802) which will prevent the closing of the connection.
I used a ts analyzer for a .ts file i have with mpeg-2 codec and i found out that it splits in 7311 packets.
I m trying to find this through matlab by using fopen to open the ts file in binary and fread to read the file but all i get is a column with a huge collection of numbers(way above the number of packets). Does anyone know how can i determine which of these data are the packets? Or if someone knows another way to find the packets would help me a lot.
Thank you in advance
From some quick googling, the MPEG-2 transport stream ('ts') format consists of packets 188-bytes in length, each having a 4-byte header followed by a 184-byte payload. Essentially, you can count the number of packets by counting the number of headers you find - but beware that, if you are only interested in counting the number of, e.g., video packets in the stream, then you will need some deeper analysis of the headers, because the stream may contain any number of interleaved "elementary streams" (which can be video, audio, or arbitrary data). Each elementary packet type in the stream is denoted by a unique "PID" which is contained in the header.
Aside from the above, you will also have to handle synchronisation - each header begins with the "synchronisation byte", which has a value 0x47 (or 01000111 in binary). According to this resource, decoders begin by looking for this synchronisation byte; once they find one, they may have found a packet header. To make sure, they try to find three consecutive synchronisation bytes (188 bytes apart in the stream); if three are found, synchronisation can occur and the packet boundaries may from then on be assumed at 188-byte intervals. Note, however, that the first byte of each assumed header should be checked to see if it is a synchronisation byte - if it is not, then this is called "sync loss" and the syncrhonisation process must start again.
Once you have some code to syncrhonise to a stream, it should be fairly easy to extract the PIDs from the header of each packet and count the number of packets associated with each unique PID you find. You should probably also check the first bit after the synchronisation byte as, if set to 1, this indicates a transport error, and the packet's payload is invalid. Detailed information on the format of packet headers can be found here.