Imagine an offer SDP that has one line of "m" with codecs 8 and 101 for DMTF and marked as sendrecv:
m = audio 35904 RTP/AVP 8 101
a = rtpmap:8 PCMA/8000
a = rtpmap:101 telephone-event/8000
a = fmtp:101 0-15
a = sendrecv
The offered SDP is answered by a SDP with one line of "m" containing codecs 8 and 120 for DTMF similarly marked as sendrecv:
m = audio 1235 RTP/AVP 8 120
a = rtpmap:8 PCMA/8000
a = rtpmap:120 telephone-event/8000
a = fmtp:101 0-15
a = sendrecv
From RFC 3264:
For streams marked as sendrecv in the answer, the "m=" line MUST
contain at least one codec the answerer is willing to both send and
receive, from amongst those listed in the offer. The stream MAY
indicate additional media formats, not listed in the corresponding
stream in the offer, that the answerer is willing to send or
receive (of course, it will not be able to send them at this time,
since it was not listed in the offer).
Above part of the RFC3264, proves that sending a different DTMF fmtp(120 to 101) in answer SDP complies with RFC3264 since the codec 8(G711a) matches with the offer SDP.
Is it okay to say the codec exchange is completed successfully and DTMF exchange will okay or is DTMF is not expected to work at this point?
In general:
RTP payload type numbers 0-95 identify a static media encoding. E.g. payload type 8 means PCMA audio with a clock rate of 8000 Hz (RFC3551). As such, this description doesn't have to (but should) be included in the media format description of the SDP offer/answer, using the "a=rtpmap:" and "a=fmtp:" attributes (RFC4566).
Payload type numbers 96-127 are dynamic. These can be used to negotiate encodings that aren't included in the static list. When using one of these numbers, an encoding specification has to be included in the media format description to specify the exact encoding parameters.
Both negotiating parties can choose their own dynamic payload type number to represent the same media encoding, this doesn't have to be the same number. This can be useful when a party already assigned a particular dynamic payload type number to another encoding. In your example one party uses 101 in the m-line and the other one uses 120, but these numbers represent the same media encoding (see "a=rtpmap:" lines). Each party tells the other 'when you send RTP using encoding X you must include payload type number Y in the RTP packet headers.
The payload type number is included in the PT field of RTP packets headers (RFC 3550)
In this case:
The "a=fmtp:" attribute in the answer specifies 101 as payload type number instead of 120. That means it doesn't apply to the telephone-events payload and no information is available as to which DTMF events are supported (RFC 4733). I think this is an implementation error and the fmtp attribute is meant to apply to the telephone-events payload.
It is an indication that you should expect DTMF issues. But it could also all work fine. Give it a try...
I want to write an HTTP implementation.
I've been looking around for a few days about sending files over HTTP with Content-Type: multipart/form-data, and I'm really interested about how browsers (or any HTTP client) creates that kind of request.
I already took a look at a lots of questions about it here at stackoverflow like:
How does HTTP file upload work?
What does enctype='multipart/form-data' mean?
I dig into RFCs 2616 (and newer versions), 2046, etc. But I didn't find a clear answer (obviously I did not get the idea behind it).At most articles and answers I found this piece of request string, that's is simple to me to interpret, all these things are documented at RFCs...
POST /upload?upload_progress_id=12344 HTTP/1.1
Host: localhost:3000
Content-Length: 1325
Origin: http://localhost:3000
... other headers ...
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryePkpFF7tjBAqx29L
------WebKitFormBoundaryePkpFF7tjBAqx29L
Content-Disposition: form-data; name="MAX_FILE_SIZE"
100000
------WebKitFormBoundaryePkpFF7tjBAqx29L
Content-Disposition: form-data; name="uploadedfile"; filename="hello.o"
Content-Type: application/x-object
... contents of file goes here ...
------WebKitFormBoundaryePkpFF7tjBAqx29L--
...and it would be simple to implement an HTTP client to construct a piece of string that way in any language.The problem becomes at ... contents of file goes here ..., there's little information about what "contents of file" is. I know it's binary data with a certain type and encoding, but It's difficult to think out of string data, how I would add a piece of binary data that has no string representation inside a string.
I would like to see examples of low level implementations of HTTP protocol with any language. And maybe in depth explanations about binary data transfer over HTTP, how client creates requests and how server read/parse it. PD. I know this question my look a duplicate but most of the answers are not focused on explaining binary data transfer (like media).
You should not try to handle strings on this part of the body, you should send binary data, see it as reading bytes from the resource and sending theses bytes unaltered.
So especially no encoding applied, no utf-8, no base64, HTTP is not a protocol with an ascii7 restriction like smtp, where base64 encoding is applied to ensure only ascii7 characters are used.
There is, by definition, no string version of this data, and looking at raw HTTP transfer (with wireshark for example) you should see binary data, bytes, stuff.
This is why most HTTP servers uses C to manage HTTP, they parse the HTTP communication byte per byte (as the protocol headers are ascii 7 only, certainly not multibytes characters) and they can also read/write arbitrary
binary data for the body quite easily (or even using system calls like readfile to let the kernel manage the binary part).
Now, about examples.
When you use Content-Length and no multipart stuff the body is exactly (content-length) bytes long, so the client parsing your sent data will just read this number of bytes and will treat this whole raw data as the body content (which may have a mime type and and encoding information, but that's just informations for layers set on top of the HTTP protocol).
When you use Transfer-Encoding: chunked, the raw binary body is separated into pieces, each part is then prefixed by an hexadecimal number (the size of the chunk) and the end of line marker. With a final null marker at the end.
If we take the wikipedia example:
4\r\n
Wiki\r\n
5\r\n
pedia\r\n
E\r\n
in\r\n
\r\n
chunks.\r\n
0\r\n
\r\n
We could replace each ascii7 letter by any byte, even a byte that would have no ascii7 representation, Ill use a * character for each real body byte:
4\r\n
****\r\n
5\r\n
*****\r\n
E\r\n
**************\r\n
0\r\n
\r\n
All the other characters are part of the HTTP protocol (here a chunked body transmission). I could also use a \n representation of binary data, and send only the null byte for each byte of the body, that would be:
4\r\n
\0\0\0\0\0\r\n
5\r\n
\0\0\0\0\0\0\r\n
E\r\n
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\r\n
0\r\n
\r\n
That's just a representation, we could also use \xNN or \NN representations, in reality these are bytes, 8 bits (too lazy to write the 0/1 representation of this body :-) ).
If the text of the example, instead of being:
Wikipedia in\r\n
\r\n
chunks.
It could have been a more complex one, with multibytes characters (here a é in utf-8):
Wikipédia in\r\n
\r\n
chunks.
This é is in fact 11000011:10101001 in utf-8, two bytes: \xc3\xa9 in \xNN representation), instead of the simple 01100101 / \x65 / echaracter. The HTTP body is now (see that second chunk size is 6 and not 5):
4\r\n
Wiki\r\n
6\r\n
p\xc3\xa9dia\r\n
E\r\n
in\r\n
\r\n
chunks.\r\n
0\r\n
\r\n
But this is only valid if the source data was effectively in utf-8, could have been another encoding. By default, unless you have some specific configuration settings available in your web server where you enforce a conversion of the source document in a specific encoding, that's not really the job of the web server to convert the source document, you take what you have, and you maybe add an header to tell the client what encoding was defined on the source document.
Finally we have the multipart way of transmitting the body, like in your question, it's a lot like the chunked version, except here boundaries and intermediary headers are used, but for the binary data between these boundaries, headers, and line endings control characters it is the same rule, everything inside are just bytes...
I'm currently using the PlayWS http client which returns an Akka stream. From my understanding, I can consume the stream and turn it into a Byte[] to calculate the size. However, this also consumes the stream and I can't use it anymore. Anyway around this?
I think there are two different aspects related to the question.
You want to know the size of the server response in advance to prepare buffer. Unfortunately there is no guaranteed way to do this. HTTP 1.1 spec explicitly allows transfer mode when the server does not know the size of the response in advance via chunked transfer encoding. See also quote from 3.3.1. Transfer-Encoding:
A recipient MUST be able to parse the chunked transfer coding
(Section 4.1) because it plays a crucial role in framing messages
when the payload body size is not known in advance.
Section 3.3.3. Message Body Length specifies how length of a message body is defined and it besides the aforementioned chunked transfer encoding it also contains quite unhelpful
Otherwise, this is a response message without a declared message
body length, so the message body length is determined by the
number of octets received prior to the server closing the
connection.
This is added for backward compatibility and discouraged from usage but is still legally allowed.
Still in many real world scenarios you can use Content-Length header field that the server may return. However there is a catch here as well: if gzip Content-Encoding is used, then Content-Length will contain size of the compressed body.
To sum up: in general case you can't get the size of the message body in advance before you fully get the server response i.e. in terms of code perform a blocking call on the response. You may try to use Content-Length and it might or might not help in your specific case.
You already have a fully downloaded response (or you are OK with blocking on your StreamedResponse) and you want to process it by first getting the size and only then processing the actual data. In such case you may first use getBodyAsBytes method which returns IndexedSeq[Byte] and thus has size, and then convert it into a new Source using Source.single which is actually exactly what the default (i.e. non-streaming) implementation of getBodyAsSource does.
8bit and binary are valid values for the Content-Transfer-Encoding header (here is a nice summary on SO).
However, trying to figure out which one was the most suitable for my needs, I encountered the following notices :
Binary encoded messages are not valid Internet messages.
and
Because not all Message Transfer Agents (MTAs) can handle 8bit data, the 8bit encoding is not a valid encoding mechanism for Internet mail.
Digging a bit I found out these warnings likely origin from Microsoft documentation.
What does it actually means ? Should one avoid these values ?
NB : It is not clear to me what the quoted "Internet messages" term specifically refers to. For my purposes, I am concerned only with multipart emails.
RFC 4180 talks about the text/csv mime type, but doesn't go into some of the more obvious variations that one would have to support, for example the separator used (ok ok so if it's Csv then it should obviously be a comma) and whether there is a header row or not.
I'd like to add parameters like separator=, and headerrow=0|1 to the mime type such that REST APIs that consume CSV files can consume them appropriately.
The alternative would seem to be to add the parameters to the URL but that feels wrong.