Allow me to be more specific...
If I want to send files, but am required to wrap them in SOAP, wouldn't I use http? I am seeing a surprisng lack of info on this online.
Sending files via SOAP doesn't have anything specifically to do with FTP. To send a file through a SOAP interface, you might base64 encode the file and stuff the whole thing into a SOAP string parameter. However, this might only be appropriate if your file size has a reasonable upper bound.
If your files can be an unbounded size, you might investigate using a different transport protocol to transfer the actual file data (eg. HTTP or even FTP), then use SOAP to transfer a pointer to the file (such as its URL). Some implementations of SOAP cannot handle arbitrary large messages.
Pretty vague question but if you're using web services you can use MTOM http://en.wikipedia.org/wiki/MTOM (SOAP Message Transmission Optimization Mechanism)
I don't know your environment but there are examples of this using .NET / WCF if you Google it.
Two standard ways of sending files along with SOAP messages are:
SOAP with Attachments
MTOM
MTOM supports either using MIME attachments or base64 encoding the file into the body, where as SOAP with Attachments only supports MIME attachments.
Related
I wanted to know which kind of protocol apache beam uses to read and write to cloud storage. Is it HTTPS or Binary(Blob).I tried to google it but I did not find. I know gsutil command uses HTTPS protocol.
You are mixing 2 things: Transport layer and data encoding.
Does Google use HTTP transport? YES, for all the API. HTTPS or gRPC (HTTP/2) are commonly use.
Does Google use binary encoding to speed up the transfert? As said before, transport can be HTTPS or gRPC. HTTPS is commonly use for REST API, and transport JSON text format. Of course, you can exchange files in binary format (GZIP for example to compress and speed up the transfert). gRPC is a binary protocol. You don't exchange JSON, but binary representation of the data that you want to exchange. And thereby, the file transfert is also in binary mode.
Now, what use Beam? As often, the Google libraries use gRPC behind the scene, and thus, the encoding is binary. If you perform yourselves REST API call, with JSON, and HTTP will be use for that; but file content is, when it can (depends on your request accept content header), transferred in binary.
EDIT 1
For BEAM, I had a look to the source code. You have here, for example, the creation of GoogleCloudStorageImpl object.
If you have a look to this full class name: import com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl;. Ok, let's see the hadoop package!!
The Javadoc is clear: JSON API is used. to confirm that, I went to the source code, and YES, the JSON format is used for the API communication.
BUT, keep in mind that is the API communication, the metadata around the file content. The file content should be sent in binary format (plain text of b64 encoding should be strange).
I am trying to figure if there is a way to stream large files chunks by chunks using HTTP transfer-encoding CHUNKED between client and server in REST. By Semantics REST service provider accepts only application/json, but I read this Chunked Transfer Encoding and thinking if this is something I can make it possible using any REST client, say for example apache Http client.
Handling large files (memory overhead will be more during normal/huge loads) is always a challenge with REST API during transfer, so is there an optimistic solution for this.
If not chunking is there any other way like reading bytes into fixed buffer and transmit over HTTP. The service provider is not willing to change the REST contract and always expects application/json media type with attachment being a part of multipart request.
The usecase that comes to my mind is how to handle attachments in email typically big in size.
Please advise.
The main use-case for Chunked transfer encoding is sending data for which you can't accurately predict the Content-Length.
So if your JSON is built up dynamically, and you can't accurately know in advance how many bytes it will be, chunked is the way to go.
However, if you can predict the Content-Length, it's not needed to use Chunked, and you shouldn't, as there is a bit of overhead in doing so. Chunked is not a requirement for streaming responses.
Also note:
Most programming languages/HTTP frameworks will automatically use Chunked encoding in circumstances where the Content-Length cannot be predicted. For Node.js this is for example any time you just write to the response stream, and for PHP this is basically always unless you explicitly set a Content-Length. So in many cases there's basically nothing you need to do to switch to this.
Chunked no longer exists in HTTP/2 as HTTP/2 has its own framing protocol that supports responses with unknown sizes.
We have a requirement to transfer documents (HL7/FHIR and others) over HTTP (REST), but where the network architecture involves multiple hops through proxies that unpack and repack the TLS. So the TLS-encryption doesn't help us much, since we break the law if the proxies in-between can see the data. (This is all within a secure semi-private network, but that doesn't help us because different organizations own the different boxes and we're required to have end-to-end encryption).
So we need payload encryption of the documents transfered with HTTP/REST. The common way to do it here is to use SOAP and encrypt the envelope.
What is the best mechanism for content/payload-encrypting REST-transported data? Is there a standard or open specification for it? In health or some other industry?
I guess one feasible way could be to add a special content type that requests encrypted content (S/MIME-based?) to the HTTP request header. A FHIR/REST-server should then be able to understand from the Accept-header of the HTTP-request that the content must be encrypted before responding. As long as the URL itself isn't sensitive, I guess this should work?
I guess also that maybe even the public key for encrypting the content could be passed in a special HTTP request header, and the server could use this for encryption? Or the keys could be shared in setting up the system?
Is this feasible and an ok approach? Has payload-encryption been discussed in the HL7-FHIR work?
It hasn't been discussed significantly. One mechanism would be to use the Binary resource. Simply zip and encrypt the Bundle/resource you want to transfer and then base-64 encode it as a Binary resource. (I suggest zipping because that makes it easy to set the mime type for the Binary.) That resource would then be the focus of a MessageHeader that would expose the necessary metadata to ensure the content is appropriately delivered across the multiple hops.
This might be a good discussion to start up on the HL7 security list server as they will likely have some additional thoughts and recommendations. (And can also ensure that wherever things land in terms of final recommendation, that that gets documented as part of the spec :>)
Based on a what I found on the internet, MIME (Multipurpose Internet Mail Extensions, now Internet Media Type (?)) is a way to describe file types (a header used by several protocols).
So, MIME itself is not a protocol, rather an extension used by other protocols, right ?
This means that the extension is used at the application layer by the applications with no protocol doing anything other than carrying the MIME header.
So, if I send a mail with a mp3 attachment, SMTP/other application layer protocol recognizes that this is an mp3 attachment or it is the duty of the application solely to recognize the file? In that sense, MIME cannot be called as an extension to SMTP but rather a feature to be used by applications.
If SMTP does not recognize that this is a different kind of file, how will it properly store it at the mail server ? (e.g. a MPEG video file needs a particular format to be stored, how will mail server store it without giving it any special treatment ? )
Sorry if my questions sound a bit vague but I want to get an idea of how different protocols (especially, SMTP) use MIME.
Thanks for your help.
RFC 822 email was originally purely plain-text, 7-bit US-ASCII. MIME specifies a facility for encapsulating other media types in email containers. It does not specify any changes to SMTP (although e.g. the 8BITMIME ESMTP extension is useful for simplifying transport of MIME messages). Thus, it is an extension of an existing protocol, not a distinct protocol in its own right. This is also demonstrated by the fact that other protocols -- notably, HTTP -- have incorporated (parts of) MIME for tagging of content types and encodings.
An Internet Media Type is only one aspect of what MIME used to codify; the mechanisms for specifying character sets and encodings are still defined in MIME proper.
Traditionally, the mail server simply stores the bare RFC822 message in its message store; it is the responsibility of the mail client to parse and possibly manipulate any MIME structure in the body for display and interaction. (The fact that RFC 822 has been superseded by 2282 and then 5322 has not fundamentally changed the actual mail message format.)
Some servers deviate from this model; for example, Microsoft Exchange seems to parse all incoming messages in order to borg them into its internal format, somewhat to the detriment of its interoperability with standard tools, and the sanity of those few of us who require reliable, felicitous access to our actual email.
The SMTP protocol itself knows nothing about the MIME format, but the SMTP server itself has to at least implement basic rfc0822 support in order to ad the Received headers, however, it does not need to implement MIME.
How does the server save the file to disk? The same way it received it from the client over the TCP/IP stream. It just saves the raw bytes sent (with the addition of the addition of a Received header I mentioned).
In other words, you are way over-thinking this. The SMTP server doesn't have to know anything about mp3 file attachments or anything else because the MIME format (it's not a protocol) is just a way to serialize the mp3 data in a message.
Whats the best way to send large files from one node.js server to another? We tried to encode it with base64 and send it over an allready existent tls socket connection, but the base64 string is to long so the socket splits it in several parts. We also thought to send it via http methods but that seems not the best way for us. Any ideas?
Unless there are no special requirements, I'd use HTTP. HTTP cliens and servers are both available and rather mature in node.js, and HTTP gives you additional features (i.e. Caching, optimistic transactional behaviour, content negotiation, partial requests, etc.).
Don't roll your own protocol based on plain sockets, you are reinventing the wheel. But you might consider other protocols such as FTP as well.