I implement the Kafka protocol for the Dart language. My implementation is based on the documentation at Kafka Protocol Guide.
I have 2 questions to that:
1.) There are 3 versions of the request header and 2 versions of the response header. Beside that, the requests and responses are also available in multiple versions. What is not documented from my point of view is which version of request uses which version of the request header (same for response and response header). Where is this information documented?
2.) The documentation references the KIP-482. Is the KIP implemented as documented or were there changes made while it was implemented?
The version is up to the client implementation to decide.
If you have no client ID and no tags, use request v0, if only client ID, then v1, both are v2
Similarly for the response, tags are used only in v1
After der discussion with OneCricketeer I looked into the sources of the Java client. There I found the file ApiMessageTypeGenerator.java.
The code there is used to generate the source of another class with the logic to determine the header version of a request/response. For the requests it looks like that every request that supports flexible versions uses a header of version 2 and all other a header of version 1. For the responses it is header version 1 for responses supporting flexible versions and otherwise header version 0. Additionally, there are 2 hardcoded exception in the linked source.
The generator works on files like ProduceRequest.json.
One way of doing is to convert the binary data to Base64 and send it along with meta data as MediaType.APPLICATION_JSON.
The problem with this is client has to convert the binary data into Base64 before sending.
I also tried sending using MediaType.APPLICATION_OCTET_STREAM format through REST. This works fine for binary data only.
One option is to send meta data in headers while using MediaType.APPLICATION_OCTET_STREAM.
Is there any better way?
Thanks
Use multipart/form-data. This is what it's meant for. Not sure which Jersey version you are using, but here is the link for the Jersey 2.x documentation for Multipart support. Here's for 1.x (not really much information). You will need to do some searching for using multipart with Javascript clients (there is a bunch of information out there)
Here is a good example of using Jersey 2.x with both the server side and the client API.
Here is an example with Jersey 1.x. You can see the API it not much different.
I have 3 client libraries that all post to the same API endpoint, just in different languages (python, ruby, php). I want to be able to tell which library a particular API call came from.
My initial thought was to just append some information to the HTTP POST/GET which would identify the client library, but I wasn't sure if that is a good method.
What have you done to verify which library an API call has originated from?
Thanks in advance for the help!
Most request libraries set a default User-Agent header. Introspect that header.
Python's urllib2 sets that header to Python-urllib/MAJOR.MINOR where the latter two values are taken from the Python version. For Python 2.7 you'd get Python-urllib/2.7 for example. Other Python libraries (like requests) have unique default User-Agent headers too.
Ruby's Net::HTTP library appears to set it to Ruby.
Of course, each of these libraries is perfectly capable of changing the User-Agent string, but most developers won't bother setting these unless the server appears to vary their response based on that string (and won't allow anything but User-Agent strings used by desktop browsers to access the server).
I am using Play Framework version 2.2 and I am trying to get the mime type of a partially uploaded file so I can do a direct upload to an Amazon S3 instance. What is the best practice for doing this?
I am currently using FlowJS but it doesn't look like they have anything in particular for dealing with mime types. Additionally, I plan on making mobile apps that will use the same API so it would be best if it was on the server side and not the client side.
The only solution I can think of is parsing the extension and mapping that to a mime type, but that sounds like a hacky way to do it.
We are going to set up a solution where the iPhone is requesting data from the server. We have the option to decide what kind of solution to put in place and we are not sure about which way to go.
Regarding SOAP I think I have the answer, there are no really stable solution for doing this (I know there are solutions, but I want something stable).
How about REST?
Or is it better to just create our own XML? It's not going to be so complicated reguest/respons-flow.
Thanks in advance!
I've created an open source application for iPhone OS 3.0 that shows how to use REST & SOAP services in iPhone application, using XML (using 8 different iPhone libraries), SOAP, JSON (using SBJSON and TouchJSON), YAML, Protocol Buffers (Google serialization format) and even CSV from a PHP sample app (included in the project).
http://github.com/akosma/iPhoneWebServicesClient
The project is modular enough to support many other formats and libraries in the future.
The following presentation in SlideShare shows my findings in terms of performance, ease of implementation and payload characteristics:
http://www.slideshare.net/akosma/web-services-3439269
Basically I've found, in my tests, that Binary Plists + REST + JSON and XML + the TBXML library are the "best" options (meaning: ease of implementation + speed of deserialization + lowest payload size).
In the Github project there's a "results" folder, with an Excel sheet summarizing the findings (and with all the raw data, too). You can launch the tests yourself, too, in 3G or wifi, and then have the results mailed to yourself for comparison and study.
Hope it helps!
REST is the way to go. There are SOAP solutions, but given that all people end up doing with SOAP can be done with RESTful services anyway, there's simply no need for the overhead (SOAP calls wrap XML for data inside of an XML envelope which must also be parsed).
The thing that makes REST as an approach great is that it makes full use of the HTTP protocol, not just for fetching data but also posting (creating) or deleting things too. HTTP has standard messages defined for problems with all those things, and a decent authentication model to boot.
Since REST is just HTTP calls, you can choose what method of data transfer best meets your needs. You could send/receive XML if you like, though JSON is easier to parse and is smaller to send. Plists are another popular format since you can send richer datatypes across and it's slightly more structured than JSON, although from the server side you generally have to find libraries to create it.
Many people use JSON but beware that it's very finicky about parsing - mess up a character at the start of a line, or accidentally get strings in there without escaping "'" characters and there can be issues.
XML Property-lists (plist) are also a common way to serialize data in Cocoa. It is also trivial to generate from other languages and some good libraries exist out there.
You are not saying how complex your data structures are and if you actually need state handling.
If you want to keep your network traffic to a minimum, while still keeping some of the structured features of XML, you might have a look at JSON. It is a very light weight data encapsulation framework.
There are some implementations available for iPhone, for instance TouchJSON
Claus
I would go with simple HTTP. NSURLConnection in the Cocoa libraries make this pretty simple. Google Toolbox for Mac also has several classes to help parsing URL-encoded data.
I think it's obvious that REST is the new king of servers communication, you should definitely use REST, the questions should be what REST methodology you should use and what coding language, in my post I present few very simple implementations for REST servers in C#, PHP, Java and node.js.