I'm new to StreamSets, thanks in advance for any help.
In my pipeline records (JSON) I have a field with Geo-coordinates (lat, lon) and I'm trying to add more meta-data to them. I'm wondering if it's possible to use a HTTP Client Processor to realize the operation described here https://nominatim.org/release-docs/develop/api/Reverse/ using the lat, lon values of my records. If it is, could you point me for some documentation or article that describes how to do it?
I've been able to use HTTP Client as Origins in other occasions, but i can't figure out how to use the values in the URL.
For example if the values of my record were
{
"lat": 41.195519,
"lon":-8.666526,
"format": "jsonv2"
}
The url should look like:
https://nominatim.openstreetmap.org/reverse?format=jsonv2&lat=41.226599&lon=-8.709737
Figured out, simply GET to
https://nominatim.openstreetmap.org/reverse?format=${record:value('/format')}&lat=${record:value('/lat')}&lon=${record:value('/lon')}
Related
I'm currently attempting to use the CoinMarketCap API but finding it frustrating.
I'm wanting to use this URL to query their API:
https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest
However, rather than finding all, or just simply filtering based upon the number, I want to find a certain few coins.
So for example, I want to only find Bitcoin, Ethereum and Cardano.
Looking at their docs, it suggests you can sort by name, but it appears this is only listing them alphabetically, which I don't want to do.
So can anyone suggest how to query their API successfully and find just Bitcoin, Ethereum and Cardano using that GET URL above?
Here's the URL to the specific URL for the API request: https://coinmarketcap.com/api/documentation/v1/#operation/getV1CryptocurrencyListingsLatest
For this purpose, you can use the endpoint Quotes Latest:
https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest
It allows you to pass a list of identifiers in a string as a parameter, like this:
1,1027,328
or a list of slugs:
bitcoin,ethereum,monero
or a list of symbols
BTC,ETH,XMR
If you trying to scrape information about new listings at crypto exchanges, you can be interested in this API:
https://rapidapi.com/Diver44/api/new-cryptocurrencies-listings/
It includes an endpoint with New Listings, New Pairs from the biggest exchanges and a very useful endpoint with information about exchanges where you can buy specific coins and prices for this coin at that exchange. It's a bit paid, but it's worth it!
I have a REST syntaxe question:
what url do you give to an endpoint to extract data similar to the record which from the id passed?
By exemple : I have a class Record:
Record {id:12, phoneNumber:"+336746563"}
I want a endpoint who will return all the records who share the same phoneNumber than the record with the id 12
which url respect the most the REST protocol ?
EDIT IMPORTANT : the client DON't know the phone number when he call the url. only the 12 id.
what url do you give to an endpoint to extract data similar to the record which from the id passed?
Anything you want -- the machines don't care what spelling you use for your resource identifiers.
I want a endpoint who will return all the records who share the same phoneNumber than the record with the id 12
/all-records-with-same-phone-number-as?id=12
/all-records-with-same-phone-number-as?12
/all-records-with-same-phone-number-as/12
All of these examples are fine. They have different trade offs -- the first one is really easy to generate using an HTML form. The last once allows you do interesting things with relative references and dot-segments.
/record/12/all-records-with-same-phone-number
similar to the above, we've just juggled the order of the path segments a little bit. Might be useful if we want to have relative references to other resources under the same /record/12 stem.
If you are expecting to need to paginate, then you might want to think about how the paging parameters fit with everything else. Again, the machines don't care, but some spellings are easier to work with than others.
I am not sure if I understand the question but let me try.
You can do this in various ways which is best suited for you w.r.t. your programming language. For example, domain.com/api/records/123456 can be the end-point. 123456 is a parameter and your code will return all the records having phoneNumber=123456.
Alternatively, the end-point can be domain.com/api/records?phoneNumber=12345.
Or Even, domain.com/api/records/123456/phonenumber.
The other option is to have the request data in the body and the domain would just look like domain.com/api/records with request as {"PhoneNumber":"123456"}
AFAIK, all these URLs respect REST protocol.
I would use something like
/service/records/{id}/similar
where the service would define similarity. It depends on the usecases for when Record becomes more complex and the client should able to specify fields.
This would sooner or later result in queries that would not be based on an existing record and my look like
/service/records?foo=1&bar=2
I could also think of
/service/records/phone-number/12345
because you are really interested in records with the same phoneNumber, not similarity?
But again, as things get more complex you will be better off with a query I think.
I'm trying to consume JIRA 2 API and trying to get custom fields. I want to further filter by passing appropriate criteria in URI itself. Current query I'm using is something similar to this:
http://localhost:8522/jira522/rest/api/2/issue/createmeta?expand=projects.issuetypes.fields
The result I'm getting from above request is about 2000 lines.. How can I further filter to get only Custom_fields and also under custom fields I need to only the ones which are required?
I'm pretty new to REST API. Please guide me If anything is wrong... TIA. I spent a lot of time browsing but don't know what exactly I need to search for or where exactly I need to get started.
You can use another queryParam just like expand and add further filtering or pagination.
http://localhost:8522/jira522/rest/api/2/issue/createmeta?expand=projects.issuetypes.fields&limit=1000
I understand the definition of GET and POST as below.
GET: List the members of the collection, complete with their member URIs for further navigation. For example, list all the cars for sale.
POST: Create a new entry in the collection where the ID is assigned automatically by the collection. The ID created is usually included as part of the data returned by this operation.
MY API searches for some detail in server with huge request payload with JSON Message in that case Which Verb should i use ?
Also can anyone please let me know the length of the characters that can be passed in query string.
The main difference between a GET and POST request is that in the former, the entire request is encoded as part of the URL itself, whereas in the latter, parameters are sent after the header. In addition, in GET request, different browsers will impose different limits on how big the URL can be. Most modern browsers will allow at least 200KB, however Internet Explorer seems to limit the URL size to 2KB.
That being said, if you have any suspicion that you will be passing in a large number of parameters which could exceed the limit imposed on GET requests by the receiving web server, you should switch to POST instead.
Here is a site which surveyed the GET behavior of most modern browsers, and it is worth a read.
Late to the party but for anyone searching for a solution, this might help.
I just came up with 2 different strategies to solve this problem. I'll create proof of concept API and test which one suites me better. Here are the solution I'm currently thinking:
1. X-HTTP-Method-Override:
Basically we would tunnel a GET request using POST/PUT method, with added X-HTTP-Method-Override request header, so that server routes the request to GET call. Simple to implement and does work in one trip.
2. Divide and Rule:
Divide requests into two separate requests. Send a POST/PUT request with all payload, to which server will create necessary response and store it in cache/db along with a key/id to access the data. Then server will respond with either "Location" header or the Key/id through which the stored response can be accessed.
Now send GET request with the key/location given by server on previous POST request. A bit complicated to implement and needs two requests, also requires a separate strategy to clean the cached responses.
If this is going to be a typical situation for your API then a RESTful approach could be to POST query data to a buffer endpoint which returns a URI from which you can GET your results.
Who knows maybe a cache of these will mitigate the need to send "huge" blobs of data about.
Well You Can Use Both To get Results From Server By Passing Some Data To server
In Case Of One Or Two Parameters like Id
Here Only One Parameter Is Used .But 3 to 4 params can Be used This Is How I Used In angularjs
Prefer : Get
Example : $http.get('/getEmployeeDataById?id=22');
In Case It Is Big Json Object
Prefer : Post
Example : var dataObj =
{
name : $scope.name,
age : $scope.age,
headoffice : $scope.headoffice
};
var res = $http.post('/getEmployeesList', dataObj);
And For Size Of Characters That Can Be Passed In Query String Here Is Already Answered
If you're getting data from the server, use GET. If you want to post something, use POST. Payload size is irrelevent. If you want to work with smaller payloads, you could implement pagination.
I have REST resource (Ex: Tickets). To be able to obtain a set of Tickets that match a given set of constraints (Ex: start date, end date, price and other criterion) a user will need to pass information. This information can be included as query parameters and the protocol can define:
GET: Tickets?start-date=date&end-date=date&price=someprice...
The set of constraints to pass could be a lot.
In such situations, is it better to use a POST and pass the set of constraints as JSON object within the body?
POST: Tickets
Body:
{
"start-date": "date"
"end-date" : "date"
. . .
}
What are the drawbacks of such an approach? Does it still agree with the REST guidelines?Ref: http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
Another alternative is the client could create a new resource called "Constraints" on the server, obtain a constraint-id (ex:123) as a response. Then it could use:
GET: Tickets?constraints-id=123
But this will mean that the server will periodically have to expire and delete "Constraint" objects, as clients might keep creating those without completing the business flow (ex: without confirming a Ticket in the end)
A third approach could be still use POST, but not create any resource. We can use a URI scheme like this:
POST: Tickets\Constraints
Body:
Body:
{
"start-date": "date"
"end-date" : "date"
. . .
}
Response:
200 OK ...
Tickets
This will mean that allthough no resource was created on the server, the need to POST the constraints to obtain Tickets is still made clear.
Which of these approaches would you recommend? What is most intuitive? Or is other any other alternative you would recommend?
Simply according to the HTTP spec, a POST is not a valid method to send a large amount of data for a query, as the intention is that the body of the request is to be stored by the server in some way, which is not the case in your example.
My current project faced the same problem and we decided to go with the more correct GET with many templated query parameters. Despite supporting over a dozen query params which can be quite long in length, most servers specify a GET request maximum length of 8KB, which I would expect to be an ample amount. I suppose this limit could be reached if you were attempting to send a GET with a large amount of the same query parameter to describe a long list, but if this is this case then it would suggest taking a step back and seeing how this has become a requirement of the API.
In my opinion a GET is the most intuitive and clearest use, and definitely seems to be the "correct" RESTful implementation. If the size of the request is an issue for you and you control the environment you are deploying to, you can even increase your server's max request size.
Yes, definitely OK and a good idea, especially if the post data is large, as it may exceed the max url length. It is better as part of the body of the message rather than on the url.