I am trying to create a nice helper class to parse some query parameters so they can be used to filter an array of data.
I am using the following syntax for my query parameters:
?filter[name,contains,string]=foo // name field contains the string "foo"
?filter[id,gte,number]=123 // item field is greater then or equal to 123
?filter[type,eq,string]=foo|bar // type field is equal to "foo" or "bar"
(you can also use , between the values to act as an AND operator, but on a single field, it doesn't have many use cases eg gt 1 AND 2 isn't a great filter)
These query params can be combined, eg
?filter[name,contains,string]=foo&filter[id,gte,number]=123
And this will act as an AND condition, meaning data is returned that matches both filters.
Great so far. But I want to implement an OR condition for multiple separate filters.
?filter[]=... is used for the actual filter query, so I can't really use that to set the OR condition.
I've tried finding REST API implementations that use somethig similar to this filtering syntax, the closest is Laravel, but the docs don't go into any details about it.
Hoping someone can point me to a good resource about REST API filtering/help me figure out a nice way to implement the OR condition.
Unfortunately I think if you want the flexibility you're looking for, it'll probably be best to define a proper grammar and accept a "filter string" that follows that syntax. This would mean you have to define the grammar and behavior of defaults, parse it, and turn it into a query that matches your underlying storage system (e.g., a SQL query or ORM API calls).
For example, this might make your filters look something like:
?filter=name:("foo" AND id >= 123) OR id <= 567
And it turns out that there's quite a lot to decide when you start introducing complex filtering operations.
For more reading, Google's API filtering syntax and guidelines are: https://google.aip.dev/160 and the EBNF grammar is https://google.aip.dev/assets/misc/ebnf-filtering.txt
Related
The standard REST verb for returning a value GET can take different parameters to select what to "get". Often there is one that takes an id to get a single value, and often some sort of search criteria to get a list.
Is there a standard way to specify the filtering and sorting of the data that is being searched for? For example, if I have an invoice record I'd like to write a GET query that says "give me all invoices for customer 123, with total > $345 and return in descending order of date".
If I were writing this myself I'd have something like:
GET http://example.com/mydata?query="customer=123&&total>345.00"&order="date"
(Note I didn't urlencode the url for clarity, though obviously that is required in practice, but I hope you get what I mean.)
I can certainly write something for this, but I am wondering if there is a standardized way to do this?
Is there a standard way to specify the filtering and sorting of the data that is being searched for?
Not that I'm aware of.
Note that HTTP doesn't really have queries (yet); HTTP has resource identifiers.
We've got a standard for resource identifiers (RFC 3986) and a standard for URI templates (RFC 6570) that describes how to produce a range of identifiers via variable expansion.
But as far as I can tell there is no published "standard" that automatically transforms a URI into a SQL query.
It's possible that one of the "convention over configuration" frameworks (ex: Rails) might have something useful here, but I haven't found it.
I originally posted multiple filters containing multiple values in JSON as part of my GET request but I believe this is bad practise, so I changed it to a POST but I don't like it as getting results from a query has nothing to do with a POST so I guess I'll have to use a query string
Most filter examples I have found are either using one filter or one value, but I am looking as to whether or not there is a best practise to pass multiple filters with multiple values for filtering as a single parameter in the query string.
For example, this is a basic one which looks for all cars that are red
GET /cars?color=red
But what if I wanted to look for all cars that are
red, blue or green and
have 2 seats or less and
their brand name starts with b and
can be bought in the US, UK or Germany
Would the following be ok?
http://myserver/api/cars?color=red|blue|green¬seats<=2¬brand[startswith]b¬country=USA|UK|Germany
I'm suggesting the use of the:
| character as a separator between each values for a given filter
¬ character as a separator between each filters
[startsWith] to handle the search type, but could contain [=, <=, >=, <>, [contains],[endswith], etc...
This would then be parsed in the server end and the relevant filters would be build accordingly based on the provided values
Hope this make sense but I'm really interested as to whether or not there is a standard/best practise used for such scenarios with REST in mind?
Thanks.
As in most design questions, the key is having a consistent design for all your APIs. You can follow certain well-known guidelines/standards to make your API easily discoverable.
For example, take a look at OData. The "Queries" section on this page is relevant to your question. Here's an example:
https://services.odata.org/v4/TripPinServiceRW/People?$top=2 & $select=FirstName, LastName & $filter=Trips/any(d:d/Budget gt 3000)
Another option is the OpenSearch standard. The relevant section is here. Here's an example:
https://opensearch.php?recordSchema=html&query=dc.creator any Mill* Grad*
Another interesting option is GraphQL, which makes it easier to map query parameters to data fetch parameters. It uses a filter payload instead of query parameters. See the spec here: GraphQL Spec.
I am starting to implement filtering through query strings in my API REST application where I can perform filtering on any field passed to a entity in my database. But there are especial parameters that I want to differentiate like; sort, page, direction, etc from the rest of fields that will be used to filter the collection.
I want to avoid using an implementation like:
/?filters=field1:value1,field2:value2&page=3&per_page=50
Because I will need to make a custom parser to the filters value.
My desired structure is something more plain, like this:
/?lastname=Halden&country=somewhere&$sort=lastname
So, all the properties that aren't prefixed with $ are used to make the filter and the other properties with the prefix are used to tweak the result.
My actual question is that if it is safe?. If there can exist a problem in the moment to parse the whole query string in some libraries.
It should be ok. As a matter of fact, ODATA, one of the widely used REST standards in enterprise software uses $ the same way you want to use. Another sensible option you could use is _ (e.g. _sort, _page, etc.)
This is sort of a follow-up to someone else's question about filtering/querying a list of cars. There the recommendation for a RESTful filtering request was to put filter expressions in the query of the URI, like this:
/cars?color=blue&type=sedan&doors=4
That's fine. But what if my filtering query becomes more complicated and I need to use Boolean operators, such as:
((color=blue OR type=sedan) AND doors=4) OR color=red
That is, I want to find a four-door blue car or a four-door sedan, but if the car is red I'll take it without caring about any of the other properties.
Is there any sort of convention for providing Boolean expressions in a RESTful URI's query parameters? I suppose I could by create some new querying expression language and put it in a POST, but that seems like a heavy and proprietary approach. How are others solving this?
It is perfectly okay to use
/cars/color:blue/type:sedan/doors:4
instead of
/cars?color=blue&type=sedan&doors=4
The URL standard says only that the path should contain the hierarchical part, and the query should contain the non-hierarchical. Since this is a map-reduce, using / is perfectly valid.
In your case you need a query language to describe your filters. If I were you I would copy an already existing solution, for example the query language of a noSQL database which has a REST API.
I think resource query language is what you need. I think you could use it like this:
/sthg?q="(foo=3|foo=bar)&price=lt=10"
or forget the default queryString parser, and like this:
/sthg?(foo=3|foo=bar)&price=lt=10
I suggest you to read the manual for further details.
Since I found no other URL compatible query language (yet), I think the only other option to serialize another query language and send it in a param, like SparSQL
http://localhost:8003/v1/graphs/sparql?query=your-urlencoded-query
by marklogic7. Hydra defines a freeTextQuery in its vocab, so they follow the same approach. But I'll ask Markus about this. It's a complicated topic, since according to the self-descriptive messages constraint you should describe somewhere what type of query language you use in the URL. I am not sure about this. :S
conclusion:
In order to support ad-hoc search queries we need a standard way to describe them in the link meta-data. Currently there are only a few standards about this. The most widely used standard is URI templates which does not support nested statements, operators, etc... for what I know. There is a draft called link descriptions which tries to fill the gap, but it is incomplete.
One possible workaround to define an URI template with a single q parameter which has rdf:type of x:SearchQuery and rdfs:range of xsd:string, and create another vocab about how to describe such a x:SearchQuery. After that the description could be used to build search forms, and validate queries sent to the server. Already existing queries could be supported too with this approach, so we don't need a new one.
So this problem can be solved with vocabs or new URI template standards.
I have seen many use a query string as you have provided - much like a SQL query string.
Here are just two examples:
Socrata (Open Data Portal company)'s SoQL (SQL variant): http://dev.socrata.com/consumers/cookbooks/querying-block-ranges.html
openFDA (API from fda.gov for open data) uses a similar string-based query parameter which maps to ElasticSearch queries, I believe: https://open.fda.gov/api/reference/#query-syntax
Try using 1 for true, 0 for false.
/_api/web/lists/getbytitle('XYZ')/items?$filter=Active eq 1
I'm looking for a good way to form a URI for a resource that filters on a collection of values contained within records. For example, say you have a recipe database and you want to search for recipes that contain "cherry" (obviously most recipes would contain multiple ingredients).
If I just want to filter on single values, I could do something similar to the following:
/recipe/search/?name=Spaghetti
But what about filtering on multiple values? I was considering something like the following:
/recipe/search/?ingredients=contains=cherry
Any thoughts on this? Is there a "standard" for a filter of this kind?
Update: One problem I have with my idea is the way it gets handled on the backend (in my case Rails). When querying the server with this particular format, Rails generates a Ruby hash that could get ugly like the following:
{"ingredients"=>"contains=cherry",
"action"=>"search",
"controller"=>"recipe"}
Your URI
First of all, in contrast to other answers I'll start from a REST perspective and then find appropriate additions to it. I am not strong in Ruby so bear with no-code on the backend.
Recipies are the entities you wanna present
your users find them at /recipies
HTTP has QUERYS for filtering
wanna have sorted those recipies by date? use /recipies/?sortby=date&sort=asc
Only recipies with cherries goes similarly: /recipies/?ingredients=cherry
So that's the REST way of structuring your basic URL.
For multiple matches there is no official way to do it, but i'd follow user1758003. This is an intuitive construction of the url and easily to parse on the backend, so we have /recipies/?ingredients=cherry,chocolate
Don't forget /recipies/search is not restful because it mirrors recipies and does not represent an independent entity. However it is a great place to host a searchform for visitors to your site.
Now there are some additional questions packed into the first, let me address them one by one:
I have a website consuming this api, how should the search form look like?
Give your visitors a /recipie/search page or a quick filtering possibility on /recipies.
Just set the <form action="/recipies" type="GET">. The browser will add all parameters as an Query string after /recipies.
Advanced functionality
A request to /recipies should list all. If you add a query every parameter of the query must be respected. so /recipies/?ingredients=cheese MUST only return cheesy recipies.
For multiply query parameter values there is no fixed standard but I'd like a service to be intuitive.
I read GET /recipies/?ingredients=cherry,chocolate as Get me the recipies which have ingredients of cherry and chocolate. To get a list of recipies containing either cherry or chocolate I'd want to write the URI like /recipies/?ingredients=cherry|chocolate which makes it visually very different from a comma and has a predefined meaning (OR) in programming contexts.
I'm not familiar with the specifics of ruby hashes but I'm guessing the hash is created to uniquely identified the query both at the http and data access levels?
Regardless, you want to be careful about URL encoding if you wish to use json in a query parameter. Another thing to keep in mind is the term "search" could be considered repetitive. If your server is being accessed using a GET method and you have criteria then hopefully you're not modifying any state in the back end. Not your question but just thought I'd mention it.
So...
https://yourserver.com/approot/recipe/search?ingredients=cherry,cheese&type=cake
HTTP doesn't define commas as a query string separator so your framework should be able to parse 2 query string parameters:
ingredients: "cherry,cheese"
which you should be able to easily covert to an an array using split or whichever equivalent function ruby provides.
and
type: "cake" (extra query term added to illustrate a point and because cherry cheese cake is awesome and cake is not an ingredient)
If I understand your example correctly you would end up with:
{
"ingredients"=>"cherry,cheese",
"type"=>"cake",
"action"=>"search",
"controller"=>"recipe"
}
Is this what you where looking for?
Most of the REST webservice using JSON data only.So use JSON format which will return single string value only. From this JSON format you can send the array value also.
For array value means you to convert that array into the JSON format like this
from php:
$ingredients = array('contains'=>array('fruits'=>'cherry,apple','vitamins'=>'a,d,e,k'));
$ingredients_json = json_encode($ingredients);
echo $ingredients_json;
it will return json format like this:
{"contains":{"fruits":"cherry,apple","vitamins":"a,d,e,k"}}
and you can use this JSON string in the url
/recipe/search/?ingredients={"contains":{"fruits":"cherry,apple","vitamins":"a,d,e,k"}}
in the server side we have the option to decode this JSON format value to the array.
{"ingredients"=>"{\"contains\":{\"fruits\":\"cherry,apple\",\"vitamins\":\"a,d,e,k\"}}",
"action"=>"search",
"controller"=>"recipe"}