Boolean logic in RESTful filtering and queries - rest

This is sort of a follow-up to someone else's question about filtering/querying a list of cars. There the recommendation for a RESTful filtering request was to put filter expressions in the query of the URI, like this:
/cars?color=blue&type=sedan&doors=4
That's fine. But what if my filtering query becomes more complicated and I need to use Boolean operators, such as:
((color=blue OR type=sedan) AND doors=4) OR color=red
That is, I want to find a four-door blue car or a four-door sedan, but if the car is red I'll take it without caring about any of the other properties.
Is there any sort of convention for providing Boolean expressions in a RESTful URI's query parameters? I suppose I could by create some new querying expression language and put it in a POST, but that seems like a heavy and proprietary approach. How are others solving this?

It is perfectly okay to use
/cars/color:blue/type:sedan/doors:4
instead of
/cars?color=blue&type=sedan&doors=4
The URL standard says only that the path should contain the hierarchical part, and the query should contain the non-hierarchical. Since this is a map-reduce, using / is perfectly valid.
In your case you need a query language to describe your filters. If I were you I would copy an already existing solution, for example the query language of a noSQL database which has a REST API.
I think resource query language is what you need. I think you could use it like this:
/sthg?q="(foo=3|foo=bar)&price=lt=10"
or forget the default queryString parser, and like this:
/sthg?(foo=3|foo=bar)&price=lt=10
I suggest you to read the manual for further details.
Since I found no other URL compatible query language (yet), I think the only other option to serialize another query language and send it in a param, like SparSQL
http://localhost:8003/v1/graphs/sparql?query=your-urlencoded-query
by marklogic7. Hydra defines a freeTextQuery in its vocab, so they follow the same approach. But I'll ask Markus about this. It's a complicated topic, since according to the self-descriptive messages constraint you should describe somewhere what type of query language you use in the URL. I am not sure about this. :S
conclusion:
In order to support ad-hoc search queries we need a standard way to describe them in the link meta-data. Currently there are only a few standards about this. The most widely used standard is URI templates which does not support nested statements, operators, etc... for what I know. There is a draft called link descriptions which tries to fill the gap, but it is incomplete.
One possible workaround to define an URI template with a single q parameter which has rdf:type of x:SearchQuery and rdfs:range of xsd:string, and create another vocab about how to describe such a x:SearchQuery. After that the description could be used to build search forms, and validate queries sent to the server. Already existing queries could be supported too with this approach, so we don't need a new one.
So this problem can be solved with vocabs or new URI template standards.

I have seen many use a query string as you have provided - much like a SQL query string.
Here are just two examples:
Socrata (Open Data Portal company)'s SoQL (SQL variant): http://dev.socrata.com/consumers/cookbooks/querying-block-ranges.html
openFDA (API from fda.gov for open data) uses a similar string-based query parameter which maps to ElasticSearch queries, I believe: https://open.fda.gov/api/reference/#query-syntax

Try using 1 for true, 0 for false.
/_api/web/lists/getbytitle('XYZ')/items?$filter=Active eq 1

Related

REST API filter queries - handling OR operations

I am trying to create a nice helper class to parse some query parameters so they can be used to filter an array of data.
I am using the following syntax for my query parameters:
?filter[name,contains,string]=foo // name field contains the string "foo"
?filter[id,gte,number]=123 // item field is greater then or equal to 123
?filter[type,eq,string]=foo|bar // type field is equal to "foo" or "bar"
(you can also use , between the values to act as an AND operator, but on a single field, it doesn't have many use cases eg gt 1 AND 2 isn't a great filter)
These query params can be combined, eg
?filter[name,contains,string]=foo&filter[id,gte,number]=123
And this will act as an AND condition, meaning data is returned that matches both filters.
Great so far. But I want to implement an OR condition for multiple separate filters.
?filter[]=... is used for the actual filter query, so I can't really use that to set the OR condition.
I've tried finding REST API implementations that use somethig similar to this filtering syntax, the closest is Laravel, but the docs don't go into any details about it.
Hoping someone can point me to a good resource about REST API filtering/help me figure out a nice way to implement the OR condition.
Unfortunately I think if you want the flexibility you're looking for, it'll probably be best to define a proper grammar and accept a "filter string" that follows that syntax. This would mean you have to define the grammar and behavior of defaults, parse it, and turn it into a query that matches your underlying storage system (e.g., a SQL query or ORM API calls).
For example, this might make your filters look something like:
?filter=name:("foo" AND id >= 123) OR id <= 567
And it turns out that there's quite a lot to decide when you start introducing complex filtering operations.
For more reading, Google's API filtering syntax and guidelines are: https://google.aip.dev/160 and the EBNF grammar is https://google.aip.dev/assets/misc/ebnf-filtering.txt

REST Protocol for searching and filtering

The standard REST verb for returning a value GET can take different parameters to select what to "get". Often there is one that takes an id to get a single value, and often some sort of search criteria to get a list.
Is there a standard way to specify the filtering and sorting of the data that is being searched for? For example, if I have an invoice record I'd like to write a GET query that says "give me all invoices for customer 123, with total > $345 and return in descending order of date".
If I were writing this myself I'd have something like:
GET http://example.com/mydata?query="customer=123&&total>345.00"&order="date"
(Note I didn't urlencode the url for clarity, though obviously that is required in practice, but I hope you get what I mean.)
I can certainly write something for this, but I am wondering if there is a standardized way to do this?
Is there a standard way to specify the filtering and sorting of the data that is being searched for?
Not that I'm aware of.
Note that HTTP doesn't really have queries (yet); HTTP has resource identifiers.
We've got a standard for resource identifiers (RFC 3986) and a standard for URI templates (RFC 6570) that describes how to produce a range of identifiers via variable expansion.
But as far as I can tell there is no published "standard" that automatically transforms a URI into a SQL query.
It's possible that one of the "convention over configuration" frameworks (ex: Rails) might have something useful here, but I haven't found it.

RESTful query API design

I want to ask what is the most RESTful way for queries, I have this existing API
/entities/users?skip=0&limit=100&queries={"$find":{"$minus":{"$find":{"username":"markzu"}}}}
Easily the first parts of the query, skip and limit are easily identifiable however I find the "queries" part quite confusing for others. What the query means is to
Find every User minus Find User entities with username 'markzu'
The reason it is defined this way is due to the internal database query behavior.
Meaning in the NoSQL database we use, the resource run two transactional queries, first is to find everything in the User table minus a find User with a username that was specified (similar to SQL) -- boolean operations. So in other words, the query means, "fetch every User except username 'markzu' "
What is the proper way to define this in RESTful way, based on standards?
What is the proper way to define this in RESTful way, based on standards?
REST doesn't care what spelling you use for resource identifiers, so long as your choice is consistent with the production rules defined in RFC 3986.
However, we do have a standard for URI Templates
A URI Template is a compact sequence of characters for describing a range of Uniform Resource Identifiers through variable expansion.
You are already aware of the most familiar form of URI template -- key-value pairs encoded in the query string.
?skip=0&limit=100&username=markzu
That's often a convenient choice, because HTML understands how to process forms into url encoded queries.
It doesn't look like you need any other parameters, you just need to be able this query from others. So a perfectly reasonable choice might be
/every-user-except?skip=0&limit=100&username=markzu
It may help to think "prepared statement", rather than "query".
The underlying details of the implementation really shouldn't enter into the calculation at all. Your REST API is a facade that makes your app look like an HTTP aware key value store.

Restful URL design full text search on any specified field

My api should support text search on specified fields. So I am thinking what kind of URL style handles it in the best way.
The below pattern, using "q" ,is mentioned in many blogs and documents to be used for full text search but I also need to specify field names:
GET /groups?q=bank+org
So I am thinking to use wildcards like below:
GET /groups?name=*bank*&owner=*org*
I am just wondering if this is aligned with the best practices in the market?
Thanks
Soheil, you are thinking right. "Search" is a "filter parameter" wich always go in the Query String.
When sending parameters that will be used to query a collection of resources you should use... Guess what! Query parameters!
As far as I know, there's no official documentation that states that. It's a common approach and it's widely adopted. The only offical documentation about query string that I'm aware of is the RFC 3986. Quoting:
3.4. Query
The query component contains non-hierarchical data that, along with
data in the path component, serves to identify a
resource within the scope of the URI's scheme and naming authority
(if any). The query component is indicated by the first question
mark ("?") character and terminated by a number sign ("#") character
or by the end of the URI. [...]
For a full text search, you can choose the parameter you find most convenient. Do you think q is a good one? Go for it! But provide a good documentation for it.

Building filter/query string for RESTful API

I'm building RESTful API using Microsoft ASP.NET WebAPI. My problem concerns GET method, or more specifficaly, optimal solution for building query/filter strings in URL's.
In my application, RESTful API acts just as proxy, or access point to my resources stored in database. It is meant to retrieve, insert, modify or delete records. The whole system stores only one type of entity, which is pretty complex, because it contains many various fields, lists and data in general. I'm using NoSQL for storing those entities, as there is pretty many of them.
The problem lies in creating proper query string for retrieving those entities by specified criteria. Classic query string looks like this:
http://localhost/api/entities?field1=val1&field2=val2&field10=val3
What's wrong with this kind of query string is that it only allows to specify fields values by equality operator, and there is no place for logic operators between each field (& is treated as somewhat AND operator).
What I need, is to allow to specify more complex query strings, more like filters. I want to allow to tell system, that I want to get records, which have for instance:
field1 >= value1 OR field2=value2 AND field3 ~ value3
~ is indicates that i want to match field3 to value3 using FUZZY algorithm (although I'm not sure if ~ is proper character to indicate intention of using fuzzy search).
Well, as you can see, classic query string is not capable of supporting that. For now, I'm passing such query string as:
http://localhost/api/entities?query=field1>=value1$OR$field2=value2$AND$field3~value3
I pass such filter as one string to my controller's method and manually parse it. My delimiter is '$', as ASP.NET WebAPI would automatically split those parameters if there was '&' character between them, and I don't want it to do that, because it is unable map logical operators (OR, AND, etc.) to method parameters in such case. I'm not sure if I'm doing the right thing by parsing this string (everything after '?' character) manually:
field1>=value1$OR$field2=value2$AND$field3~value3
My question are:
What is the proper way of representing such queries in RESTful API URLs?
Is there any existing solution for supporting such filters in ASP.NET WebAPI, other than writing my own parsing algorithm?
I would recommend you check out OData. It's built into web api (http://www.asp.net/web-api/overview/odata-support-in-aspnet-web-api). It can be challenging if you're building domain objects from the data and trying to work with those, but if your objects are straight from the database, it can be very powerful. I'm not sure about your fuzzy matching, but some examples of the filtering it's capable of can be seen at http://www.asp.net/web-api/overview/odata-support-in-aspnet-web-api/supporting-odata-query-options.