Using SQL::Abstract I need to type cast an IP column to TEXT in order to be able to search using LIKE.
I only found a "hacky" way to achieve it with:
where( { ip => { '::TEXT LIKE' => $myParameter } } )
Which generates
WHERE ( "ip" ::TEXT LIKE ? )
Question is: Is there a less hacky or official way to achieve this?
Questions are not:
Will the performance be poor?
Should I use a TEXT column instead of an IP column?
Are searches using CIDR a better alternative
The issue was in Mojo::Pg which adds a quote_char of " to the SQL::Abstract object.
When I set this to the empty string, this will work as expected:
where( { 'ip::TEXT' => { 'LIKE' => $myParameter } } )
But, to be complete, I had to use
where( { 'host(ip)' => { 'LIKE' => $myParameter } } )
because ::TEXT will give the IP with an appended /32.
I think you mix a lot of things in your question. You make it sound like it is an SQL::Abstract issue, when your real issue is with the underlying SQL itself.
First of all, I would personally avoid using SQL::Abstract in most cases (it is sometimes VERY slow to create complex queries and you can't be sure of its output) and definitely in cases like this where you want something non-standard.
So, moving on, I am not sure what you mean by IP type, from your postgres tag and the mention of CIDR I suspect you mean the inet type? If so, the equivalent of LIKE is to use subnet masks, which is basically the whole reason to use inet instead of a text/varchar field. For 192.168.* for example you would do something like below using the subnet slash notation:
SELECT * FROM users WHERE ip << inet '192.168.0.0/16'
If you don't want to treat IPs as actual IPs and take advantage of things like above, but instead you want to treat them like text (e.g. you want to search 192.16% with that producing 192.168.* results but along with 192.16.* etc which is not done with subnet masks), then you either use a text type in the first place, or as you said convert on the fly to use LIKE directly:
SELECT * FROM users WHERE TEXT(ip) LIKE '192.168.%'
There is a performance penalty over using the subnet masks, but whether that may be an issue depends on your data of course.
Note cidr works similar to inet, it won't help you with a LIKE.
Related
Let's say we have an HTTP endpoint to get all elements by name?
GET /elements?name={name}
{name} can have a value of CSV or be absent
valid:
GET /elements?name=Bill,Mary,Ann
GET /elements?name=Mike
GET /elements
invalid:
GET /elements?name=
Somehow we find out in controller that name is not passed. We know that the contract implies to return all values for elements. Possible decisions on further actions (I've seen in different projects) are:
using a NULL or a "dummy" substitution like a secret char sequence "!#$#%#$" and juggling them in database while building a query
using if (present) { executeQueryA } else { executeQueryB } logic
I am not sure I like either of these approaches, because when there is more than one optional filter these designs become unmaintainable. Something makes me believe that there is a better way to handle the situation.
What would be a proper design on back-end and in database query to handle the case "select all" when nothing is given? Just a general idea and some pseudo-code will be much appreciated.
I'm trying to write a prometheus query in grafana that will select visits_total{route!~"/api/docs/*"}
What I'm trying to say is that it should select all the instances where the route doesn't match /api/docs/* (regex) but this isn't working. It's actually just selecting all the instances. I tried to force it to select others by doing this:
visits_total{route=~"/api/order/*"} but it doesn't return anything. I found these operators in the querying basics page of prometheus. What am I doing wrong here?
May be because you have / in the regex. Try with something like visits_total{route=~".*order.*"} and see if the result is generated or not.
Try this also,
visits_total{route!~"\/api\/docs\/\*"}
If you want to exclude all the things that has the word docs you can use below,
visits_total{route!~".*docs.*"}
The main problem with your original query is that /api/docs/* will only match things like /api/docs and /api/docs//////; i.e. the * in your query will match 0 or more / characters.
I think what you meant to use was /api/docs/.*.
Using Kapacitor 1.3 and I am trying to use the following where node to keep measurements with an empty tag. Nothing is passing through and I get the same result with ==''.
| where(lambda: 'process-cpu__process-name' =~ /^$/)
I can workaround this issue using a default value for missing tags and filter on this default tag, in the following node but I am wondering if there is a better way structure the initial where statement and avoid an extra node.
| default()
.tag('process-cpu__process-name','system')
| where(lambda: \"process-cpu__process-name\" == 'system' )
Sure it doesn't pass, 'cause this
'process-cpu__process-name'
is a string literal it TICKScript, not a reference to a field, which is
"process-cpu__process-name"
You obviously got the condition always false in this case.
Quite common mistake though, especially for someone with previous experience with the languages that tolerates both single & double quote for mere string. :-)
Also, there's a function in TICKScript lambda called strLength(), find the doc here, please.
Normally a whereIn in Eloquent compares a value from a field to an array with options. I like to reverse that and compare a option to multiple options in the field:
field contains 'option1,option2,option3'
Model::whereIn('field', 'option1')->get();
Is this possible?
You can make your query using LIKE:
Model::where('field', 'LIKE', '%option1%')->get();
Documentation on the syntax is available here: http://dev.mysql.com/doc/refman/5.7/en/pattern-matching.html
If you always add a comma , even after the last choice, like option1,option2,option3, you can use a bit of a more robust filter:
Model::where('field', 'LIKE', '%option1,%')->get();
And a comma at the start (or any other separator if that matters) would make it even better:
Model::where('field', 'LIKE', '%,option1,%')->get();
Otherwise you can have issues if one of your option is similar to another one at the end (if you have fish and goldfish as possible categories, using LIKE ',fish,' will guarantee that you don't match goldfish, while LIKE 'fish,' would match both fish and goldfish).
I'd recommend to store your categories like that: /fish/goldfish/water/ and then filter using LIKE '%/yourcategory/%'
First off, here's the SO question+answer where I got my information - laravel 4 saving ip address to model.
So my table will potentially have millions of row, therefore to keep storage low I opted for option 2 - using the Schema builder's binary() column and converting/storing IPs as binary with the help of Eloquents' accessors/mutators.
Here's my table:
Schema::create('logs', function ( Blueprint $table ) {
$table->increments('id');
$table->binary('ip_address'); // postgresql reports this column as BYTEA
$table->text('route');
$table->text('user_agent');
$table->timestamp('created_at');
});
The first problem I ran into was saving the IP address. I set an accessor/mutator on my model to convert the IP string into binary using inet_pton() and inet_ntop(). Example:
public function getIpAddressAttribute( $ip )
{
return inet_ntop( $ip );
}
public function setIpAddressAttribute( $ip )
{
$this->attributes['ip_address'] = inet_pton( $ip );
}
Trying to save an IP address resulted in the whole request failing - nginx would just return a 502 bad gateway error.
OK. So I figured it had to be something with Eloquent/PostgreSQL not playing well together while passing the binary data.
I did some searching and found the pg_escape_bytea() and pg_unescape_bytea() functions. I updated my model as follows:
public function getIpAddressAttribute( $ip )
{
return inet_ntop(pg_unescape_bytea( $ip ));
}
public function setIpAddressAttribute( $ip )
{
$this->attributes['ip_address'] = pg_escape_bytea(inet_pton( $ip ));
}
Now, I'm able to save an IP address without a hitch (at least, it doesn't throw any errors).
The new problem I'm experiencing is when I try to retrieve and display the IP. pg_unescape_bytea() fails with pg_unescape_bytea() expects parameter 1 to be string, resource given.
Odd. So I dd() $ip in the accessor, the result is resource(4, stream). Is that expected? Or is Eloquent having trouble working with the column type?
I did some more searching and found it's possible that pg_unescape_bytea() is not properly unescaping the data - https://bugs.php.net/bug.php?id=45964.
After much headbanging and hairpulling, it became apparent that I might be approaching this problem from the wrong direction, and need some fresh perspective.
So, what am I doing wrong? Should I be using Postgres' BIT VARYING instead of BYTEA by altering the column type --
DB::statement("ALTER TABLE logs ALTER COLUMN ip_address TYPE BIT VARYING(16) USING CAST(ip_address AS BIT VARYING(16))");`
-- Or am I merely misusing pg_escape_bytea / pg_unescape_bytea?
All help is appreciated!
Like already said in the comments to your question: in your specific case you should use the corresponding PostgreSQL data type and handling will be much easier. Compared to MySQL you will have a lot of other types in PostgreSQL (like JSON), there is a PostgreSQL data type overview page for further reference.
That said, other people could stumble upon a similar problem with bytea fields. The reason why you got Resource instead of string was that PostgreSQL treats bytea fields as streams. A very naïve approach would be to first get the stream and then to return the data:
public function getDataAttribute($value)
{
// This will kill your server under high load with large values.
$data = fgets($value);
return pg_unescape_bytea($data);
}
You can imagine that this could be a problem where multiple people try to get big files (currently hundreds of MiB or a couple of GiB) where large data objects would need a lot of memory on the server (this could even get a problem on mobile devices without swap). In this case you should work with streams on the server and the client and just fetch the data on the client you really need.