I am facing a weird problem with router transformation in Informatica. I am using it in my mapping where I check for a particular port's value and based on the condition, I route it to appropriate flow. While I debug, I see the value of the variable as expected, but the row is identified as "filtered" in the debugger. I have tried various other methods like trimming the variable (LTRIM/RTRIM) to ensure there is no trailing spaces which makes the router condition fail, but that doesn't work either. As a result, my rows which are supposed to be inserted into the target as bypassed. Have anyone faced similar issue? I am wondering if I am missing something here.
When you run normally without the debugger are you experiencing different results?
As you know, "Filtered" means the condition is evaluating to false, so the only question should be around your condition.
What is the data type of the port and what is your exact conditional expression?
Mismatching data-types can cause unexpected boolean evaluations (ex. comparing an integer to a string without casting one side using TO_CHAR or TO_INTEGER respectively).
Related
In the apache access logs I found the following code as query string (GET), submitted multiple times each second for quite a while from one IP:
**/OR/**/ASCII(SUBSTRING((SELECT/**/COALESCE(CAST(LENGTH(rn)/**/AS/**/VARCHAR(10000))::text,(CHR(32)))/**/FROM/**/"public".belegtable/**/ORDER/**/BY/**/lv/**/OFFSET/**/1492/**/LIMIT/**/1)::text/**/FROM/**/1/**/FOR/**/1))>9
What does it mean?
Is this an attempt of breaking in via injection?
I have never seen such a statement and I don't understand its meaning. PostgreSQL is used on the server.
rn and belegtable exist. Some other attempts contain other existing fields/tables. Since the application is very costum, I don't know how the information on existing SQL fields can be known to strangers. Very weird.
**/
OR ASCII(
SUBSTRING(
( SELECT COALESCE(
CAST(LENGTH(rn) AS VARCHAR(10000))::text,
(CHR(32))
)
FROM "public".belegtable
ORDER BY lv
OFFSET 1492
LIMIT 1
)::text
FROM 1
FOR 1
)
) > 9
Is this an attempt of breaking in via injection?
The query in question does not have too many characteristics of an attempted SQL injection.
An SQL injection typically involves inserting an unwanted action into some section of a bigger query, under the disguise of a single value. Typically the injected part tries to guess what comes before it, neutralise it, do something malicious and secure the entire query from syntax errors by also neutralising what comes after the injected piece, which might not be visible to the attacker.
I don't see anything that could work as an escape sequence at the beginning or anything that would neutralise the remains of the query coming in after the injection. What this query does also isn't malicious. An SQL injection would attempt to extract some additional information about the database security, structure and configuration, or - if the attacker already gathered enough data - it would try to steal the data, encrypt it or tamper with it otherwise, depending on the aim and strategy of the attacker as well as the type of data found in the database. There also wouldn't be much point looping it like that.
As to the looping part: if someone attempted to put load on the database - as in DDoS - you'd likely see more than one node doing that and probably in a more elaborate and well disguised manner, using different and more demanding queries sent at different frequencies.
What does it mean?
It's likely someone's buggy code stuck in an unterminated loop, judging by the LIMIT and OFFSET mechanism I've seen used for looping through some set of records by taking one at a time (LIMIT 1) and incrementing which one to get next (OFFSET n). The whole expression always returns true because ASCII() returns the character code of the first character in the string. That string defaults to a space ' ', ASCII code 32, or some text representation of a number between 0 and 99999. Sice all ASCII digits are between code 48 and 57, it's effectively always comparing some bigger number than 9 to a 9, checking if it indeed is bigger.
The author of that code might not have predicted the loop to be able to run infinitely and might have misinterpreted what some of the functions used in that query do. Regardless of what really happened, I think it was a good idea to cut off that IP avoiding needless stress on the database. Double-checking your security setup is always a good idea but I wouldn't call this an attempted attack. At least not this query alone, as it might be a harmless piece of a bigger, more malicious operation - but that could be said about any query.
I have a PostgreSQL-powered web app that does some non-essential, simple calculations involving getting values from outside sources, multiplication and division for reporting purposes. Today an error where a multiplication that exceeded the value domain of a numeric( 10, 4 ) field led to an application crash. It would be much better if the relevant field had just been set to null and a notice be generated. The way the bug worked was that a wrong value in one field caused several views to become unavailable, and while a missing value in that place would have been sad but no big problem, the blocked view is still essential for the app to work.
Now I'm aware that in this particular case, setting that field to numeric( 11, 4 ) would have prevented the bailout, but that is, of course, only postponing the issue at hand. Since the error happened in a function call, I could also have written an exception handler; lastly, one could check either the multiplicands or the result for sane values (but that is in itself a little strange as I would either have to do a guess based on magnitudes or else do the multiplication in another numeric type that can probably handle a value whose magnitude is in principle not known to me with certainty, because external sources).
Exception handling is probably what this will boil down to, which, however, entails that all numeric calculations will have to be done via PL/pgSQL function calls, and will have to be implemented in many different places. None of the options seems particularly maintainable or elegant. So the question is: Can I somehow configure PostgreSQL to ignore some or all arithmetic errors and use default values in such cases? If so, can that be done per database or will I have to configure the server? If this is impossible or a Bad Idea, what are the best practices to avoid arithmetic errors?
Clarification This is not a question about how to rewrite numeric( 10, 4 ) so that the field can hold values of 1e6 and above, and also not so much about error handling in the application that uses the DB. It's more about whether there is an operator, a function call, a general configuration or a general pattern that is most commonly recommended to deal with situations where a (non-essential) computation normally results in a number (or in fact other value type) except with some inputs that cause exceptions, which is when the result could fully well and safely be discarded. Think Excel printing out #### when cell is too narrow for the digits to be displayed, or JavaScript giving you NaN in place of arithmetic errors. Returning null instead of raising an exception may be a bad idea in general programming but legitimate in specific case.
Observe that PostGreSQL error codes does have e.g. invalid_argument_for_logarithm, invalid_argument_for_ntile_function, division_by_zero all grouped together under Class 22 — Data Exception and does allow exception handling in function bodies, so I can also specifically ask: How to catch all class 22 exceptions short of listing all the error codes?, but then I still hope for a more principled approach.
Arguably the type numeric (without type modifiers) would be the right thing for you if you want to avoid overflows (that's what you seem to mean with “arithmetic error”) as much as possible.
However, there will still be the possibility of value overflows numeric format.
There is no way to configure PostgreSQL so that it ignores a numeric overflow.
If the result of an operation cannot be represented in a data type, there should be an error. If the data supplied by the application can lead to an error, the application should be ready to handle such an error rather than “crash”. Failure to do so is an application bug.
When I define a CHECK CONSTRAINT on a table, I find the condition clause stored can be different than what I entered.
Example:
Alter table T1 add constraint C1 CHECK (field1 in (1,2,3))
Looking at what is stored:
select cc.Definition from sys.check_constraints cc
inner join sys.objects o on o.object_id = cc.parent_object_id
where cc.type = 'C' and cc.name = 'T1';
I see:
([field1]=(3) OR [field1]=(2) OR [field1]=(1))
Whilst these are equivalent, they are not the same text.
(A similar behaviour occurs when using a BETWEEN clause).
My reason for wishing this did not happen is that I am trying to programatically ensure that all my CHECK constraints are correct by comparing the text I would use to define
the constraint with that stored in sys.check_constraints - and if different then drop and recreate the constraint.
However, in these cases, they are always different and so the program would always think it needs to recreate the constraint.
Question is:
Is there any known reason why SQL Server does this translation? Is it just removing a bit of syntactic sugar and storing the clause in a simpler form?
Is there a way to avoid the behaviour (other than to write my constraint clauses in the long form to match what SQL Server would change it to)?
Is there another way to tell if my check constraint is 'out of date' and needs recreating?
Is there any known reason why SQL Server does this translation? Is it just removing a bit of syntactic sugar and storing the clause in a simpler form?
I'm not aware of any reasons documented in the Books Online, or elsewhere. However, my guess is that it's normalized for some purposes that are internal to SQL Server. It might allow SQL Server to be a bit lenient in defining the expression (such as using Database for a column name), but guaranteeing that the column names are always appropriately escaped for whatever engine needs to parse the expression (ie, [Database]).
Is there a way to avoid the behaviour (other than to write my constraint clauses in the long form to match what SQL Server would change it to)?
Probably not. But if your constraints aren't terribly complicated, is re-writing the constraint clauses in the long form such a bad idea?
Is there another way to tell if my check constraint is 'out of date' and needs recreating?
Before I answer this directly, I'd point out that there's a bit of programming philosophy involved here. The API that SQL Server provides for the text of a CHECK constraint only guarantees that you'll get something equivalent to the original expression. While you could certainly build some fancy methods to try to ensure that you'll always be able to reproduce SQL Server's normalized version of the expression, there's no guarantee that Microsoft won't change its normalization rules in the future. And indeed, there's probably no guarantee that two equivalent expressions will always be normalized identically!
So, I'd first advise you to re-examine your architecture, and see if you can accomplish the same result without having to rely on undocumented API behavior.
Having said that, there are a number of methods outlined in this question (and answer).
Another alternative, which is a bit more brute-force but perhaps acceptable, would be to always assume that the expression is "out of date" and simply drop/re-create the constraint every time you check. Unless you're expecting these constraints to frequently become out-of-date (or the tables are quite large), it seems this would be a decent solution. You could probably even run it in a transaction, so that if the new constraint is already violated, simply roll-back the transaction and report the error.
I'm using the texticle (https://github.com/tenderlove/texticle) library to do full-text postgresql searches.
The library generates sql like the following:
to_tsvector('spanish', "games"."system"::text) ## plainto_tsquery('spanish', 'Genesis'::text)
If someone does a search for '&', then I get a warning in my logs:
text-search query contains only stop words or doesn't contain lexemes, ignored
How can I avoid this? Should I have the application know about the various stopwords and not send postgresql the query if the search term is comprised only of stopwords? Or can I tell postgresql to somehow ignore this warning?
the warning is perfectly descriptive. Basically you have to handle this somewhere in programming logic. If your searches are in a stored procedure you can test the value and see if it contains lexemes.
Otherwise you can set the logging level to ERROR so warnings do not get set, or you can handle it on your application level.
None of these are perfect. If I search for "the" I should get a similar warning so you probably want to be able to check after it is converted to a tsquery (i.e. in the database itself.
Otherwise, my advice is ignore the warning.
This function could be used to resolve if query contains only stop words:
numnode(plainto_tsquery('spanish','&')) = 0 )
personally i always sanitize user input. It should be considered evil for the most part. so in this case i would definitely do some filtering at the application level.
Please explain the reasoning behind making EventStoreRepository.GetById create a new stream when an aggregate doesn't exist. This behavior appears to give GetById a dual responsibility which in my case could result in undesirable results. For example, my aggregates always implement a factory method for their creation so that their first event results in the setting of the aggregate's Id. If a command was handled prior to the existance of the aggregate it may succeed even though the aggregate doesn't have it's Id set or other state initialized (crash-n-burn with null reference exception is equally as likely).
I would much rather have GetById throw an exception if an aggregate doesn't exist to prevent premature creation and command handling.
That was a dumb design decision on my part. I went back and forth on this one. To be honest, I'm still back and forth on it.
The issue is that exceptions should really be used to indicate exceptional or unexpected scenarios. The problem is that a stream not being found can be a common operation and even an expected operation in some regards. I tinkered with the idea of throwing an exception, returning null, or returning an empty stream.
The way to determine if the aggregate is empty is to check the Revision property to see if it's zero.