We are using Scala Play, and I am trying to ensure that all SQL queries are using Anorm's String Interpolation. It works with some queries, but many are not actually replacing the variables before the query is executing.
import anorm.SQL
import anorm.SqlStringInterpolation
object SecureFile
{
val table = "secure_file"
val pk = "secure_file_idx"
...
// This method works exactly as I would hope
def insert(secureFile: SecureFile): Option[Long] = {
DBExec { implicit connection =>
SQL"""
INSERT INTO secure_file (
subscriber_idx,
mime_type,
file_size_bytes,
portal_msg_idx
) VALUES (
${secureFile.subscriberIdx},
${secureFile.mimeType},
${secureFile.fileSizeBytes},
${secureFile.portalMsgIdx}
)
""" executeInsert()
}
}
def delete(secureFileIdx: Long): Int = {
DBExec { implicit connection =>
// Prints correct values
println(s"table: ${table} pk: ${pk} secureFileIdx: ${secureFileIdx} ")
// Does not work
SQL"""
DELETE FROM $table WHERE ${pk} = ${secureFileIdx}
""".executeUpdate()
// Works, but unsafe
val query = s"DELETE FROM ${table} WHERE ${pk} = ${secureFileIdx}"
SQL(query).executeUpdate()
}
}
....
}
Over in the PostgreSQL logs, it's clear that the delete statement has not acquired the correct values:
2015-01-09 17:23:03 MST ERROR: syntax error at or near "$1" at character 23
2015-01-09 17:23:03 MST STATEMENT: DELETE FROM $1 WHERE $2 = $3
2015-01-09 17:23:03 MST LOG: execute S_1: ROLLBACK
I've tried many variations of execute, executeUpdate, and executeQuery with similar results. For the moment, we are using basic string replacement but of course this is bad because it's not using PreparedStatements.
For anyone else sitting on this page scratching their head and wondering what they might be missing...
SQL("select * from mytable where id = $id")
is NOT the same as
SQL"select * from mytable where id = $id"
The former does not do String interpolation whereas the latter does.
This is easily overlooked in the aforementioned docs as all the samples provided just happen to have a (non-related) closing parenthesis on them (like this sentence does)
Anorm String interpolation was introduced to pass parameter (e.g. SQL"Select * From Test Where id = $x), with interpolation arguments (e.g. $x) set on underlying PreparedStament according proper type conversion (see use cases on https://www.playframework.com/documentation/2.3.x/ScalaAnorm ).
Next Anorm release will also have the #$foo syntax to mix interpolation for parameter with standard string interpolation. This will allow to write DELETE FROM #$table WHERE #${pk} = ${secureFileIdx} and having it executed as DELETE FROM foo WHERE bar = ? (if literal table is "foo" and pk is "bar"), with literal secureFileIdx passed as parameter. See related pull request.
Until next revision is release, you can build Anorm from its master sources ti benefit from this change.
Related
Apologies for a length post. I have been trying to beat my head around reading about mock, MagicMock, and all the time getting confused. Hence, decided to write this post.
I know several questions, and pages have been written on this. But, still not able to wrap my head around this.
My Setup:
All the test code, and the 2 module files come under one "folder" mymodule
my_module_1.py file contains
class MyOuterClass(object):
MyInnerClass(object):
attribute1: str
attribute2: str
attribute3: str
def get(self) -> MyInnerClass:
'''
pseudocode
1. a call to AWS's service is made
2. the output from call in step 1 is used to set attributes of this InnerClass
3. return innerclass instance
'''
I use the OuterClass in another file(my_module_2.py), to set some values and return a string as follows:
class MyModule2():
def get_foo(self, some_boolean_predicate):
if some_boolean_predicate:
temp = my_module_1.OuterClass().get()
statement = f'''
WITH (
BAR (
FIELD_1 = '{temp.attribute1}',
FIELD_2 = '{temp.attribute2}',
FIELD_3 = '{temp.attribute3}'
)
)
'''
else:
statement = ''
return statement
I want to write the unit tests for the file my_module_2.py file, and test the function get_foo
How I am writing the tests(or planning on)
a test file by name test_my_module2.py
I started with creating a pytest.fixture for the MyOuterClass's get function as follows since I will be reusing this piece of info again in my other tests
#pytest.fixture
def mock_get(mocker: MockerFixture) -> MagicMock:
return mocker.patch.object(MyOuterClass, 'get')
Finally,
Then I proceeded to use this fixture in my test as follows:
from unittest import mock
from unittest.mock import MagicMock, Mock, patch, PropertyMock
import pytest
from pytest_mock import MockerFixture
from my_module.my_module_1 import myOuterClass
def test_should_get_from_inner_class(self, mock_get):
# mock call to get are made
output = mock_get.get
#update the values for the InnerClass's attributes here
output.attribute1.side_effect = 'attr1'
output.attribute2.side_effect = 'attr2'
output.attribute3.side_effect = 'attr3'
mock_output_str = '''
WITH (
BAR (
FIELD_1 = 'attr1',
FIELD_2 = 'attr2',
FIELD_3 = 'attr3'
)
)
'''
module2Obj = MyModule2()
response = module2Obj.get_foo(some_boolean_predicate=True)
# the following assertion passes
assert mock_get.get.called_once()
# I would like match `response to that with mock_output_str instance above
assert response == mock_output_str
But, the assertion as you might have guessed failed, and I know I am comparing completely different types, since I see
errors such as
FAILED [100%]
WITH (
BAR (
FIELD1 = '<MagicMock name='get().attr1' id='4937943120'>',
FIELD3 = '<MagicMock name='get().attr2' id='4937962976'>',
FIELD3 = '<MagicMock name='get().attr3' id='4937982928'>'
)
)
Thank you for being patient with me till here, i know its a really lengthy post, but stuck on this for a few days, ended up creating a post here.
How do i get to validate the mock's value with the mock_output_str?
yess! the hint was in the #gold_cy's answer. I was only calling my mock and never setting its values
this is what my test case ended up looking
mock_obj = OuterClass.InnerClass()
mock_obj.attribute1='some-1'
mock_obj.attribute2='some-2'
mock_obj.attribute3='some-3'
mock_get.return_value = mock_obj
once my mock was setup properly, then the validation became easy! Thank you!
I am using h2 database to test my postgres slick functionality.
I created a below h2DbComponent:
trait H2DBComponent extends DbComponent {
val driver = slick.jdbc.H2Profile
import driver.api._
val h2Url = "jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;INIT=runscript from './test/resources/schema.sql'\\;runscript from './test/resources/schemadata.sql'"
val logger = LoggerFactory.getLogger(this.getClass)
val db: Database = {
logger.info("Creating test connection ..................................")
Database.forURL(url = h2Url, driver = "org.h2.Driver")
}
}
In the above snippet i am creating my tables using schema.sql and inserting a single row(record) with schemadata.sql.
Then i am trying to insert a record into the table as below using my test case:
class RequestRepoTest extends FunSuite with RequestRepo with H2DBComponent {
test("Add new Request") {
val response = insertRequest(Request("XYZ","tk", "DM", "RUNNING", "0.1", "l1", "file1",
Timestamp.valueOf("2016-06-22 19:10:25"), Some(Timestamp.valueOf("2016-06-22 19:10:25")), Some("scienceType")))
val actualResult=Await.result(response,10 seconds)
assert(actualResult===1)
val response2 = getAllRequest()
assert(Await.result(response2, 5 seconds).size === 2)
}
}
The above assert of insert works fine stating that the record is inserted. But the getAllRequest() assert fails as the output still contains the single row(as inserted by schemadata.sql) => which means the insertRequest change is not persisted. However the below statements states that the record is inserted as the insert returned 1 stating one record inserted.
val response = insertRequest(Request("CMP_XYZ","tesco_uk", "DM", "RUNNING", "0.1", "l1", "file1",
Timestamp.valueOf("2016-06-22 19:10:25"), Some(Timestamp.valueOf("2016-06-22 19:10:25")),
Some("scienceType")))
val actualResult=Await.result(response,10 seconds)
Below is my definition of insertRequest:
def insertRequest(request: Request):Future[Int]= {
db.run { requestTableQuery += request }
}
I am unable to figure out how can i see the inserted record. Is there any property/config which i need to add?
But the getAllRequest() assert fails as the output still contains the single row(as inserted by schemadata.sql) => which means the insertRequest change is not persisted
I would double-check that the assert(Await.result(response2, 5 seconds).size === 2) line is failing because of a size difference. Could it be failing for some other general failure?
For example, as INIT is run on each connection it could be that you are re-creating the database for each connection. Unless you're careful with the SQL, that could produce an error such as "table already exists". Adding TRACE_LEVEL_SYSTEM_OUT=2; to your H2 URL can be helpful in tracking what H2 is doing.
A couple of suggestions.
First, you could ensure your SQL only runs as needed. For example, your schema.sql could add checks to avoid trying to create the table twice:
CREATE TABLE IF NOT EXISTS my_table( my_column VARCHAR NULL );
And likewise for your schemadata.sql:
MERGE INTO my_table KEY(my_column) VALUES ('a') ;
Alternatively, you could establish schema and test data around your tests (e.g., possibly in Scala code, using Slick). Your test framework probably has a way to ensure something is run before and after a test or test suit.
My Linq Query keeps returning the null error on FirstOrDefault
The cast to value type 'System.Int32' failed because the materialized value is null
because it can't find any records to match on the ClinicalAssetID form the ClinicalReading Table, fair enough!
But I want the fields in my details page just to appear blank if the table does not have matching entry.
But how can I handle the null issue when using the order by function ?
Current Code:
var ClinicalASSPATINCVM = (from s in db.ClinicalAssets
join cp in db.ClinicalPATs on s.ClinicalAssetID equals cp.ClinicalAssetID into AP
from subASSPAT in AP.DefaultIfEmpty()
join ci in db.ClinicalINSs on s.ClinicalAssetID equals ci.ClinicalAssetID into AI
from subASSINC in AI.DefaultIfEmpty()
join co in db.ClinicalReadings on s.ClinicalAssetID equals co.ClinicalAssetID into AR
let subASSRED = AR.OrderByDescending(subASSRED => subASSRED.MeterReadingDone).FirstOrDefault()
select new ClinicalASSPATINCVM
{
ClinicalAssetID = s.ClinicalAssetID,
AssetTypeName = s.AssetTypeName,
ProductName = s.ProductName,
ModelName = s.ModelName,
SupplierName = s.SupplierName,
ManufacturerName = s.ManufacturerName,
SerialNo = s.SerialNo,
PurchaseDate = s.PurchaseDate,
PoNo = s.PoNo,
Costing = s.Costing,
TeamName = s.TeamName,
StaffName = s.StaffName,
WarrantyEndDate = subASSPAT.WarrantyEndDate,
InspectionDate = subASSPAT.InspectionDate,
InspectionOutcomeResult = subASSPAT.InspectionOutcomeResult,
InspectionDocumnets = subASSPAT.InspectionDocumnets,
LastTypeofInspection = subASSINC.LastTypeofInspection,
NextInspectionDate = subASSINC.NextInspectionDate,
NextInspectionType = subASSINC.NextInspectionType,
MeterReadingDone = subASSRED.MeterReadingDone,
MeterReadingDue = subASSRED.MeterReadingDue,
MeterReading = subASSRED.MeterReading,
MeterUnitsUsed = subASSRED.MeterUnitsUsed,
FilterReplaced = subASSRED.FilterReplaced
}).FirstOrDefault(x => x.ClinicalAssetID == id);
Tried this but doesn't work
.DefaultIfEmpty(new ClinicalASSPATINCVM())
.FirstOrDefault()
Error was:
CS1929 'IOrderedEnumerable<ClinicalReading>' does not contain a definition for 'DefaultIfEmpty' and the best extension method overload 'Queryable.DefaultIfEmpty<ClinicalASSPATINCVM>(IQueryable<ClinicalASSPATINCVM>, ClinicalASSPATINCVM)' requires a receiver of type 'IQueryable<ClinicalASSPATINCVM>'
Feel a little closer with this but still errors
let subASSRED = AR.OrderByDescending(subASSRED => (subASSRED.MeterReadingDone != null) ? subASSRED.MeterReadingDone : String.Empty).FirstOrDefault()
Error:
CS0173 Type of conditional expression cannot be determined because there is no implicit conversion between 'System.DateTime?' and 'string'
The original error means that some of the following properties of the ClinicalASSPATINCVM class - MeterReadingDone, MeterReadingDue, MeterReading, MeterUnitsUsed, or FilterReplaced is of type int.
Remember that subASSRED here
let subASSRED = AR.OrderByDescending(subASSRED => subASSRED.MeterReadingDone).FirstOrDefault()
might be null (no corresponding record).
Now look at this part of the projection:
MeterReadingDone = subASSRED.MeterReadingDone,
MeterReadingDue = subASSRED.MeterReadingDue,
MeterReading = subASSRED.MeterReading,
MeterUnitsUsed = subASSRED.MeterUnitsUsed,
FilterReplaced = subASSRED.FilterReplaced
If that was LINQ to Objects, all these would generate NRE (Null Reference Exception) at runtime. In LINQ to Entities this is converted and executed as SQL. SQL has no issues with expression like subASSRED.SomeProperty because SQL supports NULL naturally even if SomeProperty normally does not allow NULL. So the SQL query executes normally, but now EF must materialize the result into objects, and the C# object property is not nullable, hence the error in question.
To solve it, find the int property(es) and use the following pattern inside query:
SomeIntProperty = (int?)subASSRED.SomeIntProperty ?? 0 // or other meaningful default
or change receiving object property type to int? and leave the original query as is.
Do the same for any non nullable type property, e.g. DateTime, double, decimal, Guid etc.
You're problem is because your DefaultIfEmpty is executed AsQueryable. Perform it AsEnumerable and it will work:
// create the default element only once!
static readonly ClinicalAssPatInVcm defaultElement = new ClinicalAssPatInVcm ();
var result = <my big linq query>
.Where(x => x.ClinicalAssetID == id)
.AsEnumerable()
.DefaultIfEmpty(defaultElement)
.FirstOrDefault();
This won't lead to a performance penalty!
Database management systems are extremely optimized for selecting data. One of the slower parts of a database query is the transport of the selected data to your local process. Hence it is wise to let the DBMS do most of the selecting, and only after you know that you only have the data that you really plan to use, move the data to your local process.
In your case, you need at utmost one element from your DBMS, and if there is nothing, you want to use a default object instead.
AsQueryable will move the selected data to your local process in a smart way, probably per "page" of selected data.
The page size is a good compromise: not too small, so you don't have to ask for the next page too often; not too large, so that you don't transfer much more items than you actually use.
Besides, because of the Where statement you expect at utmost one element anyway. So that a full "page" is fetched is no problem, the page will contain only one element.
After the page is fetched, DefaultIfEmpty checks if the page is empty, and if so, returns a sequence containing the defaultElement. If not, it returns the complete page.
After the DefaultIfEmpty you only take the first element, which is what you want.
I want to use some extra features of PostgreSQL in my code but I don't want to fill the place with SQL string interpolations.
Currently I have:
/** Use 'now()' through Slick. */
val psqlNow = SimpleFunction.nullary[java.sql.Date]("now")
//Not really my code, but we only care for 2 lines.
def aQuery(limiter: Column[Int]) = {
myTable
.filter(_.validFrom >= psqlNow)
.filter(_.validUntil <= psqlNow)
.filter(_.fakeId === limiter).map(e => (e.fakeId, e.name)
}
But I want to use 'CURRENT_DATE', which I is a literal (and using it in place of "now" throws an exception). Can someone provide an actual example, because I can't get this to compile:
/** Use 'CURRENT_DATE' through Slick. */
val psqlNow = SimpleLiteral("CURRENT_DATE")(...WHAT GOES HERE?...)
//Not really my code, but we only care for 2 lines.
def aQuery(limiter: Column[Int]) = {
myTable
.filter(_.validFrom >= psqlNow)
.filter(_.validUntil <= psqlNow)
.filter(_.fakeId === limiter).map(e => (e.fakeId, e.name)
}
And I also want to change the following to lifted Slick, can I do it with SimpleLiteral (to somehow put 'count(*) OVER() recordsFiltered' into the generated query?
SELECT *, count(*) OVER() recordsFiltered FROM example
WHERE id = $1
The examples are trivial, the actual code is a series of folds over filtering criteria.
import scala.slick.ast.TypedType
val current_date = Column.forNode[java.sql.Date](new SimpleLiteral("CURRENT_DATE")(implicitly[TypedType[java.sql.Date]]))
does the trick. Better support is missing at the moment.
I added a PR, so in Slick 2.2 it will be supported like this:
val current_date = SimpleLiteral[java.sql.Date]("CURRENT_DATE")
See https://github.com/slick/slick/pull/981
I've coded a parser based on Scala parser combinators:
class SxmlParser extends RegexParsers with ImplicitConversions with PackratParsers {
[...]
lazy val document: PackratParser[AstNodeDocument] =
((procinst | element | comment | cdata | whitespace | text)*) ^^ {
AstNodeDocument(_)
}
[...]
}
object SxmlParser {
def parse(text: String): AstNodeDocument = {
var ast = AstNodeDocument()
val parser = new SxmlParser()
val result = parser.parseAll(parser.document, new CharArrayReader(text.toArray))
result match {
case parser.Success(x, _) => ast = x
case parser.NoSuccess(err, next) => {
tool.die("failed to parse SXML input " +
"(line " + next.pos.line + ", column " + next.pos.column + "):\n" +
err + "\n" +
next.pos.longString)
}
}
ast
}
}
Usually the resulting parsing error messages are rather nice. But sometimes it becomes just
sxml: ERROR: failed to parse SXML input (line 32, column 1):
`"' expected but `' found
^
This happens if a quote characters is not closed and the parser reaches the EOT. What I would like to see here is (1) what production the parser was in when it expected the '"' (I've multiple ones) and (2) where in the input this production started parsing (which is an indicator where the opening quote is in the input). Does anybody know how I can improve the error messages and include more information about the actual internal parsing state when the error happens (perhaps something like a production rule stacktrace or whatever can be given reasonably here to better identify the error location). BTW, the above "line 32, column 1" is actually the EOT position and hence of no use here, of course.
I don't know yet how to deal with (1), but I was also looking for (2) when I found this webpage:
https://wiki.scala-lang.org/plugins/viewsource/viewpagesrc.action?pageId=917624
I'm just copying the information:
A useful enhancement is to record the input position (line number and column number) of the significant tokens. To do this, you must do three things:
Make each output type extend scala.util.parsing.input.Positional
invoke the Parsers.positioned() combinator
Use a text source that records line and column positions
and
Finally, ensure that the source tracks positions. For streams, you can simply use scala.util.parsing.input.StreamReader; for Strings, use scala.util.parsing.input.CharArrayReader.
I'm currently playing with it so I'll try to add a simple example later
In such cases you may use err, failure and ~! with production rules designed specifically to match the error.