I have a query param of type array to collect IDs to query from a postgres table. I think I have built everything out appropriately, but the query fails with ERROR: syntax error at or near "$1"
The logs are:
SELECT
professional_leads.first_name
, professional_leads.last_name
, professional_leads.email
, professional_leads.phone_number
, professional_leads.professional_id as proId
, professional_leads.id as proLeadId
, professional_leads.user_id
, professional_leads.interview_offered_at
, professional_leads.sms_enabled
, professional_leads.email_enabled
, professional_leads.resume_pdf_object_key
, professional_leads.created_at
, professional_leads.updated_at
, professional_leads.reschedule_count
, professional_leads.experience_level
, professional_leads.waitlisted_reason
, professional_leads.resume_state
, professional_leads.interview_state
, professional_leads.state
, professional_leads.profession_id
, professional_leads.indicated_specialty_codes
, professional_leads.other_specialties
, professional_leads.professional_id
, professional_leads.license_received_on
, professional_leads.license_expires_on
, professional_leads.region_id
, professional_leads.marketing_channel
, professional_leads.newsletter
, professional_leads.referral_code
, professional_leads.asset_proof_type
, professional_leads.verification_state
, professional_leads.duplicate
FROM
professional_leads
WHERE
id IN :clause
DEBUG 2022-10-01 21:25:45,346 [[MuleRuntime].uber.15: [api-database-sapi].Copy_of_get-Flow.BLOCKING #7f0aedd] [processor: Copy_of_get-Flow/processors/0/processors/0; event: 232aba80-41f1-11ed-b583-f02f4b10a50d] org.mule.db.commons.shaded.internal.domain.executor.AbstractExecutor: Executing query:
SELECT
professional_leads.first_name
, professional_leads.last_name
, professional_leads.email
, professional_leads.phone_number
, professional_leads.professional_id as proId
, professional_leads.id as proLeadId
, professional_leads.user_id
, professional_leads.interview_offered_at
, professional_leads.sms_enabled
, professional_leads.email_enabled
, professional_leads.resume_pdf_object_key
, professional_leads.created_at
, professional_leads.updated_at
, professional_leads.reschedule_count
, professional_leads.experience_level
, professional_leads.waitlisted_reason
, professional_leads.resume_state
, professional_leads.interview_state
, professional_leads.state
, professional_leads.profession_id
, professional_leads.indicated_specialty_codes
, professional_leads.other_specialties
, professional_leads.professional_id
, professional_leads.license_received_on
, professional_leads.license_expires_on
, professional_leads.region_id
, professional_leads.marketing_channel
, professional_leads.newsletter
, professional_leads.referral_code
, professional_leads.asset_proof_type
, professional_leads.verification_state
, professional_leads.duplicate
FROM
professional_leads
WHERE
id IN ?
Parameters:
clause = ('6a379873-93f9-4b16-8752-168aa92c8846','a234570e-a739-4bcc-847a-a875f5202398')
I flatten the array in a var ids:
"(" ++ (attributes.queryParams.*id map "'$'" joinBy ",") ++ ")"
I have my query as follows:
%dw 2.0
output text
---
"SELECT
professional_leads.first_name
, professional_leads.last_name
, professional_leads.email
, professional_leads.phone_number
, professional_leads.professional_id as proId
, professional_leads.id as proLeadId
, professional_leads.user_id
, professional_leads.interview_offered_at
, professional_leads.sms_enabled
, professional_leads.email_enabled
, professional_leads.resume_pdf_object_key
, professional_leads.created_at
, professional_leads.updated_at
, professional_leads.reschedule_count
, professional_leads.experience_level
, professional_leads.waitlisted_reason
, professional_leads.resume_state
, professional_leads.interview_state
, professional_leads.state
, professional_leads.profession_id
, professional_leads.indicated_specialty_codes
, professional_leads.other_specialties
, professional_leads.professional_id
, professional_leads.license_received_on
, professional_leads.license_expires_on
, professional_leads.region_id
, professional_leads.marketing_channel
, professional_leads.newsletter
, professional_leads.referral_code
, professional_leads.asset_proof_type
, professional_leads.verification_state
, professional_leads.duplicate
FROM
professional_leads
WHERE
id IN :clause"
My input parameters in the call is:
{
"clause": vars.ids
}
Grabbing the query and using the bind variable verbatim, the query executes fine.
Is there a limitation with IN and bind variables?
So far, I am able to retrieve data from MongoDB using mongo-hadoop-core 1.4.2. The data I want to manipulate are values inside arrays inside an embed document inside each document in the collection I am querying, and I need these values as Double's. The data retrieved from collections has type RDD[(Object, org.bson.BSONObject)], which means each document is a tuple of types (Object, org.bson.BSONObject).
Whenever I want get an embed document, I do (working on spark-shell 1.5.1):
import com.mongodb.{BasicDBObject, BasicDBList} // classes I am using here.
// 'documents' already taken from collection.
scala> documents
res4: org.apache.spark.rdd.RDD[(Object, org.bson.BSONObject)] = NewHadoopRDD[0] at newAPIHadoopRDD at <console>:32
// getting one document.
scala> val doc = documents.take(1)(0)
doc: (Object, org.bson.BSONObject) = ( ... _id fields ... , ... lots of fields ...)
// getting an embed document from tuple's second element.
scala> val samples = doc._2.get("samp") match {case x: BasicDBObject => x}
samples: com.mongodb.BasicDBObject = (... some fields ...)
// getting an embed document.
scala> val latency = samples.get("latency") match {case x: BasicDBObject => x}
latency: com.mongodb.BasicDBObject = { "raw" : [ 9.71 , 8.77 , 10.16 , 9.49 , 8.54 , 10.29 , 9.55 , 9.16 , 10.78 , 10.31 , 9.54 , 10.69 , 10.33 , 9.58 , 9.07 , 9.72 , 9.48 , 8.72 , 10.59 , 9.81 , 9.31 , 10.64 , 9.87 , 9.29 , 10.38 , 9.64 , 8.86 , 10.84 , 10.06 , 9.29 , 8.45 , 9.08 , 7.55 , 9.75 , 9.05 , 10.38 , 9.64 , 8.25 , 10.27 , 9.54 , 8.52 , 10.26 , 9.53 , 7.87 , 9.76 , 9.02 , 10.27 , 7.93 , 9.73 , 9 , 10.07 , 9.35 , 7.66 , 13.68 , 11.92 , 14.72 , 14 , 12.55 , 11.77 , 11.02 , 11.59 , 10.87 , 10.4 , 9.13 , 10.28 , 9.55 , 10.43 , 8.33 , 9.66 , 8.93 , 8.05 , 11.26 , 10.53 , 9.81 , 10.2 , 9.42 , 7.73 , 9.76 , 9.04 , 8.29 , 9.34 , 7.21 , 10.05 , 9.32 , 10.28 , 8.59 , 10.15 , 9.53 , 7.88 , 9.9 , 9.15 , 13.96 , 13.19 , 11 , 13.6 , 13.01 , 12.17 , 11.39 , 10.64 , 9.9] , "xtrf" : { "...
// getting a bson array.
scala> val array = latency.get("raw") match {case x: BasicDBList => x}
array: com.mongodb.BasicDBList = [ 9.71 , 8.77 , 10.16 , 9.49 , 8.54 , 10.29 , 9.55 , 9.16 , 10.78 , 10.31 , 9.54 , 10.69 , 10.33 , 9.58 , 9.07 , 9.72 , 9.48 , 8.72 , 10.59 , 9.81 , 9.31 , 10.64 , 9.87 , 9.29 , 10.38 , 9.64 , 8.86 , 10.84 , 10.06 , 9.29 , 8.45 , 9.08 , 7.55 , 9.75 , 9.05 , 10.38 , 9.64 , 8.25 , 10.27 , 9.54 , 8.52 , 10.26 , 9.53 , 7.87 , 9.76 , 9.02 , 10.27 , 7.93 , 9.73 , 9 , 10.07 , 9.35 , 7.66 , 13.68 , 11.92 , 14.72 , 14 , 12.55 , 11.77 , 11.02 , 11.59 , 10.87 , 10.4 , 9.13 , 10.28 , 9.55 , 10.43 , 8.33 , 9.66 , 8.93 , 8.05 , 11.26 , 10.53 , 9.81 , 10.2 , 9.42 , 7.73 , 9.76 , 9.04 , 8.29 , 9.34 , 7.21 , 10.05 , 9.32 , 10.28 , 8.59 , 10.15 , 9.53 , 7.88 , 9.9 , 9.15 , 13.96 , 13.19 , 11 , 13.6 , 13.01 , 12.17 , 11.39 , 10.64 , 9.9]
Converting type Object to BasicDBObject is quite inconvenient but I need to do it in order to use get(key: String). I could also use .asInstanceOf[BasicDBObject] instead of match {case x: BasicDBObject => x} but is there any better way??.
Getting specific types, like Double, Int, String and Date, is straight forward using inhereted methods from BasicBsonObject class.
As for BasicDBList, there's a get(key: String) method, inherited from BasicBsonList, that returns an Object that can be cast to Double but only using a .asInstanceOf[Double] call and there's a toArray() inherited from java.util.ArrayList that returns an array of Object's that I can't cast to Double, even with .map(_.asInstanceOf[Double]) as I'm doing here:
scala> val arrayOfDoubles = array.toArray.map(_.asInstanceOf[Double])
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Double
at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:119)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:37)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:37)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:37)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:44)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:46)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:48)
at $iwC$$iwC$$iwC.<init>(<console>:50)
at $iwC$$iwC.<init>(<console>:52)
at $iwC.<init>(<console>:54)
at <init>(<console>:56)
at .<init>(<console>:60)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
but sometimes it works. In some documents this cast works, in other documents it doesn't and prints that error message above. Could this be a problem in the data structure given by MongoDB, to spark, but just in these documents? Smaller arrays, with 30 values, seems to always work.
my solution so far is this inefficient conversion:
scala> val arrayOfDoubles = array.toArray.map(_.toString.toDouble)
arrayOfDoubles: Array[Double] = Array(9.71, 8.77, 10.16, 9.49, 8.54, 10.29, 9.55, 9.16, 10.78, 10.31, 9.54, 10.69, 10.33, 9.58, 9.07, 9.72, 9.48, 8.72, 10.59, 9.81, 9.31, 10.64, 9.87, 9.29, 10.38, 9.64, 8.86, 10.84, 10.06, 9.29, 8.45, 9.08, 7.55, 9.75, 9.05, 10.38, 9.64, 8.25, 10.27, 9.54, 8.52, 10.26, 9.53, 7.87, 9.76, 9.02, 10.27, 7.93, 9.73, 9.0, 10.07, 9.35, 7.66, 13.68, 11.92, 14.72, 14.0, 12.55, 11.77, 11.02, 11.59, 10.87, 10.4, 9.13, 10.28, 9.55, 10.43, 8.33, 9.66, 8.93, 8.05, 11.26, 10.53, 9.81, 10.2, 9.42, 7.73, 9.76, 9.04, 8.29, 9.34, 7.21, 10.05, 9.32, 10.28, 8.59, 10.15, 9.53, 7.88, 9.9, 9.15, 13.96, 13.19, 11.0, 13.6, 13.01, 12.17, 11.39, 10.64, 9.9)
Am I missing something here or things are really inconvenient? Why do all these methods have to return Object or BSONObject? is there any way to overcome this problem I found? Where have this java.lang.Integer come from if there aren't integers in the array being cast to double?
First of all, I'd advice you to have a look at casbah if you haven't yet.
To answer your question: if you import Java conversions:
import scala.collection.JavaConversions._
You should be able to map directly over the collection without the toArray call. If your array contains either Doubles or Integers you could cast it to Number and get the double value. Like so:
array.map(_.asInstanceOf[Number].doubleValue)
I don't know what your data source looks like, but given the fact you occasionally get an Integer where you expect a Double it's likely to assume that you store round decimal numbers (e.g. 11.0) as integer (e.g. 11).
In one of my application having a requirement to download a PDF file with report details in form of table.
For creating a PDF file and writing a table in it, using cpan module available in perl. PDF::Report and PDF::Report::Table.
Please find below the code sample:
#!/usr/bin/perl
use strict;
use warnings;
use PDF::Report;
use PDF::Report::Table;
my $pdf = PDF::Report->new( PageSize => 'A4', PageOrientation => 'Portrait' );
my $table = PDF::Report::Table->new( $pdf );
my $data = [
['A1' , 'B1' , 'C1'],
['A2' , 'B2' , 'C2'],
['A3' , 'B3' , 'C3'],
['A4' , 'B4' , 'C4'],
['A5' , 'B5' , 'C5'],
['A6' , 'B6' , 'C6'],
['A7' , 'B7' , 'C7'],
['A8' , 'B8' , 'C8'],
['A9' , 'B9' , 'C9'],
['A10' , 'B10' , 'C10'],
['A11' , 'B11' , 'C11'],
['A12' , 'B12' , 'C12'],
['A13' , 'B13' , 'C13'],
['A14' , 'B14' , 'C14'],
['A15' , 'B15' , 'C15'],
['A16' , 'B16' , 'C16'],
['A17' , 'B17' , 'C17'],
['A18' , 'B18' , 'C18'],
['A19' , 'B19' , 'C19'],
['A20' , 'B20' , 'C20'],
['A21' , 'B21' , 'C21'],
['A22' , 'B22' , 'C22'],
['A23' , 'B23' , 'C23'],
['A24' , 'B24' , 'C24'],
['A25' , 'B25' , 'C25'],
['A26' , 'B26' , 'C26'],
['A27' , 'B27' , 'C27'],
['A28' , 'B28' , 'C28'],
['A29' , 'B29' , 'C29'],
['A30' , 'B30' , 'C30'],
['A31' , 'B31' , 'C31'],
['A32' , 'B32' , 'C32'],
['A33' , 'B33' , 'C33'],
['A34' , 'B34' , 'C34'],
['A35' , 'B35' , 'C35'],
['A36' , 'B36' , 'C36'],
['A37' , 'B37' , 'C37'],
['A38' , 'B38' , 'C38'],
['A39' , 'B39' , 'C39'],
['A40' , 'B40' , 'C40'],
['A41' , 'B41' , 'C41'],
];
$pdf->openpage;
$pdf->setAddTextPos( 50, 50 );
$table->addTable( $data, 400 ); # 400 is table width
$pdf->saveAs( 'table.pdf' );
Result: pdf generated with 2 pages.
at continuity of page missing a row data.
Note: i'm facing issue to attach a span shot of result.
Issues is: missing a row data. missing a row with data [A37, B37, C37].
Please help me in fixing this issues.
Thanks in advance for all your help.
well, when I run your code I get
commandPrompt > ./makepdf.pl
Useless use of greediness modifier '?' in regex; marked by <-- HERE in m/(\S{20}? <-- HERE )(?=\S)/ at /usr/local/share/perl/5.20.2/PDF/Table.pm line 386.
!!! Warning: !!! Incorrect Table Geometry! Setting bottom margin to end of sheet!
at /usr/local/share/perl/5.20.2/PDF/Report/Table.pm line 94.
!!! Warning: !!! Incorrect Table Geometry! Setting bottom margin to end of sheet!
at /usr/local/share/perl/5.20.2/PDF/Report/Table.pm line 94.
I would think that
Setting bottom margin to end of sheet!
and
!!! Warning: !!! Incorrect Table Geometry!
would have something to do with it.
Out of interest why does this work in Scala:
val exceptions = List[Char]('+')
assertTrue(exceptions.contains('+'))
but this not
val exceptions = new Array[Char]('+')
assertTrue(exceptions.contains('+'))
Because you wrote new ArrayChar. Doing that, the argument is the size of the array, and the '+' is, rather unfortunately, converted to an int to give the size. And the returned array is full of Char(0).
You should just do Array[Char]('+'), '+' would then be single element in the Array.
Try in the REPL, that makes the answer obvious:
scala> val exceptions = new Array[Char]('+')
exceptions: Array[Char] = Array( , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , )
+ has char-to-int promotion.
scala> val exceptions = Array[Char]('+')
exceptions: Array[Char] = Array(+)
scala> exceptions.contains('+')
res3: Boolean = true
is the equivalent to the List case.