I posted this to the scalikejdbc user group, and should also cross post to github issues.
I've seen in the docs examples of running raw queries against a table, i'm trying to get list of IndexEntry with the following code, and while the query is executed and returns results in the console, i'm not getting anything back in the map(rs =>... portion.
Relevant code is here - when i run this in intellij's debugger the result as "Vector" size = 0. Thanks for any guidance. i'm doing something wrong, hopefully its a simple oversight.
package company.shared.database.models
import company.shared.database.MySQLConnector
import scalikejdbc._
case class IndexEntry(model:String, indexName: String, fieldName: String)
object IndexSchemaModel extends App with MySQLConnector {
implicit val session:DBSession = AutoSession
def indexMap:List[IndexEntry] = {
val result = DB readOnly { implicit session =>
sql"""
SELECT
substring_index(t.name,'/',-1) as model,
i.name AS indexName,
f.name as tableName
FROM information_schema.innodb_sys_tables t
JOIN information_schema.innodb_sys_indexes i USING (table_id)
JOIN information_schema.innodb_sys_fields f USING (index_id)
where t.name like "MyDB/%"
""".map(rs => IndexEntry(
rs.string(0),
rs.string(1),
rs.string(2))).list().apply()
}
println(result) //always List()
List(IndexEntry("foo", "bar", "az")) //to match the return type
}
override def main(args: Array[String]): Unit = {
configureDB
indexMap
}
}
I have tried scalikejdc's other variants - withSql { queryDSL } and the entire bit as SQL Interpolation with full syntax support. The first and the last always execute against the mysql server, which returns 57 rows (small db), the middle throws an NPE, honestly i'm happy to tackle the middle second. I have an issue somewhere in .map and i've tried to have it just return the map, but it always results in an empty list.
Thanks and hopefully no syntax errors copying into SO.
Oh and FWIW, configureDb just sets a connection pool manually since the DB names and servers can vary wildly between sbt tests, dev, test and prod. Currently that is not my issue or i would see "ConnectionPool('default') not initialized" or similar.
Answered here: https://groups.google.com/forum/#!topic/scalikejdbc-users-group/yRjLjuCzuEo
should also cross post to github issues
Please avoid posting questions on GitHub issues. https://github.com/scalikejdbc/scalikejdbc/blob/master/CONTRIBUTING.md#issues
A bit of embarrassment aside. The user in question did not have the process privilege and therefore would not get any rows back from these tables, as soon as a grant process on . to user#host was added, all this worked fine. Permissions in information_schema are driven by what objects the user in question have access to. Meaning items like ROUTINES and in this case PROCESS etc have to be explicitly called out.
Related
I'm working on a web app in Rust. I'm using Tokio Postgres, Rocket and Tera (this may be relevant).
I'm using the following to connect to my DB which doesn't fail in either case.
(sql_cli, connection) = match tokio_postgres::connect("postgresql://postgres:*my_password*#localhost:8127/*AppName*", NoTls).await{
Ok((sql_cli, connection)) => (sql_cli, connection),
Err(e) => return Err(Redirect::to(uri!(error_display(MyError::new("Failed to make SQLClient").details)))),
};
My query is as follows. I keep my queries in a separate file (I'm self taught and find that easier).
let query= sql_cli.query(mycharactersquery::get_characters(user_id).as_str(), &[]).await.unwrap();
The get characters is as follows. It takes a user ID and should return the characters that they have made in the past.
pub fn get_characters(user_id: i16) -> String {
format!("SELECT * FROM player_characters WHERE user_id = {} ORDER BY char_id ASC;", user_id)
}
In my main file, I have one GET which is /mycharacters/<user_id> which works. This GET returns an HTML file. I have another GET which is /<user_id> which returns a Tera template. The first works fine and loads the characters, the second doesn't: it just loads indefinitely. I initially thought this was to do my lack of familiarity with Tera.
After some troubleshooting, I put some printouts in my code, the one before and after the SQL call work in /mycharacters/<user_id>, but only the one before writes to the terminal in /<user_id>. This makes me think that Tera isn't the issue as it isn't making it past the SQL call.
I've found exactly where it is going wrong, but I don't know why as it isn't giving an error.
Could someone please let me know if there is something obvious that I am missing or provide some assistance?
P.S. The database only has 3 columns, so an actual timeout isn't the cause.
I expected both of these SQL calls to function as I am connected to my database properly and the call is copied from the working call.
I am working on refactoring code for a Spark job written in Scala. We have a dataset of rolled up data, "rollups", that we pass through a list of rules. Each of the rollups has a "flags" value that is a list that we appended the rule information to to keep track of which "rule" was triggered by that rollup. Each rule is an object that will look at the data of a rollup and decide whether or not to add its identifying information to the rollups "flags".
So here is where the problem is. Currently each rule takes in the rollup data, adds a flag and then returns it again.
object Rule_1 extends AnomalyRule {
override def identifyAnomalies(rollupData: RollupData): RollupData = {
if (*condition on rollup data is triggered*) {
rollupData.flags = rollupData.addFlag("rule 1")
}
rollupData
}
}
This allows us to calculate the rules like:
val anomalyInfo = rollups.flatMap(x =>
AnomalyRules.rules.map(y =>
y.identifyAnomalies(x)
).filter(a => a.flags.nonEmpty)
)
# do later processing on anomalyInfo
The rollups val here is a Dataset of our rollup data and rules is a list of rule objects.
The issue with this method is that it will create a duplicate rollups for each rule. For example if we have 7 rules, each rollup will be duplicated 7 times because each rule returns the rollup passed into it. Running dropDuplicates() on the dataset will take care of this issue but its ugly and confusing.
This is why I wanted to refactor, but if I set up the rules to instead only append the rule like this:
object Rule_1 extends AnomalyRule {
override def identifyAnomalies(rollupData: RollupData): Unit= {
if (*condition on rollup data is triggered*) {
rollupData.flags = rollupData.addFlag("rule 1")
}
}
}
We can instead write
rollups.foreach(rollup =>
AnomalyRules.rules.foreach(rule =>
rule.identifyAnomalies(rollup)
)
)
# do later processing on rollups
This seems like the more intuitive approach. However, while running these rules in unit tests works fine, no "flag" information is added when the "rollups" dataset is passed through. I think it is because datasets are not mutable? This method actually does work if I collect the rollups as a list but the dataset I am testing on is much smaller than what is in production so we can't do that. This is where I am stuck and can not think of a cleaner way of writing this code. I feel like I am missing some fundamental programming concept or do not know how mutability works well enough.
I am trying to run an analyze query on a scala jdbc program as mentioned below.
try {
for(t<-tabList){
var statement = con.prepareStatement(s"analyze ${t}")
println(s"Analyze for: ${t}")
val rs = statement.execute()
println(s"Analyze completed for the table: ${t}")
println(rs)
}
} catch {
case pse:PSQLException => pse.printStackTrace()
case e:Exception => e.printStackTrace()
}
The table is on postgres and I am creating jar of my program and running it from command line as give below.
scala -cp /home/username/postgresql-42.1.4.jar analyzetables_2.11-0.1.jar schema.tablename
The code doesn't give any errors but I see the result from resultset object as false as below:
Analyze for: schema.tablename
Analyze completed for the table: schema.tablename
false
The query doesn't fail hence no exception during run time. Also the tables given are present in the database and they contain data. But I printed the resultset object and see that the boolean value false.
Does this mean the analyze query is failed or should I run this in any other way ?
Could let me know if there is any problem with the way I handled the "analyze" query and is there a better or right way to do it?
ANALYZE does not produce any result set.
If you get no java.sql.SQLException, you can assume that it worked.
You should use executeUpdate() rather than execute(), and it would be better to take a java.sql.Statement rather than a java.sql.PreparedStatement in the first place.
In the below code, I'm trying to do two operations. One, to create a customer in a db, and the other, to create an event in the db. The creation of the event, is dependent on the creation of the user.
I'm new to Scala, and confused on the role of Futures here. I'm trying to query a db and see if the user is there, and if not, create the user. The below code is supposed to check if the user exists with the customerByPhone() function, and if it doesn't, then go into the createUserAndEvent() function.
What it's actually doing, is skipping the response from customerByPhone and going straight into createUserAndEvent(). I thought that by using a flatmap, the program would automatically wait for the response and that I wouldn't have to use Await.result is that not the case? Is there a way to avoid using Await.result to not block the thread on production code?
override def findOrCreate(phoneNumber: String, creationReason: String): Future[AvroCustomer] = {
//query for customer in db
//TODO this goes into createUserAndEvent before checking that response comes back empty from querying for user
customerByPhone(phoneNumber)
.flatMap(_ => createUserAndEvent(phoneNumber, creationReason, 1.0))
}
You don't need to use Await.result or any other blocking. You do in fact have the result from customerByPhone, you're just ignoring it with the _ . I think what you want is something like this:
customerByPhone(phoneNumber)
.flatMap(customer => {
if(customer == null)
createUserAndEvent(phoneNumber, creationReason, 1.0)
else
Future(customer)
})
You need to code the logic to do something only if the customer isn't there.
I have seen similar questions, but no answers that match exactly what I need. If it's out there, I haven't seen it. I'm on Rails 3.1, Ruby 1.9.2, deployed on Heroku Cedar stack (so I'm using postgres)
I've got this model:
class User
has_many :orders
end
class Order
belongs_to :user
#scope :last_for_each_user, ???
#scope :first_for_each_user, ???
end
I need scopes (or methods) that return all orders that are all users' first orders, and all orders that are all users' last orders. My first instinct was to use GROUP BY, which worked fine for :last_for_each_user in SQLite3, but Postgres won't let you select columns that aren't in the GROUP BY clause or selected as part of aggregate functions, and I need fully formed orders objects (i.e., all of the columns, orders.*).
Any ideas? I don't want to select the whole table and sort it out in Ruby. That just doesn't feel right, and it's a lot to hold in memory.
So here's what I ended up doing, for future head-scratchers:
scope :last_for_each_user, lambda {
select("orders.*") & Order.select("max(orders.id) as id").group("orders.user_id")
}
scope :first_for_each_user, lambda {
select("orders.*") & Order.select("min(orders.id) as id").group("orders.user_id")
}
If anyone has a better idea, I am very much open to hear it. And thanks to the whole SO community (even though no one responded), this is the first time I've signed up/asked a question but every time I google a problem, I end up here!
You could use the DISTINCT ON() rules to get only the last one. Here's an example following your model structure:
scope :for_users, lambda {|ids, is_first| includes(:user).where("user_id IN (?)", ids).select("DISTINCT ON (users.id) *").order("orders.created_at ?", is_first ? : "ASC" : "DESC") }
scope :last_for_each_users, lambda { |users| for_each_users(users.map{|u| u.id }, false }
scope :first_for_each_users, lambda { |users| for_each_users(users.map{|u| u.id }, true }