I am new with Scala and I am writing a simple rss reader.
I have class Manager for managing feeds and content.
package lib
import scala.xml._
import java.net.URL
import net.liftweb.couchdb.{CouchDB, Database}
import dispatch.{Http, StatusCode}
/**
* #author smix
*
* Feeds manager
*/
object Manager {
var db = List[Database]()
/*
* Initialize CouchDb databases
*/
def init = {
this.appendDb(new Database("myserver.com", 5984, "content"))
}
/*
* Append a new database to the databases list
*/
private def appendDb(database: Database) : Unit = {
database :: db
// Strange exception if database has been already created
/* try {
this.db.head.createIfNotCreated(new Http())
} catch {
case e:java.lang.NoClassDefFoundError => {}
} */
}
/*
* Fetch articles from feed by url
*/
def fetchItems(feedUrl: String): List[scala.xml.Elem] = {
val rssFeed = XML.load( (new URL(feedUrl)).openConnection.getInputStream )
val items = rssFeed \ "channel" \ "item"
val articles: List[scala.xml.Elem] = List()
for(item <- items) {
item :: articles
}
articles
}
}
I want to store content in CouchDb. I need to have list of couch databases(feeds, articles, etc...). I wrote class but when I call appendDb i get an error:
Exception in thread "main" java.lang.NoClassDefFoundError: lib/Manager$
at collector$.main(collector.scala:5)
at collector.main(collector.scala)
Caused by: java.lang.ClassNotFoundException: lib.Manager$
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 2 more
When I rewrited db definition: var db = List[Int]() and the first line of appendDb: 1 :: this.db project ran fine... Strange.
Also, it is interesting why am I getting exception when I call createIfNotCreated for existing database(commented try-catch block in appendDb).
The exception indicates that you're missing some classes (one or more JAR files, presumably) when you run your program, though they're either irrelevant to compiling it or they are available then.
You should also note that the first line in appendDb accomplishes nothing. It builds a new List by consing database onto the front of the List referred to by db, but the resulting value is discarded. Perhaps you meant this:
db = database :: db
Related
I'm having a sudden issue with this error when I am running my AWS EMR client on our deployed server. This does not happen locally, and runs fine. Basically, I have an EMR client that I use to build and execute steps as such:
class EMRClient(emrClusterId:String) extends LazyLogging{
val accessKey = ...// access key
val secretKey = ...//secret key
val credentials = new BasicAWSCredentials(accessKey, secretKey)
val REGION = <my region>
println(">>>>>>>>>>>>>>>>>>>>Initializing EMR client for clusterId " + emrClusterId + " . The region is " + REGION)
val emr = AmazonElasticMapReduceClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withRegion(REGION)
.build()
def executeHQLStep(s3ScriptPath:String, stepName:String, args:String = ""): AddJobFlowStepsResult= {
val hqlScriptStep = buildHQLStep(hqlScriptPath, stepName, args)
val stepSet = new java.util.HashSet[StepConfig]()
//stepSet.add(enableDebugging)
stepSet.add(hqlScriptStep)
executeJobFlowSteps(stepSet)
}
/**
* Builds a StepConfig to be executed in a job flow for a given .hql file from S3
* #param hqlScriptPath the location in S3 of the script file containing the script to run
* #param args optional field for arguments for hive script.
* #param stepName An identifier to give to EMR to name your Step
* #return
*/
private def buildHQLStep(hqlScriptPath:String, stepName:String, args:String= ""): StepConfig = {
new StepConfig()
.withName(stepName)
.withActionOnFailure(ActionOnFailure.CANCEL_AND_WAIT)
.withHadoopJarStep(stepFactory.newRunHiveScriptStep(hqlScriptPath, args))
}
private def executeJobFlowSteps(steps: java.util.Set[StepConfig]): AddJobFlowStepsResult = {
emr.addJobFlowSteps(new AddJobFlowStepsRequest()
.withJobFlowId(emrClusterId)
.withSteps(steps)) // where the error is thrown
}
}
However, when this class is instantiated on the server, none of the println statements at the top are visible, my executeJobFlowSteps method is called and throws this error:
java.lang.NoSuchFieldError: SIGNING_REGION
at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.executeAddJobFlowSteps(AmazonElasticMapReduceClient.java:439)
at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.addJobFlowSteps(AmazonElasticMapReduceClient.java:421)
at emrservices.EMRClient.executeJobFlowSteps(EMRClient.scala:64)
at emrservices.EMRClient.executeHQLStep(EMRClient.scala:44)
This project is composed of several projects, and similar issues to mine have said it has to do with their AWS dependencies, but across the board all of the projects have this in the build.sbt's library dependencies: "com.amazonaws" % "aws-java-sdk" % "1.11.286"
Any idea on what the issue is?
This looks like you are mixing the 1.11.286 version of aws-java-sdk with an older version (1.11.149) of aws-java-sdk-core. The newer client is using a new field added to the core module but since your core module is out of date you are seeing the no such field error. Can you ensure all of your dependencies are in sync with one another?
I set up a test class that runs 13 integration tests. All of the tests involve 1 database, 1 schema and 3 tables in the schema. The test class is set up so that I create the test database with #BeforeClass and delete the test database with #AfterClass. In the setUp phase for each test, I migrate the schema and populate the tables with artificial data. Then in the tearDown phase I remove the schema. In each test, I run my test logic against the test database:
class MyTest {
companion object {
private val helper = MyHelper()
#BeforeClass #JvmStatic
fun createTestDatabase() {
helper.createTestDatabase()
}
#AfterClass #JvmStatic
fun removeTestDatabase() {
helper.deleteTestDatabase()
}
}
#Before
fun setUp(){
// migrate the schema and populate the tables with artificial data
helper.migrate()
helper.populate()
}
#After
fun tearDown() {
// remove the schema
helper.removeSchema()
}
#Test MyFirstTest() {
// populate the tables with artificial data
... // test logic
}
...
}
The database manipulations are all implemented in MyHelper class. In particular, I have one function in MyHelper that executes a query string:
class MyHelper {
protected val POSTGRES_USER = "postgres"
protected val POSTGRES_HOST = "localhost"
protected val POSTGRES_PORT = 5432
protected val POSTGRES_TEST_DATA_DATABASE = "my_database_test"
protected val POSTGRES_TEST_DATA_SCHEMA = "my_schema"
protected val POSTGRES_DEFAULT_DATABASE = "postgres"
protected val POSTGRES_DEFAULT_SCHEMA = "postgres"
protected val graphDataTables = listOf("my_table_1", "my_table_2", "my_table_3")
val databases = listOf(POSTGRES_DEFAULT_DATABASE, POSTGRES_TEST_DATA_DATABASE)
val dataSources = databases.associateBy({it}, { getDataSource(it)})
fun getDataSource(database: String): PGSimpleDataSource {
val ds = PGSimpleDataSource()
val url = "jdbc:postgresql://$POSTGRES_HOST:$POSTGRES_PORT/$database"
ds.setUrl(url)
ds.setUser(POSTGRES_USER)
return ds
}
fun executeQuery(sqlQuery: String,
database: String = this.POSTGRES_DEFAULT_DATABASE,
schema: String = this.POSTGRES_DEFAULT_SCHEMA
): Unit {
val ds = dataSources[database]!!
ds.currentSchema = schema
ds.connection.use {
it.createStatement().use {
it.execute(sqlQuery)
}
}
}
...
}
The function MyHelper.populate() uses the executeQuery function to insert data into the tables.
My problem is that it seems that I have many unclosed connections (leaked connections). If I keep the # of tests under 10, then all tests pass without a problem. As soon as I pass 10, I get errors like the following for tests 11 and up:
Nov 09, 2017 11:31:55 AM org.postgresql.Driver connect
SEVERE: Connection error:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
at org.postgresql.Driver$ConnectThread.getResult(Driver.java:401)
at org.postgresql.Driver.connect(Driver.java:259)
...
for the tests that got run after the 10th. And the last stack in the log points to the ds.connection.use { line in the function executeQuery.
However, if I move the helper.createTestDatabase() out of the companion object, and put it in the init block for MyTest, like so:
class MyTest {
companion object {
private val helper = MyHelper()
}
init {
helper.createTestDatabase()
}
...
}
then I don't get any error even if the # of my tests are over 10. The problem I see with this is that now I have a test database created in my postgres docker container and it still exists after I'm done with the integration tests. I have no idea how to call removeTestDatabase now.
I run my tests in debug mode and stopped in the middle of the 10th test, then with
select * from pg_stat_activity;
I can see that there are many idle connections (almost 100) in the first scenario. In the 2nd scenario, I have only 10 idle connections in the middle of the 10th test (I'd like the number of idle connections to be 0, of course). My understanding is that my default postgres setting is that max_connections is set to 100. That's why I get the error when I have more than 10 tests in the first scenario. But I have no idea how I am creating idle connections.
I am writing an application that writes data to a graph on orientDB (v 2.2.3) this graph is something like the following:
I have threads that add vertices to C vertices each C vertex has an independent thread which is responsible add D vertexes with there edges.
Each thread is working on a separate transaction, I have been getting various errors and exception like the following:
com.orientechnologies.orient.core.exception.OStorageException: Error on commit
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:253)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:189)
at com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:1271)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:549)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:109)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2665)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2634)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.commit(OrientTransactionalGraph.java:175)
at JSONManager$.commitGrap2(JSONManager.scala:371)
at JSONManager$$anonfun$main$2$$anon$1.run(JSONManager.scala:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
.....
Caused by: java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
at java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:739)
at com.orientechnologies.orient.client.remote.OStorageRemote$28.execute(OStorageRemote.java:1284)
at com.orientechnologies.orient.client.remote.OStorageRemote$28.execute(OStorageRemote.java:1271)
at com.orientechnologies.orient.client.remote.OStorageRemote$2.execute(OStorageRemote.java:192)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:224)
... 12 more
UPDATE code:
val t: Runnable = new Runnable {
override def run(): Unit = {
graph = factory.getTx
saveDUnits(dUnit, graph)
commitGrap(graph)
graph.shutdown()
}
};
pool.execute(t)
def commitGrap(graph: OrientGraph): Unit = {
var retryCount = 0
while (retryCount < 10) {
try {
graph.commit()
retryCount = 11
} catch {
case e: Exception => println("Commit Error")
e.printStackTrace()
var sleepTime = 50
if (retryCount > 5) {
sleepTime = 6000
}
Thread.sleep(sleepTime);
}
retryCount = retryCount + 1
}
}
Finally I found the error what i did, the problem was in creating OrientGraphFactory instance, the non thread safe factory is created like the follwoing
var factory: OrientGraphFactory = new OrientGraphFactory("remote:106.140.20.233/test", "root", "123")
The thread safe factory is created like the following:
var factory: OrientGraphFactory = new OrientGraphFactory("remote:106.140.20.233/test", "root", "123").setupPool(1, 20)
I missed adding .setPool(1,20)
That's it
I have successfully implemented a simple web service using Spray and Slick that passes an incoming request through a Spark ML Prediction Pipeline. Everything was working fine until I tried to add a data layer. I have chosen Slick it seems to be popular.
However, I can't quite get it to work right. I have been basing most of my code on the Hello-Slick Activator Template. I use a DAO object like so:
object dataDAO {
val datum = TableQuery[Datum]
def dbInit = {
val db = Database.forConfig("h2mem1")
try {
Await.result(db.run(DBIO.seq(
datum.schema.create
)), Duration.Inf)
} finally db.close
}
def insertData(data: Data) = {
val db = Database.forConfig("h2mem1")
try {
Await.result(db.run(DBIO.seq(
datum += data,
datum.result.map(println)
)), Duration.Inf)
} finally db.close
}
}
case class Data(data1: String, data2: String)
class Datum(tag: Tag) extends Table[Data](tag, "DATUM") {
def data1 = column[String]("DATA_ONE", O.PrimaryKey)
def data2 = column[String]("DATA_TWO")
def * = (data1, data2) <> (Data.tupled, Data.unapply)
}
I initialize my database in my Boot object
object Boot extends App {
implicit val system = ActorSystem("raatl-demo")
Classifier.initializeData
PredictionDAO.dbInit
// More service initialization code ...
}
I try to add a record to my database before completing the service request
val predictionRoute = {
path("data") {
get {
parameter('q) { query =>
// do Spark stuff to get prediction
DataDAO.insertData(data)
respondWithMediaType(`application/json`) {
complete {
DataJson(data1, data2)
}
}
}
}
}
When I send a request to my service my application crashes
java.lang.OutOfMemoryError: PermGen space
I suspect I'm implementing the Slick API incorrectly. its hard to tell from the documentation, because it stuffs all the operations into a main method.
Finally, my conf is the same as the activator ui
h2mem1 = {
url = "jdbc:h2:mem:raatl"
driver = org.h2.Driver
connectionPool = disabled
keepAliveConnection = true
}
Has anyone encountered this before? I'm using Slick 3.1
java.lang.OutOfMemoryError: PermGen space is normally not a problem with your usage, here is what oracle says about this.
The detail message PermGen space indicates that the permanent generation is full. The permanent generation is the area of the heap where class and method objects are stored. If an application loads a very large number of classes, then the size of the permanent generation might need to be increased using the -XX:MaxPermSize option.
I do not think this is because of incorrect implementation of the Slick API. This probably happens because you are using multiple frameworks that loads many classes.
Your options are:
Increase the size of perm gen size -XX:MaxPermSize
Upgrade to Java 8. The perm gen space is now replaced with MetaSpace which is tuned automatically
I am trying to run specification test in Play/Scala/ReactiveMongo project. Setup is like this:
class FeaturesSpec extends Specification {
"Features controller" should {
"create feature from JSON request" in withMongoDb { app =>
// do test
}
}
With MongoDbFixture as follows:
object MongoDBTestUtils {
def withMongoDb[T](block: Application => T): T = {
implicit val app = FakeApplication(
additionalConfiguration = Map("mongodb.uri" -> "mongodb://localhost/unittests")
)
running(app) {
def db = ReactiveMongoPlugin.db
try {
block(app)
} finally {
dropAll(db)
}
}
}
def dropAll(db: DefaultDB) = {
Await.ready(Future.sequence(Seq(
db.collection[JSONCollection]("features").drop()
)), 2 seconds)
}
}
When test runs, logs are pretty noisy, complain about resource being already closed. Although tests work correctly, this is weird and I would like to know why this occurs and how to fix it.
Error:
[info] application - ReactiveMongoPlugin stops, closing connections...
[warn] play - Error stopping plugin
java.lang.IllegalStateException: Can't get ClosableLazy value after it has been closed
at play.core.ClosableLazy.get(ClosableLazy.scala:49) ~[play_2.11-2.3.7.jar:2.3.7]
at play.api.libs.concurrent.AkkaPlugin.applicationSystem(Akka.scala:71) ~[play_2.11-2.3.7.jar:2.3.7]
at play.api.libs.concurrent.Akka$$anonfun$system$1.apply(Akka.scala:29) ~[play_2.11-2.3.7.jar:2.3.7]
at play.api.libs.concurrent.Akka$$anonfun$system$1.apply(Akka.scala:29) ~[play_2.11-2.3.7.jar:2.3.7]
at scala.Option.map(Option.scala:145) [scala-library-2.11.4.jar:na]
The exception means that you are using the ReactiveMongo plugin after the application has stopped.
You might wanna try using Around:
class withMongoDb extends Around with Scope {
val db = ReactiveMongoPlugin.db
override def around[T: AsResult](t: => T): Result = try {
val res = t
AsResult.effectively(res)
} finally {
...
}
}
You should also take a look at Flapdoodle Embedded Mongo, with that you don't have to delete databases after testing IIRC.
This problem likely occurs because your test exercises code that references a closed MongoDB instance. After each Play Specs2 test runs, the MongoDb connection is reset, thus your first test may pass, but a subsequent test may hold a stale reference to the closed instance, and as a result fail.
One way to solve this issue is to ensure the following criteria are met in your application:
Avoid using val or lazy val for MongoDb database resources
(Re)Initialize all database references on application start.
I wrote up a blog post that describes a solution to the problem within the context of a Play Controller.