I would like to programmatically enable sharding for a MongoDB database and set a shard key using the MongoDB Scala driver.
val mongoClient: MongoClient = MongoClient("mongodb://localhost")
val db = mongoClient.getDatabase("mydatabase")
val views = db.getCollection("mycollection")
val obs = views.createIndex(ascending("x"))
I was wondering whether I can use:
db.runCommand(....)
But I couldn't find anything in the documentation.
To enable sharding:
val db = mongoClient.getDatabase("admin")
val obs = db.runCommand(Document("enableSharding" -> "mydatabase"))
To shard a collection:
val db = mongoClient.getDatabase("admin")
val obs = db.runCommand(Document("shardCollection" -> "mydatabase.mycollection", "key" -> Document("xx" -> 1)))
Related
I am on Ubuntu 20.04. I want to write some data in Scala to MongoDB. Here's what I have:
import org.mongodb.scala.bson.collection.immutable.{Document => MongoDocument}
import org.mongodb.scala.{MongoClient, MongoCollection, MongoDatabase}
object Application extends App {
val mongoClient: MongoClient = MongoClient()
// Use a Connection String
//val mongoClient: MongoClient = MongoClient("mongodb://localhost")
val database: MongoDatabase = mongoClient.getDatabase("mydb")
val collection: MongoCollection[MongoDocument] = database.getCollection("user")
val doc: MongoDocument = MongoDocument("_id" -> 0, "name" -> "MongoDB", "type" -> "database",
"count" -> 1, "info" -> MongoDocument("x" -> 203, "y" -> 102))
collection.insertOne(doc)
val documents = (1 to 100) map { i: Int => MongoDocument("i" -> i) }
collection.insertMany(documents)
}
The error (not even an error, INFO level) I get:
Nov 16, 2020 1:42:08 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Cluster created with settings {hosts=[localhost:27017],
mode=SINGLE, requiredClusterType=UNKNOWN,
serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
And nothing happens to the database. No data appears there. No errors, no insertions into Mongo, nothing.
I used primarily these sources as examples:
https://mongodb.github.io/mongo-scala-driver/2.9/getting-started/quick-tour/
https://blog.knoldus.com/how-scala-interacts-with-mongodb/
MongoDB is up, status is active. Inserting data from the terminal was done successfully. So, the program's behavior I have is strange. I've been searching everywhere on the Internet for the answers but I can't seem to find it. Your help will be appreciated a lot. Thank you!
Thanks to #Luis Miguel Mejía Suárez's help. Here's what I have done so far: added an Observer implementation and a promise. Thanks to this post: Scala script wait for mongo to complete task . That's what I have now:
val mongoClient: MongoClient = MongoClient("mongodb://localhost")
val database: MongoDatabase = mongoClient.getDatabase("mydb")
val collection: MongoCollection[MongoDocument] = database.getCollection("user")
val doc: MongoDocument = MongoDocument("name" -> "MongoDB", "type" -> "database",
"count" -> 1, "info" -> MongoDocument("x" -> 203, "y" -> 102))
val observable: Observable[Completed] = collection.insertOne(doc)
val promise = Promise[Boolean]
observable.subscribe(new Observer[Completed] {
override def onNext(result: Completed): Unit = println("Inserted")
override def onError(e: Throwable): Unit = {
println("Failed")
promise.success(false)
}
override def onComplete(): Unit = {
println("Completed")
promise.success(true)
}
})
val future = promise.future
Await.result(future, Duration(5, java.util.concurrent.TimeUnit.SECONDS))
mongoClient.close()
Generally speaking, it works in most cases. Though, I didn't handle the case with insertMany method where the program has to wait for the last element insertion. My realization does not work properly with this.
P.S. Turns out insertMany also works fine with this example, I just tested it with the wrong data.
i am new to mongo scala driver i am trying to add connection pool size when initiating a mongo instance
i am doing it something like this
val settings: MongoClientSettings = MongoClientSettings.builder()
.applyToConnectionPoolSettings(ConnectionPoolSettings.Builder.maxSize(100))
.applyToClusterSettings(b => b.hosts(List(new ServerAddress("localhost")).asJava).description("Local Server"))
.build()
val mongoClient: MongoClient = MongoClient(settings)
value maxSize is not a member of object com.mongodb.connection.ConnectionPoolSettings.Builder
[error] .applyToConnectionPoolSettings(ConnectionPoolSettings.Builder.maxSize(100))
what is the right way to do this?
It is a small typo on your code
ConnectionPoolSettings.Builder.maxSize(100)
should be
ConnectionPoolSettings.builder().maxSize(100)
The code with the Block would look like this:
val settings: MongoClientSettings = MongoClientSettings.builder()
.applyToConnectionPoolSettings((t: ConnectionPoolSettings.Builder) => t.applySettings(ConnectionPoolSettings.builder().maxSize(100).build()))
.applyToClusterSettings(b => b.hosts(List(new ServerAddress("localhost")).asJava).description("Local Server"))
.build()
I use Spark 1.3.0
Let's say I have a dataframe in Spark and I need to store this to Postgres DB (postgresql-9.2.18-1-linux-x64) on a 64bit ubuntu machine.
I also use postgresql9.2jdbc41.jar as a driver to connect to postgres
I was able to read data from postgres DB using the below commands
import org.postgresql.Driver
val url="jdbc:postgresql://localhost/postgres?user=user&password=pwd"
val driver = "org.postgresql.Driver"
val users = {
sqlContext.load("jdbc", Map(
"url" -> url,
"driver" -> driver,
"dbtable" -> "cdimemployee",
"partitionColumn" -> "intempdimkey",
"lowerBound" -> "0",
"upperBound" -> "500",
"numPartitions" -> "50"
))
}
val get_all_emp = users.select("*")
val empDF = get_all_emp.toDF
get_all_emp.foreach(println)
I want to write this DF back to postgres after some processing.
Is this below code right?
empDF.write.jdbc("jdbc:postgresql://localhost/postgres", "test", Map("user" -> "user", "password" -> "pwd"))
Any pointers(scala) would be helpful.
You should follow the code below.
val database = jobConfig.getString("database")
val url: String = s"jdbc:postgresql://localhost/$database"
val tableName: String = jobConfig.getString("tableName")
val user: String = jobConfig.getString("user")
val password: String = jobConfig.getString("password")
val sql = jobConfig.getString("sql")
val df = sc.sql(sql)
val properties = new Properties()
properties.setProperty("user", user)
properties.setProperty("password", password)
properties.put("driver", "org.postgresql.Driver")
df.write.mode(SaveMode.Overwrite).jdbc(url, tableName, properties)
i want to add mongoOptions to MongoClient basically i want to add ConnectionPerHost value its default is 10 i want to increase it to 20
but i am getting errors in the code i have tried with two different ways
val SERVER:ServerAddress = {
val hostName=config.getString("db.hostname")
val port=config.getString("db.port").toInt
new ServerAddress(hostName,port)
}
val DATABASE:String = config.getString("db.dbname")
method 1
val options=MongoClientOptions.apply( connectionsPerHost=20 )
val connectionMongo = MongoConnection(SERVER).addOption(options.getConnectionsPerHost)//returning Unit instead of MongoClient
val collectionMongo = connectionMongo(DATABASE)("testdb")
getting error on last line Unit does not take parameters
method 2
val mongoOption=MongoClientOptions.builder()
.connectionsPerHost(20)
.build();
getting error on MongoClientOptions.builder() line
value builder is not a member of object com.mongodb.casbah.MongoClientOptions
-
i want to set connectionsPerHost value to 20 please help what is the right way to do this
This seems to be working.
val config = ConfigFactory.load();
val hostName = config.getString("db.hostname")
val port = config.getInt("db.port")
val server = new ServerAddress(hostName, port)
val database = config.getString("db.dbname")
val options = MongoClientOptions(connectionsPerHost = 20)
val connectionMongo = MongoClient(server, options)
val collectionMongo = connectionMongo(database)("testdb")
Note that MongoConnection is deprecated.
In my project, I'm using Casbah and I reffered http://mongodb.github.io/casbah/tutorial.html tutorial. According to that I've implemented
val mongoClient = MongoClient
val db = mongoClient("db_name")
val coll = db("coll_name")
val a = MongoDBObject("hello" -> "world")
Then next step is not going to work.
coll.insert(a)
I'm using, "org.mongodb"%% "casbah" % 2.7.0-RC0
what is this issue ?
If running mongodb on localhost and the default port try changing:
val mongoClient = MongoClient()
So you call the apply method and return an instance of MongoClient not the object..
MongoFactory.coll.save(a)
use save() to save MongoDBObject.