My program requires a large number of connections to be open (Mongo). I get the error :
Too many connections open, can't open anymore
after 819 connections. I already know we can increase this limit. But that's not what I have in mind. I'm thinking of closing the MongoClient object, and then creating a new one again after 800 connections.
My thinking is that with a new mongoClient object all the connections will be closed and when I start/create it again, the connections will be opened again until 800.
Thus not giving the error. (Let me know if this approach is totally wrong/ won't give the required results.)
For this I need to know the number of connections opened ATM. Is there any way to get this information using java?
You can get connection information by using the db.serverStatus() command. It has a connections subdocument which contains the total/available connections information.
For more information :
Documentation of server status
Details of connections block
Check the number of MongoDB connections using MongoDB Scala driver:
Create a MongoDB client:
import org.mongodb.scala._
import scala.collection.JavaConverters._
import scala.concurrent.Await
import scala.concurrent.duration._
import scala.util.{Failure, Success, Try}
// To directly connect to the default server localhost on port 27017
val mongodbClient: MongoClient = MongoClient()
// Use a Connection String
val mongodbClient: MongoClient = MongoClient("mongodb://localhost")
// or provide custom MongoClientSettings
val settings: MongoClientSettings = MongoClientSettings.builder()
.applyToClusterSettings(b => b.hosts(List(new ServerAddress("localhost")).asJava).
.build()
val mongodbClient: MongoClient = MongoClient(settings)
Call getNoOfMongodbConnection by passing mongodbClient:
val result = getNoOfMongodbConnection(mongodbClient)
Method to get the number of connections(current, available and total)
def getNoOfMongodbConnection(mongodbClient: MongoClient) = {
val adminDatabase = mongodbClient.getDatabase("admin")
val serverStatus = adminDatabase.runCommand(Document("serverStatus" -> 1)).toFuture()
Try {
Await.result(serverStatus, 10 seconds)
} match {
case Success(x) => {
val connection = x.get("connections")
logger.info("Number of mongodb connection:--> " + connection)
connection
}
case Failure(ex) => {
logger.error("Got error while getting the number of Mongodb connection:---> " + ex.printStackTrace())
None
}
}
}
Related
I am trying to connect to azure cosmos DB from intelliJ using scala language.
I am able to fetch the data from DB when I am connecting my laptop from home network. But I am getting timeout error when I connect the laptop from office network. It seems like some proxy issue, So that I have configured the proxy settings in intelliJ as shown below.
I tried to test my connection from proxy settings page, but it is throwing error as below.
Problem with connection: Request failed with status code 401
I tried to open the same from browser and received below error.
{"code":"Unauthorized","message":"Required Header authorization is missing. Ensure a valid Authorization token is passed.\r\nActivityId: af2d771f-25a3-494a-90dd-b33cd66104e2, Microsoft.Azure.Documents.Common/2.5.1"}
Below is the intelliJ code I tried and this code works.
mport org.apache.spark.sql.SparkSession
import com.microsoft.azure.cosmosdb.spark.schema._
import com.microsoft.azure.cosmosdb.spark._
import com.microsoft.azure.cosmosdb.spark.config.Config
import org.apache.log4j.Logger
object DeleteData {
/* Get the logger */
val log = Logger.getLogger(getClass.getName)
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().master("local").appName("DeleteData").getOrCreate()
// Configure connection to your collection
val readConfig = Config(Map(
"Endpoint" -> "https://***.documents.azure.com:443/",
"Masterkey" -> "***",
"Database" -> "ConnectivityDB",
"Collection" -> "historicData",
"ConnectionMode" -> "Gateway",
"SamplingRatio" -> "1.0",
"query_custom" -> "SELECT c.NUM from c where c._ts = 1564566440"
))
// Connect via azure-cosmosdb-spark to create Spark DataFrame
val docs = spark.read.cosmosDB(readConfig)
println("Total Count: " +docs.count())
docs.show(5)
spark.stop()
}
}
Is there any way to bypass the proxy for getting connected to cosmos DB from intelliJ ?
I'm afraid that the IntelliJ proxy setting doesn't affect your code,you could try to catch the request by Fiddler tool.
Maybe you could set the proxy setting in the code.Please see this link.
import java.net.Authenticator
import java.net.PasswordAuthentication
class ProxyAuthenticator(user: String, password: String) extends Authenticator {
def this() = this("default_user_name", "default_password")
override def getPasswordAuthentication(): PasswordAuthentication = {
return new PasswordAuthentication(user, password.toCharArray());
}
}
I'm prototyping a network server using Akka Streams that will listen on a port, accept incoming connections, and continuously read data off each connection. Each connected client will only send data, and will not expect to receive anything useful from the server.
Conceptually, I figured it would be fitting to model the incoming events as one single stream that only incidentally happens to be delivered via multiple TCP connections. Thus, assuming that I have a case class Msg(msg: String) that represents each data message, what I want is to represent the entirety of incoming data as a Source[Msg, _]. This makes a lot of sense for my use case, because I can very simply connect flows & sinks to this source.
Here's the code I wrote to implement my idea:
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.SourceShape
import akka.stream.scaladsl._
import akka.util.ByteString
import akka.NotUsed
import scala.concurrent.{ Await, Future }
import scala.concurrent.duration._
case class Msg(msg: String)
object tcp {
val N = 2
def main(argv: Array[String]) {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
val connections = Tcp().bind("0.0.0.0", 65432)
val delim = Framing.delimiter(
ByteString("\n"),
maximumFrameLength = 256, allowTruncation = true
)
val parser = Flow[ByteString].via(delim).map(_.utf8String).map(Msg(_))
val messages: Source[Msg, Future[Tcp.ServerBinding]] =
connections.flatMapMerge(N, {
connection =>
println(s"client connected: ${connection.remoteAddress}")
Source.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val F = builder.add(connection.flow.via(parser))
val nothing = builder.add(Source.tick(
initialDelay = 1.second,
interval = 1.second,
tick = ByteString.empty
))
F.in <~ nothing.out
SourceShape(F.out)
})
})
import scala.concurrent.ExecutionContext.Implicits.global
Await.ready(for {
_ <- messages.runWith(Sink.foreach {
msg => println(s"${System.currentTimeMillis} $msg")
})
_ <- system.terminate()
} yield (), Duration.Inf)
}
}
This code works as expected, however, note the val N = 2, which is passed into the flatMapMerge call that ultimately combines the incoming data streams into one. In practice this means that I can only read from that many streams at a time.
I don't know how many connections will be made to this server at any given time. Ideally I would want to support as many as possible, but hardcoding an upper bound doesn't seem like the right thing to do.
My question, at long last, is: How can I obtain or create a flatMapMerge stage that can read from more than a fixed number of connections at one time?
As indicated by Viktor Klang's comments I don't think this is possible in 1 stream. However, I think it would be possible to create a stream that can receive messages after materialization and use that as a "sink" for messages coming from the TCP connections.
First create the "sink" stream:
val sinkRef =
Source
.actorRef[Msg](Int.MaxValue, fail)
.to(Sink foreach {m => println(s"${System.currentTimeMillis} $m")})
.run()
This sinkRef can be used by each Connection to receive the messages:
connections foreach { conn =>
Source
.empty[ByteString]
.via(conn.flow)
.via(parser)
.runForeach(msg => sinkRef ! msg)
}
I'm using Akka 2.4.4 and trying to move from Apache HttpAsyncClient (unsuccessfully).
Below is simplified version of code that I use in my project.
The problem is that it hangs if I send more than 1-3 requests to the flow. So far after 6 hours of debugging I couldn't even locate the problem. I don't see exceptions, error logs, events in Decider. NOTHING :)
I tried reducing connection-timeout setting to 1s thinking that maybe it's waiting for response from the server but it didn't help.
What am I doing wrong ?
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.headers.Referer
import akka.http.scaladsl.model.{HttpRequest, HttpResponse}
import akka.http.scaladsl.settings.ConnectionPoolSettings
import akka.stream.Supervision.Decider
import akka.stream.scaladsl.{Sink, Source}
import akka.stream.{ActorAttributes, Supervision}
import com.typesafe.config.ConfigFactory
import scala.collection.immutable.{Seq => imSeq}
import scala.concurrent.{Await, Future}
import scala.concurrent.duration.Duration
import scala.util.Try
object Main {
implicit val system = ActorSystem("root")
implicit val executor = system.dispatcher
val config = ConfigFactory.load()
private val baseDomain = "www.google.com"
private val poolClientFlow = Http()(system).cachedHostConnectionPool[Any](baseDomain, 80, ConnectionPoolSettings(config))
private val decider: Decider = {
case ex =>
ex.printStackTrace()
Supervision.Stop
}
private def sendMultipleRequests[T](items: Seq[(HttpRequest, T)]): Future[Seq[(Try[HttpResponse], T)]] =
Source.fromIterator(() => items.toIterator)
.via(poolClientFlow)
.log("Logger")(log = myAdapter)
.recoverWith {
case ex =>
println(ex)
null
}
.withAttributes(ActorAttributes.supervisionStrategy(decider))
.runWith(Sink.seq)
.map { v =>
println(s"Got ${v.length} responses in Flow")
v.asInstanceOf[Seq[(Try[HttpResponse], T)]]
}
def main(args: Array[String]) {
val headers = imSeq(Referer("https://www.google.com/"))
val reqPair = HttpRequest(uri = "/intl/en/policies/privacy").withHeaders(headers) -> "some req ID"
val requests = List.fill(10)(reqPair)
val qwe = sendMultipleRequests(requests).map { case responses =>
println(s"Got ${responses.length} responses")
system.terminate()
}
Await.ready(system.whenTerminated, Duration.Inf)
}
}
Also what's up with proxy support ? Doesn't seem to work for me either.
You need to consume the body of the response fully so that the connection is made available for subsequent requests. If you don't care about the response entity at all, then you can just drain it to a Sink.ignore, something like this:
resp.entity.dataBytes.runWith(Sink.ignore)
By the default config, when using a host connection pool, the max connections is set to 4. Each pool has it's own queue where requests wait until one of the open connections becomes available. If that queue ever goes over 32 (default config, can be changed, must be a power of 2) then yo will start seeing failures. In your case, you only do 10 requests, so you don't hit that limit. But by not consuming the response entity you don't free up the connection and everything else just queues in behind, waiting for the connections to free up.
I am using mongo with play framework with "reactivemongo", That makes an async bridge between mongo connection and programm. For standalone projects I always use casbah lib - it has more native syntax (sometimes using of Futures in each request is not needed, and my religion does not allow me to use Async.await for blocking each request ) for me and no actors overhead, also I don't like JSON BSON conversion overhead.
But using casbah in play framework direct way(just create Mongo connection in controller ) produces connection leaks - this means you should create connection pool and control yourself, otherwords write reactivemongo.
Has anybody used casbah with mongo in production ? Where is the best and most canonical way of creating and controlling connection in play ecosystem ?
First you should check Connecting to MongoDB. Now go to this tutorial create scala project ( If you used other editor then follow scala project creation steps ).
Now check this following steps :
1> projectName/conf/application.conf add application.conf mongo Db name, collection name, port number, URL etc. play reactive mongo for ex: I added following in my application.conf
mongodb.default.host = "localhost"
mongodb.default.db = "Demo"
mongodb.default.port = "27017"
CI.default.uri = "mongodb://localhost:27017/"
2> Create a .scala file in controller folder give any name for ex. I set file name as ScalaMongoFactory and add following code in this file
import com.mongodb.casbah. {
MongoClient, MongoClientURI
}
import com.typesafe.config.ConfigFactory
object ScalaMongoFactory {
private val config = ConfigFactory.load()
private val DATABASE = config.getString("mongodb.default.db")
private val server = MongoClientURI(config.getString("CI.default.uri"))
private val client = MongoClient(server)
val database = client(DATABASE)
}
3> Now create a new .scala file in controller where you want to use mongo connection. For ex. I created checkConnection.scala file and contains like
import com.cloudinsights.scala.controllers. {
ScalaMongoFactory
}
object checkConnection {
val collection = ScalaMongoFactory.database("your collectionName")
}
There's no need to use Async.wait with reactive mongo (and you shouldn't anyway).
I guess you could use a utility object to manage your connection.
import com.mongodb.casbah.{MongoClient, MongoDB}
import play.api.Play
object MongoManager {
private val server = Play.current.configuration.getString("db.host").get
private val port = Play.current.configuration.getInt("db.port").get
object using {
def apply[A](col: String)(block: MongoDB => A): A = {
val con = MongoClient(server, port)
val a = block(con.apply(col))
con.close
a
}
def apply[A](block: MongoClient => A): A = {
val con = MongoClient(server, port)
val a = block(con)
con.close
a
}
}
object stashed {
private lazy val con = MongoClient(server, port)
def apply(col: String): MongoDB = con.apply(col)
def apply: MongoClient = con
}
}
I didn't find a play plugin for this driver.
Personally I'd recommend using the ReactiveMongo driver instead, since it can also use play's JSON library. If you're taking data from the database and feeding it through a REST api, its a nicer option.
I am trying to set up a simple server/client akka (using Akka 2.0.3) application, but it failed to connect. Beforehand here is the basic code:
import com.typesafe.config.ConfigFactory
import akka.actor.Actor
import akka.actor.ActorSystem
import akka.actor.Props
class Server extends Actor {
def receive = {
case s: String => println("Got " + s)
}
}
val serverSystem = ActorSystem("server", ConfigFactory.load(ConfigFactory.parseString("""
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
transport = "akka.remote.netty.NettyRemoteTransport"
netty {
hostname = "localhost"
port = 5678
}
}
}
""")))
val server = serverSystem.actorOf(Props[Server], name = "server")
Thread.sleep(500)
println("started")
Thread.sleep(500)
val clientSystem = ActorSystem("client", ConfigFactory.load(ConfigFactory.parseString("""
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
}
""")))
val remoteServer = clientSystem.actorFor("akka://server#XXX:5678/user/server")
remoteServer ! "HEY"
Thread.sleep(3000)
clientSystem.shutdown
serverSystem.shutdown
I know that the configurations should be placed in external files.
If you replace XXX with localhost it works:
started
Got HEY
But if I used my external (resolved) IP (PC behind home router) for XXX the HEY message never arrives. I thought it is due to some firewall problem and forwarded the related TCP and UDP ports at my router and also opened/allowed them at my Windows firewall. So after that the following code worked (also XXX replaced with my external IP). A started ServerTest can be connected by a ClientTest:
import java.net.ServerSocket
object ServerTest extends App {
println("START server")
val ss = new ServerSocket(5678)
val s = ss.accept()
println(s)
println("END")
}
import java.net.Socket
object ClientTest extends App {
println("START client")
val s = new Socket("XXX", 5678)
println(s)
println("END")
}
So it´s not a port/firewall problem, isn´t it?! So where is the problem???
localhost usually means 127.0.0.1, which is only one of the possibly many interfaces (cards) in a computer. The server binding to localhost won't receive connections connecting to the other interfaces (including the one with the external address).
You should either specify the external address in the server, or 0.0.0.0 which means "bind to all interfaces".