Close GCP SecretManagerServiceClient after getting secrets from it - scala

How can I close the client in the following code:
object GoogleSecret {
def apply(
project: String,
version: String = SecretVersion,
credentials: CredentialsProvider = defaultProvider
): Try[SecretService] = Try {
require(project != null && project.nonEmpty, "project is null or empty")
val settings = SecretManagerServiceSettings.newBuilder
.setCredentialsProvider(credentialsProvider)
.build()
val client = SecretManagerServiceClient.create(settings)
logger.info(s"connected to project: $project")
secrets => secrets.flatMap {
secret =>
logger.debug(s"getting secret: $secret")
try {
val value = client
.accessSecretVersion(SecretVersionName.of(project, secret, version))
.getPayload
.getData
.toStringUtf8
Some(secret -> value)
} catch {
case e: Throwable => {
logger.error(s"google error ($secret): ${e.getMessage}")
client.shutdown()
client.awaitTermination(30, TimeUnit.SECONDS)
client.close()
None
}
}
}.toMap
}
the above method create a client to connect to GCP Secret Manager, to get secrets values, but i have some questions:
this fragment what is supposed to do :
secrets => secrets.flatMap {
secret =>
logger.debug(s"getting secret: $secret")
try {
val value = client
.accessSecretVersion(SecretVersionName.of(project, secret, version))
.getPayload
.getData
.toStringUtf8
Some(secret -> value)
} catch {
case e: Throwable => {
logger.error(s"google error ($secret): ${e.getMessage}")
client.shutdown()
client.awaitTermination(30, TimeUnit.SECONDS)
client.close()
None
}
}
}.toMap
How i can tranform the code to return the secrets values in a Map[String, String], and close the client connection (something like that):
def apply(
project: String,
version: String = SecretVersion,
credentials: CredentialsProvider = defaultProvider
): Try[SecretService] = Try {
.
.
.
some code with the solution
.
.
.
client.shutdown()
client.awaitTermination(30, TimeUnit.SECONDS)
client.close()
secrets
}
Thanks in advance ;)

If I understand correctly, you'll need to close the client after you get the map
val client = SecretManagerServiceClient.create(settings)
// TODO: Set the type of the secrets parameter
val result = (secrets: ???) => { secrets.flatMap
...
}.toMap
}
client.shutdown()
client.awaitTermination(30, TimeUnit.SECONDS)
client.close()
return result
Here, the result is a function handle, that you're returning
However, keep in mind that the client may already have been closed, if you ever reach the exception block
Alternatively, you can
Pass a client to the function, not the project details. Close the client after your function call
Remove the Try from the function definition, and actually handle the exceptions or close the client upon a Success, then return Some[Map]

Related

Facing issue testing akka http cache

I am using Akka HTTP cache to cache my result. But i am facing issue to test it.
class GoogleAnalyticsController #Inject()(cache: Cache[String, HttpResponse],
googleAnalyticsApi: GoogleAnalyticsTrait,
googleAnalyticsHelper: GoogleAnalyticsHelper)
(implicit system: ActorSystem, materializer: ActorMaterializer) {
def routes: Route =
post {
pathPrefix("pageviews") {
path("clients" / Segment) { accountsClientId =>
entity(as[GoogleAnalyticsMetricsRequest]) { googleAnalyticsMetricsRequest =>
val googleAnalyticsMetricsKey = "key"
complete(
cache.getOrLoad(googleAnalyticsMetricsKey, _ => getGoogleAnalyticsMetricsData(accountsClientId, googleAnalyticsMetricsRequest))
)
}
}
}
}
private def getGoogleAnalyticsMetricsData(accountsClientId: String,
request: GoogleAnalyticsMetricsRequest) = {
val payload = generate(request)
val response = googleAnalyticsApi.googleAnalyticsMetricResponseHandler(accountsClientId, payload) // response from another microservice
googleAnalyticsHelper.googleAnalyticsMetricResponseHandler(
googleAnalyticsMetricsRequest.metricName, response)
}
}
class GoogleAnalyticsHelper extends LoggingHelper {
def googleAnalyticsMetricResponseHandler(metricName: String, response: Either[Throwable, Long]): Future[HttpResponse] =
response.fold({ error =>
logger.error(s"An exception has occurred while getting $metricName from behavior service and error is ${error.getMessage}")
Marshal(FailureResponse(error.getMessage)).to[HttpResponse].map(httpResponse => httpResponse.copy(status = StatusCodes.InternalServerError))
}, value =>
Marshal(MetricResponse(metricName, value)).to[HttpResponse].map(httpResponse => httpResponse.copy(status = StatusCodes.OK))
)
}
Test case: Sharing only the relevant part
"get success metric response for " + pageviews + " metric of given accounts client id" in { fixture =>
import fixture._
val metricResponse = MetricResponse(pageviews, 1)
val eventualHttpResponse = Marshal(metricResponse).to[HttpResponse].map(httpResponse => httpResponse.copy(status = StatusCodes.OK))
when(cache.getOrLoad(anyString, any[String => Future[HttpResponse]].apply)).thenReturn(eventualHttpResponse)
when(googleAnalyticsApi.getDataFromGoogleAnalytics(accountsClientId, generate(GoogleAnalyticsRequest(startDate, endDate, pageviews))))
.thenReturn(ApiResult[Long](Some("1"), None))
when(googleAnalyticsHelper.googleAnalyticsMetricResponseHandler(pageviews, Right(1))).thenReturn(eventualHttpResponse)
Post(s"/pageviews/clients/$accountsClientId").withEntity(requestEntity) ~>
googleAnalyticsController.routes ~> check {
status shouldEqual StatusCodes.OK
responseAs[String] shouldEqual generate(metricResponse)
}
}
By doing this, I am best to test if the cache has the key but not able to test if cache misses the hit. In code coverage, it misses following highlighted part
cache.getOrLoad(googleAnalyticsMetricsKey, _ =>
getGoogleAnalyticsMetricsData(accountsClientId,
googleAnalyticsMetricsRequest))
If there is a design issue, please feel free to guide me on how can I make my design testable.
Thanks in advance.
I think you don't need to mock the cache. You should create an actual object for cache instead of mocked one.
What you have done is, you have mocked the cache, in this case, the highlighted part will be not called as you are providing the mocked value. In the following stubbing, whenever cache.getOrLoad is found, eventualHttpResponse is returned:
when(cache.getOrLoad(anyString, any[String => Future[HttpResponse]].apply)).thenReturn(eventualHttpResponse)
and hence the function getGoogleAnalyticsMetricsData(accountsClientId, googleAnalyticsMetricsRequest) is never called.

Using if-guard in for-comprehension to check for existence

My am making 3 database queries, each return a Future. I am trying to use for comprehension to resolve the Futures but it seems I am not using if correctly in for
Each query depends on result of previous one. I look for a token, if found, I look for user and it found, I update the user. Each database query returns a Future[Option]] and I thought I could considitionally perform the next query depending on whether the previous one returns Some or None. I am using isDefined for this. But when I ran the code for an invalid token, I got error [NoSuchElementException: None.get] for code userOption:Option[User]<-userRepo.findUser(tokenOption.get.loginInfo); if tokenOption.isDefined
def verifyUser(token:String) = Action.async {
implicit request => {
val result:Future[Result] = for{
//generator 1 - get token from database
tokenOption:Option[UserToken] <- userTokenRepo.find(UserTokenKey(UUID.fromString(token)))
//generator2. found token, look for corresponding user to which the token belongs
userOption:Option[User] <- userRepo.findUser(tokenOption.get.loginInfo); if tokenOption.isDefined
//generator 3. found user and token. Update profile
modifiedUser:Option[User] <- confirmSignupforUser(userOption.get); if userOption.isDefined
} yield
{ //check if we have user and token and modified user here. If any is missing, return error else success
if(tokenOption.isDefined && userOption.isDefined && modifiedUser.isDefined)
Redirect("http://localhost:9000/home"+";signup=success")//TODOM - pick from config
else
if(tokenOption.isEmpty)
Redirect("http://localhost:9000/home"+";signup=error")//TODOM - pick from config
else if(userOption.isEmpty)
Redirect("http://localhost:9000/home"+";signup=error")//TODOM - pick from config
else if(modifiedUser.isEmpty)
Redirect("http://localhost:9000/home"+";signup=error")//TODOM - pick from config
else //this shouldn't happen. Unexpected
Redirect("http://localhost:9000/home"+";signup=error")//TODOM - pick from config
}
result
}
}
TL;DR
Consider using OptionT
https://typelevel.org/cats/datatypes/optiont.html
Have a look at my toned down implementation:
from https://scastie.scala-lang.org/hsXXtRAFRrGpMO1Jl1Li7A
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Await.result
import scala.concurrent.duration._
import scala.language.postfixOps
type UserToken = String
type User = String
def fromToken(token: String): Future[Option[UserToken]] = Future.successful(None)
def findUser(userToken: UserToken): Future[Option[User]] = Future.successful(None)
def modify(user: User): Future[Option[User]] = Future.successful(None)
def verifyUser(token: String) = {
val result = for {
tokenOption: Option[UserToken] <- fromToken(token) //generator 1 - get token from database
userOption: Option[User] <- findUser(tokenOption.get);
if tokenOption.isDefined //generator2. found token, look for corresponding user to which the token belongs
modifiedUser: Option[User] <- modify(userOption.get);
if userOption.isDefined //generator 3. found user and token. Update profile
} yield { //check if we have user and token and modified user here. If any is missing, return error else success
if (tokenOption.isDefined && userOption.isDefined && modifiedUser.isDefined)
println("happy")
else
println("sad")
}
result
}
result(verifyUser("hello"), 1 second)
I used the following compile time flags, the last one is important:
scalacOptions ++= Seq(
"-deprecation",
"-encoding", "UTF-8",
"-feature",
"-unchecked",
"-Xprint:typer"
)
Let's focus on this line of the compile output:
(((tokenOption: Option[Playground.this.UserToken]) => Playground.this.findUser(tokenOption.get).
withFilter(((check$ifrefutable$2: Option[Playground.this.User]) => (check$ifrefutable$2: Option[Playground.this.User] #unchecked) match {
case (userOption # (_: Option[Playground.this.User])) => true
case _ => false
...
You can see that the tokenOption.get is invoked before the withFilter. These gets are the source of the exception you get
The, almost complete output of the compile is:
[[syntax trees at end of typer]] // main.scala
....
import scala.concurrent.Future;
import scala.concurrent.ExecutionContext.Implicits.global;
import scala.concurrent.Await.result;
import scala.concurrent.duration._;
import scala.language.postfixOps;
type UserToken = String;
type User = String;
def fromToken(token: String): scala.concurrent.Future[Option[Playground.this.UserToken]] = scala.concurrent.Future.successful[None.type](scala.None);
def findUser(userToken: Playground.this.UserToken): scala.concurrent.Future[Option[Playground.this.User]] = scala.concurrent.Future.successful[None.type](scala.None);
def modify(user: Playground.this.User): scala.concurrent.Future[Option[Playground.this.User]] = scala.concurrent.Future.successful[None.type](scala.None);
def verifyUser(token: String): scala.concurrent.Future[Unit] = {
val result: scala.concurrent.Future[Unit] = Playground.this.fromToken(token).withFilter(((check$ifrefutable$1: Option[Playground.this.UserToken]) => (check$ifrefutable$1: Option[Playground.this.UserToken] #unchecked) match {
case (tokenOption # (_: Option[Playground.this.UserToken])) => true
case _ => false
}))(scala.concurrent.ExecutionContext.Implicits.global).flatMap[Unit](((tokenOption: Option[Playground.this.UserToken]) => Playground.this.findUser(tokenOption.get).withFilter(((check$ifrefutable$2: Option[Playground.this.User]) => (check$ifrefutable$2: Option[Playground.this.User] #unchecked) match {
case (userOption # (_: Option[Playground.this.User])) => true
case _ => false
}))(scala.concurrent.ExecutionContext.Implicits.global).withFilter(((userOption: Option[Playground.this.User]) => tokenOption.isDefined))(scala.concurrent.ExecutionContext.Implicits.global).flatMap[Unit](((userOption: Option[Playground.this.User]) => Playground.this.modify(userOption.get).withFilter(((check$ifrefutable$3: Option[Playground.this.User]) => (check$ifrefutable$3: Option[Playground.this.User] #unchecked) match {
case (modifiedUser # (_: Option[Playground.this.User])) => true
case _ => false
}))(scala.concurrent.ExecutionContext.Implicits.global).withFilter(((modifiedUser: Option[Playground.this.User]) => userOption.isDefined))(scala.concurrent.ExecutionContext.Implicits.global).map[Unit](((modifiedUser: Option[Playground.this.User]) => if (tokenOption.isDefined.&&(userOption.isDefined).&&(modifiedUser.isDefined))
scala.Predef.println("happy")
else
scala.Predef.println("sad")))(scala.concurrent.ExecutionContext.Implicits.global)))(scala.concurrent.ExecutionContext.Implicits.global)))(scala.concurrent.ExecutionContext.Implicits.global);
result
};
scala.Predef.locally[Unit]({
val $t: Unit = scala.concurrent.Await.result[Unit](Playground.this.verifyUser("hello"), scala.concurrent.duration.`package`.DurationInt(1).second);
Playground.this.instrumentationMap$.update(com.olegych.scastie.api.Position.apply(1199, 1236), com.olegych.scastie.api.runtime.Runtime.render[Unit]($t)((ClassTag.Unit: scala.reflect.ClassTag[Unit])));
$t
})
};
object Main extends scala.AnyRef {
def <init>(): Main.type = {
Main.super.<init>();
()
};
private[this] val playground: Playground = new Playground();
<stable> <accessor> def playground: Playground = Main.this.playground;
def main(args: Array[String]): Unit = scala.Predef.println(com.olegych.scastie.api.runtime.Runtime.write(Main.this.playground.instrumentations$))
}
}
I am not sure why you are surprised you are getting and error for None.get with invalid token: if token is invalid, tokenOption is None, so, the next statement tokenOption.get will fail with exactly this error.
You want the "guard" executed before the statement you want to short circuit, not after it:
for {
foo <- bar if foo.isDefined
baz <- foo.get
} yield baz
But this would fail in the end anyway, because there would be nothing to yield (this trick works with Options or Lists etc., but Future.withFilter will end up failing if predicate is not satisfied, there is no other alternative).
The general rule to avoid this kind of errors is never use .get on an Option (or on a Try). Also, never use .head on a List, .apply on a Map, etc.
Here is one (almost) idiomatic way to write what you want:
case object Error extends RuntimeException("")
userTokenRepo
.find(UserTokenKey(UUID.fromString(token)))
.map { _.getOrElse(throw Error)
.flatMap { userRepo.find(_.loginInfo) }
.map { _.getOrElse(throw Error) }
.flatMap(confirmSignupForUser)
.map { _.getOrElse(throw Error) }
.map { _ => "success") }
.recover { case Error => "error" }
.map { result => Redirect(s"http://localhost:9000/home;signup=$result" }
Note, I said this was "almost" idiomatic, because throwing exceptions in scala is frowned upon. A purist would object to it, and suggest using something like a Try . or a biased Either instead, or to make use of a third party library, like cats or scalaz, that provide additional tools for working with Futures of Option (namely, OptionT).
But I would not recommend getting into that right now. You should get comfortable enough with basic "vanilla" scala before starting with that advanced stuff to avoid ending up with something completely incomprehensible.
You could also write this differently, in a completely idiomatic way (without using exceptions), with something like this:
userTokenRepo.find(UserTokenKey(UUID.fromString(token)))
.flatMap {
case Some(token) => userRepo.find(token.loginInfo)
case None => Future.successful(None)
}.flatMap {
case Some(user) => confirmSignupForUser(user)
case None => Future.successful(None)
}.map {
case Some(_) => "success"
case None => "error"
}.map { result =>
Redirect(s"http://localhost:9000/home;signup=$result"
}
This is more "pure", but a little more repetitive, so my personal preference is the first variant.
Finally, you could do away with my Error thingy, and just handle the NoSuchElement exception directly. This is going to be the shortest, but kinda icky even to my taste (what if some downstream code throws this exception because of a bug?):
userTokenRepo
.find(UserTokenKey(UUID.fromString(token)))
.flatMap { userRepo.find(_.get.loginInfo) }
.flatMap(confirmSignupForUser(_.get))
.map { _.get }
.map { _ => "success") }
.recover { case _: NoSuchElementException => "error" }
.map { result =>
Redirect(s"http://localhost:9000/home;signup=$result"
}
I really don't recommend the last version though, despite it being the shortest, and arguably, the most readable one (you can even rewrite it with a for-comprehension to look even nicer). Using Option.get is commonly considered "code smell", and is almost never a good thing to do.
Motivated by How to best handle Future.filter predicate is not satisfied type errors
I rewrote like the following. While the code works, I am curious to know if I am doing it the right way (functional!). Does it look fine?
def verifyUser(token:String) = Action.async {
implicit request => {
println("verifyUser action called with token: " + token) //TODOM - add proper handling and response
val result:Future[Result] = for{tokenOption:Option[UserToken] <- userTokenRepo.find(UserTokenKey(UUID.fromString(token))) //generator 1 - get token from database
userOption:Option[User] <- if (tokenOption.isDefined) userRepo.findUser(tokenOption.get.loginInfo) else Future.successful(None) //generator2. found token, look for corresponding user to which the token belongs
modifiedUser:Option[User] <- if (userOption.isDefined) confirmSignupforUser(userOption.get) else Future.successful(None) //generator 3. found user and token. Update profile
deletedToken:Option[UserTokenKey] <- if(modifiedUser.isDefined) userTokenRepo.remove(UserTokenKey(UUID.fromString(token))) else Future.successful(None)
}
yield { //check if we have user and token and modified user here. If any is missing, return error else success
println("db query results tokenOption: "+tokenOption+", userOption: "+userOption+" : modifiedUserOption: "+modifiedUser+", deletedToken: "+deletedToken)
if(tokenOption.isDefined && userOption.isDefined && modifiedUser.isDefined && deletedToken.isDefined)
Redirect("http://localhost:9000/home"+";signup=success")//TODOM - pick from config
else
if(tokenOption.isEmpty)
Redirect("http://localhost:9000/home"+";signup=error")//TODOM - pick from config
else if(userOption.isEmpty)
Redirect("http://localhost:9000/home"+";signup=error")//TODOM - pick from config
else if(modifiedUser.isEmpty)
Redirect("http://localhost:9000/home"+";signup=error")//TODOM - pick from config
else //this shouldn't happen. Unexpected
Redirect("http://localhost:9000/home"+";signup=error")//TODOM - pick from config
}
result
}
}

Scala: getting error as "Expression of type Unit doesn't confirm to expected type..."

I am setting the connection timeout for Client like
def newClient(host: String): Client = asyncHttpClient match {
case true => {
import org.sonatype.spice.jersey.client.ahc.AhcHttpClient
AhcHttpClient.create()
}
case _ => {
import com.sun.jersey.api.client.Client
val client: Client = Client.create()
client.setConnectTimeout(5000)
//Or client.setConnectTimeout(Int.box(5000))
}
}
and getting the error message
Expression of type Unit doesn't confirm to expected type Client
Could someone help to understand the problem and properly pass the integer value?
Your method is declared to return a value of type Client. The return type of client.setConnectionTimeout is Unit, not Client, so you can't return that from your method. Instead you should return client:
def newClient(host: String): Client = asyncHttpClient match {
case true => {
import org.sonatype.spice.jersey.client.ahc.AhcHttpClient
AhcHttpClient.create()
}
case _ => {
import com.sun.jersey.api.client.Client
val client: Client = Client.create()
client.setConnectTimeout(5000)
client
}
}

Play framework 2.3.x OAuth & Actors "Cannot find HTTP Request Header here"

I'm using the example code from Play's ScalaOAuth documentation to authenticate to the twitter API. My full code is here:
object Twitter extends Controller {
val KEY = ConsumerKey("redacted")
val TWITTER = OAuth(ServiceInfo(
"https://api.twitter.com/oauth/request_token",
"https://api.twitter.com/oauth/access_token",
"https://api.twitter.com/oauth/authorize", KEY),
true)
def authenticate = Action { request =>
request.getQueryString("oauth_verifier").map { verifier =>
val tokenPair = sessionTokenPair(request).get
// We got the verifier; now get the access token, store it and back to index
TWITTER.retrieveAccessToken(tokenPair, verifier) match {
case Right(t) => {
// We received the authorized tokens in the OAuth object - store it before we proceed
Redirect(routes.Application.timeline).withSession("token" -> t.token, "secret" -> t.secret)
}
case Left(e) => throw e
}
}.getOrElse(
TWITTER.retrieveRequestToken("http://localhost:9000/auth") match {
case Right(t) => {
// We received the unauthorized tokens in the OAuth object - store it before we proceed
Redirect(TWITTER.redirectUrl(t.token)).withSession("token" -> t.token, "secret" -> t.secret)
}
case Left(e) => throw e
})
}
def sessionTokenPair(implicit request: RequestHeader): Option[RequestToken] = {
for {
token <- request.session.get("token")
secret <- request.session.get("secret")
} yield {
RequestToken(token, secret)
}
}
}
I spin up an actor that I want to use to search the Twitter API. I also use an edited version of their controller in that documentation to do this:
class TwitterParse extends Actor {
private var buildURL: String = _
private val urlBoilerPlate = "https://api.twitter.com/1.1/users/search.json?q="
private val pageCount = "&page=1&count=3"
def receive = {
case rawName: (Long, String) => {
buildURL = urlBoilerPlate + rawName._2 + pageCount
Twitter.sessionTokenPair match {
case Some(credentials) => {
WS.url(buildURL)
.sign(OAuthCalculator(Twitter.KEY, credentials))
.get
.map(result => println(result.json))
}
}
}
When I compile I get: "Cannot find any HTTP Request Header here" on the Twitter.sessionTokenPair call. This probably has to do with the fact that in their docs they include a
Action.async { implicit request =>
before calling sessionTokenPair. How would I fix this so it works in the Actor?
Thanks in advance

Scala Exception Handling - Advice Needed

I have the following piece of code that makes a request to a HTTP service to fetch a resource. It could be that the service is not available or the resource is not available. So I need to handle such error scenarios. The server actually throws an exception that I have to handle in my Scala client. Here is what I have done so far:
def getData(resource: String, requestParam: Option[Seq[(String, String)]]): Try[Elem] = {
Try {
val wc = WebClient.create(address(scheme, host, port) + "/" + resource, un, pw, null)
// Load the trust store to the HTTPConduit
if ("https" == scheme) {
val conduit = WebClient.getConfig(wc).getConduit.asInstanceOf[HTTPConduit]
loadTrustStore(conduit)
}
requestParam match {
case Some(reqParams) => reqParams.foreach((param: (String, String)) => {
wc.query(param._1, param._2)
})
case None => wc
}
XML.loadString("""<?xml version="1.0" encoding="UTF-8"?>""" + wc.get(classOf[String]))
}
}
Now, the caller of this method looks like this:
def prepareDomainObject(): MyDomainObject = {
getData(resource, params) match {
case Success(elem) => getDomainFromElem(elem)
case Failure(_) => ?? What do I do here? I actually want to get the message in the Throwable and return another type. How do I get around this?
}
How do I handle my match condition such that I return a different type in case of an error?
You have two options. Either have prepareDomainObject return a Try as well:
def prepareDomainObject(): Try[MyDomainObject] = {
getData(resource, params) map getDomainFromElem
}
or have it throw an exception on error:
def prepareDomainObject(): MyDomainObject = {
getDomainFromElem(getData(resource, params).get)
}