fixture 'pylon_config_missing_usageplan' not found while running pytest-bdd - pytest

#scenario('../features/config.feature', 'Loading a valid config')
def test_config():
pass
#given("I have provided a valid pylon config",target_fixture="pylon_config")
def pylon_config():
input_data = {
}
return input_data
#when("The configuration is loaded")
def excute_pylon_config(pylon_config):
createFilterPattern(pylon_config)
#then("It should be enriched with the expected FilterPatterns")
def no_error_message(pylon_config):
test_data1= {
}
test_data_2 = {
}
result = pylon_config
#scenario('../features/config.feature', 'Missing usagePlan section')
def test_missing_usageplan():
pass
#given("I have provided a pylon config with a missing key",target_fixture="pylon_config_missing_usageplan")
def pylon_config_missing_usageplan():
input_data = {
'metricFilters': {
'defaults': {
'xyz': []
}
}
}
return input_data
#when("The configuration is loaded")
def excute_pylon_config_missing_usageplan(pylon_config_missing_usageplan):
try:
createFilterPattern(pylon_config_missing_usageplan)
except KeyError:
pass
#then("I should receive an exception")
def error_message_pylon_config_missing_usageplan(pylon_config_missing_usageplan):
print(pylon_config_missing_usageplan)
I have written multiple test case with specifying target_fixture in both #given scenario.
While running the test case it's throwing an error with
fixture 'pylon_config_missing_usageplan' not found
available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pylon_config, pytestbdd_given_I have provided a pylon config with a missing key, pytestbdd_given_I have provided a valid pylon config, pytestbdd_given_trace, pytestbdd_then_I should receive an exception, pytestbdd_then_It should be enriched with the expected FilterPatterns, pytestbdd_then_trace, pytestbdd_when_The configuration is loaded, pytestbdd_when_trace, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
use 'pytest --fixtures [testpath]' for help on them.
Can anyone help me over here?

The issue is, that the step When The configuration is loaded has two different implementations in code:
#when("The configuration is loaded")
def excute_pylon_config(pylon_config):
createFilterPattern(pylon_config)
#when("The configuration is loaded")
def excute_pylon_config_missing_usageplan(pylon_config_missing_usageplan):
try:
createFilterPattern(pylon_config_missing_usageplan)
except KeyError:
pass
The function excute_pylon_config_missing_usageplan overrides the step implementation of excute_pylon_config and therefore if you try to load the pylon config in scenario Loading a valid config, pytest-bdd actually tries to execute the function excute_pylon_config_missing_usageplan, which is expecting the fixture pylon_config_missing_usageplan (which is not available in this scenario ...)
Solutions
Have two distinct steps for loading a valid/invalid config, e.g. When The configuration is loaded and When The invalid configuration is loaded (I would recommend this approach, since it is simpler and easier to read than solution 2)
Add a variable to the step for loading the configuration which contains the type of the configuration
Example of variable in configuration loading step:
#when(parsers.parse("The {config_type} configuration is loaded"))
def excute_pylon_config(request, config_type):
if config_type == 'valid':
# Retrieve fixture dynamically by name
pylon_config = request.getfixturevalue('pylon_config')
createFilterPattern(pylon_config)
else:
# Retrieve fixture dynamically by name
pylon_config_missing_usageplan = request.getfixturevalue('pylon_config_missing_usageplan')
try:
createFilterPattern(pylon_config_missing_usageplan)
except KeyError:
pass

Related

Scala IO wait during map external call

I will start mentioning I am very new to Scala but I have now to maintain a legacy code where some new feature are being tried to be include.
I have the following code:
Where a list is coming as a parameter where a new output needs to be processed. However it seems like code is not waiting for the response to the external service when processing.
def historyBet(jackpotListUser : List[JackpotBetHistory])(implicit MC: AppMarkerContext) : List[LegacyJackpotHistoryResponse] =
for {
bet <- jackpotListUser
prize = jackpotIntegratorService.findJackpotByJackpotHumanId(bet.jackpotHumanId) match {
case Some(jackpot : JackpotResponse) =>
...
extra code extracting price from jackpot : JackpotResponse
...
extra code generating result with prize
} yield result
How can I do a call to jackpotIntegratorService.findJackpotByJackpotHumanId to execute at that time. instead of returning something that F[Option....?
def findJackpotByJackpotHumanId(
jackpotHumanId: JackpotHumanId
)(implicit MC: AppMarkerContext): F[Option[JackpotResponse]] =
jackpotIntegratorRepo.findJackpotByJackpotHumanId(jackpotHumanId)
where it is finally implemented as:
override def findJackpotByJackpotHumanId(
jackpotHumanId: JackpotHumanId
)(implicit mc: AppMarkerContext): IO[Option[JackpotResponse]] =
... code calling an API which return the IO.
Thanks!
I thought I could do IO.await somewhere... but not sure where or how...
because in the "historyBet" function I got a F[] when it was an IO... so what is the syntax to be able to wait for the response and the continue?
Extra Comment:
The real issue we notice is that the method call is starting (the logs shows part of it) but the caller with in the maps continues too.
prize = jackpotIntegratorService.findJackpotByJackpotHumanId
this part of the code continues even when prize, which we want the final object JackpotResponse, not the IO or F.
So, if your method needs to call an IO then it must return an IO unless you unsafeRunSync them... but, as the name suggest, you should not do that.
So the return type is now: IO[List[LegacyJackpotHistoryResponse]
And can be implemented like this:
def historyBet(jackpotListUser: List[JackpotBetHistory])(implicit MC: AppMarkerContext): IO[List[LegacyJackpotHistoryResponse]] =
jackpotListUser.traverse { bet =>
jackpotIntegratorService.findJackpotByJackpotHumanId(bet.jackpotHumanId).map {
case Some(jackpot) =>
// ...
case None =>
// ...
}
}

How can I see how many documents were written & handle errors correctly?

From the documentation I can see that I should be able to use WriteResult.ok, WriteResult.code and WriteResult.n in order to understand errors and the number of updated documents but this isn't working. Here is a sample of what I'm doing (using reactiveMongoDB/Play JSON Collection Plugin):
def updateOne(collName: String, id: BSONObjectID, q: Option[String] = None) = Action.async(parse.json) { implicit request: Request[JsValue] =>
val doc = request.body.as[JsObject]
val idQueryJso = Json.obj("_id" -> id)
val query = q match {
case Some(_) => idQueryJso.deepMerge(Json.parse(q.get).as[JsObject])
case None => idQueryJso
}
mongoRepo.update(collName)(query, doc, manyBool = false).map(result => writeResultStatus(result))
}
def writeResultStatus(writeResult: WriteResult): Result = {
// NOT WORKING
if(writeResult.ok) {
if(writeResult.n > 0) Accepted else NotModified
} else BadRequest
}
Can I give an alternative approach here? You said:
"in order to understand errors and the number of updated documents but this isn't working"
Why you don't use the logging functionality that Play provides? The general idea is that:
You set the logging level (e.g., only warning and errors, or errors, etc.).
You could use the log to output a message in any case, either something is ok, or it is not.
Play saves the logs of your application while it is running.
You the maintainer/developer could look into the logs to check if there is any errors.
This approach open a great possibility in the future: you could save the logs into a third-party service and put monitoring functionalities on the top of it.
Now if we look at the documentation here, you see about different log levels, and how to use the logger.

NullPointerException in Flink custom SourceFunction

I wanted to create a SourceFunction which reads a http stream.
I used ScalaJ which does what I want (it splits the incoming text by \n-s).
Obviously the code works outside Flink, but I get a NullPointerExcetion every time I start it as a Flink job (sometimes immediately sometimes after 1-2 seconds after it transmitted 1-2 elements). It kind of looks like the Http object has some problems.
import org.apache.flink.streaming.api.functions.source.SourceFunction
import scala.io.Source.fromInputStream
import scalaj.http._
class HttpSource(url: String) extends SourceFunction[String] {
#volatile var isRunning = true
override def cancel(): Unit = isRunning = false
override def run(ctx: SourceFunction.SourceContext[String]): Unit =
httpStream(ctx.collect)
private def httpStream(f: String => Unit) = {
val request = Http(url)
request
.execute { inputStream =>
fromInputStream(inputStream)
.getLines()
.takeWhile(_ => isRunning)
.foreach(f)
}
}
}
Here's the exception I usually get:
(Sometimes it's a bit different, for example I tried to make the request value transient, then it's already null when it tries to refer to request)
Caused by: java.lang.NullPointerException
at java.io.Reader.<init>(Reader.java:78)
at java.io.InputStreamReader.<init>(InputStreamReader.java:129)
at scala.io.BufferedSource.reader(BufferedSource.scala:24)
at scala.io.BufferedSource.bufferedReader(BufferedSource.scala:25)
at scala.io.BufferedSource.scala$io$BufferedSource$$charReader$lzycompute(BufferedSource.scala:35)
at scala.io.BufferedSource.scala$io$BufferedSource$$charReader(BufferedSource.scala:33)
at scala.io.BufferedSource.scala$io$BufferedSource$$decachedReader(BufferedSource.scala:62)
at scala.io.BufferedSource$BufferedLineIterator.<init>(BufferedSource.scala:67)
at scala.io.BufferedSource.getLines(BufferedSource.scala:86)
at flinkextension.HttpSource$$anonfun$httpStream$1.apply(HttpSource.scala:21)
at flinkextension.HttpSource$$anonfun$httpStream$1.apply(HttpSource.scala:19)
at scalaj.http.HttpRequest$$anonfun$execute$1.apply(Http.scala:323)
at scalaj.http.HttpRequest$$anonfun$execute$1.apply(Http.scala:323)
at scalaj.http.HttpRequest$$anonfun$toResponse$3.apply(Http.scala:388)
at scalaj.http.HttpRequest$$anonfun$toResponse$3.apply(Http.scala:380)
at scala.Option.getOrElse(Option.scala:121)
at scalaj.http.HttpRequest.toResponse(Http.scala:380)
at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:360)
at scalaj.http.HttpRequest.exec(Http.scala:335)
at scalaj.http.HttpRequest.execute(Http.scala:323)
at flinkextension.HttpSource.httpStream(HttpSource.scala:19)
at flinkextension.HttpSource.run(HttpSource.scala:14)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Everything else seems to be working fine, when I don't use a http request, but something else like file read with the same InputStream type, just a plain while loop with strings or even when I use single http requests, which aren't streaming.
I feel like I'm missing some theoretical background, maybe flink does something in the background which destroys the Http object or the InputStream, but I didn't find anything in the documentation.
UPDATE #1:
If I put a null check into the lambda, the job usually exits immediately, sometimes processes a few elements, sometimes timeouts after hanging for a minute. Here's this version of the httpStream function:
private def httpStream(f: String => Unit) = {
val request = Http(url)
request
.execute { inputStream =>
if (inputStream == null) println("null inputstream")
else {
println("not null inputstream")
fromInputStream(inputStream)
.getLines()
.takeWhile(_ => isRunning)
.foreach(f)
}
}
}
UPDATE #2:
The code actually works in distributed mode and with StreamExecutionEnvironment.createLocalEnvironment()
I only experience the issue if I use start-local.sh and submit the jar to it.

Jetty works for HTTP but not HTTPS

I am trying to create a jetty consumer. I am able to get it successfully running using the endpoint uri:
jetty:http://0.0.0.0:8080
However, when I modify the endpoint uri for https:
jetty:https://0.0.0.0:8443
The page times out trying to load. This seems odd because the camel documentation states it should function right out of the box.
I have since loaded a signed SSL into java's default keystore, with my attempted implementation to load it below:http://camel.apache.org/jetty.html
I have a basic Jetty instance using the akka-camel library with akka and scala. ex:
class RestActor extends Actor with Consumer {
val ksp: KeyStoreParameters = new KeyStoreParameters();
ksp.setPassword("...");
val kmp: KeyManagersParameters = new KeyManagersParameters();
kmp.setKeyStore(ksp);
val scp: SSLContextParameters = new SSLContextParameters();
scp.setKeyManagers(kmp);
val jettyComponent: JettyHttpComponent = CamelExtension(context.system).context.getComponent("jetty", classOf[JettyHttpComponent])
jettyComponent.setSslContextParameters(scp);
def endpointUri = "jetty:https://0.0.0.0:8443/"
def receive = {
case msg: CamelMessage => {
...
}
...
}
...
}
This resulted in some progress, because the page does not timeout anymore, but instead gives a "The connection was interrupted" error. I am not sure where to go from here because camel is not throwing any Exceptions, but rather failing silently somewhere (apparently).
Does anybody know what would cause this behavior?
When using java's "keytool" I did not specify an output file. It didn't throw back an error, so it probably went somewhere. I created a new keystore and explicitly imported my crt into the keyfile. I then explicitly added the filepath to that keystore I created, and everything works now!
If I had to speculate, it is possible things failed silently because I was adding the certs to jetty's general bank of certs to use if eligible, instead of explicitly binding it as the SSL for the endpoint.
class RestActor extends Actor with Consumer {
val ksp: KeyStoreParameters = new KeyStoreParameters();
ksp.setResource("/path/to/keystore");
ksp.setPassword("...");
val kmp: KeyManagersParameters = new KeyManagersParameters();
kmp.setKeyStore(ksp);
val scp: SSLContextParameters = new SSLContextParameters();
scp.setKeyManagers(kmp);
val jettyComponent: JettyHttpComponent = CamelExtension(context.system).context.getComponent("jetty", classOf[JettyHttpComponent])
jettyComponent.setSslContextParameters(scp);
def endpointUri = "jetty:https://0.0.0.0:8443/"
def receive = {
case msg: CamelMessage => {
...
}
...
}
...
}
Hopefully somebody in the future can find use for this code as a template in implementing Jetty over SSL with akka-camel (surprisingly no examples seem to exist)

How can I access a resource when running an SBT runTask?

I've got an XML file that I need to read from the classpath in order to load some test data for my project with DBUnit when running a custom runTask in SBT.
The XML file is located in /src/main/resources and is copied properly to the /target/scala_2.8.1/classes during the build, but I get a MalformedURLException when trying to access it.
The weird thing is, I can access the file when this data loading functionality was part of my Scala specs unit tests.
Any ideas?
In my case the problem was that I used getClass.getResourceAsStream() in early initialiser. Had to specify the class explicitly with Class.forName() to solve it: Class.forName(<class name>).getResourceAsStream("/data.xml")
If the error is saying that the URL is malformed, it's probably true.
Here's a code I use to grab file from resource during test:
def copyFileFromResource(source: String, dest: File) {
val in = getClass.getResourceAsStream(source)
val reader = new java.io.BufferedReader(new java.io.InputStreamReader(in))
val out = new java.io.PrintWriter(new java.io.FileWriter(dest))
var line: String = null
line = reader.readLine
while (line != null) {
out.println(line)
line = reader.readLine
}
in.close
out.flush
}