i have following code in two files:
operation1.py
class App_Name():
def __init__(self):
self.type_option = ""
def Intro(self):
self.type_option = input("Chose one option: ")
...
start = App_Name()
start.Intro()
menu.py
from operation1 import App_name
aP = App_Name()
if aP.type_option == 1:
do smth
elif aP.type.type_option == 2:
do smth 2
If i type 1, i expect to run commands from first if condition. When i try to print App_name.type_option it seems to be empty. How can i pass value of aP.type_option to menu.py?
start and aP are 2 different instances. Since type_option is bound to one instance, start.type_option contains the input (as string), whereas aP.type_option contains the empty string you set in the __init__ method.
Remove start instanciation in your operation1 module or you'll be prompted when importing it!
Then fix menu.py as follows:
from operation1 import App_name
aP = App_Name()
aP.Intro()
if aP.type_option == "1":
do smth
elif aP.type.type_option == "2":
do smth 2
(note that comparison must be done against strings because Python 3 input returns strings, doesn't evaluate literals like python 2 input did)
Related
Apologies for a length post. I have been trying to beat my head around reading about mock, MagicMock, and all the time getting confused. Hence, decided to write this post.
I know several questions, and pages have been written on this. But, still not able to wrap my head around this.
My Setup:
All the test code, and the 2 module files come under one "folder" mymodule
my_module_1.py file contains
class MyOuterClass(object):
MyInnerClass(object):
attribute1: str
attribute2: str
attribute3: str
def get(self) -> MyInnerClass:
'''
pseudocode
1. a call to AWS's service is made
2. the output from call in step 1 is used to set attributes of this InnerClass
3. return innerclass instance
'''
I use the OuterClass in another file(my_module_2.py), to set some values and return a string as follows:
class MyModule2():
def get_foo(self, some_boolean_predicate):
if some_boolean_predicate:
temp = my_module_1.OuterClass().get()
statement = f'''
WITH (
BAR (
FIELD_1 = '{temp.attribute1}',
FIELD_2 = '{temp.attribute2}',
FIELD_3 = '{temp.attribute3}'
)
)
'''
else:
statement = ''
return statement
I want to write the unit tests for the file my_module_2.py file, and test the function get_foo
How I am writing the tests(or planning on)
a test file by name test_my_module2.py
I started with creating a pytest.fixture for the MyOuterClass's get function as follows since I will be reusing this piece of info again in my other tests
#pytest.fixture
def mock_get(mocker: MockerFixture) -> MagicMock:
return mocker.patch.object(MyOuterClass, 'get')
Finally,
Then I proceeded to use this fixture in my test as follows:
from unittest import mock
from unittest.mock import MagicMock, Mock, patch, PropertyMock
import pytest
from pytest_mock import MockerFixture
from my_module.my_module_1 import myOuterClass
def test_should_get_from_inner_class(self, mock_get):
# mock call to get are made
output = mock_get.get
#update the values for the InnerClass's attributes here
output.attribute1.side_effect = 'attr1'
output.attribute2.side_effect = 'attr2'
output.attribute3.side_effect = 'attr3'
mock_output_str = '''
WITH (
BAR (
FIELD_1 = 'attr1',
FIELD_2 = 'attr2',
FIELD_3 = 'attr3'
)
)
'''
module2Obj = MyModule2()
response = module2Obj.get_foo(some_boolean_predicate=True)
# the following assertion passes
assert mock_get.get.called_once()
# I would like match `response to that with mock_output_str instance above
assert response == mock_output_str
But, the assertion as you might have guessed failed, and I know I am comparing completely different types, since I see
errors such as
FAILED [100%]
WITH (
BAR (
FIELD1 = '<MagicMock name='get().attr1' id='4937943120'>',
FIELD3 = '<MagicMock name='get().attr2' id='4937962976'>',
FIELD3 = '<MagicMock name='get().attr3' id='4937982928'>'
)
)
Thank you for being patient with me till here, i know its a really lengthy post, but stuck on this for a few days, ended up creating a post here.
How do i get to validate the mock's value with the mock_output_str?
yess! the hint was in the #gold_cy's answer. I was only calling my mock and never setting its values
this is what my test case ended up looking
mock_obj = OuterClass.InnerClass()
mock_obj.attribute1='some-1'
mock_obj.attribute2='some-2'
mock_obj.attribute3='some-3'
mock_get.return_value = mock_obj
once my mock was setup properly, then the validation became easy! Thank you!
As the title implies, I'm running a Scrapy spider and storing results in MongoDB. Everything is running smoothly, except when I re-run the spider, it adds everything again, and I don't want the duplicates. My pipelines.py file looks like this:
import logging
import pymongo
from pymongo import MongoClient
from scrapy.conf import settings
from scrapy import log
class MongoPipeline(object):
collection_name = 'openings'
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
#classmethod
def from_crawler(cls, crawler):
## pull in information from settings.py
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE')
)
def open_spider(self, spider):
## initializing spider
## opening db connection
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def close_spider(self, spider):
## clean up when spider is closed
self.client.close()
def process_item(self, item, spider):
## how to handle each post
if self.db.openings.find({' quote_text': item['quote_text']}) == True:
pass
else:
self.db[self.collection_name].insert(dict(item))
logging.debug("Post added to MongoDB")
return item
My spider looks like this:
import scrapy
from ..items import QuotesItem
class QuoteSpider(scrapy.Spider):
name = 'quote'
allowed_domains = ['quotes.toscrape.com']
start_urls = ['http://quotes.toscrape.com/']
def parse(self, response):
items = QuotesItem()
quotes = response.xpath('//*[#class="quote"]')
for quote in quotes:
author = quote.xpath('.//*[#class="author"]//text()').extract_first()
quote_text = quote.xpath('.//span[#class="text"]//text()').extract_first()
items['author'] = author
items['quote_text'] = quote_text
yield items
The current syntax is obviously wrong, but is there a slight fix to the for loop to make to fix it? Should I be running this loop in the spider instead? I was also looking at upsert but was having trouble understanding how to use that effectively. Any help would be great.
Looks like you have a leading space here: self.db.openings.find({' quote_text': item['quote_text']}). I suppose it should just be 'quote_text'?
You should use is True instead of == True. This is the reason it adds everything again.
I would suggest to use findOne instead of find, will be more efficient.
Using upsert instead is indeed a good idea but the logic will be slightly different: you will update the data if the item already exists, and insert it when it doesn't exists (instead of not doing anything if the item already exists). The syntax should look something like this: self.db[self.collection_name].update({'quote_text': quote_text}, dict(item),upsert=True)
steps :
check if the collection is empty else : write in collection
if not empty and item exist : pass
else (collection not empty + item dosen't exist) : write in collection
code:
def process_item(self, item, spider):
## how to handle each post
# empty
if len(list(self.db[self.collection_name].find({}))) == 0 :
self.db[self.collection_name].insert_one(dict(item))
# not empty
elif item in list(self.db[self.collection_name].find(item,{"_id":0})) :
print("item exist")
pass
else:
print("new item")
#print("here is item",item)
self.db[self.collection_name].insert_one(dict(item))
logging.debug("Post added to MongoDB")
return item
The Click package allows a range of values to be selected from a list using the click.Choice method.
In my case the values are relatively long strings, so using:
choice_names = [u'Vulnerable BMC (IPMI)', u'IoT Vulnerability', u'SMBv1', u'BadHTTPStatus', u'Compromised']
#click.option('--category', prompt='\nPlease enter the category of incident.\n\n -- Options:\n{}\n\n'.format(
format_choices(choice_names)), type=click.Choice(choice_names))
will list the values as:
-> Vulnerable BMC (IPMI)
-> IoT Vulnerability
-> SMBv1
-> BadHTTPStatus
-> Compromised
This requires the user to enter the full string, which is inconvenient. Does Click provide a functionality to select a value using only a numeric identifier? So, the above options could be listed as:
-> Vulnerable BMC (IPMI) [1]
-> IoT Vulnerability [2]
-> SMBv1 [3]
-> BadHTTPStatus [4]
-> Compromised [5]
and to select the first option, the user would need to enter 1. This could be possible by defining a custom validation function, but I couldn't find any existing functionality offered by Click.
I came up with this:
class ChoiceOption(click.Option):
def __init__(self, param_decls=None, **attrs):
click.Option.__init__(self, param_decls, **attrs)
if not isinstance(self.type, click.Choice):
raise Exception('ChoiceOption type arg must be click.Choice')
if self.prompt:
prompt_text = '{}:\n{}\n'.format(
self.prompt,
'\n'.join(f'{idx: >4}: {c}' for idx, c in enumerate(self.type.choices, start=1))
)
self.prompt = prompt_text
def process_prompt_value(self, ctx, value, prompt_type):
if value is not None:
index = prompt_type(value, self, ctx)
return self.type.choices[index - 1]
def prompt_for_value(self, ctx):
# Calculate the default before prompting anything to be stable.
default = self.get_default(ctx)
prompt_type = click.IntRange(min=1, max=len(self.type.choices))
return click.prompt(
self.prompt, default=default, type=prompt_type,
hide_input=self.hide_input, show_choices=False,
confirmation_prompt=self.confirmation_prompt,
value_proc=lambda x: self.process_prompt_value(ctx, x, prompt_type))
#click.command()
#click.option('--hash-type', prompt='Hash', type=click.Choice(['MD5', 'SHA1'], case_sensitive=False), cls=ChoiceOption)
def cli(**kwargs):
print(kwargs)
Result:
> cli --help
Usage: cli [OPTIONS]
Options:
--hash-type [MD5|SHA1]
--help Show this message and exit.
> cli --hash-type MD5
{'hash_type': 'MD5'}
> cli
Hash:
1: MD5
2: SHA1
: 4
Error: 4 is not in the valid range of 1 to 2.
Hash:
1: MD5
2: SHA1
: 2
{'hash_type': 'SHA1'}
Edit May 25, 2020:
I recently came across questionary and integrated it with click
import click
import questionary
class QuestionaryOption(click.Option):
def __init__(self, param_decls=None, **attrs):
click.Option.__init__(self, param_decls, **attrs)
if not isinstance(self.type, click.Choice):
raise Exception('ChoiceOption type arg must be click.Choice')
def prompt_for_value(self, ctx):
val = questionary.select(self.prompt, choices=self.type.choices).unsafe_ask()
return val
#click.command()
#click.option('--hash-type', prompt='Hash', type=click.Choice(['MD5', 'SHA1'], case_sensitive=False), cls=QuestionaryOption)
def cli(**kwargs):
print(kwargs)
if __name__ == "__main__":
cli()
Since Click does not seem to provide a functionality of this kind, this custom validation function fulfills the purpose:
def validate_choice(ctx, param, value):
# Check if the passed value is an integer.
try:
index = int(value) - 1
# Return the value at the given index.
try:
return choice_names[index]
# If the index does not exist.
except IndexError:
click.echo('Please select a valid index.')
# If the value is of a different type, for example, String.
except (TypeError, ValueError):
# Return the value if it exists in the list of choices.
if value in choice_names:
return value
else:
click.echo('Please select a valid value from the choices {}.'.format(choice_names))
# Prompt the user for an input.
value = click.prompt(param.prompt)
return validate_choice(ctx, param, value)
#click.option('--category', prompt='\nPlease enter the category.\n\n -- Options:\n{}\n\n'.format(choice_names),
help='Category of the incident', callback=validate_category)
This allows a user to select a choice either by entering the choice name or by entering the index value. In case an invalid value is entered, the user is prompted again for an input.
is there a way to use the #incremental plugin like described att Pytest: how to skip the rest of tests in the class if one has failed? mixed with #pytest.mark.parametrize like below:
#pytest.mark.incremental
Class TestClass:
#pytest.mark.parametrize("input", data)
def test_preprocess_check(self,input):
# prerequisite for test
#pytest.mark.parametrize("input",data)
def test_process_check(self,input):
# test only if test_preprocess_check succeed
The problem i encountered is, at the first fail of test_preprocess_check with a given input of my data set, the following test_preprocess_check and test_process_check are labeled "xfail".
The behaviour i expect will be, at each new "input" of my parametrized data set, the test will act in an incremental fashion.
ex: data = [0,1,2]
if only test_preprocess_check(0) failed:
i got the following report:
1 failed, 5 xfailed
but i expect the report:
1 failed, 1 xfailed, 4 passed
Thanks
After some experiments i found a way to generalize the #incremental to works with parametrize annotation. Simply rewrite the _previousfailed argument to make it unique for each input. The argument _genid was excactly the need.
I added a #pytest.mark.incrementalparam to achieve this.
Code become:
def pytest_runtest_setup(item):
previousfailed_attr = getattr(item, "_genid",None)
if previousfailed_attr is not None:
previousfailed = getattr(item.parent, previousfailed_attr, None)
if previousfailed is not None:
pytest.xfail("previous test failed (%s)" %previousfailed.name)
previousfailed = getattr(item.parent, "_previousfailed", None)
if previousfailed is not None:
pytest.xfail("previous test failed (%s)" %previousfailed.name)
def pytest_runtest_makereport(item, call):
if "incrementalparam" in item.keywords:
if call.excinfo is not None:
previousfailed_attr = item._genid
setattr(item.parent,previousfailed_attr, item)
if "incremental" in item.keywords:
if call.excinfo is not None:
parent = item.parent
parent._previousfailed = item
It's interesting to mention that's it can't be used without parametrize cause parametrize annotation creates automatically _genid variable.
Hope this can helps others than me.
I have a few .csv files I want to use for the same data in Gatling. Each of these files has a certain number of ID's that I want to be accessed fairly equally. I don't want to put them all in the same file because the .csv files are generated from SQL queries and, while I may have a lot of IDs in one file, I only have a few in another. What's important to me is that I have a random sample from each of my files and a way to specify the distribution.
I found an example of how to do this but I'm having trouble applying it in my case. Here is the code I have so far. I try to both 1) print out the value from the feeder in the session and 2) try to use the value from the feeder in a get request. Both attempts fail with various errors which I detail below:
import scala.concurrent.duration._
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
import util.Random
class FeederTest extends Simulation {
//headers...
val userCreds = csv("user_creds.csv")
val sample1 = csv("sample1.csv")
val sample2 = csv("sample2.csv")
def randFeed(): String = {
val foo = Random.nextInt(2)
var retval = ""
if (foo == 0) retval = "file1"
if (foo == 1) retval = "file2"
return retval
}
val scn = scenario("feeder test")
.repeat(1) {
feed(userCreds)
.doSwitch(randFeed)(
"file1" -> feed(sample1),
"file2" -> feed(sample2)
)
.exec(http("request - login")
.post("<URL>")
.headers(headers_login)
.formParam("email", "${username}")
.formParam("password", "<not telling>"))
.exec(session => {
println(session)
println(session("first").as[String])
session})
.exec(http("goto_url")
.get("<my url>/${first}"))
}
setUp(scn.inject(atOnceUsers(1))).protocols(httpProtocol)
}
This is the error I get when I attempt to printout the feeder value in the session (as in the above code using session(<value>).as[String]):
[ERROR] [03/13/2015 10:22:38.221] [GatlingSystem-akka.actor.default-dispatcher-8] [akka://GatlingSystem/user/sessionHook-2] key not found: first
java.util.NoSuchElementException: key not found: first
at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:59)
at io.gatling.core.session.SessionAttribute.as(Session.scala:40)
at FeederTest$$anonfun$2.apply(feeder_test.scala:81)
at FeederTest$$anonfun$2.apply(feeder_test.scala:79)
at io.gatling.core.action.SessionHook.executeOrFail(SessionHook.scala:35)
at io.gatling.core.action.Failable$class.execute(Actions.scala:71)
at io.gatling.core.action.SessionHook.execute(SessionHook.scala:28)
at io.gatling.core.action.Action$$anonfun$receive$1.applyOrElse(Actions.scala:29)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at io.gatling.core.akka.BaseActor.aroundReceive(BaseActor.scala:22)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I have also tried just using the EL expression ${first} in the session. All this does is print out the string ${first}. Similarly in the last .get line I get an error saying "No attribute named 'first' is defined.
The CSV files I'm currently using just have two columns of first and last in them like so:
sample1.csv:
first, last
george, bush
bill, clinton
barak, obama
sample2.csv:
first, last
super, man
aqua, man
bat, man
I'm using Gatling 2.1.4.
doSwitch takes an Expression[Any], which is a type alias for Session => Validation[Any]. Gatling has an implicit conversion that let you pass a static value instead, see documentation.
Which is exactly what you do. Even if randFeed is a def, it still doesn't return a function, but a String.
As you want randFeed to be called every time a virtual user pass through this step, you have to wrap the randFeed inside a function, even if you don't use the Session input parameter.
doSwitch(_ => randFeed)
Then, your randFeed is both ugly (no offense) and inefficient (Random is synchronized):
import scala.concurrent.forkjoin.ThreadLocalRandom
def randFeed(): String =
ThreadLocalRandom.current().nextInt(2) match {
case 0 => "file1"
case 1 => "file2"
}
I've never used feeders, but from the documentation, I see two possible problems:
randFeed returns either "foo" or "bar" but you're mapping with "file1" and "file2", so is anything being loaded? (The documentation says "If no switch is selected, the switch is bypassed.")
The examples for feeders I see don't show accessing fed data with session(varname) but rather "${varname}".