When I call
cmdline.execute("scrapy crawl website".split())
print "Hello World"
it stops the script after cmdline.execute, and doesn't run the rest of the script and print "Hello World". How do I fix this?
By taking a look at the execute function in Scrapy's cmdline.py, you'll see the final line is:
sys.exit(cmd.exitcode)
There really is no way around this sys.exit call if you call the execute function directly, at least not without changing it. Monkey-patching is one option, albeit not a good one! A better option is to avoid calling the execute function entirely, and instead use the custom function below:
from twisted.internet import reactor
from scrapy import log, signals
from scrapy.crawler import Crawler as ScrapyCrawler
from scrapy.settings import Settings
from scrapy.xlib.pydispatch import dispatcher
from scrapy.utils.project import get_project_settings
def scrapy_crawl(name):
def stop_reactor():
reactor.stop()
dispatcher.connect(stop_reactor, signal=signals.spider_closed)
scrapy_settings = get_project_settings()
crawler = ScrapyCrawler(scrapy_settings)
crawler.configure()
spider = crawler.spiders.create(name)
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()
And you can call it like this:
scrapy_crawl("your_crawler_name")
I have just tried the following code, and it works for me:
import os
os.system("scrapy crawl website")
print("Hello World")
One can run subprocess.call. For example on Windows with powershell:
import subprocess
subprocess.call([r'C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe',
'-ExecutionPolicy',
'Unrestricted',
'scrapy crawl website -o items.json -t json'])
Related
routes.py:
from server import app
#app.route("/index")
def index():
return "Hello World"
server.py:
from bottle import Bottle, run
app = Bottle()
#app.route("/index",method=["POST"])
run.py:
from server import app
from bottle import Bottle, run
if __name__ == "__main__":
run(app, host='localhost', port=8080, debug=True)
After I split them into different files, I get the 404 error. I didn't find much information about this on google.
You're not importing routes anywhere, so routes.py is never executed.
If I were you, I would merge routes.py and server.py. But if you insist on separating them, then something like this might work:
run.py:
from bottle import Bottle, run
from server import app
import routes
...
I'm relatively confused here, and upon trying to research for an answer, I'm not seeming to find anything that makes any sense to me. I have created a discord bot with 5 cogs, and in each one I import discord, os, and from discord.ext import commands In various other cogs I import other modules such as random as the case may be, but those are the three common ones.
The problem is that in every module, import discord is grayed out (PyCharm IDE), suggesting that is never used. Despite this, my bot runs perfectly. I don't seem to be able to use things like the wait_for() command, I presume it is because it is in the discord module? Am I not setting things up correctly to use this?
I will post the initial startup module and a small snippet of another module, rather than list module. If you need more information, let me know.
initial startup:
import discord
import os
from discord.ext import commands
token = open("token.txt", "r").read()
client = commands.Bot(command_prefix = '!')
#client.command()
async def load(ctx, extension):
client.load_extension("cogs." + extension)
#client.command()
async def unload(ctx, extension):
client.unload_extension("cogs." + extension)
for filename in os.listdir("./cogs"):
if filename.endswith('.py'):
client.load_extension("cogs." + filename[:-3])
client.run(token)
another module:
import discord
from discord.ext import commands
import os
import json
from pathlib import Path
class Sheet(commands.Cog):
def __init__(self, client):
self.client = client
#commands.command()
#commands.dm_only()
async def viewchar(self, ctx):
#Snipped code here to make it shorter.
pass
#viewchar.error
async def stats_error(self, ctx, error):
if isinstance(error, commands.PrivateMessageOnly):
await ctx.send("You're an idiot, now everyone knows. Why would you want to display your character sheet "
"in a public room? PM me with the command.")
else:
raise error
def setup(client):
client.add_cog(Sheet(client))
That just means that your code doesn't directly reference the discord module anywhere. You're getting everything through the commands module.
You can remove the import discord from your code without breaking anything, because the code that relies on it will still import and use it behind the scenes.
I had a bear of a time figuring this out, and it was really bugging me, so I thought I'd post this here in case anyone hit the same problem...
(and the answer is so dang simple it hurts :-)
The Problem
The core of the issue is that sometimes, not always, when dealing with fixtures in PyTest that return objects, when you use those fixtures in a test in PyCharm, you don't get autocomplete hints. If you have objects with large numbers of methods you want to reference while writing a test, this can add a lot of overhead and inconvenience to the test writing process.
Here's a simple example to illustrate the issue:
Let's say I've got a class "event_manager" that lives in:
location.game.events
Let's further say that in my conftest.py file (PyTest standard thing for the unfamiliar), I've got a fixture that returns an instance of that class:
from location.game.events import event_manager
...
#pytest.fixture(scope="module")
def event_mgr():
"""Creates a new instance of event generate for use in tests"""
return event_manager()
I've had issues sometimes, (but not always - I can't quite figure out why) with classes like this where autocomplete will not work properly in the test code where I use the fixture, e.g.
def test_tc10657(self, evt_mgr):
"""Generates a Regmod and expects filemod to be searchable on server"""
evt_mgr.(This does not offer autocomplete hints when you type ".")
So the answer is actually quite simple, once you review type hinting in PyCharm:
http://www.jetbrains.com/help/pycharm/2016.1/type-hinting-in-pycharm.html
Here's how to fix the above test code so that autocomplete works properly:
from location.game.events import event_manager
...
def test_tc10657(self, evt_mgr: event_manager):
"""Generates a Regmod and expects filemod to be searchable on server"""
evt_mgr.(This DOES offer hints when you type "." Yay!)
Notice how I explicitly type the fixture as an input parameter of type event_manager.
Also if you add a docstring to a function and specify the type of the the parameters, you will get the code completion for those parameters.
For example using pytest and Selenium:
# The remote webdriver seems to be the base class for the other webdrivers
from selenium.webdriver.remote.webdriver import WebDriver
def test_url(url, browser_driver):
"""
This method is used to see if IBM is in the URL title
:param WebDriver browser_driver: The browser's driver
:param str url: the URL to test
"""
browser_driver.get(url)
assert "IBM" in browser_driver.title
Here's my conftest.py file as well
import pytest
from selenium import webdriver
# Method to handle the command line arguments for pytest
def pytest_addoption(parser):
parser.addoption("--driver", action="store", default="chrome", help="Type in browser type")
parser.addoption("--url", action="store", default='https://www.ibm.com', help="url")
#pytest.fixture(scope='module', autouse=True)
def browser_driver(request):
browser = request.config.getoption("--driver").lower()
# yield the driver to the specified browser
if browser == "chrome":
driver = webdriver.Chrome(executable_path='/path/to/chromedriver')
else:
raise Exception("No driver for browser " + browser)
yield driver
driver.quit()
#pytest.fixture(scope="module")
def url(request):
return request.config.getoption("--url")
Tested using Python 2.7 and PyCharm 2017.1. The docstring format is reStructuredText and the "Analyze Python code in docstrings" checkbox is checked in settings.
I'm new to Python and trying to understand classes. Not sure the following error is coming from the use of my IDE, which is Spyder, or if it is intended behaviour.
I define a class message in the file C:\mydir\class_def.py. Here is what the file contains:
class message:
def __init__(self,msg1,msg2):
self.msg1 = msg1
self.msg2 = msg2
I have another script were I want to execute code, called execute.py. In this script I import the class and make an instance of the class object. Here is the code from the script execute.py:
import os
os.chdir('C:\mydir')
from class_def import message
message_obj = message('Hello','world')
So far no problems!
Then I edit class_def.py to the following:
class message:
def __init__(self,msg1):
self.msg1 = msg1
and edit execute.py to match the new class, so removing one input tomessage:
import os
os.chdir('C:\mydir')
from class_def import message
message_obj = message('Hello')
and I get the following error:
TypeError: __init__() takes exactly 3 arguments (2 given)
It seems like Python keeps the old version of class_def.py and does not import the new one, even though it is saved.
Is this normal behaviour or is Spyder doing something funny?
If you have a .pyc file such as class_def.pyc, delete it.
I'd remove all .pyc files in your working directory and then try again. If that doesn't work, maybe you're not using the module you think you are? To be certain try something like:
import myModule
print myModule.__file__ #This will give you the path to the .pyc file your program loaded
#or
import myModule
import os
print os.path.dirname(myModule.__file__)
try those out so you can be certain that you're actually using the file you're modifying. Hope that helps!
I want to duplicate ipython notebook capability in Emacs / Pymacs; and I need some direction for a simple code that can 1) send python / "magics" code to a ipython kernel 2) receive the display output, as a string. I found this comment by minrk, the "ipython kernel" example did not work, it gave "ImportError: No module named zmq.blockingkernelmanager".
I had better luck with one his other pointers, finally I landed at ipython-1.1.0/IPython/kernel/inprocess/tests/test_kernel.py, I ripped out a minimal part, and coded an Emacs extension called pytexipy-notebook. It's on Github
goo.gl/kQzJW1
If anyone knows of better examples, such as connecting to an existing (out of process), I'd like to hear about these.
Thanks in advance,
Here is a sample for ipython 3.0.
from IPython.testing.globalipapp import get_ipython
from IPython.utils.io import capture_output
ip = get_ipython()
def run_cell(cmd):
with capture_output() as io:
res = ip.run_cell(content)
print 'suc', res.success
print 'res', res.result
res_out = io.stdout
print 'res out', res_out
content = "print (111+222)"
run_cell(content)
content = "alsdkjflajksf"
run_cell(content)
I will soon update
https://github.com/burakbayramli/emacs-ipython