I was trying out this Python code from a training website in IPython:
from bs4 import BeautifulSoup
import requests
url = raw_input("www.google.com")
r = requests.get("http://" +url)
data = r.text
soup = BeautifulSoup(data)
for link in soup.find_all('a'):
print(link.get('href'))
and found that it ran fine on the first try. I've now tried simply restarting the kernel, opening a new notebook, and generally returning the settings to how they were when I first ran the program with no luck. Why might IPython be failing to run the code and giving no response at all (as though I haven't clicked anything)?
Seems like raw_input is not supported by IPython. So it's probably just hanging there. If you change:
url = raw_input("www.google.com")
to
url = "www.google.com"
it should work.
Related
I have a problem on my nvim LSP.
according to mason plugin the pyright is downloaded and I on other IDE the lsp works great.
when i open python file with neovim it didn't recognized import and basic functions.
someone can handle this situation?
maybe it's because Lspinfo tell me that it didn't get the python root directory?import don't auto complete pythonpyright is downloaded via masonLSPInfo description
on other languages it works so don't really understand the problem
Same as me, at the moment I solved like this (it happens in some languages for me too):
local status_ok, mason = pcall(require, "mason")
if not status_ok then
return
end
local status_masonlsp_ok, mason_lspconfig = pcall(require, "mason-lspconfig")
if not status_masonlsp_ok then
return
end
local servers = {
"sumneko_lua",
"rust_analyzer",
"tailwindcss",
"bashls",
"pyright",
-- "csharp_ls",
"html",
"omnisharp",
"gopls",
}
mason.setup(settings)
mason_lspconfig.setup({
ensure_installed = servers,
automatic_installation = true,
})
So when I open nvim they are installed automatically.
Is there a way to start mitmproxy v.7.0.2 programmatically in the background?
ProxyConfig and ProxyServer have been removed since version 7.0.0, and the code below isn't working.
from mitmproxy.options import Options
from mitmproxy.proxy.config import ProxyConfig
from mitmproxy.proxy.server import ProxyServer
from mitmproxy.tools.dump import DumpMaster
import threading
import asyncio
import time
class Addon(object):
def __init__(self):
self.num = 1
def request(self, flow):
flow.request.headers["count"] = str(self.num)
def response(self, flow):
self.num = self.num + 1
flow.response.headers["count"] = str(self.num)
print(self.num)
# see source mitmproxy/master.py for details
def loop_in_thread(loop, m):
asyncio.set_event_loop(loop) # This is the key.
m.run_loop(loop.run_forever)
if __name__ == "__main__":
options = Options(listen_host='0.0.0.0', listen_port=8080, http2=True)
m = DumpMaster(options, with_termlog=False, with_dumper=False)
config = ProxyConfig(options)
m.server = ProxyServer(config)
m.addons.add(Addon())
# run mitmproxy in backgroud, especially integrated with other server
loop = asyncio.get_event_loop()
t = threading.Thread( target=loop_in_thread, args=(loop,m) )
t.start()
# Other servers, such as a web server, might be started then.
time.sleep(20)
print('going to shutdown mitmproxy')
m.shutdown()
from BigSully's gist
You can put your Addon class into your_script.py and then run mitmdump -s your_script.py. mitmdump comes without the console interface and can run in the background.
We (mitmproxy devs) officially don't support manual instantiation from Python anymore because that creates a massive amount of support burden for us. If you have some Python experience you can probably find your way around.
What if my addon has additional dependencies?
Approach 1: pip install mitmproxy is still perfectly supported and gets you the same functionality as the standalone binaries. Bonus tip: You can run venv/bin/mitmproxy or venv/Scripts/mitmproxy.exe to invoke mitmproxy in your virtualenv without having your virtualenv activated.
Approach 2: You can install mitmproxy with pipx and then run pipx inject mitmproxy <your dependency name>. See https://docs.mitmproxy.org/stable/overview-installation/#installation-from-the-python-package-index-pypi for details.
How can I debug mitmproxy itself?
If you are debugging from the command line (be it print statements or pdb), the easiest approach is to run mitmdump instead of mitmproxy, which provides the same functionality minus the console interface. Alternatively, you can use PyCharm's remote debug functionality, which also works while the console interface is active (https://github.com/mitmproxy/mitmproxy/blob/main/examples/contrib/remote-debug.py).
This example below should work fine with mitmproxy v7
from mitmproxy.tools import main
from mitmproxy.tools.dump import DumpMaster
options = main.options.Options(listen_host='0.0.0.0', listen_port=8080)
m = DumpMaster(options=options)
# the rest is same in the previous versions
from mitmproxy.addons.proxyserver import Proxyserver
from mitmproxy.options import Options
from mitmproxy.tools.dump import DumpMaster
options = Options(listen_host='127.0.0.1', listen_port=8080, http2=True)
m = DumpMaster(options, with_termlog=True, with_dumper=False)
m.server = Proxyserver()
m.addons.add(
// addons here
)
m.run()
Hi, I think that should do it
Even after a successful execution of my tests in DeviceFarm, I get an empty screenshots report. I have kept my code as simple as below -
from appium import webdriver
import time
import unittest
import os
class MyAndroidTest(unittest.TestCase):
def setUp(self):
caps = {}
self.driver = webdriver.Remote("http://127.0.0.1:4723/wd/hub", caps)
def test1(self):
self.driver.get('http://docs.aws.amazon.com/devicefarm/latest/developerguide/welcome.html')
time.sleep(5)
screenshot_folder = os.getenv('SCREENSHOT_PATH', '/tmp')
self.driver.save_screenshot(screenshot_folder + 'screen1.png')
time.sleep(5)
def tearDown(self):
self.driver.quit()
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(MyAndroidTest)
unittest.TextTestRunner(verbosity=2).run(suite)
I tested on a single device pool -
How can I make this work ?
TIA.
Missing a slash (/) before the filename (i.e., screen1.png). Line 15 should be as below -
self.driver.save_screenshot(screenshot_folder + '/screen1.png')
Though I'm not sure exactly how to write this to a file in Device Farm here are the appium docs for the screenshot endpoint and a python example.
https://github.com/appium/appium/blob/master/docs/en/commands/session/screenshot.md
It gets a base 64 encoded string which then we would just need to save it somewhere like the appium screenshot dir the other answers mentioned. Otherwise we could also save it in the /tmp dir and then export it using the custom artifacts feature.
Let me know if that link helps.
James
When you perform the same analysis in a notebook on different data files, may be handy to graphically select a data file.
In my python scripts I usually implement a QT dialog that returns the file-name of the selected file:
from PySide import QtCore, QtGui
def gui_fname(dir=None):
"""Select a file via a dialog and return the file name.
"""
if dir is None: dir ='./'
fname = QtGui.QFileDialog.getOpenFileName(None, "Select data file...",
dir, filter="All files (*);; SM Files (*.sm)")
return fname[0]
However, running this function from an notebook
full_fname = gui_fname()
causes the kernel to die (and restart):
Interestingly, puttying this 3 command in 3 separate cells works
%matplotlib qt
full_fname = gui_fname()
%matplotlib inline
but when I put those commands in one single cell the kernel dies again.
This prevents to create a function like gui_fname_ipynb() that transparently allows selecting a file with a GUI.
For convenience, I created a notebook illustrating the problem:
Open/select file with GUI (Qt Dialog)
Any suggestion on how to execute a dialog for file selection from within an IPython Notebook?
Using Anaconda 5.0.0 on windows (Python 3.6.2, IPython 6.1.0), the following two options are both working for me.
OPTION 1: Entirely in a Jupyter notebook:
CELL 1:
%gui qt
from PyQt5.QtWidgets import QFileDialog
def gui_fname(dir=None):
"""Select a file via a dialog and return the file name."""
if dir is None: dir ='./'
fname = QFileDialog.getOpenFileName(None, "Select data file...",
dir, filter="All files (*);; SM Files (*.sm)")
return fname[0]
CELL 2:
gui_fname()
This is working for me but it seems a bit...fragile. If I combine these two things into the same cell, it crashes. Or if I omit the %gui qt, it crashes. If I "restart kernel and run all cells", it doesn't work. So I kinda like this other option...
MORE RELIABLE OPTION: Separate script that opens dialog box in a new process
(Based on mkrog code here.)
PUT THE FOLLOWING IN A SEPARATE PYTHON SCRIPT CALLED blah.py:
from sys import executable, argv
from subprocess import check_output
from PyQt5.QtWidgets import QFileDialog, QApplication
def gui_fname(directory='./'):
"""Open a file dialog, starting in the given directory, and return
the chosen filename"""
# run this exact file in a separate process, and grab the result
file = check_output([executable, __file__, directory])
return file.strip()
if __name__ == "__main__":
directory = argv[1]
app = QApplication([directory])
fname = QFileDialog.getOpenFileName(None, "Select a file...",
directory, filter="All files (*)")
print(fname[0])
...AND IN YOUR JUPYTER NOTEBOOK
import blah
blah.gui_fname()
I have a universal code where it does its job without any problem. Here is my sugestion:
try:
from tkinter import Tk
from tkFileDialog import askopenfilenames
except:
from tkinter import Tk
from tkinter import filedialog
Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
filenames = filedialog.askopenfilenames() # show an "Open" dialog box and return the path to the selected file
print (filenames)
hope it can be useful
This behaviour was a bug in IPython:
https://github.com/ipython/ipython/issues/4997
that was fixed here:
https://github.com/ipython/ipython/pull/5077
The function to open a gui dialog should work on current master and on the oncoming 2.0 release.
To date, the last 1.x version (1.2.1) does not include a backport of the fix.
EDIT: The example code still crashes IPython 2.x, see this issue.
I'm writing a simple screen scraping script using python 2.7 with Eclipse PyDev. When running or debugging from within Eclipse everything works fine. However, when I run my program from the command line the server always returns a Response 500 error code. I've tried running the script and the compiled versions from the command line but get the same result -- Response 500. I've also tried some arbitrary things like adding a delay, repeated attempts, etc. but I do not know what Eclipse is doing that is different than python ran the command line.
First, where's a good place to start digging if I encounter something like this again?
Second, any ideas on how to get this working from the command line?
Code snippet below for reference
from requests import Request, Session
content_type = 'application/x-www-form-urlencoded'
headers2 = {"User-Agent" : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)',
"Content-Type" : content_type,
"Referer" : url
}
url = loginPage
payload = {"email" : username, "password" : password}
req = Request ('POST', url, data=payload, headers=headers2)
prepped = req.prepare()
s = Session()
resp = s.send(prepped)
print resp # Response 200 (good) from both within Eclipse and from cmd
resp = s.get(targetPage)
print resp # Response 200 (good) from Eclipse, Response 500 (generic web error) from cmd
s.get (logOutPage)
s.close()
Got an answer from somebody. Thanks go to user Justinsaccount from reddit.
First, I was using batch files to save typing and not directly using the command line.
Secondly, when printing out the parameters from inside the program and then comparing the eclipse version versus the .bat version, the .bat version was short a few characters which was the give away.
One of the parameters was a url that had a space character: http://somewhere.com/some page.
In strict URL, this turns into: http://somewhere.com/some%20page
When run from the command line http://somewhere.com/some%20page
works just fine. However, in a batch file the % needed to be escaped so what I got was: http://somewhere.com/some0page
which is why the server through an error -- that page didn't exist. What I needed to do was escape the % character: http://somewhere.com/some%%20page. After that change things worked just fine.