Right now I have been trying to access yahoo with python and am I am not sure why I can't seem to login.
My intended flow is
go to yahoo -> go to login -> enter username -> press submit button -> enter password -> press submit button.
Please let me know where I have made a mistake and why not code doesn't seem to work. Any alternatives to login into yahoo that are not selenium-based would be appreciated and still use python.
"""Example app to login to Yahoo using the StatefulBrowser class."""
from __future__ import print_function
import argparse
import mechanicalsoup
browser = mechanicalsoup.StatefulBrowser(
soup_config={'features': 'lxml'},
raise_on_404=True,
user_agent='MyBot/0.1: mysite.example.com/bot_info',
)
# Uncomment for a more verbose output:
browser.set_verbose(2)
browser.session.cookies.keys()
browser.open("https://login.yahoo.com/config/login?.src=fpctx&.intl=ca&.lang=en-CA&.done=https%3A%2F%2Fca.yahoo.com")
form1 = browser.select_form(nr=0)
browser['username'] = 'beta#gmail.com'
response = browser.submit_selected()
print(response.content)
browser.select_form(nr=0)
browser['passwd'] = 'badPass'
response = browser.submit_selected()
print(response)
page = browser.get_current_page()
A quick look at the login page source shows that it uses JavaScript quite extensively. It seems very likely that the form submission is handled by JavaScript, though I can't point to an exact line of code that proves this incontrovertibly.
Since MechanicalSoup does not support JavaScript, you may need to find an alternate tool that does, such as Selenium. See this FAQ for more information.
Related
Hi i know this is probably not the place to ask this but i m stumped at the moment as i cant seem to find any reference or docs relating to working with Roblox. I mean sure they have an auth route etc but nothing detailed. I want to login user using username and give them roblox based on different actions they take on the site like completing surveys etc. Can anyone please give me links to some resources that would come in handy for the particular purpose. Thank you.
Roblox does not support any OAuth systems, but you still can use HttpService:GetAsync() function to get strings/data from web site(if the page in website display that text), the way to keep data that you recieved from url(web page) safe is to store script with HttpService:GetAsync() function in server side(example: RobloxScriptService). You need to allow http requests in your GameSettings -> Security in roblox studio. Script example:
local HttpService = game:GetService("HttpService")
local stringg = HttpService:GetAsync("https://pastebin.com/raw/k7S6Ln9R")
print(string)
--Should outpud data written ot the web page, you can use any web page to store data even your own
The only two things that left is to make your web server rewrite the page, or just use some databases at your web site by placing their url into loadstring() function.
Now you just need to parse the string given by url to use it's data.
The pastebin url that i wrote into loadstring() just an example, you can write whatever you wan, but again you need to parse the data that you got from url, or just convert the string into type of text like on the page, and then just check is they written at url/webpage. Example:
local writtenpass = game.Players["anyplayer"].PlayerGui.TestGui.Frame.PasswordTextBox.text
local writtenlogin = game.Players["anyplayer"].PlayerGui.TestGui.Frame.LoginTextBox.text
local HttpService = game:GetService("HttpService")
local response = HttpService:GetAsync("https://pastebin.com/raw/k7S6Ln9R")
local istrue = string.find(response, "{ login = ".. writtenlogin .." pass = ".. writtenpass .." }")
print(istrue)
if istrue == 1 then
print("exist!")
--whatewer actions if login and pass exist
end
You can wiew the page here https://pastebin.com/raw/k7S6Ln9R
Well that a lot of damage!
If it helps mark me
I am trying to deploy a simple user bot on Google App Engine Flexible Environment running python3.7, but I get the following error. Can anyone help me with suggestions of solving this?
"File "/env/lib/python3.7/site-packages/telethon/client/auth.py", line 20, in phone: typing.Callable[[], str] = lambda: input('Please enter your phone (or bot token): '), EOFError: EOF when reading a line"
Thank you for your time
Telethon (obviously) requires you to login and make a normal user session on your account, which natively requires you to input your number when asked and enter the code received but since Google App engine doesn't allow input as #Sashidhar mentioned, depending on your userbot implementation, you can try using the userbot.session method for authentication, it can be generated locally and placed in the Google App Engine.
I try to use python on app engine to call telethon functions, upon deploy the app. I receive the Internal Server Error from browser.
I am seeing you are success on this road, would you mind shine a light on this to help me starting up, such as how to configue the app.yaml, main.py, requirements.txt , .... or any proper arrangement of librarys to make this work.
much much thanks in advance.
following is my main.py which raised server internal error on brows, the problem does not happen if this first telethon related line is remarked:
client = TelegramClient(phone, api_id, api_hash)
main.py
--------
from flask import Flask, request
import datetime
import time
import asyncio
from telethon import TelegramClient, events, sync
app = Flask(__name__)
api_id = xxxxxxxx
api_hash = 'xxxxxxxxxxxxxxxxxxxxxxxx'
phone = '+xxxxxxxxxxxx'
#app.route('/', methods=['GET'])
def hello():
reqaction = request.args.get('action', 'connect')
client = TelegramClient(phone, api_id, api_hash)
if __name__ == '__main__':
app.run(host='localhost', port=8080, debug=True)
would you mind shine a light on this to help me starting up, much thanks
I am now able to host my user BOT on GAE using the session string login method.
https://docs.telethon.dev/en/latest/concepts/sessions.html#string-sessions
I'm using the Facebook Javascript API for login in conjunction with the official Facebook PHP SDK on my server to execute the two following lines of code:
$helper = $fb->getJavaScriptHelper();
$accessToken = $helper->getAccessToken();
With the token, I'm further able to execute this code which gets the necessary details I need on the server:
$fb->setDefaultAccessToken($accessToken);
$response = $fb->get('/me?locale=en_US&fields=name,first_name,last_name,email,gender');
If I refresh the webpage I'm working with and let it fully load everything works correctly and I'm able to print to screen all of the details I get back in $response.
The problem I'm having, however, is that if I quickly refresh the screen (either by hitting F5 on Windows machines or COMMAND+R on Macs) before the Facebook javascript code executes I get the following thrown error from the Facebook API:
"This authorization code has been used"
How do I avoid this? Do I wrap the Facebook code on the client side in a jQuery document ready function? I hesitate to do that because I've been told that the Facebook Javascript code is good to go as a stand-alone script that is intelligent enough to know when the document is loaded.
I'm about ready to throw in the towel and just code a manual login process that totally bypasses the Facebook Javascript API. Thanks for your help.
Put the access code into a $_session['access_token'] and then redirect to another page to get the data.
Login Page
Callback Page (save into the session variable)
Working Page (work with the session variable)
See more: https://benmarshall.me/facebook-php-sdk/#example-login
I'm wondering how to extract text from a password protected website using Matlab's "webread" function. I have the following code (part of which i got from here):
values=inputdlg({'Url','Username:','Password'});
options=weboptions('Username',values{2},'Password',values{3},'Timeout',Inf);
html=webread(values{1},options);
txt = regexprep(html,'<script.*?/script>','');
txt = regexprep(txt,'<style.*?/style>','');
txt = regexprep(txt,'<.*?>','');
But it gets stuck at the login window for every webpage I've tried. Help? Ideas? Thanks.
The weboptions username and password parameters are for basic HTTP authentication, which is different than logging into Stack Exchange, Gmail, etc though the username and password boxes on a web page.
Some sites provide other mechanisms that might allow you to log in (like OAuth), and the File Exchange has a smattering of clients.
Here is an OAuth interface for login into the flicker using Matlab, maybe it help you. but you have to be sure that your website support this authentification method (as "Matt Krause" mentioned in his answer).
Link to Tutorial
The new graph search on facebook lets you search for current employees of a company using query token - Current Google employees (for example).
I want to scrape the results page (http://www.facebook.com/search/104958162837/employees/present) via scrapy.
Initial problem was facebook allows only a facebook user to access the information, so directing me to login.php. So, before scraping this url, I logged in via scrapy and then this result page. But even though the http response is 200 for this page, it does not scraps any data. The code is as follows:
import sys
from scrapy.spider import BaseSpider
from scrapy.http import FormRequest
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.item import Item
from scrapy.http import Request
class DmozSpider(BaseSpider):
name = "test"
start_urls = ['https://www.facebook.com/login.php'];
task_urls = [query]
def parse(self, response):
return [FormRequest.from_response(response, formname='login_form',formdata={'email':'myemailid','pass':'myfbpassword'}, callback=self.after_login)]
def after_login(self,response):
if "authentication failed" in response.body:
self.log("Login failed",level=log.ERROR)
return
return Request(query, callback=self.page_parse)
def page_parse(self,response):
hxs = HtmlXPathSelector(response)
print hxs
items = hxs.select('//div[#class="_4_yl"]')
count = 0
print items
What could I have missed or done incorrectly?
The problem is that search results (specifically div initial_browse_result) are loaded dynamically via javascript. Scrapy receives the page before those actions, so there is no results yet there.
Basically, you have two options here:
try to simulate these js (XHR) requests in scrapy, see:
Scraping ajax pages using python
Can scrapy be used to scrape dynamic content from websites that are using AJAX?
use the combination of scrapy and selenium, or scrapy and mechanize to load the whole page with the content, see:
Executing Javascript Submit form functions using scrapy in python
this answer
If you go with first option, you should analyze all requests going during the page load and figure out which one is responsible for getting the data you want to scrape.
The second is pretty straightforward, but will definitely work - you just use other tool to get the page with loaded via js data, then parse it to scrapy items.
Hope that helps.