login via HttpURLConnection return status code 200 while it should redirect - httpurlconnection

I tries to login to the website using username and password. when I make a #HttpURLConnection and post it, the status code is 200 but it actually doesn't login. when I checked the login process with Chrome #DevTools Console, I found that after entering the login button, the parameters are sent to the address I used and it returned 302 as a status code. even I add this line to my code by the result doesn't changed.
connection2.setInstanceFollowRedirects(true);
here is my code.
String loginPageURL = "https://AAAAAAAAAA";
CookieManager cookieManager = new CookieManager();
cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
cookies.forEach(cookie -> cookieManager.getCookieStore().add(null, cookie));
URL url2 = new URL(loginPageURL);
HttpURLConnection connection2 = (HttpURLConnection) url2.openConnection();
connection2.setRequestProperty("Cookie",
StringUtils.join(cookieManager.getCookieStore().getCookies(), ";"));
connection2.setInstanceFollowRedirects(true);
String loginPayload ="mypayload";
connection2.setRequestMethod("POST");
connection2.setDoOutput(true);
connection2.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
connection2.setRequestProperty("Accept-Encoding", "deflate, br");
connection2.setRequestProperty("Accept-Language", "en-US,en;q=0.9,fa;q=0.8");
connection2.setRequestProperty("Cache-Control", "max-age=0");
connection2.setRequestProperty("Connection", "keep-alive");
connection2.setRequestProperty("Content-Length", String.valueOf(loginPayload.length()));
connection2.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection2.setRequestProperty("Host", "https://BBBBBBBBBB");
connection2.setRequestProperty("Origin", "https://BBBBBBBBBB");
connection2.setRequestProperty("Referer", "https://AAAAAAAAAA");
connection2.setRequestProperty("sec-ch-ua", " Not A;Brand;v=99, Chromium;v=100, Google Chrome;v=100");
connection2.setRequestProperty("sec-ch-ua-mobile", "?0");
connection2.setRequestProperty("sec-ch-ua-platform", "Windows");
connection2.setRequestProperty("Sec-Fetch-Dest", "document");
connection2.setRequestProperty("Sec-Fetch-Mode", "navigate");
connection2.setRequestProperty("Sec-Fetch-Site", "same-origin");
connection2.setRequestProperty("Sec-Fetch-User", "?1");
connection2.setRequestProperty("Upgrade-Insecure-Requests", "1");
connection2.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36");
DataOutputStream out = new DataOutputStream(connection2.getOutputStream());
out.writeBytes(loginPayload);
System.out.println("login connection status code: "+connection2.getResponseCode());
System.out.println("content length "+loginPayload.length());
out.close();
System.out.println("*************************************************************");
int status = connection2.getResponseCode();
if (status == HttpURLConnection.HTTP_OK) {
String header = connection2.getHeaderField("Location");
System.out.println(header);
}
Anybody can help me figuring out where the problem is?
thanks in advance.

Related

Formrequest and Scrapy. Fill out a form

I would like to fill out a form using formrequest and scrapy. However, I'am stuck and I don't know how to continue to submit the form . I would be very grateful if someone could help me. This is the code I'm using:
import scrapy
def authentication_failed(response):
# TODO: Check the contents of the response and return True if
it failed
# or False if it succeeded.
pass
class IdealistaSpider(scrapy.Spider):
name = "MiPrimerSpider"
custom_settings = {'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'
}
start_urls = https://www.idealista.com/inmueble/94342193/']
def parse(self, response):
return scrapy.FormRequest.from_response(response,formxpath="//div[#class='modulecontact_form']/form[#class='formcontact']",
formdata={'contact-email': 'an email','contact-phone':'a number phone','contact-name':'costaman','privacy-policy-checkbox':' ','recommendations-checkbox':None},
clickdata={'class': 'btn action txt-bold txt-big desktop'},
callback=self.after_login
)
def after_login(self, response):
if authentication_failed(response):
self.logger.error("Login failed")
return

Python beautiful soup and requests submit form data on post url

How can i post form data on the url: http://washington.kdmid.ru/queue-en/Visitor.aspx
When i submit form data with below fields i am getting same page in response instead of redirect to next page.
import requests
from bs4 import BeautifulSoup
location_url = "http://washington.kdmid.ru/queue-en/visitor.aspx"
s = requests.Session()
main_page = s.get(location_url)
main_html = BeautifulSoup(main_page.text)
c_form = main_html.find_all("form")[0]
c_form_submit = c_form.attrs["action"]
data = {e.attrs.get("name"): e.attrs.get("value") for e in c_form.find_all("input")}
data["ctl00$MainContent$txtFam"] = "bsssabassra"
data["ctl00$MainContent$txtIm"] = "Akssssshassya"
data["ctl00$MainContent$txtOt"] = "a"
data["ctl00$MainContent$txtTel"] = "1122334455"
data["ctl00$MainContent$txtEmail"] = "akssbsars2#gmail.com"
data["ctl00$MainContent$DDL_Day"] = 1
data["ctl00$MainContent$DDL_Month"] = 1
data["ctl00$MainContent$TextBox_Year"] = 1993
data["ctl00$MainContent$DDL_Mr"] = "MR"
data["ctl00$MainContent$txtCode"] = captcha_txt
data["ctl00$MainContent$ButtonA"] = "Next"
import json; json.dumps(data)
submit_captcha_resp = s.post("http://washington.kdmid.ru/queue-en/visitor.aspx",
data=json.dumps(data))
final_page = BeautifulSoup(submit_captcha_resp.text)
It wont redirect, because it's not a browser. BS don't run the JS scripts or HTML code. But you get the response.
You should use one of these:
submit_captcha_resp = s.post("yourLongURL", json=data)
or
submit_captcha_resp = s.post("yourLongURL", data=data)
json.dumps() is used to convert a JSON to a string but you don't need that because the webpage which you are posting data uses HTML tag and form tag posts the data without converting it to string. So you shouldn't convert it to a string. You should post it in JSON format.
And as #dharmey said: If you get a 404, you should set a user agent as a popular web browser. For example:
{"User-Agent":"Mozilla/5.0"}
And I think now you have bigger problems like passing the Captcha.
I think you might be posting the data in the wrong way. You could try
submit_captcha_resp = s.post("http://washington.kdmid.ru/queue-en/visitor.aspx",
json=data)
Instead of data=json.dumps(data))
If this dosen't work / the site requires actual form data, try to pass in some headers, as they might be required for the server to recieve the request correctly.
You could just include
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36'
}
submit_captcha_resp = s.post("http://washington.kdmid.ru/queue-en/visitor.aspx",
headers=headers, data=json.dumps(data))
to start out.

403 response with HttpClient but not with browser

I’m having a problem with the HttpClient library in java.
The target web site is on SSL (https://www.betcris.com), and I can load the index page from that site just fine .
However, the different pages showing odds for the different sports returns a 403 response code with HttpClient, but loading the same pages in a browser works just fine.
Here is such a page : https://www.betcris.com/en/live-lines/soccer.
I started troubleshooting this page with the information gathered by HttpFox (a Firefox add-on that resembles LiveHttpHeaders), making sure I had all the correct request headers and cookies, but I couldn’t get it to load using HttpClient. I also determined that cookies have nothing to do with the problem, as I can remove all cookies for that web site within my browser, and then hit the page directly and it will load.
I confirmed that there’s something special going on with these pages by using the online tool at http://www.therightapi.com/test. This tool allows you to input the url of a page along with any Request header you want, and shows you the response you get from the target web site. Using that tool, I can load https://www.google.com just fine, but I get the same 403 error when trying to load https://www.betcris.com/en/live-lines/soccer.
Here's my setup at therightapi :
And the response :
Does anyone know what’s going on here ?
Thanks.
EDIT : I've created a test project, here's the java code, followed by the maven dependency you should have in your pom :
package com.yourpackage;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClientBuilder;
public class TestHttpClient {
public static void main(String[] args) {
String url = "https://www.betcris.com/en/live-lines/soccer";
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
// add request header
request.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0");
try {
HttpResponse response = client.execute(request);
System.out.println("Response Code : "
+ response.getStatusLine().getStatusCode());
BufferedReader rd = new BufferedReader(
new InputStreamReader(response.getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) {
result.append(line);
}
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.3</version>
</dependency>
I have solved this problem (avoiding 403) by setting up User-Agent property while making a request as like follow:
If you use HttpClient
HttpGet httpGet = new HttpGet(URL_HERE);
httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
If you use HttpURLConnection
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
I use the following code to consume HTTPS Urls:
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.SSLContextBuilder;
...
SSLContext sslContext =
new SSLContextBuilder().loadTrustMaterial(null, (certificate, authType) -> true).build();
try (CloseableHttpClient httpClient = HttpClients.custom().setSSLContext(sslContext)
.setSSLHostnameVerifier(new NoopHostnameVerifier()).build()) {
HttpGet httpGet = new HttpGet("YOUR_HTTPS_URL");
httpGet.setHeader("Accept", "application/xml");
httpGet.setHeader("User-Agent",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
HttpResponse response = httpClient.execute(httpGet);
logger.info("Response: " + response);
}
pom.xml:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.3</version>
</dependency>
In my case, the web server does not use a proxy to communicate with APIs.
I just disbaled the defaultproxy under system.net in web.config.
<system.net>
<defaultProxy enabled="false" />
</system.net>
403 Forbidden is used to signal an authentication requirement. In fact, the full 403 response should tell you exactly that. Luckily, HttpClient can do authentication.

PayPal Sandbox API SSL handshake error HTTPS request

With new changes in paypal , it started throwing SSL handshake exceptions those who are using old system. "PayPal SSL Certificate Changes"
https://devblog.paypal.com/paypal-ssl-certificate-changes/
This may help someone. After i got SSL Handshake exception , i spent a hell lot of time to resolve it.
Here is the Exception :
javax.net.ssl.SSLHandshakeException: Received fatal alert:
handshake_failure
Solution :
Requirements to resolve this issue :
Start from Jan 19, all sandbox API endpoint need to
1.) Use TLS 1.2 and HTTP/1.1 connection
2.) Upgrade to SHA-256 and use the G5 root certificate to make the HTTPS connection
Point 1 Solution:
If you are using java 6 then better upgrade it to java 7
https://blogs.oracle.com/java-platform-group/entry/diagnosing_tls_ssl_and_https
For my case i am using java 7 so TLSv1 (default) for JDK 7.
We have to enable it manually while starting server
**-Dhttps.protocols=TLSv1.2** passed as vm argument.
Point 2 Solution :
https://knowledge.verisign.com/support/mpki-for-ssl-support/index?page=content&actp=CROSSLINK&id=SO5624
G5 cerificate import: Save it as test.cer
Go to java home/bin then run this command
keytool -importcert -file C:/test.cer
create sanbox account. Get the facilator password and signature pass it as parameters
String encodedData = "USER=XXX-facilitator_api1.XXX.XXX"
+ "&PWD=XXXXXXXXXXXX"
+ "&SIGNATURE=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-"
+ "&VERSION=95"
+ "&METHOD=SetExpressCheckout"
+ "&PAYMENTREQUEST_0_PAYMENTACTION=Authorization"
+ "&L_PAYMENTREQUEST_0_NAME0="+URLEncoder.encode("Testing","UTF-8")
+ "&L_PAYMENTREQUEST_0_DESC0="+URLEncoder.encode("Testing","UTF-8")
+ "&L_PAYMENTREQUEST_0_AMT0="+URLEncoder.encode("99","UTF-8")
+ "&PAYMENTREQUEST_0_AMT="+URLEncoder.encode("99","UTF-8")
+ "&PAYMENTREQUEST_0_CURRENCYCODE="+URLEncoder.encode("USD","UTF-8")
+ "&LOCALECODE=en_GB"
+ "&RETURNURL=google.com"
+ "&CANCELURL=google.co.in"
+ "&LOGOIMG=imageurl";
String responsepaypal = getHTMLcontent("https://api-3t.sandbox.paypal.com/nvp",encodedData ,"UTF-8");
String token = responsepaypal.toString().replaceAll("TOKEN=(.*?)&TIMESTAMP.*", "$1");//***Token for post request on paypal***
public static String getHTMLcontent(String url,String urlParameters, String encodingDef) throws IOException {
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-length", String.valueOf(urlParameters.length()));
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36");
con.setRequestProperty("Host", "api-3t.sandbox.paypal.com");
con.setRequestProperty("Upgrade-Insecure-Requests", "1");
con.setRequestProperty("Pragma", "no-cache");
//con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
con.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
con.setRequestProperty("Connection", "keep-alive");
con.setDoOutput(true);
con.setDoInput(true);
DataOutputStream output = new DataOutputStream(con.getOutputStream());
output.writeBytes(urlParameters);
output.close();
DataInputStream input = new DataInputStream( con.getInputStream() );
StringBuffer sb = new StringBuffer();
String line;
while ((line = input.readLine()) != null) {
sb.append(line);
}
input.close();
return sb.toString();
}}
Follow the steps out here clearly mentioned:
https://developer.paypal.com/docs/classic/express-checkout/ht_ec-singleAuthPayment-curl-etc/
I was testing paypal using the sandbox account and I was getting the same error. I upgraded to java 8 and the error was not there anymore.

Marketing API Export Async Report ads_insights V2.5

I have problems to export Report of ad's statistics since code written in.NET Framework when I use ads_insights (version 2.5). Before when I used reportstats with version 2.3 I could download the report succesfully
My request is //www.facebook.com/ads/ads_insights/export_report?report_run_id=0000000&format=xls&access_token=token
When I execute the request in browser I can download the report succesfully (file xls completed), but when I execute the request since by code .NET Framework (C#) I download the file .xls incomplete [enter image description here][2]
The tasks to get report are (using code .NET C#)
1º request with method POST
graph.facebook.com/v2.5/act_countNumbrer/insights?level=ad&time_range=%7B%27since%27%3A%272015-11-02%27%2C%27until%27%3A%272015-11-02%27%7D&actions_group_by=%5B%27action_type%27%5D&fields=campaign_name%2Cad_name%2Cad_id%2Creach%2Cfrequency%2Cimpressions%2Ccpm%2Ccpp%2Cspend%2Csocial_clicks%2Cunique_clicks%2Cctr%2Cunique_ctr%2Caccount_name%2Cactions%2Ctotal_actions%2Cwebsite_clicks&time_increment=1&access_token=token
Result: successful -> I get a report_run_id
2º request with method GET
graph.facebook.com/v2.5/id_report&access_token=token
Result: successful -> I get a
{
"id": "xxxx",
"account_id": "xxx",
"time_ref": 1447171267,
"time_completed": 1447171269,
"async_status": "Job Completed",
"async_percent_completion": 100,
}
3º when "async_status" is "Job Completed", I execute request
www.facebook.com/ads/ads_insights/export_report?report_run_id=xxxx&format=xls&access_token=token
Result: I download the file .xls incomplete. If you paste the query (URL) in browser you download the report succesful (file xls completed) enter image description here
If I execute the request with code .NET Framework (C#) and saved the response as string the response said we "should update your browser " enter image description here
Why can't I download the report ?
Thank you
Code using to execute the download the report XLS
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
namespace test
{
class Program
{
static void Main(string[] args)
{
string token="token";
string report_run_id="report_number";
string url = "https://www.facebook.com/ads/ads_insights/export_report?report_run_id="+report_run_id+"format=xls&access_token"+token;
//option 1
string reportDownloadUrl = "repo"+DateTime.Now.Ticks + ".xls"; ;
Stream responseStream = null;
try
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
//request.UserAgent = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)";
var response = (HttpWebResponse)request.GetResponse();
responseStream = response.GetResponseStream(); //relleno el flujo
using (var fileStream = new FileStream(reportDownloadUrl, FileMode.Create, FileAccess.Write))
{
responseStream.CopyTo(fileStream);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (responseStream != null) responseStream.Close();
}
Console.WriteLine("File Download" +reportDownloadUrl);
/* //option 2
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
wc.Headers["User-Agent"] = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)";
wc.DownloadFile(url,"repo.xls");
}
Console.WriteLine("File Download");
*/
Console.ReadKey();
}
}
}
I had the same issue, trying to download the report using request npm in
nodejs.
Adding the User-Agent header solved my problem.
This works for me. You have some issues in your URL string.
Here is an updated url string:
string url = "https://www.facebook.com/ads/ads_insights/export_report?report_run_id=" + report_id + "&format=csv&access_token=" + accessToken;
This method works.
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
wc.Headers["User-Agent"] = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)";
wc.DownloadFile(url, "page1.csv");
}
The report gets saved to:
C:\your\path\to\project\FacebookReportPuller\bin\Debug