I created a Google dataproc cluster. After logging into master node I started spark-shell then trying to access spark history page using
http://<external_ip_masternode>:4040
It get redirected to
http://<hostname_mastername>:8088/proxy/application_1487485713573_0002/
Browser is rejecting with error "DNS address could not be found." which is understandable.
Following are VM instance setting
Public IP type Ephermal
tcp:4040 opened in firewall
ip forwarding Off: Unable to edit this configuration
Following troubleshooting done but did not help
Telnet to :4040 -> Working
Access from Ubantu host/ browser Chrome: Getting redirected and name lookup failure
Access from Ubantu host /browser Firefox: Getting redirected and name lookup failure
Access from Mac OSX host /browser Safari : Getting redirected and name lookup failure
Access from Mac OSX host/ browser chrome : Getting redirected and name lookup failure
To view Hadoop web interfaces in Dataproc, it is recommended to follow the instructions for running an SSH-based SOCKS proxy: https://cloud.google.com/dataproc/docs/concepts/cluster-web-interfaces
If you follow the instructions there, it'll also have you run a separate browser session using your SSH tunnel, and sets hostname resolution to occur on the VM side of the tunnel. That way, all the links in the Hadoop pages will automatically work, since they all reference each other using internal hostnames, and intentionally avoid any dependency on "external IP addresses".
Using the SSH tunnel is also much more secure than opening up firewall rules to visit the unencrypted HTTP traffic directly coming from the Hadoop HTTP servers (if you accidentally open up your firewall rules too broadly, then other people on the internet will be able to access your external IP addresses, and even if you don't, attackers could see your unencrypted web traffic served up by the ApplicationMaster, HistoryServer, etc.).
Related
I'm a new comer to using the overseas server. Recently I bought a vps from virmach in order to see foreign websites like google and wiki.
I've been trying for a long time configuring my shadowsocks on my server.
However, when I was using shadowsocks-qt5 to connect my server, it was timeout.
And of course I can't access google correctly.
What I want to ask is the reason why I failed.
Here are things that I do remember to do:
stop the firewall on both computers;
build the .json file which I referred to blogs in China.
Here are the outline of my shadowsocks.json on my server:
{
"server":"0.0.0.0",
"server_port":8388,
"local_address":"127.0.0.1",
"local_port":1080,
"password":"XXXX",
"timeout":600,
"method":"aes-256-cfb"
}
Other useful(maybe) information:
my client OS version: Ubuntu 18.04.3 LTS
my server OS version: Ubuntu 16.04.6 LTS
the client I choose is from: https://github.com/shadowsocks/shadowsocks-qt5
I could not help but wandered, are there any other possible reasons I've forgot? Can anyone inform me some helpful details to solve this puzzling problems? Thanks a lot!
I have not set up my own VPS but I have instead subscribed to the server provided by caonima.io, so I can't speak for any server related issues. Additionally, I have no affiliation with caonima.io. I did however successfully set up my client on Ubuntu 16.04 after having some issues connecting to GFW-blocked (China's Great FireWall) websites.
From what I understand from my solution, the client configuration is NOT the only step of setup. There are two layers of proxy access that need to be completed:
Client Configuration. Configure your client with the server and connection information. A successful connection looked like this for me with my command line interface
shadowsocks-libev command line client successful connection
System or Browser Proxy Configuration. You will need to configure either your browser or web access tool to use a proxy, or set system-wide proxy settings. To set system wide proxy settings, go to system settings > network > network proxy and enter the proxy information. Setting Socks host to localhost:1080 resulted in successful GFW-blocked website access (as shown below)!
Ubuntu network settings proxy manual configuration
I am trying to use a Virtual Box VM to create a development environment I can share with a few dozen other developers. We are all on an internal network and need to connect to external web sites via proxy server. On the VM (guest) I am putting:
windows 10 64 bit
weblogic server
Several other dev tools
My requirements are:
From the guest OS I need to be able to hit the internet (ie google.com)
From the guest OS I also need to be able to open chrome and hit the server webpages running on the guest OS using mysite.com:8007/index.html.
I have modified the guest OS host file such that 127.0.0.1 maps to mysite.com. Ideally that should mean anything going to mysite.com will get resolved to 127.0.0.1.
Our proxy to reach the internet is http-proxy.mysite.com:80. I am able to connect to the internet (pages like www.google.com) as long as I enable the proxy (http-proxy.mysite.com:80) in chrome proxy settings. However I need to be able to enter mysite.com:8007/index.html in the browser and be able to load the homepage. I am able to hit the server using localhost:8007 but I need mysite.com:8007 to work since authentication cookies wont get passed correctly to localhost:8007 urls.
When I enter mysite.com:8007/index.html in chrome it keeps trying to redirect to the IP address of the host OS 10 . * . * . * and the server is not running on the Host OS.
When I enter 'curl mysite.com:8007/index.html' in a command prompt on the guest OS it actually does prints the correct response from my server (no idea why chrome is different / not respecting the host file config).
Any idea how to configure Virtual Box or chrome such that the external pages as well as the guest server pages work?
For internet access on VM, you need to configure network card as BRIDGE, NAT or NAT NETWORK. In your case NAT NETWORK will be better solution, because this will allow you bo be you VM visible in network, just like your Host.
If you have and Internal network you mus have also another card configured for VM for internal network.
To be your VM accessible from Internal Network via http... the coomputers in network have to setup DNS entry pointing to your VM.
So I have created something small which is a image-rehost where I wish to use Python script where I have a URL such as https://i.imgur.com/VBPNX9p.jpg but with my rehost it would be
https://ip:port/abc123def456
so whenever I access that page it would give me the url that I posted here.
However the issue I am having is that I have no clue how to actually host the server that I made by node-js. Right now I just used the external IP with port of 5000. When I tried to send the image through my home ip by using the
https://external_ip:5000/abc123
the server doesn't recognize anything and nothing is being sent to the server which I in that case think I have setup something wrong.
I am using Google cloud server and I would wish to know how I can host my own server in the google cloud?
As you are having trouble adding a firewall rule, I'm going to suggest make sure port 5000 is open and not 8888.
To open the firewall rule for port 5000 in Google Cloud Platform follow these steps.
1) Navigate to VPC Network > Firewall rules > Create firewall rule.
2) In the 'Create a firewall rule' page, select these settings:
Name - choose a name for this firewall rule
Network - select the name of the network your instance belongs to, most probably
'default' unless you've configured a custom network.
Direction of traffic - 'Ingress'.
Action on match - 'Allow'.
Targets - 'All instances in the network'.
Source filter - 'IP ranges'.
Source IP ranges - '0.0.0.0/0'.
Second source filter - 'None'.
Specified protocols and ports - 'tcp:5000' or 'udp:5000' depending on whether the protocol you are using uses tcp or udp.
3) Hit 'Create'.
This will create a rule allowing traffic on port 5000 to all instances in your network from all IP address sources.
My advice would be to see if these settings work, and then once confirming this, lock down the settings by specifying a specific IP address or range of IP addresses in the 'Source IP ranges' text box, and adding a target tag to you instance and specifying 'Specified target tags' so the port is only open to the instance.
If this doesn't work, you may have a firewall rule turned on within the instance, which you would need to configure (or turn it off).
For more detailed information about setting firewall rules please see here.
For running Node.sj on GCE VM I will suggest you use the Bitnami Node.js package on GCP Marketplace which includes the latest version of Node.js, Apache, Python, and Redis. Using a pre-configured Node.js environment gets you up and running quickly because everything works out of the box. Manually configuring an environment can be a difficult and time-consuming hurdle to developing an application.
Also if you wish to do URL redirection you can use URL map feature provided with Google Cloud HTTP load balancer. This feature allows you to direct traffic to different instances based on the incoming URL. For example, you can send requests for http://www.example.com/audio to one backend service, which contains instances configured to deliver audio files, and requests for http://www.example.com/video to another backend service, which contains instances configured to deliver video files. You find steps to configure and more information here.
I am writing an application with C++ to connect to my server.
I am using gethostbyname(). It is working fine when I give host name as 'localhost'. but if i try any other site like google.com, it fails.
Can anyone please tell me what could be going wrong.
I can access those websites through browser. Note: I have proxy set up in my browser.
Is thr proxy causing issue with gethostbyname()? if yes, how can I mention proxy in my C++ code?
Based on what you describe in your question, it appears that your machine is on a corporate network that is firewalled off the Internet, all access to the web is via a web proxy, and that your network does not have Internet DNS resolution.
Even if you managed to succeed in resolving an IP address, you will not be able to make an outbound connection, unless your firewall offers a Socks proxy, or an equivalent.
If your intent is to write a client that accesses web sites via HTTP, you will need to use your corporate HTTP proxy to do that. Contact your system administrator for more information.
I have set up a google cloud sql server using the tutorial on youtube. However, Whenever I try to connect to it through the browser, the connect to it through the browser, the error i am getting is ERR_CONNECTION_REFUSED . The firewalls I have set up correlate to whats in the browser. I can also SSH into the VM. But the VM always rejects my connection. I have been trying for days.
Here are my VM Settings allowing http connections
Here is my VM External IP Address
Here is what the browser tells me when i attempt to connect.
Are you referring to this video https://www.youtube.com/watch?v=_kQXgjIfLgo?
In that video there are some steps missing such as install the web server (e.g. Apache) and PHP in the VM. After you install them, you can execute the following command in the VM:
php /var/www/index.php
If it works you can see the source code of the resulting page and if not you'll have an error to solve (maybe a package to install like php-mysql).
Once you can see the resulting page in the VM you can try to load the page through the browser of your machine. If it still doesn't work there is an Apache or network misconfiguration.