I tried to run C:\Spark\spark-1.6.1-bin-hadoop2.6\sbin>start-master.sh
, but the following mistake appeared
I also noticed that there is a warn while running bin/spark-shell
16/04/24 23:14:41 WARN : Your hostname, Pavilion resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:c0a8:867%net14, but we couldn't find any external IP address!
http://localhost/8080 is also unawaliable.
Can anybody please tell where is a mistake or did I miss any settings which are nessesary to run Master properly?
The problem is that this script is not designed to be executed on a Windows machine. Please refer to Official Spark Manual
Note: The launch scripts do not currently support Windows
The rule of thumb is that only script ending with .cmd will run on Windows. While scripts ending in .sh are designed for Linux and Mac OS. And while it should be possible to manually start Spark Master on Windows, it's probably better to just run a local[*] mode, unless you are creating a cluster of Windows machines. local[*] mode already fully utilizes power of the local machine.
Related
I am connecting remote linux and developing some application. To run the application I need to connect to another host for debugging due to resource availibility using srun command.
Seems like partial inception is possible using below in launch.json.
"miDebuggerServerAddress": "remoteHostname:9091",
And running the command on remote host as:
user#remoteHostname $ gdbserver :9091 ./myapplication arguments to my program
I have a small simple setup consisting of Jenkins & Ansible 2.7.8 running on Ubuntu 18.04.2 LTS (192.168.0.202):
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/jon/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
Jenkins ver. 2.150.3
I then have a Windows VM (192.168.0.203) that has a Powershell script stored on it and an Ansible playbook configured to connect to the Windows VM and run the Powershell script.
When I run the Ansible-Playbook directly from the command line it works fine, connects to the Windows machine and runs the script. All good.
I am having real trouble though implementing Ansible into Jenkins to run the playbook. When I run the playbook through Jenkins, I get the following error:
Building in workspace /var/lib/jenkins/workspace/Ansible-RunPS-1.0
[Ansible-RunPS-1.0] $ /usr/bin/ansible-playbook //etc/ansible/runPS.yml -f 5
PLAY [Runs remote PS script] ***************************************************
TASK [Gathering Facts] *********************************************************
[0;31mfatal: [192.168.0.203]: FAILED! => {"msg": "winrm or requests is not installed: No module named winrm"}[0m
To me, it seems to be running the playbook but then failing as it cannot find the winrm module. Could it be something to do with the account that Jenkins uses can't somehow find the winrm module, yet if I run the same command under my account, it finds it ok ?
Happy to post other configs etc. if that would help but thought I'd try and keep it as simple as possible to begin with.
As is the way with these things, you spend days trying to fix it, then as soon as you ask for help you manage to fix it.
I'd love to be able to say it was just a case of installing this or running that but in truth, I really don't know what got it going, it suddenly burst into life after a reboot, can't believe that's all it was but you never know!
I'm trying to run juinit on a remote postgresql docker.
I tried to use testcontainers.org but I can't get the configuration to work
I keep getting timeout exception
Does testcontainers support such setup? if so is there an example for that?
Thanks
as long as you have your Docker configured correctly, it should work out of the box without any additional configuration.
Just make sure that your environment variables point to the remote Docker daemon.
Well, my previous attempt to connect Pycharm from laptop to remote server did not see any ray of hope because of tcp/ip issues (which honestly I could not understand much and am still battling with) therefore I am looking to ipython as an alternative
Question: How can I configure ipython on laptop to point to the remote CentOS6 server for code processing and execution
Use Case: I want to use my laptop (using Win 7 Professional) to connect to the CentOS 6.4 master server using iPython.
Objective: To write the code in iPython on the laptop and then send the job to the server which will do the processing and should then return the result back to the laptop or to any other visualizing API.
The server and 3 namenodes already installed with pyspark and I have checked pyspark works in standalone mode on all four servers. Pyspark works in standalone mode on my laptop too.
Current setup: I use SSH to access the server. python 2.6 is installed on the server and the nodes. Able to run pyspark on all 4 servers in standalone mode
Any pointers will be helpful.
You have to start the IPython Notebook server on one cluster-node and then connect to the cluster-node's url. To do this, you have to create a profile, where you may specify hostname, port and so on. Have a look at this:
http://ipython.org/ipython-doc/1/interactive/public_server.html
I used to debug the Windows Kernel using VirtualKD, WinDBG and a single Virtual Machine.
Recently I got a Linux machine, and now I wonder- What's the easiest way to debug the Windows Kernel when your host is unable to run VirtualKD/WinDBG*?
I assume the solution will require two Virtual Machines, but I rather have two instances hosted on my actual machine rather than having an instance residing inside another virtual instance...
Is there anyway to make that work?
Thanks in advance!
*Wine is the last resort for stability reasons...
Solved! Basically, I ended up using two (VirtualBox) VMs emulating a Serial connection (null-modem cable) over a Unix domain socket (on the host). For more info, read below:
Hardware setup*:
Debuggee:
Ensure the machine is turned off and edit Serial Ports settings.
Enable Port 1, and assign values as follows: Port Number: COM1, Port Mode: Host Pipe, Create Pipe: Unchecked (client), Port/File Path: /tmp/win_link.
Debugger:
Same as above (using the same path), only this time Create Pipe should be Checked (server).
Debugger setup:
Run WinDBG and press Ctrl+K to invoke Kernel Debugging.
in COM, enter: Baudrate: 115200, Port: COM1, Resets: 0 and verify that Pipe and Reconnect are unchecked (important).
You'll be presented with the following output: Opened \\\\.\com1 Waiting to reconnect...
Debuggee setup:
Run bootcfg /debug on /port com1 /baud 115200 /id 1. To verify, run bootcfg.**
Reboot.
Quite early during the booting stage, WinDBG on the other machine should detect the debuggee is running.
*Assuming VirtualBox is used. VMWare/KVM users will probably be able to achieve the same results following similar steps. Also, for more info refer to the VirtualBox docs.
**Assuming guests are Windows XP. Later versions include bcdedit, which may be used as described here.
For QEMU\KVM follow those instructions:
http://www.linux-kvm.org/page/WindowsGuestDrivers/GuestDebugging
Very helpful but applies to Windows XP machines. You can refer to the following link if you need to configure 2 Windows7-based virtual machines on a Linux host: http://www.aldeid.com/wiki/Category:Digital-Forensics/Computer-Forensics/Debugger/Kernel
Another option nowadays is to enable local kernel debugging. This comes with some limitations, however it will enable you to access kernel data while just using one VM.
This approach only works on Windows 8.0 and Windows Server 2012 and later.
Follow these steps:
Open a Command Prompt window as Administrator.
Enter bcdedit /debug on
If the computer is not already configured as the target of a debug transport, enter bcdedit /dbgsettings local
Reboot the computer.
Once the system is rebooted, you can execute WinDBG as Administrator, press ctrl+k or go to File -> Attach to kernel -> Local and press OK.
At that point, you will be able to execute kernel-only commands and access kernel structures:
Tested under Windows 10 and with the new WinDBG version (preview).
Reference: Setting Up Local Kernel Debugging of a Single Computer Manually