I have some problems to understand the logical architecture in which I develop with Scala/Spark-shell and Hadoop environment.
For better describe the logical architecture, I drew a small schema:
As the figure shows, I have Eclipse installated on my personal PC, and I would like to run scala script from my PC to Hadoop in remote mode.
Now I have the VPN connection, and I can process my scala program with PUtty from the shell. In practice, every time that I have to launch a Scala script, I transfer the file .scala from my pc to remote machine with WinSCP, so I lanch the program directly from the remote machine. Every time I have to tranfer the file making me work wasteful.
Now the question: is there a way to launch the script from my personal PC to remote cluster, without pass through to the PUtty?
Related
I need to run some jar file in another machine in LAN.
java -jar start.jar
How to run a jar file on remote computer?
Thx!
I faced the same issue in my project while accessing resources in a distributed systems environment. If you working on Windows, you need PsExec v2.x.
Create a batch file to execute java -jar start.jar from remote cmd.
Now run the batch file from ur local machine by PSEXEC \\othermachine z:\program.exe. You will need administrator privileges on the target machine.
Visit for documentation.
Lastly, if you demand platform independency and wish to run your jar on remote machine having OS other than the OS on your local machine, then you should go with socket programming.
Via remote client like ssh, teamviewer or custom solution with server-client connection.
So I've been playing with Akka Actors for a while now, and have written some code that can distribute computation across several machines in a cluster. Before I run the "main" code, I need to have an ActorSystem waiting on each machine I will be deploying over, and I usually do this via a Python script that SSH's into all the machines and starts the process by doing something like cd /into/the/proper/folder/ and then sbt 'run-main ActorSystemCode'.
I run this Python script on one of the machines (call it "Machine X"), so I will see the output of SSH'ing into all the other machines in my Machine X SSH session. Whenever I do run the script, it seems all the machines are re-compiling the entire code before actually running it, making me sit there for a few minutes before anything useful is done.
My question is this:
Why do they need to re-compile at all? The same JVM is available on all machines, so shouldn't it just run immediately?
How do I get around this problem of making each machine compile "it's own copy"?
sbt is a build tool and not an application runner. Use sbt-assembly to build an all in one jar and put the jar on each machine and run it with scala or java command.
It's usual for cluster to have a single partition mounted on every node (via NFS or samba). You just need to copy the artifact on that partition and they will be directly accessible in each node. If it's not the case, you should ask your sysadmin to install it.
Then you will need to launch the application. Again, most clusters come
with MPI. The tools mpirun (or mpiexec) are not restricted to real MPI applications and will launch any script you want on several nodes.
During our development process the developers do code modifications, compile the code and need to deploy it on a remote machine and test it or debug it remotely.
There are manual steps that are usually needed - stop one or more services, copy the compiled files to specific place in the destination machine and other steps (maybe delete some folder etc.)
I was wondering if there is a tool that as input gets IP of remote machine and predefined steps (stop service, copy local files to remote machine etc) - and just do autmatic deployment for the developer? I'd like to automate this tiring process a bit...
Thanks.
Ant is a common tool for such tasks in Java development. You can use ant to compile your code, use an scp task to copy your binaries to a server and run scripts on that server. The configuration is done by XML and is pretty easy. You should google or search on stackoverflow for some examples.
I use rundeck to control my deployments. I like it's simplicity and the fact that all that's required is SSH access to my servers, enabling me to upload files, and run whatever scripts I require.
It has a simple XML configuration file listing the servers in my network. This makes it really easy to integrate with other CM tools.
For windows deployments you're going to require an SSH implementation installed on each node, or a more complicated deployment tool.
Is it possible to set up a remote NetBeans C++ project where the source files are only accessible via SSH?
My project needs to build on a Linux box, but I'd like to develop it on a Windows machine.
Checking out the code via SVN to my Windows machine is not an option since there are a few files that differ only by case, and NTFS is not case sensitive (unfortunately, I can not change them).
I'm well aware that Windows can be kind-of forced be case-aware and the ideal solution is to just re-name those file to something sane.
However, I'm just trying to solve this using NetBeans. Since it's a remote project anyway, why bother to keep any files locally.
Thanks
Currently, no. In general programming files with different cases of the same name is a bad practice.
You can enable case sensitivity in Windows - you may need to have a Professional version or better.
For Windows XP: http://support.microsoft.com/kb/817921
For Windows 7: http://technet.microsoft.com/en-us/library/cc732389.aspx
See also: Windows Services for Unix
Another solution would be to setup VNC/RDP on the remote Unix system. The overall solution should be to conform to a better file naming convention:
Programmer 1: "Hey man, take a look at noCamelCase.cs - I just rewrote it."
Programmer 2: "Um, nocamelcase.cs is blank."
There are two ways of doing remote builds with Netbeans. The first, the project is stored locally. You just create a regular project and on the 2nd page of the wizard you specify the network directory with the source and the remote build host. I've used this for Solaris client to Linux server, but not from Windows as we don't have the mounts exported by SMB. This uses ssh and some shared lib interposers to get the build info.
The second way is to create a remote project. In this case the project is created on the remote host and date is copied on demand to the client. I've only doe a few tests with this as I preferred the first method as it had much better latency.
Lastly, you could either use vnc or install X on your windows machine and do everything on the Linux machine.
Does anyone know of a good way to do remote administration of a Windows XP machine using just the command line?
At the moment the only things it needs to do is to be able to install applications/patches, and transfer files to and from the machine, and installing registry patches would be nice as well.
Currently we use a horrible hacked together solution that uses NetMeeting, in the past I've thrown together a proof of concept using SSH for windows (at the time windows 2000) but it didn't work to my satisfaction and was pretty buggy. Which was probably the result of the SSH Daemon I was running more then anything.
I'm pretty much open to anything, however a solution using SSH would be ideal since it's already approved for installation in my organization, and it's free. I work in the Canadian Government so anything free is best, and anything that we've already got approved for installation is even better.
psexec will allow you to run commands remotely. Some of the other PsTools can help you kill applications, get a list of processes, etc.
Why must it be
remote administration of a Windows XP machine using just the command line?
I think your very limiting yourself to what is possible by sticking to the command line. In windows environments you can easily use Group Policy to distribute most software and/or patches, and for the ones that you can't you can usually script these changes through any of the popular scripting languages such as JScript, VBScript, Kixtart, AutoIt, Powershell, etc. With these scripting languages you can easily leverage WMI to exceute and mointor processes on remotes systems, copy files, updates registry...basically everything that you're trying to accomplish....and it won't cost you anything but the cost of learning these technologies, and there many online resources and which document how to do them. Here is a link to the Microsoft Script Center, its a great start: http://www.microsoft.com/technet/scriptcenter/default.mspx
I wrote this a while back, and used it to maintain my home windows XP desktop for a while:
ssh and telnet on windows
I used the SSH option (not telnet). It worked for my purposes (killing remote tasks, copying files etc.) It uses Cygwin, but you're able to run regular windows commands as well as the bash commands that come with cygwin.
The Software Testing Automation Framework (STAF) is designed for remote access, installing software, transferring files. etc. It's open source and you can write your own service if there isn't one that does what you need. It also has a GUI component for writing, scheduling, queueing and monitoring jobs across a pool of machines.
At the moment the only things it needs to do is to be able to install
applications/patches, and transfer files to and from the machine, and
installing registry patches would be nice as well.
try to download and install eurysco to use in order:
transfer file, applications and patches with multiple-upload feature from eurysco file browser
install applications and patches in silent mode from eurysco command line feature
edit registry from eurysco system-registry feature
http://www.eurysco.com/features