What is the best way to achieve below in scala?
cat <file> | ssh user#host "aws s3 cp - s3://"
We need to go through a particular bastion host which has the privilege to upload to S3. We can easily do this in shell but not sure what is the best way to do in Scala. Is there any particular library I should use instead of using SEQ()?
I checked scala-ssh but it seems that you can't use pipe input to SSH. And also, we don't want to do "scp and then aws cp" since it will be two hops.
Please advise me.
Thanks,
Try this:
import sys.process._
val command = "cat <file>" #| """ssh user#host "aws s3 cp - s3://"""
command.!
Well, you can use Cable to do this
As an example, we pipe data to remote host tasks like this:
import cable.zssh.TypeDef._
import HostConnS._
import cable.zssh.Zssh._
val putFileI = Action("my-server", action = scriptIO("cat -", new File("My file.txt")))
val putStringI = Action("my-server", username = Some("user1"), action = scriptIO("cat -", "String data"))
val putStream = Action("my-server", action = scriptIO("cat -", inputStream))
Moreover, in the above task constructions, we just name the host to connect to, in this case
"my-server", and leave the username, password, or private key omitted, these default to the
current system user and private key, which is just like the way SSH does, if you get your machine's
ssh-copy-id -ish thing configured, indeed you can just name the host to connect to.
More info:
https://github.com/zhongwm/cable/blob/master/wiki/Piping_streams_to_remote_task.md
Related
I'm trying to list the folder in aws s3 and get only the filename out of it. The nested unix commands is not getting executed in Spark-shell and throwing error. I know we have other ways to do it by importing org.apache.hadoop.fs._
The command that I'm trying are :
import sys.process._
var cmd_exec = "aws s3 ls s3://<bucket-name>/<folder-name>/"
cmd_exec !!
If I execute it by nesting the cut command to the ls. It's throwing error.
import sys.process._
var cmd_exec = "aws s3 ls s3://<bucket-name>/<folder-name>/ | cut -d' ' -f9-"
cmd_exec !!
Error message: Unknown options: |,cut,-d',',-f9-
java.lang.RuntimeException: Nonzero exit value: 255
Any suggestion please?
AFAIK this is natural.
import scala.sys.process._
val returnValue: Int = Process("cat mycsv.csv | grep -i Lazio")!
above code also wont work...
| is redirect operator to execute another command. so instead of that....
capture the output and execute one more time..
you can see this article - A Scala shell script example as well.. where scala program can be executed as shell script... it might be useful.
TIY!
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 months ago.
The community reviewed whether to reopen this question 2 months ago and left it closed:
Original close reason(s) were not resolved
Improve this question
I forgot the password of a dev instance (irresponsible.. yeah, I am working on it). I have the connection saved in my DBeaver with the password. I am still able to connect using that connection. DBeaver is not showing it in plain text. Is there anyway I can retrieve the password? Asking DBA to reset the password is the last resort. I tried to copy paste to a notepad, copying is disabled apparently.
For DBeaver 6.1.3+
The credential file is located ~/Library/DBeaverData/workspace6/General/.dbeaver/credentials-config.json (I was on Mac) and it follows a different encryption strategy than it's predecessors. Please refer the next answer to see how to decrypt. It works like a charm.
Pre- DBeaver 6.1.3
Follow these steps (My DBeaver version was 3.5.8 and it was on Mac OsX El Capitan)
Locate the file in which DBeaver stores the connection details. For
me, it was in this location
~/.dbeaver/General/.dbeaver-data-sources.xml. This file is hidden,
so keep that in mind when you look for it.
Locate your interested Datasource Definition node in that file.
Decrypt the password: Unfortunately, everything is in plain text except password; Password is in some kind of Encrypted form. Decrypt it to plain-text using this tool.
Or
I put together a quick and dirty Java program by copying core of DBeaver's method for decrypting the password. Once you have the Encrypted password string, just execute this program, it will convert the password to plain text and prints it
How to run it
On Line Number 13, just replace OwEKLE4jpQ== with whatever encrypted password you are finding in .dbeaver-data-sources.xml file for your interested datasource. Compile it and run it, it will print the plain-text password.
https://github.com/jaisonpjohn/dbeaver-password-retriever/blob/master/SimpleStringEncrypter.java
Apparently, this is a "Popular" mistake. So I have deployed an AWS lambda function with the aforementioned code. Use this at your own risk, you will never know whether I am logging your password or not
curl https://lmqm83ysii.execute-api.us-west-2.amazonaws.com/prod/dbeaver-password-decrypter \
-X POST --data "OwEKLE4jpQ=="
Even better, here is the UI https://bugdays.com/dbeaver-password-decrypter. This goes without saying, use this at your own risk
This can be done with OpenSSL:
openssl aes-128-cbc -d \
-K babb4a9f774ab853c96c2d653dfe544a \
-iv 00000000000000000000000000000000 \
-in credentials-config.json | \
dd bs=1 skip=16 2>/dev/null
Example for macOS in one line:
openssl aes-128-cbc -d -K babb4a9f774ab853c96c2d653dfe544a -iv 00000000000000000000000000000000 -in "${HOME}/Library/DBeaverData/workspace6/General/.dbeaver/credentials-config.json" | dd bs=1 skip=16 2>/dev/null
For Linux, change the above path to ~/.local/share/DBeaverData/workspace6/General/.dbeaver/credentials-config.json.
The key is from the source and is converted to hexadecimal. This can be done in Python:
>>> import struct
>>> struct.pack('<16b', -70, -69, 74, -97, 119, 74, -72, 83, -55, 108, 45, 101, 61, -2, 84, 74).hex()
'babb4a9f774ab853c96c2d653dfe544a'
Edit: I've published the script for this here.
For DBeaver 6.1.3+ the creds are stored in a "json" file now with different encryption.
This seemed to do the job for me:
import javax.crypto.*;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.*;
public class DecryptDbeaver {
// from the DBeaver source 8/23/19 https://github.com/dbeaver/dbeaver/blob/57cec8ddfdbbf311261ebd0c7f957fdcd80a085f/plugins/org.jkiss.dbeaver.model/src/org/jkiss/dbeaver/model/impl/app/DefaultSecureStorage.java#L31
private static final byte[] LOCAL_KEY_CACHE = new byte[] { -70, -69, 74, -97, 119, 74, -72, 83, -55, 108, 45, 101, 61, -2, 84, 74 };
static String decrypt(byte[] contents) throws InvalidAlgorithmParameterException, InvalidKeyException, IOException, NoSuchPaddingException, NoSuchAlgorithmException {
try (InputStream byteStream = new ByteArrayInputStream(contents)) {
byte[] fileIv = new byte[16];
byteStream.read(fileIv);
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
SecretKey aes = new SecretKeySpec(LOCAL_KEY_CACHE, "AES");
cipher.init(Cipher.DECRYPT_MODE, aes, new IvParameterSpec(fileIv));
try (CipherInputStream cipherIn = new CipherInputStream(byteStream, cipher)) {
return inputStreamToString(cipherIn);
}
}
}
static String inputStreamToString(java.io.InputStream is) {
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}
public static void main(String[] args) throws Exception {
if (args.length != 1) {
System.err.println("syntax: param1: full path to your credentials-config.json file");
System.exit(1);
}
System.out.println(decrypt(Files.readAllBytes(Paths.get(args[0]))));
}
}
Pass it the path of your credentials-config.json file on local filesystem, for me it was
Compile it
$ javac DecryptDbeaver.java
Now run it [adjusts the paths to target your credentials-config.json file]
$ java DecryptDbeaver ~/Library/DBeaverData/workspace6/General/.dbeaver/credentials-config.json
Or if java 11+:
$ java DecryptDbeaver.java ~/Library/DBeaverData/workspace6/General/.dbeaver/credentials-config.json
It will output to the console the user+pass for connections.
{"postgres-jdbc-some-id":{"#connection":{"user":"your_user_name","password":"your_password"...
If you don't recognize which password goes to which DB based on username, you must cross link the id names it also outputs initially to the sibling data-sources.json file (which should already be present and unencrypted and contains database coordinates).
For Windows users (Tested Version 7.3.4, also tested 22.2.3)
Press File > Export > DBeaver > Project
Change the name of the export file to .zip, and unzip
Download OpenSSL, and copy \projects\General\.dbeaver\credentials-config.json into the same folder as the bin directory of openssl
Then run:
openssl aes-128-cbc -d -K babb4a9f774ab853c96c2d653dfe544a -iv 00000000000000000000000000000000 -in "credentials-config.json"
If you have WSL installed, this command can also be run from a Linux install with openssl available (which openssl) from any directory within the Linux install (Tested with Ubuntu on WSL2 copied file to \\wsl$\Ubuntu\home\me\dbeaver\credentials).
It will output to the terminal as default, if you need it in a file add > chosen_filename.json to the command.
This is the command to get the decrypted version of dbeaver credentials file on your desired destination path:
openssl aes-128-cbc -d \
-K babb4a9f774ab853c96c2d653dfe544a \
-iv 00000000000000000000000000000000 \
-in {path for the encrypted credentials file} > \
{your desired destination file}
{your desired destination file} e.g. ~/Desktop/dbeaver-credentials.json
You'll find dbeaver-credentials.json file on Desktop. But this file will have list of only usernames & passwords with some connection stanza (like mysql5-17be86ca5ea-294e2a427af47fc4). No db or server names will be there. You've to find the connection against the object id.
For Ubuntu snap package dbeaver-ce,
{path for the encrypted credentials file} = ~/snap/dbeaver-ce/current/.local/share/DBeaverData/workspace6/General/.dbeaver/credentials-config.json
if there is package declaration just compile
javac -d . SimpleStringEncrypter.java
it will put it in the correct directory structure under the current directory
Then you can give
java -cp . packagename.SimpleStringEncrypter and it will run.
Basic java.
Anyway this program is fantastic and saved a lot of time for me.
For Linux OS users, run this in Terminal:
openssl aes-128-cbc -d -K babb4a9f774ab853c96c2d653dfe544a -iv 00000000000000000000000000000000 -in "path_to/credentials-config.json" | dd bs=1 skip=16 2>/dev/null
Just replace the string "path_to/credentials-config.json" with your actual path to that file and you'll get something like this:
{"mysql8-17e009389a8-5fc414bd64e183f4":{"#connection":{"user":"root","password":"root"}},"mysql8-18099236fdf-3c3fc761c6fdde":{"#connection":{"user":"user.name","password":"your_secret_password"},"network/ssh_tunnel":{"user":"sql","jumpServer0.password":""}}}%
Look at this:
docker run -d -p 18080:8080 --name crack-dbeaver-password-18080 geekyouth/crack-dbeaver-password
https://github.com/geekyouth/crack-dbeaver-password
If you dont want all the saved connections
Just remove the --\DBeaverData\workspace6\General folder from the file system
so that it can not ask any password again.
and the workspace data will be lost.
You will loose all the custom settings and preferences.
How can I change settings in pg_hba.conf and postgresql.conf either from the command-line or programatically (especially from fabric or fabtools)?
I already found set_config, but that does not seem to work for parameters which require a server restart. The parameters to change are listen_addresses in postgresql.conf and a new line in pg_hba.conf, so connections from our sub-network will be accepted.
This is needed to write deployment scripts using fabric. It is not an option to copy template-files which then override the existing *.conf files, because the database server might be shared with other applications which bring their own configuration parameters. Thus, the existing configuration must be altered, not replaced.
Here is the currently working solution, incorporating the hint from a_horse_with_no_name. I paste a snippet from our fabfile.py (it uses require from fabtools, and it runs against Ubuntu):
db_name = env.variables['DB_NAME']
db_user = env.variables['DB_USER']
db_pass = env.variables['DB_PASSWORD']
# Require a PostgreSQL server.
require.postgres.server(version="9.4")
require.postgres.user(db_user, db_pass)
require.postgres.database(db_name, db_user)
# Listen on all addresses - use firewall to block inadequate access.
sudo(''' psql -c "ALTER SYSTEM SET listen_addresses='*';" ''', user='postgres')
# Download the remote pg_hba.conf to a temp file
tmp = tempfile.NamedTemporaryFile()
with open(tmp.name, "w") as f:
get("/etc/postgresql/9.4/main/pg_hba.conf", f, use_sudo=True)
# Define the necessary line in pg_hba.conf.
hba_line = "host all all {DB_ACCEPT_IP}/0 md5".format(**env.variables)
# Search the hba_line in the existing pg_hba.conf
with open(tmp.name, "ra") as f:
for line in f:
if hba_line in line:
found = True
break
else:
found = False
# If it does not exist, append it and upload the modified pg_hba.conf to the remote machine.
if not found:
with open(tmp.name, "a") as f:
f.write(hba_line)
put(f.name, "/etc/postgresql/9.4/main/pg_hba.conf", use_sudo=True)
# Restart the postgresql service, so the changes take effect.
sudo("service postgresql restart")
The aspect I don't like with this solution is that if I change DB_ACCEPT_IP, this will just append a new line and not remove the old one. I am sure a cleaner solution is possible.
I am trying to read a file in Spark Shell that comes with CentOS distribution of Cloudera on my local machine. Following are the commands I have entered in Spark Shell.
spark-shell
val fileData = sc.textFile("hdfs://user/home/cloudera/cm_api.py");
fileData.count
I also tried this statment for reading file:
val fileData = sc.textFile("user/home/cloudera/cm_api.py");
However I am getting
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://quickstart.cloudera:8020/user/cloudera/user/cloudera/cm_api.py
I haven't changed any settings or configurations. What am I doing wrong?
You are missing the leading slash in your url, so the path is relative. To make it absolute, use
val fileData = sc.textFile("hdfs:///user/home/cloudera/cm_api.py")
or
val fileData = sc.textFile("/user/home/cloudera/cm_api.py")
I think you need to put the file in hdfs first: hadoop fs -put, then check the file: hadoop fs -ls, then go spark-shell , val fileData = sc.textFile("cm_api.py")
In "hdfs://user/home/cloudera/cm_api.py", you are missing the hostname of the URI. You should have pass something like "hdfs://<host>:<port>/user/home/cloudera/cm_api.py", where <host> is Hadoop NameNode host and the <port> is, well, port number of Hadoop NameNode, which is 50070 by default.
The error message says hdfs://quickstart.cloudera:8020/user/cloudera/user/cloudera/cm_api.py does not exist. The path looks suspicious! The file you mean is probably at hdfs://quickstart.cloudera:8020/user/cloudera/cm_api.py.
If it is, you can access it by using that full path. Or, if the default file system is configured as hdfs://quickstart.cloudera:8020/user/cloudera/, you can use simply cm_api.py.
You may be confused between HDFS file paths and local file paths. By specifying
hdfs://quickstart.cloudera:8020/user/home/cloudera/cm_api.py
you are saying two things:
1) there is a computer by the name "quickstart.cloudera' reachable via the network (try ping to ensure that is the case), and it is running HDFS.
2) the HDFS file system contains a file at /user/home/cloudera/cm_api.py (try 'hdfs dfs -ls /user/home/cloudera/' to verify this
If you are trying to access a file on the local file system you have to use a different URI:
file:///user/home/cloudera/cm_api.py
I am using Net:Appliance::Session to login to a remote Unix server, but am not able to connect. Below is my code and the debug output:
my $s = Net::Appliance::Session->new({
personality => 'Bash',
transport => 'SSH',
host => $host,
});
$s->set_global_log_at('debug');
try {
print "Trying to connect\n";
$s->connect({ username => $user, password => $pass });
print "Executing command\n";
print $s->cmd($cmd);
}
catch {
warn "failed to execute command: $_";
}
finally {
$s->close;
};
And the output is:
Trying to connect
[ 0.019420] pr finding prompt
[ 0.028553] tr creating Net::Telnet wrapper for ssh
[ 0.031377] tr connecting with: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -l user ...
[ 3.151205] du SEEN:
Warning: Permanently added '...' (RSA) to the list of known hosts.
[ 3.183935] pr failed: [Can't call method "isa" on an undefined value at /usr/lib/perl5/site_perl/5.14/Net/CLI/Interact/Phrasebook.pm line 247.
], sending WAKE_UP and trying again
[ 3.184943] pr finding prompt
[ 4.898408] du SEEN:
Warning: Permanently added '...' (RSA) to the list of known hosts.
Password:
[ 4.920447] pr failed to find prompt! wrong phrasebook?
failed to execute command: Warning: Permanently added '...' (RSA) to the list of known hosts.
Password:
...propagated at /usr/lib/perl5/site_perl/5.14/Net/CLI/Interact/Role/Prompt.pm line 127.
When I login through Putty, I get the following response and can login successfully:
login as: user
Using keyboard-interactive authentication.
Password:
I cannot figure out what I am doing wrong. Any help is appreciated.
EDIT: I think I should mention that I am using Cygwin for this. I have manually logged in to the remote server and the keys in my .ssh/known_hosts file are also set, but still get the RSA error when running this program in Cygwin. I saw this question in SO: "Warning: Permanently added to the list of known hosts” message from Git and added the line UserKnownHostsFile ~/.ssh/known_hosts to my config file, but the error refuses to go away.
EDIT2: When I use the -vvv option in the above program, I get the following output:
Trying to connect
[ 0.020327] pr finding prompt
[ 0.062541] tr creating Net::Telnet wrapper for ssh
[ 0.063709] tr connecting with: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -vvv -l user 1.1.1.1
[ 0.731041] du SEEN:
OpenSSH_6.2p2
[ 0.851829] pr failed: [Can't call method "isa" on an undefined value at /usr/lib/perl5/site_perl/5.14/Net/CLI/Interact/Phrasebook.pm line 247.
], sending WAKE_UP and trying again
[ 0.852459] pr finding prompt
[ 0.852748] du SEEN:
OpenSSH_6.2p2, OpenSSL 1.0.1e 11 Feb 2013
[ 0.863739] pr failed to find prompt! wrong phrasebook?
failed to execute command: OpenSSH_6.2p2, OpenSSL 1.0.1e 11 Feb 2013
...propagated at /usr/lib/perl5/site_perl/5.14/Net/CLI/Interact/Role/Prompt.pm line 127.
The Net::Appliance::Session module is using is set of matching patterns called "Phrasebook" to guess password query output, command ending prompt, ...
In your case, there are 2 major issue and one minor/cosmetic one:
Net::Appliance::Session rely on connection profile. The correct one is named "bash" and not "Bash"
The bash default phrasebook (located in "~site_perl/Net/CLI/Interact/phrasebook/unix/bash/pb") is targeting ssh/bash based appliance and is not matching your everyday unix server behavior:
prompt user
match /[Uu]sername: $/
prompt pass
match /password(?: for \w+)?: $/
prompt generic
match /\w+#.+\$ $/
prompt privileged
match /^root#.+# $/
macro begin_privileged
send sudo su -
match pass or privileged
macro end_privileged
send exit
match generic
macro disconnect
send logout
As you can see, both "generic" and "pass" prompt does not match your usual linux password and prompt. You will need to adjust it to your needs:
create a library structure by creating some nested directory: "mylib\mybash\"
make a copy of the "bash" phrasebook to that nested directory and edit it to match your unix server behaviour.
There is also the ssh warning output:
Warning: Permanently added '...' (RSA) to the list of known hosts.
You just need to set ssh warnings to off using either the "-q" or "-o LogLevel=quiet" options to the ssh calling options.
So, in the end, your code would look like that:
my $s = Net::Appliance::Session->new
({ add_library => 'mylib',
personality => 'mybash',
transport => 'SSH',
host => $host,
connect_options => { opts => [ '-q', ], },
});
$s->set_global_log_at('debug');
try {
print "Trying to connect\n";
$s->connect({ username => $user, password => $pass });
print "Executing command\n";
print $s->cmd($cmd);
}
catch {
warn "failed to execute command: $_";
}
finally {
$s->close;
};
With a phrasebook like this one (quickly tuned to my freebsd server):
prompt user
match /[Uu]sername: $/
prompt pass
match /[Pp]assword:\s*$/
prompt generic
match /\w+#.+[\$>] $/
prompt privileged
match /^root#.+# $/
macro begin_privileged
send sudo su -
match pass or privileged
macro end_privileged
send exit
match generic
macro disconnect
send logout
macro paging
send terminal length %s
NOTE:
About "Net::Appliance::Session" vs "Net::OpenSSH":
both modules are handling ssh connectionx fine
"Net::Appliance::Session" is more "cisco/whatever-appliance" oriented, but should permit easily to connect to a server, execute 1 command, get its result, then go root and execute another command (very handy if you don't have direct root access from ssh)
"Net::OpenSSH" is handling command execution though ssh on 1 command only basis, that is it execute a command, get its result and exit. No direct way to set an environment, go root and execute the command (you need to use a wrapper like Net::Telnet on it to do that)
"Net::OpenSSH" requires a fairly recent version of openssh client and does not work on Windows, not even under Cygwin (see "Net::OpenSSH" manual).
Try using Net::OpenSSH instead. It would be easier to use and more reliable when talking to a Unix server.