I'm trying to integrate Kafka Connect FS & Source SFTP with Username & Passwordless Entry(Private key). But I'm getting AUTH Failure with below settings.
Its completely working fine with username:password#hostname:port format for a Test SFTP Location, but actual source doesnt allow password based authentication.
Even i tried, "fs.sftp.keyfile". but no luck.
Here is my Property file:
name=SourceConnector
connector.class=com.github.mmolimar.kafka.connect.fs.FsSourceConnector
tasks.max=1
policy.fs.fs.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem
fs.uris=sftp://username:#hostname:22/home/user/output/
fs.sftp.keyfile=/home/user/.ssh/id_rsa
topic=sampletopic
policy.class=com.github.mmolimar.kafka.connect.fs.policy.CronPolicy
policy.recursive=true
file_reader.delimited.settings.data_type_mapping_error=false
file_reader.delimited.settings.allow_nulls=true
policy.regexp=^SOURCE_1.*.gz$
policy.batch_size=0
policy.cleanup=none
file_reader.class=com.github.mmolimar.kafka.connect.fs.file.reader.CsvFileReader
file_reader.batch_size=3000
policy.cron.expression=0/30 * * ? * * *
file_reader.delimited.compression.type=gzip
Please help me to connect with private key. Thanks
ERROR FsSourceTask Cannot retrieve files to process from the FS: [[]]. There was an error executing the policy but the task tolerates this and continues: com.jcraft.jsch.JSchException: Auth fail
I'm able to resolve this error with below config:
policy.fs.fs.sftp.keyfile=/home/user/.ssh/id_rsa
Related
How can I support Kerberos based authentication with the ning http client?
I am extending existing code which has support for NTLMAuth and I want to be able to include support for Kerberos, which is used on some of the websites that I need to test.
I want to be able to put in the user and password programmatically, I do not want to use a keyTab, or setup krb5 configuration on the system where this is running.
I have the following code block;
import com.ning.http.client.RequestBuilder;
import com.ning.http.client.Realm.RealmBuilder;
....
RealmBuilder myRealmBuilder = new RealmBuilder()
.setScheme(AuthScheme.KERBEROS)
.setUsePreemptiveAuth(true)
.setNtlmDomain(getDomain())
.setNtlmHost(getHost())
.setPrincipal(getUsername())
.setPassword(getUserPassword()));
RequestBuilder rb = new RequestBuilder()
.setMethod(site.getMethod())
.setUrl(site.getUrl())
.setFollowRedirects(site.isFollowRedirects())
.setRealm(myRealmBuilder),
site)
ยดยดยด
Currently I get the error response:
FAILED: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm)
Does anyone have an good example of how to do this correctly?
I am trying to run a simple program of jcloud. The program is as follows:
String provider = "openstack-nova";
String identity = "Tenant:usename"; // tenantName:userName
String credential = "pass";
novaApi = ContextBuilder.newBuilder(provider).endpoint("http://openstack.infosys.tuwien.ac.at/identity/v2.0")
.credentials(identity, credential).modules(modules).buildApi(NovaApi.class);
regions = novaApi.getConfiguredRegions();
The openstack.infosys is connect via SOCKS proxy on port 7777. I have also enlisted the same on eclipse(Window->Preferences->General->Network Config->SOCKS(Manual)) . However, everytime I run the code I get the following error:
ERROR o.j.h.i.JavaUrlHttpCommandExecutorService - Command not considered safe to retry because request method is POST:
Which is then caused by
Caused by: java.net.SocketTimeoutException: connect timed out
I am able to access the horizon web interface of the same without any issues.
Can someone please help me in understanding what is the possible problem.
You need to tell Apache jclouds about your proxy configuration when creating the context. Have a look at these properties, and pass the ones you need to the overrides method of the ContextBuilder:
Proxy type
Proxy host
Proxy port
Proxy user
Proxy password
My goal: implement SSO on a java-based web application.
My problem: I'm not a security guy...
After some investigation I found that spring security kerberos extension is what I need (also looked into apache shiro but could only find example with a login page).
I used the samples in the following project:
https://github.com/spring-projects/spring-security-kerberos/tree/master/spring-security-kerberos-sample
I realized that I need to create a keytab. When I tried to use the keytab I got the following error:
javax.security.auth.login.LoginException: Unable to obtain password from user
Looking for some details about this error I saw that it could result from a wrong keytab location, but this is not the case here - I debugged into the source code and saw that the keytab file is loaded.
So I decided to check my keytab and see if it's ok.
First, this is last command (after a long evolution) I used to create my keytab:
ktpass /out http-web.keytab /mapuser MyUser#MYDOMAIN.COM /princ HTTP/MyUser#MYDOMAIN.COM /pass MyPass /ptype KRB5_NT_PRINCIPAL
Of course I created an SPN for MyUser with the following command:
setspn -a HTTP/MyUser#MYDOMAIN.COM MYDOMAIN.COM\MyUser
I tested the spn with the the following:
setspn -Q HTTP/MyUser#MYDOMAIN.COM
And got a successful result:
Checking domain DC=mydomain,DC=com CN=MyUser,OU=MyOrg,DC=mydomain,DC=com
HTTP/MyUser
HTTP/MyUser#MYDOMAIN.COM
Existing SPN found!
Now I wanted to test if I can obtain a ticket for MyUser by running the following command:
kinit MyUser#MYDOMAIN.COM
I got a successful result ("new ticket is stored in cache file....")
Now I wanted to test it with my keytab:
kinit MyUser#MYDOMAIN.COM -k -t http-web.keytab
Got the following exception:
Exception: krb_error 0 Do not have keys of types listed in default_tkt_enctypes available; only have keys of following type: No error KrbException: Do not have keys of types listed in default_tkt_enctypes available; only have keys of following type:
I used klist tool to see if my keytab contains any keys:
klist -e -K -k -t http-web.keytab
Got the following result:
KVNO: 8
Key type: 23
Key: 0x47bf8039a8506cd67c524a03ff84ba4e
Time stamp: Jan 01, 1970 02:00
As a last desperate attempt, I checked the following account options for MyUser:
Use Kerberos DES encryption types for this account
The account suppoerts Kerberos AES 128 bit encryption
The account suppoerts Kerberos AES 256 bit encryption
I'm not sure if setting these options caused it, but now when I run
kinit MyUser#MYDOMAIN.COM
I get the following error:
Exception: krb_error 14 KDC has no support for encryption type (14) KDC has no support for encryption type
KrbException: KDC has no support for encryption type (14)
So I'm kind of desperate here, I don't really know what I'm doing. It's all a matter of trial and error (mostly error).
If anyone can guide me through here it would be much appreciated.
Thanks,
Lior
Turned out to be a stupid mistake.
I injected in spring the user account instead of the principal name as the servicePrincipal.
I am facing this exception when I trying to create the sp.xml using the ssoadmin :
com.sun.identity.cli.CLIException: AdminTokenAction: FATAL ERROR: Cannot obtain Application SSO token.
Check AMConfig.properties for the following properties
com.sun.identity.agents.app.username
com.iplanet.am.service.password
at com.sun.identity.cli.LogWriter.log(LogWriter.java:109)
at com.sun.identity.cli.Authenticator.ldapLogin(Authenticator.java:170)
at com.sun.identity.cli.AuthenticatedCommand.ldapLogin(AuthenticatedCommand.java:144)
at com.sun.identity.federation.cli.CreateMetaDataTemplate.handleRequest(CreateMetaDataTemplate.java:113)
at com.sun.identity.cli.SubCommand.execute(SubCommand.java:291)
at com.sun.identity.cli.CLIRequest.process(CLIRequest.java:212)
at com.sun.identity.cli.CLIRequest.process(CLIRequest.java:134)
at com.sun.identity.cli.CommandManager.serviceRequestQueue(CommandManager.java:573)
at com.sun.identity.cli.CommandManager.(CommandManager.java:171)
at com.sun.identity.cli.CommandManager.main(CommandManager.java:148)
And I also tried adding something like this in the ssoamdin.bat :
-D"com.iplanet.am.naming.map.site.to.server=https://lb.example.com:443/openam=http://server1.example.com:8080/openam"
But the same exception...
How to fix it?
Thanks in advance,
The 'map-to-site' property is only needed if you have a site configured an the host where you run ssoadm is not able to talk to the siteURL.
You may set -Dcom.iplanet.services.debug.level=message -Dcom.iplanet.services.debug.directory=WRITABLE_EXISTING_DIRECTORY' as JVM options within ssoadm.bat.
You may then look into the debug directory you should find a pointer what's wrong.
The above got sorted in my case when i went to the ssoadmin folder openam/bin and found the ssoadm.bat file and opened it in the edit mode and added the following two lines in the java comaand :
-D"javax.net.ssl.trustStore=F:\tomcatsslkeystore" (tomcat keystore path)
-D"javax.net.ssl.trustStorePassword=tomcatsslkeystore" (tomcat keystore password)
i am trying to configure Google App Engine on Eclipse and use it to run a python application locally (on the local host):
for this i used following tutorial as a guide:
http://www.mkyong.com/google-app-engine/google-app-engine-python-hello-world-example-using-eclipse/
i followed the steps properly but when i try to use the configuration i get errors the console output is:
Console Output:
C:\Program Files (x86)\Google\google_appengine\google\appengine\api\search\search.py:232: UserWarning: DocumentOperationResult._code is deprecated. Use OperationResult._code instead.
'Use OperationResult.%s instead.' % (name, name))
C:\Program Files (x86)\Google\google_appengine\google\appengine\api\search\search.py:232: UserWarning: DocumentOperationResult._CODES is deprecated. Use OperationResult._CODES instead.
'Use OperationResult.%s instead.' % (name, name))
WARNING 2012-06-20 14:53:01,451 rdbms_mysqldb.py:74] The rdbms API is not available because the MySQLdb library could not be loaded.
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py", line 126, in
run_file(file, globals())
File "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py", line 122, in run_file
execfile(script_path, globals_)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_main.py", line 694, in
sys.exit(main(sys.argv))
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_main.py", line 582, in main
root_path, {}, default_partition=default_partition)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 3142, in LoadAppConfig
raise AppConfigNotFoundError
google.appengine.tools.dev_appserver.AppConfigNotFoundError
The configuration i am using is:
Windows 7 64 bit
python 2.7
Eclipse Helios
What could be the possible mistake in configuring the GAE?
Additional info : when i try to use the project with GAE manually(ie by using the launcher) it works
Update:
i experimented and discovered that since the workstation and the python installation folder is not in the same directory i get these errors
got the hint from here:
File
"C:\Program Files(x86)\Google\google_appengine\google\appengine\tools\dev_appserver.py"
line 582, in main
root_path, {}, default_partition=default_partition)
but when i made another workspace in the same partition i got this as console output and the the local host is still not working
output
C:\Program Files (x86)\Google\google_appengine\google\appengine\api\search\search.py:232: UserWarning: DocumentOperationResult._code is deprecated. Use OperationResult._code instead.
'Use OperationResult.%s instead.' % (name, name))
C:\Program Files (x86)\Google\google_appengine\google\appengine\api\search\search.py:232: UserWarning: DocumentOperationResult._CODES is deprecated. Use OperationResult._CODES instead.
'Use OperationResult.%s instead.' % (name, name))
WARNING 2012-06-20 17:20:56,719 rdbms_mysqldb.py:74] The rdbms API is not available because the MySQLdb library could not be loaded.
Runs a development application server for an application.
dev_appserver.py [options]
Application root must be the path to the application to run in this server.
Must contain a valid app.yaml or app.yml file.
Options:
--address=ADDRESS, -a ADDRESS Address to which this server should bind(Defaultlocalhost).
--clear_datastore, -c Clear the Datastore on startup. (Default false)
--debug, -d Use debug logging. (Default false)
--help, -h View this helpful message.
--port=PORT, -p PORT Port for the server to run on. (Default 8080)
--allow_skipped_files Allow access to files matched by app.yaml's
skipped_files (default False)
--auth_domain Authorization domain that this app runs in.
(Default gmail.com)
--backends Run the dev_appserver with backends support
(multiprocess mode).
--blobstore_path=DIR Path to directory to use for storing Blobstore
file stub data.
--clear_prospective_search Clear the Prospective Search subscription index
(Default false).
--datastore_path=DS_FILE Path to file to use for storing Datastore file
stub data.
(Defaultc:\users\anukoo~1\appdata\local\temp\dev_appserver.datastore)
--debug_imports Enables debug logging for module imports, showing
search paths used for finding modules and any
errors encountered during the import process.
--default_partition Default partition to use in the APPLICATION_ID.
(Default dev)
--disable_static_caching Never allow the browser to cache static files.
(Default enable if expiration set in app.yaml)
--disable_task_running When supplied, tasks will not be automatically
run after submission and must be run manually
in the local admin console.
--enable_sendmail Enable sendmail when SMTP not configured.
(Default false)
--high_replication Use the high replication datastore consistency
model. (Default false).
--history_path=PATH Path to use for storing Datastore history.
(Default c:\users\anukoo~1\appdata\local\temp\dev_appserver.datastore.history)
--multiprocess_min_port When running in multiprocess mode, specifies the
lowest port value to use when choosing ports. If
set to 0, select random ports.
(Default 9000)
--mysql_host=HOSTNAME MySQL database host.
Used by the Cloud SQL (rdbms) stub.
(Default 'localhost')
--mysql_port=PORT MySQL port to connect to.
Used by the Cloud SQL (rdbms) stub.
(Default 3306)
--mysql_user=USER MySQL user to connect as.
Used by the Cloud SQL (rdbms) stub.
(Default )
--mysql_password=PASSWORD MySQL password to use.
Used by the Cloud SQL (rdbms) stub.
(Default '')
--mysql_socket=PATH MySQL Unix socket file path.
Used by the Cloud SQL (rdbms) stub.
(Default '')
--persist_logs Enables storage of all request and application
logs to enable later access. (Default false).
--require_indexes Disallows queries that require composite indexes
not defined in index.yaml.
--show_mail_body Log the body of emails in mail stub.
(Default false)
--skip_sdk_update_check Skip checking for SDK updates. If false, fall back
to opt_in setting specified in .appcfg_nag
(Default false)
--smtp_host=HOSTNAME SMTP host to send test mail to. Leaving this
unset will disable SMTP mail sending.
(Default '')
--smtp_port=PORT SMTP port to send test mail to.
(Default 25)
--smtp_user=USER SMTP user to connect as. Stub will only attempt
to login if this field is non-empty.
(Default '').
--smtp_password=PASSWORD Password for SMTP server.
(Default '')
--task_retry_seconds How long to wait in seconds before retrying a
task after it fails during execution.
(Default '30')
--use_sqlite Use the new, SQLite based datastore stub.
(Default false)
Invalid arguments
seems like the arguments to dev_appserver.py are incorrect any ideas
If you did the same mistake as mine,then its a very high chance that you have space in the name of your directory
The warnings about deprecated things in search can be ignored. As can the message about the rdbms API if you don't plan on using that.
AppConfigNotFoundError happens when there is no app.yaml in the directory passed to dev_appserver.py. If you followed those instructions, then your app.yaml would be in the 'src' directory, and the 'program arguments' in the build command would be ${project_loc}/src - is this the case? When you run from the command-line, and see it work, what command are you running, and from what location?