Using an existing working Java example I am trying to write a python equivalent of the producer using python-kafka and confluent_kafka library. How do I configure sasl.jass.config in python with the information like that in Java below?
import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
...
Properties props = new Properties();
...
props.put("sasl.jaas.config", "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"<Kafka_Username>\" password=\"<Kafka_Password>\";");
Producer<String, String> producer = new KafkaProducer<>(props);
This works for me
from confluent_kafka import Producer
SECURITY_PROTOCOL = "SASL_SSL"
SASL_MECHANISM = "PLAIN"
conf = {
'bootstrap.servers': SERVERS,
'sasl.mechanisms': SASL_MECHANISM,
'security.protocol': SECURITY_PROTOCOL,
'sasl.username': SASL_USERNAME,
'sasl.password': SASL_PASSWORD,
...
}
consumer = Producer(conf)
Related
How do I configure Micronaut app using Vert.x and testcontainers? I'm trying:
application-test.yml
datasources:
default:
url: jdbc:tc:mysql:8:///db
driverClassName: org.testcontainers.jdbc.ContainerDatabaseDriver
vertx:
mysql:
client:
uri: jdbc:tc:mysql:8:///db
Tests with micronaut-data-jdbc work, but with micronaut-vertx-mysql-client not work:
Error:
Message: Cannot parse invalid connection URI: jdbc:tc:mysql:8:///db
I'm not very familiar with testecontainers, but it seems like it doesn't come up with a fixed port, so I don't know how to configure the connection URI.
Thanks!
It might be a problem that micronaut-vertx-mysql-client does not support the Testcontainers JDBC URL scheme (hard to say without further logs).
In this case, I would suggest to use Testcontainers with database container objects instead of the special JDBC URL.
I got a solution to the problem:
Micronaut + jdbc hikari + vertx mysql client + flyway mysql
package br.com.app;
import io.micronaut.context.ApplicationContext;
import io.micronaut.context.env.PropertySource;
import io.micronaut.core.util.CollectionUtils;
import io.micronaut.http.HttpRequest;
import io.micronaut.http.HttpStatus;
import io.micronaut.http.client.HttpClient;
import io.micronaut.runtime.EmbeddedApplication;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.MySQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
#Testcontainers
class AuthTest {
static Logger log = LoggerFactory.getLogger(AuthTest.class.getName());
#Container
static MySQLContainer mysql = new MySQLContainer("mysql:8");
private HttpClient client;
private static EmbeddedApplication application;
private static ApplicationContext context;
#BeforeAll
public static void initTests(){
log.info("Mysql is running {}, port {}", mysql.isRunning(), mysql.getFirstMappedPort());
var port = mysql.getFirstMappedPort();
var host = mysql.getHost();
var database = mysql.getDatabaseName();
var user = mysql.getUsername();
var password = mysql.getPassword();
var url = String.format("jdbc:mysql://%s:%s/%s", host, port, database);
application = ApplicationContext.run(EmbeddedApplication.class,
PropertySource.of(
"test",
CollectionUtils.mapOf(
"vertx.mysql.client.port", port,
"vertx.mysql.client.host", host,
"vertx.mysql.client.database", database,
"vertx.mysql.client.user", user,
"vertx.mysql.client.password", password,
"datasources.default.url", url,
"datasources.default.username", user,
"datasources.default.password", password,
"flyway.datasources.default.enabled", true
)
));
context = application.getApplicationContext();
}
#BeforeEach
void beforeEach(){
this.authService = context.getBean(AuthService.class);
this.client = context.getBean(HttpClient.class);
}
#Test
void testItWorks() {
Assertions.assertTrue(application.isRunning());
}
// api tests
}
Help links:
https://dev.to/major13ua/micronaut-integration-testing-using-testcontainers-2e30
https://github.com/major13ua/micronaut-tc/blob/main/src/test/java/com/example/testcontainer/controller/DemoControllerTest.java
I would create a TLS Session in Scala using Akka with mutual authentication between a client and a server. I have created two CA certificate that have to trust the respective certificates incoming from the other part.
Could you give me an exemple of how implement this?
Thank you.
I created a github project which demonstrates mutual authentication with different kind of clients, including Akka. Please have a look here: https://github.com/Hakky54/mutual-tls-ssl
It contains a full example of loading the ssl material into the client and server
A summary what you need to do is:
For the client
Create a key and a certificate and load it into a keystore
Export the certificate
Create a separate keystore for trusted certificates and import server certificate
Load the two keystores into your http client
For the server
Create a key and a certificate and load it into a keystore
Export the certificate
Create a separate keystore for trusted certificates and import client certificate
Load the two keystores into your server
It is not really clear to me what kind of server you are using, but if you are using spring-boot the example configuration would be:
server:
port: 8443
ssl:
enabled: true
key-store: classpath:identity.jks
key-password: secret
key-store-password: secret
trust-store: classpath:truststore.jks
trust-store-password: secret
client-auth: need
Akka requires a pre-configured instance of SSLContext to be able to configure HTTPS. An example of creating a client with https options would be the code snippet below.
import akka.actor.ActorSystem;
import akka.http.javadsl.ConnectionContext;
import akka.http.javadsl.Http;
import akka.http.javadsl.HttpsConnectionContext;
import com.typesafe.config.ConfigFactory;
import javax.net.ssl.SSLContext;
import java.util.Optional;
class App {
public static void main(String[] args) {
ActorSystem actorSystem = ActorSystem.create(
App.class.getSimpleName(),
ConfigFactory.defaultApplication(App.class.getClassLoader())
);
SSLContext sslContext = ...; //Initialized SSLContext
Http http = Http.get(actorSystem);
HttpsConnectionContext httpsContext = ConnectionContext.https(
sslContext,
Optional.empty(),
Optional.empty(),
Optional.empty(),
Optional.of(sslContext.getDefaultSSLParameters()));
http.setDefaultClientHttpsContext(httpsContext);
}
}
There are couple of libraries which provides easy to use utility/factory/builder classes to help you to create a SSLContext.
Apache SSLContextBuilder
Jetty SslContextFactory
SSLContext-Kickstart
There could be a bunch other libraries which provide similar functionality, but I am only aware of these three. By the way the sslcontext-kickstart is a library which is maintained by me.
Below is an overview of four ways to load the keystores and create an SSLContext. Vanilla Java and by using the three libraries.
import io.netty.handler.ssl.SslContextBuilder;
import nl.altindag.sslcontext.SSLFactory;
import org.apache.http.ssl.SSLContextBuilder;
import org.eclipse.jetty.util.ssl.SslContextFactory;
import javax.net.ssl.*;
import java.io.File;
import java.io.InputStream;
import java.security.KeyStore;
import java.security.SecureRandom;
import java.util.Objects;
class SslExample {
public static void main(String[] args) throws Exception {
//Traditional flow of creating sslContext
String keyStorePath = "keystore.p12";
String trustStorePath = "truststore.p12";
char[] keyStorePassword = "secret".toCharArray();
char[] trustStorePassword = "secret".toCharArray();
KeyStore keyStore = KeyStore.getInstance("PKCS12");
KeyStore trustStore = KeyStore.getInstance("PKCS12");
try(InputStream keyStoreInputStream = SslExample.class.getClassLoader().getResourceAsStream(keyStorePath);
InputStream trustStoreInputStream = SslExample.class.getClassLoader().getResourceAsStream(trustStorePath)) {
Objects.requireNonNull(keyStoreInputStream);
Objects.requireNonNull(trustStoreInputStream);
keyStore.load(keyStoreInputStream, keyStorePassword);
trustStore.load(trustStoreInputStream, trustStorePassword);
}
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagerFactory.init(keyStore, keyStorePassword);
KeyManager[] keyManagers = keyManagerFactory.getKeyManagers();
TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(trustStore);
TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
sslContext.init(keyManagers, trustManagers, new SecureRandom());
//creating sslContext with Apache SSLContextBuilder
SSLContext sslContext1 = SSLContextBuilder.create()
.loadKeyMaterial(new File("keystore.p12"), "secret".toCharArray(), "secret".toCharArray())
.loadTrustMaterial(new File("truststore.p12"), "secret".toCharArray())
.build();
//creating sslContext with Jetty SslContextFactory
SslContextFactory.Client sslContextFactory = new SslContextFactory.Client();
sslContextFactory.setKeyStorePath("keystore.p12");
sslContextFactory.setKeyStorePassword("secret");
sslContextFactory.setTrustStorePath("truststore.p12");
sslContextFactory.setTrustStorePassword("secret");
sslContextFactory.start();
SSLContext sslContext2 = sslContextFactory.getSslContext();
//creating sslContext with sslcontext-kickstart
SSLFactory sslFactory = SSLFactory.builder()
.withIdentity("keystore.p12", "secret".toCharArray())
.withTrustStore("truststore.p12", "secret".toCharArray())
.build();
SSLContext sslContext3 = sslFactory.getSslContext();
}
}
It is in java, but IntelliJ Idea provides a handy translate function to scala when pasting the code snippet.
I am trying to connect to Neo4j from Spark using neo4j-spark-connector. I am facing an authentication issue when I try to connect to the Neo4j org.neo4j.driver.v1.exceptions.AuthenticationException: Unsupported authentication token, scheme='none' only allowed when auth is disabled: { scheme='none' }
I have checked and the credentials I am passing are correct. Not sure why is it failing.
import org.neo4j.spark._
import org.apache.spark._
import org.graphframes._
import org.apache.spark.sql.SparkSession
import org.neo4j.driver.v1.GraphDatabase
import org.neo4j.driver.v1.AuthTokens
val config = new SparkConf()
config.set(Neo4jConfig.prefix + "url", "bolt://localhost")
config.set(Neo4jConfig.prefix + "user", "neo4j")
config.set(Neo4jConfig.prefix + "password", "root")
val sparkSession :SparkSession = SparkSession.builder.config(config).getOrCreate()
val neo = Neo4j(sparkSession.sparkContext)
val graphFrame = neo.pattern(("Person","id"),("KNOWS","null"), ("Employee","id")).partitions(3).rows(1000).loadGraphFrame
println("**********Graphframe Vertices Count************")
graphFrame.vertices.count
println("**********Graphframe Edges Count************")
graphFrame.edges.count
val pageRankFrame = graphFrame.pageRank.maxIter(5).run()
val ranked = pageRankFrame.vertices
ranked.printSchema()
val top3 = ranked.orderBy(ranked.col("pagerank").desc).take(3)
Can someone please have a look and let me know the reason for the same?
It might be a configuration issue with your neo4j.conf file. Is this line commented out:
dbms.security.auth_enabled=false
I had a similar problem, creating the following spring beans fixed the issue.
#Bean
public org.neo4j.ogm.config.Configuration getConfiguration() {
return new org.neo4j.ogm.config.Configuration.Builder()
.credentials("neo4j", "secret")
.uri("bolt://localhost:7687").build();
}
#Bean
public SessionFactory sessionFactory(org.neo4j.ogm.config.Configuration configuration) {
return new SessionFactory(configuration,
"<your base package>");
}
I'm new to the OPENTEXT Rest API and while I'm able to authenticate/create folders using it, I can't get the document upload to work. The following is the code I've been using:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStreamReader;
import java.net.URISyntaxException;
import org.apache.http.HttpHeaders;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
URIBuilder builder = new URIBuilder(https://bla.com/<restapiroot/v2/nodes");
builder.setParameter("type", "144")
.setParameter("parent_id", "123456")
.setParameter("name", "bla.pdf")
.setParameter("file", "C:\\My_Data\\bla.pdf");
MultipartEntityBuilder multiPartBuilder = MultipartEntityBuilder.create();
multiPartBuilder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
multiPartBuilder.addBinaryBody("ufile", new File (fullFileName), ContentType.DEFAULT_BINARY, fileName);
multiPartBuilder.setBoundary("aall12dk##Joey");
HttpPost httpPostRequest = new HttpPost(builder.build());
httpPostRequest.addHeader( "<auth code name>", "value" );
httpPostRequest.addHeader( HttpHeaders.CONTENT_TYPE, "multipart/form-data; boundary=aall12dk##Joey" );
httpPostRequest.addHeader( "Content-Disposition", "attachment;filename=" + "bla.pdf" );
httpPostRequest.setEntity(multiPartBuilder.build());
HttpResponse response = = httpClient.execute(httpPostRequest);
I get the following error:
00:47:47.694 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "{"error":"Could not process object, invalid action \u0027create\u0027"}"
00:47:47.695 [main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 500 Internal Server Error
I'm not sure if I'm invoking the API wrong and/or whether I'm coding the file upload logic wrong entirely. Any help would be greatly appreciated.
Thank you
I got the solution for this. Basically, the answer is to use only a generic URIBuilder. Everything else goes to the multi-part builder:
URIBuilder builder = new URIBuilder("https://bla.com/<restapiroot>/v2/nodes");
MultipartEntityBuilder multiPartBuilder = MultipartEntityBuilder.create();
multiPartBuilder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
multiPartBuilder.setBoundary("aall12dk##RandomBoundary2019"); //Random value basically.
multiPartBuilder.addPart("type", new StringBody(String.valueOf(LAPI_DOCUMENTS.DOCUMENTSUBTYPE), ContentType.MULTIPART_FORM_DATA));
multiPartBuilder.addPart("parent_id", new StringBody(String.valueOf(parentId), ContentType.MULTIPART_FORM_DATA));
multiPartBuilder.addPart("name", new StringBody(fileName, ContentType.MULTIPART_FORM_DATA));
multiPartBuilder.addPart("file", new FileBody(new File (fullFileName), ContentType.DEFAULT_BINARY));
multiPartBuilder.addPart("description", new StringBody(comments, ContentType.MULTIPART_FORM_DATA));
multiPartBuilder.addPart("external_create_date", new StringBody("2017-12-10", ContentType.MULTIPART_FORM_DATA));
multiPartBuilder.addPart("external_modify_date", new StringBody(LocalDate.now().toString(), ContentType.MULTIPART_FORM_DATA));
multiPartBuilder.addPart("external_source", new StringBody("ftp", ContentType.MULTIPART_FORM_DATA));
HttpPost httpPostRequest = new HttpPost(builder.build());
httpPostRequest.addHeader("<TicketHeaderName>", "<ticket value>");
httpPostRequest.setEntity(multiPartBuilder.build());
HttpResponse response = httpClient.execute(httpPostRequest); //need to assign response to variable or you'll end up with hanging connections.
if ( response.getStatusLine().getStatusCode() != 200 ) {
//Error handling logic
}
SimpleConsumer has been deprecated in kafka, with org.apache.kafka.clients.consumer.KafkaConsumer being the replacement. However, it doesn't have a send(...) function. How can I rewrite the below code using the new KafkaConsumer?
import scala.concurrent.duration._
import kafka.api.TopicMetadataRequest
import kafka.consumer.SimpleConsumer
....
val consumer = new SimpleConsumer(
host = "127.0.0.1",
port = 9092,
soTimeout = 2.seconds.toMillis.toInt,
bufferSize = 1024,
clientId = "health-check")
// this will fail if Kafka is unavailable
consumer.send(new TopicMetadataRequest(Nil, 1))
You can get topic metadata with .partitionsFor and .listTopics
There is no direct replacement for method ,it depends what you want to do .
If you need all partitions info there is method for that consumer.partitionFor(topic) in the new api