public class CommonClientConfigs {
private static final Logger log = LoggerFactory.getLogger(CommonClientConfigs.class);
/*
* NOTE: DO NOT CHANGE EITHER CONFIG NAMES AS THESE ARE PART OF THE PUBLIC API AND CHANGE WILL BREAK USER CODE.
*/
public static final String BOOTSTRAP_SERVERS_CONFIG = "bootstrap.servers";
public static final String BOOTSTRAP_SERVERS_DOC = "A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form "
+ "<code>host1:port1,host2:port2,...</code>. Since these servers are just used for the initial connection to "
+ "discover the full cluster membership (which may change dynamically), this list need not contain the full set of "
+ "servers (you may want more than one, though, in case a server is down).";
public static final String CLIENT_DNS_LOOKUP_CONFIG = "client.dns.lookup";
public static final String CLIENT_DNS_LOOKUP_DOC = "Controls how the client uses DNS lookups. If set to <code>use_all_dns_ips</code> then, when the lookup returns multiple IP addresses for a hostname,"
+ " they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers."
+ " If the value is <code>resolve_canonical_bootstrap_servers_only</code> each entry will be resolved and expanded into a list of canonical names.";
}
I just dont understand why they dont use the standard javadoc to comment.
anyone has idea about this?
Related
I'm trying to setup a secure Kafka cluster and having a bit of difficulty with ACLs.
The Confluent security guide for Kafka Streams (https://docs.confluent.io/current/streams/developer-guide/security.html) simply states that the Cluster Create ACL has to be given to the principal... but it doesn't say anything about how to actually handle the internal topics.
Through research and experimentation, I've determined (for Kafka version 1.0.0):
Wildcards cannot be used along with text for topic names in ACLs. For example, since all internal topics are prefixed with the application id, my first thought was to apply an acl to topics matching '<application.id>-*'. This doesn't work.
Topics created by the Streams API do not get read/write access granted to the creator automatically.
Are the exact names of the internal topics predictable and consistent? In other words, if I run my application on a dev server, will the exact same topics be created on the production server when run? If so, then I can just add ACLs derived from dev before deploying. If not, how should the ACLs be added?
Are the exact names of the internal topics predictable and consistent? In other words, if I run my application on a dev server, will the exact same topics be created on the production server when run?
Yes, you'll get the same exact topics names from run to run. The DSL generates processor names with a function that looks like this:
public String newProcessorName(final String prefix) {
return prefix + String.format("%010d", index.getAndIncrement());
}
(where index is just an incrementing integer). Those processor names are then used to create repartition topics with a function that looks like this (the parameter name is a processor name generated as above):
static <K1, V1> String createReparitionedSource(final InternalStreamsBuilder builder,
final Serde<K1> keySerde,
final Serde<V1> valSerde,
final String topicNamePrefix,
final String name) {
Serializer<K1> keySerializer = keySerde != null ? keySerde.serializer() : null;
Serializer<V1> valSerializer = valSerde != null ? valSerde.serializer() : null;
Deserializer<K1> keyDeserializer = keySerde != null ? keySerde.deserializer() : null;
Deserializer<V1> valDeserializer = valSerde != null ? valSerde.deserializer() : null;
String baseName = topicNamePrefix != null ? topicNamePrefix : name;
String repartitionTopic = baseName + REPARTITION_TOPIC_SUFFIX;
String sinkName = builder.newProcessorName(SINK_NAME);
String filterName = builder.newProcessorName(FILTER_NAME);
String sourceName = builder.newProcessorName(SOURCE_NAME);
builder.internalTopologyBuilder.addInternalTopic(repartitionTopic);
builder.internalTopologyBuilder.addProcessor(filterName, new KStreamFilter<>(new Predicate<K1, V1>() {
#Override
public boolean test(final K1 key, final V1 value) {
return key != null;
}
}, false), name);
builder.internalTopologyBuilder.addSink(sinkName, repartitionTopic, keySerializer, valSerializer,
null, filterName);
builder.internalTopologyBuilder.addSource(null, sourceName, new FailOnInvalidTimestamp(),
keyDeserializer, valDeserializer, repartitionTopic);
return sourceName;
}
If you don't change your topology—like, if don't change the order of how it's built, etc—you'll get the same results no matter where the topology is constructed (presuming you're using the same version of Kafka Streams).
If so, then I can just add ACLs derived from dev before deploying. If not, how should the ACLs be added?
I have not used ACLs, but I imagine that since these are just regular topics, then yeah, you can apply ACLs to them. The security guide does mention:
When applications are run against a secured Kafka cluster, the principal running the application must have the ACL --cluster --operation Create set so that the application has the permissions to create internal topics.
I've been wondering about this myself, though, so if I am wrong I am guessing someone from Confluent will correct me.
I've got IdentityServer3 running as a standalone identity server.
I have a separate MVC client that uses Cookies and OpenIdConnect for authentication. I'm trying to set up claims transformations amongst other things, and would like to reference the different claims types like so:
var givenName = id.FindFirst(Constants.ClaimTypes.GivenName);
var familyName = id.FindFirst(Constants.ClaimTypes.FamilyName);
var sub = id.FindFirst(Constants.ClaimTypes.Subject);
var roles = id.FindAll(Constants.ClaimTypes.Role);
On the IdentityServer3, I reference these using Thinktecture.IdentityServer.Core.Constants however on my MVC client I don't think I should need to reference Thinktecture.IdentityServer3 just for these string constants? Is there a client library that is recommended to be used in this case? I've tried Thinktecture.IdentityModel and some .NET references but none seem to replicate the ClaimTypes in Thinktecture.IdentityServer.Core.Constants. The best I've found is System.Security.Claims.ClaimTypes but that seems to have several missing e.g. FamilyName.
The first placed I looked was Thinktecture.IdentityModel but was surprised these aren't there.
So what's the magic reference? Or is it appropriate to load Thinktecture.IdentityServer3 just for these strings?
Thanks
EDIT: So I've found Thinktecture.IdentityModel.Client which contains a JwtClaimTypes that seems to mirror ClaimTypes. Why is this named with a Jwt prefix though?
The IdentityServer ClaimType constants are just a map of the OpenID Connect standard claims.
You'd be best off making your own class for these constants, as you said there's no point pulling in the full Identity Server 3 package and I don't think they are available in any other packages...
Do note that the claims come across via JSON in the JWT as snake case. For example FamilyName will be family_name.
You can install the Microsoft.AspNetCore.Authentication.JwtBearer package. It includes the JwtRegisteredClaimNames struct which you can use like:
using static Microsoft.IdentityModel.JsonWebTokens.JwtRegisteredClaimNames;
⋮
var userId = User.FindFirstValue(Sub);
Agree with Scott Brady, the best way to go about this is to create your constants class. We have created a shared library for this purpose, which contains claim type constants and have used them both in the server and the client projects.
NB: Apart from 'id_token' and 'sub' claim types, you can use your custom claim types in the implementation of 'IUserService'. This gives better clarity to claim type names also as you can use specific names based on your implementation.
Use IdentityModel
I've seen people use bare strings like "sub", but why, when the constants exist?
namespace IdentityModel
{
/// <summary>Commonly used claim types</summary>
public static class JwtClaimTypes
{
/// <summary>Unique Identifier for the End-User at the Issuer.</summary>
public const string Subject = "sub";
/// <summary>End-User's full name in displayable form including all name parts, possibly including titles and suffixes, ordered according to the End-User's locale and preferences.</summary>
public const string Name = "name";
/// <summary>Given name(s) or first name(s) of the End-User. Note that in some cultures, people can have multiple given names; all can be present, with the names being separated by space characters.</summary>
public const string GivenName = "given_name";
/// <summary>Surname(s) or last name(s) of the End-User. Note that in some cultures, people can have multiple family names or no family name; all can be present, with the names being separated by space characters.</summary>
public const string FamilyName = "family_name";
My goal is to create a strategy of different steps to get from a point-to-point communication between 2 components to a "full blown netflix" style of communication using eureka, ribbon, hystrix. With each iteration I want to add more while I try to limit the amount of changes to the actual code. Feign is my preferred client side framework to make this happen. First step is to create a FeignClient to communicate to the server:
#FeignClient(url = "http://localhost:9000")
interface Client {
#RequestMapping(method = RequestMethod.GET, value = "/author/{author}/addedValue/{addedValue}")
Result addToTotal(#RequestParam(value="author") String author, #RequestParam(value="addedValue") long addedValue);
}
This works but I don't want the URL to be hardcoded in the annotation. I would like to have this: #FeignClient()
and have a properties construct like: client.url: http://localhost:9000
So far I couldn't find any clues on how to configure that and I couldn't find a solution in the spring-cloud sources.
Can it be done and if yes; how?
It can be done with a "serviceId" instead of a "url". E.g.
#FeignClient("foo")
interface Client { ... }
and
foo.ribbon.listOfServers: localhost:9000
e.g. see http://projects.spring.io/spring-cloud/spring-cloud.html#spring-cloud-ribbon-without-eureka for docs.
This can be done like this:
#FeignClient(name="fd-mobileapi-service",url="${fdmobile.ribbon.listOfServers}")
Where fdmobile.ribbon.listOfServers : value is a property in application.properties.
I have tested it and it is working.
I got a way to pass the environment variables in a very simple way interface FeignClient,
#FeignClient(url = "https://"+"\${url}")
interface Client {
#RequestMapping(method = RequestMethod.GET, value = "/author/{author}/addedValue/{addedValue}")
Result addToTotal(#RequestParam(value="author") String author, #RequestParam(value="addedValue") long addedValue);
properties
#URL
url.client=${URL}
.env
URL=https:localhost:9000
I'm having some trouble with EntityFramework (database first) and AppHarbor.
I'm trying to use the configuration string as inserted into the web.config by AppHarbor (I've added the metadata into the Sequelizer config option on the website), and I'm trying to add some additional values using code provided.
Currently I'm being a very bad person and embedding the string directly into my apps configuration provider - not good if the hosting provider switch DBs on us, so I'm looking to do it the proper way and use the values AppHarbor supply via the web.config.
This is the string as per provided by AppHarbor (passwords and server details removed):
metadata='res://*/MyDataEntities.csdl|res://*/MyDataEntities.ssdl|res://*/MyDataEntities.msl;';provider=System.Data.SqlClient;provider connection string='Server=servername.sequelizer.com;Database=databasename;User ID=username;Password=<snip>;'
If used "as is", that generates the following error:
The specified metadata path is not valid. A valid path must be either an existing directory, an existing file with extension '.csdl', '.ssdl', or '.msl', or a URI that identifies an embedded resource.
I then use the following code (purloined off of one of the AppHarbor support discussions) to append the required extra things EF needs...
if (String.IsNullOrWhiteSpace(ProductionDatabaseConnectionString))
{
// Get it on first read and cache it
var configuration = System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration("~");
var connectionString = configuration.ConnectionStrings.ConnectionStrings["SQLAppHarbor001"].ConnectionString;
// Add the required extra metadata for EF4.x
if (!connectionString.Contains("MultipleActiveResultSets=True;"))
connectionString += "MultipleActiveResultSets=True;";
if (!connectionString.Contains("App=EntityFramework;"))
connectionString += "App=EntityFramework;";
configuration.ConnectionStrings.ConnectionStrings["SQLAppHarbor001"].ConnectionString = connectionString;
configuration.Save();
ProductionDatabaseConnectionString = connectionString;
}
return ProductionDatabaseConnectionString;
That produces the connection string as follows:
metadata='res://*/MyDataEntities.csdl|res://*/MyDataEntities.ssdl|res://*/MyDataEntities.msl;';provider=System.Data.SqlClient;provider connection string='Server=servername.sequelizer.com;Database=databasename;User ID=username;Password=<snip>;'MultipleActiveResultSets=True;App=EntityFramework;
But that produces the error:
Format of the initialization string does not conform to specification starting at index 165.
Index 165 being the start of "provider connection string".
The working connection string I use embedded, which currently works without issue, is:
metadata='res://*/MyDataEntities.csdl|res://*/MyDataEntities.ssdl|res://*/MyDataEntities.msl;';provider=System.Data.SqlClient;provider connection string='Server=servername.sequelizer.com;Database=databasename;User ID=username;Password=<snip>;multipleactiveresultsets=True;App=EntityFramework'
The only real differences being that the "multipleactiveresultsets=True;App=EntityFramework" entries are inside the "provider connection string" string rather than outside.
Other people seem to be using EntityFramework on AppHarbor using the supplied configuration variables fine, so what an I doing wrong?
Update: Multiple Active Result Sets (MARS) can now be enabled for the injected connection string by using the Sequelizer admin panel. This is the recommended approach since the web.config no longer needs to be modified, which causes an AppDomain reload during startup
I came up against this today! I did the following:
var configuration = System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration("~");
var connectionString = configuration.ConnectionStrings.ConnectionStrings["ConnStringAlias"].ConnectionString;
if (!connectionString.Contains("multipleactiveresultsets=True;"))
{
connectionString = connectionString.TrimEnd('\'');
connectionString = connectionString += "multipleactiveresultsets=True;\'";
configuration.ConnectionStrings.ConnectionStrings["ConnStringAlias"].ConnectionString = connectionString;
configuration.Save();
}
The MultipleActiveResultSets property must be inside the provider connection string, which is why you received an error regarding the format of your connection string.
I seen a few 'solutions' around but none seemed to work for me, including the solution at the bottom of a support page of how to do exactly this on AppHarbor's site. The solution provided even sends the application into an infinite loop as the application will restart every time the web.config file is saved, which is every time in the example.
So I'm working on a project where many teams are using common services and following a common architecture. One of the services in use is messaging, currently JMS with ActiveMQ. Pretty much all teams are required to follow a strict set of rules for creating and sending messages, namely, everything is pub-subscribe and the messages that are sent are somewhat like the following:
public class WorkDTO {
private String type;
private String subtype;
private String category;
private String jsonPayload; // converted custom Java object
}
The 'jsonPayload' comes from a base class that all teams extend from so it has common attributes.
So basically in JMS, everyone is always sending the same kind of message, but to different ActiveMQ Topics. When the message (WorkDTO) is sent via JMS, first it is converted into a JSON object then it is sent in a TextMessage.
Whenever a team wishes to create a subscriber for a topic, they create a DefaultMessageListenerContainer and configure it appropriately to receive messages (We are using Java-based Spring configuration). Basically every DefaultMessageListenerContainer that a team defines is pretty much the same except for maybe the destination from which to receive messages and the message handler.
I was wondering how anyone would approach further abstracting the messaging configuration via annotations in such a case? Meaning, since everyone is pretty much required to follow the same requirements, could something like the following be useful:
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.TYPE)
public #interface Listener {
String destination();
boolean durable() default false;
long receiveTimeout() default -1; // -1 use JMS default
String defaultListenerMethod() default "handleMessage";
// more config details here
}
#Listener(destination="PX.Foo", durable=true)
public class FooListener {
private ObjectMapper mapper = new ObjectMapper(); // converts JSON Strings to Java Classes
public void handleMessage(TextMessage message){
String text = message.getText();
WorkDTO dto = mapper.readValue(text, WorkDto.class);
String payload = dto.getPayload();
String type = dto.getType();
String subType = dto.getSubType();
String category = dto.getCategory();
}
}
Of course I left out the part on how to configure the DefaultMessageListenerContainer by use of the #Listener annotation. I started looking into a BeanFactoryPostProcessor to create the necessary classes and add them to the application context, but I don't know how to do all that.
The reason I ask the question is that we are switching to AMQP/RabbitMQ from JMS/ActiveMQ and would like to abstract the messaging configuration even further by use of annotations. I know AMQP is not like JMS so the configuration details would be slightly different. I don't believe we will be switching from AMQP to something else.
Here teams only need to know the name of the destination and whether they want to make their subscription durable.
This is just something that popped into my head just recently. Any thoughts on this?
I don't want to do something overly complicated though so the other alternative is to create a convenience method that returns a pre-configured DefaultMessageListenerContainer given a destination and a message handler:
#Configuration
public class MyConfig{
#Autowired
private MessageConfigFactory configFactory;
#Bean
public DefaultMessageListenerContainer fooListenerContainer(){
return configFactory.getListenerContainer("PX.Foo", new FooListener(), true);
}
}
class MessageConfigFactory {
public DefaultMessageListenerContainer getListener(String destination, Object listener, boolean durable) {
DefaultMessageListenerContainer l = new DefaultMessageListenerContainer();
// configuration details here
return l;
}
}