I have a springboot with springdata mongodb application where I am connecting to mongo change stream to save the changes to a audit collection. My application is running multiple instances (2 instances) and will be scaled up to n number instances when the load increased. When records are created in the original collection (“my collection”), the listeners will be triggered in all running instances and creates duplicate records. Following is my setup
build.gradle
…
// spring data mingodb version 3.1.5
implementation 'org.springframework.boot:spring-boot-starter-data-mongodb'
…
Listener config
#Configuration
#Slf4j
public class MongoChangeStreamListenerConfig {
#Bean
MessageListenerContainer changeStreamListenerContainer(
MongoTemplate template,
PartyConsentAuditListener consentAuditListener,
ErrorHandler errorHandler) {
MessageListenerContainer messageListenerContainer =
new MongoStreamListenerContainer(template, errorHandler);
ChangeStreamRequest<PartyConsentEntity> request =
ChangeStreamRequest.builder(consentAuditListener)
.collection("my-collection")
.filter(newAggregation(match(where("operationType").in("insert", "update", "replace"))))
.fullDocumentLookup(FullDocument.UPDATE_LOOKUP)
.build();
messageListenerContainer.register(request, MyEntity.class, errorHandler);
log.info("mongo stream listener is registered");
return messageListenerContainer;
}
#Bean
ErrorHandler getLoggingErrorHandler() {
return new ErrorHandler() {
#Override
public void handleError(Throwable throwable) {
log.error("error in creating audit records {}", throwable);
}
};
}
}
Listener container
public class MongoStreamListenerContainer extends DefaultMessageListenerContainer {
public MongoStreamListenerContainer(MongoTemplate template, ErrorHandler errorHandler) {
super(template, Executors.newFixedThreadPool(15), errorHandler);
}
#Override
public boolean isAutoStartup() {
return true;
}
}
ChangeListener
#Component
#Slf4j
#RequiredArgsConstructor
/**
* This class will listen to mongodb change stream and process changes. The onMessage will triggered
* when a record added, updated or replaced in the mongo db.
*/
public class MyEntityAuditListener
implements MessageListener<ChangeStreamDocument<Document>, MyEntity> {
#Override
public void onMessage(Message<ChangeStreamDocument<Document>, MyEntity > message) {
var update = message.getBody();
log.info("db change event received");
if (update != null) {
log.info("creating audit entries for id {}", update.getId());
// This will execute in all the instances and creating duplicating records
}
}
}
Is there a way to control the execution on one instance at a given time and share the load between nodes?. It would be really nice to know if there is a config from spring data mongodb to control the flow.
Also, I have checked the following post in stack overflow and I am not sure how to use this with spring data.
Mongo Change Streams running multiple times (kind of): Node app running multiple instances
Any help or tip to resolve this issue is highly appreciated. Thank you very much in advance.
Related
Environment: I am running Apache Ignite v2.13.0 for the cache and the cache store is persisting to a Mongo DB v3.6.0. I am also utilizing Spring Boot (Java).
Question: When I have an expiration policy set, how do I remove the corresponding data from my persistent database?
What I have attempted: I have attempted to utilize CacheEntryExpiredListener but my print statement is not getting triggered. Is this the proper way to solve the problem?
Here is a sample bit of code:
#Service
public class CacheRemovalListener implements CacheEntryExpiredListener<Long, Metrics> {
#Override
public void onExpired(Iterable<CacheEntryEvent<? extends Long, ? extends Metrics>> events) throws CacheEntryListenerException {
for (CacheEntryEvent<? extends Long, ? extends Metrics> event : events) {
System.out.println("Received a " + event);
}
}
}
Use Continuous Query to get notifications about Ignite data changes.
ExecutorService mongoUpdateExecutor = Executors.newSingleThreadExecutor();
CacheEntryUpdatedListener<Integer, Integer> lsnr = new CacheEntryUpdatedListener<Integer, Integer>() {
#Override
public void onUpdated(Iterable<CacheEntryEvent<? extends Integer, ? extends Integer>> evts) {
for (CacheEntryEvent<?, ?> e : evts) {
if (e.getEventType() == EventType.EXPIRED) {
// Use separate executor to avoid blocking Ignite threads
mongoUpdateExecutor.submit(() -> removeFromMongo(e.getKey()));
}
}
}
};
var qry = new ContinuousQuery<Integer, Integer>()
.setLocalListener(lsnr)
.setIncludeExpired(true);
// Start receiving updates.
var cursor = cache.query(qry);
// Stop receiving updates.
cursor.close();
Note 1: EXPIRED events should be enabled explicitly with ContinuousQuery#setIncludeExpired.
Note 2: Query listeners should not perform any heavy/blocking operations. Offload that work to a separate thread/executor.
Preamble
Since there are a lot of questions on StackOverflow about this already, I first want to ensure that this is not a duplicate and differentiate.
This is about
Having 2(or more) different AuthenticationProviders in 2 different AuthenticationManagers to be used on different routes.
Using the methods in Spring Security 5.5 not 3.x
Using a non XML configuration based approach
So the question is not about:
How to include several AuthenticationProvideres in on AuthenticationManager to offer "alternative authentications" (which most questions tend to be)
Case
Assume one has 2 custom AuthenticationProviders: CATApiTokenProvider and DOGApiTokenProvider. It is by design that we not talk about AOuth/JWT/Basic/Form providers, since they offer shortcuts.
Now we have 2 REST API endpoints /dog/endpoint and /cat/endpoint.
Question
How would one properly implement this today, with Spring Security 5.5:
We want the authentication provider CATApiTokenProvider to only be able to authenticate requests on /cat/endpoint
We want the authentication provider DOGApiTokenProvider to only be able to authenticate requests on /dog/endpoint
So one cannot authenticate with a cat token on /dog/endpoint and neither with a dog token on /cat/endpoint.
My Ideas/Approaches
a) I understand that since I have custom Cat/Dog filters, one can use the AuthenticationManagerResolver and pass one instance into the filter when creating the bean. This resolver might look like
public AuthenticationManagerResolver<HttpServletRequest> resolver()
{
return request -> {
if (request.getPathInfo().startsWith("/dog/")) {
try {
return ???;
} catch (Exception exception) {
log.error(exception);
}
}
if (request.getPathInfo().startsWith("/cat/")) {
try {
return ???;
} catch (Exception exception) {
log.error(exception);
}
}
};
}
Two questions with that would be:
how to return different authentication managers here? How to instantiate 2 different AM with each one CatAP and DogAP? Currently I use public void configure(AuthenticationManagerBuilder auth) but as far as I understand, I would only configure 'the one' AuthenticationManager and I could add DogAP and CatAP there, but this would let as having 1 AM with 2 APs, so when using this AM i could auth with the dog token on the cat endpoint
is this really the right way to implement this? I would have expected to be able to provide the AM on the SecurityConfiguration level
b) Somehow instantiate 2 different AuthenticationManagers and then use the SecurityConfiguration to assign them to different matchers.
Two questions:
what is the right way to spawn 2 different AMs with different providers?
I cannot understand how I would add an AM for a spec
http.authorizeRequests()
.antMatchers("/dog/**")
.?
You can either publish multiple filter chains or wire your own AuthenticationFilter with an AuthenticationManagerResolver
You may use AuthenticationManagerResolver to return different AuthenticationManagers. Since Spring Security 5.4.0, we don't need to extend the WebSecurityConfigurerAdapter to configure our SecurityFilterChain anymore, you can instead define a bean of SecurityFilterChain type.
I'll go into detail on wiring your own AuthenticationFilter.
#Configuration
#EnableWebSecurity
public class SecurityConfig {
#Bean
public SecurityFilterChain apiSecurity(HttpSecurity http) throws Exception {
http.authorizeHttpRequests((authz) -> authz
.anyRequest().authenticated());
http.addFilterBefore(apiAuthenticationFilter(), UsernamePasswordAuthenticationFilter.class);
return http.build();
}
private AuthenticationFilter apiAuthenticationFilter() {
AuthenticationFilter authenticationFilter = new AuthenticationFilter(new ApiAuthenticationManagerResolver(), new BasicAuthenticationConverter());
authenticationFilter.setSuccessHandler((request, response, authentication) -> {});
return authenticationFilter;
}
public static class ApiAuthenticationManagerResolver implements AuthenticationManagerResolver<HttpServletRequest> {
private final Map<RequestMatcher, AuthenticationManager> managers = Map.of(
new AntPathRequestMatcher("/dog/**"), new DogAuthenticationProvider()::authenticate,
new AntPathRequestMatcher("/cat/**"), new CatAuthenticationProvider()::authenticate
);
#Override
public AuthenticationManager resolve(HttpServletRequest request) {
for (Map.Entry<RequestMatcher, AuthenticationManager> entry : managers.entrySet()) {
if (entry.getKey().matches(request)) {
return entry.getValue();
}
}
throw new IllegalArgumentException("Unable to resolve AuthenticationManager");
}
}
public static class DogAuthenticationProvider implements AuthenticationProvider {
#Override
public Authentication authenticate(Authentication authentication) throws AuthenticationException {
if (authentication.getName().endsWith("_dog")) {
return new UsernamePasswordAuthenticationToken(authentication.getName(), authentication.getCredentials(),
AuthorityUtils.createAuthorityList("ROLE_DOG"));
}
throw new BadCredentialsException("Username should end with _dog");
}
#Override
public boolean supports(Class<?> authentication) {
return true;
}
}
public static class CatAuthenticationProvider implements AuthenticationProvider {
#Override
public Authentication authenticate(Authentication authentication) throws AuthenticationException {
if (authentication.getName().endsWith("_cat")) {
return new UsernamePasswordAuthenticationToken(authentication.getName(), authentication.getCredentials(),
AuthorityUtils.createAuthorityList("ROLE_CAT"));
}
throw new BadCredentialsException("Username should end with _cat");
}
#Override
public boolean supports(Class<?> authentication) {
return true;
}
}
}
In the example above, we have two AuthenticationProviders, one for cat and other for dog. They are resolved upon an AntPathRequestMatcher matching for both /dog/** and /cat/** endpoints, inside the ApiAuthenticationManagerResolver. There is no need to defined an AuthenticationManager for each dog and cat, since AuthenticationProvider/Manager have the same interface.
The ApiAuthenticationManagerResolver is then wired inside an AuthenticationFilter in your filter chain.
You can also define two different filter chains for each endpoint, like so:
#Bean
public SecurityFilterChain dogApiSecurity(HttpSecurity http) throws Exception {
http.requestMatchers((matchers) -> matchers
.antMatchers("/dog/**"));
http.authorizeRequests((authz) -> authz
.anyRequest().authenticated());
http.httpBasic();
http.authenticationProvider(new DogAuthenticationProvider());
return http.build();
}
#Bean
public SecurityFilterChain catApiSecurity(HttpSecurity http) throws Exception {
http.requestMatchers((matchers) -> matchers
.antMatchers("/cat/**"));
http.authorizeRequests((authz) -> authz
.anyRequest().authenticated());
http.httpBasic();
http.authenticationProvider(new CatAuthenticationProvider());
return http.build();
}
Please, when defining multiple filter chains, the ordering is important, make use of the #Order annotation in those scenarios.
When you do http.requestMatcher(new AntPathRequestMatcher("/endpoint/**")); you are telling Spring Security to only call the filter chain when the request matches that path.
There is also a ticket within Spring Security's repository to provide a AuthenticationManagerResolver implementation which accepts Map<RequestMatcher, AuthenticationManager>, it would be nice if you think it makes sense, give a thumbs up there.
I have a spring batch job which consists of
ItemReader<Product>: reads a product from DB
ItemProcesser<Product, List<RelatedProduct>>: reads related products from the original product
ItemWriter<List<RelatedProduct>>: writes some aspect of related products into DB
Recently, I found a case where a product had so many related products so it caused a long transaction in DB which took about an hour. This happened in our OLTP DB, so we want to split the list of related products into smaller chunks to avoid long transactions.
At first I tried to check the feasibility of code below. But it seemed this kind of code is not possible in Spring Batch.
#Bean
#JobScope
public Step step1() {
return stepBuilderFactory.get("step1")
.chunk<Product, List<RelatedProduct>>(1)
.reader(productItemReader()) // reads a product from DB
.processor(relatedProductProcessor()) // reads related products, the number of related products can be huge.
// Beginning of my hope
.reader(eachRelatedProductInListReader()) // reads each related product in the list.
.chunk(100) // re-aggregate them smaller chunks
// End of my hope
.writer(relatedProductItemWriter()) // writes info about related products
.build()
}
So now I'm thinking of storing the long list in the job context and adding one more step to process RelatedProduct in smaller chunks. But I'm wondering if there are any better ways. Any suggestions?
I implemented a new ItemReader like below to read RelatedProduct one by one using the original reader and processor:
public class FlatteningItemReader<I, O> implements ItemReader<O> {
private final ItemReader<I> originalReader;
private final ItemProcessor<I, ? extends Iterable<O>> iterableProducer;
private final Deque<O> buffer = new ArrayDeque<>();
public FlatteningItemReader(ItemReader<I> originalReader, ItemProcessor<I, ? extends Iterable<O>> iterableProducer) {
this.originalReader = Objects.requireNonNull(originalReader);
this.iterableProducer = Objects.requireNonNull(iterableProducer);
}
#Override
public O read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
if (!buffer.isEmpty()) {
return buffer.pollFirst();
}
I item;
while ((item = originalReader.read()) != null) {
Iterable<O> iter = iterableProducer.process(item);
if (iter == null) continue;
for (O o : iter) {
buffer.offerLast(o);
}
if (!buffer.isEmpty()) {
return buffer.pollFirst();
}
}
return null;
}
}
The step configuration using FlatteningItemReader looks like:
#Bean
#JobScope
public Step step1() {
return stepBuilderFactory.get("step1")
.chunk<Product, List<RelatedProduct>>(1)
// reads each RelatedItem using this new reader
.reader(new FlatteningItemReader(productItemReader(), relatedProductProcessor()))
.processor(relatedProductProcessor())
.writer(relatedProductItemWriter())
.build()
}
I'm not sure if this is good approach in Spring Batch.
Is there a work-around to get Spring to handle incoming messages from XMPP? I have tried many different configurations to get an inbound-channel-adapter to respond to incoming XMPP messages and nothing happens. I know that they show up at the Spring Integration layer (I can see that in the logs) but they are ignored. Is there any way to get them into my application layer? I hope to avoid needing to make changes to Spring Integration itself if I can.
Here is my integration configuration:
<int-xmpp:inbound-channel-adapter id="gcmIn"
channel="gcmInChannel"
xmpp-connection="gcmConnection"
auto-startup="true"
/>
<bean id="inboundBean" class="example.integration.GcmInputHandler"/>
<int:service-activator input-channel="gcmInChannel" output-channel="nullChannel" ref="inboundBean" method="handle"/>
Using the outbound-channel-adapter works fine. I can send messages over GCM 100% easily. But inbound does nothing, even though I know the messages are coming in.
Thanks
Not a very clean one, you would need to overwrite the ChatMessageListeningEndpoint, which drops all empty body messages.
This one needs then to be used as inbound-channel adapter in your config.
In addition you need to register the GCM package extension on the Smack Provider Manager, otherwise you lose the JSON message.
Working on a sample project -- so if you need more help let me know and I will post a link as soon it works somehow in a understandable way.
Here a sample GCM Input Adapter
public class GcmMessageListeningEndpoint extends ChatMessageListeningEndpoint {
private static final Logger LOG = LoggerFactory.getLogger(GcmMessageListeningEndpoint.class);
#Setter
protected PacketListener packetListener = new GcmPacketListener();
protected XmppHeaderMapper headerMapper = new DefaultXmppHeaderMapper();
public GcmMessageListeningEndpoint(XMPPConnection connection) {
super(connection);
ProviderManager.addExtensionProvider(GcmPacketExtension.GCM_ELEMENT_NAME, GcmPacketExtension.GCM_NAMESPACE,
new PacketExtensionProvider() {
#Override
public PacketExtension parseExtension(XmlPullParser parser) throws Exception {
String json = parser.nextText();
return new GcmPacketExtension(json);
}
});
}
#Override
public void setHeaderMapper(XmppHeaderMapper headerMapper) {
super.setHeaderMapper(headerMapper);
this.headerMapper = headerMapper;
if (this.headerMapper == null) throw new IllegalArgumentException("Null XmppHeaderMapper isn't supported!");
}
public String getComponentType() {
return "xmpp:inbound-channel-adapter-gcm";
}
#Override
protected void doStart() {
Assert.isTrue(this.initialized, this.getComponentName() + " [" + this.getComponentType() + "] must be initialized");
this.xmppConnection.addPacketListener(this.packetListener, null);
}
#Override
protected void doStop() {
if (this.xmppConnection != null) {
this.xmppConnection.removePacketListener(this.packetListener);
}
}
class GcmPacketListener implements PacketListener {
#Override
public void processPacket(Packet packet) throws NotConnectedException {
if (packet instanceof org.jivesoftware.smack.packet.Message) {
org.jivesoftware.smack.packet.Message xmppMessage = (org.jivesoftware.smack.packet.Message) packet;
Map<String, ?> mappedHeaders = headerMapper.toHeadersFromRequest(xmppMessage);
sendMessage(MessageBuilder.withPayload(xmppMessage).copyHeaders(mappedHeaders).build());
} else {
LOG.warn("Unsuported Packet {}", packet);
}
}
}
}
And here the new configuration for the inbound-channel-adapter remove the one in XML:
#Bean
public GcmMessageListeningEndpoint inboundAdpater(XMPPConnection connection, MessageChannel gcmInChannel) {
GcmMessageListeningEndpoint endpoint = new GcmMessageListeningEndpoint(connection);
endpoint.setOutputChannel(gcmInChannel);
return endpoint;
}
I have created GWT app, in which I have a Vertical Panel where I log the details.
Client side logging I'm doing using logger
sample code is:
public static VerticalPanel customLogArea = new VerticalPanel();
public static Logger rootLogger = Logger.getLogger("");
logerPanel.setTitle("Log");
scrollPanel.add(customLogArea);
logerPanel.add(scrollPanel);
if (LogConfiguration.loggingIsEnabled()) {
rootLogger.addHandler(new HasWidgetsLogHandler(customLogArea));
}
And I'm updating my vertical log panel using this code
rootLogger.log(Level.INFO,
"Already Present in Process Workspace\n");
But now my question is , I have to log server side details also into my vertical log panel.
My serverside GreetingServiceImpl code is:
public boolean createDirectory(String fileName)
throws IllegalArgumentException {
Boolean result = false;
try {
rootLogger.log(Level.INFO,
"I want to log this to my UI vertical log Panel");
system.out.println("log this to UI");
File dir = new File("D:/GenomeSamples/" + fileName);
if (!dir.exists()) {
result = dir.mkdir();
}
} catch (Exception e) {
e.printStackTrace();
}
return result;
}
Now I want to log sysoutprt statements to my UI from here. How can I achieve this. Now using rootLogger.log(Level.INFO,
"I want to log this to my UI vertical log Panel"); code it is logging this to eclipse console . But how to log this to my UI in client side.
Please let me know If anything wrong in this question.
If I understood you right, you want to see your server log entries in web interface. And of course, java logger and printStackTrace() won't help you in that: your gwt code is compiled to JavaScript and has nothing to do with console and log files. Besides, your server can't "push" log entries to client - it's up to client to make requests. So if you want to track new log entries and move it to client, you need to poll server for new entries. And yet another problem: you may have many clients polling your servlet and you should keep in mind this multi-threading.
This is how I see probable implementation (it's just concept, may contain some errors and misspellings):
Remote interface:
public interface GreetingService extends RemoteService {
List<String> getLogEntries();
boolean createDirectory(String fileName)throws IllegalArgumentException;
}
Remote Servlet:
public class GreetingServiceImpl extends RemoteServiceServlet implements GreetingService {
public static final String LOG_ENTRIES = "LogEntries";
public List<String> getLogEntries() {
List<String> entries = getEntriesFromSession();
List<String>copy = new ArrayList<String>(entries.size());
copy.addAll(entries);
//prevent loading the same entries twice
entries.clear();
return copy;
}
public boolean createDirectory(String fileName)throws IllegalArgumentException {
Boolean result = false;
try {
log("I want to log this to my UI vertical log Panel");
log("log this to UI");
File dir = new File("D:/GenomeSamples/" + fileName);
if (!dir.exists()) {
result = dir.mkdir();
}
} catch (Exception e) {
log("Exception occurred: " + e.getMessage());
}
return result;
}
private List<String> getEntriesFromSession() {
HttpSession session= getThreadLocalRequest().getSession();
List<String>entries = (List<String>)session.getAttribute(LOG_ENTRIES);
if (entries == null) {
entries = new ArrayList<String>();
session.setAttribute(LOG_ENTRIES,entries);
}
return entries;
}
private void log(String message) {
getEntriesFromSession().add(message);
}
Simple implementation of polling (gwt client-side):
Timer t = new Timer() {
#Override
public void run() {
greetingAsyncService.getLogEntries(new AsyncCallBack<List<String>>() {
void onSuccess(List<String>entries) {
//put entries to your vertical panel
}
void onFailure(Throwable caught){
//handle exceptions
}
});
}
};
// Schedule the timer to run once in second.
t.scheduleRepeating(1000);
greetingAsyncService.createDirectory(fileName, new AsyncCallBack<Void>(){
void onSuccess(List<String>entries) {
//no need to poll anymore
t.cancel();
}
void onFailure(Throwable caught){
//handle exceptions
}
});
}
As you can see, I have used session to keep log entries, because session is client-specific and so different clients will receive different logs. It's up to you to decide what to use - you may create your own Logger class that will track users itself and give appropriate logs to appropriate clients.
And also you may want to save level of your messages (INFO,ERROR etc.) and then display messages in different colors (red for ERROR, for instance). To do so, you need to save not List, but some your custom class.
You'd create a logging servlet that has the same methods as your logging framework to send log messages to your server via RPC.
Here are some sample RPC log methods you can use:
public interface LogService extends RemoteService {
public void logException(String logger, String priority, String message, String error, StackTraceElement[] stackTrace, String nativeStack);
}
public interface LogServiceAsync {
public void logException(String logger, String priority, String message, String error, StackTraceElement[] stackTrace, String nativeStack, AsyncCallback<Void> callback);
}
public class LogServiceImpl extends RemoteServiceServlet implements LogService {
public void logException(String loggerName, String priority, String logMessage, String errorMessage, StackTraceElement[] stackTrace, String nativeStack) {
Logger logger = getLogger(loggerName);
Level level = getLevel(priority);
// Create a Throwable to log
Throwable caught = new Throwable();
if (errorMessage != null && stackTrace != null) {
caught = new Throwable(errorMessage);
caught.setStackTrace(stackTrace);
}
//do stuff with the other passed arguments (optional)
logger.log(level, message, caught);
}
}
Although those implementations are very nice, forget about timers and repeated server queries. We've something better now.
It's possible to push data from server to client using Atmosphere which supports WebSockets.