Scala Netty is there any way to share a ReplayingDecoder - scala

I am looking to open up multiple connections using a netty client bootstrap in order to parse messages coming from multiple sources. The messages all have the same format, however, due to the amount of data that needs to be processed, I must run each connection on separate threads (This is assuming netty creates a thread per client channel, which I couldn't find a reference for - if that's not the case, how would this be achieved?).
This is the code that I use to connect to the data server:
var b = new Bootstrap()
.group(group)
.channel(classOf[NioSocketChannel])
.handler(RawFeedChannelInitializer)
var ch1 = b.clone().connect(host, port).sync().channel();
var ch2 = b.clone().connect(host, port).sync().channel();
The initializer calls RawPacketDecoder, which extends ReplayingDecoder, and is defined here.
The code works well without #Sharable when opening a single connection, but for the purpose of my application I must connect to the same server multiple times.
This results in the runtime error #Sharable annotation is not allowed pointing to my RawPacketDecoder class.
I am not entirely sure on how to get past this issue, short of reimplementing in scala an instantiable class of ReplayingDecoder as my decoder based directly on ByteToMessageDecoder.
Any help would be greatly appreciated.
Note: I am using netty 4.0.32 Final

I found the solution in this StockExchange answer.
My issue was that I was using an object based ChannelInitializer (singleton), and ReplayingDecoder as well as ByteToMessageDecoder are not sharable.
My initializer was created as a scala object, and therefore a single instance allowed. Changing the initializer to a scala class and instantiating for each bootstrap clone solved the problem. I modified the bootstrap code above as follows:
var b = new Bootstrap()
.group(group)
.channel(classOf[NioSocketChannel])
//.handler(RawFeedChannelInitializer)
var ch1 = b.clone().handler(new RawFeedChannelInitializer()).connect(host, port).sync().channel();
var ch2 = b.clone().handler(new RawFeedChannelInitializer()).connect(host, port).sync().channel();
I am not sure whether this ensures multithreading as wanted but it does allow to split the data access into multiple connections to the feed server.
Edit Update: After performing additional research on the subject, I have determined that netty does in fact create a thread per channel; this was verified by printing to console after the creation of each channel:
println("No. of active threads: " + Thread.activeCount());
The output shows an incremental number as channels are created and associated with their respective threads.
By default NioEventLoopGroup uses 2*Num_CPU_cores threads as defined here:
DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt(
"io.netty.eventLoopThreads",
Runtime.getRuntime().availableProcessors() * 2));
This value can be overriden to something else by setting
val group = new NioEventLoopGroup(16)
and then using the group to create/setup the bootstrap.

Related

How to use delta trigger in flink?

I want to use the deltatrigger in apache flink (flink 1.3) but I have some trouble with this code :
.trigger(DeltaTrigger.of(100, new DeltaFunction[uniqStruct] {
override def getDelta(oldFp: uniqStruct, newFp: uniqStruct): Double = newFp.time - oldFp.time
}, TypeInformation[uniqStruct]))
And I have this error:
error: object org.apache.flink.api.common.typeinfo.TypeInformation is not a value [ERROR] }, TypeInformation[uniqStruct]))
I don't understand why DeltaTrigger need TypeSerializer[T]
and I don't know what to do to remove this error.
Thanks a lot everyone.
I would read into this a bit https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/types_serialization.html sounds like you can create a serializer using typeInfo.createSerializer(config) on your type info. Note what you're passing in currently is a type itself and NOT the type info which is why you're getting the error you are.
You would need to do something more like
val uniqStructTypeInfo: TypeInformation[uniqStruct] = createTypeInformation[uniqStruct]
val uniqStrictTypeSerializer = typeInfo.createSerializer(config)
To quote the page above regarding the config param you need to pass to create serializer
The config parameter is of type ExecutionConfig and holds the
information about the program’s registered custom serializers. Where
ever possibly, try to pass the programs proper ExecutionConfig. You
can usually obtain it from DataStream or DataSet via calling
getExecutionConfig(). Inside functions (like MapFunction), you can get
it by making the function a Rich Function and calling
getRuntimeContext().getExecutionConfig().
DeltaTrigger needs a TypeSerializer because it uses Flink's managed state mechanism to store each element for later comparison with the next one (it just keeps one element, the last one, which is updated as new elements arrive).
You will find an example (in Java) here.
But if all you need is a window that triggers every 100msec, then it'll be easier to just use a TimeWindow, such as
input
.keyBy(<key selector>)
.timeWindow(Time.milliseconds(100)))
.apply(<window function>)
Updated:
To have hour-long windows that trigger every 100msec, you could use sliding windows. However, you would have 10 * 60 * 60 windows, and every event would be placed into each of these 36000 windows. So that's not a great idea.
If you use a GlobalWindow with a DeltaTrigger, then the window will be triggered only when events are more than 100msec apart, which isn't what you've said you want.
I suggest you look at ProcessFunction. It should be straightforward to get what you want that way.

What causes MARSHALLINGERROR when creating a znode?

I am doing a simple createAsync() with my ZooKeeperNetEx nuget package and it is throwing an exception which is triggered by a MARSHALLINGERROR.
Here's is the two-line summary (between these lines, the connection successfully confirmed to Zookeeper):
var Zoo = new ZooKeeper("localhost:50002", 5000, new ClusterWatcher());
. . .
var parentNode = Zoo.createAsync("/election", null, null, CreateMode.PERSISTENT).Result
I do not get it. ClusterWatcher is my own class derived from Watcher, of course. Yes, I am writing this in C# but this such a simple matter, I would not think it mattered. The host machine is running Windows 10 Pro, if that matters.
This exception can be triggered by not specifying the ACL mode (you seem to pass null). In Java you can pass the predefined list ZooDefs.Ids.OPEN_ACL_UNSAFE (for example, or one of the others in that class) for the ACL mode; for C# there will probably be a similarly named constant.
In the Java client library this is a convenience constant that is defined as:
/**
* This is a completely open ACL .
*/
public final ArrayList<ACL> OPEN_ACL_UNSAFE = new ArrayList<ACL>(
Collections.singletonList(new ACL(Perms.ALL, ANYONE_ID_UNSAFE))
);

Disable logging on FileConfigurationSourceChanged - LogEnabledFilter

I want Administrators to enable/disable logging at runtime by changing the enabled property of the LogEnabledFilter in the config.
There are several threads on SO that explain workarounds, but I want it this way.
I tried to change the Logging Enabled Filter like this:
private static void FileConfigurationSourceChanged(object sender, ConfigurationSourceChangedEventArgs e)
{
var fcs = sender as FileConfigurationSource;
System.Diagnostics.Debug.WriteLine("----------- FileConfigurationSourceChanged called --------");
LoggingSettings currentLogSettings = e.ConfigurationSource.GetSection("loggingConfiguration") as LoggingSettings;
var fdtl = currentLogSettings.TraceListeners.Where(tld => tld is FormattedDatabaseTraceListenerData).FirstOrDefault();
var currentLogFileFilter = currentLogSettings.LogFilters.Where(lfd => { return lfd.Name == "Logging Enabled Filter"; }).FirstOrDefault();
var filterNewValue = (bool)currentLogFileFilter.ElementInformation.Properties["enabled"].Value;
var runtimeFilter = Logger.Writer.GetFilter<LogEnabledFilter>("Logging Enabled Filter");
runtimeFilter.Enabled = filterNewValue;
var test = Logger.Writer.IsLoggingEnabled();
}
But test reveals always the initially loaded config value, it does not change.
I thought, that when changing the value in the config the changes will be propagated automatically to the runtime configuration. But this isn't the case!
Setting it programmatically as shown in the code above, doesn't work either.
It's time to rebuild Enterprise Library or shut it down.
You are right that the code you posted does not work. That code is using a config file (FileConfigurationSource) as the method to configure Enterprise Library.
Let's dig a bit deeper and see if programmatic configuration will work.
We will use the Fluent API since it is the preferred method for programmatic configuration:
var builder = new ConfigurationSourceBuilder();
builder.ConfigureLogging()
.WithOptions
.DoNotRevertImpersonation()
.FilterEnableOrDisable("EnableOrDisable").Enable()
.LogToCategoryNamed("General")
.WithOptions.SetAsDefaultCategory()
.SendTo.FlatFile("FlatFile")
.ToFile(#"fluent.log");
var configSource = new DictionaryConfigurationSource();
builder.UpdateConfigurationWithReplace(configSource);
var defaultWriter = new LogWriterFactory(configSource).Create();
defaultWriter.Write("Test1", "General");
var filter = defaultWriter.GetFilter<LogEnabledFilter>();
filter.Enabled = false;
defaultWriter.Write("Test2", "General");
If you try this code the filter will not be updated -- so another failure.
Let's try to use the "old school" programmatic configuration by using the classes directly:
var flatFileTraceListener = new FlatFileTraceListener(
#"program.log",
"----------------------------------------",
"----------------------------------------"
);
LogEnabledFilter enabledFilter = new LogEnabledFilter("Logging Enabled Filter", true);
// Build Configuration
var config = new LoggingConfiguration();
config.AddLogSource("General", SourceLevels.All, true)
.AddTraceListener(flatFileTraceListener);
config.Filters.Add(enabledFilter);
LogWriter defaultWriter = new LogWriter(config);
defaultWriter.Write("Test1", "General");
var filter = defaultWriter.GetFilter<LogEnabledFilter>();
filter.Enabled = false;
defaultWriter.Write("Test2", "General");
Success! The second ("Test2") message was not logged.
So, what is going on here? If we instantiate the filter ourselves and add it to the configuration it works but when relying on the Enterprise Library configuration the filter value is not updated.
This leads to a hypothesis: when using Enterprise Library configuration new filter instances are being returned each time which is why changing the value has no effect on the internal instance being used by Enterprise Library.
If we dig into the Enterprise Library code we (eventually) hit on LoggingSettings class and the BuildLogWriter method. This is used to create the LogWriter. Here's where the filters are created:
var filters = this.LogFilters.Select(tfd => tfd.BuildFilter());
So this line is using the configured LogFilterData and calling the BuildFilter method to instantiate the applicable filter. In this case the BuildFilter method of the configuration class LogEnabledFilterData BuildFilter method returns an instance of the LogEnabledFilter:
return new LogEnabledFilter(this.Name, this.Enabled);
The issue with this code is that this.LogFilters.Select returns a lazy evaluated enumeration that creates LogFilters and this enumeration is passed into the LogWriter to be used for all filter manipulation. Every time the filters are referenced the enumeration is evaluated and a new Filter instance is created! This confirms the original hypothesis.
To make it explicit: every time LogWriter.Write() is called a new LogEnabledFilter is created based on the original configuration. When the filters are queried by calling GetFilter() a new LogEnabledFilter is created based on the original configuration. Any changes to the object returned by GetFilter() have no affect on the internal configuration since it's a new object instance and, anyway, internally Enterprise Library will create another new instance on the next Write() call anyway.
Firstly, this is just plain wrong but it is also inefficient to create new objects on every call to Write() which could be invoked many times..
An easy fix for this issue is to evaluate the LogFilters enumeration by calling ToList():
var filters = this.LogFilters.Select(tfd => tfd.BuildFilter()).ToList();
This evaluates the enumeration only once ensuring that only one filter instance is created. Then the GetFilter() and update filter value approach posted in the question will work.
Update:
Randy Levy provided a fix in his answer above.
Implement the fix and recompile the enterprise library.
Here is the answer from Randy Levy:
Yes, you can disable logging by setting the LogEnabledFiter. The main
way to do this would be to manually edit the configuration file --
this is the main intention of that functionality (developers guide
references administrators tweaking this setting). Other similar
approaches to setting the filter are to programmatically modify the
original file-based configuration (which is essentially a
reconfiguration of the block), or reconfigure the block
programmatically (e.g. using the fluent interface). None of the
programmatic approaches are what I would call simple – Randy Levy 39
mins ago
If you try to get the filter and disable it I don't think it has any
affect without a reconfiguration. So the following code still ends up
logging: var enabledFilter = logWriter.GetFilter();
enabledFilter.Enabled = false; logWriter.Write("TEST"); One non-EntLib
approach would just to manage the enable/disable yourself with a bool
property and a helper class. But I think the priority approach is a
pretty straight forward alternative.
Conclusion:
In your custom Logger class implement a IsLoggenabled property and change/check this one at runtime.
This won't work:
var runtimeFilter = Logger.Writer.GetFilter<LogEnabledFilter>("Logging Enabled Filter");
runtimeFilter.Enabled = false/true;

Unity Entity Framework within ASP.NET WebAPI 2

I have a very weird problem with Unity here. I have the following:
public class UnityConfig
{
public static void RegisterTypes(IUnityContainer container)
container.RegisterType<IDBContext, MyDbContext>(new PerThreadLifetimeManager());
container.RegisterType<IUserDbContext>(new PerThreadLifetimeManager(), new InjectionFactory(c =>
{
var tenantConnectionString = c.Resolve<ITenantConnectionResolver>().ResolveConnectionString();
return new UserDbContext(tenantConnectionString);
}));
}
}
and then in the WebApiConfig.cs file within the Reigster method:
var container = new UnityContainer();
UnityConfig.RegisterTypes(container);
config.DependencyResolver = new UnityResolver(container);
Basically, what I want to happen in the above code is on every request to the API, I want Unity to new up a UserDbContext based on the user (multi-tenant kind of environment). Now the TenantConnectionResolver is responsible for figuring out the Connection String and then I use that connection string to new up UserDbContext.
Also note (not shown above) that TenantConnectionResolver takes an IDbConext in its constructor because I need it to figure out the connection string based on user information in that database.
But for some reason, the code within the InjectionFactory runs at random times. For example, I call //mysite.com/controller/action/1 repetitively from a browser, the code in the InjectionFactory will occasionally run but not on each request.
Am I incorrectly configuring Unity? Has anybody encountered anything similar to this?
Thanks in advance
The problem is very likely related to the LifetimeManager you are using. PerThreadLifetimeManager is not adapted in a web context, as threads are pooled and will serve multiple requests in sequence.
PerRequestLifetimeManager is probably what you want to use.

How to obtain wicket URL from PageClass and PageParameters without running Wicket application (i.e. without RequestCycle)?

In my project, there are additional (non-wicket) applications, which need to know the URL representation of some domain objects (e.g. in order to write a link like http://mydomain.com/user/someUserName/ into a notification email).
Now I'd like to create a spring bean in my wicket module, exposing the URLs I need without having a running wicket context, in order to make the other application depend on the wicket module, e.g. offering a method public String getUrlForUser(User u) returning "/user/someUserName/".
I've been stalking around the web and through the wicket source for a complete workday now, and did not find a way to retrieve the URL for a given PageClass and PageParameters without a current RequestCycle.
Any ideas how I could achieve this? Actually, all the information I need is somehow stored by my WebApplication, in which I define mount points and page classes.
Update: Because the code below caused problems under certain circumstances (in our case, being executed subsequently by a quarz scheduled job), I dived a bit deeper and finally found a more light-weight solution.
Pros:
No need to construct and run an instance of the WebApplication
No need to mock a ServletContext
Works completely independent of web application container
Contra (or not, depends on how you look at it):
Need to extract the actual mounting from your WebApplication class and encapsulate it in another class, which can then be used by standalone processes. You can no longer use WebApplication's convenient mountPage() method then, but you can easily build your own convenience implementation, just have a look at the wicket sources.
(Personally, I have never been happy with all the mount configuration making up 95% of my WebApplication class, so it felt good to finally extract it somewhere else.)
I cannot post the actual code, but having a look at this piece of code will give you an idea how you should mount your pages and how to get hold of the URL afterwards:
CompoundRequestMapper rm = new CompoundRequestMapper();
// mounting the pages
rm.add(new MountedMapper("mypage",MyPage.class));
// ... mount other pages ...
// create URL from page class and parameters
Class<? extends IRequestablePage> pageClass = MyPage.class;
PageParameters pp = new PageParameters();
pp.add("param1","value1");
IRequestHandler handler = new BookmarkablePageRequestHandler(new PageProvider(MyPage.class, pp));
Url url = rm.mapHandler(handler);
Original solution below:
After deep-diving into the intestines of the wicket sources, I was able to glue together this piece of code
IRequestMapper rm = MyWebApplication.get().getRootRequestMapper();
IRequestHandler handler = new BookmarkablePageRequestHandler(new PageProvider(pageClass, parameters));
Url url = rm.mapHandler(handler);
It works without a current RequestCycle, but still needs to have MyWebApplication running.
However, from Wicket's internal test classes, I have put the following together to construct a dummy instance of MyWebApplication:
MyWebApplication dummy = new MyWebApplication();
dummy.setName("test-app");
dummy.setServletContext(new MockServletContext(dummy, ""));
ThreadContext.setApplication(dummy);
dummy.initApplication();