Illegal access error when deleting Google Pub Sub subscription upon JVM shutdown - scala

I'm trying to delete a Google Pub Sub subscription in a JVM shutdown hook, but I'm encountering an illegal access error with the Google Pub Sub subscription admin client when the shutdown hook runs. I've tried using both sys.addShutdownHook as well as Runtime.getRuntime().addShutdownHook, but I get the same error either way.
val deleteInstanceCacheSubscriptionThread = new Thread {
override def run: Unit = {
cacheUpdateService. deleteInstanceCacheUpdateSubscription()
}
}
sys.addShutdownHook(deleteInstanceCacheSubscriptionThread.run)
// Runtime.getRuntime().addShutdownHook(deleteInstanceCacheSubscriptionThread)
This is the stack trace:
Exception in thread "shutdownHook1" java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [META-INF/services/com.google.auth.http.HttpTransportFactory]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
at org.apache.catalina.loader.WebappClassLoaderBase.findResources(WebappClassLoaderBase.java:985)
at org.apache.catalina.loader.WebappClassLoaderBase.getResources(WebappClassLoaderBase.java:1086)
at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:348)
at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
at com.google.common.collect.Iterators.getNext(Iterators.java:845)
at com.google.common.collect.Iterables.getFirst(Iterables.java:779)
at com.google.auth.oauth2.OAuth2Credentials.getFromServiceLoader(OAuth2Credentials.java:318)
at com.google.auth.oauth2.ServiceAccountCredentials.<init>(ServiceAccountCredentials.java:145)
at com.google.auth.oauth2.ServiceAccountCredentials.createScoped(ServiceAccountCredentials.java:505)
at com.google.api.gax.core.GoogleCredentialsProvider.getCredentials(GoogleCredentialsProvider.java:92)
at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:142)
at com.google.cloud.pubsub.v1.stub.GrpcSubscriberStub.create(GrpcSubscriberStub.java:263)
at com.google.cloud.pubsub.v1.stub.SubscriberStubSettings.createStub(SubscriberStubSettings.java:242)
at com.google.cloud.pubsub.v1.SubscriptionAdminClient.<init>(SubscriptionAdminClient.java:178)
at com.google.cloud.pubsub.v1.SubscriptionAdminClient.create(SubscriptionAdminClient.java:159)
at com.google.cloud.pubsub.v1.SubscriptionAdminClient.create(SubscriptionAdminClient.java:150)
at com.company.pubsub.services.GooglePubSubService.$anonfun$deleteSubscription$2(GooglePubSubService.scala:384)
at com.company.utils.TryWithResources$.withResources(TryWithResources.scala:21)
at com.company.pubsub.services.GooglePubSubService.$anonfun$deleteSubscription$1(GooglePubSubService.scala:384)
at com.company.scalalogging.Logging.time(Logging.scala:43)
at com.company.scalalogging.Logging.time$(Logging.scala:35)
at com.company.pubsub.services.GooglePubSubService.time(GooglePubSubService.scala:30)
at com.company.pubsub.services.GooglePubSubService.deleteSubscription(GooglePubSubService.scala:382)
at com.company.cache.services.CacheUpdateService.deleteInstanceCacheUpdateSubscription(CacheUpdateService.scala:109)
at com.company.cache.services.CacheUpdateHandlerService$$anon$1.run(CacheUpdateHandlerService.scala:132)
at com.company.cache.services.CacheUpdateHandlerService$.$anonfun$addSubscriptionShutdownHook$1(CacheUpdateHandlerService.scala:135)
at scala.sys.ShutdownHookThread$$anon$1.run(ShutdownHookThread.scala:37)
It seems like by the time the shutdown hook runs the Pub Sub library has already shut down, so we can't access the subscription admin client anymore. But, I was wondering if there was anyway to delete the subscription before this happens.

Related

Amazon.Runtime.ClientConfig.get_RetryMode() deadlock

I'm running a .net Framework ASP.NET WebApi application on Elastic Beanstalk and it occasionally becomes unresponsive.
We've got some process dumps of w3wp.exe and the blocking thread gets stuck on a call to Amazon.Runtime.ClientConfig.get_RetryMode(). This causes a deadlock as it doesn't release a lock obtained in Amazon.Runtime.Internal.Util.Logger.GetLogger. Subsequent calls get blocked in GetLogger() waiting for the lock to be released which never happens.
Any idea why Amazon.Runtime.ClientConfig.get_RetryMode() doesn't return?
Blocking Call Stack
Amazon.Runtime.ClientConfig.get_RetryMode()+2f
Amazon.Runtime.AmazonServiceClient.BuildRuntimePipeline()+1cd
AWS.Logger.Core.AWSLoggerCore..ctor(AWS.Logger.AWSLoggerConfig, System.String)+1b0
AWS.Logger.Log4net.AWSAppender.ActivateOptions()+131
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseAppender(System.Xml.XmlElement)+47b
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.FindAppenderByReference(System.Xml.XmlElement)+1dc
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseChildrenOfLoggerElement(System.Xml.XmlElement, log4net.Repository.Hierarchy.Logger, Boolean)+110
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseRoot(System.Xml.XmlElement)+5f
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.Configure(System.Xml.XmlElement)+554
log4net.Repository.Hierarchy.Hierarchy.XmlRepositoryConfigure(System.Xml.XmlElement)+c9
log4net.Config.XmlConfigurator.InternalConfigure(log4net.Repository.ILoggerRepository, System.IO.Stream)+2ad
log4net.Config.XmlConfigurator.InternalConfigure(log4net.Repository.ILoggerRepository, System.IO.FileInfo)+18f
log4net.Config.XmlConfigurator.Configure(log4net.Repository.ILoggerRepository, System.Uri)+77
log4net.Core.DefaultRepositorySelector.ConfigureRepository(System.Reflection.Assembly, log4net.Repository.ILoggerRepository)+2d1
log4net.Core.DefaultRepositorySelector.CreateRepository(System.Reflection.Assembly, System.Type, System.String, Boolean)+2bf
log4net.Core.DefaultRepositorySelector.GetRepository(System.Reflection.Assembly)+3e
log4net.Core.LoggerManager.GetLogger(System.Reflection.Assembly, System.Type)+45
[[DebuggerU2MCatchHandlerFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+84
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+92
Amazon.Runtime.Internal.Util.InternalLog4netLogger..ctor(System.Type)+d8
Amazon.Runtime.Internal.Util.Logger..ctor(System.Type)+5f
Amazon.Runtime.Internal.Util.Logger.GetLogger(System.Type)+af
Amazon.Runtime.AppConfigAWSCredentials..ctor()+33
Amazon.Runtime.FallbackCredentialsFactory+<>c.b__10_0()+1f
Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean)+c7
Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementClient..ctor(Amazon.RegionEndpoint)+3b
// UserCode new Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementClient(region)
Blocked Call Stack
System.Threading.Monitor.Enter(System.Object)
Amazon.Runtime.Internal.Util.Logger.GetLogger(System.Type)+68
Amazon.Runtime.AmazonServiceClient..ctor(Amazon.Runtime.AWSCredentials, Amazon.Runtime.ClientConfig)+8d
// UserCode new Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementClient(region)
I think the dead lock was caused by fallback process.
Aws Appender
-> obtain Logger
-> Would release Logger after setting up finish
-> setting up is missing some info , call fallback to figure out
-> failed connection, get_RetryMode()
-> trying to obtain Logger (dead lock generated)

java.lang.RuntimeException: Unable to start receiver (exception showed up in Google play store internal testing)

I deployed my flutter app to Google play store internal testing and after one hour, I saw this exception in the Pre-launch report details. I am not even sure where to start debugging. Could Someone please help? I don't have a lot of knowledge of Android since I am using Flutter.
Exception java.lang.RuntimeException: Unable to start receiver
com.google.android.finsky.instantapps.PhenotypeUpdateReceiver:
java.lang.IllegalStateException: Not allowed to start service Intent {
act=com.google.android.gms.phenotype.UPDATE
cmp=com.android.vending/com.google.android.finsky.instantapps.PhenotypeUpdateService
}: app is in background uid UidRecord{2a75503 u0a192 TRNB
bg:+2m53s648ms idle change:uncached procs:4 seq(0,0,0)} at
android.app.ActivityThread.handleReceiver (ActivityThread.java:4035)
at android.app.ActivityThread.access$1400 (ActivityThread.java:237)
at android.app.ActivityThread$H.handleMessage
(ActivityThread.java:1924) at android.os.Handler.dispatchMessage
(Handler.java:106) at android.os.Looper.loop (Looper.java:223) at
android.app.ActivityThread.main (ActivityThread.java:7664) at
java.lang.reflect.Method.invoke (Method.java) at
com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run
(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main
(ZygoteInit.java:947) Caused by java.lang.IllegalStateException: Not
allowed to start service Intent {
act=com.google.android.gms.phenotype.UPDATE
cmp=com.android.vending/com.google.android.finsky.instantapps.PhenotypeUpdateService
}: app is in background uid UidRecord{2a75503 u0a192 TRNB
bg:+2m53s648ms idle change:uncached procs:4 seq(0,0,0)} at
android.app.ContextImpl.startServiceCommon (ContextImpl.java:1715)
at android.app.ContextImpl.startService (ContextImpl.java:1670) at
android.content.ContextWrapper.startService (ContextWrapper.java:720)
at alj.a (alj.java:2) at alm.a (alm.java:3) at wnv.a (wnv.java:2)
at
com.google.android.finsky.instantapps.PhenotypeUpdateReceiver.onReceive
(PhenotypeUpdateReceiver.java:6) at
android.app.ActivityThread.handleReceiver (ActivityThread.java:4026)
I also managed to download the full logcat which is very large file. I am not sure how to include the file over here.
flutter downgrade to as much as possible
You have to uninstall and reinstall the app, and also have to clear the data and cache from the Settings page.
Try it.
I have solved the problem this way.

SAM Deployment failed Error- Waiter StackCreateComplete failed: Waiter encountered a terminal failure state

When I try to deploy package on SAM, the very first status comes in cloud formation console is ROLLBACK_IN_PROGRESS after that it gets changed to ROLLBACK_COMPLETE
I have tried deleting the stack and trying again, but every time same issue occurs.
Error in terminal looks like this-
Sourcing local options from ./SAMToolkit.devenv
SAM_PARAM_PKG environment variable not set
SAMToolkit will operate in legacy mode.
Please set SAM_PARAM_PKG in your .devenv file to run modern packaging.
Run 'sam help package' for more information
Runtime: java
Attempting to assume role from AWS Identity Broker using account 634668058279
Assumed role from AWS Identity Broker successfully.
Deploying stack sam-dev* from template: /home/***/1.0/runtime/sam/template.yml
sam-additional-artifacts-url.txt was not found, which is fine if there is no additional artifacts uploaded
Replacing BATS::SAM placeholders in template...
Uploading template build/private/tmp/sam-toolkit.yml to s3://***/sam-toolkit.yml
make_bucket failed: s3://sam-dev* An error occurred (BucketAlreadyOwnedByYou) when calling the CreateBucket operation: Your previous request to create the named bucket succeeded and you already own it.
upload: build/private/tmp/sam-toolkit.yml to s3://sam-dev*/sam-toolkit.yml
An error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id sam-dev* does not exist
sam-dev* will be created.
Creating ChangeSet ChangeSet-2020-01-20T12-25-56Z
Deploying stack sam-dev*. Follow in console: https://aws-identity-broker.amazon.com/federation/634668058279/CloudFormation
ChangeSet ChangeSet-2020-01-20T12-25-56Z in sam-dev* succeeded
"StackStatus": "REVIEW_IN_PROGRESS",
sam-dev* reached REVIEW_IN_PROGRESS
Deploying stack sam-dev*. Follow in console: https://console.aws.amazon.com/cloudformation/home?region=us-west-2
Waiting for stack-create-complete
Waiter StackCreateComplete failed: Waiter encountered a terminal failure state
Command failed.
Please see the logs above.
I set SQS as event source for Lambda, but didn't provided the permissions like this
- Effect: Allow
Action:
- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:GetQueueAttributes
Resource: "*"
in lambda policies.
I found this error in "Events" tab of "CloudFormation" service.

Partition is below target replica or instance count

When attempting to publish a Service Fabric application to a local cluster, the cluster fails to load the application stating the error in the title. The stack trace points me to an exception line in OwinCommunicationListener.cs:
try
{
this.eventSource.LogInfo("Starting web server on " + this.listeningAddress);
this.webApp = WebApp.Start(this.listeningAddress, appBuilder => this.startup.Invoke(appBuilder));
this.eventSource.LogInfo("Listening on " + this.publishAddress);
return Task.FromResult(this.publishAddress);
}
catch (Exception ex)
{
var logString = $"Web server failed to open endpoint {endpointName}. {ex.ToString()}";
this.eventSource.LogFatal(logString);
this.StopWebServer();
throw ex; // points to this line from cluster manager
}
I am unable to inspect the exception thrown, but there is no useful exception information other than a TargetInvocationException with a stack trace to the line noted above. Why won't this application load on my local cluster?
It's hard to say without an actual exception message or stack trace but judging by the location from which the exception was thrown and the fact that the problem resolved itself the next morning, the most likely and most common cause of this is that the port you were trying to use to open the web listener was taken by some other process at the time, and the next morning the port was free again. This, by the way, isn't really specific to Service Fabric. You're just trying to open a socket on a port that was taken by someone else.
I'm honestly more curious about why you couldn't inspect the exception. I can think of three things off the top of my head to help with that:
Use "throw" instead of "throw ex" so you don't reset the stack trace.
Look at your logs. It looks like you're writing out an ETW event in your catch block. What did it say?
Use the Visual Studio debugger: Simply set a breakpoint in the catch block and start the application with debugging by pressing F5.

Handling connection failures in apache-camel

I am writing an apache-camel RabbitMQ consumer. I would like to react somehow to connection problems (i.e. try to reconnect). Is it possible to configure apache-camel to automatically reconnect?
If not, how can I find out that a connection to the queue was interrupted? I've done the following test:
start the queue (and some producer)
start my consumer (it was getting messages as expected)
stop the queue (the messages stopped arriving, as expected, but no exception was thrown)
start the queue (no new messages were received)
I am using camel in Scala (via akka-camel), but a Java solution would be probably also OK
You can pass in the flag automaticRecoveryEnabled=true to the URI, Camel will reconnect if the connection is lost.
For automatic RabbitMQ resource recovery (Connections/Channels/Consumers/Queues/Exchanages/Bindings) when failures occur, check out Lyra (which I authored). Example usage:
Config config = new Config()
.withRecoveryPolicy(new RecoveryPolicy()
.withMaxAttempts(20)
.withInterval(Duration.seconds(1))
.withMaxDuration(Duration.minutes(5)));
ConnectionOptions options = new ConnectionOptions().withHost("localhost");
Connection connection = Connections.create(options, config);
The rest of the API is just the amqp-client API, except your resources are automatically recovered when failures occur.
I'm not sure about camel-rabbitmq specifically, but hopefully there's a way you can swap in your own resource creation via Lyra.
Current camel-rabbitmq just create a connection and the channel when the consumer or producer is started. So it don't have a chance to catch the connection exception :(.