I have code in nodeJs, which uses bunyan for logging, so what is happening is, the logs are showing access-token in error logs, which I want to mask, so for that I created custom stream such that access-token is masked in error logs, :
streams: [
{
level: 50,
stream: new MyStream()
}
]
and MyStream is :
class MyStream extends stream.Writable {
_write(chunk, encoding, done) {
process.stdout.write(maskAccesToken(chunk.toString()));
done()
}
}
Where maskAccessToken is a function that masks the access token. The logs that passes through this logger have access token masked , but certain files have this statement : throw e which is causing access token to be exposed in certain logs that are not passing through bunyan logger, how do I make it so that, the exception thrown from throw e, streams through the bunyan logger, is there a way to configure bunyan to stream exception thrown from throw statement?
Related
I'm trying to run an Apache Beam application in Kinesis Data Analytics which uses Apache Flink as the runtime. The pipeline uses the PubsubIO connector. I'm trying to authenticate with Google Cloud using code, since Kinesis Data Analytics does not allow to export environment variables, exporting GOOGLE_APPLICATION_CREDENTIALS environment variable doesn't seem to be an option.
I'm trying to authenticate using code as below.
GoogleCredentials credential = GoogleCredentials
.fromStream(credentialJsonInputStream)
.createScoped("https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/pubsub");
credential.refreshIfExpired();
options.setGcpCredential(credential);
The options reference here inherits PubsubOptions.
But when running the application it fails with the exception:
Exception in thread "main"
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403
Forbidden POST
https://pubsub.googleapis.com/v1/projects/my-project/topics/my-topic:publish
{ "code" : 403, "errors" : [ {
"domain" : "global",
"message" : "The request is missing a valid API key.",
"reason" : "forbidden" } ], "message" : "The request is missing a valid API key.", "status" : "PERMISSION_DENIED" } at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:371)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:339)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:219)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322) at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:308) at
com.amazonaws.kinesisanalytics.beam.BasicBeamStreamingJob.main(BasicBeamStreamingJob.java:67)
While debugging I noticed that the PubsubOptions reference passed to the org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient.PubsubJsonClientFactory#newClient returns null when calling GcpOptions#getGcpCredential
I'd really appreciate any insights on how to authenticate in this scenario.
GcpOptions#setGcpCredential option can’t be used with Flink runner, because the Flink runner serializes PipelineOptions, but the getGcpCredential is annotated with #JsonIgnore.
When no credential has been set explicitly via GcpOptions#setGcpCredential,
GCP services such as Pub/Sub uses a credential based upon the currently set GcpOptions#credentialFactoryClass.
So instead of calling options.setGcpCredential(credential), we can define a custom GcpCredentialFactory class. Then pass it to GcpOptions#credentialFactoryClass
options.setCredentialFactoryClass(CustomGcpCredentialFactory.class);
Your application's PipelineOptions interface would need to extend the GcpOptions interface, for you to be able to call the above method on your options reference.
public class CustomCredentialFactory extends GcpCredentialFactory {
private static CustomCredentialFactory INSTANCE = new CustomCredentialFactory();
private CustomCredentialFactory(PipelineOptions o) { }
/**
* Required by GcpOptions.GcpUserCredentialsFactory#create(org.apache.beam.sdk.options.PipelineOptions)
*/
public static CustomCredentialFactory fromOptions(PipelineOptions o) {
return new CustomCredentialFactory(o);
}
#Override
public Credentials getCredential() {
try {
// Load the GCP credential file (from S3, Jar, ..)
InputStream credentialFileInputStream = SomeUtil.getCredentialInputStream();
return GoogleCredentials
.fromStream(credentialFileInputStream)
.createScoped("https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/pubsub");
} catch (IOException e) {
return null;
}
}
}
I have a scala project that is connected to a couchbase database. I am making a get request to retrieve a document from the couchbase database. My method looks like this:
def findAll(): Future[String] = {
CouchDriver.plannerBucket.get[JsValue]("1").map(result => (result.get \ "area").as[String])
}
I'm invoking it all this in a http-akka router:
complete(
couchbaseRepository.findAll().map(v => {
HttpResponse(200, entity =
HttpEntity(ContentTypes.`application/json`, v))
})
)
However I get the following error:
Error during processing of request: 'java.lang.NullPointerException (No error message supplied)'. Completing with 500 Internal Server Error response. To change default exception handling behavior, provide a custom ExceptionHandler.
java.lang.NullPointerException: null
The odd thing is, when I change the method to this:
CouchDriver.plannerBucket.underlyingBucket.get("1")
And evaluate the expression in Intellij, it gets the document.
Not sure what the issue is!
Considering the scenario that one of the verticles throws an uncaught exception.
What happens next?
If the verticle state is removed from the system is there some mechanism similar to erlang supervisors to restart the verticle?
Documentation is not very clear about this aspect.
Update based on comments:
What interest me the most is the situation when an exception is thrown from the processing handlers of a received message (through the bus)
Regards
I have answered part of my own question (with the help of a test program)
When exception is thrown in a event handler then the exception is caught by vert.x and swallowed (ignored). The event handler will process the next message.
Update: The app can register an exception handler and have all the uncaught Throwable delivered to this handler. There you can perform additional general processing
Update2: Use Vertx.exceptionHandler to register the handler
Vert.x is all about the same style, asynchronous programming, which is mainly highlighted by callback handlers.
To handle the deployment failure case, you have first to go the programmatic way, i.e. you have to deploy your verticle programmatically through let's say a deployment verticle providing a completion handler that will be populated with deployment result, here down a sample using Java (since your haven't opt for a specific language, I will go with my best) where:
MainVerticle: is your deployment verticle (used mainly to deploy other verticles)
some.package.MyVerticle: is your real verticle, note that I used the id here and not an instance.
public class MainVerticle extends AbstractVerticle {
public void start() {
vertx.deployVerticle("some.package.MyVerticle", res -> {
if (res.succeeded()) {
// Do whatever if deployment succeeded
} else {
// Handle deployment failure here...
}
});
}
}
Now when it comes to 'messaging failures', it would be harder to highlight a specific case since it can occur at many places and on behalf of both messaging ends.
If you want to register a failure case handler when sending a message, you can instantiate a MessageProducer<T> representing the stream it can be written to, then register an exception handler on it:
EventBus eb = vertx.eventBus();
MessageProducer<String> sender = eb.sender("someAddress");
sender.exceptionHandler(e -> {
System.out.println("An error occured" + e.getCause());
});
sender.write("Hello...");
On the other side, you can handle failure case when reading the received messages pretty much the same way, but using a MessageConsumer<T> this time:
EventBus eb = vertx.eventBus();
MessageConsumer<String> receiver = eb.consumer("someAddress");
receiver.exceptionHandler(e -> {
System.out.println("An error occured while readeing data" + e.getCause());
}).handler(msg -> {
System.out.println("A message has been received: " + msg.body());
});
To add a bit to the previous answer, if you want to react to all uncaught exceptions, register handler on vertx object, as follows:
vertx.exceptionHandler(new Handler<Throwable>() {
#Override
public void handle(Throwable event) {
// do what you meant to do on uncaught exception, e.g.:
System.err.println("Error");
someLogger.error(event + " throws exception: " + event.getStackTrace());
}
});
I ran into something similar to this. When an exception happens as part of processing a message in a Verticle, I just wanted to reply with the Exception.
The idea is to just bubble up the exceptions all the way back to the entry point in the app where a decision can be made about what to do with the failure, while capturing the entire stack along the way.
To accomplish it I wrote this function:
protected ReplyException buildReplyException(Throwable cause, String message)
{
ReplyException ex = new ReplyException(ReplyFailure.RECIPIENT_FAILURE, -1, message);
ex.initCause(cause);
return ex;
}
Which I then use to build handlers, or reply handlers, like this:
reply -> {
if (reply.succeeded()) {
message.reply(true);
} else {
message.reply(buildReplyException(reply.cause(), ""));
}
});
This way the guy that sent the original message will get a failed response which contains a cause that's an exception which has a full stack trace populated on it.
This approach worked very well for me to handle errors while processing messages.
I'm using Play 2.3.7. I have a Global.onError method and it gets called when an exception is raised in an Action. However, it does not get called when an exception is raised in the constructor of a play.api.mvc.Controller. The default error page is served instead.
The code looks something like this:
object MyController extends Controller {
assert(false)
val something = Action { request => ??? }
}
The assertion failure happens the first time a request is routed to the controller. It is logged in the ! Internal server error, for ... format, but not handled by Global.onError. How could I handle this exception?
At client side:
factory.find(proxyId).fire(new Receiver<P>()
{
#Override
public void onSuccess( P response )
{
proxy = response;
...
}
#Override
public void onFailure( com.google.web.bindery.requestfactory.shared.ServerFailure error )
{
Window.alert( error.getMessage() );
}
}
at server side i use an Locator like below:
public class myLocator extends Locator<T, String>
{
#Injector LocatorHook hook;
#Override
public T find( Class<? extends T> clazz, String id )
{
T result = ...;
hook.run( result );
return result;
}
....
}
The hook.run() method may throwRunTimeException("validation exception") there, i expect to catch the
exception at client side in onFailure(), however, i did catch the exception, but the message is "Internal Server Error",
not the exception i thrown in hook.run():"validation exception".
Any ideas to let client catch the exception i throw at server side?
Updation:
As Thomas said it's weird that validating objects that come fresh from data store, but i encounter a
situation that i don't know how to use service method:
At client i get EntityProxyId object, through the factory.find( proxyId ).fire(...) i can get the entity
from datastore, but the entity may not suitable for the user to access, in this situation i need to check it at server side, but i can't find a suitable place to do the
validation, Any ideas about this?
RequestFactory doesn't expect exceptions to be thrown by locators. Exceptions should only be thrown by service methods, and will be directed to the appropriate Receiver on the client-side (the one attached to the service method that threw).
Outside service methods, the only exceptions that gets routed to the client are ReportableExceptions, that can only be thrown from a ServiceLocatorDecorator's report() methods. That means you could hook your own ServiceLocatorDecorator that catches exceptions from your locators and report()s them.
That said, validating objects that come fresh from your data store seems weird. You might want to provide a ServiceLocatorDecorator that overrides validate() (that'll validate the objects after the changes coming from the client have been applied). The errors will go back to the client in the Receiver's onConstraintViolations, and the RequestContext will be unfrozen so you can further edit your proxies and fire() again.