I'm trying to run an Apache Beam application in Kinesis Data Analytics which uses Apache Flink as the runtime. The pipeline uses the PubsubIO connector. I'm trying to authenticate with Google Cloud using code, since Kinesis Data Analytics does not allow to export environment variables, exporting GOOGLE_APPLICATION_CREDENTIALS environment variable doesn't seem to be an option.
I'm trying to authenticate using code as below.
GoogleCredentials credential = GoogleCredentials
.fromStream(credentialJsonInputStream)
.createScoped("https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/pubsub");
credential.refreshIfExpired();
options.setGcpCredential(credential);
The options reference here inherits PubsubOptions.
But when running the application it fails with the exception:
Exception in thread "main"
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403
Forbidden POST
https://pubsub.googleapis.com/v1/projects/my-project/topics/my-topic:publish
{ "code" : 403, "errors" : [ {
"domain" : "global",
"message" : "The request is missing a valid API key.",
"reason" : "forbidden" } ], "message" : "The request is missing a valid API key.", "status" : "PERMISSION_DENIED" } at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:371)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:339)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:219)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322) at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:308) at
com.amazonaws.kinesisanalytics.beam.BasicBeamStreamingJob.main(BasicBeamStreamingJob.java:67)
While debugging I noticed that the PubsubOptions reference passed to the org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient.PubsubJsonClientFactory#newClient returns null when calling GcpOptions#getGcpCredential
I'd really appreciate any insights on how to authenticate in this scenario.
GcpOptions#setGcpCredential option can’t be used with Flink runner, because the Flink runner serializes PipelineOptions, but the getGcpCredential is annotated with #JsonIgnore.
When no credential has been set explicitly via GcpOptions#setGcpCredential,
GCP services such as Pub/Sub uses a credential based upon the currently set GcpOptions#credentialFactoryClass.
So instead of calling options.setGcpCredential(credential), we can define a custom GcpCredentialFactory class. Then pass it to GcpOptions#credentialFactoryClass
options.setCredentialFactoryClass(CustomGcpCredentialFactory.class);
Your application's PipelineOptions interface would need to extend the GcpOptions interface, for you to be able to call the above method on your options reference.
public class CustomCredentialFactory extends GcpCredentialFactory {
private static CustomCredentialFactory INSTANCE = new CustomCredentialFactory();
private CustomCredentialFactory(PipelineOptions o) { }
/**
* Required by GcpOptions.GcpUserCredentialsFactory#create(org.apache.beam.sdk.options.PipelineOptions)
*/
public static CustomCredentialFactory fromOptions(PipelineOptions o) {
return new CustomCredentialFactory(o);
}
#Override
public Credentials getCredential() {
try {
// Load the GCP credential file (from S3, Jar, ..)
InputStream credentialFileInputStream = SomeUtil.getCredentialInputStream();
return GoogleCredentials
.fromStream(credentialFileInputStream)
.createScoped("https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/pubsub");
} catch (IOException e) {
return null;
}
}
}
Related
I have Java web application which implemented Apache shiro Authentication & Authorization.
Now i need to implement API Key to the existing project (which has apache shiro).
Please help me on implementation part. Even i could not find any documentation
PS:: We have already implemented 3 different types of Custom Realm(jdbc,ldap,Pac4jRealm) but now struggling to implement the API key concept with Apache Shiro.
I resolved the above issue by extending the JDBCRealm,see the below example code
public class APIRealm extends JdbcRealm {
#Override
protected AuthorizationInfo doGetAuthorizationInfo(PrincipalCollection principals) {
SimpleAuthorizationInfo info = null;
AuthAPIInfo authInfo = null;
try {
String apiKey= (String) principals.getPrimaryPrincipal();
authInfo=fetchAPIKeyInfo(apiName);
// Do all the other stuff like checking for Authorization and setting it to token
} catch (Exception e) {
insertAPILogActivity(authInfo, "User not authorized");
}
return info;
}
private AuthAPIInfo fetchAPIKeyInfo(String apiKeyName) {
//Connect to Database using JDBC connection and validate the API Key and return the AuthAPIInfo
}
}
Add the above realm in shiro.ini
apiRealm=com.example.APIRealm
securityManager.realms=$apiRealm
Trying to create an HttpTrigger v2 Azure Function with CosmosDB bidning that will look up ID from route data using SqlQuery. This is very similar to example provided by Microsoft here.
I am developing this function using VS Code. Here is the code:
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using System.Collections.Generic;
namespace some_namespace
{
public static class GetUser
{
[FunctionName("GetUser")]
public static IActionResult Run(
[HttpTrigger(
AuthorizationLevel.Anonymous,
"get",
Route = "user/{id}")] HttpRequest req,
[CosmosDB(
"DbName",
"Users",
ConnectionStringSetting = "CosmosDBConnection",
SqlQuery = "select * from Users u where u.id = {id}")]
IEnumerable<User> users,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
foreach (User user in users)
{
log.LogInformation(user.id);
}
return new OkResult();
}
}
}
User class is defined in another file. All compiles fine and I can upload it to Azure but when I navigate to the function, I get the following error:
Function (.../GetUser) Error: Microsoft.Azure.WebJobs.Host: Error indexing method 'GetUser'. System.Private.CoreLib: Could not load file or assembly 'Microsoft.Azure.WebJobs.Extensions.CosmosDB, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621). System.Private.CoreLib: Could not load file or assembly 'Microsoft.Azure.WebJobs.Extensions.CosmosDB, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.
I have modified my host.json file to include ExtensionBundle as per this article and it currently looks like this:
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
}
}
The above was supposed to automagically include the reference to CosmosDB extension with my function but it seems that it did not or I am missing something else but don't know what.
Try installing Microsoft.Azure.WebJobs.Extensions.CosmosDB as a NuGet package instead.
The ExtensionBundle seems to be for local development.
I am not sure what fixed the above function to make it work. At one point in time, I have added a redundant import for using Microsoft.Azure.WebJobs.Extensions.CosmosDB but I have subsequently removed it and the function still works. The above code will not return found objects - below is a modified, working function (logging of each returned user is not required but I left it there for debugging purposes):
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using System.Collections.Generic;
namespace some_namespace
{
public static class GetUser
{
[FunctionName("GetUser")]
public static IActionResult Run(
[HttpTrigger(
AuthorizationLevel.Anonymous,
"get",
Route = "user/{id}")] HttpRequest req,
[CosmosDB(
"DbName",
"Users",
ConnectionStringSetting = "CosmosDBConnection",
SqlQuery = "select * from Users u where u.id = {id}")]
IEnumerable<User> users,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
foreach (User user in users)
{
log.LogInformation(user.id);
}
return new OkObjectResult(users);
}
}
}
How do I configure my spring boot service so that errors such as 500 don't potentially leak implementation details such as stacktraces.
{
"timestamp": "2019/05/01 15:06:17",
"status": 500,
"error": "Internal Server Error",
"message": "Type definition error: [simple type, class net.i2p.crypto.eddsa.math.ed25519.Ed25519LittleEndianEncoding]; nested exception is com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class net.i2p.crypto.eddsa.math.ed25519.Ed25519LittleEndianEncoding and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: java.util.Collections$UnmodifiableRandomAccessList[0]->........)",
"path": "/api/test"
}
Note: here the stacktrace is in the message and not the exception part of the json.
As you can see I am already formatting the timestamp with:
#Component
public class CustomErrorAttributes extends DefaultErrorAttributes {
private static final DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
private static final String TIMESTAMP = "timestamp";
#Override
public Map<String, Object> getErrorAttributes(WebRequest webRequest, boolean includeStackTrace) {
//Let Spring handle the error first
Map<String, Object> errorAttributes = super.getErrorAttributes(webRequest, includeStackTrace);
//Format & update timestamp
Object timestamp = errorAttributes.get(TIMESTAMP);
if(timestamp == null) {
errorAttributes.put(TIMESTAMP, dateFormat.format(new Date()));
} else {
errorAttributes.put(TIMESTAMP, dateFormat.format((Date)timestamp));
}
return errorAttributes;
}
}
But I need to handle the message too.
If this 500 was the only error I could just do:
errorAttributes.put("message", "Server error. Contact support.");
However, all the errors go through here and that would override all the messages.
I could check if the status is 500 and only modify it then. However, there are other errors that can be generated that also might leak stacktraces.
Using #RestControllerAdvice seems to require knowing every exception that is generated and having an #ExceptionHandler for each and knowing which status code to respond with.
Is there a cleaner way to handle this?
It may not be the "cleanest" approach, but with projects I've been on we had a "standard format" for our Error Responses across projects, so we had a custom object with the fields that matched our orgs standard (HttpStatus, Reason, ect.) that extended RuntimeException. Then in our controllers, services, repos, ect we would catch exceptions and create this object accordingly and throw the custom one up instead. Based upon where it happened in the app (repo, service, controller, ect.) we could give our own custom verbage to it, but still log out the full exception in our server logs so we could investigate later
For example if we caught an error in our repository we would create our custom error object, set the Reason to DB unavailable (really all the consumer needs to know), set the status to HttpStatus.SERVICE_UNAVAILABLE (we tracked these with reasons and httpstatus with enums to keep status the same across modules), and throw the custom object up to the controller to be returned.
Sorry if this was a longwinded answer that may not give you what you want, I'm not too familiar with how you're trying to do it so figured I'd just give an example of other methods. I'll put some sample code as well
Custom Exception:
data class MyException(
val reason: String,
val httpStatus: HttpStatus? = null
) : RuntimeException(reason)
Method for creation:
fun createApiException(errorCode: ErrorCodeEnum) = MyException(
reason = errorCode.reason,
httpStatus = errorCode.httpStatus,
)
Spring-boot provides us with a standard method to handle exceptions using spring aop concept. You can use the #ControllerAdvice and #Exceptionhandled annotations to handle exceptions from a spring-boot rest endpoint so that a custom exception is always thrown from a rest endpoint with proper error code and error response.
The #ResponseStatus() annotation can be used to customize the response code being thrown.
For example consider the custom exception :
#ResponseStatus(HttpStatus.NOT_FOUND)
public class DataNotFoundException extends RuntimeException {
public DataNotFoundException(String exception) {
super(exception);
}
}
We can throw this error from a rest GET mapping when a data is not found like :
#GetMapping("/trains/{id}")
public Resource<Student> retrieveTrains(#PathVariable long id) {
Optional<Trains> trains = trainRepository.findById(id);
if (!train.isPresent())
throw new DataNotFoundException("id-" + id);
Resource<Trains> resource = new Resource<Trains>(train.get());
ControllerLinkBuilder linkTo = linkTo(methodOn(this.getClass()).retrieveAllTrains());
resource.add(linkTo.withRel("all-trains"));
return resource;
}
Default error response provided by Spring Boot contains all the details that are typically needed.
However, you might want to create a framework independent response structure for your organization. In that case, you can define a specific error response structure.
For example :
public class ErrorDetails {
private Date timestamp;
private String message;
private String details;
public ErrorDetails(Date timestamp, String message, String details) {
super();
this.timestamp = timestamp;
this.message = message;
this.details = details;
}
To use this error node we use :
#ControllerAdvice
public class CustomizedResponseEntityExceptionHandler extends ResponseEntityExceptionHandler {
#ExceptionHandler(DataNotFoundException.class)
public final ResponseEntity<ErrorDetails> handleUserNotFoundException(DataNotFoundException ex, WebRequest request) {
ErrorDetails errorDetails = new ErrorDetails(new Date(), ex.getMessage(),
request.getDescription(false));
return new ResponseEntity<>(errorDetails, HttpStatus.NOT_FOUND);
}
#ExceptionHandler(DataNotFoundException.class) indicates that this
method would handle exceptions of the specific type.
new ResponseEntity<>(errorDetails, HttpStatus.NOT_FOUND) - Create an
error response object and return it with a specific Http Status.
For a more generalized exception handler you can define a method that handles exception of the type Exception.class, that way you don't have to know every exception.
Like :
#ExceptionHandler(Exception.class)
public final ResponseEntity<ErrorDetails> handleAllExceptions(Exception ex, WebRequest request) {
ErrorDetails errorDetails = new ErrorDetails(new Date(), ex.getMessage(),
request.getDescription(false));
return new ResponseEntity<>(errorDetails, HttpStatus.INTERNAL_SERVER_ERROR);
}
Reference from : https://www.javaguides.net/2019/02/spring-boot-2-angular-7-crud-example-tutorial.html
Is it possible to call services or actors from one application to another in a Service Fabric Cluster ? When I tryed (using ActorProxy.Create with the proper Uri), I got a "No MethodDispatcher is found for interface"
Yes, it is possible. As long as you have the right Uri to the Service (or ActorService) and you have access to the assembly with the interface defining your service or actor the it should not be much different than calling the Service/Actor from within the same application. It you have enabled security for your service then you have to setup the certificates for the exchange as well.
If I have a simple service defined as:
public interface ICalloutService : IService
{
Task<string> SayHelloAsync();
}
internal sealed class CalloutService : StatelessService, ICalloutService
{
public CalloutService(StatelessServiceContext context)
: base(context) { }
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
yield return new ServiceInstanceListener(this.CreateServiceRemotingListener);
}
public Task<string> SayHelloAsync()
{
return Task.FromResult("hello");
}
}
and a simple actor:
public interface ICalloutActor : IActor
{
Task<string> SayHelloAsync();
}
[StatePersistence(StatePersistence.None)]
internal class CalloutActor : Actor, ICalloutActor
{
public CalloutActor(ActorService actorService, ActorId actorId)
: base(actorService, actorId) {}
public Task<string> SayHelloAsync()
{
return Task.FromResult("hello");
}
}
running in a application like this:
Then you can call it from another application within the same cluster:
// Call the service
var calloutServiceUri = new Uri(#"fabric:/ServiceFabric.SO.Answer._41655575/CalloutService");
var calloutService = ServiceProxy.Create<ICalloutService>(calloutServiceUri);
var serviceHello = await calloutService.SayHelloAsync();
// Call the actor
var calloutActorServiceUri = new Uri(#"fabric:/ServiceFabric.SO.Answer._41655575/CalloutActorService");
var calloutActor = ActorProxy.Create<ICalloutActor>(new ActorId(DateTime.Now.Millisecond), calloutActorServiceUri);
var actorHello = await calloutActor.SayHelloAsync();
You can find the right Uri in the Service Fabric Explorer if you click the service and look at the name. By default the Uri of a service is: fabric:/{applicationName}/{serviceName}.
The only tricky part is how do you get the interface from the external service to your calling service? You could simply reference the built .exe for the service you wish to call or you could package the assembly containing the interface as a NuGet package and put on a private feed.
If you don't do this and you instead just share the code between your Visual Studio solutions the Service Fabric will think these are two different interfaces, even if they share the exact same signature. If you do it for a Service you get an NotImplementedException saying "Interface id '{xxxxxxxx}' is not implemented by object '{service}'" and if you do it for an Actor you get an KeyNotfoundException saying "No MethodDispatcher is found for interface id '-{xxxxxxxxxx}'".
So, to fix your problem, make sure you reference the same assembly that is in the application you want to call in the external application that is calling.
I am creating a REST service using jersey 2.0. I am extending WebApplicationException
Method raising a particular exception
if(json.equals("") || json.equals(" ")) {
throw new ArgumentException("bad post data");
}
public class ArgumentException extends RestException {
.....
public ArgumentException(String message) {
super(Status.BAD_REQUEST,message);
}
}
public class RestException extends WebApplicationException {
...........
public RestException(Status status, String message) {
super(Response.status(status)
.entity(message)
.type("text/plain")
.build());
/*
super(Response.status(status)
.entity(new ErrorBean(status.getStatusCode(),message))
.type(MediaType.APPLICATION_JSON)
.build()); */
}
ErrorBean is a POJO
The method that returns error as plain string inside RestException works (right http code 400 and message). However when I try to pass the ErrorBean POJO and use MediaType.APPLICATION_JSON in response I get an error saying "Headers have already been sent" with http error code 500 (so some internal problem with plumbing) and empty response.
I have also looked at this question Returning JSON or XML for Exceptions in Jersey
How can I return the exception with code and message as a JSON like
{"code" : 400, "message" : .... }
Update
I have received answer on SO as well as jersey users mailing list. steps are
A non AJXB POJO does not need any annotations
Register JacksonFeature in your application
ResourceConfig rc = new ResourceConfig().packages("test").register(JacksonFeature.class);
You need to register JacksonFeature in your Application/ResourceConfig, i.e.:
// Create JAX-RS application.
final Application application = new ResourceConfig()
.packages("org.glassfish.jersey.examples.jackson")
.register(JacksonFeature.class)
// No need to register this provider if no special configuration is required.
.register(MyObjectMapperProvider.class);
Take a look at the documentation for Jackson support in Jersey and also at the example.