I want to send the rest API data to Elasticsearch from logstash. But I don't know how I can API URL and API key to logstash conf file.
enter image description here
Here I am sharing my logstash conf file.
You required to set Authorization header for api key like below:
input {
http_poller {
urls => {
test1 => {
method => get
url => "https://my-api-url"
headers => {
Accept => "application/json"
Authorization => "Basic ZWxhc3RpY"
}
}
}
}
}
Related
I'm trying to use ELK pipeline to read an email (IMAP), extract generic attachments (mainly PDF, eventually doc or ppt) and put them on ElasticSearch.
This is what I was able to do:
Loading directly to ElasticSearch from file some base64 data using Logstash, using the Ingest Attachment Processor on ElasticSearch to read the base64 content.
Loading data from IMAP (exchange email) I can correctly load all email information on ElasticSearch except the attachment (what I need).
The first solution works fine and does what I am looking for, except that it doesn't extract attachments directly from the email and that I have hardcoded base64 data inside the files.
With the second solution I have a field x-ms-has-attach: yes on Kibana, but there isn't anywhere the attachment itself. The imap plugin is intended to load only the content of the email without the attachment?
What am I missing? Could you suggest me a pipeline to achieve what I am looking for?
This is my logstash configuration for the first example:
input {
file {
path => "/my/path/to/data/*"
start_position => "beginning"
# sincedb_path => "/my/path/to/sincedb"
sincedb_path => "/dev/null"
close_older => 0
tags => ["attachment"]
}
}
output {
elasticsearch {
index => "email-attachment"
hosts => [ "localhost:9200" ]
}
}
This is the pipeline:
PUT _ingest/pipeline/email-attachment
{
"description": "Pipeline to parse an email and its attachments",
"processors": [
{
"attachment" : {
"field" : "message"
}
},
{
"remove" : {
"field" : "message"
}
},
{
"date_index_name" : {
"field" : "#timestamp",
"index_name_prefix" : "email-attachment-",
"index_name_format": "yyyy-MM",
"date_rounding" : "M"
}
}
]
}
This is my logstash configuration for the second example:
input {
imap {
host => "my.domain.it"
password => "mypassword"
user => "myuser"
port => 12345
type => "imap"
secure => true
strip_attachment => true
}
}
output {
elasticsearch {
index => "email-attachment"
hosts => [ "localhost:9200" ]
}
}
UPDATE
I'm using version 5.2.2
In the end I defined a totally different pipeline.
I read emails using a Ruby application with the mail library (you can find it on github), where it's quite easy to extract attachments.
Then I put the base64 encoding of those attachments directly on ElasticSearch, using Ingest Attachment Processor.
I filter on content_type just to be sure to load only "real" attachments, as the multiparts emails treat any multimedial content in the body (ie: images) as attachment.
P.S.
Using the mail library, you should do something like:
Mail.defaults do
retriever_method :imap, { :address => address,
:port => port,
:user_name => user_name,
:password => password,
:enable_ssl => enable_ssl,
:openssl_verify_mode => openssl_verify_mode }
and new_messages = Mail.find(keys: ['NOT','SEEN']) to retrieve unseen messages.
Then iterate over new_messages. After, you can encode a message simply using encoded = Base64.strict_encode64(attachment.body.to_s). Please inspect new_messages to check the exact field names to use.
Your problem might come from strip_attachment => true in the imap input plugin.
Can someone share me a sample application , that has REST APIs to upload images to Amazon S3 . I've googled a lot , but most of them are not working as expected .
A little help would be greatly appreciated .
For uploading images through API you can convert the images to base64 format in front end and send it to server. In, server you can convert the base64 data into the image and save it to S3 via paperclip.
class User < ActiveRecord::Base
before_save :decode_picture_data, :if => :picture_data_provided?
has_attached_file :avatar,
:storage => :s3, :s3_protocol => 'https',
:bucket => # S3_Bucket,
:url => '/user/:id/:basename_:id.:extension',
:s3_credentials => {
:access_key_id => # Access key id,
:secret_access_key => # Secret access key
},
:path => '/user/:id/:basename_:id.:extension',
:default_url => ""
validates_attachment_content_type :avatar, :content_type => %w(image/jpeg image/jpg image/png)
private
def picture_data_provided?
!self.picture_data.blank?
end
def decode_picture_data
# If cover_image_data is set, decode it and hand it over to Paperclip
data = StringIO.new(Base64.decode64(self.picture_data))
data.class.class_eval { attr_accessor :original_filename, :content_type }
data.original_filename = SecureRandom.hex(16) + ".png"
data.content_type = "image/png"
self.avatar = data
self.picture_data = nil
end
end
Here, picture_data is a column in User table which has the base64 data received from the client through API params.
You can specify your own S3 URL and path in above code.
In the above code the picture_data that came from the client side is decoded and saved in S3 via Paperclip.
I am building an Android app that is interfaced with a CakePHP 3 web API. Since a RESTful API cannot rely on cookies, I understand that I need JSON web tokens (JWT) to make this happen, and would much prefer to use a google login. I already got the Android side to request a token from Google's API, but now I am lost on how to incorporate this into my API for authentication.
I've searched around for some tutorials, such as this one: http://blog.jainsiddharth21.com/2013/04/29/login-with-google-in-cakephp/, but it relies on session data. I'm building the API in CakePHP 3, so I've looked at some of the plugins, such as ADmad/JwtAuth, so maybe I could extend on this to allow google authentication, but I am not sure how.
Login With Gmail and Specific email address to allowed to login in CakePHP 3.x
Composer to install
"google/apiclient": "^2.0"
Required gmail with login
https://console.developers.google.com/apis/credentials?project=mdpms-187410&organizationId=1095988912954
Create project and create secret key and client id
Project in set name and redirect URL
NOTE:- redirect URL must be .com and .org domain
If you develop in local machine then create follow type of virtual host
example.com and example.org
Virtual host create then
Follow this step:
Set configuration file in app_globle.php
'Google' =>
[
'googleClientID' => '123456.apps.googleusercontent.com',
'googleClientSecret' => 'abcdefghi',
'googleRedirectUrl' => 'http://example.com/oauth2callback'
]
Gmail login route
//Google login
$routes->connect('/account/google-login', ['controller' => 'Account', 'action' => 'googlelogin'], ['_name' => 'account-google-login']);
$routes->connect('/oauth2callback', ['controller' => 'Account', 'action' => 'confirmlogin'], ['_name' => 'account-google-redirect-url']);
Google login action code:
/**
* Gmail login method
*/
public function googlelogin()
{
$client = new Google_Client();
$client->setClientId(Configure::read('Google.googleClientID'));
$client->setClientSecret(Configure::read('Google.googleClientSecret'));
$client->setRedirectUri(Configure::read('Google.googleRedirectUrl'));
$client->se
tScopes([
"https://www.googleapis.com/auth/userinfo.profile",
'https://www.googleapis.com/auth/userinfo.email'
]);
$url = $client->createAuthUrl();
$this->redirect($url);
}
Google redirect url Action
/**
* Gmail auth redirect action
* #return type gmail auth data
*/
public function confirmlogin()
{
$client = new Google_Client();
$client->setClientId(Configure::read('Google.googleClientID'));
$client->setClientSecret(Configure::read('Google.googleClientSecret'));
$client->setRedirectUri(Configure::read('Google.googleRedirectUrl'));
$client->setScopes([
"https://www.googleapis.com/auth/userinfo.profile",
'https://www.googleapis.com/auth/userinfo.email'
]);
$client->setApprovalPrompt('auto');
$usersTable = TableRegistry::get('Users');
if (isset($this->request->query['code'])) {
$client->authenticate($this->request->query['code']);
$this->request->Session()->write('access_token', $client->getAccessToken());
}
if ($this->request->Session()->check('access_token') && ($this->request->Session()->read('access_token'))) {
$client->setAccessToken($this->request->Session()->read('access_token'));
}
if ($client->getAccessToken()) {
$this->request->Session()->write('access_token', $client->getAccessToken());
$oauth2 = new Google_Service_Oauth2($client);
$user = $oauth2->userinfo->get();
try {
if (!empty($user)) {
if ((preg_match("/(#example\.com)$/", $user['email'])) || (preg_match("/(#example\.in)$/", $user['email']))) {
$result = $usersTable->find('all')
->where(['email' => $user['email']])
->first();
if (!empty($result)) {
$this->AccessControl->setUser($result->toArray(), false);
$this->Flash->set(__('You have successfuly logged in.'), ['element' => 'success']);
$this->redirect(['_name' => 'dashboard']);
} else {
$data = [];
$data['email'] = $user['email'];
$data['first_name'] = $user['givenName'];
$data['last_name'] = $user['familyName'];
$data['socialId'] = $user['id'];
$data['role_id'] = Configure::read('Role.loginWithGmailUserRole');
//$data matches my Users table
$entity = $usersTable->newEntity($data);
if ($usersTable->save($entity)) {
$data['id'] = $entity->id;
$this->AccessControl->setUser($data, false);
$this->Flash->set(__('You have successfuly logged in.'), ['element' => 'success']);
$this->redirect(['_name' => 'dashboard']);
} else {
$this->Flash->error(__('Invalid login.'));
//redirect to login action
$this->redirect(['_name' => 'account-login']);
}
}
} else {
$this->Flash->error(__('Your email is invalid for this application.'));
//redirect to login action
$this->redirect(['_name' => 'account-login']);
}
} else {
$this->Flash->error(__('Gmail infos not found.'));
//redirect to login action
return $this->redirect(['_name' => 'account-login']);
}
} catch (\Exception $e) {
$this->Flash->error(__('Gmail error.'));
return $this->redirect(['_name' => 'account-login']);
}
}
}
my controller file inside api/v1/controller/
class ProfileController extends ActiveController
{
public $modelClass = 'app\models\Profile';
public function behaviors()
{
return [
[
'class' => 'yii\filters\ContentNegotiator',
'only' =>
['index', 'view', 'createnew','update','search'],
'formats' =>
['application/json' => Response::FORMAT_JSON,],
],
'verbs' => [
'class' => VerbFilter::className(),
'actions' => [
'index' => ['get'],
'view' => ['get'],
'createnew' => ['post'],
'update' => ['put'],
'delete' => ['delete'],
'deleteall' => ['post'],
'search' => ['get']
],
]
];
}
public function actionCreatenew() {
$model = new Profile();
$model->load(Yii::$app->request->post());
$model->asset = UploadedFile::getInstance($model, 'asset');
$name = $model->user_id;
if($model->asset) {
$model->asset->saveAs('uploads/'.$name.'.
'.$model->asset->extension);
$model->asset = $model->asset->name.'.'.
$model->asset->extension;
}
if($model->save()) {
echo json_encode(array('status'=>"Success",
'data'=>$model->attributes),JSON_PRETTY_PRINT);
} else {
echo json_encode(array('status'=>"Failure",
'error_code'=>400,
'errors'=>$model->errors),JSON_PRETTY_PRINT);
}
}
}
When I try to use access this from Postman like:
POST http://localhost/myapp/api/v1/profiles
I get Invalid Parameter – yii\base\InvalidParamException
Response content must not be an array.
What is the issue?? help would be grateful!! Thanks
You can easily receive single / multi-uploaded files using HTTP POST with form-data encoding in Yii2, directly in your Yii2 Controller / action.
Use this code:
$uploads = UploadedFile::getInstancesByName("upfile");
if (empty($uploads)){
return "Must upload at least 1 file in upfile form-data POST";
}
// $uploads now contains 1 or more UploadedFile instances
$savedfiles = [];
foreach ($uploads as $file){
$path = //Generate your save file path here;
$file->saveAs($path); //Your uploaded file is saved, you can process it further from here
}
If you use Postman API client to test how your API is working, you can configure the upload endpoint to work like this for multi-file uploads:
Note: The upfile[] square brackets are important! Postman will happily let you select multiple files for upload in one slot, but this will not actually work. Doing it the way shown in the screenshot makes an array of files available to the Yii2 action, through the UploadedFile mechanism. This is roughly equivalent to the standard PHP $_FILES superglobal variable but with easier handling.
Single files can be uploaded with or without the [] square brackets after the key name. And of course you can name upfile whatever you like, for your convention.
You should use \yii\web\UploadedFile::getInstanceByName('asset'); instead of getInstance() checkout this Link
I've written a script that authenticates against Google's OAuth API for Latitude, using Net::OAuth. It correctly authenticates (as I can successfully fetch data out of the API). However, when I try to add an historical entry, I get a 401 Unknown authorization header response. I'm using the following code:
my $location_data = $json->encode(\%data);
$request = Net::OAuth->request("protected resource")->new(
consumer_key => $c_key,
consumer_secret => $c_secret,
token => $token,
token_secret => $token_secret,
verifier => $verifier,
request_url => 'https://www.googleapis.com/latitude/v1/location',
request_method => 'POST',
signature_method => $s_method,
timestamp => time,
nonce => &nonce(),
extra_params => {
key => $s_key
}
);
$request->sign;
$ua->default_header("Authorization", $request->to_authorization_header);
$ua->default_header("Content-Type", "application/json");
my $res = $ua->post('https://www.googleapis.com/latitude/v1/location?key=' . $s_key,
Content => $location_data);
All of the variables are used in the fetch portion of the API, so I know those are all ok. I'm not sure if I'm using the correct URL to post against, and I've tried what's in the sample above, as well as $request->to_url.
Any suggestions would be greatly appreciated.
After some back and forth with the Latitude API team, it was determined that this error comes from the fact that the Content-Type is not actually being set to application/json. Changing the above code to:
$ua->default_header("Authorization", $request->to_authorization_header);
my $res = $ua->post('https://www.googleapis.com/latitude/v1/location?key=' . $s_key,
'Content-Type' => 'application/json',
Content => $location_data);
And everything works as expected.