Im trying to update json array. If i have JSON like this one:
{
"value":[
{
"name":{
"first":"Bob",
"last":"Pegelano"
},
"age":31,
"email":"bob#gmail.com"
},
{
"name":{
"first":"Majkl",
"last":"Skot"
},
"age":321,
"email":"gecko#gmail.com"
}]
}
I can easily update an array like this.
val jsarrayUpdate = (__ \ 'value).json.update(
__.read[JsArray].map{ o => o :+ Json.obj( "field243" -> "coucou" ) }
)
myJson.transform(jsarrayUpdate)
But I have simple array JSON without any key:
[{
"name":{
"first":"Bob",
"last":"Pegelano"
},
"age":31,
"email":"bob#gmail.com"
},
{
"name":{
"first":"Majkl",
"last":"Skot"
},
"age":321,
"email":"gecko#gmail.com"
}]
And was hoping to be able to edit it with this command:
val jsarrayUpdate2 = __.json.update(
__.read[JsArray].map{ o => o :+ Json.obj( "field243" -> "coucou" ) }
)
This is not working nor is anything else I tried in the past two hours. What am I doing wrong?
Thank you.
What about
jsArray.as[List[JsObject]].map {i => i ++ Json.obj( "field243" -> "coucou")}
This will give you a List[JsObject]. If you need you can convert it back to JsArray with
Json.toJson(listOfJsObjects).as[JsArray]
Related
I am just a newbie to the ELK and trying some testing on this, i'm able to run some tests but while i'm trying a filter with grok & mutate to remoev & replace some feilds from my syslog output i'm getting into below error..
21:58:47.976 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, {, ,, ] at line 21, column 9 (byte 496) after filter {\n if [type] == \"syslog\" {\n grok {\n match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\\[%{POSINT:pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n }\n date {\n match => [ \"syslog_timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n }\n mutate {\n remove_field => [\n \"message\",\n \"pid\",\n \"port\"\n "}
Below is my config file ....
# cat logstash-syslog2.conf
input {
file {
path => [ "/scratch/rsyslog/*/messages.log" ]
type => "syslog"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
mutate {
remove_field => [
"message",
"pid",
"port"
"_grokparsefailure"
]
}
mutate {
replace => [
"#source_host", "%{allLogs_hostname}"
"#message", "%{allLogs_message}"
]
}
mutate {
remove => [
"allLogs_hostname",
"syslog_message",
"syslog_timestamp"
]
}
}
output {
if [type] == "syslog" {
elasticsearch {
hosts => "localhost:9200"
index => "%{type}-%{+YYYY.MM.dd}"
}
}
}
please suggest what i'm doing wrong and help to understand the remove & replace functions for the lagstash..
PS: my ELK version is 5.4
The Config you posted have lot of syntactical errors , the logsatsh has it's own config language and expects the config file to abide by the rule.
This link has complete logstash config language reference.
I made some corrections to your config file and posted here , Have added my comments and explanation of what was wrong in the config file itself
input
{
file
{
path => [ "/scratch/rsyslog/*/messages.log" ]
type => "syslog"
}
}
filter
{
if [type] == "syslog"
{
grok
{
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:syslog_message}" }
}
date
{
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
# Have merged it with the remove_field option below
#mutate {
# remove_field => [
# "message",
# "pid",
# "port",
# "_grokparsefailure"
# ]
#}
mutate
{
# The replace option only accept hash data type which has a syntax as below
# For more details visit the below link
# https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-replace
replace => {
"#source_host" => "%{allLogs_hostname}"
"#message" => "%{allLogs_message}"
}
}
mutate
{
# Mutate does not have remove option i guess your intention is to remove the event field
# hence used remove_field option here
# The remove_filed option only accepts arary as value type as shown below
# For details read the below link
# https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field
remove_field => [
"message",
"pid",
"port",
"_grokparsefailure",
"allLogs_hostname",
"syslog_message",
"syslog_timestamp"
]
}
}
}
output
{
if [type] == "syslog"
{
elasticsearch
{
# The Hosts option only takes uri as a value type , originally you have provided string as it's value type
# For more info please read the below link
#https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts
hosts => ["localhost:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}
}
You can test whether the config file is syntactically correct by using logstash command line option -t this option will test and report the config file is syntactically correct
bin\logstash -f 'path-to-your-config-file' -t
Please let me know for any clarification
You have to add a comma after "port" in your logstash configuration file.
mutate {
remove_field => [
"message",
"pid",
"port",
"_grokparsefailure"
]
}
I have a search form for searching "documents", that have a small dozen of search criterions, including "entire_text", "keywords" and "description".
I'm using pg_search_scope, but I have 2 different scopes.
This is in my document.rb:
pg_search_scope :search_entire_text,
:against => :entire_text,
:using => {
:tsearch => {
:prefix => true,
:dictionary => "french"
}
}
pg_search_scope :search_keywords,
:associated_against => {
:keywords => [:keyword]
},
:using => {
:tsearch => {
:any_word => true
}
}
Each separately works fine. But I can't do this:
#resultats = Document.search_keywords(params[:ch_document][:keywords]).search_entire_text(params[:ch_document][:entire_text])
Is there any way to work around this?
Thanks
I've never used pg_search_scope but it looks like you indeed can't combine two pg_search_scope's.
What you could do is use :search_entire_text with a pg_search_scope and use the resulting id's in a Document.where([1,2,3]) that way you can use standard rails scope's for the remaining keyword searches.
Example:
# If pluck doesn't work you can also use map(&:id)
txt_res_ids = Document.search_entire_text(params[:ch_document][:entire_text]).pluck(:id)
final_results = Document.where(id: txt_res_ids).some_keyword_scope.all
It works. Here's the entire code ... if ever this could help someone :
Acte.rb (I didn't translate to english, the explanations are commented to correspond to the question above)
pg_search_scope :cherche_texte_complet, #i.e. find entire text
:against => :texte_complet,
:using => {
:tsearch => {
:prefix => true,
:dictionary => "french"
}
}
pg_search_scope :cherche_mots_clefs, #find keywords
:associated_against => {
:motclefs => [:motcle]
},
:using => {
:tsearch => {
:any_word => true
}
}
def self.cherche_date(debut, fin) #find date between
where("acte_date BETWEEN :date_debut AND :date_fin", {date_debut: debut, date_fin: fin})
end
def self.cherche_mots(mots)
if mots.present? #the if ... else is necessary, see controller.rb
cherche_mots_clefs(mots)
else
order("id DESC")
end
end
def self.ids_texte_compl(ids)
if ids.any?
where("id = any (array #{ids})")
else
where("id IS NOT NULL")
end
end
and actes_controller.rb
ids = Acte.cherche_texte_complet(params[:ch_acte][:texte_complet]).pluck(:id)
#resultats = Acte.cherche_date(params[:ch_acte][:date_debut],params[:ch_acte][:date_fin])
.ids_texte_compl(ids)
.cherche_mots(params[:ch_acte][:mots])
Thanks !
chaining works in pg_search 2.3.2 at least
SomeModel.pg_search_based_scope("abc").pg_search_based_scope("xyz")
I am not sure if I am using the correct terminology here but I am trying to loop through an array and create a hash of arrays and values.
Currently my code looks like this:
my $endResult;
my $list = $arrayRef;
my $hash;
foreach my $hash_ref ( #$list ) {
if ( substr($hash_ref->{ID_NUMBER}, 0, 3) eq 'ABC' ) {
$hash->{'ABC'}{$hash_ref->{ID_NUMBER}}->{'VEHICLES'} = $arrayRef1;
push(#$endResult, $hash);
}
... #more ID number if statementss with different id numbers
and I get an output like this:
[
{
ABC => {
ABC1234 => {
VEHICLES => [
{ X => 11, Y => 0, Z => 12 },
{ X => 2001, Y => 100000, Z => 300 },
],
},
ABC56778 => {
VEHICLES => [
{ X => 1324, Y => 0, Z => 234 },
{ X => 666, Y => 7777, Z => 555 },
],
},
...
But what I want is for ABC to point to array of hashes (ABC1234, ABC46778) instead of how it is. Let me know if I need to clarify better but I cant figure out the right syntax to make this happen when building my hash.
To get your long keys into individual array refs, you need to change the assignment of your data structure. I've added a bunch of example data.
# input data
my $vehicles = [
{
'Z' => '12',
'X' => '11',
'Y' => '0',
},
{
'Z' => '300',
'X' => '2001',
'Y' => '100000',
}
];
my $list = [ { ID_NUMBER => 'ABC1234' }, { ID_NUMBER => 'ABC56778' } ];
# output data
my $endResult;
foreach my $hash_ref (#$list) {
my $hash; # needs to be inside of the loop!
if ( substr( $hash_ref->{ID_NUMBER}, 0, 3 ) eq 'ABC' ) {
push #{ $hash->{'ABC'} }, { # this becomes the array
$hash_ref->{ID_NUMBER} => { # and everything below needs
VEHICLES => $vehicles, # to be constructed directly
}
};
push #$endResult, $hash;
}
}
This will yield the following data structure (output with Data::Printer):
\ [
[0] {
ABC [
[0] {
ABC1234 {
VEHICLES [
[0] {
X 11,
Y 0,
Z 12
},
[1] {
X 2001,
Y 100000,
Z 300
}
]
}
}
]
},
[1] {
ABC [
[0] {
ABC56778 {
VEHICLES var[0]{ABC}[0]{ABC1234}{VEHICLES}
}
}
]
}
]
Please note that your choice of variable names makes this very complicated. It's hard to read and will be hell to maintain later. Always pick variable names based on what they represent. Name them something like $vehicle, $ids or $stuff_we_need_to_take_care_off_later, but not $hash1).
1: Unless you work with a hashing algorithm and we're talking about the result. :)
I have a model. It contains a list of another model:
case class Account(
_id: ObjectId = new ObjectId,
name: String,
campaigns: List[Campaign]
)
case class Campaign(
_id: ObjectId = new ObjectId,
name: String
)
I have a form and action for display and creating new Accounts:
val accountForm = Form(
mapping(
"id" -> ignored(new ObjectId),
"name" -> nonEmptyText,
"campaigns" -> list(
mapping(
"id" -> ignored(new ObjectId),
"name" -> nonEmptyText
)(Campaign.apply)(Campaign.unapply)
)
)(Account.apply)(Account.unapply)
)
def accounts = Action {
Ok(views.html.accounts(AccountObject.all(), accountForm, CampaignObject.all()))
}
def newAccount = Action {
implicit request =>
accountForm.bindFromRequest.fold(
formWithErrors => BadRequest(views.html.accounts(AccountObject.all(), formWithErrors, CampaignObject.all())),
account => {
AccountObject.create(account)
Redirect(routes.AccountController.accounts)
}
)
}
Finally, here is my view for Accounts:
#(accounts: List[models.mongodb.Account], account_form: Form[models.mongodb.Account], campaign_list: List[models.mongodb.Campaign])
#import helper._
#args(args: (Symbol, Any)*) = #{
args
}
#main("Account List") {
<h1>#accounts.size Account(s)</h1>
<ul>
#accounts.map { account =>
<li>
#account.name
</li>
}
</ul>
<h2>Add a New Account</h2>
#form(routes.AccountController.newAccount()) {
<fieldset>
#inputText(account_form("name"), '_label -> "Account Name")
#select(
account_form("campaigns"),
options(campaign_list.map(x => x.name):List[String]),
args(
'class -> "chosen-select",
'multiple -> "multiple",
Symbol("data-placeholder") -> "Add campaigns",
'style -> "width:350px;"
): _*
)
<input type="submit" value="Create">
</fieldset>
}
}
The problem is when I submit this form, it submits it with a list of strings for the campaigns field. This gives me a 400 error when I post the form submission.
I would like to either submit submit the form with a list of campaigns instead of strings or have the form submit with a list of strings, then process the strings into a list of campaigns in my controller. Which way would be better and how would I do it? Thanks!
I ended up making a temp form to hold the strings of the nested model. Then I converted those strings to model objects in my controller:
val accountTempForm = Form(
tuple(
"id" -> ignored(new ObjectId),
"name" -> nonEmptyText,
"campaigns" -> list(text)
)
)
def newAccount = Action {
implicit request =>
accountTempForm.bindFromRequest.fold(
formWithErrors => {
println("error")
Redirect(routes.AccountController.accounts)
//BadRequest(views.html.account.accounts(AccountObject.all(), formWithErrors, CampaignObject.all()))
},
account => {
val campaign_string_list = account._3
val campaign_list = campaign_string_list.map(x => CampaignObject.getOrCreateByName(x))
val new_account = Account.apply(account._1, account._2, campaign_list)
AccountObject.create(new_account)
Redirect(routes.AccountController.accounts)
}
)
}
I have two Array of Hashes: The first contains values for a current time interval and the second contains values for a previous time interval.
#AoHcurrent=
( { node => "ABC",
link => "DEF",
time => "10:00",
value => "100",
},
{
node => "FGH",
link => "IJK",
time => "10:00",
value => "200",
},
);
#AoHprevious=
( { node => "ABC",
link => "DEF",
time => "09:45",
value => "10",
},
{ node => "FGH",
link => "IJK",
time => "09:45",
value => "50",
},
);
I want to now use HTML-Template to present this data. Something like :
NODE LINK VALUE
---------------------
ABC DEF 100(10)
FGH IJK 200 (50)
the values in brackets represent the previous value.
my %html_template_parameters =
( AOHCURRENT => \#AoHcurrent,
AOHPREVIOUS => \#AoHprevious, );
my $html_template=qq{Report.tmpl};
my $html_output=qq{Report.html};
htmlReport($html_template,$html_output,\%html_template_parameters);
where htmlReport is a function that generates the report
I require guidance on defining the Report.tmpl file.
Thanks you in advance
see also http://www.perlmonks.org/?node_id=972954
I gave an example there how this can be solved with HTML::Template::Compiled.
Basically you would navigate through the parameter stash like this:
[%= expr=".AOHPREVIOUS[__index__]{'value'}" %]
or with the classic syntax:
<TMPL_VAR expr=".AOHPREVIOUS[__index__]{'value'}" >
You can't do that with 2 separate lists just with HTML::Template. And trying to do it with HTML::Template::Expr would be a nightmare to maintain. Try collapsing them into a single list where the hash data is merged.