how to improve this perl/bash one-liner to deserialize json data - perl

I have a little bash program that calls a webservice that returns JSON data.
I have written the webservice program myself, I have complete control over its data sources, and thus I can trust the data that is returned.
Now I want to do something with the data.
The data is a simple, short key-value structure without nesting, and looks like this:
{
"asciifile" : "../tmp/data_20120720_105746-01580.txt",
"excelfile" : "../tmp/data_01580-20120720_105746.xlsx",
"from" : "Jun 19, 2012",
"msg" : "some info message, for the admin",
"outfile" : "data--recent.txt",
"outfile_excel" : "data--recent.txt.xlsx",
"resolution" : "std"
"to" : "Jul 20, 2012",
"url_comment" : "another info message, for the screen/user",
"url_outfile" : "http://www.example.com/path/tmp_cached_files/data--recent.txt",
"url_outfile_excel" : "http://www.example.com/path/tmp_cached_files/data--recent.txt.xlsx",
}
Now I am using this one-liner to deserialize the json structure returned to perl code. See last line of this snippet:
#!/bin/bash
cmd=$(curl_or_wget_call_to_webservice)
output=$(eval $cmd)
outfile_excel=$(echo "$output"| json_xs -f json -t dumper | tee | perl -n0777 -E 'eval "%h=%{$_}"; warn $# if $#; say $h{outfile_excel}')
For example, I'm not sure why I came up with the %{$_} construct. Is there a better way to do this? Is there a shorter way or a safer way to write the last line?
SE Editors: if you wish, you may move this post to the codereview stackexchange site, but I don't have an account there.
Edit: After revisiting the post after 8 months, I'd like to add that these days I use this one liner for getting the name of my github repos:
wget --quiet --auth-no-challenge --user knbknb --password secret -O -
https://api.github.com/user/repos |
perl -MJSON -n0777 -E '$r = decode_json($_); map {say $_->{name}} #$r' -

Perl can decode JSON itself, so the next should give some idea, using LWP::Simple to get some json data.
perl -MLWP::Simple -MJSON \
-e '$ref = decode_json(get("http://your.url/to/webservice")); print $ref->{outfile_excel}'
The $ref contains a perl structure of all JSON data, print out as you want it..

There is jshon. You could simply call something like
curl http://somehere.tld/data.json | jshon -e url_outfile_excel
Which would print the value for the given key.
By the way. Having control over the webservice doesn't make the input trustworthy. Be careful when calling eval.

Related

How to get objectSid value programmaticaly from Active Directory?

I'm searching for a way to get the objectSid value for an Active Directory Group programmaticaly.
I found a solution to get this value using powerShell :
Get-WMIObject win32_group -filter "name='WIKI_ADMIN'"|select sid
However what i want is to do is to get this value not from a windows machine, but from a distant Linux one.
I found a tool "adcli" that offer some fonctionnality on an AD (create user, group, ...) but no way to get the groupSid info.
Is there another tool, solution, script to get a such value ?
Thank you
Ok i found a solution using ldapsearch command line :
ldapsearch -x -LLL -h ldap_url -D 'cn=user name,OU=users,OU=myOU,DC=myou,DC=com' -w my_password -b 'CN=the_group_name,OU=users,OU=myOU,DC=myou,DC=com' -s sub '(objectClass=group)' | grep objectSid
And this gave me something like that :
objectSid:: AQUsAAAAAAUVAAAA5BLIGcudyeZ/J0pqbvoAxA==
However the value is in binary and base64 encoded, i found a script in another forum that decoded it successfully :
!/bin/bash
# Base-64 encoded objectSid
OBJECT_ID="AQUAAAAAAAUVAAAAPWW1S5rojK4mDAiG5BAAAA=="
OBJECT_ID=$1
# Decode it, hex-dump it and store it in an array
G=($(echo -n $OBJECT_ID | base64 -d -i | hexdump -v -e '1/1 " %02X"'))
# SID in HEX
# SID_HEX=${G[0]}-${G[1]}-${G[2]}${G[3]}${G[4]}${G[5]}${G[6]}${G[7]}-${G[8]}${G[9]}${G[10]}${G[11]}-${G[12]}${G[13]}${G[14]}${G[15]}-${G[16]}${G[17]}${G[18]}${G[19]}-${G[20]}${G[21]}${G[22]}${G[23]}-${G[24]}${G[25]}${G[26]}${G[27]}${G[28]}
# SID Structure: https://technet.microsoft.com/en-us/library/cc962011.aspx
# LESA = Little Endian Sub Authority
# BESA = Big Endian Sub Authority
# LERID = Little Endian Relative ID
# BERID = Big Endian Relative ID
BESA2=${G[8]}${G[9]}${G[10]}${G[11]}
BESA3=${G[12]}${G[13]}${G[14]}${G[15]}
BESA4=${G[16]}${G[17]}${G[18]}${G[19]}
BESA5=${G[20]}${G[21]}${G[22]}${G[23]}
BERID=${G[24]}${G[25]}${G[26]}${G[27]}${G[28]}
LESA1=${G[2]}${G[3]}${G[4]}${G[5]}${G[6]}${G[7]}
LESA2=${BESA2:6:2}${BESA2:4:2}${BESA2:2:2}${BESA2:0:2}
LESA3=${BESA3:6:2}${BESA3:4:2}${BESA3:2:2}${BESA3:0:2}
LESA4=${BESA4:6:2}${BESA4:4:2}${BESA4:2:2}${BESA4:0:2}
LESA5=${BESA5:6:2}${BESA5:4:2}${BESA5:2:2}${BESA5:0:2}
LERID=${BERID:6:2}${BERID:4:2}${BERID:2:2}${BERID:0:2}
LE_SID_HEX=${LESA1}-${LESA2}-${LESA3}-${LESA4}-${LESA5}-${LERID}
# Initial SID value which is used to construct actual SID
SID="S-1"
# Convert LE_SID_HEX to decimal values and append it to SID as a string
IFS='-' read -ra ADDR <<< "${LE_SID_HEX}"
for OBJECT in "${ADDR[#]}"; do
SID=${SID}-$((16#${OBJECT}))
done
echo ${SID}
and i got this :
S-1-48378511622149-21-432542436-3871972811-1783244671-3288398446
:)

How format mac address inside json array

Need help in getting below the mac address inside the json file to re-reformatted using sed
cat 251.json
cat /tmp/251.json
[
"08:f1:ea:6d:03:3c",
"08:f1:ea:6d:03:3d",
"08:f1:ea:6d:03:3e",
"08:f1:ea:6d:03:3f",
"b8:83:03:81:4b:20",
"b8:83:03:81:4b:21",
"b8:83:03:84:d5:1c",
"b8:83:03:84:d5:1d"
]
The expected format is
[
"08f1.ea6d.033c",
"08f1.ea6d.033d",
"08f1.ea6d.033e",
"08f1.ea6d.033f",
"b883.0381.4b20",
"b883.0381.4b21",
"b883.0384.d51c",
"b883.0384.d51d"
]
This should work:
sed -E 's/:(.{2}):(.{2}):(.{2}):(.{2}):/\1.\2\3.\4/g' /tmp/251.json
In this way, you get the output to stdout. If you want to modify the file, add the -i option. You can check the result here.

Retrieving Facebook reports in R

Note: this is an updated post
In Facebook I have scheduled report reports to be sent to my e-mail.
Unfortunately reports are not coming as attachment but rather as a link e.g. below
https://www.facebook.com/ads/manage/download_report.php?act=1119884838046520&report_run_id=6086936413420&format=csv&source=email_v2
Please help me to construct the URI path to retrieve the report.
Current state
I am able to get properties of the report:
RCurl::getURL("https://graph.facebook.com/v2.9/6087139848020?access_token=EAAX****")
results into
$ id : chr "6087139848020"
$ account_id : chr "1119884838046520"
$ schedule_id : chr "6085730147820"
$ time_ref : num 1.49e+09
$ time_completed : num 1.49e+09
$ async_status : chr "Job Completed"
$ async_percent_completion: num 100
$ friendly_name : chr "*****"
$ date_start : chr "2017-05-03"
$ date_stop : chr "2017-05-03"
So everything seems OK
Problem
However by trying to access the /report_ID/insights as suggested FB API reference/ad-report-run/:
RCurl::getURL("https://graph.facebook.com/v2.9/6087139848020/insights?access_token=E***")
I get an error message:
(#100) Tried accessing nonexisting summary field (results)
Any ideas how to get the body of Facebook report distributed by e-mail?
Help will be strongly appreciated.
PS Special thanks to user #phwd for suggestions
The solution is
curl -G \
-d 'report_run_id=6087450841820' \
-d 'format=csv' \
'https://www.facebook.com/ads/ads_insights/export_report/'
Source of info: FB docs for asynchronous reports

jboss-cli : How do I read one specific system property using jboss-cli?

I'm new to jboss-cli and working through the 'jboss-cli recipes'.
Question
How do I read one specific property using jboss-cli? E.g.
jboss.home.dir (e.g. "-Djboss.home.dir=/path/to/my/jboss")
Xmx ("-Xmx=4G")
Context
The "CLI Recipes" documentation has this helpful example to get all system properties. However its 'too much infomration'. I want to script reading one specific property.
https://docs.jboss.org/author/display/WFLY10/CLI+Recipes#CLIRecipes-
Overview of all system properties in JBoss AS7+ including OS system
properties and properties specified on command line using -D, -P or
--properties arguments.
Standalone
[standalone#IP_ADDRESS:9999 /] /core-service=platform-mbean/type=runtime:read-attribute(name=system-properties)
Thanks in advance
You could do a :
:resolve-expression(expression=${jboss.home.dir})
You can use the cli like this:
$JBOSS_HOME/bin/jboss-cli.sh -c --command=/system-property=MY_PROPERTY:read-resource
you get an output like this:
$JBOSS_HOME/bin/jboss-cli.sh -c --command=/system-property=MY_PROPERTY:read-resource
{
"outcome" => "success",
"result" => {"value" => "4.0"}
}
which you can extract by piping into something like this:
<cli command> | grep "{\"value\"" | sed "s/.*value\" => \"\([^\"]*\)\".*/\1/"
its a bit ugly, and there are some nasty edge cases if the values were to be something like "value" => "value =" or something hideous.
In general this works OK.
Change the sed command to be a bit more specific to fix that.
This link pointed me to the answer: I can use a groovy script to get the values. From what I see the "jboss-cli command line" does not offer this flexibility.
https://developer.jboss.org/wiki/AdvancedCLIScriptingWithGroovyRhinoJythonEtc
Solution
Here's a solution for jboss home.
[For memory you can get results from "/core-service=platform-mbean/type=memory/:read-attribute(name=heap-memory-usage)"
bash
#!/bin/sh
# Note: must set jbbin to 'jboss home /bin'
groovy -cp $jbbin/client/jboss-cli-client.jar readJbossHome.groovy
Groovy
Note: this is 'quick and dirty'.
import org.jboss.as.cli.scriptsupport.*
cli = CLI.newInstance()
cli.connect()
// Define properties
myParentProp="system-properties"
myProp="jboss.home.dir"
// Retrieve and pluck values
result = cli.cmd("/core-service=platform-mbean/type=runtime:read-resource(recursive=true,include-runtime=false)")
myResult = result.getResponse().get("result")
myParentVal = myResult.get(myParentProp)
myVal = myParentVal.get(myProp)
// Print out results
println "Property detail ${myProp} is ${myVal}"
cli.disconnect()
You can also do it via Wildfly management rest call.
http://localhost:9990/management
POST
Headers = Content-Type:application/json
Body =
{
"operation":"resolve-expression",
"expression":"${jboss.home.dir}"
}
With newer Teiid DOCs I have found some useful information I thought this might be helpful to share to people coming across a similar usecase
https://access.redhat.com/documentation/en-us/jboss_enterprise_application_platform/6.3/html/administration_and_configuration_guide/configure_system_properties_using_the_management_cli
Helps Adding, Removing & Reading System Properties with jboss-cli
jboss-cli
If you have a cli command like ehsavoie suggested :resolve-expression(expression=${jboss.home.dir}) and want to use the content of the "result" property within jboss-cli you can save it in a variable. You can use backticks (`) to evaluate expressions.
simple expression
[standalone#localhost:9990 /] :resolve-expression(expression=${jboss.home.dir})
{
"outcome" => "success",
"result" => "/home/user/wildfly"
}
use in valiable
[standalone#localhost:9990 /] set wildflydirectory=`:resolve-expression(expression=${jboss.home.dir})`
[standalone#localhost:9990 /] echo $wildflydirectory
/home/user/wildfly
PowerShell
If you happen to use the PowerShell you can use a one-liner to extract even deeply nested results with the help of the cli's --output-json option and PowerShell's ConvertFrom-Json cmdlet. In this way the parsing problem from James Roberts's approach with grep and sed are gone.
$value=(Invoke-Expression "./jboss-cli.ps1 -c --command=':resolve-expression(expression=`${jboss.home.dir})' --output-json" | ConvertFrom-Json).result
It is a bit tricky to quote the command and escape the correct PowerShell special characters.

Why does my REST request return garbage data?

I am trying to use LWP::Simple to make a GET request to a REST service. Here's the simple code:
use LWP::Simple;
$uri = "http://api.stackoverflow.com/0.8/questions/tagged/php";
$jsonresponse= get $uri;
print $jsonresponse;
On my local machine, running Ubuntu 10.4, and Perl version 5.10.1:
farhan#farhan-lnx:~$ perl --version
This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi
I can get the correct response and have it printed on the screen. E.g.:
farhan#farhan-lnx:~$ head -10 output.txt
{
"total": 1000,
"page": 1,
"pagesize": 30,
"questions": [
{
"tags": [
"php",
"arrays",
"coding-style"
(... snipped ...)
But on my host's machine to which I SSH into, I get garbage printed on the screen for the same exact code. I am assuming it has something to do with the encoding, but the REST service does not return the character set type in the response, so how do I force LWP::Simple to use the correct encoding? Any ideas what may be going on here?
Here's the version of Perl on my host's machine:
[dredd]$ perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi
I happen to have a 64 bit RHEL 5.4 box which has Perl 5.8.8 on it. I took your code and got the exact same result. I tried using Data::Dumper to dump the data, but that didn't change anything. I then went to the command line and did this:
wget -O jsonfile http://api.stackoverflow.com/0.8/questions/tagged/php
--2010-05-26 11:42:41-- http://api.stackoverflow.com/0.8/questions/tagged/php
Resolving api.stackoverflow.com... 69.59.196.211
Connecting to api.stackoverflow.com|69.59.196.211|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5430 (5.3K) [application/json]
Saving to: `jsonfile'
2010-05-26 11:42:42 (56.9 KB/s) - `jsonfile' saved [5430/5430]
When I did this:
file jsonfile
I got:
jsonfile: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT), max speed
So, the JSON data was gzipped by the web server. I tried this:
gzip -dc jsonfile
and lo and behold the results are the JSON data as you would expect.
What you can do now is to either use another module to ungzip the data, or you can check out this other thread which shows how to accept gzip using LWP::UserAgent and handle the request that way
This is bug 44435. Upgrade libwww-perl to version 5.827 or better.