What is this `job_desc_msg_t` format that I need to submit jobs to SLURM via the Perl API? - perl

The Perl API for SLURM indicates that to submit a job with the API requires that we give it a "job description" ($job_desc or $job_desc_msg), which has the structure job_desc_msg_t but it doesn't tell what job_desc_msg_t is.
update: I found it in slurm.h starting at line 1162, so I'm guessing that I will need to pass in a hash with a similar structure.

That's exactly what you must do according to the man page.
Typicaly, C structures are converted to (maybe blessed) Perl hash
references, with field names as hash keys. Arrays in C are converted to
arrays in Perl. For example, there is a structure "job_info_msg_t":
typedef struct job_info_msg {
time_t last_update; /* time of latest info */
uint32_t record_count; /* number of records */
job_info_t *job_array; /* the job records */
} job_info_msg_t;
This will be converted to a hash reference with the following
structure:
{
last_update => 1285847672,
job_array => [ {account => 'test', alloc_node => 'ln0', alloc_sid => 1234, ...},
{account => 'debug', alloc_node => 'ln2', alloc_sid => 5678, ...},
...
]
}
Note the missing of the "record_count" field in the hash. It can be
derived from the number of elements in array "job_array".
To pass parameters to the API functions, use the corresponding hash
references, for example:
$rc = $slurm->update_node({node_names => 'node[0-7]', node_state => NODE_STATE_DRAIN});
Please see "<slurm/slurm.h>" for the definition of the structures.

Related

How to filter an array of object in Mui DataGrid?

I recently changed my tables to Mui-datagrid on Material UI 5, and I have a special use case with an array of objects. I want to enable the phone number filter in this column, but the number is provided as an object list.
phone: [
{ type: "home", number: "795-946-1806" },
{ type: "mobile", number: "850-781-8104" }
]
I was expecting a 'customFilterAndSearch' or an option to customise how to search in this specific field.
customFilterAndSearch: (term, rowData) =>
!!rowData?.suppressedOptions.find(({ description }) =>
description?.toLowerCase().includes(term.toLowerCase())
),
I have made some tries with the filterOperators, but no success yet. I have made a full example here https://codesandbox.io/s/mui-data-grid-vs05fr?file=/demo.js
As far as I can see from the DataGrid documentation I don't see any way to change the filter function for a specific function.
Likely the best workaround for your use case will be converting this to a string be converting the data to a string before you pass it to the datagrid. Though you will lose the styling that you currently do by making the phone type bold.
On second though your best best would probably be to split the phone column into two columns which would probably be the cleanest way of solving your problem
Add helper function.
You could potentially add a helper function to just map all the phone lists to something like mobilePhone or homePhone
const mapPhoneObject = (rows) => {
rows.forEach((row) => {
row.phone.forEach((phone) => {
row[`${phone.type}Phone`] = phone.number;
});
});
return rows
};
I've added a fork of your snippet with my function, it is I think the most viable solution for your problem: https://codesandbox.io/s/mui-data-grid-forked-ppii8y

Two outputs in logstash. One for certain aggregations only

I'm trying to specify a second output of logstash in order to save certain aggregated data only. No clue how to achieve it at the moment. Documentation doesn't cover such a case.
At the moment I use a single input and a single output.
Input definition (logstash-udp.conf):
input {
udp {
port => 25000
codec => json
buffer_size => 5000
workers => 2
}
}
filter {
grok {
match => [ "message", "API call happened" ]
}
aggregate {
task_id => "%{example_task}"
code => "
map['api_calls'] ||= 0
map['api_calls'] += 1
map['message'] ||= event.get('message')
event.cancel()
"
timeout => 60
push_previous_map_as_event => true
timeout_code => "event.set('aggregated_calls', event.get('api_calls') > 0)"
timeout_tags => ['_aggregation']
}
}
Output definition (logstash-output.conf):
output {
elasticsearch {
hosts => ["localhost"]
manage_template => false
index => "%{[#metadata][udp]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
What I want to achieve now? I need to add a second, different aggregation (different data and conditions) which will save all the not aggregated data to Elasticsearch like now however aggregated data for this aggregation would be saved to Postgres. I'm pretty much stuck at the moment and searching the web for some docs/examples doesn't help.
I'd suggest using multiple pipelines: https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
This way you can have one pipeline for aggregation and second one for pure data.

Filter data on call to getHyperCubeData

When I run the following, I got all records from my table object (assuming i have 100 records in all). Is there a way to send the selection/filter, for example, I want to retrieve only those where department='procuring'.
table.getHyperCubeData('/qHyperCubeDef', [{
qWidth: 8,
qHeight: 100
}]).then(data => console.log(data));
I figured out the answer. Before getting the hypercube data, I need to get the field from the Doc class, then do the following:
.then(doc => doc.getField('department'))
.then(field => field.clear().then(() => field.select({qMatch: filter['procuring']})))

Assigning array/hash to data structure with arrow operator. Perl

Just asking because i can't find the answer I'm looking for...
What is the difference between these lines, I'm under the assumption that there isn't a difference.
I'm just unsure if it $stash->{hashdata} automatically becomes a reference.
my %data = { thing => 1, otherthing => 2 };
$stash->{hashdata} = \%data;
$stash->{hashdata} = { thing => 1, otherthing => 2 };
{ ... } is the syntax for a hash reference (similarly, [ ... ] is for array references).
When you assign something to a hash, it is interpreted as a list of alternating keys/values. If the list has an odd number of elements (such as 1), you get this warning:
Odd number of elements in hash assignment at ...
... unless it's only a single value that is a reference, in which case you get:
Reference found where even-sized list expected at ...
In any case, the last element is interpreted as a key with a corresponding value of undef.
Thus, if you try to assign a reference to a hash:
my %data = { ... };
A warning is emitted and the code behaves as if you had written:
my %data = ({ ... } => undef);
Hash keys are always strings, so the reference is implicitly stringified, yielding something like "HASH(0xdeadbeef)":
my %data = ('HASH(0xdeadbeef)' => undef);
This is never what you want.
The equivalent of
$stash->{hashdata} = { thing => 1, otherthing => 2 };
with a named hash would look like:
my %data = ( thing => 1, otherthing => 2 );
$stash->{hashdata} = \%data;
Note: There is no reference in the first line. We're assigning a plain list to %data.
In fact, you can think of { LIST } as syntactic sugar for:
do { my %tmp = LIST; \%tmp }
The block limits the scope of %tmp to this location in the code; the do keyword turns the block into an expression that returns the result of the last statement in the block.
This is an error.
my %data = { thing => 1, otherthing => 2 };
It should be.
my %data = ( thing => 1, otherthing => 2 );
What is the difference between these two?
$stash->{hashdata} = \%data;
$stash->{hashdata} = { thing => 1, otherthing => 2 };
The first one means any changes to %data will also happen in $stash->{hashdata} and vice versa, because it has a reference to %data.
The second means %data and $stash->{hashdata} are independent. Changes in one will not happen in the other.
Yes, there is a big difference between the two.

How do I work globally on a complex data structure using subroutines?

Specifically my data structure looks like this
{
"SomeGuy" => {
date_and_time => "11-04-2013",
Id => 7,
nr => 52,
picks => [
{ "This is an option" => "Option3" },
{ "This is another option" => "Option4" },
{ "This is another option" => "Option1" },
{ "And another one" => "Something" },
],
time_of => "06:11 AM",
total => 1,
},
"Another Guy" => { ... },
}
This is output via Data::Dump. The actual data structure contains a lot more records like "SomeGuy". The structure is identical for all of them.
I populate this data structure this way:
$guys->{$profile}{options}[$total++]{$var} = $var2;
$guys->{$profile}{Id} = $i;
$guys->{$profile}{date_and_time} = get_date($Time[0]);
$guys->{$profile}{time_of} = $Time[1];
$guys->{$profile}{total} = keys (% {$guys->{$profile}{options}[0]});
$guys->{$profile}{nr} = $pNr;
Having all this, what I want to do next is operate on this data structure. I repeat that there many many records in the data structure.
When I output the contents of it I get it in jumbled order, as to say, not in the order that it was populated. I've tried this with Data::Dumper, Data::Dump and iterating through the records myself by hand.
I know that the methods in the Data namespace are notorious for this, that's why Data::Dumper provides a way to sort through a subroutine and Data::Dump provides a default one.
So I have the data structure. It looks like I expect it to be, I have all the data as I knew it should look in it, valid. I want to sort the records according to their Id field. My thinking is that I have to use a subroutine and essentially pass a reference of the data structure to it and do the sorting there.
sub sortt {
my $dref = shift #_;
foreach my $name ( sort { $dref->{$a}{Id} <=> $dref->{$b}{Id} } keys %$dref ) {
print "$data->{$name}{Id}: $name \n";
}
}
This gets called with (in the same scope where the structure is populated, so no worries there):
sortt(\$guys);
The error is:
Not a HASH reference at perlprogram.pl line 452
So I go and use ref in the subroutine to make sure I'm passing an actual reference. And it says REF.
Next I go into desperate mode and try some stupid things like calling it with:
sortt(\%$guys)
But if I'm not mistaking this just sends a copy to the subroutine and just sorts that copy locally, so no use there.
It's no use if I make a copy and return it from the subroutine, I just want to pass a reference of my data structure and sort it and have it reflect those changes globally (or in the calling scope per se). How would I do this?
Leaving aside your syntax problem with "Not a ref", you are approaching the problem from the wrong end in the first place. I'll leave small syntactic details to others (see Ikegami's comment).
You can NOT sort them at all, because $guys is a hash, not an array. Hashes are NOT ever sorted in Perl. If you want to sort it, your have three solutions:
Store an ordered list of names as a separate array.
my #ordered_names = sort { $guys->{$a}{Id} <=> $guys->{$b}{Id} } keys %$guys
Then you use the array for ordering and go back to hashref for individual records.
Add the name to the individual guy's hash, instead of outer hash reference. $guys should be an array reference. The downside to this approach is that you can't look up a person's record by their name any more - if that functionality is needed, use #1.
#codnodder's answer shows how to do that, if you don't care about accessing records by name.
Use a Tie::* module. (Tie::IxHash, Tie::Hash::Sorted). NOT recommended since it's slower.
Perl hashes are inherently unordered. There is no way you can sort them, or reorder them at all. You have to write code to to access the elements in the order you want.
Your sortt subroutine does nothing but print the ID and the name of each hash element, sorted by the ID. Except that it doesn't, because you are trying to use the variable $data when you have actually set up $dref. That is likely to be the cause of your Not a HASH reference error, although unless you show your entire code, or at least indicate which is perlprogram.pl line 452, then we cannot help further.
The best way to do what you want is to create an array of hash keys, which you can sort in whatever order you want. Like this
my #names_by_id = sort { $guys->{$a}{Id} <=> $guys->{$b}{Id} } keys %$guys;
Then you can use this to access the hash in sorted order, like this, which prints the same output as your sortt is intended to, but formatted a little more nicely
for my $name (#names_by_id) {
printf "%4d: %s\n", $guys->{$name}{Id}, $name;
}
If you want to do anything else with the hash elements in sorted order then you have to use this technique.
$guys is already a hash ref, so you just need sortt($guys)
If you want a sorted data structure, you need something like this:
my #guys_sorted =
map { { $_ => $guys->{$_} } }
sort { $guys->{$a}{Id} <=> $guys->{$b}{Id} } keys %$guys;
print(Dumper(\#guys_sorted));
Or, in a sub:
sub sortt {
# returns a SORTED ARRAY of HASHREFS
my $ref = shift;
return map { { $_ => $ref->{$_} } }
sort { $ref->{$a}{Id} <=> $ref->{$b}{Id} } keys %$ref;
}
print(Dumper([sortt($guys)]));
That's a pretty complex data structure. If you commonly use structures like this in your program, I suggest you take your Perl programming skills up a notch, and look into learning a bit of Object Oriented Perl.
Object Oriented Perl is fairly straight forward: Your object is merely that data structure you've previously created. Methods are merely subroutines that work with that object. Most methods are getter/setter methods to set up your structure.
Initially, it's a bit more writing, but once you get the hang of it, the extra writing is easily compensated by the saving is debugging and maintaining your code.
Object Oriented Perl does two things: It first makes sure that your structure is correct. For example:
$some_guy->{Picks}->[2]->{"this is an option"} = "Foo!";
Whoops! That should have been {picks}. Imagine trying to find that error in your code.
In OO-Perl, if I mistyped a method's name, the program will immediately pick it up:
$some_guy->picks(
{
"This is an option" -> "Foo!",
"This is option 2" => "Bar!",
}
)
If I had $some_guy->Picks, I would have gotten a runtime error.
It also makes you think of your structure as an object. For example, what are you sorting on? You're sorting on your Guys' IDs, and the guys are stored in a hash called %guys.
# $a and $b are hash keys from `%guys`.
# $guys{$a} and $guys{$b} represent the guy objects.
# I can use the id method to get they guys' IDs
sort_guys_by_id {
guys{$a}->id cmp guys{$b}->id; #That was easy!
}
Take a look at the tutorial. You'll find yourself writing better programs with fewer errors.
With your heart set on a sorted data structure, I recommend the following. It is a simple array of hashes and, rather than using the name string as the key for a single-element hash, it adds a new name key to each guy's data.
I hope that, with the Data::Dump output, it is self-explanatory. It is sorted by the Id field as you requested, but it still has the disadvantage that a separate index array would allow any ordering at all without modifying or copying the original hash data.
use strict;
use warnings;
use Data::Dump;
my $guys = {
"SomeGuy " => {
date_and_time => "11-04-2013",
Id => 7,
nr => 52,
picks => [
{ "This is an option" => "Option3" },
{ "This is another option" => "Option4" },
{ "This is another option" => "Option1" },
{ "And another one" => "Something" },
],
time_of => "06:11 AM",
total => 1,
},
"Another Guy" => { Id => 9, nr => 99, total => 6 },
"My Guy" => { Id => 1, nr => 42, total => 3 },
};
my #guys_sorted = map {
my $data = $guys->{$_};
$data->{name} = $_;
$data;
}
sort {
$guys->{$a}{Id} <=> $guys->{$b}{Id}
} keys %$guys;
dd \#guys_sorted;
output
[
{ Id => 1, name => "My Guy", nr => 42, total => 3 },
{
date_and_time => "11-04-2013",
Id => 7,
name => "SomeGuy ",
nr => 52,
picks => [
{ "This is an option" => "Option3" },
{ "This is another option" => "Option4" },
{ "This is another option" => "Option1" },
{ "And another one" => "Something" },
],
time_of => "06:11 AM",
total => 1,
},
{ Id => 9, name => "Another Guy", nr => 99, total => 6 },
]