I have a shared hash using the following:
my $glue = 'data';
my %options = (
create => 1,
exclusive => 0,
mode => 0644,
destroy => 0,
);
tie %hash1, 'IPC::Shareable', $glue, { %options };
The %hash1 declared as above, is in a single perl file, but it is called by multiple applications, each application modifies its own index of the hash:
Application1 --> $hash1{app1}="alpha";
Application2 --> $hash1{app2}="betta";
...
given that applications may or may not run simultanously, will there be an data loss if application1 and application2 try to modify the hash simultaneously?
You need to use a locking mechanism. (One is provided by the module.) Otherwise, changing any value of the hash can cause the loss of any other value changed at the same time. The following program demonstrates this quite easily:
use strict;
use warnings;
use IPC::Shareable qw( );
my $glue = 'data';
my %options = (
create => 1,
exclusive => 0,
mode => 0644,
destroy => 0,
);
my ($key) = #ARGV
or die("usage\n");
tie(my %h, 'IPC::Shareable', $glue, \%options);
my $c;
while (1) {
$h{$key} = ++$c;
my $got = $h{$key};
die("$key was overwritten (got $got; expected $c)\n")
if $got != $c;
}
Run it as follows in one console:
perl a.pl foo
Then run it as follows in another console:
perl a.pl bar
The key (no pun intended) to make it work with IPC::Shareable it to provide a key as one of the initial parameters:
use IPC::SysV qw( ftok );
tie( %hash, 'IPC::Shareable', $glue, { key => ftok(__FILE__,1234567) });
It is better to use the ftok function from IPC::SysV, because if you provide a string to IPC::Shareable, it will make it a numeric id using only the first 4 characters. Thus if you have two processes with one using test_one as a key and ther other test_two, both process will use the same shared memory segment, because in both case the resulting key produced by IPC::Shareable will be 1953719668. Worse is that if the first process creates a shared memory segment of, say 2048 bytes and you want the second to create one of 524288 bytes, you will actually end up with accessing the first shared memory segment with only 2048 bytes in size and its data, with the risk of modifying the wrong data.
If you use ftok even if you spawn 10,000 processes, it will always access the shared data without loss, because IPC::Shareable uses semaphore to ensure each process access the data one after the other.
Related
I'm creating an array of hash in perl 5.34.0:
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use autodie ':default';
use DDP;
my (#a1, #a2);
my %h = ('a' => 1);
push #a1, \%h; # make #a1 a hash of array with 'a' defined
$h{b} = 2; # define b within the hash, but don't write to #a1
push #a2, \%h; # push `a` and `b` onto #a2, but *not* #a1
p #a1; # DDP gives "p" which pretty-prints
p #a2;
and this outputs:
[
[0] {
a 1,
b 2
}
]
[
[0] {
a 1,
b 2
}
]
the problem is that the b key is showing up in #a1, yet $h{b} didn't exist when data was being written to #a1.
I don't want b to show up in #a1 and, it shouldn't.
How can I modify %h so that it doesn't magically appear in different arrays?
That code adds a reference to an existing (named) hash,
push #a1, \%h;
so when that is later queried you see whatever you'd see in the hash at that time. The array only carries a pointer of sorts with the address of the data, also referred to by the hash; it's an alias of sorts. So if the hash got written to in the meanwhile then that's what you'll see via #a1.†
Add it again and, even if the hash changed, you just add the same old reference.
What you want is to make a data copy and add a reference to that -- an anonymous hash
push #a1, { %h }; # but may need deep copy instead
Now the hash data gets copied in order to populate an anonymous hash constructed by {} and we have independent data that can be changed only by writing to it via its reference in #a1.
But note that if values in that hash themselves are references then those references get copied and we have the same problem! In that case you need a deep copy, and this is best done with libraries; Storable::dclone is a good one
use Storable qw(dclone);
push #a1, dclone \%h;
Now all of the actual data is copied, for (a reference to) an independent data copy on #a1
An important exception, which is frequently used
foreach my $elem (#ary) {
my %h;
# ... code that does its work and populates the hash ...
push #res, \%h;
}
Now this is OK, because the hash %h gets created anew at every iteration, being declared inside the loop. So what happens with the data created at the previous iteration?
Since its reference was added to an array the data itself is kept, being referred to by that reference in the array, and only by that. Precisely what you want, lone data accessible only via the array.
In such a case this is preferred to push #res, { %h }, which also works, since a data copy is avoided while the data is kept by, so to say, merely changing its ownership.
† And ff data is changed via #a1, like $a1[0]->{key} = 'val';, then that new value is seen via $h{key} as well.
I have a hash which is like this:
'IRQ_VSAFE_LPM_ASC_0' => '140',
'IRQ_VSAFE_LPM_ASC_1' => '141'.......and so on
I want to replace ASC_ by ASC_1 in all keys in the hash. I tried this:
foreach $_(keys $hash)
{
s/ASC_/ASC_1/g;
}
but it's not working.
You have to delete old keys from the hash and insert new ones,
use strict;
use warnings;
sub rename_keys {
my ($hash, $func) = #_;
my #k1 = my #k2 = keys %$hash;
$func->() for #k2;
#$hash{#k2} = delete #$hash{#k1};
}
my %hash = (
'IRQ_VSAFE_LPM_ASC_0' => '140',
'IRQ_VSAFE_LPM_ASC_1' => '141',
);
rename_keys(\%hash, sub { s/ASC_/ASC_1/ });
The previous answer addressed a way to do what you want. However, it also makes sense to explain why what you tried to do didn't work.
The problem is that the syntax used for working with hashes in Perl can mislead you with its simplicity compared to the actual way the hash works underneath.
What you see in Perl code is simply two pieces of information: a hash key and a corresponding hash value: $myHash{$key} = $value; or even more misleading %myHash = ($key => $value);
However, the way the hashes work, this isn't merely storing the key and a value as a pair, as the code above may lead you into thinking. Instead, a hash is a complicated data structure, in which the key serves as an input into the addressing which is done via a formula (hash function) and an algorithm (to deal with collistions) - the details are well covered on Wikipedia article.
As such, changing a hash key as if it was merely a value isn't enough, because what is stored in the hash isn't just a value - it's a whole data structure with addressing based on that value. Therefore when you change a hash key, it would ALSO change the location of the value in the data structure, and doing that isn't possible without removing the old entry and adding a brand new entry under a new key, which will delete and re-insert the value in the correct place.
A simple way to do this may be to use pairmap from recent List::Util.
use 5.014; # so we can use the /r flag to s///
use List::Util qw( pairmap );
my %new = pairmap { ($a =~ s/ASC_/ASC_1/r) => $b } %oldhash;
Here's some perl pseudo code for what I'm asking:
my %x;
# initialize %x
my $ref = whateverSyntacticSugarIsNeeded( $x{this}{hash}{is}{deep} );
# ...
# make use of $ref multiple times
# ...
delete $ref; # ideally, this would delete $x{this}{hash}{is}{deep}
... where the idea is to avoid use of $x{this}{hash}{is}{deep} more than is absolutely necessary.
I'm fairly sure this isn't possible and the least uses possible is 2 (the initial ref/copy of the value, then to delete the key/value pair from %x). However, if I'm mistaken then feel free to correct me.
It's not clear what you want exactly. If
%x = ( very => { deep => { hash => { is => "here" } } } );
and you assign
$y = $x{very}{deep}{hash}{is}
then is's like writing
$y = 'here'
so you can't delete $y. You can, though,
$z = $x{very}{deep}{hash};
delete $z->{is};
The easiest thing is just to reference the hash one level up.
This is especially true if you give the variable a semantically appropriate name (something other than ref):
my %x;
# initialize %x
my $ref = $x{this}{hash}{is};
# ...
# make use of $ref multiple times
# ...
delete $ref->{deep};
Perl tracks whether or not a piece of memory is used by using a count to that piece of memory. If I did this:
my $ref->{this}->{is}->{a}->{deep} = "hash";
undef $ref; # Note I use "undef" and not "delete"
I free up all of that memory. All of the hash references and the actual scalar that those hash references point to. That's because I have no further way of accessing that memory.
If I do something a bit simpler:
my %hash = ( one => 1, two => 2, ref => { three => 3, four => 4 }, five => 5 );
Note that $hash{ref} is a reference to another hash. If I did this:
my $ref = $hash{ref};
I now have two variables that can access that piece of memory. Doing this:
delete $hash{ref};
does not free up that memory because $ref still points to it. However, that hash reference is no longer in my %hash hash.
If I didn't delete $hash{ref}, but did this:
$ref->{seven} = 7;
I have changed %hash because $ref and $hash{ref} point to the same piece of memory: That same hash reference. Doing this:
delete $hash{ref}->{four};
or
delete $ref->{four};
will both delete a particular entry in that hash reference.
We don't have to do something that complex either:
my %hash = ( one => 1, two => 2, three => 3 );
my $ref = \%hash; #Creating a reference to that hash
delete $ref->{three};
This will delete $hash{three} since both are pointing to the same hash in memory. However,
undef $ref;
will not undefine $hash too.
I hope this covers your question. As long as there's another way to refer to a memory location, it's not freed in Perl. However, if you point a reference to a data structure, manipulating that data structure through the reference will manipulate that data structure referenced through an array or hash variable.
I have a script that is writing entries into a hash. However, at a certain point entries exist in the hash that I think should not. So, obviously I've cocked it up somewhere, but there is only one place where I think I add elements in to the hash and I've tested that to make sure that these "rogue" elements aren't being added at this location.
What I would like to do is break on a write to the hash, something like this, but in a "global" kinda way because I don't know where this stray write is in the code - I can't see it...
So what are my options? Can I set a watch point in the EPIC debugger and if so how? (I've had a play but can;t find anything relevant).
Or could I perhaps create a extended hash that can intercept writes somehow?
Any ideas on an "easy" debugging method. Otherwise I think I'll be back to brute force debug :S Thanks in davance...
Not an EPIC-specific answer, but check out Tie::Watch. You can setup a variable (like a hash) to be watched, and your program can output something every time the variable is updated.
updated: Tie::Trace does pretty much the same thing, with a simpler interface.
Here is a DIY-version of mobs answer: Make that hash a tied hash to a class that outputs a stack trace on every write access:
package MyHash {
use Tie::Hash;
use parent -norequire, "Tie::StdHash";
use Carp "cluck";
sub STORE {
my ($self, $k, $v) = #_;
cluck "hash write to $self: $k => $v";
$self->SUPER::STORE($k => $v);
}
}
tie my %hash => "MyHash";
$hash{foo} = 42;
print "$hash{foo}\n";
Output:
hash write to MyHash=HASH(0x91be87c): foo => 42 at -e line 1.
MyHash::STORE('MyHash=HASH(0x91be87c)', 'foo', 42) called at -e line 1
42
use Hash::Util qw(
lock_keys unlock_keys
lock_hash unlock_hash
);
my %hash = (foo => 42, bar => 23);
# no keys change
lock_keys(%hash);
# no keys/values change
lock_hash(%hash);
Is there any way to force Perl to call FETCHSIZE on a tied array before each call to FETCH? My tied array knows its maximum size, but could shrink from this size depending on the results of earlier FETCH calls. here is a contrived example that filters a list to only the even elements with lazy evaluation:
use warnings;
use strict;
package VarSize;
sub TIEARRAY { bless $_[1] => $_[0] }
sub FETCH {
my ($self, $index) = #_;
splice #$self, $index, 1 while $$self[$index] % 2;
$$self[$index]
}
sub FETCHSIZE {scalar #{$_[0]}}
my #source = 1 .. 10;
tie my #output => 'VarSize', [#source];
print "#output\n"; # array changes size as it is read, perl only checks size
# at the start, so it runs off the end with warnings
print "#output\n"; # knows correct size from start, no warnings
for brevity I have omitted a bunch of error checking code (such as how to deal with accesses starting from an index other than 0)
EDIT: rather than the above two print statements, if ONE of the following two lines is used, the first will work fine, the second will throw warnings.
print "$_ " for #output; # for loop "iterator context" is fine,
# checks FETCHSIZE before each FETCH, ends properly
print join " " => #output; # however a list context expansion
# calls FETCHSIZE at the start, and runs off the end
Update:
The actual module that implements a variable sized tied array is called List::Gen which is up on CPAN. The function is filter which behaves like grep, but works with List::Gen's lazy generators. Does anyone have any ideas that could make the implementation of filter better?
(the test function is similar, but returns undef in failed slots, keeping the array size constant, but that of course has different usage semantics than grep)
sub FETCH {
my ($self, $index) = #_;
my $size = $self->FETCHSIZE;
...
}
Ta da!
I suspect what you're missing is they're just methods. Methods called by tie magic, but still just methods you can call yourself.
Listing out the contents of a tied array basically boils down to this:
my #array;
my $tied_obj = tied #array;
for my $idx (0..$tied_obj->FETCHSIZE-1) {
push #array, $tied_obj->FETCH($idx);
}
return #array;
So you don't get any opportunity to control the number of iterations. Nor can FETCH reliably tell if its being called from #array or $array[$idx] or #array[#idxs]. This sucks. Ties kinda suck, and they're really slow. About 3 times slower than a normal method call and 10 times than a regular array.
Your example already breaks expectations about arrays (10 elements go in, 5 elements come out). What happen when a user asks for $array[3]? Do they get undef? Alternatives include just using the object API, if your thing doesn't behave exactly like an array pretending it does will only add confusion. Or you can use an object with array deref overloaded.
So, what you're doing can be done, but its difficult to get it to work well. What are you really trying to accomplish?
I think that order in which perl calls FETCH/FETCHSIZE methods can't be changed. It's perls internal part.
Why not just explicitly remove warnings:
sub FETCH {
my ($self, $index) = #_;
splice #$self, $index, 1 while ($$self[$index] || 0) % 2;
exists $$self[$index] ? $$self[$index] : '' ## replace '' with default value
}