How to exclude files under conf folder for distribution? - scala

I have a application.dev.conf and application.test.conf under my conf folder in my Play 2.3 application but I do not want it to be packaged as part of my distribution? What is the right excludeFilter for it?

Actually lpiepiora's answer will do the trick, however note that filtering on mappings in Universal will only exclude application.dev.conf from the conf folder and NOT from the jar itself.
I don't know about the play framework but in general if you have something like this:
hello
├── src
│ └── main
│ ├── scala
│ │ └── com.world.hello
│ │ └── Main.scala
│ ├── resources
│ │ ├── application.dev.conf
│ │ └── application.conf
Doing:
mappings in (Universal, ) ++= {
((resourceDirectory in Compile).value * "*").get.filterNot(f => f.getName.endsWith(".dev.conf")).map { f =>
f -> s"conf/${f.name}"
}
}
would produce the following package structure:
hello/
├── lib
│ └── com.world.hello-1234-SNAPSHOT.jar
├── conf
│ └── application.conf
However if you look into the jar, you will see that your dev.conf file is still in there:
$ unzip -v com.world.hello-1234-SNAPSHOT.jar
Archive: com.world.hello-1234-SNAPSHOT.jar
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
371 Defl:N 166 55% 10-01-2018 15:20 36c30a78 META-INF/MANIFEST.MF
0 Stored 0 0% 10-01-2018 15:20 00000000 com/
0 Stored 0 0% 10-01-2018 15:20 00000000 com/world/
0 Stored 0 0% 10-01-2018 15:20 00000000 com/world/hello/
0 Stored 0 0% 10-01-2018 15:20 00000000 com/world/hello/
13646 Defl:N 4361 68% 10-01-2018 12:06 7e2dce2f com/world/hello/Main$.class
930 Defl:N 445 52% 10-01-2018 13:57 5b180d92 application.conf
930 Defl:N 445 52% 10-01-2018 13:57 5b180d92 application.dev.conf
This is actually not really harmful but if you want to remove them too, here is the answer: How to exclude resources during packaging with SBT but not during testing
mappings in (Compile, packageBin) ~= { _.filter(!_._1.getName.endsWith(".dev.conf")) }

You could use mappings to exclude both files.
mappings in Universal := {
val origMappings = (mappings in Universal).value
origMappings.filterNot { case (_, file) => file.endsWith("application.dev.conf") || file.endsWith("application.test.conf") }
}

Does an excludeFilter as follows work for you?
excludeFilter in Universal in unmanagedResources := "application.dev.conf" || "application.test.conf"
(The unmanagedResourceDirectories key refers to conf/ by default.)

Related

Python Polars - Finding min value greater than col(a) in col(b) where col(a) is a numeric and col(b) is a column of lists of numerics

How do I convert the following slow operation in pandas to a fast operation in polars?
df.to_pandas().apply(lambda x: pd.cut([x['ingame_timestamp']], list(x['time_bins']), list(x['time_bins'])[1:]), axis=1)
Assume ingame_timestamp is a float and time_bins is a list.
I basically want to be able to do something like:
df.with_columns(pl.cut(value=pl.col('val'), bins=pl.col('time_bins), labels=pl.col('time_bins')[1:]).alias('val_time_bin'))
The above code works when I use to_pandas() but obviously this loses a bunch of the speed benefits of using polars and not using apply.
The following gives you an example data frame along with a column which is the desired output:
example_df = pl.DataFrame({'values': [0,1,2], 'time_bins': [[-1, -0.5, 0.5, 1], [0, 0.5, 1.5, 2.5], [1.5, 2.5, 3, 4.5]], 'value_time_bin': [0.5, 1.5, 2.5]})
It is sufficient to find the minimum value greater than "value" in the list "time_bins".
# reproducible dataset
df = pl.DataFrame({
'values': [0, 1, 2],
'time_bins': [[-1., -0.5, 0.5, 1.], [0., 0.5, 1.5, 2.5], [1.5, 2.5, 3, 4.5]]
})
If I understood you right, you need to create column with min value from time_bins that is greater than value in values.
One way to do it:
df.explode("time_bins").groupby("values").agg([
pl.col("time_bins").list(),
pl.col("time_bins").filter(
pl.col("time_bins") > pl.col("values")
).min().alias("value_time_bin")
])
┌────────┬───────────────────────┬────────────────┐
│ values ┆ time_bins ┆ value_time_bin │
│ --- ┆ --- ┆ --- │
│ i64 ┆ list[f64] ┆ f32 │
╞════════╪═══════════════════════╪════════════════╡
│ 0 ┆ [-1.0, -0.5, ... 1.0] ┆ 0.5 │
│ 1 ┆ [0.0, 0.5, ... 2.5] ┆ 1.5 │
│ 2 ┆ [1.5, 2.5, ... 4.5] ┆ 2.5 │
└────────┴───────────────────────┴────────────────┘

Why logrotate doesn't properly postrotate only has 1 day delay

I have in /etc/logrotate.d/mikrotik :
/var/log/mikrotik.log {
rotate 2
daily
compress
dateext
dateyesterday
dateformat .%Y-%m-%d
postrotate
#/usr/sbin/invoke-rc.d syslog-ng reload >/dev/null
rsync -avH /var/log/mikrotik*.gz /backup/logs/mikrotik/
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
The mikrotik.log.YYYY-MM-DD.gz file is created daily
The problem is that rsync in postrotate doesn't copy the last file. For example, on September 25, 2021, there are such files in /var/log:
-rw-r ----- 1 root adm 37837 Sep 24 23:49 mikrotik.log. 2021-09-24.gz
-rw-r ----- 1 root adm 36980 Sep 25 23:55 mikrotik.log. 2021-09-25.gz
and in /backup/logs/mikrotik/ are only:
-rw-r ----- 1 root adm 35495 Sep 23 00:00 mikrotik.log. 2021-09-22.gz
-rw-r ----- 1 root adm 36842 Sep 23 23:58 mikrotik.log. 2021-09-23.gz
-rw-r ----- 1 root adm 37837 Sep 24 23:49 mikrotik.log. 2021-09-24.gz
There is no file mikrotik.log.2021-09-25.gz from Sep 25 23:55 it will not be copied until the next rotation.
How to make a file packed today copied by postrotate ?
Problem solved.
It relied on the order in which the operations were performed.
Lgrotate does a 'postrotate' section before compressing to .gz.
The solution to the problem was to change the name from 'postrotate' to 'lastaction'.

OSX: detect 'restricted' filesystem flag programmatically

Starting from El Capitan the system got System Integrity Protection which doesn't allow certain activities for some folders and files. Using Terminal "ls lO" command you can see flags that specific file or folder has.
drwxr-xr-x# 3 root wheel hidden 96 Aug 12 2014 opt
drwxr-xr-x 6 root wheel sunlnk,hidden 192 Nov 28 15:14 private
drwxr-xr-x# 64 root wheel restricted,hidden 2048 Nov 29 13:48 sbin
lrwxr-xr-x# 1 root wheel restricted,hidden 11 Nov 28 15:13 tmp -> private/tmp
drwxr-xr-x# 10 root wheel restricted,hidden 320 Nov 28 15:21 usr
lrwxr-xr-x# 1 root wheel restricted,hidden 11 Nov 28 15:13 var -> private/var
I'm interested in "restricted" flag. How can it be found using Swift without executing terminal commands?
I want to emphasize that executing Process() in code is not an approach that suits the needs.
"Hidden" flag can be detected via "isHidden" property from here URLResourceKey. However there is nothing about "restricted".
Could somebody point me to the right direction?
The things listed by the -O option on the ls tool are the file flags. These can be read by the stat() function in the BSD layer, and are found in the st_flags field of the resulting structure.
The "restricted" flag in ls's output corresponds to SF_RESTRICTED, so you can read it by doing something like this:
func isRestricted(at url: URL) throws -> Bool {
let flags: UInt32 = try url.withUnsafeFileSystemRepresentation { fsRep in
var info = stat()
if stat(fsRep, &info) != 0 {
guard let code = POSIXError.Code(rawValue: errno) else {
throw CocoaError(.fileReadUnknown)
}
throw POSIXError(code)
}
return info.st_flags
}
return flags & UInt32(bitPattern: SF_RESTRICTED) != 0
}

match a pattern and print subsequent lines

there are 200 files named File1_0.pdb,File1_60.pdb etc....it looks like:
ATOM 1 N VAL 1 8.897 -21.545 -7.276 1.00 0.00
ATOM 2 H1 VAL 1 9.692 -22.015 -6.868 1.00 0.00
ATOM 3 H2 VAL 1 9.228 -20.766 -7.827 1.00 0.00
ATOM 4 H3 VAL 1 8.289 -22.236 -7.693 1.00 0.00
TER
ATOM 5 CA VAL 1 8.124 -20.953 -6.203 1.00 0.00
ATOM 6 HA VAL 1 8.072 -19.874 -6.345 1.00 0.00
ATOM 7 CB VAL 1 6.693 -21.515 -6.176 1.00 0.00
ATOM 8 HB VAL 1 6.522 -22.024 -5.227 1.00 0.00
ATOM 9 CG1 VAL 1 5.684 -20.370 -6.330 1.00 0.00
ATOM 10 1HG1 VAL 1 5.854 -19.861 -7.279 1.00 0.00
i have to extract the part after TER and put in a different file...this has to be done on all 200 files. I did something like sed '1,/TER/d' File1_0.pdb > 1_0.pdb. But this will work for one file at a time...can there be a solution for all 200 files in one go... output file is named same only "File" is removed from the name...
for i in *.pdb; do sed '1,/TER/d' $i > ${i/File/}; done
This might work:
seq 0 200| xargs -i -n1 cp File1_{}.pdb 1_{}.pbd # backup files
sed -si '1,/TER/d' 1_{0..200}.pdb # edit files separately inline

Running perl script from a shell script

I've written a shell script that does stuff on Centos64
At the end of this script, I run a perl script.
It works fine as root (I have installed the perl modules), but when I run it as a system user
I get the message:
./UserActivityReport.sh
Can't locate MIME/Lite.pm in #INC (#INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /optreports/sendmailUAR.pl line 3.
BEGIN failed--compilation aborted at /opt/reports/sendmailUAR.pl line 3.
As if the modules were not installed for the user...
I run the script with this command in my shell script:
/usr/bin/perl /opt/reports/sendmailUAR.pl
Why is it not running?
NOTE: I cannot install the perl modules with the system user.
===
There you go:
perl -MMIME::Lite -MData::Dumper -e'print Dumper \%INC'
$VAR1 = {
're.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/re.pm',
'warnings/register.pm' => '/usr/lib/perl5/5.8.8/warnings/register.pm',
'XSLoader.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/XSLoader.pm',
'IO/Handle.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/IO/Handle.pm',
'MIME/Types.pm' => '/usr/lib/perl5/site_perl/5.8.8/MIME/Types.pm',
'SelectSaver.pm' => '/usr/lib/perl5/5.8.8/SelectSaver.pm',
'IO/Seekable.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/IO/Seekable.pm',
'warnings.pm' => '/usr/lib/perl5/5.8.8/warnings.pm',
'File/Basename.pm' => '/usr/lib/perl5/5.8.8/File/Basename.pm',
'Fcntl.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Fcntl.pm',
'IO.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/IO.pm',
'Symbol.pm' => '/usr/lib/perl5/5.8.8/Symbol.pm',
'bytes.pm' => '/usr/lib/perl5/5.8.8/bytes.pm',
'MIME/Type.pm' => '/usr/lib/perl5/site_perl/5.8.8/MIME/Type.pm',
'Carp.pm' => '/usr/lib/perl5/5.8.8/Carp.pm',
'MIME/Base64.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/MIME/Base64.pm',
'Exporter/Heavy.pm' => '/usr/lib/perl5/5.8.8/Exporter/Heavy.pm',
'File/Spec/Unix.pm' => '/usr/lib/perl5/5.8.8/File/Spec/Unix.pm',
'FileHandle.pm' => '/usr/lib/perl5/5.8.8/FileHandle.pm',
'strict.pm' => '/usr/lib/perl5/5.8.8/strict.pm',
'Exporter.pm' => '/usr/lib/perl5/5.8.8/Exporter.pm',
'vars.pm' => '/usr/lib/perl5/5.8.8/vars.pm',
'MIME/Lite.pm' => '/usr/lib/perl5/site_perl/5.8.8/MIME/Lite.pm',
'MIME/QuotedPrint.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/MIME/QuotedPrint.pm',
'File/Spec.pm' => '/usr/lib/perl5/5.8.8/File/Spec.pm',
'overload.pm' => '/usr/lib/perl5/5.8.8/overload.pm',
'IO/File.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/IO/File.pm',
'Mail/Address.pm' => '/usr/lib/perl5/site_perl/5.8.8/Mail/Address.pm',
'Data/Dumper.pm' => '/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Data/Dumper.pm'
};
In one of the comments, you have posted a directory listing like this:
[root~]# ls -l /usr/lib/perl5/site_perl/5.8.8/
total 60
drwxr-x--- 3 root root 4096 Sep 14 13:09 Date
drwxr-x--- 3 root root 4096 Sep 14 13:09 Email
drwxr-x--- 4 root root 4096 Sep 14 13:10 Mail
drwxr-x--- 2 root root 4096 Sep 14 13:10 MIME
drwxr-x--- 2 root root 4096 Sep 15 06:30 OLE
drwxr-x--- 2 root root 4096 Sep 15 06:30 Parse
drwxr-x--- 2 root root 4096 Sep 14 13:10 Pod
drwxr-x--- 3 root root 4096 Sep 15 06:30 Spreadsheet
drwxr-x--- 2 root root 4096 Sep 14 13:09 Time
-r--r--r-- 1 root root 6500 Aug 21 07:19 version.pm
-r--r--r-- 1 root root 9887 Aug 21 07:12 version.pod
It should be fairly evident that users other than root do not have access to these files. You should probably do more thorough audit by looping over all of #INC and changing permissions as necessary, but for a start, find /usr/lib/perl5/site_perl/5.8.8 -type d -exec chmod a+rx {} \;
I'm betting root's umask is such that files are not world readable by default.
find /usr/lib/perl5 -not -perm -o=r
Anything files listed are not world readable (and probably should be). You can fix it this way
find /usr/lib/perl5 -not -perm -o=r -exec chmod o+r {} +
If you wish to make all files matched with the first command readable by everyone.
MIME::Lite is at /usr/lib/perl5/site_perl/5.8.8/MIME/Lite.pm and /usr/lib/perl5/site_perl/5.8.8 is in system's #INC path.
As user system check that you can read that file:
cat /usr/lib/perl5/site_perl/5.8.8/MIME/Lite.pm
If you get an error doing so check the permissions of the file and the parent directories.
Otherwise, run the script as system with strace and see why it is failing to find the module file.