Perl script that checks rotated log files - perl

There is a web server running multiple websites with configured daily log files rotation.
Task: create a perl script that checks a list of current log files in source directory and compare it with a list of rotated log files in another dir. Script must print a name of log file, if one the yesterday's log was not rotated.
Source dir example:
ls -l /var/log/httpd/logs/*log
-rw-r--r-- 1 root root 0 May 20 00:01 /var/log/httpd/logs/access.log
-rw-r--r-- 1 root root 483652 May 20 12:54 /var/log/httpd/logs/othersite.com_80-access.log
-rw-r--r-- 1 root root 305 May 20 11:51 /var/log/httpd/logs/othersite.com_80-error.log
-rw-r--r-- 1 root root 0 May 20 00:01 /var/log/httpd/logs/error.log
-rw-r--r-- 1 root root 46222 May 20 12:45 /var/log/httpd/logs/www.site.com_8880-access.log
-rw-r--r-- 1 root root 0 May 20 00:01 /var/log/httpd/logs/www.site.com_8880-error.log
dir with a rotated logs:
ls -l /var/log/httpd/logs/completed/|grep 2014-05-19
-rw-r--r-- 1 root root 20 May 20 00:01 access.log.2014-05-19.gz
-rw-r--r-- 1 root root 107244 May 20 00:01 othersite.com_80-access.log.2014-05-19.gz
-rw-r--r-- 1 root root 9991 May 20 00:01 www.site.com_8880-access.log.2014-05-19.gz
-rw-r--r-- 1 root root 20 May 20 00:01 www.site.com_8880-error.log.2014-05-19.gz
In this case two yesterday's log files are absent\were not rotated:
-rw-r--r-- 1 root root 483652 May 20 12:54 /var/log/httpd/logs/othersite.com_80-access.log
-rw-r--r-- 1 root root 305 May 20 11:51 /var/log/httpd/logs/othersite.com_80-error.log
Looking forward to any suggestions!

Do a glob <> on the source directory and put in a hash key. You put the size of the log or date in the value of the hash. Then go to the destination directory and do the same thing - read the files and as you loop, check to see if you have the file in you hash. You can compare the size and date also. if you don't have it, copy it.

Related

Serialization error due to different folder structure on local and server machine

I am getting following error in my server tomcat logs:-
ERROR: The serialization policy file '/nos/js/4937711D6147993DCFE47FF203A0FEDC.gwt.rpc' was not found; did you forget to include it in this deployment?
On my local machine the above path is present and tomcat is able to locate it and hence I don't get this error. All the folders are present inside nos folder.
But on server, all the folders are present inside ROOT and nos folder is not present and hence server tomcat is not able to identify this path
I am providing folder structure on my local machine and server below :-
Windows folder structure
Directory: C:\Users\abc.metadata.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\nos
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 06-09-2022 09:07 PM classes
d----- 06-09-2022 09:09 PM com.xyz.nos.Common
d----- 06-09-2022 09:07 PM gxt
d----- 06-09-2022 09:10 PM js
d----- 06-09-2022 09:07 PM maven-status
d----- 07-09-2022 05:04 PM META-INF
d----- 06-09-2022 09:07 PM WEB-INF
Linux server folder structure
Directory
[x_qa#webapp-nos-69d5d4865-mzkxz ROOT]$ pwd
/opt/tomcat/webapps/ROOT
[x_qa#webapp-nos-69d5d4865-mzkxz webapps]$ cd ROOT
[x_qa#webapp-nos-69d5d4865-mzkxz ROOT]$ ls -lrt
total 36
drwxr-xr-x 4 root root 4096 Jan 1 1970 nos-10.0.8-SNAPSHOT
drwxr-xr-x 3 root root 4096 Jan 1 1970 maven-status
drwxr-xr-x 7 root root 4096 Jan 1 1970 js
-rw-r--r-- 1 root root 182 Jan 1 1970 index.html
drwxr-xr-x 8 root root 4096 Jan 1 1970 gxt
drwxr-xr-x 2 root root 4096 Jan 1 1970 gwt-unitCache
drwxr-xr-x 3 root root 4096 Jan 1 1970 classes
drwxr-xr-x 1 root root 4096 Jan 1 1970 WEB-INF
drwxr-xr-x 3 root root 4096 Jan 1 1970 META-INF
Can anyone suggest how can I make consistent folder structure between server and local machine by keeping both of these folders in sync ?
Let me know if my understand is incorrect ?

How to reset VSCode extensions for a workspace

I was having issues with a workspace so I tried disabling all of extensions for it. Then the only option was to enable all extensions for the workspace. I have extensions globally configured to be off or on, but I'm not sure how to get a workspace to reset to the global extensions now?
I know I can open a new window up and start manually enable/disabling the extensions in my workspace to match the new fresh window. The problem is that it would then have its own workspace extensions defined, so if I toggled one at a global level it would still have an override.
I also tried deleting the .vscode folder in the workspace, but that doesn't seem to change the extensions for the workspace.
It's not a great method but I guess this works with a lot of manual leg work.
Enable all extensions for workspace
Open a new VSCode window
Goto extensions and filter by #disabled
You can also do #enabled and disable all extensions in the first step if you have more disabled than enabled.
Now you have a list you can target. Just click Disable instead of Disable for Workspace.
I believe this should get your workspace back to normal. There has to be an easier way though. You may also have to toggle enabled extensions to disabled and then back to enabled in case it has a workspace override for them -- I didn't check how it would behave if a new window disabled the extension whether or not it would keep staying on.
I've found a way to reset all settings for a given workspace.
First, navigate to ~/.config/Code/User/workspaceStorage. Inside there, you'll find a lot of folders with seemingly random names. Each folder seems to represent one workspace.
# cd ~/.config/Code/User/workspaceStorage
# ls -la
total 156
drwxr-xr-x 39 micael micael 4096 Feb 18 15:00 .
drwxr-xr-x 6 micael micael 4096 Nov 15 13:05 ..
drwxr-xr-x 2 micael micael 4096 Jan 18 16:09 0cf549e23d37c32c70a7e30998ade1fe
drwxr-xr-x 3 micael micael 4096 Dec 29 17:42 1b637cb30f6c3acc9273df20c84be7aa
drwxr-xr-x 2 micael micael 4096 Jan 17 22:09 2010f3fb6dbb2574f12a5ba614b3b136
drwxr-xr-x 2 micael micael 4096 Feb 18 15:01 a09a9ab934662794ac48730fc950a654
...
Inside each folder, there's a workspace.json file that contains the path of the folder your workspace was created from.
We can use grep to quickly search all those folders for a matching string:
grep -r 'myfolder' ., where myfolder should be the name of the folder your workspace is using.
In this example, my workspace was created from a folder called python-playground:
# grep -r 'python-playground' .
./a09a9ab934662794ac48730fc950a654/workspace.json: "folder": "file:///home/micael/projects/python-playground"
grep: ./a09a9ab934662794ac48730fc950a654/state.vscdb: binary file matches
grep: ./a09a9ab934662794ac48730fc950a654/state.vscdb.backup: binary file matches
In my case, the folder I'm looking for is ./a09a9ab934662794ac48730fc950a654.
You'll likely get a few results, all inside the same folder. Let's first take a look inside the folder:
# ls -la a09a9ab934662794ac48730fc950a654
total 64
drwxr-xr-x 2 micael micael 4096 Feb 18 15:01 .
drwxr-xr-x 39 micael micael 4096 Feb 18 15:00 ..
-rw-r--r-- 1 micael micael 28672 Feb 18 15:01 state.vscdb
-rw-r--r-- 1 micael micael 24576 Feb 18 15:01 state.vscdb.backup
-rw-r--r-- 1 micael micael 64 Feb 18 15:00 workspace.json
# cat a09a9ab934662794ac48730fc950a654/workspace.json
{
"folder": "file:///home/micael/projects/python-playground"
}%
Now, we can either remove this folder or move it elsewhere. Make sure to close VSCode before doing so.
# rm -rf a09a9ab934662794ac48730fc950a654

Pydev tags import as “unresolved import” all compiled extensions

PyDev with Python3.5 seems unable to recognize imports from c-compiled extensions, including packages compiled via Cython.
I am working on an up-to-date debian/stretch machine with a stripped-down self-installed (in home dir) Eclipse/Neon with PyDev added via update site, if it matters.
I have both "Python 2.7.13" and "Python 3.5.2+" installed.
One of the problematic packages is lxml. I installed the debian packages and also tried installing manually via pip (and re-created the interpreter in eclipse afterwards to ensure full finding). In all cases packages do work.
In Python2 everything works as advertised.
Under Python3 PyDev flags from lxml import etree as error (but resulting program works):
Unresolved import: etree richiedi_certificato_dispositivo.py /trasmissione-telematica/Serializzazione line 8 PyDev Problem
Note: I can import lxml with no error, but then any access to lxml.etree... will be flagged in error. Data completion is consistent (i.e.: etree won't be in the offered list).
lxml is installed in the usual location:
mcon#vocore:~$ ls -l /usr/lib/python3/dist-packages/lxml
total 2156
-rw-r--r-- 1 root root 8152 Sep 5 2014 builder.py
-rw-r--r-- 1 root root 3366 May 5 2016 cssselect.py
-rw-r--r-- 1 root root 18387 May 5 2016 doctestcompare.py
-rw-r--r-- 1 root root 7641 Sep 25 2011 ElementInclude.py
-rw-r--r-- 1 root root 9490 Aug 20 06:48 _elementpath.py
-rw-r--r-- 1 root root 1710088 Aug 24 10:14 etree.cpython-35m-x86_64-linux-gnu.so
drwxr-xr-x 3 root root 4096 Jan 3 08:58 html
drwxr-xr-x 3 root root 4096 Jan 3 08:58 includes
-rw-r--r-- 1 root root 551 Oct 7 2012 __init__.py
drwxr-xr-x 4 root root 4096 Jan 3 08:58 isoschematron
-rw-r--r-- 1 root root 17450 Aug 20 06:48 lxml.etree_api.h
-rw-r--r-- 1 root root 8902 Aug 20 06:48 lxml.etree.h
-rw-r--r-- 1 root root 366440 Aug 24 10:14 objectify.cpython-35m-x86_64-linux-gnu.so
drwxr-xr-x 2 root root 4096 Jan 3 08:58 __pycache__
-rw-r--r-- 1 root root 92 Sep 5 2014 pyclasslookup.py
-rw-r--r-- 1 root root 8531 Nov 20 2014 sax.py
-rw-r--r-- 1 root root 230 Sep 25 2011 usedoctest.py
mcon#vocore:~/trasmissione-telematica$
As You can see etree is in a shared lib, as is objectify; a quick check shows also objectify is not handled by PyDev. I checked a few other "c-extension" packages (e.g.: import pycurl and from Crypto.Util import strxor) with the same result, so it seems a problem with "C" extensions.
Have you tried to put the offending packages in the forced builtins (as described in http://www.pydev.org/manual_101_interpreter.html)?
If that doesn't work, do you have some error in your error log? (see http://www.pydev.org/faq.html#PyDevFAQ-HowdoIReportaBUG%3F for details on getting it)

lb web-client export always using /tmp directory

In my lb-web-server.config file I have set the following:
[handler:delimited-file]
classname = com.logicblox.bloxweb.delim.DelimitedFileHandler
tmpdir = /my/temp/dir
When I run an export with lb web-client export --no-compress --output /some/dir/myfile.csv http://service-uri a temporary file is created in /my/temp/dir, but several files are also written to /tmp:
-rw-r--r--. 1 root root 109M Oct 9 12:13 tmp21498_2316_XXYMElJ4
-rw-r--r--. 1 root root 106M Oct 9 12:13 tmp21498_2317_XXTHg8fE
-rw-r--r--. 1 root root 114M Oct 9 12:15 tmp21498_2318_XXQhuWvW
-rw-r--r--. 1 root root 106M Oct 9 12:15 tmp21498_2319_XX7KbYmo
When the combined size of these temp files exceeds the available space on /tmp, the export fails.
Am I missing an additional setting in lb-web-server.config which would cause these files to be written to the correct location?
I think you can avoid storing these temp files under /tmp by setting the LB_TEMPDIR environment variable to another directory.
It would appear the runtime is using "generic" temporary files, whose location is controlled by the environment variable LB_TEMPDIR (defaulting to /tmp). I'm not aware of any config file setting to control it, unfortunately.
I will file a bug.

GWT + eclipse, which files are part of my source?

i created a GWT project in eclipse, and it's time to put some code back into source control. i'm not sure at this point which files are generated and can be left out of source control,
a. under war/myapp/gwt/... i see many, many files related to the standard GWT themes.
b. under war/myapp,
-rw-r--r-- 1 10102022 1602597546 1876 Jan 24 16:41 0182DE3CC529E42DA72BBD969A44841E.gwt.rpc
-rw-r--r-- 1 10102022 1602597546 1456 Jan 24 14:09 4F701266A6E52E1E409583EA9AEC39E2.gwt.rpc
-rw-r--r-- 1 10102022 1602597546 1876 Jan 25 08:38 D98FD8FE56B70659E9608109BCF8B3C1.gwt.rpc
-rw-r--r-- 1 10102022 1602597546 43 Dec 16 16:01 clear.cache.gif
drwxr-xr-x 6 10102022 1602597546 204 Jan 25 08:26 gwt
-rw-r--r-- 1 10102022 1602597546 11289 Dec 17 01:33 hosted.html
-rw-r--r-- 1 10102022 1602597546 5232 Jan 25 08:31 photodrop_web_gwt.nocache.js
normally i'd just rely on eclipse build > clean to get rid of the build time artifacts. however, i did that, and i still see WEB-INF/classes full of class files, so i know that clean isn't working.
"war/myapp" is by default GWT's output directory. So as long as you haven't saved any files there manually (you shouldn't), you can delete that directory completely.
As always, make a backup first...
I'm using source control for GWT + GAE, and this ignore file has been working great:
syntax: regexp
^war/myapp$
syntax: regexp
^war/WEB-INF/appengine-generated/datastore-indexes-auto\.xml$
syntax: regexp
^war/WEB-INF/appengine-generated/local_db\.bin