Not sure if I am explaining this well, but here goes...
I have a perl script/flow that runs various steps. Each step is basically dependent on the output of its previous step in order to run.
For example:
myflow -step1...input is file0, produces file1
myflow -step2...input is file1, produces file2
myflow -stepN...input is fileN-1, produces fileN
Right now users can run myflow -step1 -step2...-stepN to go from start to finish. I would like to somehow have the ability for the user to run myflow -stepN, have myflow check to see which steps need to be run prior to it, and then run stepN. Maybe no steps were run, so myflow -stepN would start from step1 and continue until it finishes stepN or an error occurs. Maybe step1 through step3 ran fine previously, so running -stepN would start from step4. Maybe all steps ran fine, but the user modified/deleted/touched an intermediate file, so running -stepN would detect this and rerun from that previous step.
Is there a cpan module that essentially mimics this make behavior, i.e. given steps, inputs they require, and outputs they produce, create a dependency graph and determine which steps need to be run?
I'm thinking you could use make itself instead of trying to simulate it.
The makefile rules for "building" each fileX "target" from the fileX-1 "source file" would be invoking your script for the respective step.
Related
I have a perlscript file was running fine in crontab but suddenly it stopped running without any modification.
cd /home/user/public_html/crons && ./script.pl 2>&1 >/dev/null
The top of the script file is #!/usr/bin/perl -X
The output expect from this script is changes in database
I have another script file with the same modification and still works fine
When I run the file in the browser it works fine and execute all lines without any problem
I tried full path /usr/bin/perl but it didn't work
I tried Perl at the beginning but it didn't work
I run the command from SSH using putty but nothing happened
I checked log file /var/log/cron but no errors at all
I created temporary log file cd /home/user/public_html/crons/script.pl> /tmp/temp.log 2>&1 to see the errors but the log is empty
Here is the solution:-
I found the issue, There is was a stuck process for the same cron file , so i killed this process and its fixed
You can find your file process like this
ps aux | grep 'your cron file here'
This is a really common antipattern people seem to tend toward with cron.
Cron sends you an email with the output of your script, if it generates any output. People often redirect output to /dev/null to prevent cron from sending the email. This is bad because now the output of your script is lost entirely. Even if the script has some built-in logging, it might generate errors before it gets the log file opened and those are lost. It also might crash in a way that doesn't get written to the logging mechanism.
At a bare minimum, you should just remove 2>&1 >/dev/null to start receiving the email. (and also, test your mail setup using a temporary cron job like 1 * * * * echo "Test" )
The next better solution is to change it to >> /var/log/myscript/current.log and then also set up something to rotate the log files (like logrotate) and also make sure to create that directory with permissions that the script user is allowed to write to it. By only redirecting STDOUT of the script, any errors or warnings it writes to STDERR cause you to get an email, and if there are no errors/warnings the output goes to the log file and no email gets sent.
Neither of those changes solve the root problem though, which is that when cron runs your script it does so with a different environment than you have on the command line. What you really want is a way to run the script with a consistent environment, and log it. The "ultimate solution" is to define your task in some kind of service manager, and then use cron to occasionally start it. For instance, you could use systemd and define a service that doesn't restart, then use systemctl start my_custom.service in your cron job. Now you can test independent of cron, and your tests will have the same exact environment, and be logged by the service manager. As extra bonuses, you are protected from accidentally running your script twice at once, and you get a clean way to stop a running cron job without the danger of stale pid files.
I don't particularly advocate systemd myself, but thankfully there are lots of alternatives:
Runit : http://smarden.org/runit/runsvdir.8.html
S6 : https://skarnet.org/software/s6/
Perp : http://b0llix.net/perp/site.cgi?page=perpd.8
(but installing and configuring a service manager is a bigger task than just using systemd if your distro is based on systemd) Each of these allows you to define a service that doesn't restart. Then you use a shell command to issue a "run once" directive to the supervisor, which runs the task as a child. Now you can easily launch the jobs yourself and see all the errors in the log, and then add that command to the crontab and know that it will run identically when cron starts it.
Back to your original problem, once you get some logging you are likely to discover it is a permission problem or a upgraded module in the system perl.
This issue usually is encountered when trying to run make test and sees one test fails. The README describes one can run each test individually, didn't clearly specifies how to do so.
make test uses the script called TEST in the test directory (t). To replicate make test of a single file, one would use this script as follows:
[.../perl/t]$ ./perl -I../lib TEST op/array.t
t/op/array ... ok
All tests successful.
Elapsed: 0 sec
u=0.01 s=0.00 cu=0.03 cs=0.02 scripts=1 tests=194
If you want to see the raw output of the test script, you can run perl as follows:
[.../perl/t]$ ./perl -I../lib op/array.t
1..194
ok 1
ok 2
ok 3
...
ok 192 - holes passed to sub do not lose their position (multideref, mg)
ok 193 - holes passed to sub do not lose their position (aelem)
ok 194 - holes passed to sub do not lose their position (aelem, mg)
The above information and more is found in perlhack.
This document explains how Perl development works. It includes details about the Perl 5 Porters email list, the Perl repository, the Perlbug bug tracker, patch guidelines, and commentary on Perl development philosophy.
Note that you need to run make test_prep before the above commands work. (If you've run make test, you've effectively run make test_prep already.)
Run ./perl harness ../foo/boo.t in the t directory, with foo/boo the name of the failing test.
To run a single test script, use perl, or better, prove. Assuming you are in the module's base directory:
prove -lv t/some-test-script.t
This will run the test script against the libraries in ./lib, with fallback to the libraries available to your install of Perl.
If you want to use the build libraries built by make, then this:
prove -bv t/some-test-script.t
Now the test script will be run against the libraries in ./blib, falling back to libraries installed for your Perl.
The test scripts are typically just Perl scripts that live in a t/ or xt/ or some similar path within the distribution's directory structure. So you can also run them just with Perl:
perl -Iblib t/some-test-script.t
But prove produces nicer test summary information and color coding.
That is about as granular as you can get unless tests are written to allow for targeting specific segments within a test script. If you need to target a specific test within a test script you'll usually have to dig into the test code itself.
NOTE
I've had to remove like two chunks of this post because stack overflow kept interpreting it as code when it isn't and it wouldn't let me post, I'll just make a screenshot of what the post is supposed to look like and post it here. Read this instead.
Summary
I was finally trying to learn how to use VSCode tasks and so I copied the first task example from here and created a shell script at scripts/test.sh which contains simply "echo foo". I also commented out the windows alternative script because I exclusively use WSL/Bash. Whenever I run the task I receive a "The terminal process terminated with exit code: 1" error message, which is no help whatsoever.
Testing
I ran various tests and I have no idea why this isn't working.
Proving The Task Is In The Correct Directory & In WSL
First I thought maybe the task isn't running in WSL or that the directories are out of sync, so I changed the commands to see what happens.
First, I changed it to:
"command": "pwd",
and the output was "/mnt/f/.../.../tmp/tmp.1BitOIA78E" (... are for some arbitrary path) so clearly I concluded I was running on WSL and in the right path.
Proving the Script is Executeable
Next I thought, maybe the script I'm trying to run isn't executable or something to that affect, so I changed the command to:
"command": "stat ./scripts/test.sh",
and I got the following output which shows, the file exists, it's executeable & can be accessed through "./scripts/test.sh" from whatever directory the Task is set to on construction
I run the task by typing ctrl-shift-P to open the menu, select "run tasks" and then select 'My First Task'.
Note: I don't think this is a settings problem. There're no workspace settings setup (because this is just me testing) & just in case, you can find my current user settings here which I updated immediately before posting this.
Expectations
What I'd like is either:
Someone to tell me how I can access the stderr and stdout log of the shell upon startup so I can get some actually helpful information as to why this is happening.
Someone to tell me why I can run a script perfectly fine outside of a task, but within a task it completely fails.
Whats also of note is that the script isn't the problem here. Leaving it completely blank, doesn't stop the terminal from straight up crashing.
I'm running into problems testing a new addition to a module. (Specifically - the ~ operator seems to be not working in Math::Complex for this new feature only.) It's too bizarre to be what it appears but the ideal scheme would be to add the -d option on the top line of the .t program.
Well, I was quickly disabused of that idea! It does not invoke the debugger.
If I wanted to use the debugger, I'd need to create an edit of the .t program that:
Uses (the use command) the module directly. not in the form of
BEGIN { use_ok('My::Module') };
Does not "use Test::More;"
A few other edits that cause gluteal pains
The problem with doing that is that any changes I make in the edited test program I still need to transfer back to the true test program use in "make test". Error prone as best.
I am already using "make test TEST_VERBOSE=1" so that my stdio output shows up. But there's GOT to be a simpler way to invoke the debugger on the .t
Thanks for ideas here.
-- JS
use_ok tests are great, but you should have them in test files of their own, not test files that also test other things.
I'm not sure why you would need to avoid Test::More or use_ok to run the debugger, though. What does happen when you try your test directly:
perl -d -Mblib t/yourtestfile.t?
If all else fails, you can try using Enbugger in your test script.
I have problem with setting start module and function in "Run configuration"
I'm doing it that way:
"Run -> Run Configuration" and in "start" section I'm setting
Module: mod,
Function: hello
my code:
-module(mod).
-export([hello/0]).
hello()-> io:format("42").
Now, when I hit "Run" I would like mod:hello() to be execute automatically, but it doesn't work.
What am I doing wrong?
The code does get executed...
When you hit "Run", mod:hello() does get executed. The thing is, the execution of mod:hello() is meant for system initialization, like loading library code and initializing looping states. The side effect of mod:hello(), which is the string "42" as stdout will not be reflected in your Eclipse console. To prove my point, we can create some more explicit and more persistent side effects, like creating a file in the file system called output_file.txt. Change mod.erl to something like this:
-module(mod).
-export([hello/0]).
hello() ->
os:cmd("touch output_file.txt").
Hit "Run", and you will find a output_file.txt file being created under your workspace directory. This is an evidence for the execution of mod:hello().
To achieve what you want...
In an Unix shell:
$ erlc mod.erl
$ erl -noshell -s mod hello -s init stop
42
Depending on what you want to execute, there is an alternative to the answer above: the "live expressions". There is a view with this name in the same place as the console, where you can enter an expression and enable it for evaluation every time a module is recompiled.
This works nicely for expression that aren't heavy to evaluate and that are without side effects, can be used as a form of alternative to having a test suite.