I have a couple of modules (DZP::Catalyst and DZP::OurPkgVersion) both of their purposes involve writing out files to the disk. I'm not sure how to test them, are there any good strategies for testing files written out to disk? any place I could go to read up on it?
Well, in this particular case (Dist::Zilla plugins), you're going to want to use Dist::Zilla::Tester. It takes care of much of the grunt work of creating a temporary directory, populating it with files, and cleaning up afterwards. For example, see the tests from my DZP::CJM or the tests from Dist::Zilla itself, especially the plugins directory.
Update: Dist::Zilla 4.200002 introduced the Test::DZil module, which adds some utility functions useful for testing plugins. You'll probably want to use it instead of using Dist::Zilla::Tester directly.
It depends somewhat on the module, but my general strategy is:
Ensure that file content logic is 100% separate - as far as being in different methods - from file mechanics (e.g. choosing directory/opening files/closing files/error handling).
Ensure that the file mechanics is 100% flexible, e.g. you can choose the directory/filename from the external driver.
Write tests for the file mechanics, by simply opening the specified file in specified directory, closing it, making sure no errors happen and that expected file exists and has size zero
create an array of test data, with each element of the array consisting of 3 parts
Input data for file content logic, possibly coupled with test configuration indicating which methods from file content logic to call on that data if warranted.
Expected file name to be set
Expected file contents, in the form of tar-balled expected files (exact files with exact expected content to be generated and the correct expected name).
The expected results tarballs should be in a separate sub-directory (say "expected_results" under the directory where your test script lives.
You need a tarball in case your file generation logic produces >1 file.
Then, run a loop over each test in the test array you previously created:
Create a new "actual results" temp directory (or clean up the one from prior test)
Set the directory in you module to the temp directory; set the filename of your module to the expected filename from test info.
Run the file opener method (previously tested)
Run the content generation logic from the module using test's logic directions (if applicable) and test's input data.
Run the file closer method (previously tested)
Create an "temp expected results" temp directory (or clean up the one from last test)
Copy an "expected results" tarball from "expected_results" test sub-directory to the "temp expected results" temp directory created in last bullet point
untar that tarball in "temp expected results" temp directory and delete the tarball from there.
directory-diff the "temp expected results" temp directory with "actual results" temp directory (e.g. ensure both have 100% identical list of files and that each file's contents are 100% the same, either via native Perl or using diff via system() calls.
Since the logic above is very generic, I generally abstract most of it away into a "Test::FileGenerator" module re-used by all of the unit and integration tests that test file generation ability.
Related
I'm looking for a way to automate archiving where after I plug my two external drives I can copy all my resources. The problem is that I have different file structures on my laptop and on both external drives so I need to select specific folders to be copied. It means that I can't select one root folder and copy it straightforward. I tried to find a way to declare more than one path in the cp command and in the copy command, without success. An example path:
/my_programming_stuff
/folder1
/folder2
/folder3
/folder4
I want to select only the first 3 folders to copy them into external drive1 and external drive 2. The idea is to create a .bat file that will copy everything at once ( in the best case scenario it will be copied simultaneously on both external drives, so it will be much faster). Another problem is that there needs to be a bypass the ntfs long path limitations (max. 260 characters).
Flags that I want to use:
Copy the files and directories and all of their attributes,
including ownerships and permissions.
Recursively copy directories and their contents.
When copying files from one directory to another, only
copy files that either doesn't exist or are newer than the
existing corresponding files, in the destination
directory.
data verification (so it's certain that the copy was verified)
progression bar with time eta
Until now I was using Total Commander to do this but every day I need to pick only a few folders to be copied which takes time and is inefficient.
I have experience with Bash and PowerShell but I am not sure how to handle this topic.
Create a static batch file with robocopy commands. I think /copyall is the only switch you need to specify for all this. Other defaults should satisfy requirements.
https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy
I think your time will be better spent learning how to use either FastCopy or FreeFileSynce. I used FreeFileSync some years ago but got disgusted with the it's constantly changing format of its xml file used for starting a backup, so I switched to FastCopy. But it looks like FreeFileSync may be getting their act together and I aim to do some experiments over the summer to see if I want to switch back to it.
Both can handle the long filename format issues, both can be executed by a batch file, both seem to have a lot of quality, but FreeFileSync has more features - and more bloated because of the features. But speed wise, I think FastCopy is probably one of the better products out there and very streamline in use and design.
Every time I simulate in Dymola, a number of "useless" (for me) files are created in the working directory - i.e. dsfinal.txt, dsin.txt, dslog.txt, dsmodel.c, dymosim.exe. I find it annoying as it messes up my directory.
Is there a way to select only the desired output files to be kept after the simulations, without the need of manually deleting the undesired ones?
Those are temporary, but necessary files for Dymola. As far as I know there is no option to delete them automatically. Of course you could script that, but I don't see a real point to it and those files are used by some functionality - e.g. dsfinal.txt is used when as simulation is continued.
Some notes: Those files are created in the working directory - which should contain temporary files only. The working directory can be set via the GUI using File -> Options -> Settings:
A rather common problem is, that there is a Open and a Load function in Dymola:
As the description states, Load does not influence the working directory, whereas Open sets it to the directory from which a file is opened. The latter is also true for opening files e.g. via a double-click from the explorer. So usually it is better to go with Load.
My advice would be to separate the directories in which models/packages are stored and the working directory. This way the working directories content can be fully deleted basically anytime...
I have a batch program written in Matlab, which will be deployed to a production system by packaging it using the Compiler Toolbox.
When starting the program, multiple config and data files will be loaded, where the first config file contains paths and filenames for the other files to be loaded. I have a logger in the form of a hidden global object, and its static functions allow to add to the log. Every line will be immediately written to file (Performance does not matter, but reliable logging does!).
I want to dynamically specify the location of the logfile in a text file, but I don't know where I should put that file. In the classic interpreter mode, the wd of the constructor within the loggers' classdef seems to be directory where the file is saved, but I don't really understand why.
What is a reliable way of finding a textfile in the same directory as my classdef in a Compiler-packaged program using relative paths?
Thanks!
Hi im new to programming, how do i import a .txt file? My code cant find the file, is there any specific directory it has to be put into?
My code:
object Zettel01 extends App {
import scala.io.Source
object Suchtest {
val gesch = Source.fromFile("DieUnendlicheGeschichte.txt").getLines()
for (w <- gesch) println(w)
}
}
I have tried different code but the problem is always the same, i cant find the .txt file...
Thanks in advance for any help
Flurry1337
Every Scala program that you run on your computer is ultimately a java process. This process will have a "working directory", just as every process on your computer does. By default, the working directory is the working directory of the process that started it, that is, the current directory of the shell or command-line interpreter at the time when you started your program.
Now, that means it is important to know how exactly you start your program. If you are using a command line and start your program in the fashion of java MyCoolProgram, then the current directory of the shell will become the working directory of the program. If you use an IDE like Eclipse or IntelliJ IDEA, those typically use the project folder of your IDE project as the working directory of the process that they start.
There is a simple way to find it out quickly: You can always print out the result of new java.io.File(".").getAbsolutePath(). This will print the full path to the working directory. For example, you can write a little Scala program like this:
object PrintWorkingDirectory extends App {
println(new java.io.File(".").getAbsolutePath())
}
and start it. On the console, you should find the full path of the program's working directory. If you put a file named "DieUnendlicheGeschichte.txt" in this directory, your program will find that file under exactly that file name.
Of course, you don't have to dump all your files into that one directory. You can make subdirectories in order to organize your files better. For example, you might put your file in a path like "resources/text/DieUnendlicheGeschichte.txt".
Finally, I would like to point out that there is also a different way to associate resource files with your program, and to load them. The idea is that you put the code (class files) as well as resources like texts, images, CSV files, XML files and the like into one big file. This would be a JAR file. You can then use ClassLoader to access resources inside the JAR file by a URL.
Explaining that process in detail is out of scope for this question; this is just dropping a couple of buzzwords that you (or other readers) can search for in case they want to look up a more elaborated process.
System.getProperty("user.dir") also tells you the working directory.
I am trying to find a way of monitoring directories in Perl, in particular the size of a directory, and upon detecting a change in directory size, perform a particular action.
The issue I have is with large files that require a noticeable amount of time to copy into this directory, i.e. > 100MB. What happens (in Windows, not Unix) is the system reserves enough disk space for the entire file, even though the file is still copying in progress. This causes problems for me, because my script will try to perform an action on this file that has not finished copying over. I can easily detect directory size changes in Unix via 'du', but 'du' in Windows does not behave the same way.
Are there any accurate methods of detecting directory size changes in Perl?
Edit: Some points to clarify:
- My Perl script is only monitoring a particular directory, and upon detecting a new file or a new directory, perform an action on this new file or directory. It is not copying any files; users on the network will be copying files into the directory I am monitoring.
- The problem occurs when a new file or directory appears (copied, not moved) that is significantly large (> 100MB, but usually a couple GB) and my program fires before this copy completes
- In Unix I can easily 'du' to see that the file/directory in question is growing in size, and take the appropriate action
- In Windows the size is static, so I cannot detect this change
- opendir/readdir/closedir is not feasible, as some of the directories that appear may contain thousands of files, and I want to avoid the overhead of
Ideally I would like my program to be triggered on change, but I am not sure how to do this. As of right now it busy waits until it detects a change. The change in file/directory size is not in my control.
You seem to be working around the underlying issue rather than addressing it -- your program is not properly sending a notification when it is finished copying a file. Why not do that instead of using OS-specific mechanisms to try to indirectly determine when the operation is complete?
You can use Linux::Inotify2 or Win32::ChangeNotify to detect directory/file changes.
EDIT: File::ChangeNotify seems a better option (cross-platform & used by Catalyst)
As I understand it, you are polling a directory with thousands of files. When you see a new file, there is an action that is taken on the file. This causes problems if the file is in use or still being copied, correct?
There are potentially several solutions:
1) Use flock to detect if the file is still in use by another process (test if it works properly on your OS, file system, and Perl version).
2) Use a LockFile call on Windows. If it fails, the OS or another process is using that file.
3) Change the poll interval to a non busy time on the server and take the directory off line while your process completes.
Evaluating the size of a directory is something all but the most inexperienced Perl programmers should be able to do. You can write your own portable version of du in 15 lines of code if you know about:
Either glob or opendir / readdir / closedir to iterate through the files in a directory
The filetest operators (-f file, -d file, etc.) to distinguish between regular files and directory names
The stat function or file size operator -s file to obtain the size of a file
There is a nice module called File::Monitor, it will detect new files, deleted files, changes in size and any other attribute that can be done with stat. It will then go and out put the files for you.
http://metacpan.org/pod/File::Monitor
You set up a baseline scan, then set up a call back for each item you are looking for, so new changes you can see via
$monitor->watch( {
name => 'somedir',
recurse => 1,
callback => {
files_created => sub {
my ($name, $event, $change) = #_;
# Do stuff
}
}
} );
If you need to go deeper than one level just do it to whatever level you need. After this is done and it finds new files you can trigger you application to do what you want on the files.