It appears that most SWF files, if not all are actually swf "archives" containing compressed versions of themselves. I have seen that you can extract the file using a few tools
$ flasm -x player.swf
Flasm configuration file flasm.ini not found, using default values
player.swf successfully decompressed, 206239 bytes
$ 7z x player.swf
7-Zip [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
Processing archive: player.swf
Extracting player~.swf
Everything is Ok
Size: 206239
Compressed: 106427
However I was hoping to extract from these using something a little more "conventional", e.g. tar or gzip
Relevant quote from http://www.adobe.com/content/dam/Adobe/en/devnet/swf/pdf/swf_file_format_spec_v10.pdf
The header begins with a three-byte signature of either 0x46, 0x57, 0x53 (“FWS”); or 0x43, 0x57, 0x53 (“CWS”).
An FWS signature indicates an uncompressed SWF file;
CWS indicates that the entire file after the first 8 bytes (that is, after the FileLength field) was compressed by using the ZLIB open standard.
The data format that the ZLIB library uses is described by Request for Comments (RFCs) documents 1950 to 1952. CWS file compression is permitted in SWF 6 or later only.
Update In response to the comment, here's a little bash script that is a literal translation of what the above seems to describe:
#!/bin/bash
for swf in "$#"
do
signature=$(dd if="$swf" bs=1 count=3 2> /dev/null)
case "$signature" in
FWS)
echo -e "uncompressed\t$swf"
;;
CWS)
targetname="$(dirname "$swf")/uncompressed_$(basename "$swf")"
echo "uncompressing to $targetname"
dd if="$swf" bs=1 skip=8 2>/dev/null |
(echo -n 'FWS';
dd if="$swf" bs=1 skip=3 count=5 2>/dev/null;
zlib-flate -uncompress) > "$targetname"
;;
*)
{
echo -e "unrecognized\t$swf"
file "$swf"
} > /dev/stderr
;;
esac
done
Which you'd then run across a set of *.swf files (assume you saved it as uncompress_swf.sh):
uncompress_swf.sh /some/folder/*.swf
It will say stuff like
uncompressed /some/folder/a.swf
uncompressed /some/folder/b.swf
uncompressing to /some/folder/uncompressed_c.swf
If something didn't look like a flash file, at all, it will print an error to stderr.
DISCLAIMER This is just the way I read the quoted spec. I have just checked that using this script resulted in identical output as when I had used 7z x on the input swf.
Related
In my loop, I run a dbt command and save the output to a .yml file. The following command works and generates a schema in my .yml file accurately:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > test.yml
done
However, in the example above, I am saving the test.yml file in the root directory. When I try to save the file in another path for example models/l30_mart/test.yml like this, it doesn't work:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/test.yml
done
In this case, when I open the test.ymlfile, I see this:
12:06:42 Running with dbt=1.0.1
12:06:43 Encountered an error:
Compilation Error
The schema file at models/l30_mart/test.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
https://docs.getdbt.com/docs/schemayml-files
What am I missing out on?
If I try something like this to save different files with the extracted tablename variable as the filename, it also doesn't work:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/$table.yml
done
In this case, the files either have this output:
20:39:44 Running with dbt=1.0.1
20:39:45 Encountered an error:
Compilation Error
The schema file at models/l30_mart/**firsttable.yml** is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
https://docs.getdbt.com/docs/schemayml-files
or this (eg in the secondtablename.yml file):
20:39:48 Running with dbt=1.0.1
20:39:49 Encountered an error:
Parsing Error
Error reading dbt_4flow: l30_mart/firstablename.yml - Runtime Error
Syntax error near line 2
------------------------------
1 | 20:39:44 Running with dbt=1.0.1
2 | 20:39:45 Encountered an error:
3 | Compilation Error
4 | The schema file at models/l30_mart/firsttablename.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
5 |
Raw Error:
------------------------------
mapping values are not allowed in this context
in "<unicode string>", line 2, column 31
Note that the secondtablename.yml mentions the firsttablename.yml.
I don't know dbt but the explanation that seems likely is that dbt for some reason parses all *.yml files in that target directory when you call it. Since the shell opens the pipe to the *.yml file before calling dbt, the file already exists (but initially empty) when dbt is called. Since dbt expects the file to contain a version, you get an error.
To check whether this assessment is correct, write into a temporary file:
for file in models/l30_mart/*.sql; do
target_file=$(mktemp)
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > $target_file
mv $target_file models/l30_mart/test.yml
done
(Be aware of mktemp shenanigans if you're using macOS)
Edit: Since dbt seems to be affected by the files existing, you can also try to generate all files and move them into the correct directory afterwards:
target_dir=$(mktemp -d)
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > $target_dir/$table.yml
done
mv $target_dir/*.yml models/l30_mart/
rmdir $target_dir
I am attempting to merge three ntuples (just an example but there are more) stored in a directory that are labeled as the following:
[1] mc16a_SUSY.root
[2] mc16d_SUSY.root
[3] mc16e_SUSY.root
[4] ......
To do this I am using the command hadd (hadd outputfile inputfiles..)
os.system(hadd -f Combined_SUSY_SAMPLES.root mc16*SUSY*.root)
For the output file I want to combine all files with mc16 and SUSY in the file name
But I receive the error:
hadd Target file: Combined_SUSY_SAMPLES.root
hadd compression setting for all output: 1
hadd Source file 1: mc16*SUSY*.root
Error in <TFile::TFile>: file mc16*SUSY*.root does not exist
Error in <TFileMerger::AddFile>: cannot open file mc16*SUSY*.root
hadd exiting due to error in mc16*SUSY*.root
It states that there aren't any files with mc16*SUSY*.root but these files do exist. Any solutions? Thanks for the help in advance.
I use the "*" because there are to many files to list them individually.
I'm late to respond here, but could it be that you don't have the full path to the files? Some databases will send you the file names with certain commands, but not the absolute locations. It could be that you need to append the folder to the beginning of the filename.
I am trying to use pgbadger to make html report for postgres slow query log files. My postgres logfiles are in csvlog format in folder pg_log. I transfer all logfiles
(80 files with 10 MB each) to my local windows machine and trying to generate single html report for all files. I created all one file from all files in below way,
type postgresql-2020-06-18_075333.csv > postgresql.csv
type postgresql-2020-06-18_080011.csv >> postgresql.csv
....
....
type postgresql-2020-06-18_094812.csv >> postgresql.csv
I downloaded "pgbadger-11.2" and tried below command but getting error.
D:\pgbadger-11.2>perl --version
This is perl 5, version 28, subversion 1 (v5.28.1) built for MSWin32-x64-multi-thread
D:\pgbadger-11.2>perl pgbadger "D:\June-Logs\postgresql.csv" -o postgresql.html
[========================>] Parsed 923009530 bytes of 923009530 (100.00%), queries: 1254764, events: 53
can not load incompatible binary data, binary file is from version < 4.0.
LOG: Ok, generating html report...
postgresql.html is created but no data in any tab.But it works when i create separate report for individual csv. like below.
D:\pgbadger-11.2>perl pgbadger "D:\June-Logs\postgresql-2020-06-18_075333.csv" -o postgresql-2020-06-18_075333.html
D:\pgbadger-11.2>perl pgbadger "D:\June-Logs\postgresql-2020-06-18_080011.csv" -o postgresql-2020-06-18_080011.html
...
D:\pgbadger-11.2>perl pgbadger "D:\June-Logs\postgresql-2020-06-18_094812.csv" -o postgresql-2020-06-18_094812.html
Please suggest me something to fix this issue.
I going to say this due to:
type postgresql-2020-06-18_075333.csv > postgresql.csv
type postgresql-2020-06-18_080011.csv >> postgresql.csv
Pretty sure that is introducing Windows line endings and pgBadger is looking for Unix line endings. Can you do the concatenate on the server?
UPDATE. Hmm. Ran across this
https://github.com/darold/pgbadger/releases
"This new release breaks backward compatibility with old binary or JSON
files. This also mean that incremental mode will not be able to read
old binary file [...] Add a warning about version and skip loading incompatible binary file.
Update code formatter to pgFormatter 4.0."
Not sure why it is failing on CSV logs, still what is version of pgBadger generating logs?
This may sound like a dumb question, but I'd like to know where is the .tex file saved, when I compile a pdf document from a Rmd file, using RStudio server.
I added the keep_tex option, so the header of Rmd looks like this :
---
output:
pdf_document:
keep_tex: yes
---
Then when I compiled, the output looks like this
|...................... | 33%
ordinary text without R code
|........................................... | 67%
label: plot
processing file: test.Rmd
cropping /tmp/Rtmpb1x3Q0/preview-3bfe24922427.dir/test_files/figure-latex/plot-1.pdf
PDFCROP 1.33, 2012/02/01 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `/tmp/Rtmpb1x3Q0/preview-3bfe24922427.dir/test_files/figure-latex/plot-1.pdf'.
|.................................................................| 100%
ordinary text without R code
/usr/lib/rstudio-server/bin/pandoc/pandoc test.utf8.md --to latex --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash-implicit_figures --output /tmp/Rtmpb1x3Q0/preview-3bfe24922427.dir/test.tex --template /home/myusername/R/x86_64-pc-linux-gnu-library/3.1/rmarkdown/rmd/latex/default.tex --highlight-style tango --latex-engine pdflatex --variable 'geometry:margin=1in'
output file: test.knit.md
/usr/lib/rstudio-server/bin/pandoc/pandoc test.utf8.md --to latex --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash-implicit_figures --output /tmp/Rtmpb1x3Q0/preview-3bfe24922427.dir/test.pdf --template /home/myusername/R/x86_64-pc-linux-gnu-library/3.1/rmarkdown/rmd/latex/default.tex --highlight-style tango --latex-engine pdflatex --variable 'geometry:margin=1in'
Output created: /tmp/Rtmpb1x3Q0/preview-3bfe24922427.dir/test.pdf
I'd like to find the intermediate .tex file (or test.knit.md), and do a bit of editing. Except it is no where to be found. Not in the working directory, or /home/myusername/, or /, or /tmp/Rtmpb1x3Q0/.
I'd really appreciate it if someone has the answer.
Actually, this first line in the console
/usr/lib/rstudio-server/bin/pandoc/pandoc test.utf8.md --to latex --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash-implicit_figures --output /tmp/Rtmpb1x3Q0/preview-3bfe24922427.dir/test.tex --template /home/myusername/R/x86_64-pc-linux-gnu-library/3.1/rmarkdown/rmd/latex/default.tex --highlight-style tango --latex-engine pdflatex --variable 'geometry:margin=1in'
told us that the output .tex file is in /tmp/Rtmpb1x3Q0/preview-3bfe24922427.dir/test.tex
Somehow I did not find the file last time, but on a recent instance, the .tex file is actually there, so that answers the question.
The initial code from the question
---
output:
pdf_document:
keep_tex: yes
---
Throws an error for me, while the following does not:
---
output:
pdf_document: default
keep_tex: T
---
However, I was still unable to find the .tex file following the console output. It appears to still not be saved. Instead what worked easily was running the following lines in the R studio console:
#install.packages(rmarkdown)
rmarkdown::render("FileName.Rmd", output_format = latex_document())
The file "FileName.Rmd" needs to be in the current working directory - which is where the .tex file will be saved.
So, site that used to use FTP now has an HTTP front-end and won't allow FTP connections. The site in question (for an example directory) will show a page with links to different dates. Inside each of these different dates, there are many files, and I typically just need to get some file with some clear pattern e.g. *h17v04*.hdf. I thought this could work:
wget -I "${PLATFORM}/${PRODUCT}/${YEAR}.*" -r -l 4 \
--user-agent="Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" \
--verbose -c -np -nc -nd \
-A "*h17v04*.hdf" http://e4ftl01.cr.usgs.gov/$PLATFORM/$PRODUCT/
where PLATFORM=MOLT, PRODUCT=MOD09GA.005 and YEAR=2004, for example. This seems to start looking into all the useful dates, finds the index.html, and then just skips to the next directory, without downloading the relevant hdf file:
--2013-06-14 13:09:18-- http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.01/
Reusing existing connection to e4ftl01.cr.usgs.gov:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.01/index.html'
[ <=> ] 174,182 134K/s in 1.3s
2013-06-14 13:09:20 (134 KB/s) - `e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.01/index.html' saved [174182]
Removing e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.01/index.html since it should be rejected.
--2013-06-14 13:09:20-- http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.02/
[...]
If I ignore the -A option, only the index.html file is downloaded to my system, but it appears it's not parsed and the links are not followed. I don't really know what more is required to make this work, as I can't see why it doesn't!!!
SOLUTION
In the end, the problem was due to an old bug in the local version of wget. However, I ended up writing my own script for downloading MODIS data from the server above. The script is pure Python, and is available from here.
Consider to use pyModis instead of wget which is a Free and Open Source Python based library to work with MODIS data. It offers bulk-download for user selected time ranges, mosaicking of MODIS tiles, and the reprojection from Sinusoidal to other projections, convert HDF format to other formats. See
http://www.pymodis.org/