I'm trying to download a page from a website with the wget command in Linux, what I want to get is the information contained in the html code of the page, for instance.
I've tried using R's Rvest and Python's beautiful soup for it but it didn't work, so now I'm trying with wget and mostly I get scripts in .css and .js which I can't extract useful information from (as far as I know), so to summarize I would like to extract information of a from a dynamic site using the wget command.
I've used the arguments -F, -i and some others, bear in mind that I'm fairly new to computer science and Linux in general, so if this question is trivial please explain it nonetheless, thanks.
Also, if necessary I can post what I've done up until now.
with VBA and https://phantomjs.org/
sDir = ThisWorkbook.Path
sCmd = "cmd.exe /K cd """ & sDir & """ & phantomjs.exe phm_save.js " & sURL & " > " & sFile
pid = Shell(sCmd, vbHide)
phm_save.js
var system = require('system');
var page = require('webpage').create();
page.open(system.args[1], function(){
console.log(page.content);
phantom.exit();
}
);
Related
Question about running a custom function in Powershell.
I'm on Windows 10 and I'd like to somehow print my monorepository's directory tree structure excluding node_modules. This is not supported out of the box but requires a custom function to be defined. I found one solution on StackOverflow (https://stackoverflow.com/a/43810460/9654273), which would enable using a command like:
tree -Exclude node_modules -Ascii > tree.txt
The problem is I don't know what to do with the provided source code :D The answer says "add to your $PROFILE, for instance", so I ran notepad $PROFILE in PowerShell, pasted the code snippet there, saved it and tried running the command. It didn't work because I did something wrong. According to the StackOverflow post's comments from anand_v.singh and mklement0 I was still running some other tree command, not the one I just attempted to define.
So how do I use a custom function in PowerShell? Starting point is that source code is on StackOverflow and I don't know where to paste it. Or do you know some other, easier way to print a directory tree on Windows 10 excluding node_modules?
I had the same problem with that function. The issue is the special characters in the hashtable at line 106:
$chars = #{
interior = ('├', '+')[$ndx]
last = ('└', '\')[$ndx] #'
hline = ('─', '-')[$ndx]
vline = ('│', '|')[$ndx]
space = ' '
}
I changed the special characters to ascii as follows:
$chars = #{
interior = ('+', '+')[$ndx]
last = ('\', '\')[$ndx] #'
hline = ('-', '-')[$ndx]
vline = ('|', '|')[$ndx]
space = ' '
}
The only downside is that you do not now have the option of using special graphics characters (the Ascii switch is still there, but does nothing). Maybe someone could tell us how to embed them properly.
i'm trying to use gettext to translate the string in my site
gettext doesn't have problem detecting strings such as
<? echo _("Donations"); ?>
or
<? echo _("Donate to this site");?>
but obviously, usually we'll use codes like this in our site
<? echo _("$siteName was developed with one thing in mind"); ?>
Of course in the website, the $siteName is displayed correctly as
My Website was developed with one thing in mind
if we put
$siteName = "My Website";
previously.
My problem is, i'm using poedit to extract all the strings in my codes that needs to be translated, and it seems poedit doesn't extract all string with php codes like I described above. So how do I get poedit extract strings with php code inside it too? Or is there any other tools I should use?
One possibility is to use sprintf. Just make sure you keep the percent (%) in the poedit string!
echo sprintf( _("This %s can be translated "), 'string');
Or when using multiple variables
echo vsprintf( _("This %s can be %s"), ['string', 'translated']);
I wrote a script analyzing perl-files (totally without PPI, because it will be used on Servers where the admins don't want PPI to be installed and so on and so forth, but let's not talk about that).
Now, let's say I have this code:
my $object = MySQL->new();
my $ob2 = $object;
$ob2->
(Where MySQL is one of our modules).
My script correctly identifies that $ob2 is a MySQL-Object and sees where it came from, and then returns a list of found subs in that module.
My idea was, that, since I use vim for editing, this could be a really cool way for "CTRL-n"-Completetion.
So, when...
$ob2->[CTRL-n]
It shows the CTRL-n-Box which opens my Perl-Script and gives it a few parameters (I would need: The line that I am actually on, the cursor position and the whole file as it is in vim).
I already found things like vim-perl, which allows me to write something like
if has('perl')
function DefPerl()
perl << EOF
use MyModule;
return call_to_my_function(); # returns all the methods from the object for example
EOF
endfunction
call DefPerl()
endif
But somehow this does not get executed (I tried writing something to a file with a system call just for the sake of testing)...
So, in short:
Does anyone here know how to achieve that? Calling a perl-function from vim by pressing CTRL-n with the full file-code and the line vim is actually in and the position, and then opening a completetion-menu with the results it got from the perl-script?
I hope someone knows what I mean here. Any help would be appreciated.
The details and tips for invoking embedded Perl code from Vim can be found in this Vim Tips Wiki article. Your attempts are already pretty close, but to return stuff from Perl, you need to use Vim's Perl API:
VIM::DoCommand "let retVal=". aMeaningfullThingToReturn
For the completion menu, your Perl code needs to return a List of Vim objects that adhere to the format as described by :help complete-items. And :help complete-functions shows how to trigger the completion. Basically, you define an insert-mode mapping that sets 'completefunc' and then trigger your function via <C-x><C-u>. Here's a skeleton to get your started:
function! ExampleComplete( findstart, base )
if a:findstart
" Locate the start of the keyword.
let l:startCol = searchpos('\k*\%#', 'bn', line('.'))[1]
if l:startCol == 0
let l:startCol = col('.')
endif
return l:startCol - 1 " Return byte index, not column.
else
" Find matches starting with a:base.
let l:matches = [{'word': 'example1'}, {'word': 'example2'}]
" TODO: Invoke your Perl function here, input: a:base, output: l:matches
return l:matches
endif
endfunction
function! ExampleCompleteExpr()
set completefunc=ExampleComplete
return "\<C-x>\<C-u>"
endfunction
inoremap <script> <expr> <Plug>(ExampleComplete) ExampleCompleteExpr()
if ! hasmapto('<Plug>(ExampleComplete)', 'i')
imap <C-x><C-z> <Plug>(ExampleComplete)
endif
I need to prepare list of strings for translation of my iPhone application.
I have extracted strings from *.m files using genstring and from the XIB files using ibtool command.
But I have also lots of texts to translate in plist files (String field types enclosed in string tag).
Is there a nice bash script / command to extract those strings into a flat txt file?
I could review and filter it so my translators can work with nice list but not with alien looking XML file.
I made a custom shell script which tries to figure out the values needed. You can then use the localize.py script in a modified way (see below) to automatically create the translation files. (The line break where somehow very important) If there more entities to be translated, the shell script can be modified accordingly
#!/bin/bash
rm -f $2
sed -n 'N;/<key>Title<\/key>/{N;/<string>.*<\/string>/{s/.*<string>\(.*\)<\/string>.*/\/* \1 *\/\
"\1" = "\1";\
/p;};}' $1 >> $2
sed -n 'N;/<key>FooterText<\/key>/{N;/<string>.*<\/string>/{s/.*<string>\(.*\)<\/string>.*/\/* \1 *\/\
\"\1" = "\1";\
/p;}
;}' $1 >> $2
sed -n 'N;/<key>Titles<\/key>/{N;/<array>/{:a
N;/<\/array>/!{
/<string>.*<\/string>/{s/.*<string>\(.*\)<\/string>.*/\/* \1 *\/\
\"\1" = "\1";\
/p;}
ba
;};};}' $1 >> $2
the localize.py script needed some modification. Therefore I created a small package containing the localizer for the source code and for the plist Files. The new script even supports Duplikates (meaning it will kick them)
We recently made a small online application to do that, please take a look on: http://www.icapps.be/plist-translator/
I can't think of any command off the top of my head. However, plists are glorified xml files and there are various parsers available for them.
It shouldn't be too difficult to create a simple python script to get all the strings from the file.
Does this help?
http://www.icanlocalize.com/site/tutorials/how-to-translate-plist-files/
We much prefer paying clients who use our translation system with our translators, but you can translate yourself in our GUI at no charge.
I am trying to create my first zsh completion script, in this case for the command netcfg.
Lame as it may sound I have stuck on the first hurdle, disclaimer, I know how to do this crudely, however I seek the "ZSH WAY" to do this.
I need to list the files in /etc/networking but only the files, not the directory component, so I do the following.
echo $(ls /etc/network.d/*(.))
/etc/network.d/ethernet-dhcp /etc/network.d/wireless-wpa-config
What I wanted was:
ethernet-dhcp wireless-wpa-config
So I try (excuse my naivity) :
echo ${(s/*\/)$(ls /etc/network.d/*(.))}
/etc/network.d/ethernet-dhcp /etc/network.d/wireless-wpa-config
It seems that this doesn't work, I'm sure there must be some clever way of doing this by splitting into an array and getting the last part but as I say, I'm complete noob at this.
Any advice gratefully received.
General note: There is no need to use ls to generate the filenames. You might as well use echo some*glob. But if you want to protect the possible embedded newline characters even that is a bad idea. The first example below globs directly into an array to protect embedded newlines. The second one uses printf to generate NUL terminated data to accomplish the same thing without using a variable.
It is easy to do if you are willing to use a variable:
typeset -a entries
entries=(/etc/network.d/*(.)) # generate the list
echo ${entries#/etc/network.d/} # strip the prefix from each one
You can also do it without a variable, but the extra stuff to isolate individual entries is a bit ugly:
# From the inside, to the outside:
# * glob the entries
# * NUL terminate them into a single string
# * split at NUL
# * strip the prefix from each one
echo ${${(0)"$(printf '%s\0' /etc/network.d/*(.))"}#/etc/network.d/}
Or, if you are going to use a subshell anyway (i.e. the command substitution in the previous example), just cd to the directory so it is not part of the glob expansion (plus, you do not have to repeat the directory name):
echo ${(0)"$(cd /etc/network.d && printf '%s\0' *(.))"}
Chris Johnsen's answer is full of useful information about zsh, however it doesn't mention the much simpler solution that works in this particular case:
echo /etc/network.d/*(:t)
This is using the t history modifier as a glob qualifier.
Thanks for your suggestions guys, having done yet more reading of ZSH and coming back to the problem a couple of days later, I think I've got a very terse solution which I would like to share for your benefit.
echo ${$(print /etc/network.d/*(.)):t}
I'm used to seeing basename(1) stripping off directory components; also, you can use echo /etc/network/* to get the file listing without running the external ls program. (Running external programs can slow down completion more than you'd like; I didn't find a zsh-builtin for basename, but that doesn't mean that there isn't one.)
Here's something I hope will help:
haig% for f in /etc/network/* ; do basename $f ; done
if-down.d
if-post-down.d
if-pre-up.d
if-up.d
interfaces