About half the time when I click on WGET script following a CMIP6 data search on the ESGF (LLNL node), I get a wget script that only points to one, unrelated file. It's always the same one, too. Here's the relevant line that shows up in each wget file:
download_files="$(cat <<EOF--dataset.file.url.chksum_type.chksum
'famipc5_ne120_v0.3_00001_01_198001_198401_climo.nc' 'http://esgf.anl.gov/thredds/fileServer/esg_dataroot/ACME/climo/amip/v0_3/atm/mon/native/ne120/ens1/famipc5_ne120_v0.3_00001_01_198001_198401_climo.nc' 'SHA256' 'e5040c5df9d080437418943f02a41e84712dbe1c4a69982447712d7c7334241d'
EOF--dataset.file.url.chksum_type.chksum
)"
This happens with a wide variety of datasets. Here's one file where that happens, for example:
CMIP6.CMIP.CCCma.CanESM5.amip.r1i1p1f1.day.pr.gn
I've been searching for a reason, so far without success. A workaround is to hit the "download HTML" button 1000 times for each individual needed file instead (or set up a Globus endpoint for the files where that's possible), but it's very inconvenient and doesn't provide the functionality of a bash script.
Does anyone know what may be causing this? Is there some sort of limit to how many wget scripts an ESGF user can generate per day and these are downloaded as placeholders afterwards instead?
Grateful for any insight!
PS: I apologize for the cdo tag; I know this isn't a cdo problem, but it's hard to find relevant tags for this, and I figured that community may know what's up.
Turns out this is a browser issue. Repeating the search with chrome fixed it.
(Also that stack overflow may have not been the right venue to post this question; but I want this to be searchable somewhere at least)
Related
I often have to move a large number of files from one part of sharepoint to another, and the GUI often has issues with data loss or duplication, as well as being extremely repetitive and time intensive.
Ideally I'd really like there to be a way to just navigate sharepoint files as if they were just any other files on a command line in a computer file system. Is such a thing even possible? If not, is there at least a way to cp files from one directory into another?
Things like these make it seem easy, except the file paths don't actually match up in reality to any expected path:
https://www.sharepointdiary.com/2018/03/sharepoint-online-move-files-using-powershell.html
When attempting to use the SPO or PNP module in powershell, the documentation is pretty unclear. Get-PNPfile either always returns file not found if I try to use /Documents/Foldername like one would think. Even if I right click and copy link and get that messy url and make sure to deal with the ampersands, it still doesn't work. For example
Get-PnPFile -Url "https://domain.sharepoint.com/sites/team/Documents/file"
I would expect this to well, return an object that contains something pointing to the file, but it never works.
One possibility is that MFA is a requirement in the environment I'm using and it seems requires a flag -UseWebLogin which appears to work without errors, but it also appears to work when I mistyped the url of the team name when I used the command Connect PnPOnline, so maybe there is an issue there?
First for MFA, it will be better to user Connect-PnPOnline -Url "https://domain.sharepoint.com" -Interactive just like the comment on your original post.
And for the file not found error, it seems you are not using the correct URL. Have a try on the site relative URL.
Here is what I have tested
So, the solution ended up being the use of the "sync" feature which then creates an alias that can be manipulated as if it was a regular file on a machine using powershell. There doesn't seem to be a straightforward way to interact with the sharepoint filesystem via command line which is bonkers.
I am an absolute newbie as far as Powershell goes. I found a script here - https://www.litigationsupporttipofthenight.com/single-post/2020/04/19/powershell-script-to-count-words-lines-and-characters-in-multiple-pdfs - that I thought was exactly what I was looking for but when I run it, it seems that it sees my image based pdfs as just text files and says there are thousands of words/characters in them. I have a feeling that I am missing something.....I see various forum postings on the web regarding itextsharp and searching words in pdfs (no idea if this is it or not)?
Hoping someone can point me in the right direction (a specific example of this on the web somewhere) would be very much appreciated.
Gully
Is there a possibility to searching inside all JSON objects from all available responses in the network tab? Currently it works, but very randomly and isn't much reliable. Sometimes and especially in a smaller responses it's ok but when you have more assets almost always looking for, e.g. specific params value ends unsuccessfully. Do you know any smart solution of that issue? I've checked and first question associated with it has already few years and Google devs still haven't responded.
Example: I have object ID in response body, but cannot find it by search CTRL+F
I think one way is to save all the response in a file (manually or automatically, if possibile by using a browser extension).
After you have stored all the responses in a file you can parse the file and find things inside the file by using a script or just regex.
You can save the answers (as HAR file) manually (I use firefox) by right clicking on a network response inside the developer console panel.
I found that is the same for chrome.
Look here:
https://developers.google.com/web/tools/chrome-devtools/network/reference
I didn't search if there is a way to automatically store all the responses received by a browser. I'm not sure, but I think it isn't possible :/
I'm using wget to archive a discussion from a forum. The discussion is over several pages, navigated to with next and previous buttons.
I generated a list of the page urls and used that for the input-file, however the convert-links option is not converting the next and previous links, only the images.
Is there any way to make it do that?
I could use -r, but that would need a depth of 64 to get the whole discussion, and therefore it would get a whole load of extra unwanted stuff as well.
I figured out a workaround. It was easy enough to change the input file to html and upload it. Then with -r and -l1 it correctly converted the links.
I'm writing a script using Perl and Net::FTP, which is trying to upload a rather large file to the mainframe and I can't seem to get the primary/secondary allocations right.
I try to change them but it appears that, because the data sets are SMS-managed, they go to a different volume each time.
So I went looking the the quote site option which would (hopefully) lock the upload to a specific volume.
The current options I have are:
$ftp->site ("bl=10204 lr=170 rec=vb cy pri=100 sec=100")
and space is pretty tight on most volumes. However, I've located a volume with the required space but don't know how to specify that in the FTP options.
Is there a document somewhere detailing them? I couldn't find anything in the CommServer stuff nor did a Google search for ftp zos quote site turn up anything useful.
Alternatively, does anyone know the option to specify a fixed volume? Obviously I'd prefer the former since there may be other useful things in there but, at this point, I'd take the latter gratefully - I've darn near pulled out all my hair :-)
Log onto your mainframe's FTP using an interactive FTP program (e.g. FTP). Then type "quote help site" and you should get a good list of the options.
Just a little explanation... "site" is a command for the mainframe's FTP server. Typing "help" in front of it gets you help on the function. However, just typing "help site" will look for help on your FTP client (which probably never heard of the site command), so you have to pump the command to the server, which is what the "quote" command does.
BTW, your site command is asking for 100 primary and 100 secondary cylinders. That's a hefty chunk of DASD. Do you really need all that?
Good luck... let me know if you have any questions. I practically live on FTP on the mainframe and have written dozens (maybe hundreds) of Perl scripts to make things easier.
As a general FTP reference for z/os try Z/OS V1R7.0 Comm Svr: IP User's Guide and Commands.
Site commands can be found here.
Why not let SMS find the appropriate volume by using a DataClass appropriate for the file size?
How much hair did you start out with? Some can't afford to pull too much!