I am trying to mirror a site index similar to
https://index.gd.workers.dev/
using wget. I run this code
wget -e robots=off --content-on-error --mirror -np -R "index.html*" https://index.gd.workers.dev/
It's giving this output:
--2020-03-11 01:20:05-- http://index.gd.workers.dev/ Resolving index.gd.workers.dev (index.gd.workers.dev)... 104.31.87.133,
104.31.86.133, 2606:4700:3035::681f:5785, ... Connecting to index.gd.workers.dev (index.gd.workers.dev)|104.31.87.133|:80...
connected. HTTP request sent, awaiting response... 200 OK Length: 361
[text/html] Saving to: ‘index.gd.workers.dev/index.html.tmp’
index.gd.workers.de 100%[===================>] 361 --.-KB/s in
0s
Last-modified header missing -- time-stamps turned off. 2020-03-11
01:20:05 (2.1 MB/s) - ‘index.gd.workers.dev/index.html.tmp’ saved
[361/361]
Removing index.gd.workers.dev/index.html.tmp since it should be
rejected.
FINISHED --2020-03-11 01:20:05-- Total wall clock time: 0.1s
Downloaded: 1 files, 361 in 0s (2.1 MB/s)
The end result is just a folder with the site name and nothing inside it. What am I doing wrong here? and is there any other way to mirror the directory?
Related
I'm trying to understand the following, maybe one of you guys can help me out by explaining on what exactly happens here:
My goal is to write a script, which copies files from the find-command, parallels rsync-commands and backup those, by re-creating the folder-structure from the source on the destination as well.
Somehow my initial script does not work like that, and i don't really have an idea on how to fix it to behave that way.
Initially, i've copied the files with "ls -1 /foldername" as sourceFolder, which is not best-practice. So i've tried to change it to using "find" as described beyond.
find "$sourceFolder" -maxdepth 2 -mindepth 2 -print0 | xargs --verbose -0 -I {} -P $maxThreads -n 1 rsync -valhP {}/ "$destFolder"/ --checksum --human-readable --stats --dry-run >> "$logDir"/result.log
--
If i run this script, it literaly would recopy the file(s), although those exist in the destination folder i would expect it to be.
sending incremental file list
.d..tp..... ./
>f+++++++++ macs.txt
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 242 bytes
Total transferred file size: 242 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.484 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 83
Total bytes received: 22
So as it turned out, sourceFolder and destFolder have to match, and i can't run anything like sourceFolder/folder1/folder1a destFolder/.
Can anyone help me out on what's wrong and why it behaves like that?
Thanks a Lot,
M.
As part of CI pipeline on ADO, I make REST API GET calls to get a list of requirement work items objects and a list of test results object. I sort and match the list and then I do multiple POST call to add information from that list as a attachment of my ADO test item object. Everything is done thought the ADO Matlab plugin task by using system function to execute curl through the command line via Matlab. Everything seem to work, I see the attachment on every test uploaded well and it even prints ''Done'' after all curl POST request to indicate, I'm done with my Matlab script.
I would expect the ADO CI job task to complete and pass to the next task in YAML file. But after running for 5 mins and completing everything, the job task stall and keep running for another 10h(Max ADO pipeline time)
The STDIO streams did not close within 10 seconds of the exit event from process 'C:\agent_work_tasks\RunMATLABCommand_28fdff80-51b4-4b6e-83e1-cfcf3f3b25a6\0.6.3\bin\run_matlab_command.bat'. This may indicate a child process inherited the STDIO streams and has not yet exited.
Any Ideas how to resolve this bug?
What I Tried:
I tried to run my script locally, I saw all my curl outputs via my matlab command window, but I saw that upon completion it opened a empty command prompt window.
looking for similar incidents and tried closing all processes(#4) as indicated here: https://developercommunity.visualstudio.com/t/the-stdio-streams-did-not-close-within-10-seconds/523146
I tried adding pause/wait in matlab script or via system command call to give time for asynchronous process to complete.
I added quit to my matlab script, and I tried using Taskkill/IM cmd.exe to kill all open command windows when my scripts ends.
Both didn't work on the CI pipeline. it still runs forever until i manually stop it or it reaches max time.
Error message & partial log:
2022-04-04T18:39:45.6947848Z curl -u :**ADOPAT** -X POST -H "Content-Type: application/json" -d "{\"stream\":\"***B64 encoded content***\", \"fileName\": \"requirementPath.txt\", \"comment\":\"Testattachmentupload\",\"attachmentType\":\"GeneralAttachment\"}" http://dev.azure.com/{organization}/{project}/_apis/test/Runs/**runID**/Results/**testResultID**/attachments?api-version=6.0-preview
2022-04-04T18:39:46.3140720Z
2022-04-04T18:39:46.3141125Z status =
2022-04-04T18:39:46.3141240Z
2022-04-04T18:39:46.3141367Z 0
2022-04-04T18:39:46.3141523Z
2022-04-04T18:39:46.3141598Z
2022-04-04T18:39:46.3141697Z cmdout =
2022-04-04T18:39:46.3141781Z
2022-04-04T18:39:46.3141992Z ' % Total % Received % Xferd Average Speed Time Time Time Current
2022-04-04T18:39:46.3142311Z Dload Upload Total Spent Left Speed
2022-04-04T18:39:46.3142510Z
2022-04-04T18:39:46.3142760Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
2022-04-04T18:39:46.3143126Z 100 354 100 139 100 215 269 417 --:--:-- --:--:-- --:--:-- 692
2022-04-04T18:39:46.3143506Z 100 354 100 139 100 215 269 416 --:--:-- --:--:-- --:--:-- 692
2022-04-04T18:39:46.3143948Z {"id":**attachementID**,"url":"http://dev.azure.com/{organization}/{project}/_apis/test/Runs/**RunID**/Results/**testResultID**/Attachments/**attachmentID**"}'
2022-04-04T18:39:46.3144224Z
2022-04-04T18:43:06.5720937Z done
2022-04-04T18:43:29.0053832Z ##[debug]Exit code 0 received from tool 'C:\agent\_work\_tasks\RunMATLABCommand_28fdff80-51b4-4b6e-83e1-cfcf3f3b25a6\0.6.3\bin\run_matlab_command.bat'
2022-04-04T18:43:39.0075174Z The STDIO streams did not close within 10 seconds of the exit event from process 'C:\agent\_work\_tasks\RunMATLABCommand_28fdff80-51b4-4b6e-83e1-cfcf3f3b25a6\0.6.3\bin\run_matlab_command.bat'. This may indicate a child process inherited the STDIO streams and has not yet exited.
2022-04-04T18:43:39.0076952Z ##[debug]The STDIO streams did not close within 10 seconds of the exit event from process 'C:\agent\_work\_tasks\RunMATLABCommand_28fdff80-51b4-4b6e-83e1-cfcf3f3b25a6\0.6.3\bin\run_matlab_command.bat'. This may indicate a child process inherited the STDIO streams and has not yet exited.
2022-04-05T04:10:04.2626714Z ##[debug]Re-evaluate condition on job cancellation for step: 'Matlab Rest API Test Update Action'.
2022-04-05T04:10:04.2919553Z ##[error]The operation was canceled.
2022-04-05T04:10:04.2932116Z ##[debug]System.OperationCanceledException: The operation was canceled.
Could someone please give me advice to make an openbmc image for Raspberrypi platform ?
Before I tried, I looked through related documents and believed an openbmc image can be worked on Raspberrypi.
Like OpenBMC with Raspberry Pi (2 or 3) and build bmcweb?
and https://kevinleeblog.github.io/project1/2019/11/25/openbmc-for-raspberry-pi-zero/.
So, I followed these instructions and tried the following steps.
#1: Git clone openbmc.git to my local PC.
tm#tm-VB1:~/Rpi4-64$ git clone https://github.com/openbmc/openbmc.git
Snip the logs but it looks no problem.
Receiving objects: 100% (182121/182121), 84.10 MiB | 5.55 MiB/s, done.
Resolving deltas: 100% (96860/96860), done.
#2: set TEMPLATECONF for raspberrypi
tm#tm-VB1:~/Rpi4-64$ export TEMPLATECONF=meta-evb/meta-evb-raspberrypi/conf
tm#tm-VB1:~/Rpi4-64$ echo $TEMPLATECONF
meta-evb/meta-evb-raspberrypi/conf
#3: set up the environment by "openbmc-env"
tm#tm-VB1:~/Rpi4-64/openbmc$ . openbmc-env
### Initializing OE build env ###
Snip the logs but it looks no problem. As you know, the script automatically creates a subdirectory,build, under openbmc.
Common targets are:
obmc-phosphor-image
tm#tm-VB1:~/Rpi4-64/openbmc/build$
#4: Change the directory and edit local.conf for my Raspberrypi platform.
tm#tm-VB1:~/Rpi4-64/openbmc/build$ cat ./conf/local.conf
Snip the log for unchanged part.
MACHINE ??= "raspberrypi4-64" <<< Change here for my platform.
DL_DIR ?= "/home/tm/Yocto/downloads" <<< Add here for build-time reduction at retry.
SSTATE_DIR ?= "/home/tm/Yocto/sstate-cache" <<< Add here for build-time reduction at retry.
#5: Change FLASH_SIZE variable based on the following sugestion. https://github.com/openbmc/openbmc/issues/3590
tm#tm-VB1:~/Rpi4-64/openbmc/meta-phosphor/classes$ cat image_types_phosphor.bbclass
Snip the log.
# Flash characteristics in KB unless otherwise noted
FLASH_SIZE ?= "131072" <<< I changed only this variable from 32768 to 131072.
#6: bitbake starts.
tm#tm-VB1:~/Rpi4-64/openbmc/bitbake obmc-phosphor-image
Then, ERROR happened.
ERROR: Logfile of failure stored in: /home/tm/Rpi/openbmc/build/tmp/work/raspberrypi-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2055074
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.09147 s, 367 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=495980 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/u-boot.bin
DEBUG: Spanning start=0K end=512K
DEBUG: Compare needed=495980 available=524288 margin=28308
484+1 records in
484+1 records out
495980 bytes (496 kB, 484 KiB) copied, 0.00120141 s, 413 MB/s
DEBUG: Considering file size=8266960 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi
DEBUG: Spanning start=512K end=4864K
>>>DEBUG: Compare needed=8266960 available=4456448 margin=-3810512
ERROR: Image '/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi' is too large!
DEBUG: Python function do_generate_static finished
It said margin=-3810512.
Now, my 2nd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 262144.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3915 tasks of which 2633 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc/build$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.177223 s, 189 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc/build$
It said margin=-23936.
OK. Image is too large. So,my 3rd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 9437184.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3935 tasks of which 0 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 4 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc$
tm#tm-VB1:~/Rpi4/openbmc$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.173685 s, 193 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc$
It said the same margin as 256MB case.
My 4th try.
I removed the whole openbmc directory and did the same steps above.
I changed MACHINE ??= "raspberrypi4-64" to "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 33554432.
It is the same result before.
My 5th try.
I removed the whole openbmc directory and did the same steps above.
I used MACHINE ??= "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 67108864.
It is the same result before.
After I tried several variations, it always said "image is too large" although I changed FLASH_SIZE to much much larger one.
So, I am wondering if I have missed some important configuration or it needs another parameter to fix this except FLASH_SIZE.
By the way, I tried romulus and made it.
My environment is ubuntu-20.04.2.0-desktop-amd64.
I really appreciate someone could kindly give me advice to make this work.
Interesting, I don't have a quick fix for you but I did notice the partition that is over sized is the uboot partition. The uboot is a smaller separate binary installed on the machine. It looks as if your uboot build is over 512k and the partition is set for 512k. Your flash size is massize
FLASH_SIZE = 9437184" that is more then a gig, (because FLASH_SIZE is in K)
If I were you I would first try to build an older version of openbmc for raspberry pi. (It used to work so you just need to find the commit before uboot grew to big). Use git to move back a month until you find it works.
If that does not work I would try to modify the partition table.
here is where you failing
this looks fine building the uboot image looks fine
increasing the kernel offset make if build, but the other targets in openbmc will not be happy with this solution. So maybe meta-raspberry-pi will have to override the partition table (if uboot can not be shrunk)
What ever you do, open an issue on the github and share you changes. Also use the discord, and gerrit.
I just replicated this issue. We should fix it
I cloned yay through git clone https://aur.archlinux.org/yay.git. I Enter the directory and run makepkg -sic but I unfortunately get the error:
==> Making package: yay 10.1.0-1 (Mon 26 Oct 2020 06:25:36 AM +0330)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
-> Downloading yay-10.1.0.tar.gz...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 118 100 118 0 0 97 0 0:00:01 0:00:01 --:--:-- 97
100 339k 100 339k 0 0 103k 0 0:00:03 0:00:03 --:--:-- 168k
==> Validating source files with sha256sums...
yay-10.1.0.tar.gz ... Passed
==> Extracting sources...
-> Extracting yay-10.1.0.tar.gz with bsdtar
==> Starting build()...
go build -v -trimpath -mod=readonly -modcacherw -ldflags '-s -w -extldflags "-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" -X "main.yayVersion=10.1.0" -X "main.localePath=/usr/share/locale/"' -buildmode=pie -o yay
go: github.com/Jguer/go-alpm/v2#v2.0.1: Get "https://gocenter.io/github.com/%21jguer/go-alpm/v2/#v/v2.0.1.mod": dial tcp 35.230.74.213:443: i/o timeout
make: *** [Makefile:127: yay] Error 1
==> ERROR: A failure occurred in build().
Aborting...
Are you getting the error consistently? If so, it looks like the host you are using to build does not have access to GoCenter.
exec: "gcc": executable file not found in $PATH
check prerequisites. see https://wiki.archlinux.org/index.php/Ar … Repository
But most of the time there is a problem with gcc, which can be solved in the following way :
pamac install base-devel
choose a gcc number from list
This is the first time I have every used make. I am trying to install the Julia language. So I cloned from GitHub
git clone git://github.com/JuliaLang/julia.git
The instructions then say enter in the Julia directory and type make. It ran for a very long time - I ate a pizza.
When I got back, typing Julia did not work. Towards the end of the installation, I got a long error message:
/usr/bin/install -c -m 644 libpcre.pc libpcreposix.pc libpcrecpp.pc '/home/john/Downloads/julia/usr/lib/pkgconfig'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 130 100 130 0 0 243 0 --:--:-- --:--:-- --:--:-- 337
0 0 0 8773k 0 0 310k 0 --:--:-- 0:00:28 --:--:-- 0
curl: (28) Operation too slow. Less than 1 bytes/sec transferred the last 15 seconds
curl: (6) name lookup timed out
make[2]: *** [openblas-v0.2.8.tar.gz] Error 6
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2
I tried sudo make - putting sudo in front seems to solve everything but not this:
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
make[2]: *** [openblas-v0.2.8/config.status] Error 2
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2
What steps can I take to makes sure Julia installs properly?
I need version 2.0 so I can use iJulia with my iPython notebook. If there is an easier way without compiling directly, I would just do that.
The problem is that the makefile is trying to download a file (curl is a command line program that acts like a web browser, and is often used to download files from websites).
However, for whatever reason (maybe the internet was tired) the download failed and timed out.
The reason it fails now with the unexpected end of file error is that (a) the download gave you part of a file before it failed, and (b) the makefile you're using is badly written so it didn't clean up the partly-downloaded file on failure.
So, that file exists and thus make won't try to download it, but it's only partial so when you try to uncompress it, it fails.
You should delete the file it tried to download by hand (with something like rm -f openblas-v0.2.8.tar.gz) then re-run make. Maybe the internet has woken up, or drunk some coffee, and the download will work this time.