External command download progress output writes zillion lines in PowerShell remote session - powershell

Note: I do not know if this choco specific issue or not (probably not), but this is the only scenario when I can demonstrate it
Context
I use an external command (this case choco) in my PowerShell script. When the command displays progress (not with the PowerShell top [ooooo....]) then in the standard output in the last line the progress overwrites itself so the final output is always one line.
When I run the very same script in a remote context like this:
Invoke-Command -Session $remoteSession -ScriptBlock {
& "./myscript.ps1" # <-- This is the script where the external choco executes
}
then the local PowerShell output is flooded with zillion separate lines of one updating line like this:
Progress: 0% - Saving 128 KB of 43.81 MB
Progress: 0% - Saving 287.99 KB of 43.81 MB
Progress: 0% - Saving 432 KB of 43.81 MB
Progress: 1% - Saving 591.99 KB of 43.81 MB
Progress: 1% - Saving 751.98 KB of 43.81 MB
Progress: 2% - Saving 911.97 KB of 43.81 MB
This is not only disturbing, it overflows the command window buffer also, what I set the maximum 9999.
Question
Is it by design, or is there a setting something about new line or similar in remote case?

Related

Broken terminal output when running docker-compose with -T

I have a script that I use to start commands on some docker-compose containers.
The script runs docker-compose exec -T {container} {command}.
I have to use the -T (Disable pseudo-TTY allocation. By default docker compose exec allocates a TTY) option because otherwise my precommit hook, which also runs this script, errors with a panic: provided file is not a console.
However when I run the script on my own terminal I get a broken output that looks like this :
run v1.22.10
$ vite build
vite v2.7.13 building for production...
transforming...
✓ 378 modules transformed.
rendering chunks...
public/front/assets/Inter-Thin.77d96c1c.woff2 97.30 KiB
public/front/assets/Inter-ExtraLight.b6cd094a.woff2 101.79 KiB
public/front/assets/Inter-ThinItalic.d82beee8.woff2 104.00 KiB
public/front/assets/Inter-ExtraLightItalic.db229bf3.woff2 108.78 KiB
public/front/assets/Inter-Light.36b86832.woff2 101.89 KiB
public/front/assets/Inter-LightItalic.737ac201.woff2 108.72 KiB
public/front/assets/Inter-Regular.d612f121.woff2 96.55 KiB
public/front/assets/Inter-Medium.1b498b95.woff2 103.44 KiB
public/front/assets/Inter-Italic.900058df.woff2 104.37 KiB
public/front/assets/Inter-MediumItalic.81600858.woff2 109.55 KiB
public/front/assets/Inter-SemiBold.15226129.woff2 103.32 KiB
public/front/assets/Inter-SemiBoldItalic.3b6df7d0.woff2 109.42 KiB
public/front/assets/Inter-BoldItalic.3f211964.woff2 109.19 KiB
public/front/assets/Inter-Bold.c63158ba.woff2 103.65 KiB
public/front/assets/Inter-ExtraBoldItalic.cf6b1d6c.woff2 109.09 KiB
public/front/assets/Inter-ExtraBold.307d9809.woff2 103.62 KiB
public/front/assets/Inter-Black.fc10113c.woff2 100.46 KiB
public/front/assets/Inter-BlackItalic.bc80081d.woff2 106.20 KiB
public/front/assets/TRYVesterbro-Light.59d4e0df.woff2 134.08 KiB
public/front/assets/TRYVesterbro-Regular.cf9a26a9.woff2 139.92 KiB
public/front/assets/TRYVesterbro-Medium.116a42a4.woff2 142.17 KiB
public/front/assets/TRYVesterbro-Bold.ac6caaee.woff2 144.62 KiB
public/front/assets/TRYVesterbro-ExtraBold.9be0564d.woff2 142.90 KiB
Without -T the output is fine (every line is aligned correctly) but the hook will not work with the previously stated error.
If that's relevant I use zsh for my shell but I tried running in bash and it doesn't solve the issue either.
Is there any way to fix the terminal output while retaining -T in docker-compose ?
So after investigating I found that the output only contained \n and not \n\r. The missing \r is the reason the output skips a line but doesn't go back to the start (carriage return).
I made a simple bash function that I added to my .zshrc (you can add it to your .bashrc too).
clout() {
$# | sed 's/$/\r/'
}
Use it like this
clout your_command args
This function pipes the output of your command and adds a \r when it sees a \n.
This is the best solution I found. I am still open to better solutions if someone has one.

"image is too large" keeps on happening to openbmc image for Raspberrypi platform

Could someone please give me advice to make an openbmc image for Raspberrypi platform ?
Before I tried, I looked through related documents and believed an openbmc image can be worked on Raspberrypi.
Like OpenBMC with Raspberry Pi (2 or 3) and build bmcweb?
and https://kevinleeblog.github.io/project1/2019/11/25/openbmc-for-raspberry-pi-zero/.
So, I followed these instructions and tried the following steps.
#1: Git clone openbmc.git to my local PC.
tm#tm-VB1:~/Rpi4-64$ git clone https://github.com/openbmc/openbmc.git
Snip the logs but it looks no problem.
Receiving objects: 100% (182121/182121), 84.10 MiB | 5.55 MiB/s, done.
Resolving deltas: 100% (96860/96860), done.
#2: set TEMPLATECONF for raspberrypi
tm#tm-VB1:~/Rpi4-64$ export TEMPLATECONF=meta-evb/meta-evb-raspberrypi/conf
tm#tm-VB1:~/Rpi4-64$ echo $TEMPLATECONF
meta-evb/meta-evb-raspberrypi/conf
#3: set up the environment by "openbmc-env"
tm#tm-VB1:~/Rpi4-64/openbmc$ . openbmc-env
### Initializing OE build env ###
Snip the logs but it looks no problem. As you know, the script automatically creates a subdirectory,build, under openbmc.
Common targets are:
obmc-phosphor-image
tm#tm-VB1:~/Rpi4-64/openbmc/build$
#4: Change the directory and edit local.conf for my Raspberrypi platform.
tm#tm-VB1:~/Rpi4-64/openbmc/build$ cat ./conf/local.conf
Snip the log for unchanged part.
MACHINE ??= "raspberrypi4-64" <<< Change here for my platform.
DL_DIR ?= "/home/tm/Yocto/downloads" <<< Add here for build-time reduction at retry.
SSTATE_DIR ?= "/home/tm/Yocto/sstate-cache" <<< Add here for build-time reduction at retry.
#5: Change FLASH_SIZE variable based on the following sugestion. https://github.com/openbmc/openbmc/issues/3590
tm#tm-VB1:~/Rpi4-64/openbmc/meta-phosphor/classes$ cat image_types_phosphor.bbclass
Snip the log.
# Flash characteristics in KB unless otherwise noted
FLASH_SIZE ?= "131072" <<< I changed only this variable from 32768 to 131072.
#6: bitbake starts.
tm#tm-VB1:~/Rpi4-64/openbmc/bitbake obmc-phosphor-image
Then, ERROR happened.
ERROR: Logfile of failure stored in: /home/tm/Rpi/openbmc/build/tmp/work/raspberrypi-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2055074
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.09147 s, 367 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=495980 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/u-boot.bin
DEBUG: Spanning start=0K end=512K
DEBUG: Compare needed=495980 available=524288 margin=28308
484+1 records in
484+1 records out
495980 bytes (496 kB, 484 KiB) copied, 0.00120141 s, 413 MB/s
DEBUG: Considering file size=8266960 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi
DEBUG: Spanning start=512K end=4864K
>>>DEBUG: Compare needed=8266960 available=4456448 margin=-3810512
ERROR: Image '/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi' is too large!
DEBUG: Python function do_generate_static finished
It said margin=-3810512.
Now, my 2nd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 262144.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3915 tasks of which 2633 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc/build$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.177223 s, 189 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc/build$
It said margin=-23936.
OK. Image is too large. So,my 3rd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 9437184.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3935 tasks of which 0 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 4 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc$
tm#tm-VB1:~/Rpi4/openbmc$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.173685 s, 193 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc$
It said the same margin as 256MB case.
My 4th try.
I removed the whole openbmc directory and did the same steps above.
I changed MACHINE ??= "raspberrypi4-64" to "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 33554432.
It is the same result before.
My 5th try.
I removed the whole openbmc directory and did the same steps above.
I used MACHINE ??= "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 67108864.
It is the same result before.
After I tried several variations, it always said "image is too large" although I changed FLASH_SIZE to much much larger one.
So, I am wondering if I have missed some important configuration or it needs another parameter to fix this except FLASH_SIZE.
By the way, I tried romulus and made it.
My environment is ubuntu-20.04.2.0-desktop-amd64.
I really appreciate someone could kindly give me advice to make this work.
Interesting, I don't have a quick fix for you but I did notice the partition that is over sized is the uboot partition. The uboot is a smaller separate binary installed on the machine. It looks as if your uboot build is over 512k and the partition is set for 512k. Your flash size is massize
FLASH_SIZE = 9437184" that is more then a gig, (because FLASH_SIZE is in K)
If I were you I would first try to build an older version of openbmc for raspberry pi. (It used to work so you just need to find the commit before uboot grew to big). Use git to move back a month until you find it works.
If that does not work I would try to modify the partition table.
here is where you failing
this looks fine building the uboot image looks fine
increasing the kernel offset make if build, but the other targets in openbmc will not be happy with this solution. So maybe meta-raspberry-pi will have to override the partition table (if uboot can not be shrunk)
What ever you do, open an issue on the github and share you changes. Also use the discord, and gerrit.
I just replicated this issue. We should fix it

Error with !runaway command

I am looking a dump file collected from production environment for high cpu usage. I ran !threadpool and !runaway command as follows
0:000> !ThreadPool
CPU utilization: 100%
Worker Thread: Total: 6 Running: 2 Idle: 4 MaxLimit: 32767 MinLimit: 4
Work Request in Queue: 0
--------------------------------------
Number of Timers: 8
--------------------------------------
Completion Port Thread:Total: 8 Free: 3 MaxFree: 8 CurrentLimit: 8 MaxLimit: 1000 MinLimit: 4
0:000> !runaway
ERROR: !runaway: extension exception 0x80004002.
"Unable to get thread times - dumps may not have time information"
I want to know what threads are consuming most cpu time but I cannot run !runaway command. Are there any other commands in sos, sosex or any other extension that could be helpful in this case?
You need a tool that adds the necessary information to the dump.
In WinDbg, the .dump command has the /mt MiniOption, which
Adds additional thread information to the minidump. This includes thread times, which can be displayed by using the !runaway extension or the .ttime (Display Thread Times) command when debugging the minidump.
(Emphasis: links in WinDbg)
The t option is included in the a option as well, so .dump /ma is fine as well.
To find out whether or not your dump has that information, use the undocumented command .dumpdebug like this:
.shell -ci ".dumpdebug" findstr "MiniDump"
If there is a line
1000 MiniDumpWithThreadInfo
the information is contained and you have a different issue. If it's not there, the time info is not available.
Most other tools I know do not provide such detailed settings, so it's more or less luck, whether this info is included or not.

Mongodb build/compile error: not enough memory on Ubuntu

Preface so this isn't marked as a duplicate: I've seen lots of mongodb memory issues posted on stack overflow, but none that have to do with errors on the compilation.
I just freshly downloaded and ran Ubuntu on Virtualbox (on a mac), so I feel like there should be enough memory. However, when I try to compile Mongodb from the source code I've gotten the following errors about an hour into the compilation (I have done this a few times now)
scons: *** [<whatever file it was working on>] No space left on device
scons: building terminated because of errors
and on a separate occasion
IOError: [Errno 28] No space left on device:
File "/usr/lib/scons/SCons/Script/Main.py", line 1359:
_exec_main(parser, values)
File "/usr/lib/scons/SCons/Script/Main.py", line 1323:
_main(parser)
File "/usr/lib/scons/SCons/Script/Main.py", line 1072:
nodes = _build_targets(fs, options, targets, target_top)
File "/usr/lib/scons/SCons/Script/Main.py", line 1281:
jobs.run(postfunc = jobs_postfunc)
File "/usr/lib/scons/SCons/Job.py", line 113:
postfunc()
File "/usr/lib/scons/SCons/Script/Main.py", line 1278:
SCons.SConsign.write()
File "/usr/lib/scons/SCons/SConsign.py", line 109:
syncmethod()
File "/usr/lib/scons/SCons/dblite.py", line 117:
self._pickle_dump(self._dict, f, 1)
Exception IOError: (28, 'No space left on device') in <bound method dblite.__del__ of <SCons.dblite.dblite object at 0x7fbe2a577dd0>> ignored
I've tried both of the following build commands:
scons all --dbg=on -j1
scons --dbg=on -j1
According to VirtualBox the virtual size is 8 GB and the Actual size is 4.09 GB. Also, if it makes the difference, the odds that the memory on my mac is actually full is slim to none.
Any help would be greatly appreciated, thanks in advance.
EDIT: I've tried creating more memory (24 GB) and resizing partitions but I still cannot complete a build.
Here is the output of the df -T command:
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/sda1 ext4 15345648 14304904 238184 99% /
none tmpfs 4 0 4 0% /sys/fs/cgroup
udev devtmpfs 1014316 12 1014304 1% /dev
tmpfs tempfs 205012 860 204152 1% /run
none tempfs 5120 0 5120 0% /run/lock
none tempfs 1025052 152 1024900 1% /run/shm
none tempfs 102400 40 102360 1% /run/user
When you say memory, I believe you mean disk space. Try running the command
df -T to see what % usage you really have. You will probably need to resize the amount of space virtualbox has assigned to your image, as well as resize your repartition. It may be simpler to just create a new virtualbox image with 16 or 24GB of disk space.
If you decide to go the resize partition route, here is a helpful resource: https://askubuntu.com/questions/126153/how-to-resize-partitions

Process information in dump

I learnt that .tlist command in windbg dumps all the processes running in the system at the time of creating crash dump.
I would like to see the Memory Information of each process. So that it will help me to see if the system is over loaded by any specific process.
!process 0 1 will list all the processes and show memory related info for each. I issued this command using livekd and got all the processes. And here's my chrome process (which I picked out from the output):
PROCESS fffffa8007cb4200
SessionId: 1 Cid: 1158 Peb: 7efdf000 ParentCid: 0ff8
DirBase: 1b7962000 ObjectTable: fffff8a00addb010 HandleCount: 135.
Image: chrome.exe
VadRoot fffffa80090a6f80 Vads 169 Clone 0 Private 4037. Modified 3702. Locked 0.
DeviceMap 0000000000000000
Token fffff8a0091f9120
ElapsedTime 00:05:49.161
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (8020, 50, 345) (32080KB, 200KB, 1380KB)
PeakWorkingSetSize 10137
VirtualSize 144 Mb
PeakVirtualSize 151 Mb
PageFaultCount 66631
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 5784
Job fffffa8009822e30
Note memory related properties such as "Working Set Sizes", "Virtual Size", etc.
ps. Works with livekd and with system memory dumps (which I believe is what livekd does).
Marc
This information is not contained in process dump. .tlist queries your current system, not the state when the dump was taken. If you can take a system dump, than you can check out processes and their memory usage, as Marc Sherman already answered.