Broken terminal output when running docker-compose with -T - docker-compose

I have a script that I use to start commands on some docker-compose containers.
The script runs docker-compose exec -T {container} {command}.
I have to use the -T (Disable pseudo-TTY allocation. By default docker compose exec allocates a TTY) option because otherwise my precommit hook, which also runs this script, errors with a panic: provided file is not a console.
However when I run the script on my own terminal I get a broken output that looks like this :
run v1.22.10
$ vite build
vite v2.7.13 building for production...
transforming...
✓ 378 modules transformed.
rendering chunks...
public/front/assets/Inter-Thin.77d96c1c.woff2 97.30 KiB
public/front/assets/Inter-ExtraLight.b6cd094a.woff2 101.79 KiB
public/front/assets/Inter-ThinItalic.d82beee8.woff2 104.00 KiB
public/front/assets/Inter-ExtraLightItalic.db229bf3.woff2 108.78 KiB
public/front/assets/Inter-Light.36b86832.woff2 101.89 KiB
public/front/assets/Inter-LightItalic.737ac201.woff2 108.72 KiB
public/front/assets/Inter-Regular.d612f121.woff2 96.55 KiB
public/front/assets/Inter-Medium.1b498b95.woff2 103.44 KiB
public/front/assets/Inter-Italic.900058df.woff2 104.37 KiB
public/front/assets/Inter-MediumItalic.81600858.woff2 109.55 KiB
public/front/assets/Inter-SemiBold.15226129.woff2 103.32 KiB
public/front/assets/Inter-SemiBoldItalic.3b6df7d0.woff2 109.42 KiB
public/front/assets/Inter-BoldItalic.3f211964.woff2 109.19 KiB
public/front/assets/Inter-Bold.c63158ba.woff2 103.65 KiB
public/front/assets/Inter-ExtraBoldItalic.cf6b1d6c.woff2 109.09 KiB
public/front/assets/Inter-ExtraBold.307d9809.woff2 103.62 KiB
public/front/assets/Inter-Black.fc10113c.woff2 100.46 KiB
public/front/assets/Inter-BlackItalic.bc80081d.woff2 106.20 KiB
public/front/assets/TRYVesterbro-Light.59d4e0df.woff2 134.08 KiB
public/front/assets/TRYVesterbro-Regular.cf9a26a9.woff2 139.92 KiB
public/front/assets/TRYVesterbro-Medium.116a42a4.woff2 142.17 KiB
public/front/assets/TRYVesterbro-Bold.ac6caaee.woff2 144.62 KiB
public/front/assets/TRYVesterbro-ExtraBold.9be0564d.woff2 142.90 KiB
Without -T the output is fine (every line is aligned correctly) but the hook will not work with the previously stated error.
If that's relevant I use zsh for my shell but I tried running in bash and it doesn't solve the issue either.
Is there any way to fix the terminal output while retaining -T in docker-compose ?

So after investigating I found that the output only contained \n and not \n\r. The missing \r is the reason the output skips a line but doesn't go back to the start (carriage return).
I made a simple bash function that I added to my .zshrc (you can add it to your .bashrc too).
clout() {
$# | sed 's/$/\r/'
}
Use it like this
clout your_command args
This function pipes the output of your command and adds a \r when it sees a \n.
This is the best solution I found. I am still open to better solutions if someone has one.

Related

"image is too large" keeps on happening to openbmc image for Raspberrypi platform

Could someone please give me advice to make an openbmc image for Raspberrypi platform ?
Before I tried, I looked through related documents and believed an openbmc image can be worked on Raspberrypi.
Like OpenBMC with Raspberry Pi (2 or 3) and build bmcweb?
and https://kevinleeblog.github.io/project1/2019/11/25/openbmc-for-raspberry-pi-zero/.
So, I followed these instructions and tried the following steps.
#1: Git clone openbmc.git to my local PC.
tm#tm-VB1:~/Rpi4-64$ git clone https://github.com/openbmc/openbmc.git
Snip the logs but it looks no problem.
Receiving objects: 100% (182121/182121), 84.10 MiB | 5.55 MiB/s, done.
Resolving deltas: 100% (96860/96860), done.
#2: set TEMPLATECONF for raspberrypi
tm#tm-VB1:~/Rpi4-64$ export TEMPLATECONF=meta-evb/meta-evb-raspberrypi/conf
tm#tm-VB1:~/Rpi4-64$ echo $TEMPLATECONF
meta-evb/meta-evb-raspberrypi/conf
#3: set up the environment by "openbmc-env"
tm#tm-VB1:~/Rpi4-64/openbmc$ . openbmc-env
### Initializing OE build env ###
Snip the logs but it looks no problem. As you know, the script automatically creates a subdirectory,build, under openbmc.
Common targets are:
obmc-phosphor-image
tm#tm-VB1:~/Rpi4-64/openbmc/build$
#4: Change the directory and edit local.conf for my Raspberrypi platform.
tm#tm-VB1:~/Rpi4-64/openbmc/build$ cat ./conf/local.conf
Snip the log for unchanged part.
MACHINE ??= "raspberrypi4-64" <<< Change here for my platform.
DL_DIR ?= "/home/tm/Yocto/downloads" <<< Add here for build-time reduction at retry.
SSTATE_DIR ?= "/home/tm/Yocto/sstate-cache" <<< Add here for build-time reduction at retry.
#5: Change FLASH_SIZE variable based on the following sugestion. https://github.com/openbmc/openbmc/issues/3590
tm#tm-VB1:~/Rpi4-64/openbmc/meta-phosphor/classes$ cat image_types_phosphor.bbclass
Snip the log.
# Flash characteristics in KB unless otherwise noted
FLASH_SIZE ?= "131072" <<< I changed only this variable from 32768 to 131072.
#6: bitbake starts.
tm#tm-VB1:~/Rpi4-64/openbmc/bitbake obmc-phosphor-image
Then, ERROR happened.
ERROR: Logfile of failure stored in: /home/tm/Rpi/openbmc/build/tmp/work/raspberrypi-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2055074
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.09147 s, 367 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=495980 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/u-boot.bin
DEBUG: Spanning start=0K end=512K
DEBUG: Compare needed=495980 available=524288 margin=28308
484+1 records in
484+1 records out
495980 bytes (496 kB, 484 KiB) copied, 0.00120141 s, 413 MB/s
DEBUG: Considering file size=8266960 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi
DEBUG: Spanning start=512K end=4864K
>>>DEBUG: Compare needed=8266960 available=4456448 margin=-3810512
ERROR: Image '/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi' is too large!
DEBUG: Python function do_generate_static finished
It said margin=-3810512.
Now, my 2nd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 262144.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3915 tasks of which 2633 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc/build$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.177223 s, 189 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc/build$
It said margin=-23936.
OK. Image is too large. So,my 3rd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 9437184.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3935 tasks of which 0 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 4 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc$
tm#tm-VB1:~/Rpi4/openbmc$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.173685 s, 193 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc$
It said the same margin as 256MB case.
My 4th try.
I removed the whole openbmc directory and did the same steps above.
I changed MACHINE ??= "raspberrypi4-64" to "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 33554432.
It is the same result before.
My 5th try.
I removed the whole openbmc directory and did the same steps above.
I used MACHINE ??= "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 67108864.
It is the same result before.
After I tried several variations, it always said "image is too large" although I changed FLASH_SIZE to much much larger one.
So, I am wondering if I have missed some important configuration or it needs another parameter to fix this except FLASH_SIZE.
By the way, I tried romulus and made it.
My environment is ubuntu-20.04.2.0-desktop-amd64.
I really appreciate someone could kindly give me advice to make this work.
Interesting, I don't have a quick fix for you but I did notice the partition that is over sized is the uboot partition. The uboot is a smaller separate binary installed on the machine. It looks as if your uboot build is over 512k and the partition is set for 512k. Your flash size is massize
FLASH_SIZE = 9437184" that is more then a gig, (because FLASH_SIZE is in K)
If I were you I would first try to build an older version of openbmc for raspberry pi. (It used to work so you just need to find the commit before uboot grew to big). Use git to move back a month until you find it works.
If that does not work I would try to modify the partition table.
here is where you failing
this looks fine building the uboot image looks fine
increasing the kernel offset make if build, but the other targets in openbmc will not be happy with this solution. So maybe meta-raspberry-pi will have to override the partition table (if uboot can not be shrunk)
What ever you do, open an issue on the github and share you changes. Also use the discord, and gerrit.
I just replicated this issue. We should fix it

why when i want to push my repo i have 40209 object pushed [duplicate]

Here is an example:
$ git push -u myserver master
Counting objects: 22, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (14/14), done.
Writing objects: 100% (14/14), 1.89 KiB | 0 bytes/s, done.
Total 14 (delta 10), reused 0 (delta 0)
To myserver.com:git/myrepo.git
ed46414..0cae272 master -> master
Branch master set up to track remote branch master from myserver.
Basically the only number above that makes any sense to me is the 8 in "using up to 8 threads", because I have a quad-core processor with hyperthreading, therefore I can run 8 threads.
What are there 22 of? Why does it say 22 and then 14, what happened to the other 8? Why does it say 0 bytes/s, given that it did stuff and took finite time? What is "total" and "delta" and "reused"?
Details about the example: The example above is copy-pasted from Terminal on Mac OS X. I manually did a find-replace to substitute in "myrepo" and "myserver.com", everything else is verbatim. The repository has 910 commits, and I made about 3 commits since the prior push. The 3 new commits affected at least 3 files. The repo contains over a thousand files.
Short answer
This is merely the output of the git command git count-objects -v for the push (the same command is called for output when gc, pull and clone). More info in the man pages : git-count-objects(1).
$ git count-objects -v
...
size: 14 # The "Compressing objects: 100% (14/14)" part (the size in KiB)
in-pack: 22 # The "Counting objects: 22" part (the number of objects)
...
Long answer
Counting objects: 22, done.
This is git 22 internal objects being counted for that specific commit. Pretty much everything in git is an object, and are basically blobs saved in your .git/objects folder under their respective hash. More info in the man pages : 9.2 Git Internals - Git Objects.
Compressing objects: 100% (14/14), done.
This is git compressing the objects before send. The 14/14 is the progression in KiB of the compression (14 KiB to compress).
Writing objects: 100% (14/14), 1.89 KiB | 0 bytes/s, done.
This is git sending (if remote) and writing the objects. The 1.89 KiB | 0 bytes/s is the progression in KiB and the speed (0 bytes/s when finished).
Total 14 (delta 10), reused 0 (delta 0)
This is the output of the packfile algorithm in git (see 9.4 Git Internals - Packfiles) and is fairly obscure. It basically packs the unused objects, typically older history, in .git/objects/pack. After packing, git checks if it can reuse packs (hence the reused 0 part). The delta 0 part is the gain in KiB from the packing or from the reuse.
Git is a content addressable file system. i.e., it takes an object(file, tree, commit) and stores it in files addressable by hashes.
Suppose you make a very small change in the file. Should git store the full file as a different object? Well it does. But occasionally(during push, pull), git computes file changes as deltas and store them instead of full files.
That means, the most recent version of file is stored in full (since it should be available most readily), older version of the same file is just an object containing the difference between the two and so on.
This way git saves space while still able to reconstruct the file for any revision you throw at it.
Now coming to your question:
Counting objects: 22, done.: Git is counting the object related to your commits which you are pushing.
Total 14 (delta 10): Git was able to reduce the number of objects by finding 10 deltas.
reused 0 (delta 0): Git can reuse the delta objects if same exists already. For example if the similar changes might have been introduced in some other file, the delta may be similar and reusable. Here, there was nothing to reuse.
Writing objects: 100% (14/14), 1.89 KiB | 0 bytes/s, done. Here Git is sending (or writing) the objects over the network, and you can see progress and speed statistics as it's doing that.
Hope this helps.

External command download progress output writes zillion lines in PowerShell remote session

Note: I do not know if this choco specific issue or not (probably not), but this is the only scenario when I can demonstrate it
Context
I use an external command (this case choco) in my PowerShell script. When the command displays progress (not with the PowerShell top [ooooo....]) then in the standard output in the last line the progress overwrites itself so the final output is always one line.
When I run the very same script in a remote context like this:
Invoke-Command -Session $remoteSession -ScriptBlock {
& "./myscript.ps1" # <-- This is the script where the external choco executes
}
then the local PowerShell output is flooded with zillion separate lines of one updating line like this:
Progress: 0% - Saving 128 KB of 43.81 MB
Progress: 0% - Saving 287.99 KB of 43.81 MB
Progress: 0% - Saving 432 KB of 43.81 MB
Progress: 1% - Saving 591.99 KB of 43.81 MB
Progress: 1% - Saving 751.98 KB of 43.81 MB
Progress: 2% - Saving 911.97 KB of 43.81 MB
This is not only disturbing, it overflows the command window buffer also, what I set the maximum 9999.
Question
Is it by design, or is there a setting something about new line or similar in remote case?

Memory not freed upon image.close() in Pillow

I am struggling to de-allocate from a Pillow Image.
If I run the following:
#profile
def do_image_things():
im = Image.open(foo.png')
im.close()
del im
Then I get the following output from Python's memory_profiler:
Line # Mem usage Increment Line Contents
================================================
8 33.645 MiB 33.645 MiB #profile
9 def do_image_things():
12 37.383 MiB 3.738 MiB im = Image.open(u'foo.png')
13 37.387 MiB 0.004 MiB im.close()
14 37.387 MiB 0.000 MiB del im
The im.close() call appears not to have de-allocated the memory that the Image.open() reserved. The is a bare-bones reproduction of an issue occurring in a large-scale image processing deployment that we've noticed memory problems in.
Has anyone been able to resolve this problem?
I'm running Pillow Version 5.0.0 and 2.7.14 on Mac OS X.

What uses the memory on raspberry pi?

On my pi after start there is no free memory, but i can not found, waht uses it:
pi#node1 ~ $ cat /proc/cpuinfo
processor : 0
model name : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS : 2.00
Features : half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xb76
CPU revision : 7
Hardware : BCM2708
Revision : 0013
Serial : 00000000bf2e5e5c
pi#node1 ~ $ uname -a
Linux node1 4.0.7+ #801 PREEMPT Tue Jun 30 18:15:24 BST 2015 armv6l GNU/Linux
pi#node1 ~ $ head -n1 /etc/issue
Raspbian GNU/Linux 7 \n \l
pi#node1 ~ $ grep MemTotal /proc/meminfo
MemTotal: 493868 kB
pi#node1 ~ $ grep "model name" /proc/cpuinfo
model name : ARMv6-compatible processor rev 7 (v6l)
pi#node1 ~ $ ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5
0.6 0.2 6244 2377 -bash
0.3 0.0 6748 2458 sort -k 1 -nr
0.3 0.0 4140 2457 ps -eo pmem,pcpu,vsize,pid,cmd
0.2 0.1 9484 2376 sshd: pi#pts/0
0.2 0.1 5600 2236 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 104:107
pi#node1 ~ $ free
total used free shared buffers cached
Mem: 493868 478364 15504 0 500 4956
-/+ buffers/cache: 472908 20960
Swap: 102396 116 102280
I am not a linux expert, but if I understand it right, there is just 15Mb free memory, but no task uses more than 0.6%. Than why is not there more free?
Memory is not exclusively allocated by Processes.
The bootloader and the init ram filesystem is stored in memory.
The kernel (could be very big) is loaded into memory.
The kernel reserve memory for it's processes. ps shows 0.0% for these system processes.
Driver allocate buffer memory
The graphics card needs memory
If you have not configured your swap space on a harddrive or SD card, it uses memory.
The network system allocates memory for unix sockets and shared memory.
100 processes with 0.1 % are 10%.
And, if you start a process and stop it not all of it memory will be released.
Try it. Show the memory usage with free. Start a process that need some memory. Stop the process and use free again. I would bet that there is more memory usage than before.
Edit
Here is an example of a pi with less memory usage. I have no problems running java on it. I have a WLAN Dongle and a original NOIR CAM installed.
I installed Raspbian Wheezy. I used a kernel that I compiled from sources:
> uname -a
Linux raspberrypi 3.18.14+ #2 PREEMPT Sun May 31 20:19:04 UTC 2015 armv6l GNU/Linux
> head -n1 /etc/issue
Raspbian GNU/Linux 7 \n \l
On this pi I can run java -version in an acceptable period of time.
time java -version
java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) Client VM (build 25.0-b70, mixed mode)
real 0m1.012s
user 0m0.800s
sys 0m0.190s
Here is my memory footprint
> free
total used free shared buffers cached
Mem: 380816 138304 242512 0 8916 96728
-/+ buffers/cache: 32660 348156
Swap: 102396 0 102396