i was trying to import a large collection in mongodb but everytime i try to import
-rw-r--r-- 1 root root 3491368960 Jul 15 06:15 activity-june.json
mongoimport --db analytics --collection reports < activity-june.json
it gives me error like this:
fatal error: runtime: out of memory
goroutine 37 [running]:
runtime.throw(0xcba857)
/usr/local/go/src/pkg/runtime/panic.c:520 +0x69 fp=0x7fd4984fea68 sp=0x7fd4984fea50
runtime.SysMap(0xc308100000, 0x100000000, 0x42b700, 0xcd7eb8)
/usr/local/go/src/pkg/runtime/mem_linux.c:147 +0x93 fp=0x7fd4984fea98 sp=0x7fd4984fea68
runtime.MHeap_SysAlloc(0xce3ea0, 0x100000000)
/usr/local/go/src/pkg/runtime/malloc.goc:616 +0x15b fp=0x7fd4984feaf0 sp=0x7fd4984fea98
MHeap_Grow(0xce3ea0, 0x80000)
/usr/local/go/src/pkg/runtime/mheap.c:319 +0x5d fp=0x7fd4984feb30 sp=0x7fd4984feaf0
MHeap_AllocLocked(0xce3ea0, 0x80000, 0x0)...
heres my disk space:
-bash-4.1# swapon -s
Filename Type Size Used Priority
/dev/dm-1 partition 4128764 133416 -1
and heres my last query on my database, its not inserting anything.
> db.reports.find().count();
7
Kindly check on this
1) Your computer should be having 64 bit, if it 32 bit then we cannot import more than 2GB of data
2) Each of your document should not exceed 16MB
http://docs.mongodb.org/manual/reference/program/mongoimport/
also check this post,
https://forum.syncthing.net/t/fatal-error-runtime-out-of-memory/2190
Related
Could someone please give me advice to make an openbmc image for Raspberrypi platform ?
Before I tried, I looked through related documents and believed an openbmc image can be worked on Raspberrypi.
Like OpenBMC with Raspberry Pi (2 or 3) and build bmcweb?
and https://kevinleeblog.github.io/project1/2019/11/25/openbmc-for-raspberry-pi-zero/.
So, I followed these instructions and tried the following steps.
#1: Git clone openbmc.git to my local PC.
tm#tm-VB1:~/Rpi4-64$ git clone https://github.com/openbmc/openbmc.git
Snip the logs but it looks no problem.
Receiving objects: 100% (182121/182121), 84.10 MiB | 5.55 MiB/s, done.
Resolving deltas: 100% (96860/96860), done.
#2: set TEMPLATECONF for raspberrypi
tm#tm-VB1:~/Rpi4-64$ export TEMPLATECONF=meta-evb/meta-evb-raspberrypi/conf
tm#tm-VB1:~/Rpi4-64$ echo $TEMPLATECONF
meta-evb/meta-evb-raspberrypi/conf
#3: set up the environment by "openbmc-env"
tm#tm-VB1:~/Rpi4-64/openbmc$ . openbmc-env
### Initializing OE build env ###
Snip the logs but it looks no problem. As you know, the script automatically creates a subdirectory,build, under openbmc.
Common targets are:
obmc-phosphor-image
tm#tm-VB1:~/Rpi4-64/openbmc/build$
#4: Change the directory and edit local.conf for my Raspberrypi platform.
tm#tm-VB1:~/Rpi4-64/openbmc/build$ cat ./conf/local.conf
Snip the log for unchanged part.
MACHINE ??= "raspberrypi4-64" <<< Change here for my platform.
DL_DIR ?= "/home/tm/Yocto/downloads" <<< Add here for build-time reduction at retry.
SSTATE_DIR ?= "/home/tm/Yocto/sstate-cache" <<< Add here for build-time reduction at retry.
#5: Change FLASH_SIZE variable based on the following sugestion. https://github.com/openbmc/openbmc/issues/3590
tm#tm-VB1:~/Rpi4-64/openbmc/meta-phosphor/classes$ cat image_types_phosphor.bbclass
Snip the log.
# Flash characteristics in KB unless otherwise noted
FLASH_SIZE ?= "131072" <<< I changed only this variable from 32768 to 131072.
#6: bitbake starts.
tm#tm-VB1:~/Rpi4-64/openbmc/bitbake obmc-phosphor-image
Then, ERROR happened.
ERROR: Logfile of failure stored in: /home/tm/Rpi/openbmc/build/tmp/work/raspberrypi-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2055074
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.09147 s, 367 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=495980 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/u-boot.bin
DEBUG: Spanning start=0K end=512K
DEBUG: Compare needed=495980 available=524288 margin=28308
484+1 records in
484+1 records out
495980 bytes (496 kB, 484 KiB) copied, 0.00120141 s, 413 MB/s
DEBUG: Considering file size=8266960 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi
DEBUG: Spanning start=512K end=4864K
>>>DEBUG: Compare needed=8266960 available=4456448 margin=-3810512
ERROR: Image '/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi' is too large!
DEBUG: Python function do_generate_static finished
It said margin=-3810512.
Now, my 2nd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 262144.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3915 tasks of which 2633 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc/build$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.177223 s, 189 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc/build$
It said margin=-23936.
OK. Image is too large. So,my 3rd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 9437184.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3935 tasks of which 0 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 4 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc$
tm#tm-VB1:~/Rpi4/openbmc$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.173685 s, 193 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc$
It said the same margin as 256MB case.
My 4th try.
I removed the whole openbmc directory and did the same steps above.
I changed MACHINE ??= "raspberrypi4-64" to "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 33554432.
It is the same result before.
My 5th try.
I removed the whole openbmc directory and did the same steps above.
I used MACHINE ??= "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 67108864.
It is the same result before.
After I tried several variations, it always said "image is too large" although I changed FLASH_SIZE to much much larger one.
So, I am wondering if I have missed some important configuration or it needs another parameter to fix this except FLASH_SIZE.
By the way, I tried romulus and made it.
My environment is ubuntu-20.04.2.0-desktop-amd64.
I really appreciate someone could kindly give me advice to make this work.
Interesting, I don't have a quick fix for you but I did notice the partition that is over sized is the uboot partition. The uboot is a smaller separate binary installed on the machine. It looks as if your uboot build is over 512k and the partition is set for 512k. Your flash size is massize
FLASH_SIZE = 9437184" that is more then a gig, (because FLASH_SIZE is in K)
If I were you I would first try to build an older version of openbmc for raspberry pi. (It used to work so you just need to find the commit before uboot grew to big). Use git to move back a month until you find it works.
If that does not work I would try to modify the partition table.
here is where you failing
this looks fine building the uboot image looks fine
increasing the kernel offset make if build, but the other targets in openbmc will not be happy with this solution. So maybe meta-raspberry-pi will have to override the partition table (if uboot can not be shrunk)
What ever you do, open an issue on the github and share you changes. Also use the discord, and gerrit.
I just replicated this issue. We should fix it
I've got a chrome os app crashing regularly and causing all other chrome processes to crash as well.
I'm seeing crash reports in chrome://crashes, but no way to see the details of the report. I also can't find any minidump files to analyse.
What is the way to get crash report internals in chrome os?
Try the solution in this SO post.
root#localhost:-$ mkdir /tmp/misc && chmod 777 /tmp/misc
root#localhost:-$ cd /tmp
root#localhost:-$ watch -n 1 'find . -mmin -1 -exec cp {} /tmp/misc/ \;'
Then, as a regular user (not root):
google-chrome --enable-logging --v=1
Once you see files created by the watch command, run:
root#localhost:-$ ls -l
-rw------- 1 root root 230432 Apr 16 09:06 chromium-renderer-minidump-2113a256de381bce.dmp
-rw------- 1 root root 230264 Apr 16 09:12 chromium-renderer-minidump-95889ebac3d8ac81.dmp
-rw------- 1 root root 231264 Apr 16 09:13 chromium-renderer-minidump-da0752adcba4e7ca.dmp
-rw------- 1 root root 236246 Apr 16 09:12 chromium-upload-56dc27ccc3570a10
-rw------- 1 root root 237247 Apr 16 09:13 chromium-upload-5cebb028232dd944
Now you can use breakpad to work on the *.dmp files.
You need to be in dev mode in order to access the crash reports. There's no way otherwise to access where the crashes are saved (system crashes under /var/spool/crash and browser/user crashes under /home/chronos/*/crash/).
However, if you're using official Chrome OS builds, we don't currently publish the symbols for the binaries, so it'll probably be a bit difficult to debug using those minidumps.
Process Explorer has columns for CPU time (down to milliseconds) and CPU Cycles. For WinDbg I am aware of the !runaway command, also !runaway 7 for more details, but it shows CPU time only.
Are the CPU cycles also available somehow in a user mode crash dump?
What I have tried:
I looked at dt nt!_KTHREAD and I see it has a CycleTime property
ntdll!_KTHREAD
+0x000 Header : _DISPATCHER_HEADER
+0x018 CycleTime : Uint8B
I tried to query that property in a !for_each_thread, but WinDbg responds that it's available in kernel mode only.
Why do I want those CPU cycles?
I am working on a training for JetBrains dotTrace. It has an option to count CPU cycles and I'd like to explain where this cycles come from. Above kernel structure and Process Explorer is probably enough, but it would be awesome to see it live or post mortem in a user mode dump. I explain a lot of basics with WinDbg.
Following the implementation of GetProcessTimes() in ReactOS, you can see that the information is copied from the process' KPROCESS. So, indeed, it's only physically present in a dump that includes kernel memory.
C:\tw>ls -l
total 0
C:\tw>cdb -c ".dump /ma .\tw.dmp;q" calc.exe | grep writ
Dump successfully written
C:\tw>cdb -c "lm;!peb;.dump /ma .\tw1.dmp;q" calc.exe | grep writ
Dump successfully written
C:\tw>cdb -c ".ttime;q" -z tw.dmp | grep -B 3 quit
Created: Wed Apr 5 20:03:55.919 2017 ()
Kernel: 0 days 0:00:00.046
User: 0 days 0:00:00.000
quit:
C:\tw>cdb -c ".ttime;q" -z tw1.dmp | grep -B 3 quit
Created: Wed Apr 5 20:04:28.682 2017 ()
Kernel: 0 days 0:00:00.031
User: 0 days 0:00:00.000
quit:
C:\tw>
I just encountered some strange behavior with Perl 5.16.3 on FreeBSD 9.3-RELEASE-p3. We've got a cron job which runs every five minutes and generates some text status files. I just happened to list the contents of the output directory and saw that the timestamps for some of the files were in the future! The files are created like this:
if (open(OUT, "> $status_file_path")) {
print OUT "$status_info\n";
close OUT;
}
Now, the file handle OUT is used in several places, however it is opened and closed within the same block as shown above. And like I said, out of ten files, only a few had future dates when displayed using ls.
For example, files with the current date had timestamps like 04/02/2015 20:29:46, files with future timestamps were out in November, e.g. 11/10/2015 09:38:41.
What might be going on here?
EDIT
I've got two tests running:
1) a perl script running a loop of 1000 iterations, sleeping a random time up to 10 seconds between iterations, using the open/print/close logic to create an output file and abort the script if the file's modification time is in the future.
2) a cron entry to touch a test file every minute, e.g. touch /home/test/test_file_date_with_cron.txt
TEST RESULTS
Neither of the tests generated output files with a timestamp in the future.
This is scary.
EDIT 2
Here is the filesystem info, the files are written in the /usr directory.
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/gprootfs 2G 133M 1.7G 7% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/gpt/gpusrfs 431G 3.8G 392G 1% /usr
procfs 4.0k 4.0k 0B 100% /proc
EDIT 3
Running the script outside of cron for several hundred iterations didn't duplicate the problem. HOWEVER, I just found some other files, which are created by a CGI script which have the future dates:
-rw-r--r-- 1 test test 5783 Nov 10 2015 Config.xml_20150210_104151
-rw-r--r-- 1 test test 34548 Nov 10 2015 Config2.xml_20150210_104151
-rw-r--r-- 1 test test 6105 Nov 10 2015 Config.xml_20151109_232210
-rw-r--r-- 1 test test 34554 Nov 10 2015 Config2.xml_20151109_232210
-rw-rw-r-- 1 root test 2075 Nov 9 2015 Config.xml_20151109_231055
-rw-rw-r-- 1 root test 1232 Nov 9 2015 Config2.xml_20151109_231055
These are archive files, which get moved and renamed with the file's mtime timestamp. Note that BOTH ls and Perl's stat() function report the future date -- stat() is used to generate the file's timestamp portion of the name.
Looking at the first entry, ls reports "Nov 10 2015", whereas when the CGI script processed it, Perl's stat() reported "20150210_104151", i.e. "Feb 02 2015" which is most likely correct.
Further down, we see ls showing "Nov 10 2015" and stat() reported "20151109_232210", i.e. "Nov 09 2015".
Finding those additional archived config files helped me track down the cause, which was as others have suggested, that the system date and timezone changed.
From: 1447147328 and America/Adak
To: 1426637771 and America/New_York
What was throwing me off, was that I thought the cron script wrote ALL of the output files each time it executes, but that's not the case. The files have different "refresh intervals".
I followed some online guides trying to get some headless VMs to start/suspend automatically at boot/shutdown on my Mac. I can't get it to work at all. This is my first time trying to get script runs on Startup/Shutdown, so it could be that I'm just missing something very basic and if that's the case I apologize.
These are the steps I followed:
Created a directory /Library/StartupItems/HeadlessVM
Created two files within that directory:
-rwxr--r-- 1 root wheel 242 Feb 19 19:05 HeadlessVM
-rw-r--r-- 1 root wheel 188 Feb 20 12:42 StartupParameters.plist
Contents for HeadlessVM
$ cat HeadlessVM
#!/bin/sh
. /etc/rc.common
StartService ()
{
ConsoleMessage "Starting HeadlessVM"
/usr/local/bin/runvmheadless
}
StopService ()
{
ConsoleMessage "Suspending HeadlessVM"
/usr/local/bin/suspendvmheadless
}
RunService "$1"
Contents for StartupParameters.plist
$ cat StartupParameters.plist
{
Description = "Runs/Suspends Virtual Machine Headless on OS X Startup/Shutdown";
Provides = ("HeadlessVM");
Uses = ("Disks");
OrderPreference = ("Late");
}
Created my script files, that will perform both actions:
-rwxr-xr-x# 1 xxxxxxx admin 164 Feb 19 01:06 runvmheadless
-rwxr-xr-x# 1 xxxxxxx admin 160 Feb 19 01:19 suspendvmheadless
Contents for runvmheadless
$ cat runvmheadless
#!/bin/bash
"/Applications/VMware Fusion.app/Contents/Library/vmrun" -T fusion start "/Volumes/Archive/Virtual Machines/vm.vmwarevm/vm.vmx" nogui
Contents for suspendvmheadless
$ cat suspendvmheadless
#!/bin/bash
"/Applications/VMware Fusion.app/Contents/Library/vmrun" -T fusion suspend "/Volumes/StaticData/Virtual Machines/vm.vmwarevm/vm.vmx"
My troubleshooting so far:
If I run the scripts from the terminal, they work as intended.
If I run sudo /sbin/SystemStarter (start or stop) "HeadlessVM" it also works.
On console I only see the following when I reboot, no clue on what's wrong tho.
2/20/12 12:11:09.128 PM SystemStarter: Runs/Suspends Virtual Machine Headless on OS X Startup/Shutdown (100) did not complete successfully
Appreciate any help, Thank you.
I found what was wrong. The code above is fine, the problem is that my scripts are trying to get data from an encrypted secondary disk which was not available at boot time.
I used this in order to bypass this problem:https://github.com/jridgewell/Unlock
Thanks