I am trying to restore a mongo db dump. Running mongorestore on a PC works fine. But when i run it on a linux server (64bit MongoDB), no error is thrown, but the data is not imported.
Given below is the message on the terminal :
#mongorestore --drop dump
connected to: 127.0.0.1
Sun Nov 4 11:56:13 dump/m101/hw1.bson
Sun Nov 4 11:56:13 going into namespace [m101.hw1]
Sun Nov 4 11:56:13 dropping
1 objects found
Sun Nov 4 11:56:13 Creating index: { key: { _id: 1 }, ns: "m101.hw1", name: "_id_" }
Sun Nov 4 11:56:13 dump/m101/funnynumbers.bson
Sun Nov 4 11:56:13 going into namespace [m101.funnynumbers]
Sun Nov 4 11:56:13 dropping
100 objects found
Sun Nov 4 11:56:13 Creating index: { key: { _id: 1 }, ns: "m101.funnynumbers", name: "_id_" }
Edit:
This was a non-issue. The problem was not with the insertion, but with my querying. I was trying to do db.m101.funnynumbers.count(). The correct way of doing it is db.funnynumbers.count().
Related
Starting from El Capitan the system got System Integrity Protection which doesn't allow certain activities for some folders and files. Using Terminal "ls lO" command you can see flags that specific file or folder has.
drwxr-xr-x# 3 root wheel hidden 96 Aug 12 2014 opt
drwxr-xr-x 6 root wheel sunlnk,hidden 192 Nov 28 15:14 private
drwxr-xr-x# 64 root wheel restricted,hidden 2048 Nov 29 13:48 sbin
lrwxr-xr-x# 1 root wheel restricted,hidden 11 Nov 28 15:13 tmp -> private/tmp
drwxr-xr-x# 10 root wheel restricted,hidden 320 Nov 28 15:21 usr
lrwxr-xr-x# 1 root wheel restricted,hidden 11 Nov 28 15:13 var -> private/var
I'm interested in "restricted" flag. How can it be found using Swift without executing terminal commands?
I want to emphasize that executing Process() in code is not an approach that suits the needs.
"Hidden" flag can be detected via "isHidden" property from here URLResourceKey. However there is nothing about "restricted".
Could somebody point me to the right direction?
The things listed by the -O option on the ls tool are the file flags. These can be read by the stat() function in the BSD layer, and are found in the st_flags field of the resulting structure.
The "restricted" flag in ls's output corresponds to SF_RESTRICTED, so you can read it by doing something like this:
func isRestricted(at url: URL) throws -> Bool {
let flags: UInt32 = try url.withUnsafeFileSystemRepresentation { fsRep in
var info = stat()
if stat(fsRep, &info) != 0 {
guard let code = POSIXError.Code(rawValue: errno) else {
throw CocoaError(.fileReadUnknown)
}
throw POSIXError(code)
}
return info.st_flags
}
return flags & UInt32(bitPattern: SF_RESTRICTED) != 0
}
I am trying to touch a file(for referencing date) with a future date something like -
Current date - $date
Fri Jan 6 03:59:55 EST 2017
touch -t 201702032359.59 /var/tmp/ME_FILE_END
on checking the timestamp of the file as -
$ ls -lrt /var/tmp/ME_FILE_END
getting an output with only date and not the entire timestamp(hhmm.sec)
-rw-r--r-- 1 abcproc abc 0 Feb 3 2017 /var/tmp/ME_FILE_END
But for a date with is less than or equal to current it gives correct result -
touch -t 201612010000.00 /var/tmp/ME_FILE_START
ls -lrt /var/tmp/ME_FILE_START
-rw-r--r-- 1 abcproc abc 0 Dec 1 00:00 /var/tmp/ME_FILE_START
Can someone please suggest why this discrepancy ?
It's just the way ls displays the date. When far from now, the modification time is not displayed.
If you want details regarding the last access / modification / change time, you should be using stat.
stat /var/tmp/ME_FILE_END
You will see the expected output.
For example:
[10:29:41]dabi#gaia:~$ touch -t 201702032359.59 /var/tmp/ME_FILE_END
[10:29:43]dabi#gaia:~$ ls -ltr /var/tmp/ME_FILE_END
-rw-rw-r-- 1 dabi dabi 0 feb. 3 2017 /var/tmp/ME_FILE_END
[10:29:47]dabi#gaia:~$ stat /var/tmp/ME_FILE_END
File : '/var/tmp/ME_FILE_END'
Size : 0 Blocks : 0 I/O blocks : 4096 empty file
Device : 803h/2051d Inode : 5374373 Links : 1
Access : (0664/-rw-rw-r--) UID : ( 1000/ dabi) GID : ( 1000/ dabi)
Access : 2017-02-03 23:59:59.000000000 +0100
Change : 2017-02-03 23:59:59.000000000 +0100
Change : 2017-01-06 10:29:43.364630503 +0100
Birth : -
mongodb has unexpectedly crashed with following stack:
Sun Dec 29 11:30:43 [conn410] build index XXX { referral: 1 }
Sun Dec 29 11:30:43 [conn410] build index done 26597 records 0.056 secs
Sun Dec 29 11:30:43 Invalid access at address: 0
Sun Dec 29 11:30:43 Got signal: 11 (Segmentation fault).
Sun Dec 29 11:30:43 Backtrace:
0xa83fc9 0xa845a0 0x7fdad2200490 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x9493e3 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d40e0 0x8d6200 0x94360d 0x948375 0x8823bc 0x885405 0xa96a46
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
/lib64/libpthread.so.0(+0xf490) [0x7fdad2200490]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
/usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
/usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
/usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
/usr/bin/mongod(_ZN5mongo8UpdateOp4nextEv+0x253) [0x9493e3]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
/usr/bin/mongod(_ZN5mongo11MultiCursor10nextClauseEv+0x70) [0x8d40e0]
/usr/bin/mongod(_ZN5mongo11MultiCursorC1EPKcRKNS_7BSONObjES5_N5boost10shared_ptrINS0_8CursorOpEEEb+0x220) [0x8d6200]
/usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverE+0x35d) [0x94360d]
/usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugE+0x125) [0x948375]
/usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x47c) [0x8823bc]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x1105) [0x885405]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
Logstream::get called in uninitialized state
Sun Dec 29 11:30:43 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Sun Dec 29 11:30:43 ERROR: Client::shutdown not called: conn
how can I know what caused the crashing? Is there another log that describes more details?
Not sure the best way to accomplish this.
I get these results to a text file from a mysql query. I would like to the sort 4th column with the oldest entry first.
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Wed Apr 3 17:00:52 2013 Mon Apr 15 09:42:33 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Mon Apr 8 14:01:05 2013 Mon Apr 15 09:42:33 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Fri Apr 5 13:00:56 2013 Mon Apr 15 09:42:33 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Mon Apr 8 08:00:59 2013 Mon Apr 8 08:00:59 2013 10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Thu Mar 28 14:15:12 2013 Fri Apr 5 09:00:55 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Thu Mar 28 14:15:12 2013 Fri Apr 5 07:00:53 2013
sort -r -k10 test does not seem to cut it.
A Perl solution using Time::Piece (in the Perl standard library since 5.10.0) and a Schwartzian Transform.
#!/usr/bin/perl
use strict;
use warnings;
use 5.010;
use Time::Piece;
say map { $_->[0] }
sort { $a->[1] cmp $b->[1] }
map { [ $_, sortdate($_) ] } <DATA>;
sub sortdate {
my $date = join ' ', (split)[2 .. 6];
return Time::Piece->strptime($date, '%a %b %d %H:%M:%S %Y')->datetime;
}
__END__
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Wed Apr 3 17:00:52 2013 Mon Apr 15 09:42:33 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Mon Apr 8 14:01:05 2013 Mon Apr 15 09:42:33 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Fri Apr 5 13:00:56 2013 Mon Apr 15 09:42:33 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Mon Apr 8 08:00:59 2013 Mon Apr 8 08:00:59 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Thu Mar 28 14:15:12 2013 Fri Apr 5 09:00:55 2013
10.xxx.xxx.xxx 70:xx:xx:xx:xx:xx Thu Mar 28 14:15:12 2013 Fri Apr 5 07:00:53 2013
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
(
FOR /f "tokens=1-7*" %%a IN (sqlout.txt) DO (
SET "month="
FOR %%m IN (01 Jan 02 Feb 03 Mar 04 Apr 11 Nov 12 Dec) DO IF NOT DEFINED month (
IF %%d==%%m SET month=!prev!
SET prev=%%m
)
SET /a day=100+%%e
ECHO(%%g!month!!day!%%f*%%a %%b %%c %%d %%e %%f %%g %%h
)
)>tempfile.1
(
FOR /f "delims=" %%i IN ('sort tempfile.1') DO (
SET line=%%i
ECHO(!line:**=!
)
) >sortedoutput.txt
DEL tempfile.1 /F /Q
On the sample data, there are 8 significant columns. Of these, the fourth is month, fifth date, sixth time and seventh year. I have no idea what you mean by the "fourth" column being "the" date since there are two separate dates in each line.
The process simply picks up the fourth column and looks through a list of month numbers (2-digit) and month abbreviations. It's easier to save the previous element for use when the next is compared, so when the monthname matches the fourth column in %%d, the month variable is set, and that will turn off any further processing of the month.
You may have noticed I've not listed all of the months This gives you something to do. It obviously won't work unless all of the months are entered in the list.
Next we deal with the fifth column, and add 100 to the day number, producing a number from 101 to 131. These numbers all have three characters.
Next we ECHO out a string of
the year from column 7 in %%g
the 2-character month number in month
the 3-digit augmented day number
the time from column 6
an asterisk
each of the elements of the original line
Next step is to read each of those lines after sorting.
set the sorted line into line
echo everything from line except the part up to the first asterisk.
I have a replica set of 3 MongoDB instances. The instances have 8GB of RAM and Dual Core 2.27 GHz CPUs. All instances are running version 2.2.2 (I saw the same behavior from 2.0.1).
Here's my issue: Our primary instance (master of the replica set) recently acquired the habit of crawling to 100% CPU every 2 days. Tracking down the cause, I decided to run the MongoDB profiler. I found hundreds of extremely slow queries. Here is an example:
> db.system.profile.find()
{
"ts" : ISODate("2012-12-16T20:31:39.078Z"),
"op" : "command",
"ns" : "stylesaint.$cmd",
"command" : {
"count" : "tears",
"query" : {
"_id" : { "$gt" : ObjectId("50cdeadeaf58d3de96000294") },
"active" : true,
"is_image_processed" : true,
"hidden_from_feed" : false,
"hidden_from_public_feeds" : false
},
"fields" : null
},
"ntoreturn" : 1,
"responseLength" : 48,
"millis" : 13930,
"client" : "#########"
}
From what I've read about mongodb, the natural next step in these situations is to try explain()ing those queries. However, explain() does not explain the slowness of the query:
> db.tears.find({ "_id" : { "$gt" : ObjectId("50cdeadeaf58d3de96000294") }, "active" : true, "is_image_processed" : true, "hidden_from_feed" : false, "hidden_from_public_feeds" : false }).explain()
{
"cursor" : "BtreeCursor id",
"isMultiKey" : false,
"n" : 4,
"nscannedObjects" : 5,
"nscanned" : 5,
"nscannedObjectsAllPlans" : 23,
"nscannedAllPlans" : 25,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 0,
"indexBounds" : {
"_id" : [
[
ObjectId("50cdeadeaf58d3de96000294"),
ObjectId("ffffffffffffffffffffffff")
]
]
},
"server" : "#########"
}
Scanning 5 documents should not take 13 seconds. Something else is going on that is slowing down the query. Maybe some other query is starving the server's resources? However, I don't know where to look. Any advice you can offer is appreciated.
MongoDB Logs
I couldn't find any warnings in the startup process:
***** SERVER RESTARTED *****
Sun Dec 16 21:02:56 [initandlisten] MongoDB starting : pid=...
Sun Dec 16 21:02:56 [initandlisten] db version v2.2.2, pdfile version 4.5
Sun Dec 16 21:02:56 [initandlisten] git version: ...
Sun Dec 16 21:02:56 [initandlisten] build info: Linux 2.6.21.7-2 ...
Sun Dec 16 21:02:56 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/data/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", replSet: "...", rest: "true" }
Sun Dec 16 21:02:56 [initandlisten] journal dir=/data/mongodb/journal
Sun Dec 16 21:02:56 [initandlisten] recover : no journal files present, no recovery needed
Sun Dec 16 21:02:56 [initandlisten] waiting for connections on port ...
Sun Dec 16 21:02:56 [websvr] admin web console waiting for connections on port ...
Sun Dec 16 21:02:56 [initandlisten] connection accepted from ...
Sun Dec 16 21:02:56 [conn1] end connection ... (0 connections now open)
Sun Dec 16 21:02:56 [initandlisten] connection accepted from ... #2 (1 connection now open)
Sun Dec 16 21:02:56 [rsStart] replSet I am ...
Sun Dec 16 21:02:56 [rsStart] replSet STARTUP2
Sun Dec 16 21:02:56 [rsHealthPoll] replSet member ... is up
Sun Dec 16 21:02:56 [rsHealthPoll] replSet member ... is now in state SECONDARY
Sun Dec 16 21:02:57 [initandlisten] connection accepted from ... #3 (2 connections now open)
Sun Dec 16 21:02:57 [rsSync] replSet SECONDARY
Sun Dec 16 21:02:58 [initandlisten] connection accepted from ... #4 (3 connections now open)
Sun Dec 16 21:02:58 [initandlisten] connection accepted from ... #5 (4 connections now open)
Sun Dec 16 21:02:58 [conn5] end connection ... (3 connections now open)
Sun Dec 16 21:02:58 [rsHealthPoll] replSet member ... is up
Sun Dec 16 21:02:58 [rsHealthPoll] replSet member ... is now in state PRIMARY
Sun Dec 16 21:02:59 [initandlisten] connection accepted from ... #6 (4 connections now open)
Sun Dec 16 21:03:00 [initandlisten] connection accepted from ... #7 (5 connections now open)
Sun Dec 16 21:03:02 [conn7] end connection ... (4 connections now open)
Sun Dec 16 21:03:03 [rsBackgroundSync] replSet syncing to: ...
Sun Dec 16 21:03:04 [rsSyncNotifier] replset setting oplog notifier to ...
Sun Dec 16 21:03:06 [conn2] end connection ... (3 connections now open)
Sun Dec 16 21:03:06 [initandlisten] connection accepted from ... #8 (4 connections now open)
Sun Dec 16 21:03:08 [initandlisten] connection accepted from ... #9 (5 connections now open)
Sun Dec 16 21:03:13 [initandlisten] connection accepted from ... #10 (6 connections now open)
Sun Dec 16 21:03:13 [conn10] end connection ... (5 connections now open)
Sun Dec 16 21:03:13 [initandlisten] connection accepted from ... #11 (6 connections now open)
Sun Dec 16 21:03:15 [conn3] end connection ... (5 connections now open)
Sun Dec 16 21:03:16 [rsHealthPoll] replSet member .... is now in state SECONDARY
Sun Dec 16 21:03:16 [rsMgr] replSet info electSelf 1
Sun Dec 16 21:03:16 [rsMgr] replSet PRIMARY
Re: Request for more info
At the moment, MongoDB is functioning normally; there are no queries above 100ms. As soon as 100% CPU happens again, I'll post more info about system resources.
First off, I think the queries are probably a red herring. Are you running these servers under a NUMA architecture? You might read over the Mongo docs for usage on NUMA systems.
If you are running on a NUMA system, then using numactl to run your daemon with an interleave policy will probably fix your issue.
You can check to see if you have any startup warnings. They will appear in your log while you're booting the daemon, and and you can find them after the fact while the daemon is running, though I don't recall how off the top of my head.
Failing that, you might check your IO operations while making those queries. If I had to guess, you're hitting your disk and not operating with your working set in memory. What do your memory usage stats (free -h and the memory usage metrics from inside the mongo console) look like?