I'm getting throttling error while getting data from Exchange online with PowerShell command and I'm having trouble understand what exactly exceeds the policy. I haven't found any documentation and all the numbers in the error message looks fine for me.
Category: 1003 Message:This operation exceeds the throttling budget for policy part 'LocalTime', policy value '3000000', Budget type: 'PowerShell'. Suggested backoff time 226271 ms.
PowerShell BudgetType: PowerShell
ActiveRunspaces: 0/20
Balance: -2960943/2160000/-3000000
PowerShellCmdletsLeft: 397/400
ExchangeCmdletsLeft: 197/200
CmdletTimePeriod: 5
DestructiveCmdletsLeft: 120/120
DestructiveCmdletTimePeriod: 60
QueueDepth: 100
MaxRunspacesTimePeriod: 60
RunSpacesRemaining: 20/20
LastTimeFrameUpdate: 6/12/2022 6:45:06 AM
LastTimeFrameUpdateDestructiveCmdlets: 6/12/2022 6:44:57 AM
LastTimeFrameUpdateMaxRunspaces: 6/12/2022 6:44:57 AM
Locked: True
LockRemaining: 00:03:46.2710274 ;
Computer: )
Reason:OverBudgetException
I want to have the policy change to avoid those issues but I'm not really sure what should be changed?
We have set up a three member replica set consisting of the following, all on MongoDB version 3.4:
Primary. Physical local server, Windows Server 2012, 64 GB RAM, 6 cores. Hosted in Scandinavia.
Secondary. Amazon EC2, Windows Server 2016, m4.2xlarge, 32 GB RAM, 8 vCPUs. Hosted in Germany.
Arbiter. Tiny cloud based Linux instance.
The problem we are seeing is that the secondary is unable to keep up with the primary. As we seed it with data (copy over from the primary) and add it to the replica set, it typically manages to get in sync, but an hour later it might lag behind by 10 minutes; a few hours later, it's an hour behind, and so on, until a day or two later, it goes stale.
We are trying to figure out why this is. The primary is consistently using 0-1% CPU, while the secondary is under constant heavy load at 20-80% CPU. This seems to be the only potential resource constraint. Disk and network load does not seem to be an issue. There seems to be some locking going on on the secondary, as operations in the mongo shell (such as db.getReplicationInfo()) frequently takes 5 minutes or more to complete, and mongostat rarely works (it just says i/o timeout). Here is output from mongostat during a rare instance when it reported stats for the secondary:
host insert query update delete getmore command dirty used flushes vsize res qrw arw net_in net_out conn set repl time
localhost:27017 *0 33 743 *0 0 166|0 1.0% 78.7% 0 27.9G 27.0G 0|0 0|1 2.33m 337k 739 rs PRI Mar 27 14:41:54.578
primary.XXX.com:27017 *0 36 825 *0 0 131|0 1.0% 78.7% 0 27.9G 27.0G 0|0 0|0 1.73m 322k 739 rs PRI Mar 27 14:41:53.614
secondary.XXX.com:27017 *0 *0 *0 *0 0 109|0 4.3% 80.0% 0 8.69G 7.54G 0|0 0|10 6.69k 134k 592 rs SEC Mar 27 14:41:53.673
I ran db.serverStatus() on the secondary, and compared to the primary, and one number that stood out was the following:
"locks" : {"Global" : {"timeAcquiringMicros" : {"r" : NumberLong("21188001783")
The secondary had an uptime of 14000 seconds at the time.
Would appreciate any ideas on what this could be, or how to debug this issue! We could upgrade the Amazon instance to something beefier, but we've done that three times already, and at this point we figure that something else must be wrong.
I'll include output from db.currentOp() on the secondary below, in case it helps. (That command took 5 minutes to run, after which the following was logged: Restarting oplog query due to error: CursorNotFound: Cursor not found, cursor id: 15728290121. Last fetched optime (with hash): { ts: Timestamp 1490613628000|756, t: 48 }[-5363878314895774690]. Restarts remaining: 3)
"desc":"conn605",
"connectionId":605,"client":"127.0.0.1:61098",
"appName":"MongoDB Shell",
"secs_running":0,
"microsecs_running":NumberLong(16),
"op":"command",
"ns":"admin.$cmd",
"query":{"currentOp":1},
"locks":{},
"waitingForLock":false,
"lockStats":{}
"desc":"repl writer worker 10",
"secs_running":0,
"microsecs_running":NumberLong(14046),
"op":"none",
"ns":"CustomerDB.ed2112ec779f",
"locks":{"Global":"W","Database":"W"},
"waitingForLock":false,
"lockStats":{"Global":{"acquireCount":{"w":NumberLong(1),"W":NumberLong(1)}},"Database":{"acquireCount":{"W":NumberLong(1)}}}
"desc":"ApplyBatchFinalizerForJournal",
"op":"none",
"ns":"",
"locks":{},
"waitingForLock":false,
"lockStats":{}
"desc":"ReplBatcher",
"secs_running":11545,
"microsecs_running":NumberLong("11545663961"),
"op":"none",
"ns":"local.oplog.rs",
"locks":{},
"waitingForLock":false,
"lockStats":{"Global":{"acquireCount":{"r":NumberLong(2)}},"Database":{"acquireCount":{"r":NumberLong(1)}},"oplog":{"acquireCount":{"r":NumberLong(1)}}}
"desc":"rsBackgroundSync",
"secs_running":11545,
"microsecs_running":NumberLong("11545281690"),
"op":"none",
"ns":"local.replset.minvalid",
"locks":{},
"waitingForLock":false,
"lockStats":{"Global":{"acquireCount":{"r":NumberLong(5),"w":NumberLong(1)}},"Database":{"acquireCount":{"r":NumberLong(2),"W":NumberLong(1)}},"Collection":{"acquireCount":{"r":NumberLong(2)}}}
"desc":"TTLMonitor",
"op":"none",
"ns":"",
"locks":{"Global":"r"},
"waitingForLock":true,
"lockStats":{"Global":{"acquireCount":{"r":NumberLong(35)},"acquireWaitCount":{"r":NumberLong(2)},"timeAcquiringMicros":{"r":NumberLong(341534123)}},"Database":{"acquireCount":{"r":NumberLong(17)}},"Collection":{"acquireCount":{"r":NumberLong(17)}}}
"desc":"SyncSourceFeedback",
"op":"none",
"ns":"",
"locks":{},
"waitingForLock":false,
"lockStats":{}
"desc":"WT RecordStoreThread: local.oplog.rs",
"secs_running":1163,
"microsecs_running":NumberLong(1163137036),
"op":"none",
"ns":"local.oplog.rs",
"locks":{},
"waitingForLock":false,
"lockStats":{"Global":{"acquireCount":{"r":NumberLong(1),"w":NumberLong(1)}},"Database":{"acquireCount":{"w":NumberLong(1)}},"oplog":{"acquireCount":{"w":NumberLong(1)}}}
"desc":"rsSync",
"secs_running":11545,
"microsecs_running":NumberLong("11545663926"),
"op":"none",
"ns":"local.replset.minvalid",
"locks":{"Global":"W"},
"waitingForLock":false,
"lockStats":{"Global":{"acquireCount":{"r":NumberLong(272095),"w":NumberLong(298255),"R":NumberLong(1),"W":NumberLong(74564)},"acquireWaitCount":{"W":NumberLong(3293)},"timeAcquiringMicros":{"W":NumberLong(17685)}},"Database":{"acquireCount":{"r":NumberLong(197529),"W":NumberLong(298255)},"acquireWaitCount":{"W":NumberLong(146)},"timeAcquiringMicros":{"W":NumberLong(651947)}},"Collection":{"acquireCount":{"r":NumberLong(2)}}}
"desc":"clientcursormon",
"secs_running":0,
"microsecs_running":NumberLong(15649),
"op":"none",
"ns":"CustomerDB.b72ac80177ef",
"locks":{"Global":"r"},
"waitingForLock":true,
"lockStats":{"Global":{"acquireCount":{"r":NumberLong(387)},"acquireWaitCount":{"r":NumberLong(2)},"timeAcquiringMicros":{"r":NumberLong(397538606)}},"Database":{"acquireCount":{"r":NumberLong(193)}},"Collection":{"acquireCount":{"r":NumberLong(193)}}}}],"ok":1}
JJussi was exactly right (thanks!). The problem was that the active data is bigger than the memory available, and we were using Amazon EBS "Throughput Optimized HDD". We changed this to "General Purpose SSD" and the problem immediately went away. We were even able to downgrade the server from m4.2xlarge to m4.large.
We were confused by the fact that this manifested as high CPU load. We thought disk load was not an issue based on the fairly low amount of data written to disk per second. But when we tried making the AWS server the primary instance, we noticed that there was a very strong correlation between high CPU load and disk queue length. Further testing on the disk showed that it had very bad performance for the type of traffic that Mongo has.
I have 2 raspberry pi's that I wanted to benchmark for load balancing purpose.
Raspberry pi Model B v1.1 - running Raspbian Jessie
Raspberry pi Model B+ v1.2 - running Raspbian Jessie
I installed sysbench on both systems and ran: sysbench --num-threads=1 --test=cpu --cpu-max-prime=10000 --validate run on the first and changed --num-threads=4 on the second, as its quadcore and ran both.
The results are not at all what I expected (I obviously expected the multithreaded benchmark to severely outperform the single threaded benchmark). When I ran a the command with a single thread, performance was about the same on both systems. But when I changed the number of threads to 4 on the second Pi it still took the same amount of time, except that the per request statistics showed that the average request took about 4 times as much time. I can seem to grasp why this is.
Here are the results:
Raspberry pi v1.1
Single thread
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 1325.0229s
total number of events: 10000
total time taken by event execution: 1324.9665
per-request statistics:
min: 131.00ms
avg: 132.50ms
max: 171.58ms
approx. 95 percentile: 137.39ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 1324.9665/0.00
Raspberry pi v1.2
Four threads
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 1321.0618s
total number of events: 10000
total time taken by event execution: 5283.8876
per-request statistics:
min: 486.45ms
avg: 528.39ms
max: 591.60ms
approx. 95 percentile: 553.98ms
Threads fairness:
events (avg/stddev): 2500.0000/0.00
execution time (avg/stddev): 1320.9719/0.03
"Raspberry pi Model B+ v1.2" has the same CPU as "Raspberry pi Model B v1.1". Both boards are from the first generation of Raspberry Pi and they have 1 core CPU.
For 4 CPU you need Raspberry Pi 2 Model B instead of Raspberry pi Model B+.
Yeah, the naming is a bit confusing :(
In my project, I have to update documents in MongoDB many times. I find MongoDB support insert a lot of documents with insert_many use a single command, but I can't use update_many to update a lot at once, they have difference condition. I have to update them one by one.
With insert_many, I can insert more than 7000 documents per second. At the same environment, there are only about 1500 documents could be updated per second. It just seems inefficient to send thousands of commands when one will do.
Is it possible to send multiple update commands to MongoDB server at once?
Thanks for your explain #Blakes Seven, I have rewritten my program with Bulk and update documents with "unordered" operation. There is the speed report on my test environment.
1 thread: 12655 doc/s cpu: 150 - 200%
2 threads: 19005 doc/s cpu: 200 - 300%
3 threads: 24433 doc/s cpu: 300 - 400%
4 threads: 28957 doc/s cpu: 400 - 500%
5 threads: 35586 doc/s cpu: 500 - 600%
6 threads: 32942 doc/s cpu: 600+%
On my test environment, test program and MongoDB server running on the same machine, It seems not perfect for multiple threads. The CPU usage of MongoDB when run the program with a single thread, It was between 150% and 200%. MongoDB executed the operations in parallel exactly, seems have a limit of the threads with a client connect.
Anyway, a single thread is enough for me, besides, fewer thread has higher efficiency.
Another report on the online environment that client and server running on a different machine:
1 thread: 14719 doc/s
2 threads: 26837 doc/s
3 threads: 34908 doc/s
4 threads: 46151 doc/s
5 threads: 47842 doc/s
6 threads: 52522 doc/s
You can do that with
db.collection.findAndModify()
Please go through documentation :
https://docs.mongodb.com/manual/reference/method/db.collection.findAndModify/
Background Information
I just recently started to get these errors with a production ColdFusion 10 server:
Service Temporary Unavailable!
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Jakarta/ISAPI/isapi_redirector/1.2.32 ()
After spending a few hours with google I came across this thread on the adobe forums:
https://forums.adobe.com/thread/1016323?start=0&tstart=0
There is a lot of information in this thread, but I focused on two areas.
Get current usage of threads/sessions/memory metrics.
Use the metrics information to tune the ColdFusion IIS Connector configuration
My goal was to finish with this blog post:
http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration
The blog post was referenced in this reported bug:
https://bugbase.adobe.com/index.cfm?event=bug&id=3318104
Problem
I'm currently stuck on #1; getting current usage of threads/sessions/memory metrics.
I checked: CFadmin > Debugging & Logging > Debug Output Settings > Enable Metrics Logging.
ColdFusion 10 metrics.log:
"Information","scheduler-1","07/20/14","15:12:24",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 1055964040 Total memory: 1570766848 Active Sessions: 679"
"Information","scheduler-1","07/20/14","15:13:24",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 1136605536 Total memory: 1572864000 Active Sessions: 674"
"Information","scheduler-1","07/20/14","15:14:24",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 939095720 Total memory: 1572339712 Active Sessions: 673"
On a ColdFusion 11 development server I turned on Enable Metrics Logging just to see what it reported. The metrics.log for that file looks like this:
"Information","scheduler-1","07/20/14","15:20:59",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 589971080 Total memory: 1320157184 Active Sessions: 40"
"Information","scheduler-2","07/20/14","15:21:59",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 583831160 Total memory: 1320157184 Active Sessions: 41"
"Information","scheduler-2","07/20/14","15:22:59",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 907572096 Total memory: 1431830528 Active Sessions: 40"
The problem is that almost all of the information is coming back as "null".
ColdFusion 10 environment:
Windows Server 2008 R2,
ColdFusion 10 Standard 64-bit,
Java 7u60
ColdFusion 11 environment:
Windows Server 2012 R2,
ColdFusion 11 Standard 64-bit,
Java 7u65
Additional Note
This was found in the coldfusion-error.log on ColdFusion 10 (not in CF 11 though):
java.lang.NullPointerException
at coldfusion.server.jrun4.metrics.SimpleLoadMetric.run(SimpleLoadMetric.java:157)
at coldfusion.scheduling.ThreadPool.run(ThreadPool.java:211)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:71)
The Question
Does anyone know how to get the Enable Metrics Logging to actually report threads used?
In the first link, "carl type3" posted a sample of his metrics.log file and it had all the information that I want to get.
CF Admin Metrics Settings
ColdFusion 10 workers.properties:
worker.list=cfusion
worker.cfusion.type=ajp13
worker.cfusion.host=localhost
worker.cfusion.port=8012
worker.cfusion.max_reuse_connections=250
worker.cfusion.connection_pool_size=500
worker.cfusion.connection_pool_timeout=60
ColdFusion 10 server.xml connector:
<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500" connectionTimeout ="60000" />
To enable metric logging, go to Debugging & Logging>Debug Output Settings and then update the values highlighted below.
The "Max threads" shows null value in your logs, which further infer, that the metric logs are not enabled. Use the Current Thread Count as an input to the connection_pool_size and then set the max_reuse_connections. You will also need to add connectionTimeout and maxThreads in server.xml, as suggested in http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning. This is applicable for CF10 as well.
The correct port you need to set in the CF Admin can be found in the server.xml of cfusion.
In most setups with IIS as a frontend webserver it's 8012, so you need to change this setting in the CF Admin to this value.
Restart coldfusion and you should finally see some values with cfstat as well as in the metrics log.