I've just installed (uploaded) a new extension onto my Magento2 store, however after compiling and deploying I constantly run into the following error:
1 exception(s):
Exception #0 (Exception): Warning: file_put_contents(/var/www/html/pub/static/frontend/smart_media/smart_media1/th_TH/css/themes.css): failed to open stream: No such file or directory in /var/www/html/vendor/oyejorge/less.php/lessc.inc.php on line 177
Exception #0 (Exception): Warning: file_put_contents(/var/www/html/pub/static/frontend/smart_media/smart_media1/th_TH/css/themes.css): failed to open stream: No such file or directory in /var/www/html/vendor/oyejorge/less.php/lessc.inc.php on line 177
#0 [internal function]: Magento\Framework\App\ErrorHandler->handler(2, 'file_put_conten...', '/var/www/html/v...', 177, Array)
#1 /var/www/html/vendor/oyejorge/less.php/lessc.inc.php(177): file_put_contents('/var/www/html/p...', '/* Less Variabl...')
#2 /var/www/html/app/code/Rokanthemes/Themeoption/Observer/RefreshStyleObserver.php(58): lessc->compileFile('/var/www/html/a...', '/var/www/html/p...')
#3 /var/www/html/vendor/magento/framework/Event/Invoker/InvokerDefault.php(72): Rokanthemes\Themeoption\Observer\RefreshStyleObserver->execute(Object(Magento\Framework\Event\Observer))
#4 /var/www/html/vendor/magento/framework/Event/Invoker/InvokerDefault.php(60): Magento\Framework\Event\Invoker\InvokerDefault->_callObserverMethod(Object(Rokanthemes\Themeoption\Observer\RefreshStyleObserver), Object(Magento\Framework\Event\Observer))
#5 /var/www/html/vendor/magento/framework/Event/Manager.php(66): Magento\Framework\Event\Invoker\InvokerDefault->dispatch(Array, Object(Magento\Framework\Event\Observer))
#6 /var/www/html/generated/code/Magento/Framework/Event/Manager/Proxy.php(95): Magento\Framework\Event\Manager->dispatch('layout_generate...', Array)
#7 /var/www/html/vendor/magento/framework/View/Layout/Builder.php(134): Magento\Framework\Event\Manager\Proxy->dispatch('layout_generate...', Array)
#8 /var/www/html/vendor/magento/framework/View/Page/Builder.php(55): Magento\Framework\View\Layout\Builder->generateLayoutBlocks()
#9 /var/www/html/vendor/magento/framework/View/Layout/Builder.php(65): Magento\Framework\View\Page\Builder->generateLayoutBlocks()
#10 /var/www/html/vendor/magento/framework/View/Layout.php(254): Magento\Framework\View\Layout\Builder->build()
#11 /var/www/html/vendor/magento/framework/View/Layout.php(876): Magento\Framework\View\Layout->build()
#12 /var/www/html/vendor/magento/module-cms/Helper/Page.php(171): Magento\Framework\View\Layout->getBlock('page_content_he...')
#13 /var/www/html/vendor/magento/module-cms/Controller/Index/Index.php(43): Magento\Cms\Helper\Page->prepareResultPage(Object(Magento\Cms\Controller\Index\Index\Interceptor), 'home')
#14 /var/www/html/vendor/magento/framework/App/Action/Action.php(107): Magento\Cms\Controller\Index\Index->execute()
#15 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(58): Magento\Framework\App\Action\Action->dispatch(Object(Magento\Framework\App\Request\Http))
#16 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Cms\Controller\Index\Index\Interceptor->___callParent('dispatch', Array)
#17 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Cms\Controller\Index\Index\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#18 /var/www/html/generated/code/Magento/Cms/Controller/Index/Index/Interceptor.php(26): Magento\Cms\Controller\Index\Index\Interceptor->___callPlugins('dispatch', Array, Array)
#19 /var/www/html/vendor/magento/framework/App/FrontController.php(55): Magento\Cms\Controller\Index\Index\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#20 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(58): Magento\Framework\App\FrontController->dispatch(Object(Magento\Framework\App\Request\Http))
#21 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\App\FrontController\Interceptor->___callParent('dispatch', Array)
#22 /var/www/html/vendor/magento/module-store/App/FrontController/Plugin/RequestPreprocessor.php(94): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#23 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(135): Magento\Store\App\FrontController\Plugin\RequestPreprocessor->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#24 /var/www/html/vendor/magento/module-page-cache/Model/App/FrontController/BuiltinPlugin.php(73): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#25 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(135): Magento\PageCache\Model\App\FrontController\BuiltinPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#26 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#27 /var/www/html/generated/code/Magento/Framework/App/FrontController/Interceptor.php(26): Magento\Framework\App\FrontController\Interceptor->___callPlugins('dispatch', Array, Array)
#28 /var/www/html/vendor/magento/framework/App/Http.php(135): Magento\Framework\App\FrontController\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#29 /var/www/html/vendor/magento/framework/App/Bootstrap.php(256): Magento\Framework\App\Http->launch()
#30 /var/www/html/pub/index.php(37): Magento\Framework\App\Bootstrap->run(Object(Magento\Framework\App\Http\Interceptor))
#31 {main}
I've searched for many solutions, tried to fix permissions, re-deploy static content, remove generated folder, change permissions etc. but to no avail. Some expert insight would be greatly appreciated. Thank you very much
It sounds like potentially the directory target it is trying to write to (smart_media/smart_media1/th_TH/css) does not exist. You may need to modify the extension so that it creates the directory prior to attempting to write to it.
Looks like there are permission issues. You can check magento docs to set the right permissions. You can try these commands
sudo find . -type d -exec chmod 755 {} \;
sudo find . -type f -exec chmod 644 {} \;
One of our Postgresql 9.2.4 database servers has got "stuck". The are 20 connections to it showing up in htop as stuck on BIND and the server is at 100% CPU and load avg of 20 or so:
postgres: chicken accounts x.x.x.x (46317) BIND
Update: It turns out that we had a bunch of simple selects with long IN lists in the same tx. These were the real sources of the problem.
These are all doing inserts into different tables using JDBC prepared statements and batching. SELECT's are still working. Occasional inserts into other databases by different connections are working. The database has a few hundred schemas all with exactly the same tables (we shard by client). The problem only seems to happen with inserts into a specific table in a couple of schemas.
There is nothing in the logs to indicate a problem. I have tried restarting the server.
What does BIND mean and how do I fix this?
Here are 3 stack traces taken a couple of seconds apart on one of the "stuck on BIND" back ends:
#0 0x00007fa2b41efb47 in FunctionCall2Coll (flinfo=<optimized out>, collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/fmgr/fmgr.c:1333
#1 0x00007fa2b41d7484 in SearchCatCache (cache=0x7fa2b4b5ba40, v1=<optimized out>, v2=<optimized out>, v3=<optimized out>, v4=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/cache/catcache.c:1114
#2 0x00007fa2b41e5548 in get_attstatsslot (statstuple=0x7fa0a0840fd8, atttype=<optimized out>, atttypmod=<optimized out>, reqkind=<optimized out>, reqop=<optimized out>, actualop=<optimized out>, values=0x7fff26999260, nvalues=0x7fff26999274, numbers=0x7fff26999268, nnumbers=0x7fff26999278) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/cache/lsyscache.c:2778
#3 0x00007fa2b4194909 in var_eq_const (varonleft=1 '\001', constval=140336814239560, operator=98, vardata=0x7fff269992f0, constisnull=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/adt/selfuncs.c:301
#4 var_eq_const (vardata=0x7fff269992f0, operator=98, constval=140336814239560, constisnull=<optimized out>, varonleft=1 '\001') at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/adt/selfuncs.c:258
#5 0x00007fa2b4194daa in eqsel (fcinfo=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/adt/selfuncs.c:239
#6 0x00007fa2b41efcbd in FunctionCall4Coll (flinfo=<optimized out>, collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/fmgr/fmgr.c:1378
#7 0x00007fa2b4190fbe in scalararraysel (root=0x7fa2b634dd50, clause=0x7fa2b634e890, is_join_clause=0 '\000', varRelid=-183960723, jointype=JOIN_INNER, sjinfo=0x0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/adt/selfuncs.c:1871
#8 0x00007fa2b409d32d in clause_selectivity (root=0x7fa2b634dd50, clause=0x7fa2b634e890, varRelid=0, jointype=JOIN_INNER, sjinfo=0x0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/path/clausesel.c:705
#9 0x00007fa2b409d5b4 in clauselist_selectivity (root=0x7fa2b634dd50, clauses=<optimized out>, varRelid=0, jointype=JOIN_INNER, sjinfo=0x0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/path/clausesel.c:108
#10 0x00007fa2b40a122d in set_baserel_size_estimates (root=0x7fa2b634dd50, rel=0x7fa2b5c0f140) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/path/costsize.c:3412
#11 0x00007fa2b409b9e2 in set_plain_rel_size (rel=0x7fa2b5c0f140, root=0x7fa2b634dd50, rte=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/path/allpaths.c:361
#12 set_rel_size (root=0x7fa2b634dd50, rel=0x7fa2b5c0f140, rti=1, rte=0x7fa2b4b6fec0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/path/allpaths.c:254
#13 0x00007fa2b409cc67 in set_base_rel_sizes (root=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/path/allpaths.c:179
#14 make_one_rel (root=0x7fa2b634dd50, joinlist=0x7fa2b5c0f5a8) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/path/allpaths.c:137
#15 0x00007fa2b40b3d41 in query_planner (root=0x7fa2b634dd50, tlist=<optimized out>, tuple_fraction=0, limit_tuples=-1, qp_callback=<optimized out>, qp_extra=0x7fff26999c60, cheapest_path=0x7fff26999c80, sorted_path=0x7fff26999c88, num_groups=0x7fff26999c78) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/plan/planmain.c:251
#16 0x00007fa2b40b55f7 in grouping_planner (root=<optimized out>, tuple_fraction=0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/plan/planner.c:1159
#17 0x00007fa2b40b74b9 in subquery_planner (glob=0x7fa2b634dcc0, parse=0x7fa2b4b6fd60, parent_root=<optimized out>, hasRecursion=<optimized out>, tuple_fraction=0, subroot=0x7fff26999d78) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/plan/planner.c:533
#18 0x00007fa2b40b7782 in standard_planner (parse=0x7fa2b4b6fd60, cursorOptions=0, boundParams=0x7fa2b64c4090) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/optimizer/plan/planner.c:204
#19 0x00007fa2b411e1ee in pg_plan_query (querytree=<optimized out>, cursorOptions=<optimized out>, boundParams=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/postgres.c:751
#20 0x00007fa2b411e2d4 in pg_plan_queries (querytrees=<optimized out>, cursorOptions=0, boundParams=0x7fa2b64c4090) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/postgres.c:810
#21 0x00007fa2b41d9f2c in BuildCachedPlan (plansource=0x7fa2b5132d30, qlist=0x7fa2b4b6fd10, boundParams=0x7fa2b64c4090) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/cache/plancache.c:903
#22 0x00007fa2b41da13a in GetCachedPlan (plansource=0x7fa2b5132d30, boundParams=0x7fa2b64c4090, useResOwner=0 '\000') at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/cache/plancache.c:1166
#23 0x00007fa2b411ffb5 in exec_bind_message (input_message=0x7fff2699a080) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/postgres.c:1715
#24 PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=0x7fa2b4aa61a8 "accounts", username=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/postgres.c:4008
#25 0x00007fa2b40dc23b in BackendRun (port=0x7fa2b4ac1aa0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:3614
#26 BackendStartup (port=0x7fa2b4ac1aa0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:3304
#27 ServerLoop () at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:1367
#28 0x00007fa2b40dcd11 in PostmasterMain (argc=<optimized out>, argv=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:1127
#29 0x00007fa2b3f17332 in main (argc=5, argv=0x7fa2b4aa4190) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/main/main.c:199
#0 0x00007fa2b1f37898 in do_compare (weights=0x7fa2ae712394 "", position=0, seq2=0x7fff269985d0, seq1=0x7fff26998560) at strcoll_l.c:447
#1 __GI___strcoll_l (s1=0x7fff269986a0 "http://twitter.com/AmandaMurphy92/statuses/408714411285966848", s2=0x7fff26998aa0 "http://twitter.com/alwhufc_/statuses/408714228712087552", l=<optimized out>) at strcoll_l.c:626
#2 0x00007fa2b41a5c7d in varstr_cmp (arg1=0x7fa2b660e310 "http://twitter.com/AmandaMurphy92/statuses/408714411285966848", len1=61, arg2=0x7fa2b6610d0c "http://twitter.com/alwhufc_/statuses/408714228712087552", len2=55, collid=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/adt/varlena.c:1488
#3 0x00007fa2b41a6101 in bttextcmp (fcinfo=0x7fff26998f20) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/adt/varlena.c:1669
#4 0x00007fa2b41efb03 in FunctionCall2Coll (flinfo=<optimized out>, collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/fmgr/fmgr.c:1326
#5 0x00007fa2b3f43267 in _bt_compare_array_elements (a=<optimized out>, b=<optimized out>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/nbtree/nbtutils.c:508
#6 0x00007fa2b421cf4c in med3 (a=0x7fa2b646e9b8 "\f\343`\266\242\177", b=0x7fa2b646ea00 "H\344^\266\242\177", c=0x7fa2b646ea48 "\b\ra\266\242\177", cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:99
#7 0x00007fa2b421d313 in qsort_arg (a=0x7fa2b646e9b8, n=78, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:145
#8 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b646e518, n=294, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#9 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b646dc30, n=1201, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#10 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b646dc30, n=2288, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#11 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b646dc30, n=3651, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#12 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b646dc30, n=7714, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#13 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b646dc30, n=10000, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#14 0x00007fa2b3f43c61 in _bt_sort_array_elements (nelems=<optimized out>, elems=0x7fa2b646dc30, reverse=0 '\000', skey=<optimized out>, scan=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/nbtree/nbtutils.c:477
#15 _bt_preprocess_array_keys (scan=0x7fa2b6469e40) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/nbtree/nbtutils.c:340
#16 0x00007fa2b3f412a7 in btrescan (fcinfo=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/nbtree/nbtree.c:512
#17 0x00007fa2b41efdb2 in FunctionCall5Coll (flinfo=<optimized out>, collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=<optimized out>, arg5=0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/fmgr/fmgr.c:1407
#18 0x00007fa2b3f3ac91 in index_rescan (scan=0x7fa2b6469e40, keys=0x7fa2b6469d30, nkeys=1, orderbys=0x7fa2b6469e10, norderbys=0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/index/indexam.c:343
#19 0x00007fa2b4059382 in ExecInitIndexScan (node=<optimized out>, estate=0x7fa2b638c7a0, eflags=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/executor/nodeIndexscan.c:617
#20 0x00007fa2b40461d5 in ExecInitNode (node=0x7fa2b58f15d0, estate=0x7fa2b638c7a0, eflags=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/executor/execProcnode.c:192
#21 0x00007fa2b4044dc6 in InitPlan (eflags=16, queryDesc=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/executor/execMain.c:879
#22 standard_ExecutorStart (queryDesc=<optimized out>, eflags=16) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/executor/execMain.c:203
#23 0x00007fa2b4122859 in PortalStart (portal=0x7fa2b4bb6250, params=0x7fa2b64c4090, eflags=0, snapshot=0x0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/pquery.c:533
#24 0x00007fa2b411fffc in exec_bind_message (input_message=0x7fff2699a080) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/postgres.c:1737
#25 PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=0x7fa2b4aa61a8 "accounts", username=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/postgres.c:4008
#26 0x00007fa2b40dc23b in BackendRun (port=0x7fa2b4ac1aa0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:3614
#27 BackendStartup (port=0x7fa2b4ac1aa0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:3304
#28 ServerLoop () at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:1367
#29 0x00007fa2b40dcd11 in PostmasterMain (argc=<optimized out>, argv=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:1127
#30 0x00007fa2b3f17332 in main (argc=5, argv=0x7fa2b4aa4190) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/main/main.c:199
#0 get_next_seq (seq=0x7fff26998600, nrules=4, rulesets=0x7fa2ae711f88 "\001\002\001\005\001\001\001\005\002\002\002\005", weights=0x7fa2ae712394 "", table=0x7fa2ae711f94, extra=0x7fa2ae73fd18 "", indirect=0x7fa2ae747b7c) at strcoll_l.c:165
#1 0x00007fa2b1f376ac in __GI___strcoll_l (s1=0x7fff269986d0 "http://twitter.com/JustJayWright/statuses/408715518095687680", s2=0x7fff26998ad0 "http://twitter.com/JuveeReds/statuses/408714997922295808", l=<optimized out>) at strcoll_l.c:600
#2 0x00007fa2b41a5c7d in varstr_cmp (arg1=0x7fa2b6679654 "http://twitter.com/JustJayWright/statuses/40871551809568768", <incomplete sequence \354>, len1=60, arg2=0x7fa2b660dbd4 "http://twitter.com/JuveeReds/statuses/40871499792229580", <incomplete sequence \360>, len2=56, collid=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/adt/varlena.c:1488
#3 0x00007fa2b41a6101 in bttextcmp (fcinfo=0x7fff26998f50) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/adt/varlena.c:1669
#4 0x00007fa2b41efb03 in FunctionCall2Coll (flinfo=<optimized out>, collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/fmgr/fmgr.c:1326
#5 0x00007fa2b3f43267 in _bt_compare_array_elements (a=<optimized out>, b=<optimized out>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/nbtree/nbtutils.c:508
#6 0x00007fa2b421d15e in qsort_arg (a=0x7fa2b6477550, n=8, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:156
#7 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b6477550, n=36, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#8 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b6477550, n=51, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#9 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b6477550, n=88, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#10 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b6477228, n=541, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#11 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b6474e50, n=4062, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#12 0x00007fa2b421d2c2 in qsort_arg (a=0x7fa2b646dc30, n=10000, es=8, cmp=0x7fa2b3f43250 <_bt_compare_array_elements>, arg=0x7fff26999770) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/port/qsort_arg.c:186
#13 0x00007fa2b3f43c61 in _bt_sort_array_elements (nelems=<optimized out>, elems=0x7fa2b646dc30, reverse=0 '\000', skey=<optimized out>, scan=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/nbtree/nbtutils.c:477
#14 _bt_preprocess_array_keys (scan=0x7fa2b6469e40) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/nbtree/nbtutils.c:340
#15 0x00007fa2b3f412a7 in btrescan (fcinfo=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/nbtree/nbtree.c:512
#16 0x00007fa2b41efdb2 in FunctionCall5Coll (flinfo=<optimized out>, collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=<optimized out>, arg5=0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/utils/fmgr/fmgr.c:1407
#17 0x00007fa2b3f3ac91 in index_rescan (scan=0x7fa2b6469e40, keys=0x7fa2b6469d30, nkeys=1, orderbys=0x7fa2b6469e10, norderbys=0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/access/index/indexam.c:343
#18 0x00007fa2b4059382 in ExecInitIndexScan (node=<optimized out>, estate=0x7fa2b638c7a0, eflags=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/executor/nodeIndexscan.c:617
#19 0x00007fa2b40461d5 in ExecInitNode (node=0x7fa2b58f15d0, estate=0x7fa2b638c7a0, eflags=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/executor/execProcnode.c:192
#20 0x00007fa2b4044dc6 in InitPlan (eflags=16, queryDesc=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/executor/execMain.c:879
#21 standard_ExecutorStart (queryDesc=<optimized out>, eflags=16) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/executor/execMain.c:203
#22 0x00007fa2b4122859 in PortalStart (portal=0x7fa2b4bb6250, params=0x7fa2b64c4090, eflags=0, snapshot=0x0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/pquery.c:533
#23 0x00007fa2b411fffc in exec_bind_message (input_message=0x7fff2699a080) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/postgres.c:1737
#24 PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=0x7fa2b4aa61a8 "accounts", username=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/tcop/postgres.c:4008
#25 0x00007fa2b40dc23b in BackendRun (port=0x7fa2b4ac1aa0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:3614
#26 BackendStartup (port=0x7fa2b4ac1aa0) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:3304
#27 ServerLoop () at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:1367
#28 0x00007fa2b40dcd11 in PostmasterMain (argc=<optimized out>, argv=<optimized out>) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/postmaster/postmaster.c:1127
#29 0x00007fa2b3f17332 in main (argc=5, argv=0x7fa2b4aa4190) at /tmp/buildd/postgresql-9.2-9.2.5/build/../src/backend/main/main.c:199
Here is a sample htop (Postgres mostly doing BIND):
1 [||||||||||||||||||||||||||||||||||||||||||100.0%] 5 [||||||||||||||||||||||||||||||||||||||||||100.0%]
2 [||||||||||||||||||||||||||||||||||||||||||100.0%] 6 [||||||||||||||||||||||||||||||||||||||||||100.0%]
3 [||||||||||||||||||||||||||||||||||||||||||100.0%] 7 [||||||||||||||||||||||||||||||||||||||||||100.0%]
4 [||||||||||||||||||||||||||||||||||||||||||100.0%] 8 [||||||||||||||||||||||||||||||||||||||||||100.0%]
Mem[||||||||||||||||||||||||||||||||||||3142/32077MB] Tasks: 112, 5 thr; 22 running
Swp[||| 756/16375MB] Load average: 12.14 4.70 2.61
Uptime: 21 days, 16:53:13
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
25190 postgres 20 0 8537M 792M 760M R 66.0 2.5 1:23.90 postgres: chicken accounts xxx.xx.xx.55(37696) BIND
25001 postgres 20 0 8567M 1283M 1225M R 55.0 4.0 2:13.37 postgres: chicken accounts xxx.xx.xx.55(37547) BIND
25245 postgres 20 0 8545M 1346M 1308M R 55.0 4.2 0:32.43 postgres: chicken accounts xxx.xx.xx.54(44788) BIND
25246 postgres 20 0 8551M 1087M 1040M R 51.0 3.4 0:36.89 postgres: chicken accounts xxx.xx.xx.55(37811) BIND
25002 postgres 20 0 8549M 1141M 1097M R 48.0 3.6 0:37.10 postgres: chicken accounts xxx.xx.xx.55(37546) BIND
25248 postgres 20 0 8538M 872M 839M R 48.0 2.7 0:31.32 postgres: chicken accounts xxx.xx.xx.54(44794) SELECT
25247 postgres 20 0 8532M 791M 762M R 42.0 2.5 0:31.28 postgres: chicken accounts xxx.xx.xx.55(37810) BIND
25367 postgres 20 0 8525M 542M 519M R 40.0 1.7 0:26.89 postgres: chicken accounts xxx.xx.xx.55(38157) BIND
25370 postgres 20 0 8525M 653M 630M R 40.0 2.0 0:26.24 postgres: chicken accounts xxx.xx.xx.54(48307) SELECT
24267 postgres 20 0 8583M 2614M 2536M R 40.0 8.1 2:10.62 postgres: chicken accounts xxx.xx.xx.55(35060) BIND
23945 postgres 20 0 8587M 2671M 2588M R 36.0 8.3 2:41.28 postgres: chicken accounts xxx.xx.xx.55(33708) BIND
24437 postgres 20 0 8579M 2966M 2892M R 36.0 9.2 3:01.71 postgres: chicken accounts xxx.xx.xx.54(50743) BIND
25039 postgres 20 0 8549M 1147M 1103M R 33.0 3.6 1:30.71 postgres: chicken accounts xxx.xx.xx.54(43718) BIND
24999 postgres 20 0 8547M 916M 875M R 31.0 2.9 0:33.86 postgres: chicken accounts xxx.xx.xx.54(42010) BIND
24793 postgres 20 0 8556M 1262M 1210M R 31.0 3.9 0:41.88 postgres: chicken accounts xxx.xx.xx.54(36794) BIND
25368 postgres 20 0 8529M 645M 622M R 31.0 2.0 0:22.30 postgres: chicken accounts xxx.xx.xx.55(38163) PARSE
24868 postgres 20 0 8554M 1833M 1782M R 31.0 5.7 2:18.14 postgres: chicken accounts xxx.xx.xx.54(39693) BIND
23040 postgres 20 0 8603M 3770M 3677M R 29.0 11.8 3:07.17 postgres: chicken accounts xxx.xx.xx.55(58410) BIND
25000 postgres 20 0 8549M 1152M 1107M R 28.0 3.6 1:56.71 postgres: chicken accounts xxx.xx.xx.54(42018) BIND
25374 postgres 20 0 8513M 378M 367M R 13.0 1.2 0:10.85 postgres: chicken accounts xxx.xx.xx.55(38173) idle in
I start to think so it is PostgreSQL bug - I seen this pathological event more time in our customer, when database was overloaded. When we try to debug this issue, we connections waits to spinlocks and all activity is related to spinlocks processing
354246.00 93.0% s_lock /usr/lib/postgresql/9.2/bin/postgres
10503.00 2.8% LWLockRelease /usr/lib/postgresql/9.2/bin/postgres
8802.00 2.3% LWLockAcquire /usr/lib/postgresql/9.2/bin/postgres
828.00 0.2% _raw_spin_lock [kernel.kallsyms]
559.00 0.1% _raw_spin_lock_irqsave [kernel.kallsyms]
340.00 0.1% switch_mm [kernel.kallsyms]
305.00 0.1% poll_schedule_timeout [kernel.kallsyms]
274.00 0.1% native_write_msr_safe [kernel.kallsyms]
257.00 0.1% _raw_spin_lock_irq [kernel.kallsyms]
238.00 0.1% apic_timer_interrupt [kernel.kallsyms]
236.00 0.1% __schedule [kernel.kallsyms]
213.00 0.1% HeapTupleSatisfiesMVCC
This issue is related to high number of opened connections (more than 200). It is first time so somebody reports it with small number of connections. Please, recheck a number of opened connections.
BIND is the second phase of the PARSE / BIND / EXECUTE extended query protocol in PostgreSQL; see message flow in the docs.
To work out what it's doing in the bind phase and why it's taking so long, you'll need to use a tool like gdb to attach to a process and get a backtrace. Or several, since it sounds like it could be an infinite loop. See this wiki page for guidance.
If you want to do it non-intrusively you can potentially use the perf tool to capture a profile. I wrote about this recently: Tracing PostgreSQL performance with perf. There's also the PostgreSQL wiki page on using perf. It can be a bit tricky to convince perf to produce useful user-space stack traces on x64 on most distros, as noted in my post, unless you recompile PostgreSQL with -fno-omit-frame-pointer -Og -g3, which you probably won't want to do.
For that reason, gdb is probably easier. So long as you don't do anything silly it won't crash the backend you're attaching to. Even if you did crash it, by say running a call command that caused a segfault, your PostgreSQL server will just interrupt all connections and restart.
Don't leave gdb attached with execution stopped for ages, as it'll stop the master from doing necessary cleanup work. There's no rush though; even on a busy server, having gdb stopping a backend for a few minutes is not going to bother anything much. If gdb is attached but the program is running ("continue" in gdb) then having it attached won't affect anything at all.
Consider popping over to PostgreSQL's IRC channel if you're stuck with debugging the server. I won't be there at the moment, but usually lots of others will.