Related
Same as title
I'm Using AWS Glue, Script, Glue3 and Python3 also using psycopg2 library
When i delete AWS Aurora(postgresql) record
my syntax is very simple, like this
tuple_list : [(value1,value2),(value1,value2), ....]
value 1 : String, value 2 : timestamp
query = "DELETE FROM {table} WHERE (col1, col2) IN ( VALUES %s)".format(table=table)
extras.execute_values(db.cur(), query, tuple_list, template="(%s, %s)", page_size=2000)
db_conn.commit()
i got the message "where syntax 'col2' is not in list error"
i have no idea why i didn't work....
thx for your interesting and have a good day
all my error message
Job aborted due to stage failure: Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 24) (IP executor 1): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/types.py", line 1571, in __getattr__
idx = self.__fields__.index(item)
ValueError: 'col2' is not in list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 604, in main
process()
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 594, in process
out_iter = func(split_index, iterator)
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 2916, in pipeline_func
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 2916, in pipeline_func
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 2916, in pipeline_func
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 418, in func
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 932, in func
File "/tmp/job_test", line 136, in process_partition
File "/tmp/job_test", line 88, in execute_batch_delete
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/types.py", line 1576, in __getattr__
raise AttributeError(item)
AttributeError: col2
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:517)
at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:652)
at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:635)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:470)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1030)
at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2278)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Driver stacktrace:
I'm developing a ruby application using Ruby 2.6.3, Rails 6.0.2.2 and Postgres 12.
Everything was running fine, until I added an index to a table using a migration:
class AddIndexToUserEmail < ActiveRecord::Migration[6.0]
def change
add_index :users, :email, unique: true
end
end
Running rails test:models return the following infinite dump, and I wasn't able to get rid of.
Any help will be appreciated
% rails test:models
Run options: --seed 62890
# Running:
Traceback (most recent call last):
20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1158:in `with_friend'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:565:in `dump'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:565:in `dump': no _dump_data is defined for class PG::Connection (TypeError)
25: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
24: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
23: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
22: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
21: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1158:in `with_friend'
14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:564:in `dump'
5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:567:in `rescue in dump'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `make_proxy'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `new'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1101:in `initialize'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1807:in `to_id'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1738:in `current_server': DRb::DRbServerNotFound (DRb::DRbServerNotFound)
25: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
24: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
23: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
22: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
21: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1158:in `with_friend'
14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:564:in `dump'
5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:567:in `rescue in dump'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `make_proxy'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `new'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1101:in `initialize'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1807:in `to_id'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1738:in `current_server': DRb::DRbServerNotFound (DRb::DRbConnError)
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:94:in `block (2 levels) in start'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:97:in `rescue in block (2 levels) in start'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:97:in `each'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:98:in `block (3 levels) in start': undefined method `exception=' for #<Minitest::UnexpectedError: Unexpected exception> (NoMethodError)
Did you mean? exception
Traceback (most recent call last):
Traceback (most recent call last):
20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1158:in `with_friend'
15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:565:in `dump'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:565:in `dump' 10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.: 0/drb/drb.rb:1158:in `with_friend'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
no _dump_data is defined for class PG::Connection ( 8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
TypeError 7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
) 5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
25: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
24: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:565:in `dump'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:565:in `dump': 23: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
no _dump_data is defined for class PG::Connection ( 22: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
TypeError 21: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
)
20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
25: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
24: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1158:in `with_friend'
14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
23: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
22: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
21: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:564:in `dump'
20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:567:in `rescue in dump'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `make_proxy'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/ 17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
2.6.0/drb/drb.rb:650:in `new'
16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1158:in `with_friend'
14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1101:in `initialize'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1807:in `to_id'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1738:in `current_server': DRb::DRbServerNotFound 9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
( 8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
DRb::DRbServerNotFound 7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
) 6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:564:in `dump'
5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:567:in `rescue in dump'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `make_proxy'
25: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `new'
24: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1101:in `initialize'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1807:in `to_id'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1738:in `current_server': 23: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
DRb::DRbServerNotFound ( 22: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
DRb::DRbServerNotFound 21: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
) 20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1158:in `with_friend'
14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
25: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
24: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
23: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
22: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelizat 7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
ion.rb:74:in `map'
21: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:564:in `dump'
20: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
5: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:567:in `rescue in dump'
19: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `block in start'
4: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `make_proxy'
18: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:75:in `fork'
3: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:650:in `new'
2: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1101:in `initialize'
17: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:95:in `block (2 levels) in start'
1: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1807:in `to_id'
/Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1738:in `current_server' 16: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1138:in `method_missing'
: 15: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1158:in `with_friend'
DRb::DRbServerNotFound (DRb::DRbConnError 14: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1139:in `block in method_missing'
)
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
13: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1227:in `open'
12: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1140:in `block (2 levels) in method_missing'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:138:in `run'
11: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:1251:in `send_message'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `start'
10: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:929:in `send_request'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `map'
9: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `send_request'
7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `each'
8: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:610:in `each'
6: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2.2/lib/active_support/testing/parallelization.rb:74:in `times'
7: from /Users/pdipietro/.rbenv/versions/2.6.3/lib/ruby/2.6.0/drb/drb.rb:611:in `block in send_request'
... more dump existed but limited to 30000 characters
This problem is due to the last minitest version (5.14). They have fixed that in rails but the fix has not been released yet
Open you gemfile and explicitly set your minitest version:
gem 'minitest', '<=5.13'
then run bundle update minitest. It will then show you what the real issue with your database in your error message.
I am trying to do some text mining tasks with pySpark. I am new to Spark and I've been following this example http://mccarroll.net/blog/pyspark2/index.html to build the bag of words for my data.
Originally my data looked something like this
df.show(5)
+------------+---------+----------------+--------------------+
|Title |Month | Author | Document|
+------------+---------+----------------+--------------------+
| a | Jan| John |This is a document |
| b | Feb| Mary |A book by Mary |
| c | Mar| Luke |Newspaper article |
+------------+---------+----------------+--------------------+
So far I have extracted the terms of each document with
bow0 = df.rdd\
.map( lambda x: x.Document.replace(',',' ').replace('.',' ').replace('-',' ').lower())\
.flatMap(lambda x: x.split())\
.map(lambda x: (x, 1))
Which gives me
[('This', 1),
('is', 1),
('a', 1),
('document', 1)]
But when I try to compute the frequency with reduceByKey and try to see the result
bow0.reduceByKey(lambda x,y:x+y).take(50)
I get this error:
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
<ipython-input-53-966f90775397> in <module>()
----> 1 bow0.reduceByKey(lambda x,y:x+y).take(50)
/usr/local/spark/python/pyspark/rdd.py in take(self, num)
1341
1342 p = range(partsScanned, min(partsScanned + numPartsToTry, totalParts))
-> 1343 res = self.context.runJob(self, takeUpToNumLeft, p)
1344
1345 items += res
/usr/local/spark/python/pyspark/context.py in runJob(self, rdd, partitionFunc, partitions, allowLocal)
990 # SparkContext#runJob.
991 mappedRDD = rdd.mapPartitions(partitionFunc)
--> 992 port = self._jvm.PythonRDD.runJob(self._jsc.sc(), mappedRDD._jrdd, partitions)
993 return list(_load_from_socket(port, mappedRDD._jrdd_deserializer))
994
/usr/local/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in __call__(self, *args)
1131 answer = self.gateway_client.send_command(command)
1132 return_value = get_return_value(
-> 1133 answer, self.gateway_client, self.target_id, self.name)
1134
1135 for temp_arg in temp_args:
/usr/local/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
61 def deco(*a, **kw):
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()
/usr/local/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
317 raise Py4JJavaError(
318 "An error occurred while calling {0}{1}{2}.\n".
--> 319 format(target_id, ".", name), value)
320 else:
321 raise Py4JError(
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 31.0 failed 4 times, most recent failure: Lost task 1.3 in stage 31.0 (TID 84, 9.242.64.15, executor 7): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 177, in main
process()
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 172, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/usr/local/spark/python/pyspark/rdd.py", line 2423, in pipeline_func
return func(split, prev_func(split, iterator))
File "/usr/local/spark/python/pyspark/rdd.py", line 2423, in pipeline_func
return func(split, prev_func(split, iterator))
File "/usr/local/spark/python/pyspark/rdd.py", line 346, in func
return f(iterator)
File "/usr/local/spark/python/pyspark/rdd.py", line 1842, in combineLocally
merger.mergeValues(iterator)
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/shuffle.py", line 236, in mergeValues
for k, v in iterator:
File "<ipython-input-48-5c0753c6b152>", line 1, in <lambda>
AttributeError: 'NoneType' object has no attribute 'replace'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:404)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1517)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1505)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1504)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1504)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1732)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1687)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1676)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2029)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2050)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2069)
at org.apache.spark.api.python.PythonRDD$.runJob(PythonRDD.scala:455)
at org.apache.spark.api.python.PythonRDD.runJob(PythonRDD.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 177, in main
process()
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 172, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/usr/local/spark/python/pyspark/rdd.py", line 2423, in pipeline_func
return func(split, prev_func(split, iterator))
File "/usr/local/spark/python/pyspark/rdd.py", line 2423, in pipeline_func
return func(split, prev_func(split, iterator))
File "/usr/local/spark/python/pyspark/rdd.py", line 346, in func
return f(iterator)
File "/usr/local/spark/python/pyspark/rdd.py", line 1842, in combineLocally
merger.mergeValues(iterator)
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/shuffle.py", line 236, in mergeValues
for k, v in iterator:
File "<ipython-input-48-5c0753c6b152>", line 1, in <lambda>
AttributeError: 'NoneType' object has no attribute 'replace'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:404)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
To expand on my comment, the error you are receiving is due to the presence of a null value in your Document column. Here's a small example to demonstrate:
data = [
['a', 'Jan', 'John', 'This is a document'],
['b', 'Feb', 'Mary', 'A book by Mary'],
['c', 'Mar', 'Luke', 'Newspaper article'],
['d', 'Apr', 'Mark', None]
]
columns = ['Title', 'Month', 'Author', 'Document']
df = spark.createDataFrame(data, columns)
df.show()
#+-----+-----+------+------------------+
#|Title|Month|Author| Document|
#+-----+-----+------+------------------+
#| a| Jan| John|This is a document|
#| b| Feb| Mary| A book by Mary|
#| c| Mar| Luke| Newspaper article|
#| d| Apr| Mark| null|
#+-----+-----+------+------------------+
For the last row, the value in the Document column is null. When you compute bow0 as in your question, when the map function operates on that row it tries to call x.Document.replace where x is None. This results in AttributeError: 'NoneType' object has no attribute 'replace'.
One way to overcome this is to filter out the bad values before calling map:
bow0 = df.rdd\
.filter(lambda x: x.Document)\
.map( lambda x: x.Document.replace(',',' ').replace('.',' ').replace('-',' ').lower())\
.flatMap(lambda x: x.split())\
.map(lambda x: (x, 1))
bow0.reduceByKey(lambda x,y:x+y).take(50)
#[(u'a', 2),
# (u'this', 1),
# (u'is', 1),
# (u'newspaper', 1),
# (u'article', 1),
# (u'by', 1),
# (u'book', 1),
# (u'mary', 1),
# (u'document', 1)]
Or you can build in the check for None condition inside of your map function. In general, it is good practice to make your map function robust to bad inputs.
As an aside, you can do the same thing using the DataFrame API functions. In this case:
from pyspark.sql.functions import explode, split, regexp_replace, col, lower
df.select(explode(split(regexp_replace("Document", "[,.-]", " "), "\s+")).alias("word"))\
.groupby(lower(col("word")).alias("lower"))\
.count()\
.show()
#+---------+-----+
#| lower|count|
#+---------+-----+
#| document| 1|
#| by| 1|
#|newspaper| 1|
#| article| 1|
#| mary| 1|
#| is| 1|
#| a| 2|
#| this| 1|
#| book| 1|
#+---------+-----+
So I have a feature that I want to add in a program I'm writing. Basically, if a user has cookies that say what his username and password are it will look up username and the hashed value of the cookie in a stored postgres database and return a positive response and JWT token if I get a match (made as secure as I can with bcrypt hashing - whether storing hashed passwords is a good idea is maybe another question).
This request is made to the server every time a user goes to a page that requires a saved password.
My problem is that every now and then I get the following
inside CookieLogin
app_1 | search_userinfo_table started
db_1 | FATAL: sorry, too many clients already
app_1 | 2018/08/26 16:22:25 pq: sorry, too many clients already
app_1 | 2018/08/26 16:22:25 http: panic serving 172.21.0.1:37304: pq: sorry, too many clients already
app_1 | goroutine 704 [running]:
app_1 | net/http.(*conn).serve.func1(0xc420156f00)
app_1 | /usr/local/go/src/net/http/server.go:1721 +0xd0
app_1 | panic(0x6d63e0, 0xc420102c20)
app_1 | /usr/local/go/src/runtime/panic.go:489 +0x2cf
app_1 | log.Panic(0xc4200438c0, 0x1, 0x1)
app_1 | /usr/local/go/src/log/log.go:322 +0xc0
app_1 | github.com/patientplatypus/webserver/database.Search_userinfo_table(0xc42017f29d, 0xb, 0xc4201500c0, 0xc4200c22e0, 0xc420043a10)
app_1 | /go/src/github.com/patientplatypus/webserver/database/search.go:18 +0x169
app_1 | github.com/patientplatypus/webserver/authentication.loginHandler(0xc42017f29d, 0xb, 0xc42017f2ba, 0x3, 0x899a80, 0xc420148460)
app_1 | /go/src/github.com/patientplatypus/webserver/authentication/login.go:53 +0x39
app_1 | github.com/patientplatypus/webserver/authentication.CookieLogin(0x899a80, 0xc420148460, 0xc4203fc800)
app_1 | /go/src/github.com/patientplatypus/webserver/authentication/login.go:35 +0x279
app_1 | net/http.HandlerFunc.ServeHTTP(0x75c658, 0x899a80, 0xc420148460, 0xc4203fc800)
app_1 | /usr/local/go/src/net/http/server.go:1942 +0x44
app_1 | github.com/gorilla/mux.(*Router).ServeHTTP(0xc420052460, 0x899a80, 0xc420148460, 0xc4203fc800)
app_1 | /go/src/github.com/gorilla/mux/mux.go:162 +0x101
app_1 | main.JWTHandler.func1(0x899a80, 0xc420148460, 0xc4203fc600)
app_1 | /go/src/github.com/patientplatypus/webserver/main.go:34 +0x342
app_1 | net/http.HandlerFunc.ServeHTTP(0xc4200b12a0, 0x899a80, 0xc420148460, 0xc4203fc600)
app_1 | /usr/local/go/src/net/http/server.go:1942 +0x44
app_1 | github.com/rs/cors.(*Cors).Handler.func1(0x899a80, 0xc420148460, 0xc4203fc600)
app_1 | /go/src/github.com/rs/cors/cors.go:200 +0xe9
app_1 | net/http.HandlerFunc.ServeHTTP(0xc4200b12c0, 0x899a80, 0xc420148460, 0xc4203fc600)
app_1 | /usr/local/go/src/net/http/server.go:1942 +0x44
app_1 | net/http.serverHandler.ServeHTTP(0xc42008a630, 0x899a80, 0xc420148460, 0xc4203fc600)
app_1 | /usr/local/go/src/net/http/server.go:2568 +0x92
app_1 | net/http.(*conn).serve(0xc420156f00, 0x899fc0, 0xc4200506c0)
app_1 | /usr/local/go/src/net/http/server.go:1825 +0x612
app_1 | created by net/http.(*Server).Serve
app_1 | /usr/local/go/src/net/http/server.go:2668 +0x2ce
...and then the golang client will crash and I will have to manually restart. This is an error that happens only sometimes which is incredibly frustrating as it makes debugging difficult (sorry for the verbose terminal log - I'm not sure when the error will fire again).
Does anyone have any advice or suggestions? Is this a common problem with a no-duh solution?
Thanks!
EDIT:
As far as I can tell, here is the offending postgres search query:
package data
import(
"fmt"
"log"
_ "github.com/lib/pq"
)
func Search_userinfo_table(searchEmail string)(bool, string) {
fmt.Println("search_userinfo_table started")
rows, err1 := db.Query("SELECT * FROM userinfo")
if err1 != nil {
log.Panic(err1)
}
for rows.Next() {
var email string
var password string
var regString string
var regBool bool
var uid int
err2 := rows.Scan(&email, &password, ®String, ®Bool, &uid)
if err2 != nil {
log.Panic(err2)
}
fmt.Println("email | password")
fmt.Printf("%s | %s", email, password)
if email == searchEmail{
return false, password
}
}
return true, ""
}
Which returns the whether the user is in the database or not as well as the hashed password. Talking with a few people the answer appears to be that postgres adds a new connection every time it makes this read and then eventually crashes, but I want it to reuse connections (although I thought that this happened automagically?). Anyway, more information.
You're missing the required defer rows.Close().
Each instance of *sql.Rows has a connection associated with it. When you close a rows instance the associated connection gets released and put back into the pool (*DB).
Not closing a rows instance, on the other hand, will cause its connection not to be released. And not releasing connections will in turn lead to the connection pool to attempt to open one too many new connections.
1) Bundle identifier same as one at linked apps
2) Using a tester account
3) Window->G Play Games->Setup->Android Setup - done
4) SDK Manager's packages up to date, play-games-plugin-for-unity up to date
5) My setup code(taken from the Minimal sample and modified a litte bit):
void Start()
{
PlayGamesPlatform.DebugLogEnabled = true;
PlayGamesPlatform.Activate();
Social.localUser.Authenticate(success =>
{
Debug.Log(success);
});
var config = new PlayGamesClientConfiguration.Builder().Build();
PlayGamesPlatform.InitializeInstance(config);
}
I get the following logcat:
05-11 05:52:22.719 666-676/? I/ActivityManager﹕ START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity bnds=[240,427][240,427]} from pid 1212
05-11 05:52:22.744 666-1206/? I/ActivityManager﹕ Start proc com.Toughwin.MemoryRun for activity com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity: pid=937 uid=10305 gids={50305, 3003}
05-11 05:52:23.098 937-937/? D/ActivityThread﹕ ACT-AM_ON_RESUME_CALLED ActivityRecord{41fdc2c8 token=android.os.BinderProxy#41fdba30 {com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}}
05-11 05:52:23.103 937-937/? D/ActivityThread﹕ ACT-LAUNCH_ACTIVITY handled : 0 / ActivityRecord{41fdc2c8 token=android.os.BinderProxy#41fdba30 {com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}}
05-11 05:52:23.114 137-137/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:0,p:-1,c:137) setConsumerName: com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity
05-11 05:52:23.114 137-137/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:0,p:-1,c:137) setDefaultBufferSize: w=854, h=480
05-11 05:52:23.118 666-684/? I/WindowManager﹕ Gaining focus: Window{420f9d90 u0 com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}
05-11 05:52:23.197 137-338/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:0,p:937,c:137) connect: api=2 producer=(937:com.Toughwin.MemoryRun) producerControlledByApp=true
05-11 05:52:23.197 137-336/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:2,p:937,c:137) new GraphicBuffer needed
05-11 05:52:23.218 137-137/? I/GLConsumer﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e65b30,api:2) [void* android::GLConsumer::createImage(EGLDisplay, const android::sp<android::GraphicBuffer>&, const android::Rect&)]
05-11 05:52:23.224 666-684/? I/ActivityManager﹕ [AppLaunch] Displayed Displayed com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity: +491ms
05-11 05:52:23.224 666-684/? D/ActivityManager﹕ AP_PROF:AppLaunch_LaunchTime:com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity:491:152769531
05-11 05:52:23.370 937-963/? D/MALI﹕ #06 pc 0049f42c /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.370 937-963/? D/MALI﹕ #07 pc 0049f2f0 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.370 937-963/? D/MALI﹕ #08 pc 0049fa98 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.370 937-963/? D/MALI﹕ #09 pc 002e10e8 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.370 937-963/? D/MALI﹕ #10 pc 004b7c9c /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.370 937-963/? D/MALI﹕ #11 pc 004b90a8 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.371 937-963/? D/MALI﹕ #12 pc 004bde28 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.371 937-963/? E/Unity﹕ [EGL] eglChooseConfig(m_EGLDisplay, configAttribs, NULL, 0, &eglConfigCount): EGL_BAD_ATTRIBUTE: An unrecognized attribute or attribute value was passed in the attribute list.
(Filename: ./Runtime/GfxDevice/egl/ConfigEGL.cpp Line: 222)
05-11 05:52:23.374 937-963/? D/MALI﹕ #06 pc 004a08c0 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.374 937-963/? D/MALI﹕ #07 pc 004a0c88 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.374 937-963/? D/MALI﹕ #08 pc 0049f550 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.374 937-963/? D/MALI﹕ #09 pc 0049fb00 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.374 937-963/? D/MALI﹕ #10 pc 002e10e8 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.374 937-963/? D/MALI﹕ #11 pc 004b7c9c /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.374 937-963/? D/MALI﹕ #12 pc 004b90a8 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.374 937-963/? D/MALI﹕ #13 pc 004bde28 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.377 937-963/? D/MALI﹕ #06 pc 004a08c0 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.377 937-963/? D/MALI﹕ #07 pc 004a0ca0 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.377 937-963/? D/MALI﹕ #08 pc 0049f550 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.377 937-963/? D/MALI﹕ #09 pc 0049fb00 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.377 937-963/? D/MALI﹕ #10 pc 002e10e8 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.377 937-963/? D/MALI﹕ #11 pc 004b7c9c /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.377 937-963/? D/MALI﹕ #12 pc 004b90a8 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.377 937-963/? D/MALI﹕ #13 pc 004bde28 /data/app-lib/com.Toughwin.MemoryRun-1/libunity.so
05-11 05:52:23.391 937-963/? D/Unity﹕ GL_EXT_debug_marker GL_OES_texture_npot GL_OES_compressed_ETC1_RGB8_texture GL_OES_standard_derivatives GL_OES_EGL_image GL_OES_depth24 GL_ARM_rgba8 GL_ARM_mali_shader_binary GL_OES_depth_texture GL_OES_packed_depth_stencil GL_EXT_texture_format_BGRA8888 GL_OES_vertex_half_float GL_EXT_blend_minmax GL_OES_EGL_image_external GL_OES_EGL_sync GL_OES_rgb8_rgba8 GL_EXT_multisampled_render_to_texture GL_EXT_discard_framebuffer GL_OES_get_program_binary GL_ARM_mali_program_binary GL_EXT_shader_texture_lod GL_EXT_robustness GL_OES_depth_texture_cube_map GL_KHR_debug
05-11 05:52:23.569 133-598/? D/FrameworkListener﹕ dispatchCommand data = (getaddrinfo stats.unity3d.com ^ 1024 0 1 0 ^)
05-11 05:52:23.570 133-990/? D/libc-netbsd﹕ res_queryN name = stats.unity3d.com, class = 1, type = 1
05-11 05:52:23.570 133-990/? D/libc﹕ QUERY: RECURSIVE stats.unity3d.com (A)
05-11 05:52:23.634 133-990/? D/libc﹕ QUERY: RECURSIVE stats.unity3d.com (A)
05-11 05:52:23.634 133-990/? D/libc﹕ ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61990
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3
;; stats.unity3d.com, type = A, class = IN
stats.unity3d.com. 1H IN A 62.116.219.119
unity3d.com. 44m24s IN NS ns01.unity3d.com.
unity3d.com. 44m24s IN NS ns02.unity3d.com.
unity3d.com. 44m24s IN NS ns03.unity3d.com.
ns01.unity3d.com. 1h44m IN A 62.116.219.114
ns02.unity3d.com. 1h44m IN A 75.126.59.154
ns03.unity3d.com. 1h44m IN A 54.248.81.61
1h3m22ih
05-11 05:52:23.635 133-990/? D/libc﹕ f2 26 81 80 00 01 00 01 00 03 00 03 05 73 74 61 .&...........sta
74 73 07 75 6e 69 74 79 33 64 03 63 6f 6d 00 00 ts.unity3d.com..
01 00 01 c0 0c 00 01 00 01 00 00 0e 10 00 04 3e ...............>
74 db 77 c0 12 00 02 00 01 00 00 0a 68 00 07 04 t.w.........h...
6e 73 30 31 c0 12 c0 12 00 02 00 01 00 00 0a 68 ns01...........h
00 07 04 6e 73 30 32 c0 12 c0 12 00 02 00 01 00 ...ns02.........
00 0a 68 00 07 04 6e 73 30 33 c0 12 c0 3f 00 01 ..h...ns03...?..
00 01 00 00 18 60 00 04 3e 74 db 72 c0 52 00 01 .....`..>t.r.R..
00 01 00 00 18 60 00 04 4b 7e 3b 9a c0 65 00 01 .....`..K~;..e..
00 01 00 00 18 60 00 04 36 f8 51 3d .....`..6.Q=
05-11 05:52:23.635 133-990/? D/libc-netbsd﹕ res_queryN name = stats.unity3d.com succeed
05-11 05:52:23.635 937-986/? D/libc-netbsd﹕ getaddrinfo: stats.unity3d.com get result from proxy >>
05-11 05:52:23.637 937-986/? I/System.out﹕ [socket][0] connection stats.unity3d.com/62.116.219.119:80;LocalPort=45598(0)
05-11 05:52:23.637 937-986/? I/System.out﹕ [CDS]connect[stats.unity3d.com/62.116.219.119:80] tm:90
05-11 05:52:24.318 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: Activating PlayGamesPlatform.
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.321 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: Instance was not initialized, using default configuration.
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.412 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: PlayGamesPlatform activated: GooglePlayGames.PlayGamesPlatform
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.414 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: Creating platform-specific Play Games client.
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.416 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: Creating real IPlayGamesClient
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.496 937-963/? D/GamesUnitySDK﹕ Performing Android initialization of the GPG SDK
05-11 05:52:24.749 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: Entering state: BeforeRoomCreateStartedState
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.750 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: BeforeRoomCreateStartedState.OnStateEntered: Defaulting to no-op.
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.752 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: Entering state: ShutdownState
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.752 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 DEBUG: ShutdownState.OnStateEntered: Defaulting to no-op.
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:24.762 937-963/? W/Unity﹕ !!! [Play Games Plugin DLL] 05/11/15 5:52:24 +03:00 WARNING: PlayGamesPlatform already initialized. Ignoring this call.
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:25.024 937-1000/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:25 +03:00 DEBUG: Starting Auth Transition. Op: SIGN_IN status: ERROR_NOT_AUTHORIZED
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:25.093 666-684/? I/WindowManager﹕ Losing focus: Window{420f9d90 u0 com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}
05-11 05:52:25.142 937-937/? D/ActivityThread﹕ ACT-AM_ON_PAUSE_CALLED ActivityRecord{41fdc2c8 token=android.os.BinderProxy#41fdba30 {com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}}
05-11 05:52:30.209 937-937/? D/ActivityThread﹕ ACT-AM_ON_RESUME_CALLED ActivityRecord{41fdc2c8 token=android.os.BinderProxy#41fdba30 {com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}}
05-11 05:52:30.225 666-684/? I/WindowManager﹕ Gaining focus: Window{420f9d90 u0 com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}
05-11 05:52:33.932 937-1000/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:33 +03:00 DEBUG: Starting Auth Transition. Op: SIGN_IN status: ERROR_NOT_AUTHORIZED
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:33.967 937-963/? I/Unity﹕ [Play Games Plugin DLL] 05/11/15 5:52:33 +03:00 DEBUG: Invoking user callback on game thread
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:52:33.968 937-963/? I/Unity﹕ False
(Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
05-11 05:54:16.504 937-937/? D/ActivityThread﹕ ACT-AM_ON_PAUSE_CALLED ActivityRecord{41fdc2c8 token=android.os.BinderProxy#41fdba30 {com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}}
05-11 05:54:16.557 137-137/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:2,p:937,c:137) setDefaultBufferSize: w=480, h=854
05-11 05:54:16.568 666-684/? I/WindowManager﹕ Losing focus: Window{420f9d90 u0 com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}
05-11 05:54:16.622 137-618/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:2,p:-1,c:137) disconnect: api=2
05-11 05:54:16.622 137-618/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:0,p:-1,c:137) getReleasedBuffers: returning mask 0xffffffff
05-11 05:54:16.622 137-618/? I/GLConsumer﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e65b30,api:0) destroying EGLImage dpy=0x1 img=0x10000004
05-11 05:54:16.629 137-137/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:0,p:-1,c:-1) consumerDisconnect
05-11 05:54:16.630 137-137/? I/BufferQueue﹕ [com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity](this:0xb8e9bab8,id:1863,api:0,p:-1,c:-1) ~BufferQueue
05-11 05:54:18.535 666-22270/? I/WindowState﹕ WIN DEATH: Window{420f9d90 u0 com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}
05-11 05:54:18.535 666-22270/? W/WindowManager﹕ Force-removing child win Window{421009f8 u0 SurfaceView} from container Window{420f9d90 u0 com.Toughwin.MemoryRun/com.unity3d.player.UnityPlayerActivity}
Do I miss something? What could I possibly be missing?
SUCCESS!
Add a Keystore.(only if you've uploaded the apk to the dev console)
I was missing to add the keystore to my app, since I uploaded the app to the dev console and in order to be able to upload it, a keystore was needed.