Spark Dataframe Union giving duplicates - scala

I have a base dataset, and one of the columns is having null and not null values.
so I do:
val nonTrained_ds = base_ds.filter(col("col_name").isNull)
val trained_ds = base_ds.filter(col("col_name").isNotNull)
When I print that out, I get clear separate of rows. But when I do,
val combined_ds = nonTrained_ds.union(trained_ds)
I get duplicate records of rows from nonTrained_ds, and the strange thing is, rows from trained_ds are no longer in the combined ds.
Why does this happen?
the values of trained_ds are:
+----------+----------------+
|unique_no | running_id|
+----------+----------------+
|0456700001|16 |
|0456700004|16 |
|0456700007|16 |
|0456700010|16 |
|0456700013|16 |
|0456700016|16 |
|0456700019|16 |
|0456700022|16 |
|0456700025|16 |
|0456700028|16 |
|0456700031|16 |
|0456700034|16 |
|0456700037|16 |
|0456700040|16 |
|0456700043|16 |
|0456700046|16 |
|0456700049|16 |
|0456700052|16 |
|0456700055|16 |
|0456700058|16 |
|0456700061|16 |
|0456700064|16 |
|0456700067|16 |
|0456700070|16 |
+----------+----------------+
the values of nonTrained_ds are:
+----------+----------------+
|unique_no | running_id|
+----------+----------------+
|0456700002|null |
|0456700003|null |
|0456700005|null |
|0456700006|null |
|0456700008|null |
|0456700009|null |
|0456700011|null |
|0456700012|null |
|0456700014|null |
|0456700015|null |
|0456700017|null |
|0456700018|null |
|0456700020|null |
|0456700021|null |
|0456700023|null |
|0456700024|null |
|0456700026|null |
|0456700027|null |
|0456700029|null |
|0456700030|null |
|0456700032|null |
|0456700033|null |
|0456700035|null |
|0456700036|null |
|0456700038|null |
|0456700039|null |
|0456700041|null |
|0456700042|null |
|0456700044|null |
|0456700045|null |
|0456700047|null |
|0456700048|null |
|0456700050|null |
|0456700051|null |
|0456700053|null |
|0456700054|null |
|0456700056|null |
|0456700057|null |
|0456700059|null |
|0456700060|null |
|0456700062|null |
|0456700063|null |
|0456700065|null |
|0456700066|null |
|0456700068|null |
|0456700069|null |
|0456700071|null |
|0456700072|null |
+----------+----------------+
the values of the combined ds are:
+----------+----------------+
|unique_no | running_id|
+----------+----------------+
|0456700002|null |
|0456700003|null |
|0456700005|null |
|0456700006|null |
|0456700008|null |
|0456700009|null |
|0456700011|null |
|0456700012|null |
|0456700014|null |
|0456700015|null |
|0456700017|null |
|0456700018|null |
|0456700020|null |
|0456700021|null |
|0456700023|null |
|0456700024|null |
|0456700026|null |
|0456700027|null |
|0456700029|null |
|0456700030|null |
|0456700032|null |
|0456700033|null |
|0456700035|null |
|0456700036|null |
|0456700038|null |
|0456700039|null |
|0456700041|null |
|0456700042|null |
|0456700044|null |
|0456700045|null |
|0456700047|null |
|0456700048|null |
|0456700050|null |
|0456700051|null |
|0456700053|null |
|0456700054|null |
|0456700056|null |
|0456700057|null |
|0456700059|null |
|0456700060|null |
|0456700062|null |
|0456700063|null |
|0456700065|null |
|0456700066|null |
|0456700068|null |
|0456700069|null |
|0456700071|null |
|0456700072|null |
|0456700002|16 |
|0456700005|16 |
|0456700008|16 |
|0456700011|16 |
|0456700014|16 |
|0456700017|16 |
|0456700020|16 |
|0456700023|16 |
|0456700026|16 |
|0456700029|16 |
|0456700032|16 |
|0456700035|16 |
|0456700038|16 |
|0456700041|16 |
|0456700044|16 |
|0456700047|16 |
|0456700050|16 |
|0456700053|16 |
|0456700056|16 |
|0456700059|16 |
|0456700062|16 |
|0456700065|16 |
|0456700068|16 |
|0456700071|16 |
+----------+----------------+


This did the trick,
val nonTrained_ds = base_ds.filter(col("primary_offer_id").isNull).distinct()
val trained_ds = base_ds.filter(col("primary_offer_id").isNotNull).distinct()

Related

Can I specify the output in Test2 test

Is there a way to edit the following Test2 test:
perl -MTest2::V0 -MRegexp::Common=URI -wE 'is( shift, match $RE{URI}, "should match a URI" )' foo
Such that the CHECK output is less verbose than this:
# Seeded srand with seed '20200813' from local date.
1..1
not ok 1 - should match a URI
# Failed test 'should match a URI'
# at -e line 1.
# +-----+----+-------------------------------------------------------+-----+
# | GOT | OP | CHECK | LNs |
# +-----+----+-------------------------------------------------------+-----+
# | foo | =~ | (?:(?:(?:file)://(?:(?:(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][ | 1 |
# | | | -a-zA-Z0-9]*)?[a-zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9 | |
# | | | ]*[a-zA-Z0-9]|[a-zA-Z]))|(?:[0-9]+[.][0-9]+[.][0-9]+[ | |
# | | | .][0-9]+))|localhost)?)(?:/(?:(?:(?:(?:[-a-zA-Z0-9$_. | |
# | | | +!*'(),:#&=]|(?:%[a-fA-F0-9][a-fA-F0-9]))*)(?:/(?:(?: | |
# | | | [-a-zA-Z0-9$_.+!*'(),:#&=]|(?:%[a-fA-F0-9][a-fA-F0-9] | |
# | | | ))*))*)))))|(?:(?:telnet)://(?:(?:(?:(?:(?:[-a-zA-Z0- | |
# | | | 9$_.+!*'(),;?&=]|(?:%[a-fA-F0-9][a-fA-F0-9]))*))(?::( | |
# | | | ?:(?:(?:[-a-zA-Z0-9$_.+!*'(),;?&=]|(?:%[a-fA-F0-9][a- | |
# | | | fA-F0-9]))*)))?)#)?(?:(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][- | |
# | | | a-zA-Z0-9]*)?[a-zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9] | |
# | | | *[a-zA-Z0-9]|[a-zA-Z]))|(?:[0-9]+[.][0-9]+[.][0-9]+[. | |
# | | | ][0-9]+)))(?::(?:(?:[0-9]+)))?)(?:/)?)|(?:(?:https?): | |
# | | | //(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z | |
# | | | 0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z | |
# | | | ])[.]?)|(?:[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(? | |
# | | | :(?:[0-9]*)))?(?:/(?:(?:(?:(?:(?:(?:[a-zA-Z0-9\-_.!~* | |
# | | | '():#&=+$,]+|(?:%[a-fA-F0-9][a-fA-F0-9]))*)(?:;(?:(?: | |
# | | | [a-zA-Z0-9\-_.!~*'():#&=+$,]+|(?:%[a-fA-F0-9][a-fA-F0 | |
# | | | -9]))*))*)(?:/(?:(?:(?:[a-zA-Z0-9\-_.!~*'():#&=+$,]+| | |
# | | | (?:%[a-fA-F0-9][a-fA-F0-9]))*)(?:;(?:(?:[a-zA-Z0-9\-_ | |
# | | | .!~*'():#&=+$,]+|(?:%[a-fA-F0-9][a-fA-F0-9]))*))*))*) | |
# | | | )(?:[?](?:(?:(?:[;/?:#&=+$,a-zA-Z0-9\-_.!~*'()]+|(?:% | |
# | | | [a-fA-F0-9][a-fA-F0-9]))*)))?))?)|(?:(?:gopher)://(?: | |
# | | | (?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0-9]) | |
# | | | [.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z]))|( | |
# | | | ?:[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(?:(?:[0-9] | |
# | | | +)))?/(?:(?:(?:[0-9+IgT]))(?:(?:(?:[-a-zA-Z0-9$_.+!*' | |
# | | | (),:#&=]+|(?:%[a-fA-F0-9][a-fA-F0-9]))*))))|(?:(?:pop | |
# | | | )://(?:(?:(?:(?:[-a-zA-Z0-9$_.+!*'(),&=~]+|(?:%[a-fA- | |
# | | | F0-9][a-fA-F0-9]))+))(?:;AUTH=(?:[*]|(?:(?:(?:[-a-zA- | |
# | | | Z0-9$_.+!*'(),&=~]+|(?:%[a-fA-F0-9][a-fA-F0-9]))+)|(? | |
# | | | :[+](?:APOP|(?:(?:[-a-zA-Z0-9$_.+!*'(),&=~]+|(?:%[a-f | |
# | | | A-F0-9][a-fA-F0-9]))+))))))?#)?(?:(?:(?:(?:(?:(?:[a-z | |
# | | | A-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0-9])[.])*(?:[a-zA-Z][-a | |
# | | | -zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z]))|(?:[0-9]+[.][0-9]+[. | |
# | | | ][0-9]+[.][0-9]+)))(?::(?:(?:[0-9]+)))?)|(?:(?:prospe | |
# | | | ro)://(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a- | |
# | | | zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a- | |
# | | | zA-Z]))|(?:[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(? | |
# | | | :(?:[0-9]+)))?/(?:(?:(?:(?:[-a-zA-Z0-9$_.+!*'(),?:#&= | |
# | | | ]|(?:%[a-fA-F0-9][a-fA-F0-9]))*)(?:/(?:(?:[-a-zA-Z0-9 | |
# | | | $_.+!*'(),?:#&=]|(?:%[a-fA-F0-9][a-fA-F0-9]))*))*))(? | |
# | | | :(?:;(?:(?:[-a-zA-Z0-9$_.+!*'(),?:#&]|(?:%[a-fA-F0-9] | |
# | | | [a-fA-F0-9]))*)=(?:(?:[-a-zA-Z0-9$_.+!*'(),?:#&]|(?:% | |
# | | | [a-fA-F0-9][a-fA-F0-9]))*))*))|(?:(?:wais)://(?:(?:(? | |
# | | | :(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0-9])[.])* | |
# | | | (?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z]))|(?:[0- | |
# | | | 9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(?:(?:[0-9]+)))? | |
# | | | /(?:(?:(?:(?:[-a-zA-Z0-9$_.+!*'(),]|(?:%[a-fA-F0-9][a | |
# | | | -fA-F0-9]))*))(?:[?](?:(?:(?:[-a-zA-Z0-9$_.+!*'(),;:# | |
# | | | &=]|(?:%[a-fA-F0-9][a-fA-F0-9]))*))|/(?:(?:(?:[-a-zA- | |
# | | | Z0-9$_.+!*'(),]|(?:%[a-fA-F0-9][a-fA-F0-9]))*))/(?:(? | |
# | | | :(?:[-a-zA-Z0-9$_.+!*'(),]|(?:%[a-fA-F0-9][a-fA-F0-9] | |
# | | | ))*)))?))|(?:(?:tel):(?:(?:(?:[+](?:[0-9\-.()]+)(?:;i | |
# | | | sub=[0-9\-.()]+)?(?:;postd=[0-9\-.()*#ABCDwp]+)?(?:(? | |
# | | | :;(?:phone-context)=(?:(?:(?:[+][0-9\-.()]+)|(?:[0-9\ | |
# | | | -.()*#ABCDwp]+))|(?:(?:[!'E-OQ-VX-Z_e-oq-vx-z~]|(?:%( | |
# | | | ?:2[124-7CFcf]|3[AC-Fac-f]|4[05-9A-Fa-f]|5[1-689A-Fa- | |
# | | | f]|6[05-9A-Fa-f]|7[1-689A-Ea-e])))(?:[!'()*\-.0-9A-Z_ | |
# | | | a-z~]+|(?:%(?:2[1-9A-Fa-f]|3[AC-Fac-f]|[4-6][0-9A-Fa- | |
# | | | f]|7[0-9A-Ea-e])))*)))|(?:;(?:tsp)=(?: |(?:(?:(?:[A-Z | |
# | | | a-z](?:(?:(?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?)(?:[. | |
# | | | ](?:[A-Za-z](?:(?:(?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9] | |
# | | | )?))*))))|(?:;(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7AB | |
# | | | DEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[ | |
# | | | 0-9ACEace]))*)(?:=(?:(?:(?:(?:[!'*\-.0-9A-Z_a-z~]+|%( | |
# | | | ?:2[13-7ABDEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9 | |
# | | | A-Fa-f]|7[0-9ACEace]))*)(?:[?](?:(?:[!'*\-.0-9A-Z_a-z | |
# | | | ~]+|%(?:2[13-7ABDEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef] | |
# | | | |6[0-9A-Fa-f]|7[0-9ACEace]))*))?)|(?:%22(?:(?:%5C(?:[ | |
# | | | a-zA-Z0-9\-_.!~*'()]|(?:%[a-fA-F0-9][a-fA-F0-9])))|[a | |
# | | | -zA-Z0-9\-_.!~*'()]+|(?:%(?:[01][a-fA-F0-9])|2[013-9A | |
# | | | -Fa-f]|[3-9A-Fa-f][a-fA-F0-9]))*%22)))?))*)|(?:[0-9\- | |
# | | | .()*#ABCDwp]+(?:;isub=[0-9\-.()]+)?(?:;postd=[0-9\-.( | |
# | | | )*#ABCDwp]+)?(?:;(?:phone-context)=(?:(?:(?:[+][0-9\- | |
# | | | .()]+)|(?:[0-9\-.()*#ABCDwp]+))|(?:(?:[!'E-OQ-VX-Z_e- | |
# | | | oq-vx-z~]|(?:%(?:2[124-7CFcf]|3[AC-Fac-f]|4[05-9A-Fa- | |
# | | | f]|5[1-689A-Fa-f]|6[05-9A-Fa-f]|7[1-689A-Ea-e])))(?:[ | |
# | | | !'()*\-.0-9A-Z_a-z~]+|(?:%(?:2[1-9A-Fa-f]|3[AC-Fac-f] | |
# | | | |[4-6][0-9A-Fa-f]|7[0-9A-Ea-e])))*)))(?:(?:;(?:phone- | |
# | | | context)=(?:(?:(?:[+][0-9\-.()]+)|(?:[0-9\-.()*#ABCDw | |
# | | | p]+))|(?:(?:[!'E-OQ-VX-Z_e-oq-vx-z~]|(?:%(?:2[124-7CF | |
# | | | cf]|3[AC-Fac-f]|4[05-9A-Fa-f]|5[1-689A-Fa-f]|6[05-9A- | |
# | | | Fa-f]|7[1-689A-Ea-e])))(?:[!'()*\-.0-9A-Z_a-z~]+|(?:% | |
# | | | (?:2[1-9A-Fa-f]|3[AC-Fac-f]|[4-6][0-9A-Fa-f]|7[0-9A-E | |
# | | | a-e])))*)))|(?:;(?:tsp)=(?: |(?:(?:(?:[A-Za-z](?:(?:( | |
# | | | ?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?)(?:[.](?:[A-Za-z | |
# | | | ](?:(?:(?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?))*))))|( | |
# | | | ?:;(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]|3[0 | |
# | | | -9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEace]) | |
# | | | )*)(?:=(?:(?:(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABD | |
# | | | Eabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0 | |
# | | | -9ACEace]))*)(?:[?](?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[1 | |
# | | | 3-7ABDEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa- | |
# | | | f]|7[0-9ACEace]))*))?)|(?:%22(?:(?:%5C(?:[a-zA-Z0-9\- | |
# | | | _.!~*'()]|(?:%[a-fA-F0-9][a-fA-F0-9])))|[a-zA-Z0-9\-_ | |
# | | | .!~*'()]+|(?:%(?:[01][a-fA-F0-9])|2[013-9A-Fa-f]|[3-9 | |
# | | | A-Fa-f][a-fA-F0-9]))*%22)))?))*))))|(?:(?:ftp)://(?:( | |
# | | | ?:(?:(?:[a-zA-Z0-9\-_.!~*'();:&=+$,]+|(?:%[a-fA-F0-9] | |
# | | | [a-fA-F0-9]))*))(?:)#)?(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][ | |
# | | | -a-zA-Z0-9]*)?[a-zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9 | |
# | | | ]*[a-zA-Z0-9]|[a-zA-Z])[.]?)|(?:[0-9]+[.][0-9]+[.][0- | |
# | | | 9]+[.][0-9]+)))(?::(?:(?:[0-9]*)))?(?:/(?:(?:(?:(?:(? | |
# | | | :[a-zA-Z0-9\-_.!~*'():#&=+$,]+|(?:%[a-fA-F0-9][a-fA-F | |
# | | | 0-9]))*)(?:/(?:(?:[a-zA-Z0-9\-_.!~*'():#&=+$,]+|(?:%[ | |
# | | | a-fA-F0-9][a-fA-F0-9]))*))*))(?:;type=(?:[AIai]))?))? | |
# | | | )|(?:(?:tv):(?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)? | |
# | | | [a-zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]| | |
# | | | [a-zA-Z])[.]?))?)|(?:(?:news):(?:(?:[*]|(?:(?:[-a-zA- | |
# | | | Z0-9$_.+!*'(),;/?:&=]|(?:%[a-fA-F0-9][a-fA-F0-9]))+#( | |
# | | | ?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0-9])[ | |
# | | | .])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z]))|(? | |
# | | | :[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))|(?:[a-zA-Z][-A- | |
# | | | Za-z0-9.+_]*))))|(?:(?:fax):(?:(?:(?:[+](?:[0-9\-.()] | |
# | | | +)(?:;isub=[0-9\-.()]+)?(?:;tsub=[0-9\-.()]+)?(?:;pos | |
# | | | td=[0-9\-.()*#ABCDwp]+)?(?:(?:;(?:phone-context)=(?:( | |
# | | | ?:(?:[+][0-9\-.()]+)|(?:[0-9\-.()*#ABCDwp]+))|(?:(?:[ | |
# | | | !'E-OQ-VX-Z_e-oq-vx-z~]|(?:%(?:2[124-7CFcf]|3[AC-Fac- | |
# | | | f]|4[05-9A-Fa-f]|5[1-689A-Fa-f]|6[05-9A-Fa-f]|7[1-689 | |
# | | | A-Ea-e])))(?:[!'()*\-.0-9A-Z_a-z~]+|(?:%(?:2[1-9A-Fa- | |
# | | | f]|3[AC-Fac-f]|[4-6][0-9A-Fa-f]|7[0-9A-Ea-e])))*)))|( | |
# | | | ?:;(?:tsp)=(?: |(?:(?:(?:[A-Za-z](?:(?:(?:[-A-Za-z0-9 | |
# | | | ]+)){0,61}[A-Za-z0-9])?)(?:[.](?:[A-Za-z](?:(?:(?:[-A | |
# | | | -Za-z0-9]+)){0,61}[A-Za-z0-9])?))*))))|(?:;(?:(?:[!'* | |
# | | | \-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]|3[0-9]|4[1-9A-Fa | |
# | | | -f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEace]))*)(?:=(?:(?: | |
# | | | (?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]|3[0-9] | |
# | | | |4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEace]))*) | |
# | | | (?:[?](?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]| | |
# | | | 3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEac | |
# | | | e]))*))?)|(?:%22(?:(?:%5C(?:[a-zA-Z0-9\-_.!~*'()]|(?: | |
# | | | %[a-fA-F0-9][a-fA-F0-9])))|[a-zA-Z0-9\-_.!~*'()]+|(?: | |
# | | | %(?:[01][a-fA-F0-9])|2[013-9A-Fa-f]|[3-9A-Fa-f][a-fA- | |
# | | | F0-9]))*%22)))?))*)|(?:[0-9\-.()*#ABCDwp]+(?:;isub=[0 | |
# | | | -9\-.()]+)?(?:;tsub=[0-9\-.()]+)?(?:;postd=[0-9\-.()* | |
# | | | #ABCDwp]+)?(?:;(?:phone-context)=(?:(?:(?:[+][0-9\-.( | |
# | | | )]+)|(?:[0-9\-.()*#ABCDwp]+))|(?:(?:[!'E-OQ-VX-Z_e-oq | |
# | | | -vx-z~]|(?:%(?:2[124-7CFcf]|3[AC-Fac-f]|4[05-9A-Fa-f] | |
# | | | |5[1-689A-Fa-f]|6[05-9A-Fa-f]|7[1-689A-Ea-e])))(?:[!' | |
# | | | ()*\-.0-9A-Z_a-z~]+|(?:%(?:2[1-9A-Fa-f]|3[AC-Fac-f]|[ | |
# | | | 4-6][0-9A-Fa-f]|7[0-9A-Ea-e])))*)))(?:(?:;(?:phone-co | |
# | | | ntext)=(?:(?:(?:[+][0-9\-.()]+)|(?:[0-9\-.()*#ABCDwp] | |
# | | | +))|(?:(?:[!'E-OQ-VX-Z_e-oq-vx-z~]|(?:%(?:2[124-7CFcf | |
# | | | ]|3[AC-Fac-f]|4[05-9A-Fa-f]|5[1-689A-Fa-f]|6[05-9A-Fa | |
# | | | -f]|7[1-689A-Ea-e])))(?:[!'()*\-.0-9A-Z_a-z~]+|(?:%(? | |
# | | | :2[1-9A-Fa-f]|3[AC-Fac-f]|[4-6][0-9A-Fa-f]|7[0-9A-Ea- | |
# | | | e])))*)))|(?:;(?:tsp)=(?: |(?:(?:(?:[A-Za-z](?:(?:(?: | |
# | | | [-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?)(?:[.](?:[A-Za-z]( | |
# | | | ?:(?:(?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?))*))))|(?: | |
# | | | ;(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]|3[0-9 | |
# | | | ]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEace]))* | |
# | | | )(?:=(?:(?:(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEa | |
# | | | bde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9 | |
# | | | ACEace]))*)(?:[?](?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13- | |
# | | | 7ABDEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f] | |
# | | | |7[0-9ACEace]))*))?)|(?:%22(?:(?:%5C(?:[a-zA-Z0-9\-_. | |
# | | | !~*'()]|(?:%[a-fA-F0-9][a-fA-F0-9])))|[a-zA-Z0-9\-_.! | |
# | | | ~*'()]+|(?:%(?:[01][a-fA-F0-9])|2[013-9A-Fa-f]|[3-9A- | |
# | | | Fa-f][a-fA-F0-9]))*%22)))?))*))))|(?:(?:nntp)://(?:(? | |
# | | | :(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0 | |
# | | | -9])[.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z] | |
# | | | ))|(?:[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(?:(?:[ | |
# | | | 0-9]+)))?)/(?:(?:[a-zA-Z][-A-Za-z0-9.+_]*))(?:/(?:[0- | |
# | | | 9]+))?))) | |
# +-----+----+-------------------------------------------------------+-----+
Can I do anything to make the output look something like:
# +-----+----+-------------------------------------------------------+-----+
# | GOT | OP | CHECK | LNs |
# +-----+----+-------------------------------------------------------+-----+
# | foo | =~ | $RE{URI} | 1 |
# +-----+----+-------------------------------------------------------+-----+

Seems like it is not possible to format the check message. Looking at the source:
Test2::Tools::Compare::is() calls
Test2::Compare::Delta::diag() which calls
Test2::Compare::Delta::table() to format the output.
You may try ask for a new feature at the GitHub issue tracker.

How can I make my dataframe sample in each run using apache spark

I have a peculiar scenario in which the sample obtained in two consecutive samplings are not consistent even when I've provided a seed value. I'm using the following code (Which was an outcome of a discussion here:
var conversionSample = sortedConversionSubset.sample(true, (sampleSize + 0.05), 3*x).limit((conversionCount * sampleSize).toInt)
var nonConversionSample = sortedNonConversionSubset.sample(true, (sampleSize + 0.05), 3*x).limit((nonConversionCount * sampleSize).toInt)
Here
'sampleSize' is a constant fraction value less than 0.8
'x' is a constant int, which represents xth iteration in a for loop
'conversionCount' and 'nonConversionCount' are int values representing number of rows in each subset
Now the observation being that in two successive runs the sample generated is different in both cases which was not the expected behavior.
sortedConversionSubset
+--------------------------------------+----------+
|clientid |Conversion|
+--------------------------------------+----------+
|02438b66-2de4-4765-bae3-de7453647ea7_1|1 |
|203865ed-f02a-4ed9-9098-82691de707a4_0|1 |
|203865ed-f02a-4ed9-9098-82691de707a4_1|1 |
|674e2337-aec5-434e-b56e-8c2efcc42894_1|1 |
|6d6036d3-c161-4f5d-8557-80b85dd87bd9_0|1 |
|6d6036d3-c161-4f5d-8557-80b85dd87bd9_1|1 |
|7797aba3-3eea-4556-856e-753812b4b551_0|1 |
|7797aba3-3eea-4556-856e-753812b4b551_1|1 |
|870ab2a5-0650-42b8-9e6f-bde3859f64fd_0|1 |
|870ab2a5-0650-42b8-9e6f-bde3859f64fd_1|1 |
|9b606693-4ffa-44a5-bd7c-cc6974ce3e83_0|1 |
|be218b72-c664-40cf-adf5-e3519095e941_0|1 |
|e7dc7fd9-32df-46a1-b3bd-793bbda09f6f_0|1 |
|eaf434da-6a8f-4ab0-a744-62bea663ed5e_0|1 |
|eaf434da-6a8f-4ab0-a744-62bea663ed5e_1|1 |
+--------------------------------------+----------+
sortedNonConversionSubset
+--------------------------------------+----------+
|clientid |Conversion|
+--------------------------------------+----------+
|03358d8f-9b9c-4258-9c99-234ab102c29b_1|0 |
|040d213c-e91a-42f4-9bf7-90671670dc17_0|0 |
|04fe5148-1c56-4c88-aed0-1f01220bffd6_0|0 |
|0ed2e621-9ba4-46f0-8793-a84d32538c39_0|0 |
|0f9bcf42-e7fa-49a0-9d75-6c9bbc38b4d5_0|0 |
|108c5478-abc0-44d9-968b-47f81c4f5a37_0|0 |
|129eb883-159d-49be-b8ae-9aa44a3e2919_0|0 |
|13e3d779-026b-4d12-8619-aa5fe6ca99ed_0|0 |
|14497295-eebd-44aa-9f26-fc5e4810fb54_0|0 |
|1855d96d-3647-4c4f-a20f-7e46f7635798_0|0 |
|1911caf0-a470-4898-9b62-57c604422727_0|0 |
|1b91b8dc-09b8-47e2-b892-f5c14b650019_0|0 |
|1dfa820c-77e0-4927-8a39-ecd8e842b09b_0|0 |
|1e48e346-4ada-4a8d-896b-7658cc2499cd_0|0 |
|252be902-4204-40a5-9d3c-dd3a7d0f0355_0|0 |
|2995b49d-525b-43e9-ab36-8b8910a4607c_0|0 |
|2bc06b59-4624-4ddd-87a3-ed04cba88233_0|0 |
|2d4538a5-20e6-4742-ae46-aad0a5ed3fff_0|0 |
|31563716-9380-4662-90e5-7f63a1ab9072_0|0 |
|34442a3e-0437-4c41-86fb-1ac55062993a_0|0 |
|35151629-2f86-4917-90d2-42daa5ae4f5c_0|0 |
|3c37e066-dff5-4bd9-84ab-b9e73f3f3fdd_0|0 |
|3e998096-3a4b-4b57-a1de-69d2dbd19abd_0|0 |
|3f8ace3c-d378-4423-97a0-3d9cf35ba256_0|0 |
|49a0cfb8-490f-4252-84fa-2b9e250e9333_0|0 |
|4c3f11fa-e3ba-4eb1-977a-06f034bf8a54_0|0 |
|4ee484f4-e877-44c3-9390-c4e4072c5dee_0|0 |
|4fa035b3-dcd5-40e1-9107-0a0c943ff597_1|0 |
|529704d2-5a60-4718-a03f-639e040f6634_0|0 |
|560f6978-028b-4a37-9f97-d97e93976bf7_0|0 |
|57b47c74-b071-4278-89c9-f7b4cb1225d1_0|0 |
|58305773-f944-4039-8452-f5eb8d62f0cf_0|0 |
|58dfa9dd-43cf-4eb7-ade6-7235004a9815_0|0 |
|5b146218-9bb6-46f0-8c83-df131d78f591_0|0 |
|5ca3b5bc-35a9-42a5-bd37-a8fc94366dc6_0|0 |
|5d5f2ea0-aed9-4c2d-8c22-68859ec35e8e_0|0 |
|5f9ebf92-3b1b-4628-b949-44a32e6d3659_0|0 |
|64822b8c-009e-48ab-b6ca-1a7ece1106fa_0|0 |
|6b352714-af74-4773-854b-073e644e8684_0|0 |
|6e528e49-472e-48c7-baa9-edc25303e427_0|0 |
|73203f58-8be2-4716-b8f0-79c64400c57b_0|0 |
|741630e0-1c99-497d-a127-5c4c562952c5_0|0 |
|778e3b8a-2ca5-469a-9697-f646962e8308_0|0 |
|8029c542-d933-43fb-b359-f2438dcd5660_0|0 |
|8b06ba24-2af3-4eec-811a-4d1779f37876_0|0 |
|8fb43dff-260d-4ece-85e2-3bc2cb636ac1_0|0 |
|90f8a4cb-1956-43c4-ac7d-8c6514cd023a_0|0 |
|916f2e2a-6135-4004-8d54-d80b822ce394_0|0 |
|968a7ca3-1649-4586-9e60-b7e8565e708a_0|0 |
|a32782cc-8c4c-403b-aa83-09f1cec45fdb_0|0 |
|a63f44d5-a4d5-45a0-8a4b-cebf05df810b_0|0 |
|a6f958bc-e050-4216-b981-d51f1c0ff60d_0|0 |
|a7dba1bb-d7ff-44e6-9c4c-997ae59a2337_1|0 |
|ac33d675-d9cc-43b5-94fb-7d412773db14_0|0 |
|b1227816-9bf2-474f-8e82-5739acf6c895_0|0 |
|b1c27a2e-6efc-4869-880b-9ce0a4962edc_0|0 |
|b4ff6d43-cf0a-4f1d-9431-1edcb8ee1fb6_0|0 |
|b9e477ab-2065-42bb-832b-5d0e98ee05c7_0|0 |
|ba8c4efe-e71c-468c-b1bf-37efff596907_0|0 |
|c21eefc8-43d0-4be0-a252-b9fc4dbb7ad0_0|0 |
|c3785311-87c8-43bc-99a8-01d64f5eaa87_0|0 |
|c543bde7-deb8-4484-b0be-353c44baf6eb_1|0 |
|ca31e550-9d28-4628-bfe8-53648a2007f7_0|0 |
|cbc33697-20cb-4f8b-accd-0a6396a4ea41_0|0 |
|cc7810aa-08fc-44e7-acdc-ac948a28f9b9_0|0 |
|d1efdc7c-afb0-4995-bbbd-a76f731d2492_0|0 |
|d6a4b928-e576-41d7-9628-18709765199d_0|0 |
|d7311ec7-6c50-448d-8a6e-f690c3070d57_1|0 |
|d86b09f9-70a0-4101-a13b-129fe3a37b86_0|0 |
|d911be5b-aceb-45c8-a79e-73ccfa1b96f0_0|0 |
|db0c7b10-80f7-4071-aa53-fe0e2dc5ebce_0|0 |
|dce14c51-fa57-4e98-987d-708e2a9aa293_0|0 |
|dd026fb8-f818-4d1e-aaa4-4c9b3fd24994_0|0 |
|dfa9c55c-1e75-4010-be86-a6b1eb723672_0|0 |
|ea29f600-9e85-40f4-9f88-dcef46beb0c1_0|0 |
|eb5e58fc-eaac-4059-8ebc-1fab1ccf3555_1|0 |
|eb7568ab-83ac-45a7-bf4b-3b048d6c7c53_0|0 |
|f5b1cfc4-e397-4699-adab-0af6ee0e1b76_0|0 |
|facbfc8c-d477-4b27-bf15-52a56c26cbf6_0|0 |
|ffd03bca-ef40-4fa4-913e-73c002f29796_0|0 |
+--------------------------------------+----------+
1st Run Sample
+--------------------------------------+----------+
|clientid |Conversion|
+--------------------------------------+----------+
|203865ed-f02a-4ed9-9098-82691de707a4_1|1 |
|6d6036d3-c161-4f5d-8557-80b85dd87bd9_0|1 |
|6d6036d3-c161-4f5d-8557-80b85dd87bd9_1|1 |
|02438b66-2de4-4765-bae3-de7453647ea7_1|1 |
|7797aba3-3eea-4556-856e-753812b4b551_0|1 |
|870ab2a5-0650-42b8-9e6f-bde3859f64fd_0|1 |
|1dfa820c-77e0-4927-8a39-ecd8e842b09b_0|0 |
|252be902-4204-40a5-9d3c-dd3a7d0f0355_0|0 |
|2995b49d-525b-43e9-ab36-8b8910a4607c_0|0 |
|2bc06b59-4624-4ddd-87a3-ed04cba88233_0|0 |
|31563716-9380-4662-90e5-7f63a1ab9072_0|0 |
|5ca3b5bc-35a9-42a5-bd37-a8fc94366dc6_0|0 |
|5d5f2ea0-aed9-4c2d-8c22-68859ec35e8e_0|0 |
|5f9ebf92-3b1b-4628-b949-44a32e6d3659_0|0 |
|5f9ebf92-3b1b-4628-b949-44a32e6d3659_0|0 |
|5f9ebf92-3b1b-4628-b949-44a32e6d3659_0|0 |
|6b352714-af74-4773-854b-073e644e8684_0|0 |
|6e528e49-472e-48c7-baa9-edc25303e427_0|0 |
|6e528e49-472e-48c7-baa9-edc25303e427_0|0 |
|741630e0-1c99-497d-a127-5c4c562952c5_0|0 |
|03358d8f-9b9c-4258-9c99-234ab102c29b_1|0 |
|040d213c-e91a-42f4-9bf7-90671670dc17_0|0 |
|040d213c-e91a-42f4-9bf7-90671670dc17_0|0 |
|04fe5148-1c56-4c88-aed0-1f01220bffd6_0|0 |
|129eb883-159d-49be-b8ae-9aa44a3e2919_0|0 |
|1855d96d-3647-4c4f-a20f-7e46f7635798_0|0 |
|3c37e066-dff5-4bd9-84ab-b9e73f3f3fdd_0|0 |
|3e998096-3a4b-4b57-a1de-69d2dbd19abd_0|0 |
|3f8ace3c-d378-4423-97a0-3d9cf35ba256_0|0 |
|49a0cfb8-490f-4252-84fa-2b9e250e9333_0|0 |
|4fa035b3-dcd5-40e1-9107-0a0c943ff597_1|0 |
|4fa035b3-dcd5-40e1-9107-0a0c943ff597_1|0 |
|529704d2-5a60-4718-a03f-639e040f6634_0|0 |
|560f6978-028b-4a37-9f97-d97e93976bf7_0|0 |
|560f6978-028b-4a37-9f97-d97e93976bf7_0|0 |
|560f6978-028b-4a37-9f97-d97e93976bf7_0|0 |
|778e3b8a-2ca5-469a-9697-f646962e8308_0|0 |
|8b06ba24-2af3-4eec-811a-4d1779f37876_0|0 |
+--------------------------------------+----------+
2nd Run Sample
+--------------------------------------+----------+
|clientid |Conversion|
+--------------------------------------+----------+
|02438b66-2de4-4765-bae3-de7453647ea7_1|1 |
|7797aba3-3eea-4556-856e-753812b4b551_0|1 |
|870ab2a5-0650-42b8-9e6f-bde3859f64fd_0|1 |
|870ab2a5-0650-42b8-9e6f-bde3859f64fd_1|1 |
|be218b72-c664-40cf-adf5-e3519095e941_0|1 |
|be218b72-c664-40cf-adf5-e3519095e941_0|1 |
|1dfa820c-77e0-4927-8a39-ecd8e842b09b_0|0 |
|252be902-4204-40a5-9d3c-dd3a7d0f0355_0|0 |
|2995b49d-525b-43e9-ab36-8b8910a4607c_0|0 |
|2bc06b59-4624-4ddd-87a3-ed04cba88233_0|0 |
|31563716-9380-4662-90e5-7f63a1ab9072_0|0 |
|5ca3b5bc-35a9-42a5-bd37-a8fc94366dc6_0|0 |
|5d5f2ea0-aed9-4c2d-8c22-68859ec35e8e_0|0 |
|5f9ebf92-3b1b-4628-b949-44a32e6d3659_0|0 |
|5f9ebf92-3b1b-4628-b949-44a32e6d3659_0|0 |
|5f9ebf92-3b1b-4628-b949-44a32e6d3659_0|0 |
|6b352714-af74-4773-854b-073e644e8684_0|0 |
|6e528e49-472e-48c7-baa9-edc25303e427_0|0 |
|6e528e49-472e-48c7-baa9-edc25303e427_0|0 |
|741630e0-1c99-497d-a127-5c4c562952c5_0|0 |
|03358d8f-9b9c-4258-9c99-234ab102c29b_1|0 |
|040d213c-e91a-42f4-9bf7-90671670dc17_0|0 |
|040d213c-e91a-42f4-9bf7-90671670dc17_0|0 |
|04fe5148-1c56-4c88-aed0-1f01220bffd6_0|0 |
|129eb883-159d-49be-b8ae-9aa44a3e2919_0|0 |
|1855d96d-3647-4c4f-a20f-7e46f7635798_0|0 |
|3c37e066-dff5-4bd9-84ab-b9e73f3f3fdd_0|0 |
|3e998096-3a4b-4b57-a1de-69d2dbd19abd_0|0 |
|3f8ace3c-d378-4423-97a0-3d9cf35ba256_0|0 |
|49a0cfb8-490f-4252-84fa-2b9e250e9333_0|0 |
|4fa035b3-dcd5-40e1-9107-0a0c943ff597_1|0 |
|4fa035b3-dcd5-40e1-9107-0a0c943ff597_1|0 |
|529704d2-5a60-4718-a03f-639e040f6634_0|0 |
|560f6978-028b-4a37-9f97-d97e93976bf7_0|0 |
|560f6978-028b-4a37-9f97-d97e93976bf7_0|0 |
|560f6978-028b-4a37-9f97-d97e93976bf7_0|0 |
|778e3b8a-2ca5-469a-9697-f646962e8308_0|0 |
|8b06ba24-2af3-4eec-811a-4d1779f37876_0|0 |
+--------------------------------------+----------+
The two samples being different could be a road blocker for me and just want to check how I could make these consistent

Emacs Orgmode table $> References does not work

GNU Emacs 24.4.1 org-mode
Here is an org-mode table
#+TBLNAME: revenue
| / | < | | < | | < | | | | | | | | | | | |
| Product | Year_SUM | Month_SUM | Platform | Platform_SUM | adwo | AdMob | adChina | adSage | appfigures | appdriver | coco | Domob | Dianru | Limei | guohead | youmi |
| | | | | | | | | | | | | | | | | |
|---------+----------+-----------+----------+------------------+------+-------+---------+--------+------------+-----------+------+-------+--------+-------+---------+-------|
| Jan | | | iOS | #ERROR | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| | | | Android | =vsum($6..$>);NE | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 |
|---------+----------+-----------+----------+------------------+------+-------+---------+--------+------------+-----------+------+-------+--------+-------+---------+-------|
| | | | | | | | | | | | | | | | | |
#+TBLFM: $5=vsum($6..$>);NE
As you see ,the formula $5=vsum($6..$>);NE can't be calculated! Here is debug info:
Substitution history of formula
Orig: vsum($6..$>)
$xyz-> vsum($6..$>)
#r$c-> vsum($6..$>)
$1-> vsum((0)..$>)
--------^
Error: Expected `)'
But if I replace the formula with $5=vsum($6..$17) and then it works ,I can't figure out where is the problem?
I need some help ,appreciate it!

Comonad example in Scala

What is Comonad, if it's possible describe in Scala syntax.
I found scalaz library implementation, but it's not clear where it can be useful.

Well, monads allow you to add values to them, change them based on a computation from a non-monad to a monad. Comonads allow you to extract values from them, and change them based on a computation from a comonad to a non-comonad.
The natural intuition is that they'll usually appear where you have a CM[A] and want to extract A.
See this very interesting post that touches on comonads a bit casually, but, to me at least, making them very clear.

What follows is a literal translation of code from this blog post.
case class U[X](left: Stream[X], center: X, right: Stream[X]) {
def shiftRight = this match {
case U(a, b, c #:: cs) => U(b #:: a, c, cs)
}
def shiftLeft = this match {
case U(a #:: as, b, c) => U(as, a, b #:: c)
}
}
// Not necessary, as Comonad also has fmap.
/*
implicit object uFunctor extends Functor[U] {
def fmap[A, B](x: U[A], f: A => B): U[B] = U(x.left.map(f), f(x.center), x.right.map(f))
}
*/
implicit object uComonad extends Comonad[U] {
def copure[A](u: U[A]): A = u.center
def cojoin[A](a: U[A]): U[U[A]] = U(Stream.iterate(a)(_.shiftLeft).tail, a, Stream.iterate(a)(_.shiftRight).tail)
def fmap[A, B](x: U[A], f: A => B): U[B] = U(x.left.map(f), x.center |> f, x.right.map(f))
}
def rule(u: U[Boolean]) = u match {
case U(a #:: _, b, c #:: _) => !(a && b && !c || (a == b))
}
def shift[A](i: Int, u: U[A]) = {
Stream.iterate(u)(x => if (i < 0) x.shiftLeft else x.shiftRight).apply(i.abs)
}
def half[A](u: U[A]) = u match {
case U(_, b, c) => Stream(b) ++ c
}
def toList[A](i: Int, j: Int, u: U[A]) = half(shift(i, u)).take(j - i)
val u = U(Stream continually false, true, Stream continually false)
val s = Stream.iterate(u)(_ =>> rule)
val s0 = s.map(r => toList(-20, 20, r).map(x => if(x) '#' else ' '))
val s1 = s.map(r => toList(-20, 20, r).map(x => if(x) '#' else ' ').mkString("|")).take(20).force.mkString("\n")
println(s1)
Output:
| | | | | | | | | | | | | | | | | | | |#| | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#| | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| |#| | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#|#|#| | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| | | |#| | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#| | |#|#| | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| |#| |#| |#| | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#|#|#|#|#|#|#| | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| | | | | | | |#| | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#| | | | | | |#|#| | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| |#| | | | | |#| |#| | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#|#|#| | | | |#|#|#|#| | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| | | |#| | | |#| | | |#| | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#| | |#|#| | |#|#| | |#|#| | | | | |
| | | | | | | | | | | | | | | | | | | |#| |#| |#| |#| |#| |#| |#| |#| | | | |
| | | | | | | | | | | | | | | | | | | |#|#|#|#|#|#|#|#|#|#|#|#|#|#|#|#| | | |
| | | | | | | | | | | | | | | | | | | |#| | | | | | | | | | | | | | | |#| | |
| | | | | | | | | | | | | | | | | | | |#|#| | | | | | | | | | | | | | |#|#| |
| | | | | | | | | | | | | | | | | | | |#| |#| | | | | | | | | | | | | |#| |#|
| | | | | | | | | | | | | | | | | | | |#|#|#|#| | | | | | | | | | | | |#|#|#|#

The scalaz library provides a ComonadStore which extends the property of Comonad. It is defined like this:
trait ComonadStore[F[_], S] extends Comonad[F] { self =>
def pos[A](w: F[A]): S
def peek[A](s: S, w: F[A]): A
def peeks[A](s: S => S, w: F[A]): A =
peek(s(pos(w)), w)
def seek[A](s: S, w: F[A]): F[A] =
peek(s, cojoin(w))
def seeks[A](s: S => S, w: F[A]): F[A] =
peeks(s, cojoin(w))
def experiment[G[_], A](s: S => G[S], w: F[A])(implicit FG: Functor[G]): G[A] =
FG.map(s(pos(w)))(peek(_, w))
}
And a Store( which is analogous to (S => A, S)) has an instance of Comonad. You can look at this question that explains what it is more specifically.
You also have the Coreader and Cowriter Comonads that are the dual of the Reader and Writer Monads, here is an excellent blog post that talks about it in Scala.

Per-table constants are not being substituted

Here is my table
| | Name | Vert | Horz | Area | Cost | USD |
|---+-------------+---------+-------+--------+------+-----|
| $ | $price = 75 | $Hi=2.9 | | | | |
| # | Kitchen | 4.160 | 3.630 | #ERROR | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
#+TBLFM: $5=$4*$Hi*2+$3*$Hi*2
Here is trace output:
Substitution history of formula
Orig: $4*$Hi*2+$3*$Hi*2
$xyz-> $4*(#UNDEFINED_NAME)*2+$3*(#UNDEFINED_NAME)*2
#r$c-> $4*(#UNDEFINED_NAME)*2+$3*(#UNDEFINED_NAME)*2
$1-> (3.630)*(#UNDEFINED_NAME)*2+(4.160)*(#UNDEFINED_NAME)*2
---------^
Error: #'s not allowed in this context
What's wrong? Why $Hi was not substituted?

Ah, found it by myself. Here is wrong row, and the fixed row:
| $ | $price = 75 | $Hi=2.9 | | | | |
| $ | price = 75 | Hi=2.9 | | | | |