What is | in diff output? - diff
I am diff'ing two .md5 files with diff -yb --suppress-common-lines --width=250
and getting
0b92397d4978b7b5ba1ae2d4be0ca639 1.__Ms._Eldoris_McCondichie_3-5-04-Apple_ProRes_422_for_Interlaced_material_copy | 66c0a190ccf79e6ca1e34b86bfe89788 1. Ms. Eldoris McCondichie 3:5:04.mov
66c0a190ccf79e6ca1e34b86bfe89788 1.__Ms._Eldoris_McCondichie_3-5-04.mov | 7519e0c6d5f8f56be15b5c6eb82f1678 10. Mr. Clyde Eddy.mov
9ca5150c4f399ad58aae1e4a4a97f809 10._Mr._Clyde_Eddy-Apple_ProRes_422_for_Interlaced_material_copy.mov | 1eedc2c35fdbbc033dbc7bea8b3e4c0d 11. Ms. Jewel Smitherman Rogers 1a w TC.mov
7519e0c6d5f8f56be15b5c6eb82f1678 10._Mr._Clyde_Eddy.mov | 535acdb76a8f56f50bad418c8ff18ec7 12. Ms. Jewel Smitherman Rogers 1b w TC.mov
11411fac492a50e7692669849281dd8a 11._Ms._Jewel_Smitherman_Rogers_1a_w_TC-Apple_ProRes_422_for_Interlaced_material | 2f5d941751409dfd407e8f181aa745c6 13. Ms. Jewel Smitherman Rogers 2 w TC.mov
1eedc2c35fdbbc033dbc7bea8b3e4c0d 11._Ms._Jewel_Smitherman_Rogers_1a_w_TC.mov | 4a0c342444fedb2096c8495fae5e1459 14. Ms. Thelma Knight w TC.mov
acf43bd1b507f1370238dd9d7855f177 12._Ms._Jewel_Smitherman_Rogers_1b_w_TC-Apple_ProRes_422_for_Interlaced_material | 56b0d7952f01b48f47d90a5c300411ef 15. Mr. Robert Holloway w TC.mov
535acdb76a8f56f50bad418c8ff18ec7 12._Ms._Jewel_Smitherman_Rogers_1b_w_TC.mov | 3117d375927b0032f7b804d1f272f97a 16. Mr. Archie Franklin w TC.mov
1e2c9a47ef1ae1869e35e1c439af054f 13._Ms._Jewel_Smitherman_Rogers_2_w_TC-Apple_ProRes_422_for_Interlaced_material_ | 552230c7e504e0f3fa819a46b8169bd4 17. Mr. John Hope Franklin by Charles Ogletree.mov
2f5d941751409dfd407e8f181aa745c6 13._Ms._Jewel_Smitherman_Rogers_2_w_TC.mov | 933b65571efefd2ea1e642f478f4cc94 18. Dr. Olivia Hooker - Congress 03:07.mov
86841f59d7660a99feb5d3ce65c827a0 14._Ms._Thelma_Knight_w_TC-Apple_ProRes_422_for_Interlaced_material_copy.mov | 5fdea1ec2667544cb59e0a4b5b377092 19. Mr. John Hope Franklin - Congress 03:07.mov
4a0c342444fedb2096c8495fae5e1459 14._Ms._Thelma_Knight_w_TC.mov | 83a52017b776abd0a0c82e3866ac08b5 2. Dr. Olivia Hooker 11:16:04.mov
d6853f5f8fda9f130073fb3e3dbf16e6 15._Mr._Robert_Holloway_w_TC-Apple_ProRes_422_for_Interlaced_material_copy.mov | 3611978acf5e564efe7567262b9960f8 20. Bill O'Brian - Historian.mov
56b0d7952f01b48f47d90a5c300411ef 15._Mr._Robert_Holloway_w_TC.mov | 09108415da82b24c114f3c135db611eb 21. John Rogers - Descendant.mov
eafa27fce895e52bc7b6071668e0c10f 16._Mr._Archie_Franklin_w_TC-Apple_ProRes_422_for_Interlaced_material_copy.mov | 1d74cad5bf3e0887d31c4df709b91957 22. Ms. Eddie Faye Gates - Historian 3:4:04.mov
3117d375927b0032f7b804d1f272f97a 16._Mr._Archie_Franklin_w_TC.mov | d6a5dbc85ed5b60b4fc678ba9ad9672d 23. Scott Elsworth - Historian.mov
3969d4eb70bf9595bb7a3bac283fda28 17._Mr.__John_Hope_Franklin_by_Charles_Ogletree-Apple_ProRes_422_for_Interlaced_ | 529a6ad514dca022836f43a780f8d1dd 24. Dr. Olivia Hooker MV 8:07.mov
552230c7e504e0f3fa819a46b8169bd4 17._Mr.__John_Hope_Franklin_by_Charles_Ogletree.mov | 1ca96afc9135bc73e2be7f6702469416 25. Mr. Otis Clark MV 8:07.mov
7947f017d3e0dc194dd3085c2474583f 18._Dr._Olivia_Hooker_-_Congress_03-07-Apple_ProRes_422_for_Interlaced_material_ | 2bf6c9f2cc68f7e8d54b89d9585d90d9 26. Mr. Wes Young MV 8:07.mov
933b65571efefd2ea1e642f478f4cc94 18._Dr._Olivia_Hooker_-_Congress_03-07.mov | b609627fef5619d4d735f46352d0effb 27. Survivors Supreme Court 3:9:05.mov
eb837afbd180e1c9217b5cb02ca69849 19._Mr._John_Hope_Franklin_-_Congress_03-07-Apple_ProRes_422_for_Interlaced_mate | 91c74d7902d1bd4895e77c64fd163d7d 3. Ms. Jimmie Lily Franklin 7:8:04.mov
5fdea1ec2667544cb59e0a4b5b377092 19._Mr._John_Hope_Franklin_-_Congress_03-07.mov | ac6c3ac0437a93425f1b6be05c9a4914 4. Mr. Otis Clark 3:5:04.mov
1968980fe9a3d19650fa7c3ec5507a2e 2.__Dr._Olivia_Hooker_11-16-04-Apple_ProRes_422_for_Interlaced_material_copy.mov | 978f5771c0c6a821124387ce029983c8 5. Ms. Juanita Arnold 03:04.mov
83a52017b776abd0a0c82e3866ac08b5 2.__Dr._Olivia_Hooker_11-16-04.mov | b9fc1244ffbd733f4de4e014e5bef209 6. Ms. Eulis Jackson 03:04.mov
ada33ffb5e5e2083856f52b5709f1b31 20._Bill_O_Brian_-_Historian-Apple_ProRes_422_for_Interlaced_material_copy.mov | f7e5822a03f8bad047b701fd5c2c6704 7. Mr. Wes Young 3:25:04.mov
3611978acf5e564efe7567262b9960f8 20._Bill_O_Brian_-_Historian.mov | 1fc6342a34e1efc6b6c99225543a2900 8. Mr. J.B. Bates 3:5:04.mov
94e6f75688705b0e0a1b73e61cb367d7 21._John_Rogers_-_Descendant-Apple_ProRes_422_for_Interlaced_material_copy.mov | b760f9ef2c13fbdf17c56001860176b6 9. Ms. Beatrice Campbell Webster.mov
09108415da82b24c114f3c135db611eb 21._John_Rogers_-_Descendant.mov <
9345c8828f917026d88592d095190556 22._Ms._Eddie_Faye_Gates_-_Historian_3-4-04-Apple_ProRes_422_for_Interlaced_mate <
1d74cad5bf3e0887d31c4df709b91957 22._Ms._Eddie_Faye_Gates_-_Historian_3-4-04.mov <
65772091bcc8ebc75cca81a2fa29ecf4 23._Scott_Elsworth_-_Historian-Apple_ProRes_422_for_Interlaced_material_copy.mov <
d6a5dbc85ed5b60b4fc678ba9ad9672d 23._Scott_Elsworth_-_Historian.mov <
dd53a3132b03327ccf1660905aca1884 24._Dr._Olivia_Hooker_MV_8-07-Apple_ProRes_422_for_Interlaced_material_copy.mov <
529a6ad514dca022836f43a780f8d1dd 24._Dr._Olivia_Hooker_MV_8-07.mov <
f75eb9a79e4e4fccf53d0463a1c7d520 25._Mr._Otis_Clark_MV_8-07-Apple_ProRes_422_for_Interlaced_material_copy.mov <
1ca96afc9135bc73e2be7f6702469416 25._Mr._Otis_Clark_MV_8-07.mov <
f992aa8cf233d306fc7cea0a32d1d1a6 26._Mr._Wes_Young__MV_8-07-Apple_ProRes_422_for_Interlaced_material_copy.mov <
2bf6c9f2cc68f7e8d54b89d9585d90d9 26._Mr._Wes_Young__MV_8-07.mov <
10f38ecde7590255e88529e9ba41cd06 27._Survivors_Supreme_Court_3-9-05-Apple_ProRes_422_for_Interlaced_material_copy <
b609627fef5619d4d735f46352d0effb 27._Survivors_Supreme_Court_3-9-05.mov <
6d566d7dbbf79652a6545e89c8a0e5a6 3.__Ms._Jimmie_Lily_Franklin_7-8-04-Apple_ProRes_422_for_Interlaced_material_cop <
91c74d7902d1bd4895e77c64fd163d7d 3.__Ms._Jimmie_Lily_Franklin_7-8-04.mov <
What is the | symbol telling me? I can't find that anywhere in the diff documentation. Some of the lines match on checksums, yes, but the filenames are slightly different. For other lines, both the checksums and filenames only exist in the first file, not the second, yet in both instances the | is used.
Further down I get the < symbol, which I understand.
I don't get it.
Related
How to use stata svy etregress postestimation assumption check
When using survey data and etregress with an endogenous treatment effect in Stata number of diagnostics and post estimate parts stop being available for the use. svy: etregress logwage i.race gender, treat(training = i.education gender) -------------------------------------------------------------------------------------------------- | Linearized | Coef. Std. Err. t P>|t| [95% Conf. Interval] ---------------------------------+---------------------------------------------------------------- logwage | race | African American | .3891554 .0031105 12.20 0.000 .2000000 .8474752 Asian American | .1487310 .0002843 04.11 0.000 .027113 .8765290 | gender | female | -.0230411 .010445 -6.85 0.000 -.115341 -.0107295 | 1.training | .3703371 .0451778 10.61 0.000 .2018037 .4186134 ---------------------------------+---------------------------------------------------------------- training | i.education | Highschool | -.0715731 .0490565 1.28 0.098 -.1106579 .1291781 College | .1271380 .0401052 3.95 0.003 .0329516 .2107563 Grad School | .8522143 .0085337 8.99 0.000 .8271381 .9573284 | gender | female | .0127444 .0100058 5.33 0.041 .0100558 .0866312 _cons | -1.260083 .0327235 -26.12 0.000 -1.531405 -1.098524 ---------------------------------+---------------------------------------------------------------- /athrho | .0051552 .031410 0.17 0.827 -.0722533 .0810246 /lnsigma | -1.872551 .0166818 -73.50 0.000 -1.928624 -1.278064 ---------------------------------+---------------------------------------------------------------- rho | .0084120 .0421116 -.0649947 .0888529 sigma | .4000831 .0038170 .1925127 .5067780 lambda | .0012673 .0226365 -.0324029 When I have this model simple assumptions related to a linear model like: Check linearity or assumption of independence and the homoscedasticity, normality, or goodness of fit diagnostics do not give output. A residuals versus predicted values plot could have been a rvfplot but this gives the error: last estimates not found Trying estat gofgives invalid subcommand gof and the same for the estat hettest help etregress postestimation does not discuss model assumption tests or goodness of fit tests which we normally see with regress or log-linear model in Stata. When I try the predict residual or predict rstudent nothing is reported making plotting not possible again. I can provide reproducible example of the problem with the reference given by others: webuse nhanes2f, clear qui svyset psuid [pweight=finalwgt], strata(stratid) qui svy: etregress loglead i.female i.diabetes, treat(diabetes = weight age height i.female) // coefl nlcom pct_eff:(100*(exp(_b[loglead:1.female])-1)) Here also the etregress is used with a log transformed dependent variable and a treatment component. Following this model like asked above, how do we check the assumptions and goodness of fit?
how to use estat vif in the right way
I have 2 questions concerning estat vif to test multicollinearity: Is it correct that you can only calculate estat vif after the regress command? If I execute this command Stata only gives me the vif of one independent variable. How do I get the vif of all the independent variables?
Q1. I find estat vif documented under regress postestimation. If you can find it documented under any other postestimation heading, then it is applicable after that command. Q2. You don't give any examples, reproducible or otherwise, of your problem. But estat vif by default gives a result for each predictor (independent variable). . sysuse auto, clear (1978 Automobile Data) . regress mpg weight price Source | SS df MS Number of obs = 74 -------------+---------------------------------- F(2, 71) = 66.85 Model | 1595.93249 2 797.966246 Prob > F = 0.0000 Residual | 847.526967 71 11.9369995 R-squared = 0.6531 -------------+---------------------------------- Adj R-squared = 0.6434 Total | 2443.45946 73 33.4720474 Root MSE = 3.455 ------------------------------------------------------------------------------ mpg | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- weight | -.0058175 .0006175 -9.42 0.000 -.0070489 -.0045862 price | -.0000935 .0001627 -0.57 0.567 -.000418 .0002309 _cons | 39.43966 1.621563 24.32 0.000 36.20635 42.67296 ------------------------------------------------------------------------------ . estat vif Variable | VIF 1/VIF -------------+---------------------- price | 1.41 0.709898 weight | 1.41 0.709898 -------------+---------------------- Mean VIF | 1.41
Spark: All RDD data not getting saved to Cassandra table
Hi, I am trying to load RDD data to a Cassandra Column family using Scala. Out of a total 50 rows , only 28 are getting stored into cassandra table.
Below is the Code snippet:
val states = sc.textFile("state.txt")
//list o fall the 50 states of the USA
var n =0 // corrected to var
val statesRDD = states.map{a =>
n=n+1
(n, a)
}
scala> statesRDD.count
res2: Long = 50
cqlsh:brs> CREATE TABLE BRS.state(state_id int PRIMARY KEY, state_name text);
statesRDD.saveToCassandra("brs","state", SomeColumns("state_id","state_name"))
// this statement saves only 28 rows out of 50, not sure why!!!!
cqlsh:brs> select * from state;
state_id | state_name
----------+-------------
23 | Minnesota
5 | California
28 | Nevada
10 | Georgia
16 | Kansas
13 | Illinois
11 | Hawaii
1 | Alabama
19 | Maine
8 | Oklahoma
2 | Alaska
4 | New York
18 | Virginia
15 | Iowa
22 | Wyoming
27 | Nebraska
20 | Maryland
7 | Ohio
6 | Colorado
9 | Florida
14 | Indiana
26 | Montana
21 | Wisconsin
17 | Vermont
24 | Mississippi
25 | Missouri
12 | Idaho
3 | Arizona
(28 rows)
Can anyone please help me in finding where the issue is?
Edit:
I understood why only 28 rows are getting stored in Cassandra, it's because I have made the first column a PRIMARY KEY and It looks like in my code, n is incremented maximum to 28 and then it starts again with 1 till 22 (total 50).
val states = sc.textFile("states.txt")
var n =0
var statesRDD = states.map{a =>
n+=1
(n, a)
}
I tried making n an accumulator variable as well(viz. val n = sc.accumulator(0,"Counter")), but I don't see any differnce in the output.
scala> statesRDD.foreach(println)
[Stage 2:> (0 + 0) / 2]
(1,New Hampshire)
(2,New Jersey)
(3,New Mexico)
(4,New York)
(5,North Carolina)
(6,North Dakota)
(7,Ohio)
(8,Oklahoma)
(9,Oregon)
(10,Pennsylvania)
(11,Rhode Island)
(12,South Carolina)
(13,South Dakota)
(14,Tennessee)
(15,Texas)
(16,Utah)
(17,Vermont)
(18,Virginia)
(19,Washington)
(20,West Virginia)
(21,Wisconsin)
(22,Wyoming)
(1,Alabama)
(2,Alaska)
(3,Arizona)
(4,Arkansas)
(5,California)
(6,Colorado)
(7,Connecticut)
(8,Delaware)
(9,Florida)
(10,Georgia)
(11,Hawaii)
(12,Idaho)
(13,Illinois)
(14,Indiana)
(15,Iowa)
(16,Kansas)
(17,Kentucky)
(18,Louisiana)
(19,Maine)
(20,Maryland)
(21,Massachusetts)
(22,Michigan)
(23,Minnesota)
(24,Mississippi)
(25,Missouri)
(26,Montana)
(27,Nebraska)
(28,Nevada)
I am curious to know what is causing n to not getting updated after value 28? Also, what are the ways in which I can create a counter which I can use for creating RDD?
There are some misconceptions about distributed systems embedded inside your question. The real heart of this is "How do I have a counter in a distributed system?"
The short answer is you don't. For example what you've done in your code example originally is something like this.
Task One {
var x = 0
record 1: x = 1
record 2: x = 2
}
Task Two {
var x = 0
record 20: x = 1
record 21: x = 2
}
Each machine is independently creating a new x variable set at 0 which gets incremented within it's own context, independently over the other nodes.
For most use cases the "counter" question can be replaced with "How can I get a Unique Identifier per Record in a distributed system?"
For this most users end up using a UUID which can be generated on independent machines with infinitesimal chances of conflicts.
If the question can be "How can I get a monotonic increasing unique indentifier?"
Then you can use zipWithUniqueIndex which will not count but will generate monotonically increasing ids.
If you just want them number to start with it's best to do it on the local system.
Edit; Why can't I use an accumulator?
Accumulators store their state (surprise) per task. You can see this with a little example:
val x = sc.accumulator(0, "x")
sc.parallelize(1 to 50).foreachPartition{ it => it.foreach(y => x+= 1); println(x)}
/*
6
7
6
6
6
6
6
7
*/
x.value
// res38: Int = 50
The accumulators combine their state after finishing their tasks, which means you can't use them as a global distributed counter.
getting wget error while trying to upgrade openwrt kernel
BusyBox v1.22.1 (2014-11-14 10:11:32 CST) built-in shell (ash) Enter 'help' for a list of built-in commands. _______ ________ __ | |.-----.-----.-----.| | | |.----.| |_ | - || _ | -| || | | || _|| _| |_______|| __|_____||||________||| |____| |__| W I R E L E S S F R E E D O M CHAOS CALMER (Bleeding Edge, unknown) 1 1/2 oz Gin Shake with a glassful 1/4 oz Triple Sec of broken ice and pour 3/4 oz Lime Juice unstrained into a goblet. 1 1/2 oz Orange Juice 1 tsp. Grenadine Syrup root#OpenWrt:~# wget https://downloads.openwrt.org/chaos_calmer/15.05.1/ramips/m t7620/openwrt-15.05.1-ramips-mt7620-ArcherC20i-squashfs-sysupgrade.bin wget: not an http or ftp url: https://downloads.openwrt.org/chaos_calmer/15.05.1/ramips/mt7620/openwrt-15.05.1-ramips-mt7620-ArcherC20i-squashfs-sysupgrade.bin root#OpenWrt:~#
Replace https in your link with http: wget http://downloads.openwrt.org/chaos_calmer/15.05.1/ramips/mt7620/openwrt-15.05.1-ramips-mt7620-ArcherC20i-squashfs-sysupgrade.bin Your installed version of wget doesn't support TLS (libopenssl is not installed). Also, I would change to /tmp dir first, so that downloaded image is stored in RAM (you probably don't have enough space in flash): cd /tmp wget http://...
Copy a categorical variable with its value labels
Is it possible to copy a labeled categorical variable in a single line, or do I generally have to copy over labels as a separate step? In the case I'm looking at, egen ... group() comes close, but changes the underlying integers: sysuse auto ** starts them from different indices egen mycut = cut(mpg), at(0 20 30 50) label icodes egen mycut_copy = group(mycut), label ** does weird stuff egen mycut2 = cut(mpg), at(0 20 30 50) label icodes replace mycut2 = group(mycut2) egen mycut_copy2 = group(mycut2), label ** the correct approach? gen mycut3 = cut(mpg), at(0 20 30 50) label icodes gen mycut_copy3 = mycut3 label values mycut_copy3 mycut3
You can do what you want very easily using the less-known clonevar command: sysuse auto, clear egen mycut = cut(mpg), at(0 20 30 50) label icodes clonevar mycut2 = mycut list mycut* in 1/10, separator(0) +----------------+ | mycut mycut2 | |----------------| 1. | 20- 20- | 2. | 0- 0- | 3. | 20- 20- | 4. | 20- 20- | 5. | 0- 0- | 6. | 0- 0- | 7. | 20- 20- | 8. | 20- 20- | 9. | 0- 0- | 10. | 0- 0- | +----------------+ Note that group() refers to different functions when used with generate and egen, which is why you do not get the same results.