Split RDD into many RDDs and Cache - scala

I have an rdd like so
(aid, session, sessionnew, date)
(55-BHA, 58, 15, 2017-05-09)
(07-YET, 18, 5, 2017-05-09)
(32-KXD, 27, 20, 2017-05-09)
(19-OJD, 10, 1, 2017-05-09)
(55-BHA, 1, 0, 2017-05-09)
(55-BHA, 19, 3, 2017-05-09)
(32-KXD, 787, 345, 2017-05-09)
(07-YET, 4578, 1947, 2017-05-09)
(07-YET, 23, 5, 2017-05-09)
(32-KXD, 85, 11, 2017-05-09)
I want to split everything with the same aid to a new rdd and then cache that for use later, so one rdd per unique aid. I saw some other answers but they are saving the rdds to files. Is there a problem with saving this many rdds in memory? It will likely be around 30k+
I save the cached rdd with spark jobserver.


I would suggest you to cache the grouped rdd as below
lets say you have rdd data as :
val rddData = sparkContext.parallelize(Seq(
("55-BHA", 58, 15, "2017-05-09"),
("07-YET", 18, 5, "2017-05-09"),
("32-KXD", 27, 20, "2017-05-09"),
("19-OJD", 10, 1, "2017-05-09"),
("55-BHA", 1, 0, "2017-05-09"),
("55-BHA", 19, 3, "2017-05-09"),
("32-KXD", 787, 345, "2017-05-09"),
("07-YET", 4578, 1947, "2017-05-09"),
("07-YET", 23, 5, "2017-05-09"),
("32-KXD", 85, 11, "2017-05-09")))
You can cache the data by grouping with "aid" and use filter to select the grouped data you need as :
val grouped = rddData.groupBy(_._1).cache
val filtered = grouped.filter(_._1 equals("32-KXD"))
But I would suggest you to use DataFrame as below which is efficient and improved than rdds
import sqlContext.implicits._
val dataFrame = Seq(
("55-BHA", 58, 15, "2017-05-09"),
("07-YET", 18, 5, "2017-05-09"),
("32-KXD", 27, 20, "2017-05-09"),
("19-OJD", 10, 1, "2017-05-09"),
("55-BHA", 1, 0, "2017-05-09"),
("55-BHA", 19, 3, "2017-05-09"),
("32-KXD", 787, 345, "2017-05-09"),
("07-YET", 4578, 1947, "2017-05-09"),
("07-YET", 23, 5, "2017-05-09"),
("32-KXD", 85, 11, "2017-05-09")).toDF("aid", "session", "sessionnew", "date").cache
val newDF = dataFrame.select("*").where(dataFrame("aid") === "32-KXD")
newDF.show
I hope it helps

Related

Flutter Convert To String Without "[ ]" [duplicate]

This question already has answers here:
How to convert list<String> into String in Dart without iteration?
(3 answers)
Closed 2 years ago.
I have a list. I want to convert this list to String data. I try to convert like this below, but brackets [] could not delete.
My List datas: [6, 8, 9, 11, 14, 15, 16, 133, 134, 135, 136, 138]
I tried like this to convert.
List value = [6, 8, 9, 11, 14, 15, 16, 133, 134, 135, 136, 138]
data = value.toString();
My data Output like:
print(data);
[6, 8, 9, 11, 14, 15, 16, 133, 134, 135, 136, 138]
But it doesn't work.
How can I convert this list to string without [].
Thanks.

I would suggest using join
https://api.dart.dev/stable/2.10.4/dart-core/Iterable/join.html
List value = [6, 8, 9, 11, 14, 15, 16, 133, 134, 135, 136, 138]
var data = value.join(',')

Try:
List<int> value = [6, 8, 9, 11, 14, 15, 16, 133, 134, 135, 136, 138];
List<String> string_values = value.map((value) => value.toString()).toList();

void main() {
String output='';
List value = [6, 8, 9, 11, 14, 15, 16, 133, 134, 135, 136, 138];
value.forEach((x)=>x.toString());
output = value.join(',');
print(output);
}

Intersection of Two Map rdd's in Scala

I have two RDD's, for example:
firstmapRDD - (0-14,List(0, 4, 19, 19079, 42697, 444, 42748))
secondmapRdd-(0-14,List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94))
I want to find the intersection.
I tried, var interResult = firstmapRDD.intersection(secondmapRdd), which shows no result in output file.
I also tried , cogrouping based on keys, mapRDD.cogroup(secondMapRDD).filter(x=>), but I don't know how to find the intersection between both the values, is it x=>x._1.intersect(x._2), Can someone help me with the syntax?
Even this throws a compile time error, mapRDD.cogroup(secondMapRDD).filter(x=>x._1.intersect(x._2))
var mapRDD = sc.parallelize(map.toList)
var secondMapRDD = sc.parallelize(secondMap.toList)
var interResult = mapRDD.intersection(secondMapRDD)
It may be because of ArrayBuffer[List[]] values, because of which the intersection is not working. Is there any hack to remove it?
I tried doing this
var interResult = mapRDD.cogroup(secondMapRDD).filter{case (_, (l,r)) => l.nonEmpty && r.nonEmpty }. map{case (k,(l,r)) => (k, l.toList.intersect(r.toList))}
Still getting an empty list!

Since you are looking intersect on values, you need to join both RDDs, get all the matched values, then do the intersect on values.
sample code:
val firstMap = Map(1 -> List(1,2,3,4,5))
val secondMap = Map(1 -> List(1,2,5))
val firstKeyRDD = sparkContext.parallelize(firstMap.toList, 2)
val secondKeyRDD = sparkContext.parallelize(secondMap.toList, 2)
val joinedRDD = firstKeyRDD.join(secondKeyRDD)
val finalResult = joinedRDD.map(tuple => {
val matchedLists = tuple._2
val intersectValues = matchedLists._1.intersect(matchedLists._2)
(tuple._1, intersectValues)
})
finalResult.foreach(println)
The output will be
(1,List(1, 2, 5))

Is it bad practice to populate this array using a for loop?

Please forgive me for asking what is probably a real beginners question. My search on google and stackoverflow didn't produce anything conclusive.
My array needs to contain the numbers 0 through 59. Here is a simple for loop to populate the array:
var timeArray = [0]
count = 1
while count < 60 {
timeArray.append(count)
count++
}
On the other hand, I could do this:
var timeArray = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
The second I guess is faster and maybe more readable. The first is maybe more concise.
What is general best practice in this case? Is there another, beter alternative?
Thanks.

Yes you are right the for loop will be slower that the second one.
I would use the second option but with slightly different syntax, just to save typings:
var timeArray = Array(0..<60)

WinBUGS "Array index is greater than array upper bound for a"

I have a simple Bayesian hierachical model (linear mixed model with random intercepts) which should be easy to run. The problem is that after successfully loading the model and data, I get the following error when I try to compile the model, "array index is greater than array upper bound for a." It seems like this should be an easy fix, but I've repeatedly checked the indexing, and the data, and cannot find a problem. I have experimented on different example datasets, and don't have any issue, which makes me think something other then indexing is the problem.
Any suggestions would be greatly appreciated!
model{
for (i in 1:n) {
y[i] ~ dnorm(y.hat[i],tau.y)
y.hat[i]<- a[d[i]] + beta0 + beta1*x[i]
}
beta0 ~ dnorm(50, 0.001) I(0, )
beta1 ~ dnorm(0, 0.001)
tau.y <- pow(sigma.y,-2)
sigma.y ~ dunif(0,100)
for (j in 1:J){
a[j] ~ dnorm(0, .0001)
a.hat[j] <- mu.a
}
mu.a ~ dnorm(0, .0001)
tau.a <- pow(sigma.a, -2)
sigma.a ~ dunif(0, 100)
}
list( n = 904, J = 44, y = c( 61.46, 86, 75.11, 77.31, 65.7, 71.82, 61.42, 63.8, 45.53, 48.29,
43.77, 51.6, 80.1, 51.1, 65.9, 71.9, 66.73, 68.96, 86, 74.3, 72.9, 66.9, 54.4,
66, 70, 64, 78.6, 70.6, 80.2, 75.2, 63.6, 62.7, 71.9, 56.1, 59.6, 57.6, 61,
59.9, 57.2, 51.4, 62.8, 70, 60.7, 63.3, 60.4, 65.3, 75.45, 63.8, 68.21, 70.45,
59.28, 53.05, 43.6, 30.82, 79.74, 56.03, 66.88, 49.31, 64.17, 56.57, 54.37,
51.99, 49.56, 42.6, 48.52, 51.72, 59.43, 57.81, 57.29, 57.79, 67.14, 70, 61.38,
54.23, 63.41, 58.19, 67.23, 31.91, 40.95, 53.83, 49.84, 53.32, 61.2, 52.11,
53.83, 58.13, 54.21, 52.72, 49.12, 43.7, 50.2, 36.93, 59.7, 57.8, 50.7, 68.2,
47.2, 70.27, 90.75, 72.16, 75.05, 75.2, 56.58, 63.59, 43.53, 35.6, 49.19,
64.36, 45.77, 53.85, 30.92, 56.99, 85.54, 77.73, 65, 42.11, 47.47, 25.3, 34.82,
21.5, 72.76, 81.08, 77.18, 69.41, 58.54, 72.73, 59.45, 63, 63.9, 62.15, 55.58,
49.72, 47.11, 54.85, 47.63, 47.7, 49.75, 48.32, 59.15, 77.3, 73.7, 87.73, 55,
61.3, 52.2, 52.9, 48.3, 58.7, 53.48, 78.88, 77.43, 81.94, 73.24, 71.75, 64.99,
66.99, 63.82, 57.23, 54.15, 57.08, 54.35, 65.24, 78.26, 63.63, 68.72, 65.21,
71.4, 44.54, 57.89, 60.9, 51.3, 64.84, 54, 52.76, 91.03, 93.05, 75.1, 71.89,
77.68, 79.88, 74.01, 56.67, 59.06, 48.66, 67.09, 41.25, 61.18, 68.31, 46.74,
50.87, 72.13, 79.7, 77.18, 58.28, 44.42, 60.12, 66.9, 67.73, 77.59, 62.12,
63.01, 84.65, 100, 69.56, 62.48, 47.2, 72.83, 52.83, 68.04, 69.11, 63.91,
56.25, 57.43, 86.5, 65.44, 70, 69, 65, 68, 67, 54, 50, 46, 55, 42, 45, 49.7,
48.7, 40.1, 51.8, 51.9, 62.5, 50.04, 63.2, 53.8, 57.46, 35.5, 69.94, 87.04,
82.1, 73.44, 67.49, 76.53, 73.21, 59.37, 63.41, 63.54, 59.56, 58.93, 56.38,
51.99, 50.94, 53.63, 57.79, 50.8, 53.66, 50.12, 47.58, 51.46, 56.57, 60.64,
63.23, 60.7, 61.2, 64.03, 68.85, 77.96, 74.98, 56.38, 83.7, 62.29, 68.38,
40.18, 71.12, 37.69, 68.43, 53.69, 76.56, 71.15, 67.67, 45.75, 75.21, 48.96,
53, 52.7, 34.6, 60.09, 52.18, 65.92, 71, 60.95, 61.79, 57.6, 52.91, 59.26,
48.98, 60.66, 42.7, 52.8, 41.14, 56.52, 42.54, 36.26, 35.68, 74.61, 61.22,
69.58, 63.14, 58.76, 56.82, 46.7, 54.82, 48.04, 56.47, 60.6, 53.11, 67, 51.88,
45.41, 46.76, 83.91, 55.55, 22.41, 38.54, 44.98, 41.47, 39.44, 54.04, 55.9,
67.3, 56.4, 68, 67.7, 62.8, 46.1, 86.44, 55.7, 61.98, 66.12, 77.48, 65.55,
62.58, 77.43, 99.05, 57.97, 49.57, 60.8, 52.68, 77.39, 63.34, 66.21, 67.25,
83.61, 64.02, 63.07, 58.1, 57.29, 55.68, 41, 45.65, 49.31, 55.91, 56.47, 58.5,
68.64, 61.64, 78.46, 72.22, 40.86, 50.36, 47.65, 50.06, 64, 63.35, 61.06,
47.24, 39.05, 45.22, 40.13, 33.94, 31.59, 34.2, 52.5, 38.46, 50.89, 63.63,
43.12, 56.92, 53.04, 63.26, 61.13, 63.7, 44.6, 45.37, 55.28, 84.3, 66.2, 64,
84, 47.8, 36, 36.9, 37.6, 50.3, 55.9, 42.9, 47, 43.8, 29.7, 42.8, 36.2, 35.3,
33.8, 38.2, 35.9, 37.1, 30.6, 44.6, 49.3, 34.9, 37, 37.6, 26.7, 40.8, 40.45,
53.8, 45.12, 29.48, 47.19, 66.23, 63.68, 63.91, 55.36, 55.96, 44.9, 36.8,
46.47, 47.76, 40.35, 41.97, 42, 60, 47.2, 49.72, 34.69, 43.14, 51.18, 52.81,
51.02, 45.89, 46.9, 28.6, 44.48, 41.31, 66.09, 56.26, 47.43, 31.9, 54.59,
61.56, 67.03, 69.74, 61.55, 63.02, 50.9, 52.9, 50.68, 48.36, 41.53, 35.13, 50,
49.21, 59.69, 39.45, 56.26, 48.78, 50.65, 45.26, 47.19, 50, 30.2, 46.81, 56.3,
48.24, 37.56, 57.35, 50.4, 42.39, 44.31, 45.43, 61.9, 55.03, 55.19, 52.83,
46.51, 45.66, 55.14, 58.89, 56.74, 46.58, 62.51, 45.15, 51.56, 44.95, 48.11,
44.33, 58.3, 49.5, 54.3, 46.5, 53.9, 37.7, 43.51, 39.86, 58.43, 68.21, 54.07,
55.67, 71.11, 67.25, 72.5, 63.41, 51.36, 51.74, 53.44, 55.8, 58.79, 44.12,
31.4, 46.9, 43.6, 46.18, 36.24, 41.08, 47.85, 53.69, 37.53, 37.43, 40.3, 21.3,
42.49, 55.71, 62.2, 58.45, 63.45, 58.13, 53.59, 57.82, 55.94, 53.44, 50.86,
20.85, 27.08, 42.8, 34.29, 38.9, 34.59, 26.45, 72.34, 104.88, 73.93, 66.95,
75.72, 75.1, 53.9, 74.67, 73.97, 85.5, 85.38, 59.7, 59.36, 44.51, 62.25, 56.26,
62.77, 60.24, 55.13, 61.35, 66.21, 56.06, 60.46, 70.45, 50, 66.44, 48.24,
55.56, 71.13, 60.97, 37.94, 43.6, 36, 51.22, 57.84, 58.56, 57.14, 63.51, 50.35,
58.68, 62.46, 59.02, 62.23, 59.44, 66.82, 65.38, 53.92, 57.73, 43.34, 58.41,
65.06, 56.91, 49.01, 53.5, 48.6, 52.47, 46.04, 58.81, 39.29, 43.59, 77.4, 30.9,
50.5, 31.66, 125, 37.8, 53.91, 59.84, 50, 52.31, 39.53, 47.87, 39.58, 46.97,
56.16, 57.78, 44.52, 40.22, 52.56, 45.45, 47.17, 76, 91.95, 78.97, 64.81,
55.56, 53.1, 51.71, 55.9, 93.6, 74.62, 53.37, 57.1, 62.57, 40.58, 56.75, 61.58,
58.08, 61.59, 65.45, 64.1, 57.03, 56.5, 49.56, 50, 32.84, 21.69, 51.9, 28.59,
45.2, 28.17, 39.29, 68.6, 57.07, 61.11, 48.05, 56.43, 63.2, 71.71, 55.25,
50.18, 43.23, 41.67, 30.37, 37.5, 38.39, 32.8, 37.36, 39.39, 54.41, 64.94,
60.38, 52.36, 53.57, 59.87, 70.77, 56.8, 55.1, 47.46, 48.98, 61.2, 30.41, 38.4,
43.19, 76.56, 74.65, 57.64, 60.87, 81.58, 83.3, 63.41, 54.05, 51.48, 54.01,
48.4, 52.24, 49.9, 43.4, 47.17, 43.6, 36.6, 40, 36.46, 38.56, 47.99, 49.58,
54.2, 42.51, 41.6, 33.2, 43.95, 41.31, 86.05, 66.98, 53.75, 58.36, 45.73,
53.42, 55.88, 69.72, 73.8, 67.85, 54.74, 52.36, 61, 56.58, 40.82, 40.77, 42.42,
38.44, 52.79, 52.73, 35.2, 46.03, 62.01, 64.46, 56.4, 36, 46.79, 48.64, 66.67,
72.03, 65.42, 82.25, 74.69, 62.61, 61.21, 53.09, 56.19, 57.95, 57.21, 43.76,
66.5, 55.76, 57.69, 59.67, 74.96, 58.78, 67.3, 69.15, 68.33, 71.56, 75.29,
67.05, 68.86, 78.61, 59.19, 51.66, 56.31, 64.66, 47.29, 58.39, 60.22, 40, 41.4,
41.38, 46.76, 61.8, 61.4, 68.63, 87.7, 78.46, 70.51, 59.77, 65.42, 72.1, 59.38,
61.21, 46.28, 41.42, 54.18, 47.31, 45.51, 36.31, 51.03, 61.08, 25.08, 59.45,
58.81, 60.37, 50, 80, 61.5, 89.44, 88.1, 75, 61.9, 80.56, 42.86, 67.31, 85.71,
60, 59.4, 43.99, 64.52, 69.4, 55.14, 43.75, 75, 76.71, 52.93, 52.34, 45.21,
38.28, 50.91, 61.54, 61.11, 45.87, 47.6, 45.05, 44.43, 52.9, 34.52, 40.57,
40.5, 37.3, 38, 45.06, 33.71, 32.12, 48.4, 39.78, 39.86, 36.8, 31.8, 47.86,
41.78, 48, 60, 49.8, 55.2, 44.6, 60.4, 57.4, 65.9, 59.1, 52.2, 54.41, 62.32,
57.33, 68.17, 62.96, 50.33, 51.48, 48.73, 59.56, 61.96, 56.68, 32, 49.6, 45.8,
53.16, 39.58, 58.55, 54.7, 42.53, 39.59, 33.33, 69.67, 66.79, 44.14, 47.75,
65.38, 39.7, 76.97, 34) , d = c( 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13,
13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15,
15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 17, 17,
17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18,
18, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19,
19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 20, 20,
20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20,
20, 20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21,
21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22,
22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22,
22, 22, 22, 22, 22, 22, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25,
25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 29, 29, 29, 29, 29, 29, 29, 29,
29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 30, 30, 30, 30,
30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 31, 31,
31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 34, 34, 34, 34, 34, 35, 35, 35, 35,
35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36,
36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37,
37, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 39,
39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40,
40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40,
40, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41,
41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42,
42, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 51,
51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51,
51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52,
52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 53, 53, 53, 53, 53, 53,
53, 53, 53, 53, 53, 53, 53) , x = c( 0.54, 0.5350262, 0.554, 0.582, 0.624,
0.638, 0.652, 0.666, 0.6834799, 0.681, 0.682, 0.684, 0.685, 0.686, 0.687,
0.688, 0.689, 0.6918783, 10.13947, 10.252, 10.364, 10.476, 10.588, 10.7,
10.812, 10.924, 11.036, 11.148, 11.25654, 11.757, 12.254, 12.751, 13.248,
13.745, 14.242, 14.739, 15.236, 15.733, 16.22779, 16.232, 16.234, 16.236,
16.238, 16.24, 16.242, 16.244, 16.246, 16.248, 16.24809, 10.28, 10.28, 10.28,
10.28, 10.28, 10.28, 10.28, 10.28, 10.498, 10.716, 10.934, 11.152, 11.37,
11.588, 11.806, 12.242, 12.45755, 12.464, 12.466, 12.468, 12.472, 12.474,
12.476, 12.478, 12.48307, 20.445, 20.495, 20.57, 20.62, 20.67224, 21.01, 21.35,
21.69, 22.03, 22.37, 22.71, 23.05, 23.39, 23.73, 24.06687, 24.098, 24.126,
24.154, 24.182, 24.21, 24.266, 24.322, 24.34951, 2.517, 2.614, 2.711, 3.002,
3.099, 3.293, 3.386308, 3.584, 3.778, 3.972, 4.166, 4.748, 4.942, 5.329492,
5.34, 5.345, 5.35, 5.355, 5.36, 5.365, 5.37, 5.375, 5.384694, 8.443984, 8.604,
8.768, 8.932, 9.096, 9.26, 9.424, 9.752, 9.916, 10.07591, 10.391, 10.702,
11.013, 11.324, 11.635, 12.257, 12.568, 12.879, 13.19029, 13.211, 13.232,
13.253, 13.274, 13.295, 13.316, 13.337, 13.358, 13.379, 13.3975, 10.23494,
10.26, 10.29, 10.32, 10.35, 10.41, 10.47, 10.52766, 11.038, 11.546, 12.054,
12.308, 12.816, 13.06939, 13.582, 14.094, 14.606, 15.118, 15.63, 16.142,
16.654, 17.166, 17.678, 18.19218, 10.25687, 10.412, 10.716, 10.868, 11.172,
11.324, 11.628, 11.945, 12.11, 12.44, 12.605, 12.77, 12.935, 13.1, 13.265,
13.4293, 13.499, 13.568, 13.637, 13.706, 13.775, 13.844, 13.913, 13.982,
14.051, 14.12134, 31.87835, 31.952, 32.024, 32.06, 32.096, 32.23761, 32.322,
32.527, 32.609, 32.821, 32.992, 33.163, 33.505, 33.676, 34.018, 34.189, 8.564,
8.568, 8.572, 8.58, 8.588, 8.596, 8.596099, 8.625, 8.675, 8.7, 8.725, 8.775,
8.8, 8.825, 8.96, 9.29, 9.4, 9.51, 9.62, 9.73, 9.84, 9.950349, 21.62867, 21.88,
22.13, 22.38, 22.63, 22.88, 23.13, 23.38, 23.63, 23.88, 24.12896, 24.482,
24.834, 25.186, 25.538, 25.89, 26.242, 26.594, 27.298, 27.6469, 28.052, 28.253,
28.454, 28.655, 28.856, 29.057, 29.258, 29.459, 29.65551, 16.064, 16.092,
16.12, 16.148, 16.176, 16.206, 16.238, 16.286, 16.302, 16.318, 16.334,
16.34918, 16.382, 16.414, 16.446, 16.478, 16.51, 16.542, 16.574, 16.606,
16.638, 16.66627, 15.708, 15.864, 16.098, 16.254, 16.40707, 16.424, 16.438,
16.452, 16.466, 16.48, 16.494, 16.508, 16.522, 16.536, 16.55052, 16.816,
17.082, 17.348, 17.614, 17.88, 18.146, 18.412, 18.678, 18.944, 19.20852,
17.284, 17.328, 17.372, 17.416, 17.45741, 23.314, 23.572, 24.088, 28.775,
30.237, 31.699, 32.43322, 32.979, 33.528, 34.077, 34.626, 35.175, 35.724,
36.273, 36.822, 37.371, 37.92301, 36.728, 36.977, 37.314, 37.441, 37.695,
38.076, 38.3302, 38.686, 38.864, 39.22, 39.576, 39.932, 13.29665, 13.53, 13.76,
13.99, 14.22, 14.68, 15.14, 15.59616, 16.44, 17.28, 18.12, 19.79912, 19.922,
20.044, 20.105, 20.166, 20.288, 20.349, 20.41163, 13.208, 13.252, 13.296,
13.34, 13.384, 13.428, 13.472, 13.516, 13.56361, 13.746, 13.932, 14.118,
14.304, 14.49, 14.676, 14.862, 15.048, 15.234, 15.42321, 15.54, 15.66, 15.78,
15.9, 16.02, 16.14, 16.26, 16.38, 16.5, 16.6215, 17.40591, 17.485, 17.56,
17.635, 17.71, 17.785, 17.86, 17.935, 18.01, 18.085, 18.16367, 18.167, 18.174,
18.181, 18.188, 18.195, 18.202, 18.209, 18.216, 18.223, 18.2299, 18.23, 18.23,
18.23, 18.23, 18.23, 18.23, 18.23, 18.23, 18.2299, 9.266, 9.738, 9.974, 10.21,
10.446, 10.682, 10.918, 11.154, 11.38546, 11.586, 11.782, 11.978, 12.174,
12.37, 12.566, 12.762, 12.958, 13.154, 13.355, 13.365, 13.38, 13.395, 13.41,
13.425, 13.44, 13.455, 13.47, 13.485, 13.49869, 7.424, 7.472, 7.496, 7.52,
7.544, 7.568, 7.592, 7.616, 7.641526, 7.678, 7.716, 7.754, 7.792, 7.83, 7.868,
7.906, 7.944, 7.982, 8.02042, 8.085, 8.15, 8.215, 8.28, 8.345, 8.41, 8.475,
8.54, 8.605, 8.666293, 7.663, 8.116, 11.196, 11.249, 11.302, 11.355, 11.408,
11.461, 11.514, 11.567, 11.61951, 12.205, 12.79, 13.375, 13.96, 14.545, 15.13,
15.715, 16.3, 16.885, 17.46967, 17.484, 17.498, 17.512, 17.526, 17.54, 17.554,
17.568, 17.582, 17.596, 17.61099, 4.495, 4.745, 4.995, 5.12, 5.245, 5.37,
5.495, 5.624713, 5.658, 5.696, 5.734, 5.772, 5.81, 5.848, 5.886, 5.924, 5.962,
6.004034, 6.305, 6.61, 6.915, 7.22, 7.525, 7.83, 8.135, 8.44, 8.745, 9.054572,
10.14, 10.43, 10.71782, 11.13, 11.335, 11.54, 11.745, 11.95, 12.77435, 14.808,
16.846, 18.884, 19.903, 20.922, 21.941, 22.95846, 37.338, 37.401, 37.527,
37.814, 38.038, 38.598, 39.258, 39.532, 40.354, 40.902, 41.44835, 23.081,
23.294, 23.507, 23.71729, 24.077, 24.434, 24.791, 25.148, 25.505, 25.862,
26.219, 26.576, 26.933, 27.2913, 27.299, 27.308, 27.317, 27.326, 27.335,
27.344, 27.353, 27.362, 27.371, 27.38188, 23.306, 23.788, 24.27268, 24.57,
25.17, 25.47, 25.77, 26.07, 26.37, 26.67, 26.97, 27.2698, 27.577, 27.884,
28.191, 28.498, 28.805, 29.112, 29.419, 29.726, 30.033, 30.34219, 16.134,
24.71, 24.71, 24.71, 24.71, 24.71, 24.71, 31.94, 32.16, 32.38, 32.6, 32.82,
33.26, 33.48, 33.70422, 34.137, 34.574, 35.011, 35.448, 36.322, 37.196, 37.633,
38.07169, 38.072, 38.076, 38.078, 38.08, 38.082, 38.084, 38.086, 38.088,
38.08755, 46.095, 46.406, 46.717, 47.028, 47.339, 47.65467, 23.06, 23.1,
23.14316, 23.622, 23.863, 24.104, 24.345, 24.586, 25.54903, 25.694, 25.838,
26.27, 26.414, 26.558, 26.702, 26.846, 26.99223, 23.826, 23.863, 23.90286,
24.322, 24.533, 24.744, 24.955, 25.166, 26.01203, 26.094, 26.178, 26.346,
26.514, 26.598, 26.682, 26.766, 26.8545, 14.18, 14.365, 14.54838, 15.208,
15.537, 15.866, 16.195, 16.524, 17.511, 17.84261, 18.423, 21.338, 21.921,
22.504, 23.66658, 19.814, 23.12309, 24.952, 25.41, 26.326, 27.242, 23.696,
24.058, 24.42368, 24.597, 24.774, 24.951, 25.128, 25.305, 25.482, 25.659,
25.836, 26.013, 26.19191, 26.201, 26.212, 26.234, 26.245, 26.256, 26.267,
26.278, 26.289, 26.30484, 23.55656, 24.102, 24.373, 24.644, 24.915, 25.186,
25.457, 25.728, 25.999, 26.27382, 26.632, 26.994, 27.356, 27.718, 28.08,
28.442, 29.166, 29.528, 29.88617, 30.124, 30.358, 30.826, 31.06, 31.294,
31.528, 31.762, 31.996, 32.22849, 2.742582, 2.774, 2.808, 2.842, 2.876, 2.91,
2.978, 3.046, 3.144, 3.272, 3.528, 3.656, 3.95, 4.41, 4.64, 4.87, 5.1, 5.33,
5.56, 5.79, 6.017525, 16.63745, 16.884, 17.128, 17.372, 17.494, 17.738,
17.86075, 18.016, 18.172, 18.328, 18.484, 18.796, 19.108, 19.42257, 19.64,
19.86, 20.08, 20.3, 20.52438, 9.794, 9.868, 9.942, 10.016, 10.09, 10.238,
10.386, 10.597, 10.871, 11.145, 11.282, 11.419, 11.556, 11.693, 11.82756,
11.97, 12.11, 12.39, 12.53, 12.67, 12.95, 13.09, 13.2299, 23.206, 26.195,
26.778, 27.944, 28.527, 29.10834, 29.503, 29.896, 30.682, 31.075, 31.468,
31.861, 32.254, 32.647, 33.04166, 21.83835, 21.946, 21.999, 22.052, 22.105,
22.158, 22.211, 22.264, 22.317, 22.368, 22.622, 22.874, 23.126, 23.378, 23.63,
23.882, 24.134, 24.386, 24.638, 24.88723, 24.933, 24.976, 25.062, 25.105,
25.148, 25.191, 25.234, 25.277, 25.32213, 18.59, 18.81, 19.03, 19.25, 19.47,
19.68829, 20.517, 21.344, 22.171, 22.998, 23.825, 24.652, 25.479, 26.306,
27.133, 27.96245, 28.009, 28.058, 28.107, 28.156, 28.205, 28.254, 28.303,
28.352, 28.401, 28.44787, 18.224, 18.652, 18.866, 19.08125, 19.785, 20.49,
21.9, 22.605, 23.31, 24.015, 24.72, 25.425, 26.13229) )

The error occurs as in y.hat[i] <- you allow a to take values from d (anything from 1,..,53 given in the data) which can be greater than the your upper bound for a (J = 44 given in the data).

Histogram from two vectors in Matlab

Thanks in advance for the help.
I have two sets of parallel vectors:
x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 55];
x_count = [7721, 6475, 3890, 2138, 1152, 784, 674, 492, 424, 365, 309, 302, 232, 250, 220, 208, 190, 162, 144, 134, 97, 93, 89, 97, 92, 85, 77, 87, 64, 75, 72, 82, 61, 48, 46, 44, 35, 20, 28, 20, 21, 10, 6, 8, 4, 4, 4, 3, 1, 1];
y = [1, 2, 3, 4, 5, 6, 7, 8, 9, 55];
y_count = [88, 40, 24, 12, 8, 5, 1, 1, 1, 100];
where x, y are the categories, and x_count, y_count are the frequency of each categories. x and y can be of unequal lengths, and need not contain the same categories.
I want to create a side-by-side bar/histogram plot, where the x-axis is the categories, placed side-by-side like this: side by side multiply histogram in matlab. The frequency counts go along the y-axis.
I've tried googling around, but still stuck on this. If someone could help, that would be great. The solution in side by side multiply histogram in matlab works only if x and y have the same length, but mine's not.

You can try this:
% create unique bins
bins = unique([x y]);
% create vectors with zeros same size as bins
xBins = zeros(size(bins));
yBins = zeros(size(bins));
% fill in counts in the respective spots
xBins(ismember(x, bins)) = x_count;
yBins(ismember(y, bins)) = y_count;
bar(bins, [xBins' yBins']);