Recursive function to generate Permutations in Scala - scala
I am trying to learn recursive functions in Scala. This functions generates the
permutations without repetition.
I find little difficult to understand this. Added three print statements to understand the flow.
def permute(nums:List[Int], f:List[Int]=>Unit, p:List[Int]):Unit = {
if(nums.isEmpty) {
println("Inside If")
f(p)
} else {
var before = List[Int]()
var after = nums
while(after.nonEmpty) {
println("beforecall "+"before:"+ (before)+" after:"+(after)+" p:"+p)
permute(before ::: after.tail, f, after.head::p)
before ::= after.head
after = after.tail
println("aftercall "+"before:"+before+" after:"+after+" p:"+p)
}
}
}
And i am running the program for permute( List(1,2,3), println, Nil)
This is the first 10 lines of console output.
beforecall before:List() after:List(1, 2, 3) p:List()
beforecall before:List() after:List(2, 3) p:List(1)
beforecall before:List() after:List(3) p:List(2, 1)
Inside If
List(3, 2, 1)
aftercall before:List(3) after:List() p:List(2, 1)
aftercall before:List(2) after:List(3) p:List(1)
beforecall before:List(2) after:List(3) p:List(1)
beforecall before:List() after:List(2) p:List(3, 1)
Inside If
List(2, 3, 1)
Here we can clearly see beforecall was getting printed 3 times. So there were 3 recursive calls.
permute ( List(2,3), f, List(1) )
permute ( List(3), f, List(2,1) )
permute ( List(), f, List(3,21) )
Then for the last function call since nums is empty, it went inside If and prints List(3,2,1)
Here my doubt is why the aftercall statement was getting printed only two times. For the earlier 3 function calls, we should see 3 corresponding aftercall statements right.
I am little confused. Please guide me.
Also how to mentally approach these kind of problems and write recursive solution from scratch coming from imperative world.
Here is my sample implementation. It uses Stream instead of List just like standard Scala library does because the total output space could be huge.
object Perms extends App {
def perms[T](xs: List[T]): Stream[List[T]] =
xs match {
case Nil => Stream.empty
case single#List(e) => Stream(single)
case l =>
val s = for {
i <- (0 until l.length).toStream
(pref, suf) = l.splitAt(i)
} yield (suf.head, pref ::: suf.tail)
s.flatMap { case (e, rem) => perms(rem).map(ys => e :: ys) }
}
val testData = (1 to 5).toList
val isMyImplSubsetOfRefImpl = (perms(testData).toSet -- testData.permutations.toStream).size == 0
val isRefImplSubsetOfMyImpl = (testData.permutations.toSet -- perms(testData)).size == 0
val isCorrect = isMyImplSubsetOfRefImpl && isRefImplSubsetOfMyImpl
println(s"Matches library implementation results?: $isCorrect")
val l = (1 to 1000).toList
println("First 10 results of a very large stream (n = 1000):")
perms(l) take 10 foreach println
}
Prints:
Matches library implementation results?: true
First 10 results of a very large stream (n = 1000):
List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000)
List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 1000, 999)
...
There are 3 cases you deal with:
empty list - nothing to do, return empty result
list with one element - there can be only one permutation, return it
list with 2 or more elements - take every element from that list and prepend it to the permutation of remaining elements.
You can reach base cases either with initial empty input or through recursion - input decreases by one elem every call. Hope that helps you to understand the reasoning process.
Related
Scipy for Heart Rate data processing, Scipy.signal find_peaks returning empty array
Had an array with HR data in the form of BPMs (beats per minute).
Need to split the data into segments where the HR was increasing, decreasing, and stayed the same to find the up/down amplitude and the associated times as well as the times when the heart rate did not change.
To do so, tried using find_peaks to find the peaks and troughs of the HR data.
To find troughs, I did a transformation of the original HR by multiplying it by -1 to find its peaks which in turn is the real troughs of the original HR data.
find_peaks while specifying plateau and height, gives outputs that contains the peak location as well as the plateau locations and the peak values, example from below:
find_peaks(arr, height=0.5, prominence=0.1, plateau_size=0.5)
(array([ 9, 18, 33, 64, 70, 80, 87]),
{'plateau_sizes': array([5, 2, 2, 3, 1, 3, 2]),
'left_edges': array([ 7, 18, 33, 63, 70, 79, 87]),
'right_edges': array([11, 19, 34, 65, 70, 81, 88]),
'peak_heights': array([ 80., 87., 81., 107., 106., 105., 105.]),
'prominences': array([ 2., 11., 2., 18., 3., 8., 8.]),
'left_bases': array([ 3, 3, 29, 43, 68, 74, 74]),
'right_bases': array([13, 40, 40, 98, 98, 98, 98])})
however, upon specifying height for the negative version of the original Heart rate Data to find the troughs and the heights, it returned empty array. If I remove the height=0.5, it outputs valid values except the peak_heights.
find_peaks(trougharr,plateau_size=0.5,height=0.5)
(array([], dtype=int64),
{'plateau_sizes': array([], dtype=int64),
'left_edges': array([], dtype=int64),
'right_edges': array([], dtype=int64),
'peak_heights': array([], dtype=float64)})
Is there something wrong calling height with negative numbered arrays?
If there's a easier method or simpler method for splitting up the data into up and down and constant portions, that would be much appreciated.
the original sample HR data is as such:
HR
array([ 77, 77, 77, 76, 77, 78, 79, 80, 80, 80, 80, 80, 79,
78, 78, 79, 83, 85, 87, 87, 86, 86, 86, 85, 83, 81,
80, 79, 79, 79, 80, 80, 80, 81, 81, 80, 79, 79, 79,
74, 69, 69, 69, 69, 70, 70, 70, 70, 70, 71, 72, 80,
82, 89, 92, 95, 97, 99, 100, 102, 103, 105, 106, 107, 107,
107, 105, 104, 103, 105, 106, 102, 100, 97, 97, 98, 101, 102,
104, 105, 105, 105, 104, 104, 104, 104, 104, 105, 105, 104, 104,
104, 104, 98, 96, 93, 92, 90, 89])
-1*HR. (the problematic one)
array([ -77, -77, -77, -76, -77, -78, -79, -80, -80, -80, -80,
-80, -79, -78, -78, -79, -83, -85, -87, -87, -86, -86,
-86, -85, -83, -81, -80, -79, -79, -79, -80, -80, -80,
-81, -81, -80, -79, -79, -79, -74, -69, -69, -69, -69,
-70, -70, -70, -70, -70, -71, -72, -80, -82, -89, -92,
-95, -97, -99, -100, -102, -103, -105, -106, -107, -107, -107,
-105, -104, -103, -105, -106, -102, -100, -97, -97, -98, -101,
-102, -104, -105, -105, -105, -104, -104, -104, -104, -104, -105,
-105, -104, -104, -104, -104, -98, -96, -93, -92, -90, -89])
Tried to split Heart Rate data into sections where it was increasing/decreasing/staying constant to find the up_amplitude,up_time, down_amplitude, down_time, time_constant.
Found find_peaks to do it but it may not be the simplest, if there's a simpler method, please point me to it.
Tried to use peak_prominences, but it did not detect all prominences for some reason, the codes tried were:
peaks, _ = find_peaks(x)
prominences = peak_prominences(x, peaks)[0]
contour_heights = x[peaks] - prominences
plt.plot(x)
plt.plot(peaks, x[peaks], "x")
plt.vlines(x=peaks, ymin=contour_heights, ymax=x[peaks])
plt.show()
The prominences missed several locations.
unable to enable rabbitmq rabbitmq_auth_backend_oauth2 plugin
Rabbitmq version 3.8.16
followed this guide.
I tried enabling the plugin.
sudo rabbitmq-plugins enable rabbitmq_auth_backend_oauth2
However it throws back an error.
** (CaseClauseError) no case clause matching: {:could_not_start, :jose, {:jose, {{:shutdown, {:failed_to_start_child, :jose_server, {{:case_clause, {:ECPrivateKey, 1, <<104, 152, 88, 12, 19, 82, 251, 156, 171, 31, 222, 207, 0, 76, 115, 88, 210, 229, 36, 106, 137, 192, 81, 153, 154, 254, 226, 38, 247, 70, 226, 157>>, {:namedCurve, {1, 2, 840, 10045, 3, 1, 7}}, <<4, 46, 75, 29, 46, 150, 77, 222, 40, 220, 159, 244, 193, 125, 18, 190, 254, 216, 38, 191, 11, 52, 115, 159, 213, 230, 77, 27, 131, 94, 17, ...>>, :asn1_NOVALUE}}, [{:jose_server, :check_ec_key_mode, 2, [file: 'src/jose_server.erl', line: 189]}, {:lists, :foldl, 3, [file: 'lists.erl', line: 1267]}, {:jose_server, :support_check, 0, [file: 'src/jose_server.erl', line: 153]}, {:jose_server, :init, 1, [file: 'src/jose_server.erl', line: 93]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 423]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 390]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}}}, {:jose_app, :start, [:normal, []]}}}}
(rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/plugins/plugins_helpers.ex:210: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/2
(rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/plugins/plugins_helpers.ex:107: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/4
(rabbitmqctl 3.8.0-dev) lib/rabbitmq/cli/plugins/commands/enable_command.ex:121: anonymous fn/6 in RabbitMQ.CLI.Plugins.Commands.EnableCommand.do_run/2
(elixir 1.10.4) lib/stream.ex:1325: anonymous fn/2 in Stream.iterate/2
(elixir 1.10.4) lib/stream.ex:1538: Stream.do_unfold/4
(elixir 1.10.4) lib/stream.ex:1609: Enumerable.Stream.do_each/4
(elixir 1.10.4) lib/stream.ex:956: Stream.do_enum_transform/7
(elixir 1.10.4) lib/stream.ex:1609: Enumerable.Stream.do_each/4
{:case_clause, {:could_not_start, :jose, {:jose, {{:shutdown, {:failed_to_start_child, :jose_server, {{:case_clause, {:ECPrivateKey, 1, <<104, 152, 88, 12, 19, 82, 251, 156, 171, 31, 222, 207, 0, 76, 115, 88, 210, 229, 36, 106, 137, 192, 81, 153, 154, 254, 226, 38, 247, 70, 226, ...>>, {:namedCurve, {1, 2, 840, 10045, 3, 1, 7}}, <<4, 46, 75, 29, 46, 150, 77, 222, 40, 220, 159, 244, 193, 125, 18, 190, 254, 216, 38, 191, 11, 52, 115, 159, 213, 230, 77, 27, 131, ...>>, :asn1_NOVALUE}}, [{:jose_server, :check_ec_key_mode, 2, [file: 'src/jose_server.erl', line: 189]}, {:lists, :foldl, 3, [file: 'lists.erl', line: 1267]}, {:jose_server, :support_check, 0, [file: 'src/jose_server.erl', line: 153]}, {:jose_server, :init, 1, [file: 'src/jose_server.erl', line: 93]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 423]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 390]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}}}, {:jose_app, :start, [:normal, []]}}}}}
Any pointers or documentation for this configuration.
Thanks,
Sajith
Well, rabbitmq 3.8.5 seems to work. I assume the plugin built with 3.8.16 has a problem.
Maximize profit with disregard to number of pickup-delivery done OR tools
I am trying to maximize profit, whether I deliver all pickups-deliveries does not matter.
I tried setting SetArcCostEvaluatorOfAllVehicles to use negative profit instead of cost callback; however, this does not work, an I get no solution often, my thought is that cost cannot be negative. Is this correct?
Adding AddVariableMaximizedByFinalizer with profit callback does not seem to work, because it runs after the solutions are already found, and therefore it will not eliminate deliveries that are not profitable. Is this correct?
My gut feeling is that I need set a metric (profit dimension?) that evaluates the performance of solver, and use AddDisjunction with punishment for missing pickup-delivery set to 0, to eliminate not profitable deliveries. Is something like this possible? If not, what is the recommended approach?
Edit:
Here is my code, it is a very small modification of: https://developers.google.com/optimization/routing/pickup_delivery
"""Simple Pickup Delivery Problem (PDP)."""
from __future__ import print_function
from ortools.constraint_solver import routing_enums_pb2
from ortools.constraint_solver import pywrapcp
def create_data_model():
"""Stores the data for the problem."""
data = {}
data['distance_matrix'] = [
[
0, 548, 776, 696, 582, 274, 502, 194, 308, 194, 536, 502, 388, 354,
468, 776, 662
],
[
548, 0, 684, 308, 194, 502, 730, 354, 696, 742, 1084, 594, 480, 674,
1016, 868, 1210
],
[
776, 684, 0, 992, 878, 502, 274, 810, 468, 742, 400, 1278, 1164,
1130, 788, 1552, 754
],
[
696, 308, 992, 0, 114, 650, 878, 502, 844, 890, 1232, 514, 628, 822,
1164, 560, 1358
],
[
582, 194, 878, 114, 0, 536, 764, 388, 730, 776, 1118, 400, 514, 708,
1050, 674, 1244
],
[
274, 502, 502, 650, 536, 0, 228, 308, 194, 240, 582, 776, 662, 628,
514, 1050, 708
],
[
502, 730, 274, 878, 764, 228, 0, 536, 194, 468, 354, 1004, 890, 856,
514, 1278, 480
],
[
194, 354, 810, 502, 388, 308, 536, 0, 342, 388, 730, 468, 354, 320,
662, 742, 856
],
[
308, 696, 468, 844, 730, 194, 194, 342, 0, 274, 388, 810, 696, 662,
320, 1084, 514
],
[
194, 742, 742, 890, 776, 240, 468, 388, 274, 0, 342, 536, 422, 388,
274, 810, 468
],
[
536, 1084, 400, 1232, 1118, 582, 354, 730, 388, 342, 0, 878, 764,
730, 388, 1152, 354
],
[
502, 594, 1278, 514, 400, 776, 1004, 468, 810, 536, 878, 0, 114,
308, 650, 274, 844
],
[
388, 480, 1164, 628, 514, 662, 890, 354, 696, 422, 764, 114, 0, 194,
536, 388, 730
],
[
354, 674, 1130, 822, 708, 628, 856, 320, 662, 388, 730, 308, 194, 0,
342, 422, 536
],
[
468, 1016, 788, 1164, 1050, 514, 514, 662, 320, 274, 388, 650, 536,
342, 0, 764, 194
],
[
776, 868, 1552, 560, 674, 1050, 1278, 742, 1084, 810, 1152, 274,
388, 422, 764, 0, 798
],
[
662, 1210, 754, 1358, 1244, 708, 480, 856, 514, 468, 354, 844, 730,
536, 194, 798, 0
],
]
data['pickups_deliveries'] = [
[1, 6],
[2, 10],
[4, 3],
[5, 9],
[7, 8],
[15, 11],
[13, 12],
[16, 14],
]
data['num_vehicles'] = 4
data['depot'] = 0
data['revenue'] = {6: 1000000,
10: 100,
3: 100,
9: 100,
8: 100,
11: 100,
12: 100,
14: 100
}
return data
def print_solution(data, manager, routing, solution):
"""Prints solution on console."""
total_distance = 0
for vehicle_id in range(data['num_vehicles']):
index = routing.Start(vehicle_id)
plan_output = 'Route for vehicle {}:\n'.format(vehicle_id)
route_distance = 0
while not routing.IsEnd(index):
plan_output += ' {} -> '.format(manager.IndexToNode(index))
previous_index = index
index = solution.Value(routing.NextVar(index))
route_distance += routing.GetArcCostForVehicle(
previous_index, index, vehicle_id)
plan_output += '{}\n'.format(manager.IndexToNode(index))
plan_output += 'Distance of the route: {}m\n'.format(route_distance)
print(plan_output)
total_distance += route_distance
print('Total Distance of all routes: {}m'.format(total_distance))
def main():
"""Entry point of the program."""
# Instantiate the data problem.
data = create_data_model()
# Create the routing index manager.
manager = pywrapcp.RoutingIndexManager(len(data['distance_matrix']),
data['num_vehicles'], data['depot'])
# Create Routing Model.
routing = pywrapcp.RoutingModel(manager)
# Define cost of each arc.
def distance_callback(from_index, to_index):
"""Returns the manhattan distance between the two nodes."""
# Convert from routing variable Index to distance matrix NodeIndex.
from_node = manager.IndexToNode(from_index)
to_node = manager.IndexToNode(to_index)
return data['distance_matrix'][from_node][to_node]
transit_callback_index = routing.RegisterTransitCallback(distance_callback)
routing.SetArcCostEvaluatorOfAllVehicles(transit_callback_index)
# Add Distance constraint.
dimension_name = 'Distance'
routing.AddDimension(
transit_callback_index,
0, # no slack
3000, # vehicle maximum travel distance
True, # start cumul to zero
dimension_name)
distance_dimension = routing.GetDimensionOrDie(dimension_name)
distance_dimension.SetGlobalSpanCostCoefficient(100)
# Define Transportation Requests.
for request in data['pickups_deliveries']:
pickup_index = manager.NodeToIndex(request[0])
delivery_index = manager.NodeToIndex(request[1])
routing.AddPickupAndDelivery(pickup_index, delivery_index)
routing.solver().Add(
routing.VehicleVar(pickup_index) == routing.VehicleVar(
delivery_index))
routing.solver().Add(
distance_dimension.CumulVar(pickup_index) <=
distance_dimension.CumulVar(delivery_index))
for node, revenue in data["revenue"].items():
routing.AddDisjunction(
[manager.NodeToIndex(node)], revenue
)
# Setting first solution heuristic.
search_parameters = pywrapcp.DefaultRoutingSearchParameters()
search_parameters.first_solution_strategy = (
routing_enums_pb2.FirstSolutionStrategy.PARALLEL_CHEAPEST_INSERTION)
# Solve the problem.
solution = routing.SolveWithParameters(search_parameters)
# Print solution on console.
if solution:
print_solution(data, manager, routing, solution)
if __name__ == '__main__':
main()
I addeded:
data['revenue'] = {6: 1000000,
10: 100,
3: 100,
9: 100,
8: 100,
11: 100,
12: 100,
14: 100
}
and
for node, revenue in data["revenue"].items():
routing.AddDisjunction(
[manager.NodeToIndex(node)], revenue)
When I run this code all pickup-deliveries get delivered (even if I hard code revenue to 0). I am looking for a solution where only [1, 6] is delivered, because it is the only one that gives profit.
I think I see where my issue is coming from. When pickup deliveries constraints are setup, the cost is minimized given these constraints. And since all of these constraints can be satisfied all pickup-deliveries get delivered.
Is there a way to make pickup-deliveries constraints soft, and focus on minimizing the cost (plus the punishment)?
If you add a penalty of 0 to deliveries, the solver will happily drop all of them. Also, non profitable is in the context of the route. Therefore, you need to tweak the penalties to get what you want. Response to edit: To make a PDP soft, you need to add 2 disjunctions, one for the pickup, and one for the delivery.
Here is the code that I ended up using:
"""Simple Pickup Delivery Problem (PDP)."""
from __future__ import print_function
from ortools.constraint_solver import routing_enums_pb2
from ortools.constraint_solver import pywrapcp
def create_data_model():
"""Stores the data for the problem."""
data = {}
data['distance_matrix'] = [
[
0, 548, 776, 696, 582, 274, 502, 194, 308, 194, 536, 502, 388, 354,
468, 776, 662
],
[
548, 0, 684, 308, 194, 502, 730, 354, 696, 742, 1084, 594, 480, 674,
1016, 868, 1210
],
[
776, 684, 0, 992, 878, 502, 274, 810, 468, 742, 400, 1278, 1164,
1130, 788, 1552, 754
],
[
696, 308, 992, 0, 114, 650, 878, 502, 844, 890, 1232, 514, 628, 822,
1164, 560, 1358
],
[
582, 194, 878, 114, 0, 536, 764, 388, 730, 776, 1118, 400, 514, 708,
1050, 674, 1244
],
[
274, 502, 502, 650, 536, 0, 228, 308, 194, 240, 582, 776, 662, 628,
514, 1050, 708
],
[
502, 730, 274, 878, 764, 228, 0, 536, 194, 468, 354, 1004, 890, 856,
514, 1278, 480
],
[
194, 354, 810, 502, 388, 308, 536, 0, 342, 388, 730, 468, 354, 320,
662, 742, 856
],
[
308, 696, 468, 844, 730, 194, 194, 342, 0, 274, 388, 810, 696, 662,
320, 1084, 514
],
[
194, 742, 742, 890, 776, 240, 468, 388, 274, 0, 342, 536, 422, 388,
274, 810, 468
],
[
536, 1084, 400, 1232, 1118, 582, 354, 730, 388, 342, 0, 878, 764,
730, 388, 1152, 354
],
[
502, 594, 1278, 514, 400, 776, 1004, 468, 810, 536, 878, 0, 114,
308, 650, 274, 844
],
[
388, 480, 1164, 628, 514, 662, 890, 354, 696, 422, 764, 114, 0, 194,
536, 388, 730
],
[
354, 674, 1130, 822, 708, 628, 856, 320, 662, 388, 730, 308, 194, 0,
342, 422, 536
],
[
468, 1016, 788, 1164, 1050, 514, 514, 662, 320, 274, 388, 650, 536,
342, 0, 764, 194
],
[
776, 868, 1552, 560, 674, 1050, 1278, 742, 1084, 810, 1152, 274,
388, 422, 764, 0, 798
],
[
662, 1210, 754, 1358, 1244, 708, 480, 856, 514, 468, 354, 844, 730,
536, 194, 798, 0
],
]
data['pickups_deliveries'] = [
[1, 6],
[2, 10],
[4, 3],
[5, 9],
[7, 8],
[15, 11],
[13, 12],
[16, 14],
]
data['num_vehicles'] = 4
data['depot'] = 0
data['revenue'] = {(1, 6): 1000000,
(2, 10): 100,
(4, 3): 100,
(5, 9): 10000,
(7, 8): 100,
(15, 11): 100,
(13, 12): 100,
(16, 14): 100
}
return data
def print_solution(data, manager, routing, solution):
"""Prints solution on console."""
total_distance = 0
for vehicle_id in range(data['num_vehicles']):
index = routing.Start(vehicle_id)
plan_output = 'Route for vehicle {}:\n'.format(vehicle_id)
route_distance = 0
while not routing.IsEnd(index):
plan_output += ' {} -> '.format(manager.IndexToNode(index))
previous_index = index
index = solution.Value(routing.NextVar(index))
route_distance += routing.GetArcCostForVehicle(
previous_index, index, vehicle_id)
plan_output += '{}\n'.format(manager.IndexToNode(index))
plan_output += 'Distance of the route: {}m\n'.format(route_distance)
print(plan_output)
total_distance += route_distance
print('Total Distance of all routes: {}m'.format(total_distance))
def main():
"""Entry point of the program."""
# Instantiate the data problem.
data = create_data_model()
# Create the routing index manager.
manager = pywrapcp.RoutingIndexManager(len(data['distance_matrix']),
data['num_vehicles'], data['depot'])
# Create Routing Model.
routing = pywrapcp.RoutingModel(manager)
# Define cost of each arc.
def distance_callback(from_index, to_index):
"""Returns the manhattan distance between the two nodes."""
# Convert from routing variable Index to distance matrix NodeIndex.
from_node = manager.IndexToNode(from_index)
to_node = manager.IndexToNode(to_index)
return data['distance_matrix'][from_node][to_node]
transit_callback_index = routing.RegisterTransitCallback(distance_callback)
routing.SetArcCostEvaluatorOfAllVehicles(transit_callback_index)
# Add Distance constraint.
dimension_name = 'Distance'
routing.AddDimension(
transit_callback_index,
0, # no slack
3000, # vehicle maximum travel distance
True, # start cumul to zero
dimension_name)
distance_dimension = routing.GetDimensionOrDie(dimension_name)
distance_dimension.SetGlobalSpanCostCoefficient(100)
# Define Transportation Requests.
for request in data['pickups_deliveries']:
pickup_index = manager.NodeToIndex(request[0])
delivery_index = manager.NodeToIndex(request[1])
routing.AddPickupAndDelivery(pickup_index, delivery_index)
routing.solver().Add(
routing.VehicleVar(pickup_index) == routing.VehicleVar(
delivery_index))
routing.solver().Add(
distance_dimension.CumulVar(pickup_index) <=
distance_dimension.CumulVar(delivery_index))
for node, revenue in data["revenue"].items():
start, end = node
routing.AddDisjunction(
[manager.NodeToIndex(end)], revenue
)
routing.AddDisjunction(
[manager.NodeToIndex(start)], 0
)
# Setting first solution heuristic.
search_parameters = pywrapcp.DefaultRoutingSearchParameters()
search_parameters.first_solution_strategy = (
routing_enums_pb2.FirstSolutionStrategy.PARALLEL_CHEAPEST_INSERTION)
# Solve the problem.
solution = routing.SolveWithParameters(search_parameters)
# Print solution on console.
if solution:
print_solution(data, manager, routing, solution)
if __name__ == '__main__':
main()
polyhedron not filled (openscad)
i have a polyhedron that seams to be well-formed with no overlap. i have no error or warning when i press f6, and when i press f12 to check if i have missordered faces , no pink face are displayed from outside ( all faces from inside the object are all pink which is consistent ) . i have to do a difference between this object and another one but my polyhedron is never solid. Did i missunderstood something ? thanks a lot for any advices. points = [[5.01, -10.505, -1.5], [6.5345, -10.5048, -1.5], [8.059, -10.5045, -1.5], [9.5835, -10.5042, -1.5], [11.108, -10.504, -1.5], [12.6325, -10.5038, -1.5], [14.157, -10.5035, -1.5], [15.6815, -10.5033, -1.5], [17.206, -10.503, -1.5], [18.7305, -10.5028, -1.5], [20.255, -10.5025, -1.5], [21.7795, -10.5023, -1.5], [23.304, -10.502, -1.5], [24.8285, -10.5018, -1.5], [26.353, -10.5015, -1.5], [27.8775, -10.5012, -1.5], [29.402, -10.501, -1.5], [30.9265, -10.5008, -1.5], [32.451, -10.5005, -1.5], [33.9755, -10.5002, -1.5], [35.5, -10.5, -1.5], [5, -10.5, -1.5], [5.31731, -10.5, -0.733875], [5.9485, -10.5, -0.081], [6.86244, -10.5, 0.465375], [8.028, -10.5, 0.912], [9.41406, -10.5, 1.26562], [10.9895, -10.5, 1.533], [12.7232, -10.5, 1.72087], [14.584, -10.5, 1.836], [16.5408, -10.5, 1.88512], [18.5625, -10.5, 1.875], [20.6179, -10.5, 1.81238], [22.676, -10.5, 1.704], [24.7056, -10.5, 1.55662], [26.6755, -10.5, 1.377], [28.5547, -10.5, 1.17187], [30.312, -10.5, 0.948], [31.9163, -10.5, 0.712125], [33.3365, -10.5, 0.471], [34.5414, -10.5, 0.231375], [35.5, -10.5, 0], [2.5, -8.5, -1.5], [3.54437, -8.78162, 0.442813], [4.72, -9.028, 2.0875], [6.01562, -9.24137, 3.45844], [7.42, -9.424, 4.58], [8.92188, -9.57812, 5.47656], [10.51, -9.706, 6.1725], [12.1731, -9.80988, 6.69219], [13.9, -9.892, 7.06], [15.6794, -9.95463, 7.30031], [17.5, -10, 7.4375], [19.3506, -10.0304, 7.49594], [21.22, -10.048, 7.5], [23.0969, -10.0551, 7.47406], [24.97, -10.054, 7.4425], [26.8281, -10.0469, 7.42969], [28.66, -10.036, 7.46], [30.4544, -10.0236, 7.55781], [32.2, -10.012, 7.7475], [33.8856, -10.0034, 8.05344], [35.5, -10, 8.5], [-1, -5, -1.5], [0.543563, -5.30675, 0.389375], [2.1685, -5.624, 2.02], [3.86619, -5.94725, 3.41063], [5.628, -6.272, 4.58], [7.44531, -6.59375, 5.54688], [9.3095, -6.908, 6.33], [11.2119, -7.21025, 6.94812], [13.144, -7.496, 7.42], [15.0971, -7.76075, 7.76438], [17.0625, -8, 8], [19.0317, -8.20925, 8.14562], [20.996, -8.384, 8.22], [22.9468, -8.51975, 8.24188], [24.8755, -8.612, 8.23], [26.7734, -8.65625, 8.20312], [28.632, -8.648, 8.18], [30.4426, -8.58275, 8.17938], [32.1965, -8.456, 8.22], [33.8852, -8.26325, 8.32063], [35.5, -8, 8.5], [-2, -3, -1.5], [-0.584562, -3.30662, 0.788375], [0.9535, -3.623, 2.722], [2.60181, -3.94387, 4.32863], [4.348, -4.264, 5.636], [6.17969, -4.57812, 6.67188], [8.0845, -4.881, 7.464], [10.0501, -5.16738, 8.04012], [12.064, -5.432, 8.428], [14.1139, -5.66962, 8.65537], [16.1875, -5.875, 8.75], [18.2723, -6.04288, 8.73963], [20.356, -6.168, 8.652], [22.4262, -6.24512, 8.51487], [24.4705, -6.269, 8.356], [26.4766, -6.23438, 8.20312], [28.432, -6.136, 8.084], [30.3244, -5.96863, 8.02638], [32.1415, -5.727, 8.058], [33.8708, -5.40587, 8.20663], [35.5, -5, 8.5], [-3, 0, -1.5], [-1.13556, 0, 1.20887], [0.8455, 0, 3.506], [2.92481, 0, 5.42212], [5.084, 0, 6.988], [7.30469, 0, 8.23438], [9.5685, 0, 9.192], [11.8571, 0, 9.89162], [14.152, 0, 10.364], [16.4349, 0, 10.6399], [18.6875, 0, 10.75], [20.8913, 0, 10.7251], [23.028, 0, 10.596], [25.0792, 0, 10.3934], [27.0265, 0, 10.148], [28.8516, 0, 9.89062], [30.536, 0, 9.652], [32.0614, 0, 9.46287], [33.4095, 0, 9.354], [34.5618, 0, 9.35613], [35.5, 0, 9.5], [-2, 3, -1.5], [-0.584562, 3.30662, 0.788375], [0.9535, 3.623, 2.722], [2.60181, 3.94387, 4.32863], [4.348, 4.264, 5.636], [6.17969, 4.57812, 6.67188], [8.0845, 4.881, 7.464], [10.0501, 5.16738, 8.04012], [12.064, 5.432, 8.428], [14.1139, 5.66962, 8.65537], [16.1875, 5.875, 8.75], [18.2723, 6.04288, 8.73963], [20.356, 6.168, 8.652], [22.4262, 6.24512, 8.51487], [24.4705, 6.269, 8.356], [26.4766, 6.23438, 8.20312], [28.432, 6.136, 8.084], [30.3244, 5.96863, 8.02638], [32.1415, 5.727, 8.058], [33.8708, 5.40587, 8.20663], [35.5, 5, 8.5], [-1, 5, -1.5], [0.543563, 5.30675, 0.389375], [2.1685, 5.624, 2.02], [3.86619, 5.94725, 3.41063], [5.628, 6.272, 4.58], [7.44531, 6.59375, 5.54688], [9.3095, 6.908, 6.33], [11.2119, 7.21025, 6.94812], [13.144, 7.496, 7.42], [15.0971, 7.76075, 7.76438], [17.0625, 8, 8], [19.0317, 8.20925, 8.14562], [20.996, 8.384, 8.22], [22.9468, 8.51975, 8.24188], [24.8755, 8.612, 8.23], [26.7734, 8.65625, 8.20312], [28.632, 8.648, 8.18], [30.4426, 8.58275, 8.17938], [32.1965, 8.456, 8.22], [33.8852, 8.26325, 8.32063], [35.5, 8, 8.5], [2.5, 8.5, -1.5], [3.54437, 8.78162, 0.442813], [4.72, 9.028, 2.0875], [6.01562, 9.24137, 3.45844], [7.42, 9.424, 4.58], [8.92188, 9.57812, 5.47656], [10.51, 9.706, 6.1725], [12.1731, 9.80988, 6.69219], [13.9, 9.892, 7.06], [15.6794, 9.95463, 7.30031], [17.5, 10, 7.4375], [19.3506, 10.0304, 7.49594], [21.22, 10.048, 7.5], [23.0969, 10.0551, 7.47406], [24.97, 10.054, 7.4425], [26.8281, 10.0469, 7.42969], [28.66, 10.036, 7.46], [30.4544, 10.0236, 7.55781], [32.2, 10.012, 7.7475], [33.8856, 10.0034, 8.05344], [35.5, 10, 8.5], [5, 10.5, -1.5], [5.31731, 10.5, -0.733875], [5.9485, 10.5, -0.081], [6.86244, 10.5, 0.465375], [8.028, 10.5, 0.912], [9.41406, 10.5, 1.26562], [10.9895, 10.5, 1.533], [12.7232, 10.5, 1.72087], [14.584, 10.5, 1.836], [16.5408, 10.5, 1.88512], [18.5625, 10.5, 1.875], [20.6179, 10.5, 1.81238], [22.676, 10.5, 1.704], [24.7056, 10.5, 1.55662], [26.6755, 10.5, 1.377], [28.5547, 10.5, 1.17187], [30.312, 10.5, 0.948], [31.9163, 10.5, 0.712125], [33.3365, 10.5, 0.471], [34.5414, 10.5, 0.231375], [35.5, 10.5, 0], [5.01, 10.505, -1.5], [6.5345, 10.5048, -1.5], [8.059, 10.5045, -1.5], [9.5835, 10.5042, -1.5], [11.108, 10.504, -1.5], [12.6325, 10.5038, -1.5], [14.157, 10.5035, -1.5], [15.6815, 10.5033, -1.5], [17.206, 10.503, -1.5], [18.7305, 10.5028, -1.5], [20.255, 10.5025, -1.5], [21.7795, 10.5023, -1.5], [23.304, 10.502, -1.5], [24.8285, 10.5018, -1.5], [26.353, 10.5015, -1.5], [27.8775, 10.5012, -1.5], [29.402, 10.501, -1.5], [30.9265, 10.5008, -1.5], [32.451, 10.5005, -1.5], [33.9755, 10.5002, -1.5], [35.5, 10.5, -1.5]]; faces = [[0, 21, 1], [21, 22, 1], [1, 22, 2], [22, 23, 2], [2, 23, 3], [23, 24, 3], [3, 24, 4], [24, 25, 4], [4, 25, 5], [25, 26, 5], [5, 26, 6], [26, 27, 6], [6, 27, 7], [27, 28, 7], [7, 28, 8], [28, 29, 8], [8, 29, 9], [29, 30, 9], [9, 30, 10], [30, 31, 10], [10, 31, 11], [31, 32, 11], [11, 32, 12], [32, 33, 12], [12, 33, 13], [33, 34, 13], [13, 34, 14], [34, 35, 14], [14, 35, 15], [35, 36, 15], [15, 36, 16], [36, 37, 16], [16, 37, 17], [37, 38, 17], [17, 38, 18], [38, 39, 18], [18, 39, 19], [39, 40, 19], [19, 40, 20], [40, 41, 20], [21, 42, 22], [42, 43, 22], [22, 43, 23], [43, 44, 23], [23, 44, 24], [44, 45, 24], [24, 45, 25], [45, 46, 25], [25, 46, 26], [46, 47, 26], [26, 47, 27], [47, 48, 27], [27, 48, 28], [48, 49, 28], [28, 49, 29], [49, 50, 29], [29, 50, 30], [50, 51, 30], [30, 51, 31], [51, 52, 31], [31, 52, 32], [52, 53, 32], [32, 53, 33], [53, 54, 33], [33, 54, 34], [54, 55, 34], [34, 55, 35], [55, 56, 35], [35, 56, 36], [56, 57, 36], [36, 57, 37], [57, 58, 37], [37, 58, 38], [58, 59, 38], [38, 59, 39], [59, 60, 39], [39, 60, 40], [60, 61, 40], [40, 61, 41], [61, 62, 41], [42, 63, 43], [63, 64, 43], [43, 64, 44], [64, 65, 44], [44, 65, 45], [65, 66, 45], [45, 66, 46], [66, 67, 46], [46, 67, 47], [67, 68, 47], [47, 68, 48], [68, 69, 48], [48, 69, 49], [69, 70, 49], [49, 70, 50], [70, 71, 50], [50, 71, 51], [71, 72, 51], [51, 72, 52], [72, 73, 52], [52, 73, 53], [73, 74, 53], [53, 74, 54], [74, 75, 54], [54, 75, 55], [75, 76, 55], [55, 76, 56], [76, 77, 56], [56, 77, 57], [77, 78, 57], [57, 78, 58], [78, 79, 58], [58, 79, 59], [79, 80, 59], [59, 80, 60], [80, 81, 60], [60, 81, 61], [81, 82, 61], [61, 82, 62], [82, 83, 62], [63, 84, 64], [84, 85, 64], [64, 85, 65], [85, 86, 65], [65, 86, 66], [86, 87, 66], [66, 87, 67], [87, 88, 67], [67, 88, 68], [88, 89, 68], [68, 89, 69], [89, 90, 69], [69, 90, 70], [90, 91, 70], [70, 91, 71], [91, 92, 71], [71, 92, 72], [92, 93, 72], [72, 93, 73], [93, 94, 73], [73, 94, 74], [94, 95, 74], [74, 95, 75], [95, 96, 75], [75, 96, 76], [96, 97, 76], [76, 97, 77], [97, 98, 77], [77, 98, 78], [98, 99, 78], [78, 99, 79], [99, 100, 79], [79, 100, 80], [100, 101, 80], [80, 101, 81], [101, 102, 81], [81, 102, 82], [102, 103, 82], [82, 103, 83], [103, 104, 83], [84, 105, 85], [105, 106, 85], [85, 106, 86], [106, 107, 86], [86, 107, 87], [107, 108, 87], [87, 108, 88], [108, 109, 88], [88, 109, 89], [109, 110, 89], [89, 110, 90], [110, 111, 90], [90, 111, 91], [111, 112, 91], [91, 112, 92], [112, 113, 92], [92, 113, 93], [113, 114, 93], [93, 114, 94], [114, 115, 94], [94, 115, 95], [115, 116, 95], [95, 116, 96], [116, 117, 96], [96, 117, 97], [117, 118, 97], [97, 118, 98], [118, 119, 98], [98, 119, 99], [119, 120, 99], [99, 120, 100], [120, 121, 100], [100, 121, 101], [121, 122, 101], [101, 122, 102], [122, 123, 102], [102, 123, 103], [123, 124, 103], [103, 124, 104], [124, 125, 104], [105, 126, 106], [126, 127, 106], [106, 127, 107], [127, 128, 107], [107, 128, 108], [128, 129, 108], [108, 129, 109], [129, 130, 109], [109, 130, 110], [130, 131, 110], [110, 131, 111], [131, 132, 111], [111, 132, 112], [132, 133, 112], [112, 133, 113], [133, 134, 113], [113, 134, 114], [134, 135, 114], [114, 135, 115], [135, 136, 115], [115, 136, 116], [136, 137, 116], [116, 137, 117], [137, 138, 117], [117, 138, 118], [138, 139, 118], [118, 139, 119], [139, 140, 119], [119, 140, 120], [140, 141, 120], [120, 141, 121], [141, 142, 121], [121, 142, 122], [142, 143, 122], [122, 143, 123], [143, 144, 123], [123, 144, 124], [144, 145, 124], [124, 145, 125], [145, 146, 125], [126, 147, 127], [147, 148, 127], [127, 148, 128], [148, 149, 128], [128, 149, 129], [149, 150, 129], [129, 150, 130], [150, 151, 130], [130, 151, 131], [151, 152, 131], [131, 152, 132], [152, 153, 132], [132, 153, 133], [153, 154, 133], [133, 154, 134], [154, 155, 134], [134, 155, 135], [155, 156, 135], [135, 156, 136], [156, 157, 136], [136, 157, 137], [157, 158, 137], [137, 158, 138], [158, 159, 138], [138, 159, 139], [159, 160, 139], [139, 160, 140], [160, 161, 140], [140, 161, 141], [161, 162, 141], [141, 162, 142], [162, 163, 142], [142, 163, 143], [163, 164, 143], [143, 164, 144], [164, 165, 144], [144, 165, 145], [165, 166, 145], [145, 166, 146], [166, 167, 146], [147, 168, 148], [168, 169, 148], [148, 169, 149], [169, 170, 149], [149, 170, 150], [170, 171, 150], [150, 171, 151], [171, 172, 151], [151, 172, 152], [172, 173, 152], [152, 173, 153], [173, 174, 153], [153, 174, 154], [174, 175, 154], [154, 175, 155], [175, 176, 155], [155, 176, 156], [176, 177, 156], [156, 177, 157], [177, 178, 157], [157, 178, 158], [178, 179, 158], [158, 179, 159], [179, 180, 159], [159, 180, 160], [180, 181, 160], [160, 181, 161], [181, 182, 161], [161, 182, 162], [182, 183, 162], [162, 183, 163], [183, 184, 163], [163, 184, 164], [184, 185, 164], [164, 185, 165], [185, 186, 165], [165, 186, 166], [186, 187, 166], [166, 187, 167], [187, 188, 167], [168, 189, 169], [189, 190, 169], [169, 190, 170], [190, 191, 170], [170, 191, 171], [191, 192, 171], [171, 192, 172], [192, 193, 172], [172, 193, 173], [193, 194, 173], [173, 194, 174], [194, 195, 174], [174, 195, 175], [195, 196, 175], [175, 196, 176], [196, 197, 176], [176, 197, 177], [197, 198, 177], [177, 198, 178], [198, 199, 178], [178, 199, 179], [199, 200, 179], [179, 200, 180], [200, 201, 180], [180, 201, 181], [201, 202, 181], [181, 202, 182], [202, 203, 182], [182, 203, 183], [203, 204, 183], [183, 204, 184], [204, 205, 184], [184, 205, 185], [205, 206, 185], [185, 206, 186], [206, 207, 186], [186, 207, 187], [207, 208, 187], [187, 208, 188], [208, 209, 188], [189, 210, 190], [210, 211, 190], [190, 211, 191], [211, 212, 191], [191, 212, 192], [212, 213, 192], [192, 213, 193], [213, 214, 193], [193, 214, 194], [214, 215, 194], [194, 215, 195], [215, 216, 195], [195, 216, 196], [216, 217, 196], [196, 217, 197], [217, 218, 197], [197, 218, 198], [218, 219, 198], [198, 219, 199], [219, 220, 199], [199, 220, 200], [220, 221, 200], [200, 221, 201], [221, 222, 201], [201, 222, 202], [222, 223, 202], [202, 223, 203], [223, 224, 203], [203, 224, 204], [224, 225, 204], [204, 225, 205], [225, 226, 205], [205, 226, 206], [226, 227, 206], [206, 227, 207], [227, 228, 207], [207, 228, 208], [228, 229, 208], [208, 229, 209], [229, 230, 209], [20, 41, 62, 83, 104, 125, 146, 167, 188, 209, 230], [20, 230, 210, 189, 168, 147, 126, 105, 84, 63, 42, 21, 0]]; polyhedron(points,faces);
Spark Scala TF-IDF value sorted vectors
So far I have been able to tokenize all of my documents, and use CountVectorizer and IDF from Spark's MLLib. I am trying to get the top 50 words from each document, but I am not sure how to sort the output of IDF.
onePer is a dataframe of document IDs and tokenized documents.
val tf = new CountVectorizer()
.setInputCol("text")
.setOutputCol("features").fit(onePer)
.transform(onePer).select("features").rdd
.map{x:Row => x.getAs[Vector](0)}
tf.cache()
val idf = new IDF().fit(tf)
val tfidf: RDD[Vector] = idf.transform(tf)
This is what my output looks like (number of words in vocab, id of word, word score). I would like to sort by score and get the top k:
(440,[0,2,3,4,5,6,7,8,9,10,12,15,17,18,19,22,23,24,25,26,27,28,30,31,32,33,34,35,39,41,43,45,47,49,51,52,53,55,57,63,66,69,70,71,74,76,79,80,83,84,85,88,94,95,96,97,99,102,106,107,109,111,117,120,121,124,127,128,129,138,142,145,146,149,154,156,164,166,167,170,171,176,187,189,199,203,204,217,218,219,232,234,236,237,238,240,248,250,251,254,259,263,265,267,280,291,296,302,304,309,319,322,328,333,347,361,364,371,375,384,388,393,395,401,403,433,438,439],[1.3559553712291716,3.9422868018213513,0.6369074622370692,7.795697904781566,3.153829441457081,0.0,5.519201522549892,0.3184537311185346,0.3184537311185346,1.3559553712291716,0.4519851237430572,0.4519851237430572,0.6061358035703155,1.0116009116784799,0.4519851237430572,0.7884573603642703,0.4519851237430572,2.0232018233569597,0.7884573603642703,8.523740461192126,0.6061358035703155,0.6061358035703155,0.6061358035703155,0.6061358035703155,0.7884573603642703,0.6061358035703155,0.6061358035703155,0.6061358035703155,0.7884573603642703,0.7884573603642703,1.0116009116784799,1.0116009116784799,2.0232018233569597,0.7884573603642703,0.7884573603642703,3.897848952390783,0.7884573603642703,0.7884573603642703,1.0116009116784799,5.114244276715276,1.0116009116784799,1.0116009116784799,2.5985659682605218,1.2992829841302609,1.2992829841302609,1.0116009116784799,1.0116009116784799,1.0116009116784799,1.0116009116784799,1.0116009116784799,2.5985659682605218,1.0116009116784799,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,3.4094961844768505,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,3.4094961844768505,1.2992829841302609,1.2992829841302609,1.2992829841302609,3.4094961844768505,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253])
Update
I was able to get this working by doing the following:
tfidf.map(x => x.toSparse).map{x => x.indices.zip(x.values)
.sortBy(-_._2)
.take(10)
.map(_._1)
}
This might help:
scala> val x = (440,Array[Int](0,2,3,4,5,6,7,8,9,10,12,15,17,18,19,22,23,24,25,26,27,28,30,31,32,33,34,35,39,41,43,45,47,49,51,52,53,55,57,63,66,69,70,71,74,76,79,80,83,84,85,88,94,95,96,97,99,102,106,107,109,111,117,120,121,124,127,128,129,138,142,145,146,149,154,156,164,166,167,170,171,176,187,189,199,203,204,217,218,219,232,234,236,237,238,240,248,250,251,254,259,263,265,267,280,291,296,302,304,309,319,322,328,333,347,361,364,371,375,384,388,393,395,401,403,433,438,439),Array[Double](1.3559553712291716,3.9422868018213513,0.6369074622370692,7.795697904781566,3.153829441457081,0.0,5.519201522549892,0.3184537311185346,0.3184537311185346,1.3559553712291716,0.4519851237430572,0.4519851237430572,0.6061358035703155,1.0116009116784799,0.4519851237430572,0.7884573603642703,0.4519851237430572,2.0232018233569597,0.7884573603642703,8.523740461192126,0.6061358035703155,0.6061358035703155,0.6061358035703155,0.6061358035703155,0.7884573603642703,0.6061358035703155,0.6061358035703155,0.6061358035703155,0.7884573603642703,0.7884573603642703,1.0116009116784799,1.0116009116784799,2.0232018233569597,0.7884573603642703,0.7884573603642703,3.897848952390783,0.7884573603642703,0.7884573603642703,1.0116009116784799,5.114244276715276,1.0116009116784799,1.0116009116784799,2.5985659682605218,1.2992829841302609,1.2992829841302609,1.0116009116784799,1.0116009116784799,1.0116009116784799,1.0116009116784799,1.0116009116784799,2.5985659682605218,1.0116009116784799,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,3.4094961844768505,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,3.4094961844768505,1.2992829841302609,1.2992829841302609,1.2992829841302609,3.4094961844768505,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.2992829841302609,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253,1.7047480922384253))
scala> val (r, indices, values) = x
r: Int = 440
indices: Array[Int] = Array(0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 17, 18, 19, 22, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 35, 39, 41, 43, 45, 47, 49, 51, 52, 53, 55, 57, 63, 66, 69, 70, 71, 74, 76, 79, 80, 83, 84, 85, 88, 94, 95, 96, 97, 99, 102, 106, 107, 109, 111, 117, 120, 121, 124, 127, 128, 129, 138, 142, 145, 146, 149, 154, 156, 164, 166, 167, 170, 171, 176, 187, 189, 199, 203, 204, 217, 218, 219, 232, 234, 236, 237, 238, 240, 248, 250, 251, 254, 259, 263, 265, 267, 280, 291, 296, 302, 304, 309, 319, 322, 328, 333, 347, 361, 364, 371, 375, 384, 388, 393, 395, 401, 403, 433, 438, 439)
values: Array[Double] = Array(1.3559553712291716, 3.9422868018213513, 0.6369074622370692, 7.795697904781566, 3.153829441457081, 0.0, 5.519201522549892, 0.3184537311185346, 0.31845373...
scala> val topTermIds = indices.zip(values).sortBy( - _._2).take(50).map(_._1)
topTermIds: Array[Int] = Array(26, 4, 7, 63, 2, 52, 109, 124, 138, 5, 70, 85, 24, 47, 176, 187, 189, 199, 203, 204, 217, 218, 219, 232, 234, 236, 237, 238, 240, 248, 250, 251, 254, 259, 263, 265, 267, 280, 291, 296, 302, 304, 309, 319, 322, 328, 333, 347, 361, 364)
Now you need to plug in above code into a closure, something like:
val topTermsByScore = rdd.map { v: Vector =>
// to sort decreasing use -
v.indices.zip(v.values).sortBy( - _._2).take(50).map(_._1)
}