Related
I'm experimenting with huggingface transformers to finetune microsoft/layoutlmv2-base-uncased through AutoModelForTokenClassification on my custom dataset that is similar to FUNSD (pre-processed and normalized). After a few iterations of training I get this error :
Traceback (most recent call last):
File "layoutlmV2/train.py", line 137, in <module>
trainer.train()
File "..../lib/python3.8/site-packages/transformers/trainer.py", line 1409, in train
return inner_training_loop(
File "..../lib/python3.8/site-packages/transformers/trainer.py", line 1651, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "..../lib/python3.8/site-packages/transformers/trainer.py", line 2345, in training_step
loss = self.compute_loss(model, inputs)
File "..../lib/python3.8/site-packages/transformers/trainer.py", line 2377, in compute_loss
outputs = model(**inputs)
File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
return forward_call(*input, **kwargs)
File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 1228, in forward
outputs = self.layoutlmv2(
File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
return forward_call(*input, **kwargs)
File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 902, in forward
text_layout_emb = self._calc_text_embeddings(
File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 753, in _calc_text_embeddings
spatial_position_embeddings = self.embeddings._calc_spatial_position_embeddings(bbox)
File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 93, in _calc_spatial_position_embeddings
h_position_embeddings = self.h_position_embeddings(bbox[:, :, 3] - bbox[:, :, 1])
File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
return forward_call(*input, **kwargs)
File "..../lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
return F.embedding(
File "..../lib/python3.8/site-packages/torch/nn/functional.py", line 2203, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
After further inspection (vocab size, bboxes, dimensions, classes...) I noticed that there's negative values inside the input tensor causing the error. While input tensors of successful previous iterations have unsigned integers only. These negative numbers are returned by _calc_spatial_position_embeddings(self, bbox) in modeling_layoutlmv2.py
line 92 :
h_position_embeddings = self.h_position_embeddings(bbox[:, :, 3] - bbox[:, :, 1])
What may cause the returned input values to be negative?
What could I do to prevent this error from happening?
Example of the input tensor that triggers the error in torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) :
tensor([[ 0, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 10, 10, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12,
12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
8, 5, 5, 5, 5, 5, 5, -6, -6, -6, -6, -6, -6, 1, 1, 1, 1, 1,
5, 5, 5, 5, 5, 5, 7, 5, 7, 7, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0]])
After double checking the dataset and specifically the coordinates of the labels, I've found that some rows bbox coordinates lead to zero width or height. Here's a simplified example:
x1, y1, x2, y2 = dataset_row["bbox"]
print((x2-x1 < 1) or (y2-y1 < 1)) #output is sometimes True
After removing these labels from the dataset, the issue was resolved.
I'm in the process of writing a utility in Swift which can be used to calculate CRC32 checksum of an input data . A similar utility exists in Java which we are using extensively and has been working well for us.
The Java Utility uses java.util.zip.CRC32 to calculate the checksum. Pseudo code is as follows :
Java code:
private void transferFileData(short index, byte[] data, long dataSize) {
CRC32 crc32 = new CRC32();
long crc = crc32.update(data, (int) dataSize);
System.out.println("CRC32 : " + crc);
}
The Swift uses the CRC32 (import CryptoSwift) CryptoSwift code for generating the checksum in swift is as follows :
Swift code:
func crc32Func(_ item:[Int8] ) {
let data = Data(bytes: item, count: item.count)
let byte = data.bytes
let crc32 = byte.crc32()
print("checksum == \(crc32)"
}
The output from the Java code is :
Checksum in Java : 3771181957
The output from the swift code is :
Checksum in swift : 1894162356
Why the checksum values are not the same?
This is the code applied by Swift as below:
The data type is [Int8]
The data size is 450 bytes.
let item:
[Int8] = [45, 35, 76, 70, 67, 68, 95, 70, 79, 84, 65, 95, 70, 87, 95,
70, 85, 76, 76, 10, 80, 75, 71, 95, 86 , 69, 82, 83, 73, 79, 78, 58,
51, 46, 48, 46, 48, 10, 66, 65, 83, 69, 95, 86, 69, 82, 83, 73, 79, 78
, 58, 10, 72, 65, 83, 72, 58, 51, 97, 57, 98, 51, 99, 52, 56, 57, 52,
56, 56, 53, 49, 50, 53, 52, 55 , 48, 51, 49, 54, 48, 57, 56, 48, 101,
98, 101, 51, 54, 54, 10, 80, 75, 71, 95, 83, 73, 90, 69, 58, 49 , 51,
49, 48, 55, 50, 48, 10, 26, 0, 0, 5, 32, -79, 11, 5, 8, -39, -5, 4, 8,
-35, -5, 4, 8, -15, -5, 4, 8, -13, -5, 4, 8, -11, -5, 4, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -31, -15, 10, 8, -9, -5, 4, 8,
0, 0, 0, 0, -127, -14, 10, 8, -27, -14, 10, 8, 1, 12, 5, 8, 1, 12, 5,
8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1,
12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12,
5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8,
-7, -5, 4, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 9, -4,
4, 8, 25, -4, 4, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5,
8, 1, 12, 5, 8, 57, -4, 4, 8, 1, 12, 5, 8, 73, -4, 4, 8, 89, -4, 4, 8,
105, -4, 4, 8, 1, 12, 5, 8, -39, -4, 4, 8, 1, 12, 5, 8, 1, 12, 5, 8,
1, 12, 5, 8, 41, -4, 4, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1,
12, 5 , 8, 1, 12, 5, 8, 1, 12, 5, 8, 121, -4, 4, 8, -119, -4, 4, 8, 1,
12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12,
5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8,
1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, -103, -4]
is input.
Java input:
byte[] data
The type is long and the data entered is the same as [Int8] of swift data.
The CRC you are getting from your Swift code (whatever that actual code is) is correct. The CRC you are getting from your Java code (whatever that actual code is) is not correct.
There is no way to know what you're doing wrong in your Java code without at least being able to see that code.
The title might seem a bit confusing, but what I'm trying to do is rather simple.
I have 2 lists, one is:
List(12, 14, 16, 24, 26, 32, 36, 43, 44, 46, 52, 54, 56, 62, 66, 72, 74, 76)
And the second one is:
List(11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29, 31, 32)
I want to iterate through the first list and see if there exists a value in the first list that would be the sum of a previous value and another value from the second list. If a value like that exists, it should get summed with the same value from the second list and check if a value like that exists in the first list and so on until it reaches the maximum obtainable value in the first list.
For example: If we take the value 16 from the first list, and keep summing it with 28, we get the values:
16, 44 and 72
Or: We take 24 from the first list, keep adding 19 to it and getting the values:
24, 43 and 62
If, including the starting value, at least 2 more values can't be found by adding the same number, it should just take the next value in the list and do the same calculations.
Any help would be appreciated, I tried using the fold and reduce functions, but I couldn't get them to work.
EDIT:
Tried using Stream.iterate to do the job, but It didn't really work
val iTried = Stream.iterate(0)(i => i + 1)
.takeWhile(i => coordinatesRows(i)
.toString exists coordinatesRows + valuesForDiag(i))
Also tried making a function of my own:
def sortDiagonally(xs: List[Int]) ={
/**
**/
val start = xs.head
Stream.from(1).flatMap{ value =>
xs exists (start + valuesForDiag == xs)
}
I'm still learning the language, so I'm not too familiar with all the inns and outs of Scala.
EDIT 2:
By using these 2 lists, I would like to get the output of:
List(List(16, 44, 72), List(24, 43, 62), List(32, 54, 76), List(36, 54, 72), List(12, 24, 36), List(12, 44, 76))
I also removed the value of 10, 20 and 30 from the 2nd list, as it shouldn't have been there.
val vals = List(12, 14, 16, 24, 26, 32, 36, 43, 44, 46, 52, 54, 56, 62, 66, 72, 74, 76)
val mods = List(10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32)
def find(l: List[Int]): Set[Int] = l match {
case Nil => Set.empty[Int]
case x :: xs =>
xs.filter{
i => mods.exists(j => (i - x) % j == 0)
}.toSet ++ find(xs)
}
find(vals)
Returns:
Set(56, 24, 52, 46, 74, 32, 44, 54, 76, 66, 72, 43, 26, 36, 62)
All matching values:
def find(l: List[Int]): List[(Int, Int, Int)] = l match {
case Nil => Nil
case x :: xs =>
xs.flatMap{
i => mods.collect{ case j if (i - x) % j == 0 => (x, i, j) }
} ++ find(xs)
}
println(find(vals))
Returns:
List((12,24,12), (12,26,14), (12,32,10), (12,32,20), (12,36,12), (12,36,24), (12,43,31), (12,44,16), (12,44,32), (12,46,17), (12,52,10), (12,52,20), (12,54,14), (12,54,21), (12,56,11), (12,56,22), (12,62,10), (12,62,25), (12,66,18), (12,66,27), (12,72,10), (12,72,12), (12,72,15), (12,72,20), (12,72,30), (12,74,31), (12,76,16), (12,76,32), (14,24,10), (14,26,12), (14,32,18), (14,36,11), (14,36,22), (14,43,29), (14,44,10), (14,44,15), (14,44,30), (14,46,16), (14,46,32), (14,52,19), (14,54,10), (14,54,20), (14,56,14), (14,56,21), (14,62,12), (14,62,16), (14,62,24), (14,66,13), (14,66,26), (14,72,29), (14,74,10), (14,74,12), (14,74,15), (14,74,20), (14,74,30), (14,76,31), (16,26,10), (16,32,16), (16,36,10), (16,36,20), (16,43,27), (16,44,14), (16,44,28), (16,46,10), (16,46,15), (16,46,30), (16,52,12), (16,52,18), (16,54,19), (16,56,10), (16,56,20), (16,62,23), (16,66,10), (16,66,25), (16,72,14), (16,72,28), (16,74,29), (16,76,10), (16,76,12), (16,76,15), (16,76,20), (16,76,30), (24,36,12), (24,43,19), (24,44,10), (24,44,20), (24,46,11), (24,46,22), (24,52,14), (24,52,28), (24,54,10), (24,54,15), (24,54,30), (24,56,16), (24,56,32), (24,62,19), (24,66,14), (24,66,21), (24,72,12), (24,72,16), (24,72,24), (24,74,10), (24,74,25), (24,76,13), (24,76,26), (26,36,10), (26,43,17), (26,44,18), (26,46,10), (26,46,20), (26,52,13), (26,52,26), (26,54,14), (26,54,28), (26,56,10), (26,56,15), (26,56,30), (26,62,12), (26,62,18), (26,66,10), (26,66,20), (26,72,23), (26,74,12), (26,74,16), (26,74,24), (26,76,10), (26,76,25), (32,43,11), (32,44,12), (32,46,14), (32,52,10), (32,52,20), (32,54,11), (32,54,22), (32,56,12), (32,56,24), (32,62,10), (32,62,15), (32,62,30), (32,66,17), (32,72,10), (32,72,20), (32,74,14), (32,74,21), (32,76,11), (32,76,22), (36,46,10), (36,52,16), (36,54,18), (36,56,10), (36,56,20), (36,62,13), (36,62,26), (36,66,10), (36,66,15), (36,66,30), (36,72,12), (36,72,18), (36,74,19), (36,76,10), (36,76,20), (43,54,11), (43,56,13), (43,62,19), (43,66,23), (43,72,29), (43,74,31), (43,76,11), (44,54,10), (44,56,12), (44,62,18), (44,66,11), (44,66,22), (44,72,14), (44,72,28), (44,74,10), (44,74,15), (44,74,30), (44,76,16), (44,76,32), (46,56,10), (46,62,16), (46,66,10), (46,66,20), (46,72,13), (46,72,26), (46,74,14), (46,74,28), (46,76,10), (46,76,15), (46,76,30), (52,62,10), (52,66,14), (52,72,10), (52,72,20), (52,74,11), (52,74,22), (52,76,12), (52,76,24), (54,66,12), (54,72,18), (54,74,10), (54,74,20), (54,76,11), (54,76,22), (56,66,10), (56,72,16), (56,74,18), (56,76,10), (56,76,20), (62,72,10), (62,74,12), (62,76,14), (66,76,10))
Finally
def find(l: List[Int]): List[List[Int]] = l match {
case Nil => Nil
case x :: xs =>
mods.map {
mod => x :: xs.collect { case c if (c - x) % mod == 0 => c }
}.filter(_.size > 1) ++ find(xs)
}
find(vals)
Returns:
List(List(12, 32, 52, 62, 72), List(12, 56), List(12, 24, 36, 72), List(12, 26, 54), List(12, 72), List(12, 44, 76), List(12, 46), List(12, 66), List(12, 32, 52, 72), List(12, 54), List(12, 56), List(12, 36), List(12, 62), List(12, 66), List(12, 72), List(12, 43, 74), List(12, 44, 76), List(14, 24, 44, 54, 74), List(14, 36), List(14, 26, 62, 74), List(14, 66), List(14, 56), List(14, 44, 74), List(14, 46, 62), List(14, 32), List(14, 52), List(14, 54, 74), List(14, 56), List(14, 36), List(14, 62), List(14, 66), List(14, 43, 72), List(14, 44, 74), List(14, 76), List(14, 46), List(16, 26, 36, 46, 56, 66, 76), List(16, 52, 76), List(16, 44, 72), List(16, 46, 76), List(16, 32), List(16, 52), List(16, 54), List(16, 36, 56, 76), List(16, 62), List(16, 66), List(16, 43), List(16, 44, 72), List(16, 74), List(16, 46, 76), List(24, 44, 54, 74), List(24, 46), List(24, 36, 72), List(24, 76), List(24, 52, 66), List(24, 54), List(24, 56, 72), List(24, 43, 62), List(24, 44), List(24, 66), List(24, 46), List(24, 72), List(24, 74), List(24, 76), List(24, 52), List(24, 54), List(24, 56), List(26, 36, 46, 56, 66, 76), List(26, 62, 74), List(26, 52), List(26, 54), List(26, 56), List(26, 74), List(26, 43), List(26, 44, 62), List(26, 46, 66), List(26, 72), List(26, 74), List(26, 76), List(26, 52), List(26, 54), List(26, 56), List(32, 52, 62, 72), List(32, 43, 54, 76), List(32, 44, 56), List(32, 46, 74), List(32, 62), List(32, 66), List(32, 52, 72), List(32, 74), List(32, 54, 76), List(32, 56), List(32, 62), List(36, 46, 56, 66, 76), List(36, 72), List(36, 62), List(36, 66), List(36, 52), List(36, 54, 72), List(36, 74), List(36, 56, 76), List(36, 62), List(36, 66), List(43, 54, 76), List(43, 56), List(43, 62), List(43, 66), List(43, 72), List(43, 74), List(44, 54, 74), List(44, 66), List(44, 56), List(44, 72), List(44, 74), List(44, 76), List(44, 62), List(44, 66), List(44, 72), List(44, 74), List(44, 76), List(46, 56, 66, 76), List(46, 72), List(46, 74), List(46, 76), List(46, 62), List(46, 66), List(46, 72), List(46, 74), List(46, 76), List(52, 62, 72), List(52, 74), List(52, 76), List(52, 66), List(52, 72), List(52, 74), List(52, 76), List(54, 74), List(54, 76), List(54, 66), List(54, 72), List(54, 74), List(54, 76), List(56, 66, 76), List(56, 72), List(56, 74), List(56, 76), List(62, 72), List(62, 74), List(62, 76), List(66, 76))
I am using Perl's GD::Graph
module to plot a signal profile. I've managed to get almost everything to look just about right, except that for some reason there is an irritating and unwanted gap between the y-axis and the first plotted point. How can I eliminate this?
Here is the code for the test page I've written (it display on my localhost server, but I tried installing it on my live server so that others here could see the result, it errors - I think there must be some Perl modules missing from my live site):
#!perl
use POSIX;
use CGI ':standard';
use GD::Graph::area;
use GD::Graph::colour qw(:colours);
use strict;
# Graph constants
GD::Graph::colour::add_colour( axes => [ 0x00, 0x00, 0x00 ] );
GD::Graph::colour::add_colour( earth => [ 0x00, 0x00, 0x00 ] );
GD::Graph::colour::add_colour( beam => [ 0x00, 0x00, 0x00 ] );
GD::Graph::colour::add_colour( clutter => [ 0x66, 0x33, 0x00 ] );
GD::Graph::colour::add_colour( terrain => [ 0x00, 0x99, 0x00 ] );
GD::Graph::colour::add_colour( earth => [ 0x00, 0x00, 0x00 ] );
GD::Graph::colour::add_colour( fre1 => [ 0xFF, 0x33, 0x00 ] );
GD::Graph::colour::add_colour( fre2 => [ 0xFF, 0x66, 0x00 ] );
GD::Graph::colour::add_colour( sky1 => [ 0x33, 0x66, 0x99 ] );
GD::Graph::colour::add_colour( sky2 => [ 0x66, 0x99, 0xCC ] );
# All arrays should same number of entries.
my #data = (
[ 0, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 5, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 10, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 15, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 20, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 25, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 30, undef, undef, undef, undef, undef, undef, undef, undef, undef ],
[ 450, 458, 459, 458, 457, 455, 453, 451, 449, 447, 444, 441, 438, 435, 432, 429, 426, 422, 420, 416, 413, 409, 406, 402, 399, 395, 391, 388, 384, 381, 376, 373, 369, 365, 361, 358, 353, 350, 345, 341, 337, 333, 329, 325, 321, 316, 313, 308, 304, 299, 296, 291, 286, 282, 277, 273, 268, 265, 259, 256, 250, 247, 241, 236, 232, 227, 223, 217, 213, 208, 204, 198, 194, 188, 184, 178, 173, 168, 163, 158, 152, 148, 142, 137, 131, 126, 120, 115, 109, 102, 97, 90, 85, 78, 72, 64, 58, 50, 43, 32 ],
[ 450, 453, 451, 449, 447, 444, 441, 438, 435, 432, 428, 426, 422, 418, 415, 411, 408, 404, 400, 396, 393, 389, 386, 381, 378, 374, 369, 366, 361, 358, 353, 350, 345, 342, 337, 334, 329, 326, 321, 316, 313, 308, 305, 300, 296, 291, 288, 283, 279, 274, 271, 266, 261, 257, 252, 249, 244, 240, 235, 231, 226, 222, 217, 212, 208, 203, 199, 194, 190, 185, 181, 176, 172, 166, 162, 157, 151, 147, 142, 138, 132, 128, 123, 118, 113, 108, 103, 98, 92, 87, 82, 76, 71, 65, 60, 54, 49, 42, 36, 27 ],
[ 450, 445, 440, 436, 432, 428, 423, 419, 414, 411, 406, 402, 397, 392, 389, 384, 380, 375, 372, 367, 363, 358, 354, 350, 346, 341, 336, 332, 328, 324, 319, 315, 310, 307, 302, 298, 293, 290, 285, 280, 276, 271, 268, 263, 259, 254, 250, 246, 242, 237, 233, 228, 224, 220, 215, 211, 206, 203, 198, 194, 189, 186, 181, 176, 172, 167, 164, 159, 155, 150, 147, 142, 138, 133, 129, 125, 120, 116, 111, 107, 102, 99, 94, 90, 85, 82, 77, 73, 68, 63, 60, 55, 51, 46, 43, 38, 34, 29, 25, 21 ],
[ 450, 437, 429, 423, 416, 411, 405, 400, 394, 389, 383, 379, 373, 367, 363, 357, 353, 347, 343, 337, 333, 328, 323, 318, 314, 309, 303, 299, 294, 290, 285, 281, 275, 272, 266, 262, 257, 253, 248, 243, 239, 234, 231, 226, 222, 217, 213, 208, 205, 200, 196, 191, 186, 183, 178, 174, 169, 166, 161, 157, 153, 149, 144, 140, 136, 132, 128, 124, 120, 116, 112, 108, 104, 100, 97, 92, 88, 84, 80, 77, 73, 69, 65, 62, 58, 55, 51, 48, 44, 40, 37, 34, 31, 27, 25, 22, 19, 17, 15, 14 ],
[ 450, 432, 422, 415, 406, 400, 393, 387, 380, 375, 368, 363, 357, 350, 345, 339, 334, 328, 324, 318, 313, 307, 303, 297, 293, 287, 281, 277, 271, 267, 262, 258, 252, 248, 243, 239, 233, 229, 224, 219, 215, 210, 206, 201, 197, 192, 188, 183, 180, 175, 171, 166, 161, 158, 153, 149, 145, 141, 136, 133, 128, 125, 120, 116, 112, 108, 105, 100, 97, 93, 89, 85, 82, 78, 75, 71, 67, 64, 60, 57, 53, 50, 46, 43, 40, 37, 34, 31, 28, 25, 22, 19, 17, 15, 13, 11, 10, 8, 8, 9 ],
[ 387, 344, 266, 240, 222, 215, 190, 156, 120, 76, 99, 151, 167, 158, 167, 158, 157, 148, 114, 89, 87, 28, 11, 11, 12, 12, 12, 13, 13, 13, 13, 23, 15, 34, 39, 38, 72, 107, 122, 134, 128, 128, 121, 100, 82, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 15, 20, 20, 15, 15, 15, 15, 14, 14, 14, 14, 60, 53, 13, 13, 12, 12, 12, 12, 11, 11, 11, 10, 10, 9, 9, 9, 8, 8, 7, 7, 6, 6, 5, 5, 4, 4, 3, 2, 2, 1, 26 ],
[ 386, 343, 261, 235, 217, 210, 185, 151, 105, 61, 84, 136, 152, 143, 152, 153, 152, 143, 109, 84, 82, 23, 11, 11, 12, 12, 12, 13, 13, 13, 13, 23, 15, 34, 34, 33, 67, 102, 117, 129, 123, 128, 121, 100, 82, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 15, 15, 15, 15, 15, 15, 15, 14, 14, 14, 14, 60, 53, 13, 13, 12, 12, 12, 12, 11, 11, 11, 10, 10, 9, 9, 9, 8, 8, 7, 7, 6, 6, 5, 5, 4, 4, 3, 2, 2, 1, 26 ],
[ 0, 1, 1, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 9, 10, 10, 11, 11, 11, 12, 12, 12, 12, 13, 13, 13, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 15, 15, 15, 15, 15, 15, 15, 14, 14, 14, 14, 13, 13, 13, 13, 12, 12, 12, 11, 11, 11, 10, 10, 10, 9, 9, 8, 8, 8, 7, 7, 6, 6, 5, 5, 4, 4, 3, 2, 2, 1, 1 ]
);
my $dist = 32.745;
my $mygraph = GD::Graph::area->new( 800, 300 );
$mygraph->set(
transparent => 0,
l_margin => 3,
b_margin => 3,
t_margin => 5,
r_margin => 5,
text_space => 3,
axis_space => 7,
tick_length => -5,
accent_treshold => 100,
x_label_position => 0.5,
bgclr => 'white',
fgclr => 'black',
boxclr => 'sky1',
labelclr => 'black',
axislabelclr => 'black',
textclr => 'black',
x_label => 'Kilometres',
y_label => 'Metres',
x_label_skip => 15.27,
x_last_label_skip => 1,
x_min_value => 0,
x_max_value => $dist,
borderclrs => [qw(fre2 fre1 black fre2 sky1 clutter terrain axes)],
dclrs => [qw(fre2 fre1 fre1 fre2 sky1 clutter terrain axes)]
) or warn $mygraph->error;
my $myimage = $mygraph->plot( \#data ) or die $mygraph->error;
print "Content-type: image/png\n\n";
print $myimage->png;
Background Information
I currently have a very successful web page which enables UK citizens to align a terrestrial TV aerial towards a given transmitter.
Here's an example of it in action
Once loading has completed, and the relevant buttons enabled, there three available visual components to help align the aerial and predict the likely usefulness of the signal:
A Google map for finding a landmark to point the aerial towards
A UK Ordnance Survey map showing the signal path
A vertical signal profile showing potential obstructions
It is the last of these, which is obtained by pressing the Signal Profile button at the bottom of the form and is displayed at the bottom of the page beneath the two maps, that is of concern here. I'm looking to replace it because it currently uses the Google Chart Tools API which unfortunately has been deprecated by Google and was supposed to end in 2015, but fortunately so far they've kept it running.
I've been looking ever since for a replacement come the day that Google throw the switch, but have had very limited success. If anyone knows of a package I could install on my own server to replace Google's chart drawing functionality, I'd be most grateful to know about it.
This feels like an off-by-one error in the code, but I don't have time to investigate further now. I'll just note that a work-around seems to be to replace:
x_min_value => 0,
with
x_min_value => 1,
I have Nepali date in employment table of kep database and i want to convert this date in to English date using postgresql.Guide me please.
Here is my table
id date
1 2071/1/4
2 2071/1/29
3 2069/4/24
SQLFiddle
1) Import list of nepali years and length of each month (I copied data from here). In first column there is nepali year and in other columns there is length of each month in days (second column in table is length of first month in every year).
-- drop table if exists tmpcal;
create table tmpcal (nyear int, a int, b int, c int, d int, e int, f int, g int, h int, i int, j int, k int, l int);
insert into tmpcal values
(2000 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2001 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2002 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2003 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2004 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2005 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2006 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2007 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2008 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 29, 31),
(2009 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2010 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2011 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2012 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2013 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2014 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2015 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2016 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2017 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2018 , 31, 32, 31, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2019 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2020 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2021 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2022 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2023 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2024 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2025 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2026 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2027 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2028 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2029 , 31, 31, 32, 31, 32, 30, 30, 29, 30, 29, 30, 30),
(2030 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2031 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2032 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2033 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2034 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2035 , 30, 32, 31, 32, 31, 31, 29, 30, 30, 29, 29, 31),
(2036 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2037 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2038 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2039 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2040 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2041 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2042 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2043 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2044 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2045 , 31, 32, 31, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2046 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2047 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2048 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2049 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2050 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2051 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2052 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2053 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2054 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2055 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2056 , 31, 31, 32, 31, 32, 30, 30, 29, 30, 29, 30, 30),
(2057 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2058 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2059 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2060 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2061 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2062 , 30, 32, 31, 32, 31, 31, 29, 30, 29, 30, 29, 31),
(2063 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2064 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2065 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2066 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 29, 31),
(2067 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2068 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2069 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2070 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2071 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2072 , 31, 32, 31, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2073 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2074 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2075 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2076 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2077 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2078 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2079 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2080 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2081 , 31, 31, 32, 32, 31, 30, 30, 30, 29, 30, 30, 30),
(2082 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 30, 30),
(2083 , 31, 31, 32, 31, 31, 30, 30, 30, 29, 30, 30, 30),
(2084 , 31, 31, 32, 31, 31, 30, 30, 30, 29, 30, 30, 30),
(2085 , 31, 32, 31, 32, 30, 31, 30, 30, 29, 30, 30, 30),
(2086 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 30, 30),
(2087 , 31, 31, 32, 31, 31, 31, 30, 30, 29, 30, 30, 30),
(2088 , 30, 31, 32, 32, 30, 31, 30, 30, 29, 30, 30, 30),
(2089 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 30, 30),
(2090 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 30, 30);
2) Assign date to every nepali date:
first convert columns into rows (using union all)
then we generate nepali days (using generate_series())
at the end we number rows (using row_number()) and add this number to 1943-04-14 date and substract 1 (used this converter to match nepali date 2000/01/01 to common date).
.
-- drop table if exists cal_conversion;
create table cal_conversion as (
with tmp as (
select nyear, 1::int as nmonth, a as nday from tmpcal union all
select nyear, 2, b from tmpcal union all
select nyear, 3, c from tmpcal union all
select nyear, 4, d from tmpcal union all
select nyear, 5, e from tmpcal union all
select nyear, 6, f from tmpcal union all
select nyear, 7, g from tmpcal union all
select nyear, 8, h from tmpcal union all
select nyear, 9, i from tmpcal union all
select nyear, 10, j from tmpcal union all
select nyear, 11, k from tmpcal union all
select nyear, 12, l from tmpcal
)
select
*,
nyear || '/' || nmonth || '/' || nday as ndate,
'1943-04-14'::date + row_number() over(order by nyear, nmonth, nday)::int - 1 as edate
from (
select
nyear,
nmonth,
generate_series(1, nday) as nday
from tmp) x
);
3) Finally, use our conversion table:
Sample data:
-- drop table if exists test_data;
create table test_data (
id int,
ndate varchar);
insert into test_data values
(1,'2071/1/4'),
(2,'2071/1/29'),
(3,'2069/4/24');
Usage (simple join):
select
ndate,
id,
edate
from
test_data
join cal_conversion using (ndate);
Result:
2069/4/24;3;2012-08-08
2071/1/29;2;2014-05-12
2071/1/4;1;2014-04-17