Finetuning LayoutLM on FUNSD-like dataset - index out of range in self - neural-network

I'm experimenting with huggingface transformers to finetune microsoft/layoutlmv2-base-uncased through AutoModelForTokenClassification on my custom dataset that is similar to FUNSD (pre-processed and normalized). After a few iterations of training I get this error :
Traceback (most recent call last):
File "layoutlmV2/train.py", line 137, in <module>
trainer.train()
File "..../lib/python3.8/site-packages/transformers/trainer.py", line 1409, in train
return inner_training_loop(
File "..../lib/python3.8/site-packages/transformers/trainer.py", line 1651, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "..../lib/python3.8/site-packages/transformers/trainer.py", line 2345, in training_step
loss = self.compute_loss(model, inputs)
File "..../lib/python3.8/site-packages/transformers/trainer.py", line 2377, in compute_loss
outputs = model(**inputs)
File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
return forward_call(*input, **kwargs)
File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 1228, in forward
outputs = self.layoutlmv2(
File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
return forward_call(*input, **kwargs)
File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 902, in forward
text_layout_emb = self._calc_text_embeddings(
File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 753, in _calc_text_embeddings
spatial_position_embeddings = self.embeddings._calc_spatial_position_embeddings(bbox)
File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 93, in _calc_spatial_position_embeddings
h_position_embeddings = self.h_position_embeddings(bbox[:, :, 3] - bbox[:, :, 1])
File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
return forward_call(*input, **kwargs)
File "..../lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
return F.embedding(
File "..../lib/python3.8/site-packages/torch/nn/functional.py", line 2203, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
After further inspection (vocab size, bboxes, dimensions, classes...) I noticed that there's negative values inside the input tensor causing the error. While input tensors of successful previous iterations have unsigned integers only. These negative numbers are returned by _calc_spatial_position_embeddings(self, bbox) in modeling_layoutlmv2.py
line 92 :
h_position_embeddings = self.h_position_embeddings(bbox[:, :, 3] - bbox[:, :, 1])
What may cause the returned input values to be negative?
What could I do to prevent this error from happening?
Example of the input tensor that triggers the error in torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) :
tensor([[ 0, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 10, 10, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12,
12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
8, 5, 5, 5, 5, 5, 5, -6, -6, -6, -6, -6, -6, 1, 1, 1, 1, 1,
5, 5, 5, 5, 5, 5, 7, 5, 7, 7, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0]])

After double checking the dataset and specifically the coordinates of the labels, I've found that some rows bbox coordinates lead to zero width or height. Here's a simplified example:
x1, y1, x2, y2 = dataset_row["bbox"]
print((x2-x1 < 1) or (y2-y1 < 1)) #output is sometimes True
After removing these labels from the dataset, the issue was resolved.

Related

Can rrule start at half past hour?

Is it possible to create a rrule that runs every 30 minutes but starts and ends at the middle of an hour? If rrule would accept non-integer parameters like byhour=range(16.5,19) that would be great, but unfortunately only integer is accepted.
In my case the times would be 16:30, 17:00, 17:30, 18:00 and 18:30 every weekday. All I can do is from 16:00 to 18:30 as follows:
from dateutil.rrule import *
from dateutil.parser import parse
list(rrule(MINUTELY, interval=30, count=20, byhour=range(16,19), byminute=(0,30), byweekday=(MO,TU,WE,TH,FR), dtstart=parse("20220602T070000")))
...with the following result:
[datetime.datetime(2022, 6, 2, 16, 0),
datetime.datetime(2022, 6, 2, 16, 30),
datetime.datetime(2022, 6, 2, 17, 0),
datetime.datetime(2022, 6, 2, 17, 30),
datetime.datetime(2022, 6, 2, 18, 0),
datetime.datetime(2022, 6, 2, 18, 30),
datetime.datetime(2022, 6, 3, 16, 0),
datetime.datetime(2022, 6, 3, 16, 30),
datetime.datetime(2022, 6, 3, 17, 0),
datetime.datetime(2022, 6, 3, 17, 30),
datetime.datetime(2022, 6, 3, 18, 0),
datetime.datetime(2022, 6, 3, 18, 30),
datetime.datetime(2022, 6, 6, 16, 0),
datetime.datetime(2022, 6, 6, 16, 30),
datetime.datetime(2022, 6, 6, 17, 0),
datetime.datetime(2022, 6, 6, 17, 30),
datetime.datetime(2022, 6, 6, 18, 0),
datetime.datetime(2022, 6, 6, 18, 30),
datetime.datetime(2022, 6, 7, 16, 0),
datetime.datetime(2022, 6, 7, 16, 30)]
I was hoping rrule has more flexibility than crontab...
I found a solution using rrset.exrule:
from dateutil.rrule import *
from dateutil.parser import parse
myrrule = rrule(MINUTELY, interval=30, count=20, byhour=range(16,19), byminute=(0,30), byweekday=(MO,TU,WE,TH,FR), dtstart=parse("20220602T070000"))
myexrule = rrule(DAILY, interval=1, count=4, byhour=16, byminute=0, byweekday=(MO,TU,WE,TH,FR), dtstart=parse("20220602T070000"))
rrset = rruleset()
rrset.rrule(myrrule)
rrset.exrule(myexrule)
list(rrset)
with the following result:
[datetime.datetime(2022, 6, 2, 16, 30),
datetime.datetime(2022, 6, 2, 17, 0),
datetime.datetime(2022, 6, 2, 17, 30),
datetime.datetime(2022, 6, 2, 18, 0),
datetime.datetime(2022, 6, 2, 18, 30),
datetime.datetime(2022, 6, 3, 16, 30),
datetime.datetime(2022, 6, 3, 17, 0),
datetime.datetime(2022, 6, 3, 17, 30),
datetime.datetime(2022, 6, 3, 18, 0),
datetime.datetime(2022, 6, 3, 18, 30),
datetime.datetime(2022, 6, 6, 16, 30),
datetime.datetime(2022, 6, 6, 17, 0),
datetime.datetime(2022, 6, 6, 17, 30),
datetime.datetime(2022, 6, 6, 18, 0),
datetime.datetime(2022, 6, 6, 18, 30),
datetime.datetime(2022, 6, 7, 16, 30)]

Filter Scala Matrix with condition

I have the given matrix in Scala:
val matrix = Array([30, 0, 13, 21, 25, 15],
[55, 47, 26, 54, 44, 3],
[21, 19, 23, 47, 29, 13],
[52, 50, 44, 14, 21, 24],
[10, 37, 0, 22, 17, 58],
[36, 55, 48, 27, 13, 35])
I need to filter the matrix (values from 2nd column > 40 and values fom 4rd column <45)
Can i do this somehow with the matrix.filter method?
You can try this way:
scala> :paste
// Entering paste mode (ctrl-D to finish)
val matrix = Array(Array(30, 0, 13, 21, 25, 15),
Array(55, 47, 26, 54, 44, 3),
Array(21, 19, 23, 47, 29, 13),
Array(52, 50, 44, 14, 21, 24),
Array(10, 37, 0, 22, 17, 58),
Array(36, 55, 48, 27, 13, 35))
// Exiting paste mode, now interpreting.
matrix: Array[Array[Int]] = Array(Array(30, 0, 13, 21, 25, 15), Array(55, 47, 26, 54, 44, 3), Array(21, 19, 23, 47, 29, 13), Array(52, 50, 44, 14, 21, 24), Array(10, 37, 0, 22, 17, 58), Array(36, 55, 48, 27, 13, 35))
scala> matrix.filter(x => x(1) > 40 && x(3) < 45)
res0: Array[Array[Int]] = Array(Array(52, 50, 44, 14, 21, 24), Array(36, 55, 48, 27, 13, 35))

Different CRC32 values in Java and swift

I'm in the process of writing a utility in Swift which can be used to calculate CRC32 checksum of an input data . A similar utility exists in Java which we are using extensively and has been working well for us.
The Java Utility uses java.util.zip.CRC32 to calculate the checksum. Pseudo code is as follows :
Java code:
private void transferFileData(short index, byte[] data, long dataSize) {
CRC32 crc32 = new CRC32();
long crc = crc32.update(data, (int) dataSize);
System.out.println("CRC32 : " + crc);
}
The Swift uses the CRC32 (import CryptoSwift) CryptoSwift code for generating the checksum in swift is as follows :
Swift code:
func crc32Func(_ item:[Int8] ) {
let data = Data(bytes: item, count: item.count)
let byte = data.bytes
let crc32 = byte.crc32()
print("checksum == \(crc32)"
}
The output from the Java code is :
Checksum in Java : 3771181957
The output from the swift code is :
Checksum in swift : 1894162356
Why the checksum values are not the same?
This is the code applied by Swift as below:
The data type is [Int8]
The data size is 450 bytes.
let item:
[Int8] = [45, 35, 76, 70, 67, 68, 95, 70, 79, 84, 65, 95, 70, 87, 95,
70, 85, 76, 76, 10, 80, 75, 71, 95, 86 , 69, 82, 83, 73, 79, 78, 58,
51, 46, 48, 46, 48, 10, 66, 65, 83, 69, 95, 86, 69, 82, 83, 73, 79, 78
, 58, 10, 72, 65, 83, 72, 58, 51, 97, 57, 98, 51, 99, 52, 56, 57, 52,
56, 56, 53, 49, 50, 53, 52, 55 , 48, 51, 49, 54, 48, 57, 56, 48, 101,
98, 101, 51, 54, 54, 10, 80, 75, 71, 95, 83, 73, 90, 69, 58, 49 , 51,
49, 48, 55, 50, 48, 10, 26, 0, 0, 5, 32, -79, 11, 5, 8, -39, -5, 4, 8,
-35, -5, 4, 8, -15, -5, 4, 8, -13, -5, 4, 8, -11, -5, 4, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -31, -15, 10, 8, -9, -5, 4, 8,
0, 0, 0, 0, -127, -14, 10, 8, -27, -14, 10, 8, 1, 12, 5, 8, 1, 12, 5,
8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1,
12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12,
5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8,
-7, -5, 4, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 9, -4,
4, 8, 25, -4, 4, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5,
8, 1, 12, 5, 8, 57, -4, 4, 8, 1, 12, 5, 8, 73, -4, 4, 8, 89, -4, 4, 8,
105, -4, 4, 8, 1, 12, 5, 8, -39, -4, 4, 8, 1, 12, 5, 8, 1, 12, 5, 8,
1, 12, 5, 8, 41, -4, 4, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1,
12, 5 , 8, 1, 12, 5, 8, 1, 12, 5, 8, 121, -4, 4, 8, -119, -4, 4, 8, 1,
12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12,
5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8,
1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, 1, 12, 5, 8, -103, -4]
is input.
Java input:
byte[] data
The type is long and the data entered is the same as [Int8] of swift data.
The CRC you are getting from your Swift code (whatever that actual code is) is correct. The CRC you are getting from your Java code (whatever that actual code is) is not correct.
There is no way to know what you're doing wrong in your Java code without at least being able to see that code.

Perl - GD::Graph - Gap between plot and y-axis

I am using Perl's GD::Graph
module to plot a signal profile. I've managed to get almost everything to look just about right, except that for some reason there is an irritating and unwanted gap between the y-axis and the first plotted point. How can I eliminate this?
Here is the code for the test page I've written (it display on my localhost server, but I tried installing it on my live server so that others here could see the result, it errors - I think there must be some Perl modules missing from my live site):
#!perl
use POSIX;
use CGI ':standard';
use GD::Graph::area;
use GD::Graph::colour qw(:colours);
use strict;
# Graph constants
GD::Graph::colour::add_colour( axes => [ 0x00, 0x00, 0x00 ] );
GD::Graph::colour::add_colour( earth => [ 0x00, 0x00, 0x00 ] );
GD::Graph::colour::add_colour( beam => [ 0x00, 0x00, 0x00 ] );
GD::Graph::colour::add_colour( clutter => [ 0x66, 0x33, 0x00 ] );
GD::Graph::colour::add_colour( terrain => [ 0x00, 0x99, 0x00 ] );
GD::Graph::colour::add_colour( earth => [ 0x00, 0x00, 0x00 ] );
GD::Graph::colour::add_colour( fre1 => [ 0xFF, 0x33, 0x00 ] );
GD::Graph::colour::add_colour( fre2 => [ 0xFF, 0x66, 0x00 ] );
GD::Graph::colour::add_colour( sky1 => [ 0x33, 0x66, 0x99 ] );
GD::Graph::colour::add_colour( sky2 => [ 0x66, 0x99, 0xCC ] );
# All arrays should same number of entries.
my #data = (
[ 0, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 5, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 10, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 15, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 20, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 25, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, undef, 30, undef, undef, undef, undef, undef, undef, undef, undef, undef ],
[ 450, 458, 459, 458, 457, 455, 453, 451, 449, 447, 444, 441, 438, 435, 432, 429, 426, 422, 420, 416, 413, 409, 406, 402, 399, 395, 391, 388, 384, 381, 376, 373, 369, 365, 361, 358, 353, 350, 345, 341, 337, 333, 329, 325, 321, 316, 313, 308, 304, 299, 296, 291, 286, 282, 277, 273, 268, 265, 259, 256, 250, 247, 241, 236, 232, 227, 223, 217, 213, 208, 204, 198, 194, 188, 184, 178, 173, 168, 163, 158, 152, 148, 142, 137, 131, 126, 120, 115, 109, 102, 97, 90, 85, 78, 72, 64, 58, 50, 43, 32 ],
[ 450, 453, 451, 449, 447, 444, 441, 438, 435, 432, 428, 426, 422, 418, 415, 411, 408, 404, 400, 396, 393, 389, 386, 381, 378, 374, 369, 366, 361, 358, 353, 350, 345, 342, 337, 334, 329, 326, 321, 316, 313, 308, 305, 300, 296, 291, 288, 283, 279, 274, 271, 266, 261, 257, 252, 249, 244, 240, 235, 231, 226, 222, 217, 212, 208, 203, 199, 194, 190, 185, 181, 176, 172, 166, 162, 157, 151, 147, 142, 138, 132, 128, 123, 118, 113, 108, 103, 98, 92, 87, 82, 76, 71, 65, 60, 54, 49, 42, 36, 27 ],
[ 450, 445, 440, 436, 432, 428, 423, 419, 414, 411, 406, 402, 397, 392, 389, 384, 380, 375, 372, 367, 363, 358, 354, 350, 346, 341, 336, 332, 328, 324, 319, 315, 310, 307, 302, 298, 293, 290, 285, 280, 276, 271, 268, 263, 259, 254, 250, 246, 242, 237, 233, 228, 224, 220, 215, 211, 206, 203, 198, 194, 189, 186, 181, 176, 172, 167, 164, 159, 155, 150, 147, 142, 138, 133, 129, 125, 120, 116, 111, 107, 102, 99, 94, 90, 85, 82, 77, 73, 68, 63, 60, 55, 51, 46, 43, 38, 34, 29, 25, 21 ],
[ 450, 437, 429, 423, 416, 411, 405, 400, 394, 389, 383, 379, 373, 367, 363, 357, 353, 347, 343, 337, 333, 328, 323, 318, 314, 309, 303, 299, 294, 290, 285, 281, 275, 272, 266, 262, 257, 253, 248, 243, 239, 234, 231, 226, 222, 217, 213, 208, 205, 200, 196, 191, 186, 183, 178, 174, 169, 166, 161, 157, 153, 149, 144, 140, 136, 132, 128, 124, 120, 116, 112, 108, 104, 100, 97, 92, 88, 84, 80, 77, 73, 69, 65, 62, 58, 55, 51, 48, 44, 40, 37, 34, 31, 27, 25, 22, 19, 17, 15, 14 ],
[ 450, 432, 422, 415, 406, 400, 393, 387, 380, 375, 368, 363, 357, 350, 345, 339, 334, 328, 324, 318, 313, 307, 303, 297, 293, 287, 281, 277, 271, 267, 262, 258, 252, 248, 243, 239, 233, 229, 224, 219, 215, 210, 206, 201, 197, 192, 188, 183, 180, 175, 171, 166, 161, 158, 153, 149, 145, 141, 136, 133, 128, 125, 120, 116, 112, 108, 105, 100, 97, 93, 89, 85, 82, 78, 75, 71, 67, 64, 60, 57, 53, 50, 46, 43, 40, 37, 34, 31, 28, 25, 22, 19, 17, 15, 13, 11, 10, 8, 8, 9 ],
[ 387, 344, 266, 240, 222, 215, 190, 156, 120, 76, 99, 151, 167, 158, 167, 158, 157, 148, 114, 89, 87, 28, 11, 11, 12, 12, 12, 13, 13, 13, 13, 23, 15, 34, 39, 38, 72, 107, 122, 134, 128, 128, 121, 100, 82, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 15, 20, 20, 15, 15, 15, 15, 14, 14, 14, 14, 60, 53, 13, 13, 12, 12, 12, 12, 11, 11, 11, 10, 10, 9, 9, 9, 8, 8, 7, 7, 6, 6, 5, 5, 4, 4, 3, 2, 2, 1, 26 ],
[ 386, 343, 261, 235, 217, 210, 185, 151, 105, 61, 84, 136, 152, 143, 152, 153, 152, 143, 109, 84, 82, 23, 11, 11, 12, 12, 12, 13, 13, 13, 13, 23, 15, 34, 34, 33, 67, 102, 117, 129, 123, 128, 121, 100, 82, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 15, 15, 15, 15, 15, 15, 15, 14, 14, 14, 14, 60, 53, 13, 13, 12, 12, 12, 12, 11, 11, 11, 10, 10, 9, 9, 9, 8, 8, 7, 7, 6, 6, 5, 5, 4, 4, 3, 2, 2, 1, 26 ],
[ 0, 1, 1, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 9, 10, 10, 11, 11, 11, 12, 12, 12, 12, 13, 13, 13, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 15, 15, 15, 15, 15, 15, 15, 14, 14, 14, 14, 13, 13, 13, 13, 12, 12, 12, 11, 11, 11, 10, 10, 10, 9, 9, 8, 8, 8, 7, 7, 6, 6, 5, 5, 4, 4, 3, 2, 2, 1, 1 ]
);
my $dist = 32.745;
my $mygraph = GD::Graph::area->new( 800, 300 );
$mygraph->set(
transparent => 0,
l_margin => 3,
b_margin => 3,
t_margin => 5,
r_margin => 5,
text_space => 3,
axis_space => 7,
tick_length => -5,
accent_treshold => 100,
x_label_position => 0.5,
bgclr => 'white',
fgclr => 'black',
boxclr => 'sky1',
labelclr => 'black',
axislabelclr => 'black',
textclr => 'black',
x_label => 'Kilometres',
y_label => 'Metres',
x_label_skip => 15.27,
x_last_label_skip => 1,
x_min_value => 0,
x_max_value => $dist,
borderclrs => [qw(fre2 fre1 black fre2 sky1 clutter terrain axes)],
dclrs => [qw(fre2 fre1 fre1 fre2 sky1 clutter terrain axes)]
) or warn $mygraph->error;
my $myimage = $mygraph->plot( \#data ) or die $mygraph->error;
print "Content-type: image/png\n\n";
print $myimage->png;
Background Information
I currently have a very successful web page which enables UK citizens to align a terrestrial TV aerial towards a given transmitter.
Here's an example of it in action
Once loading has completed, and the relevant buttons enabled, there three available visual components to help align the aerial and predict the likely usefulness of the signal:
A Google map for finding a landmark to point the aerial towards
A UK Ordnance Survey map showing the signal path
A vertical signal profile showing potential obstructions
It is the last of these, which is obtained by pressing the Signal Profile button at the bottom of the form and is displayed at the bottom of the page beneath the two maps, that is of concern here. I'm looking to replace it because it currently uses the Google Chart Tools API which unfortunately has been deprecated by Google and was supposed to end in 2015, but fortunately so far they've kept it running.
I've been looking ever since for a replacement come the day that Google throw the switch, but have had very limited success. If anyone knows of a package I could install on my own server to replace Google's chart drawing functionality, I'd be most grateful to know about it.
This feels like an off-by-one error in the code, but I don't have time to investigate further now. I'll just note that a work-around seems to be to replace:
x_min_value => 0,
with
x_min_value => 1,

Convert Nepali date to english date using postgresql

I have Nepali date in employment table of kep database and i want to convert this date in to English date using postgresql.Guide me please.
Here is my table
id date
1 2071/1/4
2 2071/1/29
3 2069/4/24
SQLFiddle
1) Import list of nepali years and length of each month (I copied data from here). In first column there is nepali year and in other columns there is length of each month in days (second column in table is length of first month in every year).
-- drop table if exists tmpcal;
create table tmpcal (nyear int, a int, b int, c int, d int, e int, f int, g int, h int, i int, j int, k int, l int);
insert into tmpcal values
(2000 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2001 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2002 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2003 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2004 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2005 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2006 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2007 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2008 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 29, 31),
(2009 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2010 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2011 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2012 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2013 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2014 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2015 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2016 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2017 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2018 , 31, 32, 31, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2019 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2020 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2021 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2022 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2023 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2024 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2025 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2026 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2027 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2028 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2029 , 31, 31, 32, 31, 32, 30, 30, 29, 30, 29, 30, 30),
(2030 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2031 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2032 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2033 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2034 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2035 , 30, 32, 31, 32, 31, 31, 29, 30, 30, 29, 29, 31),
(2036 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2037 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2038 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2039 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2040 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2041 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2042 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2043 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2044 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2045 , 31, 32, 31, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2046 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2047 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2048 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2049 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2050 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2051 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2052 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2053 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2054 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2055 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2056 , 31, 31, 32, 31, 32, 30, 30, 29, 30, 29, 30, 30),
(2057 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2058 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2059 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2060 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2061 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2062 , 30, 32, 31, 32, 31, 31, 29, 30, 29, 30, 29, 31),
(2063 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2064 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2065 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2066 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 29, 31),
(2067 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2068 , 31, 31, 32, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2069 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2070 , 31, 31, 31, 32, 31, 31, 29, 30, 30, 29, 30, 30),
(2071 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2072 , 31, 32, 31, 32, 31, 30, 30, 29, 30, 29, 30, 30),
(2073 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 31),
(2074 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2075 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2076 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2077 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31),
(2078 , 31, 31, 31, 32, 31, 31, 30, 29, 30, 29, 30, 30),
(2079 , 31, 31, 32, 31, 31, 31, 30, 29, 30, 29, 30, 30),
(2080 , 31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30),
(2081 , 31, 31, 32, 32, 31, 30, 30, 30, 29, 30, 30, 30),
(2082 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 30, 30),
(2083 , 31, 31, 32, 31, 31, 30, 30, 30, 29, 30, 30, 30),
(2084 , 31, 31, 32, 31, 31, 30, 30, 30, 29, 30, 30, 30),
(2085 , 31, 32, 31, 32, 30, 31, 30, 30, 29, 30, 30, 30),
(2086 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 30, 30),
(2087 , 31, 31, 32, 31, 31, 31, 30, 30, 29, 30, 30, 30),
(2088 , 30, 31, 32, 32, 30, 31, 30, 30, 29, 30, 30, 30),
(2089 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 30, 30),
(2090 , 30, 32, 31, 32, 31, 30, 30, 30, 29, 30, 30, 30);
2) Assign date to every nepali date:
first convert columns into rows (using union all)
then we generate nepali days (using generate_series())
at the end we number rows (using row_number()) and add this number to 1943-04-14 date and substract 1 (used this converter to match nepali date 2000/01/01 to common date).
.
-- drop table if exists cal_conversion;
create table cal_conversion as (
with tmp as (
select nyear, 1::int as nmonth, a as nday from tmpcal union all
select nyear, 2, b from tmpcal union all
select nyear, 3, c from tmpcal union all
select nyear, 4, d from tmpcal union all
select nyear, 5, e from tmpcal union all
select nyear, 6, f from tmpcal union all
select nyear, 7, g from tmpcal union all
select nyear, 8, h from tmpcal union all
select nyear, 9, i from tmpcal union all
select nyear, 10, j from tmpcal union all
select nyear, 11, k from tmpcal union all
select nyear, 12, l from tmpcal
)
select
*,
nyear || '/' || nmonth || '/' || nday as ndate,
'1943-04-14'::date + row_number() over(order by nyear, nmonth, nday)::int - 1 as edate
from (
select
nyear,
nmonth,
generate_series(1, nday) as nday
from tmp) x
);
3) Finally, use our conversion table:
Sample data:
-- drop table if exists test_data;
create table test_data (
id int,
ndate varchar);
insert into test_data values
(1,'2071/1/4'),
(2,'2071/1/29'),
(3,'2069/4/24');
Usage (simple join):
select
ndate,
id,
edate
from
test_data
join cal_conversion using (ndate);
Result:
2069/4/24;3;2012-08-08
2071/1/29;2;2014-05-12
2071/1/4;1;2014-04-17