I am using mongodb for our Application .
I ran the db.setProfilingLevel(1,25) on the mongo shell to identify the slow queries from system.profile collection .
I observed that the read operations are fast , but the update operations are very slow
This is a sample of my mongostat
insert/s query/s update/s delete/s getmore/s command/s flushes/s mapped vsize res faults/s locked % idx miss % q t|r|w conn time
0 950 469 0 0 471 0 10396 12295 3207 27 34.9 0 0|0|0 152 07:18:49
0 838 418 0 0 422 0 10396 12295 3209 21 34.6 0 0|0|0 152 07:18:50
0 1005 502 0 0 504 0 10396 12295 3211 21 35.5 0 0|0|0 152 07:18:51
0 837 410 0 0 418 0 10396 12295 3212 20 35.7 0 0|0|0 152 07:18:52
0 754 377 0 0 379 0 10396 12295 3214 19 36.7 0 0|0|0 152 07:18:53
0 841 420 0 0 422 0 10396 12295 3216 24 35.9 0 0|0|0 152 07:18:54
0 877 438 0 0 442 0 10396 12295 3217 23 37.2 0 0|0|0 152 07:18:55
0 799 393 0 0 395 0 10396 12295 3219 21 37 0 0|0|0 152 07:18:56
0 947 471 0 0 479 0 10396 12295 3221 26 39 0 0|0|0 152 07:18:57
0 855 427 0 0 429 0 10396 12295 3222 24 38.4 0 0|0|0 152 07:18:58
0 1007 504 0 0 506 0 10396 12295 3224 31 36 0 0|0|0 152 07:18:59
0 841 413 0 0 417 0 10396 12295 3226 23 37.2 0 0|0|0 152 07:19:00
The stats are from dev environment , cant assume really from prod environment .
As per the architecture i cannot reduce the index size on that collection , but i saw that usePowerOf2Sizes can help me in this case in improving the write/update response time in mongodb .
I have heard lot of usePowerOf2Sizes , which says that
As usePowerOf2Sizes can reduce fragmentation .
all data will be set in mmemory and performance will be great .
With this option MongoDB will be able to more effectively reuse space.
usePowerOf2Sizes is useful for collections where you will be inserting and deleting large numbers of documents to ensure that MongoDB will effectively use space on disk.
I want to know if there will be any negative impact in using the option usePowerOf2Sizes ?? I have got 17 collections in my mongodb and want to use usePowerOf2Sizes for only one collection .
Please let me know , thanks in advance .
Related
I have a bunch of time-dependent data, non-decreasing and right-continuous, always with come category involved, that can be repeated. I am looking for a sort of a variable width-bar chart, or cascade chart, flipped on it's side, from the right. For example,
set term postscript eps enhanced
set output "so.eps"
$Data <<EOD
# date source reg sic pic total
2000-07-25 2000glider C-FJSN 8 0 216
2000-07-28 2000glider C-FJSN 10 0 226
2000-07-28 2000glider C-FJSN 11 0 237
2000-07-28 2000glider C-GCLB 4 0 241
2000-07-29 2000glider C-GCLY 3 0 244
2000-07-29 2000glider C-GCLY 2 0 246
2000-07-29 2000glider C-GCLY 17 0 263
2000-07-30 2000glider C-GCLB 15 0 278
2000-07-30 2000glider C-GCLB 0 13 291
2000-07-30 2000glider C-GCLK 11 0 302
2000-07-30 2000glider C-FJSN 0 16 318
2000-07-30 2000glider C-GCLB 0 10 328
2000-08-02 2000glider C-GQRT 0 13 341
2000-08-04 2000glider C-GCLY 0 11 352
2000-08-05 2000glider C-GCLB 12 0 364
2000-08-05 2000glider C-GCLB 0 12 376
2000-08-06 2000glider C-GCLB 0 11 387
2000-08-07 2000glider C-GFMB 0 12 399
2000-08-07 2000glider C-GCMB 0 11 410
2000-08-08 2000glider C-GCLK 0 12 422
2000-08-09 2000glider C-GCLB 14 0 436
2000-08-09 2000glider C-GCLB 0 9 445
2000-08-10 2000glider C-GCLL 0 10 455
EOD
set monochrome
set xdata time
set timefmt "%Y-%m-%d"
set xtics format "%Y-%m-%d" rotate by -30
set ylabel "hours"
set grid
unset key
unset border
plot $Data using 1:($6/60) with fillsteps lw 2, \
$Data using 1:($6/60):3 with labels
Which gives,
I'm trying to create like a cumulative distribution function, where hours, in this case, are added. I want the line to go to zero on the left and the maximum value on the right. The labels, instead of printing, should be in the key and mapped to the data, grouped with other consecutive data, from the right. Is there such a plot, and how to I create it?
I have full control over the input, so I was thinking maybe an inverse-transform and rotate would be easier?
Not sure if I fully understood all your points correctly.
Maybe the following script might bring you closer to what you want.
Update:
If you want color bars for intervals then you could use the plotting style with boxxyerror (check help boxxy).
with boxxy requires either 4 or 6 columns. 4 columns would be (x):(y):(dx):(dy) for a box centered at x,y with +/-dx and +/-dy.
sum up your durations which are either in column 4 or 5 in the variable total. No need for column 6.
the difference dx=43200 (i.e. half a day in seconds)
There are different ways of plottings steps.
Script:
### cumulative plot with color code
reset session
$Data <<EOD
# date source reg sic pic total
2000-07-25 2000glider C-FJSN 8 0 216
2000-07-28 2000glider C-FJSN 10 0 226
2000-07-28 2000glider C-FJSN 11 0 237
2000-07-28 2000glider C-GCLB 4 0 241
2000-07-29 2000glider C-GCLY 3 0 244
2000-07-29 2000glider C-GCLY 2 0 246
2000-07-29 2000glider C-GCLY 17 0 263
2000-07-30 2000glider C-GCLB 15 0 278
2000-07-30 2000glider C-GCLB 0 13 291
2000-07-30 2000glider C-GCLK 11 0 302
2000-07-30 2000glider C-FJSN 0 16 318
2000-07-30 2000glider C-GCLB 0 10 328
2000-08-02 2000glider C-GQRT 0 13 341
2000-08-04 2000glider C-GCLY 0 11 352
2000-08-05 2000glider C-GCLB 12 0 364
2000-08-05 2000glider C-GCLB 0 12 376
2000-08-06 2000glider C-GCLB 0 11 387
2000-08-07 2000glider C-GFMB 0 12 399
2000-08-07 2000glider C-GCMB 0 11 410
2000-08-08 2000glider C-GCLK 0 12 422
2000-08-09 2000glider C-GCLB 14 0 436
2000-08-09 2000glider C-GCLB 0 9 445
2000-08-10 2000glider C-GCLL 0 10 455
EOD
# get a unique list from datablock
addToList(list,col) = list.( strstrt(list,'"'.strcol(col).'"') > 0 ? \
'' : ' "'.strcol(col).'"')
Uniqs = ''
stats $Data u (Uniqs=addToList(Uniqs,3)) nooutput
Uniq(i) = word(Uniqs,i)
getIndex(s) = sum [_i=1:words(Uniqs)] s eq word(Uniqs,_i) ? _i : 0
myTimeFmt = "%Y-%m-%d"
set format x myTimeFmt timedate
set xtics format myTimeFmt rotate by -30
set ylabel "hours"
set format y "%tH:%tM" timedate
set grid
set key out reverse Left noautotitle
set style fill solid 0.5
plot total=0 $Data u (timecolumn(1,myTimeFmt)):(dy=($4+$5)*60,total=total+dy) w steps lc "black" dt 3, \
total=0 '' u (timecolumn(1,myTimeFmt)):(dy=($4+$5)*60,total=total+dy,total-dy/2.): \
(43200):(dy/2.):(getIndex(strcol(3))) w boxxy lc var, \
for [i=1:words(Uniqs)] keyentry w boxxy lc i ti Uniq(i)
### end of script
Result:
We are testing our application for performance, which is using Postgresql 13 as a database. It is very insert and update heavy and we cannot get more than 65k TPS on the database. But none of the most common bottlenecks do apply (CPU/network/disk IO).
We have also run multiple combinations with pgbench but also those cannot go past 65k.
E.g.
pgbench -i -s 50 -U postgres -h <DB_HOST> -d <DB_NAME>
pgbench -c 64 -j 32 -t 100000 -h <DB_HOST> -U postgres <DB_NAME>
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 64
number of threads: 32
number of transactions per client: 100000
number of transactions actually processed: 6400000/6400000
latency average = 0.976 ms
tps = 65584.664360 (including connections establishing)
tps = 65594.330678 (excluding connections establishing)
(Using more clients resulted in fewer TPS)
The database server has the following specs:
64 Cores (Intel Xeon Gold 6130 (Skylake, 2.10GHz, 2 CPUs/node, 16 cores/CPU))
192 GiB RAM (12 * 16GiB DIMM DDR4 Synchronous Registered (Buffered) 2666 MHz (0.4 ns))
2 * SSD SATA Samsung MZ7KM240HMHQ0D3 (one is used for the WAL and the other for the data)
10 Gbps network link
OS: Debian 11
And we are using the following configuration:
shared_buffers=65551953kB
effective_cache_size=147491895kB
huge_pages=on
min_wal_size=4GB
max_wal_size=16GB
wal_buffers=1GB
work_mem=2GB
maintenance_work_mem=4GB
checkpoint_completion_target=0.9
checkpoint_timeout = 15min
random_page_cost=1.1
bgwriter_flush_after = 2MB
effective_io_concurrency = 200
# Disabled just for performance experiments
fsync = off
synchronous_commit = off
full_page_writes = on
max_worker_processes=64
max_parallel_workers=64
max_parallel_workers_per_gather=10
max_parallel_maintenance_workers=12
When we reach the 65k TPS we have the following resource usage on the server:
Our application:
CPU: 25%
I/O Utilization: 30% resp. 60% (WAL)
Network: in 200Mbps out 125Mbps
pgbench:
CPU: 75%
I/O Utilization: 2% resp. 5% (WAL)
Network: in/out ~500Mbps
We have also temporarily placed the WAL and data into RAM to verify that the disks are not the issue, but this had no impact either.
Does anyone have a guess where our bottleneck might be?
(We have thought about sharding already but would like to use the single server as much as possible)
Edit:
vmstat 1 -S M while running pgbench:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 92009 25 18316 0 0 3 318 405 22 26 11 63 0 0
0 0 0 92009 25 18316 0 0 0 0 261 320 0 0 100 0 0
0 0 0 92009 25 18316 0 0 0 46616 216 257 0 0 100 0 0
45 0 0 92260 25 17996 0 0 0 0 509124 781029 35 13 52 0 0
44 1 0 92252 25 17998 0 0 0 82192 743920 1140285 51 20 30 0 0
50 0 0 92247 25 18003 0 0 0 65564 740366 1152212 52 18 30 0 0
41 0 0 92243 25 18005 0 0 0 0 736052 1138409 50 20 30 0 0
35 0 0 92240 25 18009 0 0 0 0 734265 1138483 50 20 30 0 0
36 0 0 92238 25 18012 0 0 0 264 741315 1152409 52 19 29 0 0
43 0 0 92236 25 18016 0 0 0 36 740374 1147759 51 20 29 0 0
46 0 0 92231 25 18020 0 0 0 163856 737491 1166498 53 19 28 0 0
47 0 0 92228 25 18023 0 0 0 0 741582 1171814 52 19 28 0 0
43 0 0 92223 25 18027 0 0 0 61584 741537 1168119 53 19 28 0 0
46 0 0 92220 25 18030 0 0 0 36 739695 1167456 53 19 28 0 0
43 0 0 92216 25 18034 0 0 0 228 741992 1150333 52 20 29 0 0
38 0 0 92214 25 18036 0 0 0 147464 740589 1166289 52 19 28 0 0
41 0 0 92209 25 18042 0 0 0 0 737148 1162946 52 19 28 0 0
44 0 0 92207 25 18044 0 0 0 2480 741757 1173128 53 19 28 0 0
39 0 0 92205 25 18049 0 0 0 48 740404 1170644 53 19 28 0 0
52 0 0 92201 25 18051 0 0 0 292 739032 1159037 52 19 28 0 0
42 1 0 92198 25 18054 0 0 0 20072 740101 1165594 52 20 28 0 0
51 0 0 92194 25 18059 0 0 0 45464 738055 1165382 53 19 28 0 0
41 0 0 92190 25 18062 0 0 0 0 742838 1172377 53 19 28 0 0
45 0 0 92185 25 18067 0 0 0 36 740704 1174534 53 19 28 0 0
50 0 0 92182 25 18069 0 0 0 92 741691 1150716 52 19 28 0 0
42 0 0 92177 25 18073 0 0 0 28 740220 1168488 53 18 28 0 0
44 0 0 92174 25 18077 0 0 0 0 738818 1164769 53 19 28 0 0
46 0 0 92172 25 18080 0 0 0 0 740720 1169902 53 19 28 0 0
46 0 0 92166 25 18083 0 0 0 90404 592524 945810 42 15 43 0 0
45 0 0 92161 25 18087 0 0 0 3884 746310 1159898 52 19 28 0 0
49 0 0 92157 25 18090 0 0 0 20 747242 1177750 53 19 28 0 0
36 0 0 92152 25 18094 0 0 0 0 744477 1173832 53 19 28 0 0
46 0 0 92149 25 18098 0 0 0 0 746194 1172700 53 19 28 0 0
39 0 0 92147 26 18101 0 0 0 49768 745651 1177462 54 18 28 0 0
43 0 0 92143 26 18105 0 0 0 212 744968 1161110 53 19 28 0 0
43 0 0 92138 26 18109 0 0 0 0 743223 1176960 54 19 28 0 0
43 0 0 92135 26 18112 0 0 0 81920 745168 1173574 53 19 28 0 0
48 0 0 92132 26 18116 0 0 0 0 743174 1169255 53 19 28 0 0
48 0 0 92129 26 18120 0 0 0 68 592295 933445 42 15 43 0 0
41 0 0 92124 26 18123 0 0 0 76 740354 1162221 52 19 28 0 0
49 0 0 92120 26 18125 0 0 0 0 738456 1158291 53 19 28 0 0
39 0 0 92117 26 18129 0 0 0 147536 740735 1162479 52 20 28 0 0
49 0 0 92113 26 18133 0 0 0 0 737209 1165532 53 20 28 0 0
49 0 0 92111 26 18137 0 0 0 40 741185 1168133 54 19 28 0 0
45 0 0 92110 26 18140 0 0 0 4000 740693 1141945 52 20 28 0 0
42 0 0 92105 26 18144 0 0 0 0 741857 1168830 53 19 28 0 0
43 0 0 92102 26 18147 0 0 0 8 742546 1168867 54 18 28 0 0
43 0 0 92101 26 18150 0 0 0 147456 741941 1166646 53 19 28 0 0
41 1 0 92097 26 18154 0 0 0 64192 740052 1169040 53 19 28 0 0
48 0 0 92094 26 18158 0 0 0 27484 737224 1139511 52 20 28 0 0
47 0 0 92087 26 18162 0 0 0 0 740821 1165037 53 19 28 0 0
54 0 0 92059 26 18164 0 0 0 4 737109 1155098 53 19 27 0 0
38 0 0 92051 26 18170 0 0 0 147456 701847 1075069 55 20 25 0 0
35 0 0 92064 26 18174 0 0 0 44 723153 1125736 54 19 27 0 0
48 0 0 92056 26 18179 0 0 0 53008 734590 1134838 52 19 29 0 0
46 0 0 92053 26 18183 0 0 0 0 741595 1166891 53 19 28 0 0
46 0 0 92049 26 18186 0 0 0 0 740196 1170838 54 19 27 0 0
31 1 0 92045 26 18191 0 0 0 98304 741800 1170076 54 18 28 0 0
44 0 0 92043 26 18194 0 0 0 49188 733352 1173652 53 20 27 0 0
43 0 0 92028 26 18198 0 0 0 116 733522 1151497 53 20 27 0 0
44 0 0 92037 26 18201 0 0 0 0 730364 1137665 53 19 27 0 0
37 0 0 92035 26 18204 0 0 0 0 742348 1164945 53 19 28 0 0
44 0 0 92031 26 18208 0 0 0 0 739273 1165044 53 19 28 0 0
52 0 0 92028 26 18211 0 0 0 147888 739274 1164496 53 19 28 0 0
50 0 0 92024 26 18215 0 0 0 144 739684 1145210 53 19 28 0 0
50 0 0 92020 26 18219 0 0 0 0 742847 1167779 54 18 28 0 0
38 0 0 92016 26 18223 0 0 0 0 738079 1166580 53 19 28 0 0
36 0 0 92013 26 18226 0 0 0 0 742687 1171101 54 18 27 0 0
48 0 0 92009 26 18229 0 0 0 147500 741536 1166846 53 19 28 0 0
40 0 0 92006 26 18233 0 0 0 94600 740746 1147102 52 20 28 0 0
45 0 0 92001 26 18238 0 0 0 0 741119 1163851 53 19 28 0 0
48 0 0 91999 26 18241 0 0 0 0 740995 1167197 53 19 28 0 0
35 0 0 91996 26 18244 0 0 0 0 742235 1165666 53 19 28 0 0
44 1 0 91993 26 18248 0 0 0 49192 741392 1164506 53 19 28 0 0
43 1 0 91990 26 18251 0 0 0 124876 743695 1144639 52 19 29 0 0
48 0 0 91987 26 18255 0 0 0 24864 737759 1159383 52 20 28 0 0
44 0 0 91983 26 18258 0 0 0 0 740224 1164983 53 19 28 0 0
43 0 0 91980 26 18262 0 0 0 0 741742 1168140 54 19 27 0 0
18 0 0 91976 26 18267 0 0 0 36 737449 1162293 53 19 28 0 0
49 0 0 91973 26 18269 0 0 0 147576 741462 1148048 52 20 28 0 0
43 0 0 91969 26 18274 0 0 0 0 742408 1168332 54 19 27 0 0
43 0 0 91966 26 18277 0 0 0 0 738803 1164992 53 19 28 0 0
39 0 0 91963 26 18280 0 0 0 4 737891 1159372 52 19 28 0 0
43 0 0 91962 26 18283 0 0 0 40 741888 1166835 53 19 28 0 0
48 0 0 91958 26 18287 0 0 0 164144 738677 1145900 52 20 28 0 0
46 0 0 91955 26 18291 0 0 0 0 740956 1165789 53 19 28 0 0
44 0 0 91952 26 18295 0 0 0 0 741055 1166460 53 19 28 0 0
44 0 0 91948 26 18299 0 0 0 8 739414 1165698 53 19 28 0 0
46 0 0 91945 26 18301 0 0 0 48 743218 1165277 53 19 28 0 0
36 0 0 91941 26 18305 0 0 0 208 736320 1134425 51 20 29 0 0
47 0 0 91936 26 18309 0 0 0 239096 739799 1159730 52 19 28 0 0
45 0 0 91932 26 18312 0 0 0 0 742477 1167618 53 20 28 0 0
45 0 0 91928 26 18316 0 0 0 0 736442 1159690 52 20 28 0 0
47 0 0 91926 26 18319 0 0 0 76 737145 1157620 52 20 28 0 0
48 0 0 91921 26 18323 0 0 0 64 739999 1146323 52 19 29 0 0
50 0 0 91918 26 18326 0 0 0 197176 739590 1159797 52 19 28 0 0
50 0 0 91915 26 18330 0 0 0 0 740533 1166111 53 19 28 0 0
52 0 0 91911 26 18334 0 0 0 0 739776 1161328 52 20 28 0 0
42 0 0 91907 26 18338 0 0 0 60 590783 929545 41 16 43 0 0
39 0 0 91904 26 18341 0 0 0 4248 744434 1161062 52 19 29 0 0
41 1 0 91900 26 18345 0 0 0 114688 741817 1163511 53 19 28 0 0
14 0 0 91928 26 18349 0 0 0 32768 598242 996868 43 15 42 0 0
0 0 0 91951 26 18349 0 0 0 0 41914 84357 3 1 96 0 0
0 0 0 91952 26 18349 0 0 0 36 174 204 0 0 100 0 0
0 0 0 91954 26 18349 0 0 0 276 7897 13403 0 0 99 0 0
0 0 0 91954 26 18349 0 0 0 0 1911 3678 0 0 100 0 0
0 0 0 91954 26 18349 0 0 0 147456 330 351 0 0 100 0 0
Edit:
The same pgbench benchmark as above gave about 68k when it was run on the same system as the database and select only 540k (507k from a remote system).
Is CPU:75% user or user + system?
Anyway, 75% of OS CPU with hyper-threading means all cores 100% busy.
You use simple protocol tpc-b which is stressing network and context switches.
See https://franckpachot.medium.com/do-you-know-what-you-are-measuring-with-pgbench-d8692a33e3d6
I tried to generate the image with the Quartz MS font as follows.
Then use jTessBoxEditorFX to generate the box file, as follows.
0 26 23 97 125 0
1 169 26 189 122 0
2 209 23 279 124 0
3 305 23 370 124 0
4 391 25 461 121 0
5 481 23 551 124 0
6 571 23 641 124 0
7 665 27 731 124 0
8 753 24 822 124 0
9 842 24 912 125 0
The traineddata cannot be recognized normally. Is there any problem with my practice?
Whether there are similar experiences of seniors can guide me, thank you.
I am creating a form control in Libre office and am exporting the document to pdf.
Trying to set the text of the control (a textbox) using itextsharp (in other words c# program) only empties the box.
However, if I open the pdf using acrobat reader and edits the text in the box, saving the document results in a pdf where it is possible to write to that textbox.
Why do I have to do that?
Error reproduction
Cliking the toolbar icon in libre office.
Dragging out a square in the document.
Double clicking that box, giving it the name currenttime.
Exporting to pdf:
c# code
if (openFileDialog1.ShowDialog() == DialogResult.OK)
{
if (saveFileDialog1.ShowDialog() == DialogResult.OK)
{
using (var fs = new FileStream(saveFileDialog1.FileName, FileMode.Create))
{
var reader = new PdfReader(openFileDialog1.FileName);
{
using (var pdfStamper = new PdfStamper(reader, fs))
{
var acroFields = pdfStamper.AcroFields;
acroFields.SetField("currentdate", DateTime.Now.ToString());
pdfStamper.FormFlattening = true;
pdfStamper.FreeTextFlattening = true;
pdfStamper.Writer.CloseStream = false;
}
}
reader.Close();
fs.Close();
}
}
}
edit
Here comes textual dumps of the pdf. I have changed some binary data places with "some binary data". The textbox has been given the default value "123".
pdf after it has been created with libre office is of version 1.4
%PDF-1.4
some binary data
2 0 obj
<</Length 3 0 R/Filter/FlateDecode>>
stream
some binary data
endstream
endobj
3 0 obj
78
endobj
7 0 obj
<</Type/FontDescriptor/FontName/LiberationSans
/Flags 4
/FontBBox[-543 -303 1301 980]/ItalicAngle 0
/Ascent 905
/Descent -211
/CapHeight 979
/StemV 80
>>
endobj
8 0 obj
<</Type/Font/Subtype/TrueType/BaseFont/LiberationSans
/Encoding/WinAnsiEncoding
/FirstChar 32 /LastChar 255
/Widths[277 277 354 556 556 889 666 190 333 333 389 583 277 333 277 277
556 556 556 556 556 556 556 556 556 556 277 277 583 583 583 556
1015 666 666 722 722 666 610 777 722 277 500 666 556 833 722 777
666 777 722 666 610 722 666 943 666 666 610 277 277 277 469 556
333 556 556 500 556 556 277 556 556 222 222 500 222 833 556 556
556 556 333 500 277 556 500 722 500 500 500 333 259 333 583 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
277 333 556 556 556 556 259 556 333 736 370 556 583 333 736 552
399 548 333 333 333 576 537 333 333 333 365 556 833 833 833 610
666 666 666 666 666 666 1000 722 666 666 666 666 277 277 277 277
722 722 777 777 777 777 777 583 777 722 722 722 722 666 666 610
556 556 556 556 556 556 889 500 556 556 556 556 277 277 277 277
556 556 556 556 556 556 556 548 610 556 556 556 556 500 556 500
]
/FontDescriptor 7 0 R>>
endobj
5 0 obj
<</F1 8 0 R
>>
endobj
9 0 obj
<</Font 5 0 R
/ProcSet[/PDF/Text]
>>
endobj
1 0 obj
<</Type/Page/Parent 6 0 R/Resources 9 0 R/MediaBox[0 0 595 842]/Annots[
4 0 R ]
/Group<</S/Transparency/CS/DeviceRGB/I true>>/Contents 2 0 R>>
endobj
6 0 obj
<</Type/Pages
/Resources 9 0 R
/MediaBox[ 0 0 595 842 ]
/Kids[ 1 0 R ]
/Count 1>>
endobj
10 0 obj
<</Type/XObject
/Subtype/Form
/BBox[0 0 82.7 23.1]
/Resources 9 0 R
/Length 18
/Filter/FlateDecode
>>
stream
some binary data
endstream
endobj
4 0 obj
<</Type/Annot/Subtype/Widget/F 4
/Rect[59.6 759.3 142.5 782.2]
/FT/Tx
/P 1 0 R
/T(currenttime)
/Ff 4096
/V <FEFF003100320033>
/DV <FEFF003100320033>
/DR<</Font 5 0 R>>
/DA(0 0 0 rg /F1 12 Tf)
/AP<<
/N 10 0 R
>>
>>
endobj
11 0 obj
<</Type/Catalog/Pages 6 0 R
/OpenAction[1 0 R /XYZ null null 0]
/Lang(sv-SE)
/AcroForm<</Fields[
4 0 R
]/DR 9 0 R/NeedAppearances true>>
>>
endobj
12 0 obj
<</Creator<FEFF005700720069007400650072>
/Producer<FEFF004C0069006200720065004F0066006600690063006500200035002E0033>
/CreationDate(D:20170606104859+02'00')>>
endobj
xref
0 13
0000000000 65535 f
0000001431 00000 n
0000000019 00000 n
0000000168 00000 n
0000001843 00000 n
0000001347 00000 n
0000001590 00000 n
0000000187 00000 n
0000000357 00000 n
0000001378 00000 n
0000001688 00000 n
0000002073 00000 n
0000002231 00000 n
trailer
<</Size 13/Root 11 0 R
/Info 12 0 R
/ID [ <5F5DD24A5E7FF740A8BB6B15F88EF602>
<5F5DD24A5E7FF740A8BB6B15F88EF602> ]
/DocChecksum /BFFAD3050AA9FF87945C97B9608B3C6C
>>
startxref
2406
%%EOF
after it has been edited in acrobat reader (I changed the default value of the textbox from "123" to "12"), it will be saved in version 1.6 and an interesting x:xmpmeta information is inserted. Also a lot of empty lines are inserted in the document. At this point, it is programmatically editable.
%PDF-1.6
%âãÏÓ
7 0 obj
<</Linearized 1/L 6449/O 9/E 2599/N 1/T 6160/H [ 451 149]>>
endobj
13 0 obj
<</DecodeParms<</Columns 4/Predictor 12>>/Filter/FlateDecode/ID[<5F5DD24A5E7FF740A8BB6B15F88EF602><FAE65369E246E7409111A7D5BDED1E6F>]/Index[7 17]/Info 6 0 R/Length 52/Prev 6161/Root 8 0 R/Size 24/Type/XRef/W[1 2 1]>>stream
some binary data
endstream
endobj
startxref
0
%%EOF
23 0 obj
<</Filter/FlateDecode/I 92/Length 65/S 38/V 69>>stream
some binary data
endstream
endobj
8 0 obj
<</AcroForm<</DA(/Helv 0 Tf 0 g )/DR 22 0 R/Fields[14 0 R]>>/Lang(sv-SE)/Metadata 1 0 R/OpenAction[9 0 R/XYZ null null 0]/Pages 5 0 R/Type/Catalog>>
endobj
9 0 obj
<</Annots[14 0 R]/Contents 12 0 R/CropBox[0 0 595 842]/Group<</CS/DeviceRGB/I true/S/Transparency>>/MediaBox[0 0 595 842]/Parent 5 0 R/Resources 22 0 R/Rotate 0/Type/Page>>
endobj
10 0 obj
<</BBox[0.0 0.0 82.9 22.9]/Filter/FlateDecode/Length 68/Resources 15 0 R>>stream
some binary data
endstream
endobj
11 0 obj
<</Filter/FlateDecode/First 66/Length 1226/N 9/Type/ObjStm>>stream
some binary data
endstream
endobj
12 0 obj
<</Filter/FlateDecode/Length 78>>stream
some binary data
endstream
endobj
1 0 obj
<</Length 3146/Subtype/XML/Type/Metadata>>stream
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.4-c005 78.147326, 2012/08/23-13:03:03 ">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:xmp="http://ns.adobe.com/xap/1.0/"
xmlns:pdf="http://ns.adobe.com/pdf/1.3/"
xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"
xmlns:dc="http://purl.org/dc/elements/1.1/">
<xmp:CreateDate>2017-06-06T10:48:59+02:00</xmp:CreateDate>
<xmp:CreatorTool>Writer</xmp:CreatorTool>
<xmp:ModifyDate>2017-06-06T11:20:41+02:00</xmp:ModifyDate>
<xmp:MetadataDate>2017-06-06T11:20:41+02:00</xmp:MetadataDate>
<pdf:Producer>LibreOffice 5.3</pdf:Producer>
<xmpMM:DocumentID>uuid:fcdf7344-18ca-44b6-934c-8d5ab8fc8ea3</xmpMM:DocumentID>
<xmpMM:InstanceID>uuid:895fdc09-0aaa-4421-86b2-418c75f88d22</xmpMM:InstanceID>
<dc:format>application/pdf</dc:format>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>
endstream
endobj
2 0 obj
<</Filter/FlateDecode/First 4/Length 48/N 1/Type/ObjStm>>stream
some binary data
endstream
endobj
3 0 obj
<</Filter/FlateDecode/First 4/Length 106/N 1/Type/ObjStm>>stream
some binary data
endstream
endobj
4 0 obj
<</DecodeParms<</Columns 3/Predictor 12>>/Filter/FlateDecode/ID[<5F5DD24A5E7FF740A8BB6B15F88EF602><FAE65369E246E7409111A7D5BDED1E6F>]/Info 6 0 R/Length 37/Root 8 0 R/Size 7/Type/XRef/W[1 2 0]>>stream
some binary data
endstream
endobj
startxref
116
%%EOF
edit2
I was putting the files on my dropbox.
https://www.dropbox.com/sh/5btzl9qqzua18q1/AACIjCrvNZ5cunuLj9sze-l3a?dl=0
As already surmised in a comment, the problem is caused by Libre Office creating the PDF with NeedAppearances set to true in the AcroForm dictionary. Furthermore the wrong field name is used.
Field name
In your code you set the field "currentdate" while in your sample PDFs the field is called "currenttime". Obviously you have to use the correct field name.
NeedAppearances flag
This flag tells a PDF viewer that it shall construct appearance streams and appearance dictionaries for all widget annotations in the document. iText, therefore, when filling in the form field
acroFields.SetField("currentdate", DateTime.Now.ToString());
does not create an appearance for that field - any viewer is required to construct new appearances anyways.
Unfortunately form flattening
pdfStamper.FormFlattening = true;
is implemented by using the existing appearances and only them. As no appearance has been created when setting the field, its flattened form turns out to be empty.
(Strictly speaking this implementation of form flattening is wrong: In this case iText is the PDF processor that wants to make use of the appearances; thus, it should create all appearances here, even ignoring existing ones.)
You can fix this by telling iText to create appearances during form fill-ins in spite of the NeedAppearances flag:
using (var pdfStamper = new PdfStamper(reader, fs))
{
var acroFields = pdfStamper.AcroFields;
acroFields.GenerateAppearances = true;// <<<<<<<<<<<<<<<<<<
acroFields.SetField("currenttime", DateTime.Now.ToString());
pdfStamper.FormFlattening = true;
pdfStamper.FreeTextFlattening = true;
pdfStamper.Writer.CloseStream = false;
}
After adding the marked line above, the output of the code includes the changed newly set value.
Additionally Libre Office does not embed the LiberationSans font. As I have not installed it on my system, I only see dots. I would propose you make LibreOffice embed such fonts or else use standard 14 fonts. Otherwise your PDFs won't display as desired on a number of computers.
hello i need to perform quantization to dct cofficents for an image, for a block
size of 8*8 pixles in matlab. can you help me with the syntax, thank you.
There is a built-in function in MATLAB for DCT.
You need the signal processing tool box. Type 'ver' (without quotes) in the MATLAB command to see if you have it.
The code:
image = image; % define your image
[m,n] = size(image); % get size of your image
imvector = reshape(image, m*n, 1); % reshape your image to a vector to compute DCT
imdct = dct(imvector); % compute DCT
imagedct = reshape(imdct,m,n); \ reshape result back to original form of your image
There is an example in the help file as well which is very nice:
I = imread('cameraman.tif');
I = im2double(I);
T = dctmtx(8);
dct = #(block_struct) T * block_struct.data * T';
B = blockproc(I,[8 8],dct);
mask = [1 1 1 1 0 0 0 0
1 1 1 0 0 0 0 0
1 1 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0];
B2 = blockproc(B,[8 8],#(block_struct) mask .* block_struct.data);
invdct = #(block_struct) T' * block_struct.data * T;
I2 = blockproc(B2,[8 8],invdct);
imshow(I), figure, imshow(I2)
To quantize DCT coefficients, you simply divide each coefficient by a quantization term and round to integers. The quantization terms are often unique for each coefficient, and are stored in a quantization matrix.
Wikipedia has a nice example. Here is how to implement that example in Matlab.
coef = [
-415 -33 -58 35 58 -51 -15 -12;
5 -34 49 18 27 1 -5 3;
-46 14 80 -35 -50 19 7 -18;
-53 21 34 -20 2 34 36 12;
9 -2 9 -5 -32 -15 45 37;
-8 15 -16 7 -8 11 4 7;
19 -28 -2 -26 -2 7 -44 -21;
18 25 -12 -44 35 48 -37 -3
];
quant = [
16 11 10 16 24 40 51 61;
12 12 14 19 26 58 60 55;
14 13 16 24 40 57 69 56;
14 17 22 29 51 87 80 62;
18 22 37 56 68 109 103 77;
24 35 55 64 81 104 113 92;
49 64 78 87 103 121 120 101;
72 92 95 98 112 100 103 99
];
quantCoef = round(coef ./ quant)
quantCoef =
-26 -3 -6 2 2 -1 0 0
0 -3 4 1 1 0 0 0
-3 1 5 -1 -1 0 0 0
-4 1 2 -1 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
clc
clear all
close all
image=imread('Water lilies.jpg');
Q=[8 36 36 36 39 45 52 65;
36 36 36 37 41 47 56 68;
36 36 38 42 47 54 64 78;
36 37 42 50 59 69 81 98;
39 41 47 54 73 89 108 130;
45 47 54 69 89 115 144 178;
53 56 64 81 108 144 190 243;
65 68 78 98 130 178 243 255];
GrayImage=rgb2gray(image);
NewImage=uint8(zeros(size(GrayImage)));
Q=uint32(Q);
for i=1:size(GrayImage,1)/8
for j=1:size(GrayImage,2)/8
block=GrayImage(((i-1)*8)+1:((i-1)*8)+8,((j-1)*8)+1:((j-1)*8)+8);
dct=dct2(block);
dct=uint32(dct);
a=dct./Q;
z=a.*Q;
idct=idct2(z);
NewImage(((i-1)*8)+1:((i-1)*8)+8,((j-1)*8)+1:((j-1)*8)+8)=uint8(idct);
end
end
imwrite(NewImage,'NewImage.jpg');
GrayImage=double(GrayImage);
NewImage=double(NewImage);
MSE=0;
for i=1:size(GrayImage,1)
for j=1:size(GrayImage,2)
d=(GrayImage(i,j)-NewImage(i,j))^2;
MSE=d+MSE;
end
end