hello i need to perform quantization to dct cofficents for an image, for a block
size of 8*8 pixles in matlab. can you help me with the syntax, thank you.
There is a built-in function in MATLAB for DCT.
You need the signal processing tool box. Type 'ver' (without quotes) in the MATLAB command to see if you have it.
The code:
image = image; % define your image
[m,n] = size(image); % get size of your image
imvector = reshape(image, m*n, 1); % reshape your image to a vector to compute DCT
imdct = dct(imvector); % compute DCT
imagedct = reshape(imdct,m,n); \ reshape result back to original form of your image
There is an example in the help file as well which is very nice:
I = imread('cameraman.tif');
I = im2double(I);
T = dctmtx(8);
dct = #(block_struct) T * block_struct.data * T';
B = blockproc(I,[8 8],dct);
mask = [1 1 1 1 0 0 0 0
1 1 1 0 0 0 0 0
1 1 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0];
B2 = blockproc(B,[8 8],#(block_struct) mask .* block_struct.data);
invdct = #(block_struct) T' * block_struct.data * T;
I2 = blockproc(B2,[8 8],invdct);
imshow(I), figure, imshow(I2)
To quantize DCT coefficients, you simply divide each coefficient by a quantization term and round to integers. The quantization terms are often unique for each coefficient, and are stored in a quantization matrix.
Wikipedia has a nice example. Here is how to implement that example in Matlab.
coef = [
-415 -33 -58 35 58 -51 -15 -12;
5 -34 49 18 27 1 -5 3;
-46 14 80 -35 -50 19 7 -18;
-53 21 34 -20 2 34 36 12;
9 -2 9 -5 -32 -15 45 37;
-8 15 -16 7 -8 11 4 7;
19 -28 -2 -26 -2 7 -44 -21;
18 25 -12 -44 35 48 -37 -3
];
quant = [
16 11 10 16 24 40 51 61;
12 12 14 19 26 58 60 55;
14 13 16 24 40 57 69 56;
14 17 22 29 51 87 80 62;
18 22 37 56 68 109 103 77;
24 35 55 64 81 104 113 92;
49 64 78 87 103 121 120 101;
72 92 95 98 112 100 103 99
];
quantCoef = round(coef ./ quant)
quantCoef =
-26 -3 -6 2 2 -1 0 0
0 -3 4 1 1 0 0 0
-3 1 5 -1 -1 0 0 0
-4 1 2 -1 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
clc
clear all
close all
image=imread('Water lilies.jpg');
Q=[8 36 36 36 39 45 52 65;
36 36 36 37 41 47 56 68;
36 36 38 42 47 54 64 78;
36 37 42 50 59 69 81 98;
39 41 47 54 73 89 108 130;
45 47 54 69 89 115 144 178;
53 56 64 81 108 144 190 243;
65 68 78 98 130 178 243 255];
GrayImage=rgb2gray(image);
NewImage=uint8(zeros(size(GrayImage)));
Q=uint32(Q);
for i=1:size(GrayImage,1)/8
for j=1:size(GrayImage,2)/8
block=GrayImage(((i-1)*8)+1:((i-1)*8)+8,((j-1)*8)+1:((j-1)*8)+8);
dct=dct2(block);
dct=uint32(dct);
a=dct./Q;
z=a.*Q;
idct=idct2(z);
NewImage(((i-1)*8)+1:((i-1)*8)+8,((j-1)*8)+1:((j-1)*8)+8)=uint8(idct);
end
end
imwrite(NewImage,'NewImage.jpg');
GrayImage=double(GrayImage);
NewImage=double(NewImage);
MSE=0;
for i=1:size(GrayImage,1)
for j=1:size(GrayImage,2)
d=(GrayImage(i,j)-NewImage(i,j))^2;
MSE=d+MSE;
end
end
Related
We are testing our application for performance, which is using Postgresql 13 as a database. It is very insert and update heavy and we cannot get more than 65k TPS on the database. But none of the most common bottlenecks do apply (CPU/network/disk IO).
We have also run multiple combinations with pgbench but also those cannot go past 65k.
E.g.
pgbench -i -s 50 -U postgres -h <DB_HOST> -d <DB_NAME>
pgbench -c 64 -j 32 -t 100000 -h <DB_HOST> -U postgres <DB_NAME>
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 64
number of threads: 32
number of transactions per client: 100000
number of transactions actually processed: 6400000/6400000
latency average = 0.976 ms
tps = 65584.664360 (including connections establishing)
tps = 65594.330678 (excluding connections establishing)
(Using more clients resulted in fewer TPS)
The database server has the following specs:
64 Cores (Intel Xeon Gold 6130 (Skylake, 2.10GHz, 2 CPUs/node, 16 cores/CPU))
192 GiB RAM (12 * 16GiB DIMM DDR4 Synchronous Registered (Buffered) 2666 MHz (0.4 ns))
2 * SSD SATA Samsung MZ7KM240HMHQ0D3 (one is used for the WAL and the other for the data)
10 Gbps network link
OS: Debian 11
And we are using the following configuration:
shared_buffers=65551953kB
effective_cache_size=147491895kB
huge_pages=on
min_wal_size=4GB
max_wal_size=16GB
wal_buffers=1GB
work_mem=2GB
maintenance_work_mem=4GB
checkpoint_completion_target=0.9
checkpoint_timeout = 15min
random_page_cost=1.1
bgwriter_flush_after = 2MB
effective_io_concurrency = 200
# Disabled just for performance experiments
fsync = off
synchronous_commit = off
full_page_writes = on
max_worker_processes=64
max_parallel_workers=64
max_parallel_workers_per_gather=10
max_parallel_maintenance_workers=12
When we reach the 65k TPS we have the following resource usage on the server:
Our application:
CPU: 25%
I/O Utilization: 30% resp. 60% (WAL)
Network: in 200Mbps out 125Mbps
pgbench:
CPU: 75%
I/O Utilization: 2% resp. 5% (WAL)
Network: in/out ~500Mbps
We have also temporarily placed the WAL and data into RAM to verify that the disks are not the issue, but this had no impact either.
Does anyone have a guess where our bottleneck might be?
(We have thought about sharding already but would like to use the single server as much as possible)
Edit:
vmstat 1 -S M while running pgbench:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 92009 25 18316 0 0 3 318 405 22 26 11 63 0 0
0 0 0 92009 25 18316 0 0 0 0 261 320 0 0 100 0 0
0 0 0 92009 25 18316 0 0 0 46616 216 257 0 0 100 0 0
45 0 0 92260 25 17996 0 0 0 0 509124 781029 35 13 52 0 0
44 1 0 92252 25 17998 0 0 0 82192 743920 1140285 51 20 30 0 0
50 0 0 92247 25 18003 0 0 0 65564 740366 1152212 52 18 30 0 0
41 0 0 92243 25 18005 0 0 0 0 736052 1138409 50 20 30 0 0
35 0 0 92240 25 18009 0 0 0 0 734265 1138483 50 20 30 0 0
36 0 0 92238 25 18012 0 0 0 264 741315 1152409 52 19 29 0 0
43 0 0 92236 25 18016 0 0 0 36 740374 1147759 51 20 29 0 0
46 0 0 92231 25 18020 0 0 0 163856 737491 1166498 53 19 28 0 0
47 0 0 92228 25 18023 0 0 0 0 741582 1171814 52 19 28 0 0
43 0 0 92223 25 18027 0 0 0 61584 741537 1168119 53 19 28 0 0
46 0 0 92220 25 18030 0 0 0 36 739695 1167456 53 19 28 0 0
43 0 0 92216 25 18034 0 0 0 228 741992 1150333 52 20 29 0 0
38 0 0 92214 25 18036 0 0 0 147464 740589 1166289 52 19 28 0 0
41 0 0 92209 25 18042 0 0 0 0 737148 1162946 52 19 28 0 0
44 0 0 92207 25 18044 0 0 0 2480 741757 1173128 53 19 28 0 0
39 0 0 92205 25 18049 0 0 0 48 740404 1170644 53 19 28 0 0
52 0 0 92201 25 18051 0 0 0 292 739032 1159037 52 19 28 0 0
42 1 0 92198 25 18054 0 0 0 20072 740101 1165594 52 20 28 0 0
51 0 0 92194 25 18059 0 0 0 45464 738055 1165382 53 19 28 0 0
41 0 0 92190 25 18062 0 0 0 0 742838 1172377 53 19 28 0 0
45 0 0 92185 25 18067 0 0 0 36 740704 1174534 53 19 28 0 0
50 0 0 92182 25 18069 0 0 0 92 741691 1150716 52 19 28 0 0
42 0 0 92177 25 18073 0 0 0 28 740220 1168488 53 18 28 0 0
44 0 0 92174 25 18077 0 0 0 0 738818 1164769 53 19 28 0 0
46 0 0 92172 25 18080 0 0 0 0 740720 1169902 53 19 28 0 0
46 0 0 92166 25 18083 0 0 0 90404 592524 945810 42 15 43 0 0
45 0 0 92161 25 18087 0 0 0 3884 746310 1159898 52 19 28 0 0
49 0 0 92157 25 18090 0 0 0 20 747242 1177750 53 19 28 0 0
36 0 0 92152 25 18094 0 0 0 0 744477 1173832 53 19 28 0 0
46 0 0 92149 25 18098 0 0 0 0 746194 1172700 53 19 28 0 0
39 0 0 92147 26 18101 0 0 0 49768 745651 1177462 54 18 28 0 0
43 0 0 92143 26 18105 0 0 0 212 744968 1161110 53 19 28 0 0
43 0 0 92138 26 18109 0 0 0 0 743223 1176960 54 19 28 0 0
43 0 0 92135 26 18112 0 0 0 81920 745168 1173574 53 19 28 0 0
48 0 0 92132 26 18116 0 0 0 0 743174 1169255 53 19 28 0 0
48 0 0 92129 26 18120 0 0 0 68 592295 933445 42 15 43 0 0
41 0 0 92124 26 18123 0 0 0 76 740354 1162221 52 19 28 0 0
49 0 0 92120 26 18125 0 0 0 0 738456 1158291 53 19 28 0 0
39 0 0 92117 26 18129 0 0 0 147536 740735 1162479 52 20 28 0 0
49 0 0 92113 26 18133 0 0 0 0 737209 1165532 53 20 28 0 0
49 0 0 92111 26 18137 0 0 0 40 741185 1168133 54 19 28 0 0
45 0 0 92110 26 18140 0 0 0 4000 740693 1141945 52 20 28 0 0
42 0 0 92105 26 18144 0 0 0 0 741857 1168830 53 19 28 0 0
43 0 0 92102 26 18147 0 0 0 8 742546 1168867 54 18 28 0 0
43 0 0 92101 26 18150 0 0 0 147456 741941 1166646 53 19 28 0 0
41 1 0 92097 26 18154 0 0 0 64192 740052 1169040 53 19 28 0 0
48 0 0 92094 26 18158 0 0 0 27484 737224 1139511 52 20 28 0 0
47 0 0 92087 26 18162 0 0 0 0 740821 1165037 53 19 28 0 0
54 0 0 92059 26 18164 0 0 0 4 737109 1155098 53 19 27 0 0
38 0 0 92051 26 18170 0 0 0 147456 701847 1075069 55 20 25 0 0
35 0 0 92064 26 18174 0 0 0 44 723153 1125736 54 19 27 0 0
48 0 0 92056 26 18179 0 0 0 53008 734590 1134838 52 19 29 0 0
46 0 0 92053 26 18183 0 0 0 0 741595 1166891 53 19 28 0 0
46 0 0 92049 26 18186 0 0 0 0 740196 1170838 54 19 27 0 0
31 1 0 92045 26 18191 0 0 0 98304 741800 1170076 54 18 28 0 0
44 0 0 92043 26 18194 0 0 0 49188 733352 1173652 53 20 27 0 0
43 0 0 92028 26 18198 0 0 0 116 733522 1151497 53 20 27 0 0
44 0 0 92037 26 18201 0 0 0 0 730364 1137665 53 19 27 0 0
37 0 0 92035 26 18204 0 0 0 0 742348 1164945 53 19 28 0 0
44 0 0 92031 26 18208 0 0 0 0 739273 1165044 53 19 28 0 0
52 0 0 92028 26 18211 0 0 0 147888 739274 1164496 53 19 28 0 0
50 0 0 92024 26 18215 0 0 0 144 739684 1145210 53 19 28 0 0
50 0 0 92020 26 18219 0 0 0 0 742847 1167779 54 18 28 0 0
38 0 0 92016 26 18223 0 0 0 0 738079 1166580 53 19 28 0 0
36 0 0 92013 26 18226 0 0 0 0 742687 1171101 54 18 27 0 0
48 0 0 92009 26 18229 0 0 0 147500 741536 1166846 53 19 28 0 0
40 0 0 92006 26 18233 0 0 0 94600 740746 1147102 52 20 28 0 0
45 0 0 92001 26 18238 0 0 0 0 741119 1163851 53 19 28 0 0
48 0 0 91999 26 18241 0 0 0 0 740995 1167197 53 19 28 0 0
35 0 0 91996 26 18244 0 0 0 0 742235 1165666 53 19 28 0 0
44 1 0 91993 26 18248 0 0 0 49192 741392 1164506 53 19 28 0 0
43 1 0 91990 26 18251 0 0 0 124876 743695 1144639 52 19 29 0 0
48 0 0 91987 26 18255 0 0 0 24864 737759 1159383 52 20 28 0 0
44 0 0 91983 26 18258 0 0 0 0 740224 1164983 53 19 28 0 0
43 0 0 91980 26 18262 0 0 0 0 741742 1168140 54 19 27 0 0
18 0 0 91976 26 18267 0 0 0 36 737449 1162293 53 19 28 0 0
49 0 0 91973 26 18269 0 0 0 147576 741462 1148048 52 20 28 0 0
43 0 0 91969 26 18274 0 0 0 0 742408 1168332 54 19 27 0 0
43 0 0 91966 26 18277 0 0 0 0 738803 1164992 53 19 28 0 0
39 0 0 91963 26 18280 0 0 0 4 737891 1159372 52 19 28 0 0
43 0 0 91962 26 18283 0 0 0 40 741888 1166835 53 19 28 0 0
48 0 0 91958 26 18287 0 0 0 164144 738677 1145900 52 20 28 0 0
46 0 0 91955 26 18291 0 0 0 0 740956 1165789 53 19 28 0 0
44 0 0 91952 26 18295 0 0 0 0 741055 1166460 53 19 28 0 0
44 0 0 91948 26 18299 0 0 0 8 739414 1165698 53 19 28 0 0
46 0 0 91945 26 18301 0 0 0 48 743218 1165277 53 19 28 0 0
36 0 0 91941 26 18305 0 0 0 208 736320 1134425 51 20 29 0 0
47 0 0 91936 26 18309 0 0 0 239096 739799 1159730 52 19 28 0 0
45 0 0 91932 26 18312 0 0 0 0 742477 1167618 53 20 28 0 0
45 0 0 91928 26 18316 0 0 0 0 736442 1159690 52 20 28 0 0
47 0 0 91926 26 18319 0 0 0 76 737145 1157620 52 20 28 0 0
48 0 0 91921 26 18323 0 0 0 64 739999 1146323 52 19 29 0 0
50 0 0 91918 26 18326 0 0 0 197176 739590 1159797 52 19 28 0 0
50 0 0 91915 26 18330 0 0 0 0 740533 1166111 53 19 28 0 0
52 0 0 91911 26 18334 0 0 0 0 739776 1161328 52 20 28 0 0
42 0 0 91907 26 18338 0 0 0 60 590783 929545 41 16 43 0 0
39 0 0 91904 26 18341 0 0 0 4248 744434 1161062 52 19 29 0 0
41 1 0 91900 26 18345 0 0 0 114688 741817 1163511 53 19 28 0 0
14 0 0 91928 26 18349 0 0 0 32768 598242 996868 43 15 42 0 0
0 0 0 91951 26 18349 0 0 0 0 41914 84357 3 1 96 0 0
0 0 0 91952 26 18349 0 0 0 36 174 204 0 0 100 0 0
0 0 0 91954 26 18349 0 0 0 276 7897 13403 0 0 99 0 0
0 0 0 91954 26 18349 0 0 0 0 1911 3678 0 0 100 0 0
0 0 0 91954 26 18349 0 0 0 147456 330 351 0 0 100 0 0
Edit:
The same pgbench benchmark as above gave about 68k when it was run on the same system as the database and select only 540k (507k from a remote system).
Is CPU:75% user or user + system?
Anyway, 75% of OS CPU with hyper-threading means all cores 100% busy.
You use simple protocol tpc-b which is stressing network and context switches.
See https://franckpachot.medium.com/do-you-know-what-you-are-measuring-with-pgbench-d8692a33e3d6
I have this matrix:
A =[22 22 142 142 142 92 92 92 0 0
0 109 109 151 151 151 23 23 149 149
0 0 0 152 152 152 38 38 0 0
0 13 13 113 113 113 119 119 119 0
0 8 8 8 84 84 14 14 14 0
0 0 144 144 144 0 0 0 66 66
139 139 139 34 34 34 0 0 0 0
0 0 64 64 64 128 128 59 59 59
83 83 83 65 65 65 67 67 67 0];
How can I find indices (row, column) from matrix with zero value respectively 2 or more?
You can use find as follows:
[r,c] = find(A==0)
[rows,cols] = ind2sub(size(A),find(A==0))
find gives you the indices and ind2sub converts them in column-row format.
I'm working on a program that's supposed to write out the multiplication table as shown in the picture.
This is what I've done
A = (1:10)'*(1:10);
tril (A)
And this is my output. Is there a way I can do this without the zeros?
Or should I go with a different approach? Any help is greatly appreciated.
1 0 0 0 0 0 0 0 0 0
2 4 0 0 0 0 0 0 0 0
3 6 9 0 0 0 0 0 0 0
4 8 12 16 0 0 0 0 0 0
5 10 15 20 25 0 0 0 0 0
6 12 18 24 30 36 0 0 0 0
7 14 21 28 35 42 49 0 0 0
8 16 24 32 40 48 56 64 0 0
9 18 27 36 45 54 63 72 81 0
10 20 30 40 50 60 70 80 90 100
Here's one way:
A=tril((1:10)'*(1:10))
A(A==0)=NaN;
S=num2str(A);
S(S==78|S==97)=' '
The second line distinguishes plain old '0' from a pesky '0' in, say, '20'.
The third line converts the array to a string.
The last line replaces capital 'N' (character 78) and lowercase 'a' (character 97) with blank space.
Your approach is good, but if you want to avoid the zeros you want strings, not numbers. For example,
>> n = 10;
>> char(arrayfun(#(k) {sprintf('%i ', k:k:k^2)}, 1:n).')
ans =
1
2 4
3 6 9
4 8 12 16
5 10 15 20 25
6 12 18 24 30 36
7 14 21 28 35 42 49
8 16 24 32 40 48 56 64
9 18 27 36 45 54 63 72 81
10 20 30 40 50 60 70 80 90 100
How it works
sprintf('%i ', k:k:k^2) generates each row of the table as a string. cellfun is used to iterate over all rows. The rows are packed into a cell array of strings, and that cell array is converted to a char matrix, which automatically pads with spaces.
We have a MxN matrix and a constrain cstrn = 100;.
The constrain is the summarize limit of column's elements (per column):
sum(matrix(:,:))<=cstrn.
For a given example as the following:
Columns 1 to 5:
15 18 -5 22 19
50 98 -15 39 -8
70 -15 80 45 38
31 52 9 80 72
-2 63 52 71 6
7 99 32 58 41
I want to find the max number of element per column who fulfill this constrain.
How can i summarize every column element with the others elements in same column and find which sum combinations uses the max number of elements per column?
In the given example solution is:
4 3 5 2 5
where
column 1: 15 + 50 + 31 +7 +(-2)
column 2: 18 +(-15) + 52 or 63 etc.
Thank you in advance.
Since it is always easier to fit small elements into a sum, you can do a sort, followed by the cumulative sum:
m= [
15 18 -5 22 19
50 98 -15 39 -8
70 -15 80 45 38
31 52 9 80 72
-2 63 52 71 6
7 99 32 58 41];
cs = cumsum(sort(m))
cs =
-2 -15 -15 22 -8
5 3 -20 61 -2
20 55 -11 106 17
51 118 21 164 55
101 216 73 235 96
171 315 153 315 168
Now you easily identify at which element you cross the threshold cnstrn (thanks, #sevenless)!
out = sum(cs <= cnstrn)
out =
4 3 5 2 5
I'd add to Jonas's answer, that you can impose your constraint in a way that outputs a logical matrix then sum over the 1's and 0's of that matrix like so:
cstrn = 100
m= [
15 18 -5 22 19
50 98 -15 39 -8
70 -15 80 45 38
31 52 9 80 72
-2 63 52 71 6
7 99 32 58 41];
val_matrix = cumsum(sort(m))
logical_matrix = val_matrix<=cstrn
output = sum(logical_matrix)
Giving output:
cstrn =
100
val_matrix =
-2 -15 -15 22 -8
5 3 -20 61 -2
20 55 -11 106 17
51 118 21 164 55
101 216 73 235 96
171 315 153 315 168
logical_matrix =
1 1 1 1 1
1 1 1 1 1
1 1 1 0 1
1 0 1 0 1
0 0 1 0 1
0 0 0 0 0
output =
4 3 5 2 5
Here is a logic, on mobile so can't give a code.
Check this out. Go to a column, sort it ascending order, loop to sum, break when hits <=100. Get counter. Refer back to original column, get the indices of elements matching the elements you just summed up :-)
I have a matrix 'eff_tot' with dimension (m x n) which I want to rearrange according to a matrix called 'matches' (e.g. [n2 n3; n4 n5]) and put all the collumns not specified in 'matches' at the end.
That is, I want to have [eff_tot(:,n2) eff_tot(:,n3) ; eff_tot(:,n4) eff_tot(:,n5) ; eff_tot(:,n1)].
That's all folks!
Taking the example in the first answer, what I would like to have is:
eff_tot =
81 15 45 15 24
44 86 11 14 42
92 63 97 87 5
19 36 1 58 91
27 52 78 55 95
82 41 0 0 0
87 8 0 0 0
9 24 0 0 0
40 13 0 0 0
26 19 0 0 0
Regards.
Create a vector listing the indices of all the columns in eff_tot and then use SETDIFF to determine which columns do not occur in [n2 n3 n4 n5]. These columns are the unmatched ones. Now concatenate the matched and unmatched column indices to create your column-reordered eff_tot matrix.
>> eff_tot = randi(100, 5, 7)
eff_tot =
45 82 81 15 15 41 24
11 87 44 14 86 8 42
97 9 92 87 63 24 5
1 40 19 58 36 13 91
78 26 27 55 52 19 95
>> n2 = 3; n3 = 5; n4 = 2; n5 = 6;
>> missingColumn = setdiff(1:size(eff_tot, 2), [n2 n3 n4 n5])
missingColumn =
1 4 7
>> eff_tot = [eff_tot(:,n2) eff_tot(:,n3) eff_tot(:,missingIndex); eff_tot(:,n4) eff_tot(:,n5) zeros(size(eff_tot, 1), length(missingIndex))];
eff_tot =
81 15 45 15 24
44 86 11 14 42
92 63 97 87 5
19 36 1 58 91
27 52 78 55 95
82 41 0 0 0
87 8 0 0 0
9 24 0 0 0
40 13 0 0 0
26 19 0 0 0