How to to identify letters on a license plate with varying perspectives - matlab

Question

I am making a script in Matlab that takes in an image of the rear of a car. After some image processing I would like to output the original image of the car with a rectangle around the license plate of the car. Here is what I have written so far:
origImg = imread('CAR_IMAGE.jpg');
I = imresize(origImg, [500, NaN]); % easier viewing and edge connecting
G = rgb2gray(I);
M = imgaussfilt(G); % blur to remove some noise
E = edge(M, 'Canny', 0.4);
% I can assume all letters are somewhat upright
RP = regionprops(E, 'PixelIdxList', 'BoundingBox');
W = vertcat(RP.BoundingBox); W = W(:,3); % get the widths of the BBs
H = vertcat(RP.BoundingBox); H = H(:,4); % get the heights of the BBs
FATTIES = W > H; % find the BBs that are more wide than tall
RP = RP(FATTIES);
E(vertcat(RP.PixelIdxList)) = false; % remove more wide than tall regions
D = imdilate(E, strel('disk', 1)); % dilate for easier viewing
figure();
imshowpair(I, D, 'montage'); % display original image and processed image
Here are some examples:
From here I am unsure how to isolate the letters of the license plate, particularly like in the second example above where each letter has a decreased area due to the perspective of the image. My first idea was to get the bounding box of all regions and keep only the regions where the perimeter to area ratio is "similar" but this resulted in removing the letters of the plate that were connected when I dilate the image like the K and V in the fourth example above.
I would appreciate some suggestions on how I should go about isolating these letters. No code is necessary, and any advice is appreciated.

Answer 1

So I continued to work despite not receiving any answers here on SO and managed to get a working version through trial and error. All of the following code comes after the code in my original question and all plots below are from the first example image above. First, I found the variance for every single pixel row of the image and plotted them like so:
V = var(D, 0, 2);
X = 1:length(V);
figure();
hold on;
scatter(X, V);
I then fit a very high order polynomial to this scatter plot and saved the values where the slope of the polynomial was zero and the variance value was very low (i.e. the dark row of pixels immediately before or after a row with some white):
P = polyfit(X', V, 25);
PV = polyval(P, X);
Z = X(find(PV < 0.03 & abs(gradient(PV)) < 0.0001));
plot(X, PV); % red curve on plot
scatter(Z, zeros(1,length(Z))); % orange circles on x-axis
I then calculate the integral of the polynomial between any consecutive Z values (my dark rows), and save the two Z values between which the integral is the largest, which I mark with lines on the plot:
MAX_INTEG = -1;
MIN_ROW = -1;
MAX_ROW = -1;
for i = 1:(length(Z)-1)
TEMP_MIN = Z(i);
TEMP_MAX = Z(i+1);
Q = polyint(P);
TEMP_INTEG = diff(polyval(Q, [TEMP_MIN, TEMP_MAX]));
if (TEMP_INTEG > MAX_INTEG)
MAX_INTEG = TEMP_INTEG;
MIN_ROW = TEMP_MIN;
MAX_ROW = TEMP_MAX;
end
end
line([MIN_ROW, MIN_ROW], [-0.1, max(V)+0.1]);
line([MAX_ROW, MAX_ROW], [-0.1, max(V)+0.1]);
hold off;
Since the X-values of these lines correspond row numbers in the original image, I can crop my image between MIN_ROW and MAX_ROW:
I repeat the above steps now for the columns of pixels, crop, and remove any excess black rows of columns to result in the identified plate:
I then perform 2D cross correlation between this cropped image and the edged image D using Matlab's xcorr2 to locate the plate in the original image. After finding the location I just draw a rectangle around the discovered plate like so: