Coordinates from getBoundingClientRect() are off on Ionic 3 app - ionic-framework

Question

I am writing an Ionic 3 application to take small pictures.
I am using the Camera Preview plugin to do this:
Then I get the coordinates of the green box using this:
<div id="mask" style="width:100%;height:100px;border:10px solid rgb(125, 255, 0);" *ngIf="!picture"> </div>
var box = document.getElementById("mask");
var rect = box.getBoundingClientRect();
console.log("MASK: "+rect.left+" "+rect.top+" "+rect.width+" "+rect.height)
Then I crop the image using the coordinates I got from the box to get this:
I am using the following code to crop the image:
generateFromImage(img, x, y, w, h, quality: number = 1, callback) {
var canvas: any = document.createElement("canvas");
var image = new Image();
image.src = img;
image.onload = () => {
canvas.width = w;
canvas.height = h;
var ctx = canvas.getContext("2d");
ctx.drawImage(image, x+this.offsetX, y+this.offsetY, w*this.scaleX, h*this.scaleY, 0, 0, w, h);
var dataUrl = canvas.toDataURL('image/jpeg', quality);
callback(dataUrl)
}
}
Where x,y,w,h are the coordinates I got from box.getBoundingClientRect().
As you can see, I had to introduce an offsetX, offsetY, scaleX, scaleY to adjust the coordinates because it was not working.
drawImage() parameters are sx,sy,sw,sh (source box -> coordinates from the original image) and x,y,w,h (destination -> coordinate of the destination image). I do not understand why it is off.
After some trial and error, I found that the following configuration works for my iPhone 8:
offsetX=125
offsetY=48
scaleX=1.7
scaleY=1.4
The Y coordinate difference I suspect that has something to do with the application toolbar. I have no idea why x is off and why using the same w and h on both source and destination does not keep the aspect ratio.
I did not test on another devices but it will most certainly fail because the offsets and scale factors will be different.
Why is it happening? Why do I need to fix the coordinates?
Thanks for your help!

Answer 1

Your main problem is probably that you don't take the CSS scaling into account.
The coordinates we use in drawImage are relative to the image's natural size (i.e, the one of the media, not of the <img>). This means that you can't come from the rendered transform matrix (getBoundingClientRect) to the media's one directly.
You need to first determine the scale that has been applied by CSS, so you can transform your screen space coordinates to the media's space.
var img_bbox = img.getBoundingClientRect();
// the ratio by which our image has been scaled by CSS
var scale_x = img_bbox.width / img.naturalWidth;
var scale_y = img_bbox.height / img.naturalHeight;
// ...then
ctx.drawImage(img,
rect.left / scale_x,
rect.top / scale_y,
rect.width/ scale_x,
rect.height / scale_y,
0,
0,
rect.width,
rect.height
);
Now, you will also have to take into account the difference between the position of your rectangle and the one of your <img>, so e.g x will actually be (rect_bbox.left - img_bbox.left) / scale_x.
One thing you didn't cleared up though, is if you wanted to crop inside, or outside the border, i.e should the border itself be part of the cropped area.
Here is an example which does take the border area too.
onload = crop;
function crop() {
// get the rendered bounding box of our elements
var r_bbox = rect.getBoundingClientRect();
var img_bbox = img.getBoundingClientRect();
// the ratio by which our image has been scaled by CSS
var scale_x = img_bbox.width / img.naturalWidth;
var scale_y = img_bbox.height / img.naturalHeight;
// our output coords
var output = {
x: r_bbox.left - img_bbox.left,
y: r_bbox.top - img_bbox.top,
w: r_bbox.width,
h: r_bbox.height
};
var ctx = canvas.getContext('2d');
canvas.width = output.w;
canvas.height = output.h;
ctx.drawImage(img,
// source
// we need to scale all coords by the CSS scaling
output.x / scale_x,
output.y / scale_y,
output.w / scale_x,
output.h / scale_y,
// destination, to rendered space, no scaling
0,
0,
output.w,
output.h
);
}
img{
width: 200px;
height: 200px;
}
#rect{
position:absolute;
left: 42px;
top: 50px;
width: 100px;
height: 50px;
z-index: 2;
border: 5px solid rgba(0,255,0,.5);
}
*{
vertical-align: top;
}
<img id="img" src="https://upload.wikimedia.org/wikipedia/commons/5/55/John_William_Waterhouse_A_Mermaid.jpg">
<div id="rect"></div>
<canvas id="canvas"></canvas>
And one which takes only what is inside the border-area
onload = crop;
function crop() {
// get the rendered bounding box of our elements
var r_bbox = rect.getBoundingClientRect();
var img_bbox = img.getBoundingClientRect();
// the ratio by which our image has been scaled by CSS
var scale_x = img_bbox.width / img.naturalWidth;
var scale_y = img_bbox.height / img.naturalHeight;
// our output coords
var output = {
// rect.clientLeft is the size of the left border
// so add it to 'x'
x: r_bbox.left - img_bbox.left + rect.clientLeft,
// same as for 'x'
y: r_bbox.top - img_bbox.top + rect.clientTop,
// size of padding box
w: rect.clientWidth,
h: rect.clientHeight
};
var ctx = canvas.getContext('2d');
canvas.width = output.w;
canvas.height = output.h;
ctx.drawImage(img,
output.x / scale_x,
output.y / scale_y,
output.w / scale_x,
output.h / scale_y,
0,
0,
output.w,
output.h
);
}
/* scale our image through CSS */
img{
width: 200px;
height: 200px;
}
#rect{
position:absolute;
left: 42px;
top: 50px;
width: 100px;
height: 50px;
z-index: 2;
border: 5px solid rgba(255,0,0,.5);
background: rgba(0,255,0,.5);
}
*{
vertical-align: top;
}
<img id="img" src="https://upload.wikimedia.org/wikipedia/commons/5/55/John_William_Waterhouse_A_Mermaid.jpg">
<div id="rect"></div>
<canvas id="canvas"></canvas>