SpriteKit Inelastic Collision Reducing Velocity - swift

Question

I'm building a pong/breaker game with a ball and non-static blocks. I'd like the ball to never stop moving, but whenever it hits a block it loses velocity.
I have the restitusion = 1 for all sprites involved, I've tried setting the mass equal to each other and the density and the friction = 0. But, the ball still loses velocity on a bounce.
When the ball hits a block I'm removing it in the didBegin(contact:) function. I've also tried delaying the removal and it didn't help.
I'd like for the ball to have a constant velocity, but still be able to interact with the blocks as later I'd like to add blocks that can be hit without immediately being broken. So, the blocks can't be static but the ball needs to have a constant velocity.
My code for creating the ball node:
func ballNode(_ position: CGPoint?) -> SKSpriteNode {
let node = SKSpriteNode()
node.position = position == nil ? CGPoint(x: size.width/2, y: 100) : position!
node.size = CGSize(width: 17, height: 17)
//background
let background = SKShapeNode(circleOfRadius: 8.5)
background.fillColor = UIColor.white
node.addChild(background)
//physics
node.physicsBody = SKPhysicsBody(circleOfRadius: 8.5)
node.physicsBody?.allowsRotation = true
node.physicsBody?.friction = 0
node.physicsBody?.restitution = 1
node.physicsBody?.linearDamping = 0
node.physicsBody?.angularDamping = 0
node.physicsBody?.categoryBitMask = BallCategory
node.physicsBody?.contactTestBitMask = AddBlockBorderCategory | PaddleCategory
node.physicsBody?.collisionBitMask = PaddleCategory | BlockCategory | BorderCategory
return node
}
My code for creating the block node:
func createBlockNode() -> BlockNode {
let width = (size.width-CGFloat(6*layout[0].count))/CGFloat(layout[0].count)
let height = width*0.5
let nodeSize = CGSize(width: width, height: height)
let node = BlockNode(texture: nil, size: nodeSize)
let background = SKShapeNode(rectOf: nodeSize)
background.fillColor = .darkGray
background.strokeColor = .lightGray
//physics
node.physicsBody = SKPhysicsBody(rectangleOf: nodeSize)
node.physicsBody?.restitution = 1
node.physicsBody?.allowsRotation = true
node.physicsBody?.friction = 0
node.physicsBody?.categoryBitMask = BlockCategory
node.physicsBody?.contactTestBitMask = BallCategory
node.addChild(background)
return node
}
And a screen recording:
screen recording of the ball losing velocity
I'm starting the ball using this:
ball!.physicsBody?.applyForce(CGVector(dx: 0, dy: 50))