I am making a simulation where you create different balls of certain mass, connected by springs which you can define (in the program below all springs have natural length L and spring constant k). How I do it is I created a function accel(b,BALLS), (note b is the specific ball and BALLS are all of the ball objects in various stages of update) which gets me acceleration on this one ball from calculating all the forces acting on it (tensions from ball the springs connected to it and gravity) and I would think this function is definitely correct and problems lie elsewhere in the while loop. I then use the RK4 method described on this website: http://spiff.rit.edu/richmond/nbody/OrbitRungeKutta4.pdf in the while loop to update velocity and position of each ball. To test my understanding of the method I first made a simulation where only two balls and one spring is involved on Desmos: https://www.desmos.com/calculator/4ag5gkerag
I allowed for energy display and saw that indeed RK4 is much better than Euler method. Now I made it in python in the hope that it should work with arbitrary config of balls and springs, but energy isn't even conserved when I have two balls and one spring! I couldn't see what I did differently, at least when two balls on involved. And when I introduce a third ball and a second spring to the system, energy increases by the hundreds every second. This is my first time coding a simulation with RK4, and I expect you guys can find mistakes in it. I have an idea that maybe the problem is caused by because there are multiple bodies and difficulties arises when I update their kas or kvs at the same time but then again I can't spot any difference between what this code is doing when simulating two balls and my method used in the Desmos file. Here is my code in python:
import pygame
import sys
import math
import numpy as np
pygame.init()
width = 1200
height = 900
SCREEN = pygame.display.set_mode((width, height))
font = pygame.font.Font(None, 25)
TIME = pygame.time.Clock()
dampwall = 1
dt = 0.003
g = 20
k=10
L=200
def dist(a, b):
return math.sqrt((a[0] - b[0])*(a[0] - b[0]) + (a[1] - b[1])*(a[1] - b[1]))
def mag(a):
return dist(a, [0, 0])
def dp(a, b):
return a[0]*b[0]+a[1]*b[1]
def norm(a):
return list(np.array(a)/mag(a))
def reflect(a, b):
return norm([2*a[1]*b[0]*b[1]+a[0]*(b[0]**2 - b[1]**2), 2*a[0]*b[0]*b[1]+a[1]*(-b[0]**2 + b[1]**2)])
class ball:
def __init__(self, x, y, vx, vy, mass,spr,index,ka,kv):
self.r = [x, y]
self.v = [vx, vy]
self.radius = 5
self.mass = mass
self.spr=spr
self.index = index
self.ka=ka
self.kv=kv
def detectbounce(self,width,height):
if self.r[0] + self.radius > width/2 and self.r[0]+self.v[0] > self.r[0] or self.r[0] - self.radius < -width/2 and self.r[0]+self.v[0] < self.r[0] or self.r[1] + self.radius > height/2 and self.r[1]+self.v[1] > self.r[1] or self.r[1] - self.radius < -height/2 and self.r[1]+self.v[1] < self.r[1]:
return True
def bounce_walls(self, width, height):
if self.r[0] + self.radius > width/2 and self.r[0]+self.v[0] > self.r[0]:
self.v[0] *= -dampwall
if self.r[0] - self.radius < -width/2 and self.r[0]+self.v[0] < self.r[0]:
self.v[0] *= -dampwall
if self.r[1] + self.radius > height/2 and self.r[1]+self.v[1] > self.r[1]:
self.v[1] *= -dampwall
if self.r[1] - self.radius < -height/2 and self.r[1]+self.v[1] < self.r[1]:
self.v[1] *= -dampwall
def update_r(self,v, h):
self.r[0] += v[0] * h
self.r[1] += v[1] * h
def un_update_r(self,v, h):
self.r[0] += -v[0] * h
self.r[1] += -v[1] * h
def KE(self):
return 0.5 * self.mass * mag(self.v)**2
def GPE(self):
return self.mass * g * (-self.r[1] + height)
def draw(self, screen, width, height):
pygame.draw.circle(screen, (0, 0, 255), (self.r[0] +
width / 2, self.r[1] + height / 2), self.radius)
#(self, x, y, vx, vy, mass,spr,index,ka,kv):
# balls = [ball(1, 19, 0, 0,5,[1],0,[0,0,0,0],[0,0,0,0]), ball(250, 20, 0,0,1,[0],1,[0,0,0,0],[0,0,0,0])]
# springs = [[0, 1]]
balls = [ball(1, 19, 0, 0,5,[1,3],0,[0,0,0,0],[0,0,0,0]), ball(250, 20, 0,0,2,[0,2,3],1,[0,0,0,0],[0,0,0,0]),ball(450, 0, 0,0,2,[1,3],1,[0,0,0,0],[0,0,0,0]),ball(250, -60, 0,0,2,[0,1,2],1,[0,0,0,0],[0,0,0,0])]
springs = [[0, 1],[1,2],[0,3],[1,3],[2,3]]
def accel(b,BALLS):
A=[0,g]
for i in range(0,len(b.spr)):
ball1=b
ball2=BALLS[b.spr[i]]
r1 = norm(list(np.array(ball2.r) - np.array(ball1.r)))
lnow = dist(ball1.r, ball2.r)
force = k * (lnow - L)
A[0]+=force/ball1.mass*r1[0]
A[1]+=force/ball1.mass*r1[1]
return A
initE=0
while True:
TIME.tick(200)
SCREEN.fill((0, 0, 0))
for event in pygame.event.get():
if event.type == pygame.QUIT:
pygame.quit()
sys.exit()
#compute k1a and k1v for all balls
for ball in balls:
ball.ka[0]=accel(ball,balls)
ball.kv[0]=ball.v
#create newb1 based on 'updated' position of all balls with their own k1v
newb=[]
for ball in balls:
ball.update_r(ball.kv[0], dt/2)
newb.append(ball)
ball.un_update_r(ball.kv[0], dt/2)
#compute k2a and k2v for all balls based on newb1
for ball in balls:
ball.update_r(ball.kv[0], dt/2)
ball.ka[1]=accel(ball,newb)
ball.un_update_r(ball.kv[0], dt/2)
ball.kv[1]=[ball.v[0]+0.5*dt*ball.ka[0][0],ball.v[1]+0.5*dt*ball.ka[0][1]]
#create newb2 based on 'updated' position of all balls with their own k2v
newb=[]
for ball in balls:
ball.update_r(ball.kv[1], dt/2)
newb.append(ball)
ball.un_update_r(ball.kv[1], dt/2)
#compute k3a and k3v for all balls
for ball in balls:
ball.update_r(ball.kv[1], dt/2)
ball.ka[2]=accel(ball,newb)
ball.un_update_r(ball.kv[1], dt/2)
ball.kv[2]=[ball.v[0]+0.5*dt*ball.ka[1][0],ball.v[1]+0.5*dt*ball.ka[1][1]]
newb=[]
for ball in balls:
ball.update_r(ball.kv[2], dt)
newb.append(ball)
ball.un_update_r(ball.kv[2], dt)
#compute k4a and k4v for all balls
for ball in balls:
ball.update_r(ball.kv[2], dt)
ball.ka[3]=accel(ball,newb)
ball.un_update_r(ball.kv[2], dt)
ball.kv[3]=[ball.v[0]+dt*ball.ka[2][0],ball.v[1]+dt*ball.ka[2][1]]
#final stage of update
for ball in balls:
if ball.detectbounce(width,height)==True:
ball.bounce_walls(width, height)
else:
ball.v=[ball.v[0]+dt*(ball.ka[0][0]+2*ball.ka[1][0]+2*ball.ka[2][0]+ball.ka[3][0])/6, ball.v[1]+dt*(ball.ka[0][1]+2*ball.ka[1][1]+2*ball.ka[2][1]+ball.ka[3][1])/6]
ball.r=[ball.r[0]+dt*(ball.kv[0][0]+2*ball.kv[1][0]+2*ball.kv[2][0]+ball.kv[3][0])/6, ball.r[1]+dt*(ball.kv[0][1]+2*ball.kv[1][1]+2*ball.kv[2][1]+ball.kv[3][1])/6]
for ball in balls:
ball.draw(SCREEN, width, height)
for i in range(0,len(ball.spr)):
ball1=ball
ball2=balls[ball.spr[i]]
pygame.draw.line(SCREEN, (0, 0, 155), (
ball1.r[0]+width/2, ball1.r[1]+height/2), (ball2.r[0]+width/2, ball2.r[1]+height/2))
#check for energy
KE = 0
EPE = 0
GPE = 0
for i in range(0, len(springs)):
EPE += 1/2 * k * \
(L - dist(balls[springs[i][0]].r,
balls[springs[i][1]].r))**2
for i in range(0, len(balls)):
KE += balls[i].KE()
GPE += balls[i].GPE()
if initE == 0:
initE += KE+EPE+GPE
text = font.render('init Energy: ' + str(round(initE,1))+' '+'KE: ' + str(round(KE, 1)) + ' '+'EPE: ' + str(round(EPE, 1))+' ' + 'GPE: ' + str(round(GPE, 1)) + ' ' + 'Total: ' + str(round(KE+EPE+GPE, 1)) + ' ' + 'Diff: ' + str(round((KE+EPE+GPE-initE), 1)),
True, (255, 255, 255))
textRect = text.get_rect()
textRect.center = (370, 70)
SCREEN.blit(text, textRect)
pygame.display.flip()
This is the edited, corrected by Lutz Lehmann and with some extra improvements:
import pygame
import sys
import math
import numpy as np
pygame.init()
width = 1200
height = 900
SCREEN = pygame.display.set_mode((width, height))
font = pygame.font.Font(None, 25)
TIME = pygame.time.Clock()
dampwall = 1
dt = 0.003
g = 5
k = 10
L = 200
digits = 6
def dist(a, b):
return math.sqrt((a[0] - b[0])*(a[0] - b[0]) + (a[1] - b[1])*(a[1] - b[1]))
def mag(a):
return dist(a, [0, 0])
def dp(a, b):
return a[0]*b[0]+a[1]*b[1]
def norm(a):
return list(np.array(a)/mag(a))
def reflect(a, b):
return norm([2*a[1]*b[0]*b[1]+a[0]*(b[0]**2 - b[1]**2), 2*a[0]*b[0]*b[1]+a[1]*(-b[0]**2 + b[1]**2)])
class Ball:
def __init__(self, x, y, vx, vy, mass, spr, index, ka, kv):
self.r = [x, y]
self.v = [vx, vy]
self.radius = 5
self.mass = mass
self.spr = spr
self.index = index
self.ka = ka
self.kv = kv
def copy(self):
return Ball(self.r[0], self.r[1], self.v[0], self.v[1], self.mass, self.spr, self.index, self.ka, self.kv)
def detectbounce(self, width, height):
if self.r[0] + self.radius > width/2 and self.r[0]+self.v[0] > self.r[0] or self.r[0] - self.radius < -width/2 and self.r[0]+self.v[0] < self.r[0] or self.r[1] + self.radius > height/2 and self.r[1]+self.v[1] > self.r[1] or self.r[1] - self.radius < -height/2 and self.r[1]+self.v[1] < self.r[1]:
return True
def bounce_walls(self, width, height):
if self.r[0] + self.radius > width/2 and self.r[0]+self.v[0] > self.r[0]:
self.v[0] *= -dampwall
if self.r[0] - self.radius < -width/2 and self.r[0]+self.v[0] < self.r[0]:
self.v[0] *= -dampwall
if self.r[1] + self.radius > height/2 and self.r[1]+self.v[1] > self.r[1]:
self.v[1] *= -dampwall
if self.r[1] - self.radius < -height/2 and self.r[1]+self.v[1] < self.r[1]:
self.v[1] *= -dampwall
def update_r(self, v, h):
self.r[0] += v[0] * h
self.r[1] += v[1] * h
def un_update_r(self, v, h):
self.r[0] += -v[0] * h
self.r[1] += -v[1] * h
def KE(self):
return 0.5 * self.mass * mag(self.v)**2
def GPE(self):
return self.mass * g * (-self.r[1] + height)
def draw(self, screen, width, height):
pygame.draw.circle(screen, (0, 0, 255), (self.r[0] +
width / 2, self.r[1] + height / 2), self.radius)
# (self, x, y, vx, vy, mass,spr,index,ka,kv):
# balls = [Ball(1, 19, 0, 0, 1, [1], 0, [0, 0, 0, 0], [0, 0, 0, 0]),
# Ball(250, 20, 0, 0, 1, [0], 1, [0, 0, 0, 0], [0, 0, 0, 0])]
# springs = [[0, 1]]
balls = [Ball(1, 19, 0, 0,5,[1,3],0,[0,0,0,0],[0,0,0,0]), Ball(250, 20, 0,0,2,[0,2,3],1,[0,0,0,0],[0,0,0,0]),Ball(450, 0, 0,0,2,[1,3],1,[0,0,0,0],[0,0,0,0]),Ball(250, -60, 0,0,2,[0,1,2],1,[0,0,0,0],[0,0,0,0])]
# n=5
# resprings=[]
# for i in range(0,n):
# for j in range(0,n):
# if i==0 and j==0:
# resprings.append([1,2,n,n+1,2*n])
# if i==n and j==0:
# resprings.apend([n*(n-1)+1,n*(n-1)+2,n*(n-2),n*(n-3),n*(n-2)+1])
# if j==0 and i!=0 or i!=n:
# resprings.append([(i-1)*n+1,(i-1)*n+2,(i-2)*n,(i-2)*n+1,(i)*n,(i)*n+1])
def getsprings(B):
S=[]
for i in range(0,len(B)):
theball=B[i]
for j in range(len(theball.spr)):
spring=sorted([i,theball.spr[j]])
if spring not in S:
S.append(spring)
return S
springs = getsprings(balls)
def accel(b, BALLS):
A = [0, g]
for i in range(0, len(b.spr)):
ball1 = b
ball2 = BALLS[b.spr[i]]
r1 = norm(list(np.array(ball2.r) - np.array(ball1.r)))
lnow = dist(ball1.r, ball2.r)
force = k * (lnow - L)
A[0] += force/ball1.mass*r1[0]
A[1] += force/ball1.mass*r1[1]
return A
initE = 0
while True:
TIME.tick(200)
SCREEN.fill((0, 0, 0))
for event in pygame.event.get():
if event.type == pygame.QUIT:
pygame.quit()
sys.exit()
for ball in balls:
ball.bounce_walls(width, height)
# compute k1a and k1v for all balls
for ball in balls:
ball.ka[0] = accel(ball, balls)
ball.kv[0] = ball.v
# create newb1 based on 'updated' position of all balls with their own k1v
newb = []
for ball in balls:
ball.update_r(ball.kv[0], dt/2)
newb.append(ball.copy())
ball.un_update_r(ball.kv[0], dt/2)
# compute k2a and k2v for all balls based on newb1
for ball in balls:
ball.update_r(ball.kv[0], dt/2)
ball.ka[1] = accel(ball, newb)
ball.un_update_r(ball.kv[0], dt/2)
ball.kv[1] = [ball.v[0]+0.5*dt*ball.ka[0]
[0], ball.v[1]+0.5*dt*ball.ka[0][1]]
# create newb2 based on 'updated' position of all balls with their own k2v
newb = []
for ball in balls:
ball.update_r(ball.kv[1], dt/2)
newb.append(ball.copy())
ball.un_update_r(ball.kv[1], dt/2)
# compute k3a and k3v for all balls
for ball in balls:
ball.update_r(ball.kv[1], dt/2)
ball.ka[2] = accel(ball, newb)
ball.un_update_r(ball.kv[1], dt/2)
ball.kv[2] = [ball.v[0]+0.5*dt*ball.ka[1]
[0], ball.v[1]+0.5*dt*ball.ka[1][1]]
newb = []
for ball in balls:
ball.update_r(ball.kv[2], dt)
newb.append(ball.copy())
ball.un_update_r(ball.kv[2], dt)
# compute k4a and k4v for all balls
for ball in balls:
ball.update_r(ball.kv[2], dt)
ball.ka[3] = accel(ball, newb)
ball.un_update_r(ball.kv[2], dt)
ball.kv[3] = [ball.v[0]+dt*ball.ka[2][0], ball.v[1]+dt*ball.ka[2][1]]
# final stage of update
for ball in balls:
ball.v = [ball.v[0]+dt*(ball.ka[0][0]+2*ball.ka[1][0]+2*ball.ka[2][0]+ball.ka[3][0])/6,
ball.v[1]+dt*(ball.ka[0][1]+2*ball.ka[1][1]+2*ball.ka[2][1]+ball.ka[3][1])/6]
ball.r = [ball.r[0]+dt*(ball.kv[0][0]+2*ball.kv[1][0]+2*ball.kv[2][0]+ball.kv[3][0])/6,
ball.r[1]+dt*(ball.kv[0][1]+2*ball.kv[1][1]+2*ball.kv[2][1]+ball.kv[3][1])/6]
for ball in balls:
ball.draw(SCREEN, width, height)
for i in range(0, len(ball.spr)):
ball1 = ball
ball2 = balls[ball.spr[i]]
pygame.draw.line(SCREEN, (0, 0, 155), (
ball1.r[0]+width/2, ball1.r[1]+height/2), (ball2.r[0]+width/2, ball2.r[1]+height/2))
# check for energy
KE = 0
EPE = 0
GPE = 0
for i in range(0, len(springs)):
EPE += 1/2 * k * \
(L - dist(balls[springs[i][0]].r,
balls[springs[i][1]].r))**2
for i in range(0, len(balls)):
KE += balls[i].KE()
GPE += balls[i].GPE()
if initE == 0:
initE += KE+EPE+GPE
text1 = font.render(f"initial energy: {str(round(initE, digits))}", True, (255, 255, 255))
text2 = font.render(f"kinetic energy: {str(round(KE, digits))}", True, (255, 255, 255))
text3 = font.render(f"elastic potential energy: {str(round(EPE, digits))}", True, (255, 255, 255))
text4 = font.render(f"gravitational energy: {str(round(GPE, digits))}", True, (255, 255, 255))
text5 = font.render(f"total energy: {str(round(KE + EPE + GPE, digits))}", True, (255, 255, 255))
text6 = font.render(f"change in energy: {str(round(KE + EPE + GPE - initE, digits))}", True, (255, 255, 255))
SCREEN.blit(text1, (10, 10))
SCREEN.blit(text2, (10, 60))
SCREEN.blit(text3, (10, 110))
SCREEN.blit(text4, (10, 160))
SCREEN.blit(text5, (10, 210))
SCREEN.blit(text6, (10, 260))
pygame.display.flip()
The immediate error seems to be this
for ball in balls:
...
newb1.append(ball)
...
as ball is just a reference to the class ball instance, thus newb1 is a list of references to the objects in balls, it makes no difference if you manipulate the one or the other, it is always the same data records that get changed.
You need to apply a copy mechanism, as you have lists of lists, you need a deep copy, or a dedicated copy member method, else you just copy the array references in the ball instances, so you get different instances, but pointing to the same arrays.
It is probably not an error but still a bad idea to have the class name also as variable name in the same scope.
I have a pinescript strategy below in which I am trying to calculate the Pearsons R correlation value. I got the code from here and modified it a bit (https://www.tradingview.com/script/CD7yUWRV-Linear-Regression-Trend-Channel/).
His code does not include the Pearson's R correlation which is very important to the trading strategy I'm trying to use since it indicates the strength of the trend and it's direction (up or down). To see a working examle of the Pearson's R, add the default indicator linear regression and it will be the number to the bottom left. I'll attach a screenshot for an example.
How do I calculate the Pearson's R Value from the code I have so far?
I have looked for sample pine scripts that have the Pearson's R calculation for linear regression in pinescript and could not find anything.
strategy(title="Linear Regression Trend Channel Strategy", overlay=true,initial_capital=1000,commission_type=strategy.commission.percent,commission_value=0.26,default_qty_type=strategy.percent_of_equity,default_qty_value=100)
period = input( 240, "Period" , input.integer, minval=3)//288
deviations = input( 2.0, "Deviation(s)" , input.float , minval=0.1, step=0.1)
extendType = input("Right", "Extend Method", input.string , options=["Right","None"])=="Right" ? extend.right : extend.none
periodMinusOne = period-1
Ex = 0.0, Ey = 0.0, Ex2 = 0.0, Exy = 0.0, for i=0 to periodMinusOne
closeI = nz(close[i]), Ex := Ex + i, Ey := Ey + closeI, Ex2 := Ex2 + (i * i), Exy := Exy + (closeI * i)
ExEx = Ex * Ex, slope = Ex2==ExEx ? 0.0 : (period * Exy - Ex * Ey) / (period * Ex2 - ExEx)
linearRegression = (Ey - slope * Ex) / period
intercept = linearRegression + bar_index * slope
deviation = 0.0, for i=0 to periodMinusOne
deviation := deviation + pow(nz(close[i]) - (intercept - slope * (bar_index[i])), 2.0)
deviation := deviations * sqrt(deviation / periodMinusOne)
startingPointY = linearRegression + slope * periodMinusOne
var line upperChannelLine = na , var line medianChannelLine = na , var line lowerChannelLine = na
line.delete(upperChannelLine[1]), line.delete(medianChannelLine[1]), line.delete(lowerChannelLine[1])
upperChannelLine := line.new(bar_index - period + 1, startingPointY + deviation, bar_index, linearRegression + deviation, xloc.bar_index, extendType, color.new(#FF0000, 0), line.style_solid , 2)
medianChannelLine := line.new(bar_index - period + 1, startingPointY , bar_index, linearRegression , xloc.bar_index, extendType, color.new(#C0C000, 0), line.style_solid , 1)
lowerChannelLine := line.new(bar_index - period + 1, startingPointY - deviation, bar_index, linearRegression - deviation, xloc.bar_index, extendType, color.new(#00FF00, 0), line.style_solid , 2)
if(crossunder(close,line.get_y2(lowerChannelLine)))
strategy.entry("Long", strategy.long)
if(crossover(close,line.get_y2(upperChannelLine)))
strategy.entry("Short", strategy.short)
// This source code is subject to the terms of the Mozilla Public License 2.0 at https://mozilla.org/MPL/2.0/
// © x11joe
// Credit given to #midtownsk8rguy for original source code. I simply modified to add Pearson's R
//#version=4
study("Linear Regression Trend Channel With Pearson's R", "LRTCWPR", true, format.inherit)
period = input( 20, "Period" , input.integer, minval=3)
deviations = input( 2.0, "Deviation(s)" , input.float , minval=0.1, step=0.1)
extendType = input("Right", "Extend Method", input.string , options=["Right","None"])=="Right" ? extend.right : extend.none
periodMinusOne = period-1
Ex = 0.0, Ey = 0.0, Ex2 = 0.0,Ey2 =0.0, Exy = 0.0, for i=0 to periodMinusOne
closeI = nz(close[i]), Ex := Ex + i, Ey := Ey + closeI, Ex2 := Ex2 + (i * i),Ey2 := Ey2 + (closeI * closeI), Exy := Exy + (closeI * i)
ExT2 = pow(Ex,2.0) //Sum of X THEN Squared
EyT2 = pow(Ey,2.0) //Sym of Y THEN Squared
PearsonsR = (Exy - ((Ex*Ey)/period))/(sqrt(Ex2-(ExT2/period))*sqrt(Ey2-(EyT2/period)))
ExEx = Ex * Ex, slope = Ex2==ExEx ? 0.0 : (period * Exy - Ex * Ey) / (period * Ex2 - ExEx)
linearRegression = (Ey - slope * Ex) / period
intercept = linearRegression + bar_index * slope
deviation = 0.0, for i=0 to periodMinusOne
deviation := deviation + pow(nz(close[i]) - (intercept - slope * (bar_index[i])), 2.0)
deviation := deviations * sqrt(deviation / periodMinusOne)
startingPointY = linearRegression + slope * periodMinusOne
var label pearsonsRLabel = na
label.delete(pearsonsRLabel[1])
pearsonsRLabel := label.new(bar_index,startingPointY - deviation*2,text=tostring(PearsonsR), color=color.black,style=label.style_labeldown,textcolor=color.white,size=size.large)
var line upperChannelLine = na , var line medianChannelLine = na , var line lowerChannelLine = na
line.delete(upperChannelLine[1]), line.delete(medianChannelLine[1]), line.delete(lowerChannelLine[1])
upperChannelLine := line.new(bar_index - period + 1, startingPointY + deviation, bar_index, linearRegression + deviation, xloc.bar_index, extendType, color.new(#FF0000, 0), line.style_solid , 2)
medianChannelLine := line.new(bar_index - period + 1, startingPointY , bar_index, linearRegression , xloc.bar_index, extendType, color.new(#C0C000, 0), line.style_solid , 1)
lowerChannelLine := line.new(bar_index - period + 1, startingPointY - deviation, bar_index, linearRegression - deviation, xloc.bar_index, extendType, color.new(#00FF00, 0), line.style_solid , 2)
Okay so I posted my solution below. I got the idea thanks to a youtube video here, https://www.youtube.com/watch?v=2B_UW-RweSE
This really helped me figure out the formula. Hope this helps someone else on tradingView that needed this!
i think this
PearsonsR = (Exy - ((Ex*Ey)/period))/(sqrt(Ex2-(ExT2/period))*sqrt(Ey2-(EyT2/period)))
should be
PearsonsR = (((Ex*Ey)/period)-Exy)/(sqrt(Ex2-(ExT2/period))*sqrt(Ey2-(EyT2/period)))
according with:https://it.wikipedia.org/wiki/Indice_di_correlazione_di_Pearson#/media/File:Correlation_coefficient.png
How can we measure the similarity distance between categorical data ?
Example:
Gender: Male, Female
Numerical values: [0 - 100], [200 - 300]
Strings: Professionals, beginners, etc,...
Thanks in advance.
There are different ways to do this. One of the simplest would be as follows.
1) Assign numeric value to each property so the order matches the meaning behind the property if possible. It is important to order property values from lower to higher if property can be measured. If it is not possible and property is categorical (like gender, profession, etc), just assign number to each possible value.
P1 - Gender
-------------------
0 - Male
1 - Female
P2 - Experience
-----------
0 - Beginner
5 - Average
10 - Professional
P3 - Age
-----------
[0 - 100]
P4 - Body height, cm
-----------
[50 - 250]
2) For each concept find scale factor and offset so all property values fall in the same chosen range, say [0-100]
Sx = 100 / (Px max - Px min)
Ox = -Px min
In sample provided you would get:
S1 = 100
O1 = 0
S2 = 10
O2 = 0
S3 = 1
O3 = 0
S4 = 0.5
O4 = -50
3) Now you can create a vector containing all the property values.
V = (S1 * P1 + O1, S2 * P2 + O2, S3 * P3 + O3, S4 * P4 + O4)
In sample provided it would be:
V = (100 * P1, 10 * P2, P3, 0.5 * P4 - 50)
4) Now you can compare two vectors V1 and V2 by subtracting one from other. The length of resulting vector will tell how different they are.
delta = |V1 - V2|
Vectors are subtracted by subtracting each dimension. Vector length can be calculated as square root of sum of squared vector dimensions.
Imagine we have 3 persons:
John
P1 = 0 (male)
P2 = 0 (beginner)
P3 = 20 (20 years old)
P4 = 190 (body height is 190 cm)
Kevin
P1 = 0 (male)
P2 = 10 (professional)
P3 = 25 (25 years old)
P4 = 186 (body height is 186 cm)
Lea
P1 = 1 (female)
P2 = 10 (professional)
P3 = 40 (40 years old)
P4 = 178 (body height is 178 cm)
Vectors would be:
J = (100 * 0, 10 * 0, 20, 0.5 * 190 - 50) = (0, 0, 20, 45)
K = (100 * 0, 10 * 10, 25, 0.5 * 186 - 50) = (0, 100, 25, 43)
L = (100 * 1, 10 * 10, 40, 0.5 * 178 - 50) = (100, 100, 40, 39)
To determine we need to subtract vectors:
delta JK = |J - K| =
= |(0 - 0, 0 - 100, 20 - 25, 45 - 43)| =
= |(0, -100, -5, 2)| =
= SQRT(0 ^ 2 + (-100) ^ 2 + (-5) ^ 2 + 2 ^ 2) =
= SQRT(10000 + 25 + 4) =
= 100,14
delta KL = |K - L| =
= |(0 - 100, 100 - 100, 25 - 40, 43 - 39)| =
= |(-100, 0, -15, 4)| =
= SQRT((-100) ^ 2 + 0 ^ 2 + (-15) ^ 2 + 4 ^ 2) =
= SQRT(10000 + 225 + 16) =
= 101,20
delta LJ = |L - J| =
= |(100 - 0, 100 - 0, 40 - 20, 39 - 45)| =
= |(100, 100, 20, -6)| =
= SQRT(100 ^ 2 + 100 ^ 2 + (20) ^ 2 + (-6) ^ 2) =
= SQRT(10000 + 10000 + 400 + 36) =
= 142,95
From this you can see that John and Kevin are more similar than any other as delta is smaller.
There are a number of measures for finding similarity between categorical data. The following paper discuses briefly about these measures.
https://conservancy.umn.edu/bitstream/handle/11299/215736/07-022.pdf?sequence=1&isAllowed=y
If you're trying to do this in R, there's a package named 'nomclust', which has all these similarity measures readily available.
Hope this helps!
If you are using python, there is a latest library which helps in finding the proximity matrix based on similarity measures such as Eskin, overlap, IOF, OF, Lin, Lin1, etc.
After obtaining the proximity matrix we can go on clustering using Hierarchical Cluster Analysis.
Check this link to the library named "Categorical_similarity_measures":
https://pypi.org/project/Categorical-similarity-measures/0.4/
Just a thought, We can also apply euclidean distance between two variables to find a drift value. If it is 0, then there is no drift or else call as similar. But the vector should be sorted and same length before calculation.