Beruflich Dokumente
Kultur Dokumente
Multivariate Calculus
Formula sheet
Dr Samuel J. Cooper
Prof. David Dye
Dr A. Freddie Page
- Power Rule:
Derivative structures
Given f (x) = axb ,
Given f = f (x, y, z)
then f 0 (x) = abx(b−1)
- Product Rule:
- Jacobian:
Given A(x) = f (x)g(x),
∂f ∂f ∂f
then A0 (x) = f 0 (x)g(x) + f (x)g 0 (x) Jf = , ,
∂x ∂y ∂z
- Chain Rule:
Given h = h(p) and p = p(m), - Hessian:
dh dh dp
then = × ∂2f ∂2f ∂2f
dm dp dm
∂x2 ∂x∂y ∂x∂z
Hf = ∂2f ∂2f ∂2f
- Total derivative:
∂y∂x ∂y 2 ∂y∂z
For the function f (x, y, z, ...), where each variable
∂2f ∂2f ∂2f
is a function of parameter t, the total derivative is ∂z∂x ∂z∂y ∂z 2
df ∂f dx ∂f dy ∂f dz
= + + + ...
dt ∂x dt ∂y dt ∂z dt
1
Neural networks - Grad :
- Activation function: ∂f
∂x
ex − e−x
σ(x) = tanh(x) = ∇f = ∂f
ex + e−x ∂y
d 1 4
(σ(x)) = 2 = ∂f
dx cosh (x) (e + e−x )2
x ∂z
- Directional Gradient:
Taylor Series
∇f.r̂
- Univariate:
- Gradient Descent:
1
f (x) = f (c) + f (c)(x − c) + f 00 (c)(x − c)2 + ...
0
2 sn+1 = sn − γ∇f
∞ (n)
X f (c)
= (x − c)n
n! - Lagrange Multipliers λ:
n=0
- Multivariate: ∇f = λ∇g
criterion: ∇χ2 = 0
Optimization and Vector Calculus
- Newton-Raphson:
anext = acur − γ∇χ2
n
f (xi ) X (yi − y(xi ; ak )) ∂y
xi+1 = xi − = acur + γ
f 0 (xi ) σi ∂ak
i