Vectors are a foundational concept in linear algebra and have broad applications across the sciences. Among the most fundamental ideas related to vectors are the concepts of basis, orthogonality, and orthonormality.
Let’s take a closer look at each of these ideas with examples to deepen our understanding.
- Basis Vectors
- Orthogonal Vectors
- Orthonormal Vectors
1. Basis Vectors
In linear algebra, a set of vectors is considered a basis for a vector space if:
- They are linearly independent: No vector can be represented as a linear combination of the other vectors.
- They span the space: Any vector in the space can be expressed as a linear combination of these basis vectors.
Data Science Context: When dealing with datasets, you can think of each data point as a vector in some space. Basis vectors can serve as a “coordinate system” for this space. Principal Component Analysis (PCA), a popular dimensionality reduction technique, essentially tries to find a new basis for representing data where each basis vector (or principal component) captures a decreasing amount of variance in the dataset.
Example: Suppose you have data points in 3D space, and you want to represent them in 2D without much loss of information. PCA will find two basis vectors (the first two principal components) that capture the most variance, allowing you to project your data onto these vectors and move from 3D to 2D.
Consider the vectors in $R^{2}$
$ e1 = \begin{bmatrix} 1, 0\ \end{bmatrix} $
$ e2 = \begin{bmatrix} 0, 1\ \end{bmatrix} $
These vectors are a basis for $R^{2}$ because any vector in $R^{2}$ can be written as a combination of e1 and e2.
import numpy as np
# Define the potential basis vectors
e1 = np.array([1, 0])
e2 = np.array([0, 1])
# Check if they are linearly independent using the determinant
matrix = np.vstack([e1, e2])
det = np.linalg.det(matrix)
print("Determinant:", det)
if det != 0:
print("e1 and e2 form a basis for R^2.")
else:
print("e1 and e2 do not form a basis for R^2.")
Determinant: 1.0
e1 and e2 form a basis for R^2.
2. Orthogonal Vectors
Two vectors are orthogonal if their dot product is zero. This means they are perpendicular to each other in geometric terms.
Data Science Context: Orthogonality is valuable in data science because orthogonal features (or vectors) are uncorrelated. This property is especially useful in regression analysis, where multicollinearity (correlation among predictor variables) can distort results and interpretations.
Example: Imagine you have two features: height and height_squared. These features are not orthogonal since they are correlated. But height and, say, weight, could be orthogonal if they are uncorrelated in your datase
For vectors u and v, if u · v = 0, then u and v are orthogonal.
Consider the vectors:
$ u = \begin{bmatrix} 1, 2\ \end{bmatrix} $
$ v = \begin{bmatrix} 2, -1\ \end{bmatrix} $
The dot product is:
$u · v = 1(2) + 2(-1) = 0$
Thus, u and v are orthogonal.
# Vectors are orthogonal if their dot product is zero. Let's check for orthogonality
u = np.array([1, 2])
v = np.array([2, -1])
dot_product = np.dot(u, v)
print("Dot product:", dot_product)
if dot_product == 0:
print("u and v are orthogonal.")
else:
print("u and v are not orthogonal.")
Dot product: 0
u and v are orthogonal.
3. Orthonormal Vectors
Orthonormal vectors are not only orthogonal but also of unit length (magnitude of 1). An orthonormal set of vectors provides a “normalized” basis for a space.
- They are orthogonal to each other.
- Each vector has a magnitude (or length) of 1.
Data Science Context: Orthonormal vectors simplify computations as their lengths are 1, and they are mutually perpendicular. This property is heavily leveraged in algorithms like the Gram-Schmidt process, which orthogonalizes a set of vectors and then normalizes them to generate an orthonormal set.
Example: In the aforementioned PCA, the principal components are not only orthogonal but also orthonormal. This means that when you project your data onto these components, the length of the projections remains unchanged, ensuring that the relative distances between data points are maintained.
$u = [1/sqrt(2), 1/sqrt(2)]$
$v = [-1/sqrt(2), 1/sqrt(2)]$
These vectors are orthonormal because:
- Their dot product is zero, making them orthogonal.
- The magnitude of each vector is 1.
# For vectors to be orthonormal, they must be orthogonal, and each vector's magnitude must be 1. Let's check
u_norm = np.array([1/np.sqrt(2), 1/np.sqrt(2)])
v_norm = np.array([-1/np.sqrt(2), 1/np.sqrt(2)])
dot_product = np.dot(u_norm, v_norm)
u_magnitude = np.linalg.norm(u_norm)
v_magnitude = np.linalg.norm(v_norm)
print("Dot product:", dot_product)
print("Magnitude of u:", u_magnitude)
print("Magnitude of v:", v_magnitude)
if dot_product == 0 and u_magnitude == 1 and v_magnitude == 1:
print("u_norm and v_norm are orthonormal.")
else:
print("u_norm and v_norm are not orthonormal.")
Dot product: 0.0
Magnitude of u: 0.9999999999999999
Magnitude of v: 0.9999999999999999
u_norm and v_norm are not orthonormal.
Conclusion
Understanding the nuances between basis, orthogonal, and orthonormal vectors is key for various applications across mathematics and science. While at first glance they might appear as mere academic concepts, their real-world applications span from quantum physics to computer graphics, making them indispensable tools in the modern world.
As with many mathematical concepts, the true power and beauty of these ideas come to light when applied to solve complex problems.
