Matrix Operations – A Comprehensive Guide on Matrix Tensors and its operations with Examples

Tensors, in general, are mathematical objects that generalize vectors, matrices, and higher-order structures, allowing us to describe more complex interactions, particularly in Data Science. A second-order tensor is often simply referred to as a matrix, but when discussing tensors, the context usually involves a broader mathematical or physical framework.

Matrix or Second-Order Tensor

A matrix is a grid of numbers arranged in rows and columns. Matrices are second-order tensors. They are often used in machine learning to represent datasets and transformations, for instance, rotation or scaling.

Example:
$ V = \begin{bmatrix}
1 & 2 \\
3 & 4\\
5 & 6\\
\end{bmatrix} $

Matrix Operations:

Matrix operations are foundational to many areas of mathematics, computer graphics, data science, and other disciplines. Python doesn’t natively support matrices, but the numpy library makes matrix operations quite easy.

1. Matrix Addition
2. Matrix Subtraction
3. Scalar Multiplication
4. Matrix Multiplication (Element-wise / Dot Product)
5. Matrix Multiplication
6. Transpose of a Matrix
7. Inverse of a Matrix
8. Determinant of a Matrix
9. Reshaping a Matrix
10. Identity and Inverse

1. Matrix Addition:

Just like vectors, you can add matrices element-wise. If you have

$ A = \begin{bmatrix}
1 & 2 \\
3 & 4\\
\end{bmatrix} $

$ B = \begin{bmatrix}
4 & 3 \\
2 & 1 \\
\end{bmatrix} $

then A+B is

$ \begin{bmatrix}
5 & 5 \\
5 & 5 \\
\end{bmatrix} $

import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[4, 3], [2, 1]])

C = A + B
print(C)

[[5 5]
 [5 5]]

2. Matrix Subtraction:

Just like vectors, you can subtract matrices element-wise

$ A = \begin{bmatrix}
1 & 2 \\
3 & 4\\
\end{bmatrix} $

$ B = \begin{bmatrix}
4 & 3 \\
2 & 1 \\
\end{bmatrix} $

then A – B is

$ \begin{bmatrix}
-3 & -1 \\
1 & 3 \\
\end{bmatrix} $

C = A - B
print(C)

[[-3 -1]
 [ 1  3]]

3. Scalar Multiplication:

Matrices can be multiplied by scalars.

If $c = 3$ and

$ A = \begin{bmatrix}
1 & 2 \\
3 & 4\\
\end{bmatrix} $

then $c ⋅ A$ is

$ \begin{bmatrix}
3 & 6 \\
9 & 12\\
\end{bmatrix} $

I = 3 * A
print(I)

[[ 3  6]
 [ 9 12]]

4. Matrix Multiplication (Element-wise / Dot Product):

Element-wise multiplication is straightforward: each element in the resulting matrix is just the product of the corresponding elements in the two matrices being multiplied. Both matrices need to be of the same dimension for this to work.

Let’s break it down with an example:

Given matrices:

$ A = \begin{bmatrix}
1 & 2 \\
3 & 4\\
\end{bmatrix} $

$ B = \begin{bmatrix}
4 & 3 \\
2 & 1 \\
\end{bmatrix} $

C = A ⊙ B (where ⊙ denotes element-wise multiplication)

$ C = \begin{bmatrix}
4 & 6 \\
6 & 4 \\
\end{bmatrix} $

E = A * B
print(E)

[[4 6]
 [6 4]]

5. Matrix Multiplication:

To multiply two matrices, the number of columns in the first matrix must be equal to the number of rows in the second matrix. If matrix A is of size m×n and matrix B is of size n×p, then their product will be of size m×p.

$

c_{ij} = \sum_{k=1}^{n} a_{ik} \times b_{kj}
$

This equation indicates that to get the element $c_{ij}$ of matrix $C$, you take the dot product of the $i^{th}$ row of matrix $A$ with the $j^{th}$ column of matrix $B$.

import numpy as np

# Define the matrices A and B
A = np.array([[1, 2], [3, 4]])
B = np.array([[4, 3], [2, 1]])

# Multiply the matrices
C = np.matmul(A, B)

# Print the result
print(C)

[[ 8  5]
 [20 13]]

6. Transpose of a Matrix:

The transpose of a matrix is obtained by flipping the matrix over its main diagonal. This main diagonal starts from the top left of the matrix and extends to the bottom right. Essentially, in the transpose operation, rows become columns and columns become rows.

Mathematically, if matrix ( A ) has an element ( a_{ij} ) at the ( i^{th} ) row and ( j^{th} ) column, then its transpose, denoted ( A^T ) or ( A’ ), will have this element at the ( j^{th} ) row and ( i^{th} ) column. In other words:

$
(A^T){ij} = a{ji}
$

Example:

Given matrix ( A ):

$
A =
\begin{bmatrix}
1 & 2 & 3 \\
4 & 5 & 6 \\
\end{bmatrix}
$

To compute the transpose ( A^T ):

The resulting matrix ( A^T ) will have:
– The first column as the first row of ( A )
– The second column as the second row of ( A )
– And so on…

Thus:

$
A^T =
\begin{bmatrix}
1 & 4 \\
2 & 5 \\
3 & 6 \\
\end{bmatrix}
$

So, for this 2×3 matrix ( A ), its transpose ( A^T ) is a 3×2 matrix.

For square matrices (matrices with equal rows and columns), the transpose simply swaps the elements on either side of the main diagonal.

import numpy as np
A = np.array([[1, 2, 3], [4, 5, 6]])

G = A.T
print(G)

[[1 4]
 [2 5]
 [3 6]]

7. Inverse of a Matrix:

In linear algebra, the inverse of a matrix (A) is denoted (A^{-1}) and it is the matrix such that when you multiply (A) by its inverse (A^{-1}), you get the identity matrix (I). Mathematically, this can be expressed as:

$A \times A^{-1} = A^{-1} \times A = I$

Where (I) is the identity matrix, a matrix that has ones on its diagonal and zeros everywhere else. It’s called the identity matrix because when you multiply any matrix by the identity matrix, the original matrix is unchanged.

Note: Not all matrices have inverses. For a matrix to have an inverse, it must be a square matrix (same number of rows and columns) and its determinant must not be zero.

How to Find the Inverse:

One of the common methods to find the inverse of a matrix is to use the formula:

$ A^{-1} = \frac{1}{\text{det}(A)} \times \text{adj}(A)$

Where:
– $ \text{det}(A) $ is the determinant of matrix (A).
– $ \text{adj}(A) $ is the adjugate (or adjoint) of matrix (A), which is the transpose of the cofactor matrix of (A).

Example:

Consider the matrix
$ A = \begin{bmatrix}
2 & 1 \\
1 & 1
\end{bmatrix} $

To find its inverse:
1. Calculate the determinant:
$ \text{det}(A) = (2)(1) – (1)(1) = 2 – 1 = 1 $

1. Find the cofactor matrix:
   $ \begin{bmatrix}
   1 & -1 \\
   -1 & 2
   \end{bmatrix} $

2. Take the transpose to get the adjugate:
   $ \begin{bmatrix}
   1 & -1 \\
   -1 & 2
   \end{bmatrix} $

3. Multiply by $ \frac{1}{\text{det}(A)}$:
   $ A^{-1} = \frac{1}{1} \times \begin{bmatrix}
   1 & -1 \\
   -1 & 2
   \end{bmatrix} = \begin{bmatrix}
   1 & -1 \\
   -1 & 2
   \end{bmatrix} $

Thus, the inverse of matrix (A) is:
$ A^{-1} = \begin{bmatrix}
1 & -1 \\
-1 & 2
\end{bmatrix} $

You can verify this by multiplying (A) with its inverse to ensure you get the identity matrix.

import numpy as np
A = np.array([[2, 1], [1, 1]])

H = np.linalg.inv(A)
print(H)

[[ 1. -1.]
 [-1.  2.]]

8. Determinant of a Matrix:

The determinant is a scalar value that is derived from a square matrix and is denoted by det(A) or |A| for a matrix A. It has multiple applications in linear algebra, including understanding whether a matrix is invertible (a matrix is invertible if its determinant is not zero) and for finding the volume of a parallelepiped defined by vectors.

Calculating the Determinant:

• For a 2×2 Matrix:
  Given the matrix
  $ A = \begin{bmatrix} a & b \\ c & d \end{bmatrix} $

The determinant is
$ \text{det}(A) = ad – bc $

• For a 3×3 Matrix:
  Given the matrix
  $ A = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix} $

The determinant is
$ \text{det}(A) = aei + bfg + cdh – ceg – afh – bdi $

For matrices larger than 3×3, the determinant is commonly found using a method called Laplace’s expansion, which recursively calculates the determinant by breaking down the matrix into smaller matrices.

Example:

Let’s determine the determinant of a 2×2 matrix:
$ A = \begin{bmatrix} 3 & 4 \\ 1 & 2 \end{bmatrix} $

Using our formula for a 2×2 matrix:
$ \text{det}(A) = (3)(2) – (4)(1) = 6 – 4 = 2 $

So, the determinant of matrix A is 2.

import numpy as np
A = np.array([[3, 4], [1, 2]])

det_A = np.linalg.det(A)
print(det_A)

2.0000000000000004

9. Reshaping a Matrix :

Reshaping a matrix involves changing its structure to have a different number of rows and columns, while still retaining the same data and the same number of elements. It is commonly used in data manipulation, machine learning, and computer vision tasks.

However, when reshaping, the total number of elements must remain constant. If matrix A has dimensions m x n (where m is the number of rows and n is the number of columns), and you want to reshape it to have dimensions p x q, then m*n must equal p*q.

Example:

Original Matrix A:
$ A = \begin{bmatrix} 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \end{bmatrix} $

The matrix A has dimensions 2 x 4 and contains 8 elements in total.

If we want to reshape it into a matrix with dimensions 4 x 2, it will look like this:

$ B = \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \\ 7 & 8 \end{bmatrix} $

import numpy as np

A = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
B = A.reshape(4, 2)

print(B)

[[1 2]
 [3 4]
 [5 6]
 [7 8]]

10. Identity Matrix:

Just as with numbers where we have 1 as the multiplicative identity (meaning any number multiplied by 1 remains unchanged), for matrices, we have the identity matrix. For a 2×2 matrix,

$ I = \begin{bmatrix}
1 & 0 \\
0 & 1 \\
\end{bmatrix} $

Multiplying any 2×2 matrix by $I$ yields the original matrix. Additionally, some matrices have inverses, such that multiplying the matrix by its inverse yields the identity matrix.

I = np.eye(3)
print(I)

[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]

Conclusion

Second order matrix tensors, or simply matrices, are everywhere. From the equations that power our most cutting-edge machine learning algorithms to fundamental concepts in physics and engineering, understanding matrices is key to grasping the language of the modern world.

As you delve deeper into any technical field, the matrix and its higher-dimensional siblings will undoubtedly be your constant companions.