Machine Learning Begins with Numbers
We have always perceived machine learning as some mysterious black box. It takes in our heterogeneous data, and somehow it is able to see a pattern and give us the prediction based on that pattern. But, in reality, it is just a bunch of numbers playing around. And the best way, known till now, to organise a set of number, while maintaining the computational ease, is to map them in a matrix.
A machine learning feature set consists of data points, and features. Consider following dataset. Here we wish to predict the daily sales of each type of coffee.
| Coffee | Avg. Temp. (°C) | Rain | Serving Type | Previous Sales | Next Day Sales |
|---|---|---|---|---|---|
| Latte | 20 | 1 | Hot | 80 | 75 |
| Espresso | 20 | 1 | Hot | 45 | 42 |
| Cold Brew | 30 | 0 | Cold | 70 | 82 |
| Cappuccino | 20 | 1 | Hot | 65 | 120 |
| Mocha | 30 | 0 | Hot | 55 | 65 |
| Iced Americano | 28 | 0 | Cold | 100 | 135 |
In linear algebra terms, one data point is a vector.
\[
\mathbf{x}_{\mathrm{Latte}} =
\begin{bmatrix}
20\
1\
Hot\
80\
75
\end{bmatrix}
\]
But if we want to work on entire dataset(which is ideal), we need to take help of matrix. Hence, in terms of machine learning, a matrix is a collection of data points.
\[
X =
\begin{bmatrix}
20 & 1 & Hot & 80 & 75 \\
20 & 1 & Hot & 45 & 42 \\
30 & 0 & Cold & 70 & 82\\
20 & 1 & Hot & 65 & 120 \\
30 & 0 & Hot & 55 & 65 \\
28 & 0 & Cold & 100 & 135 \\
\end{bmatrix}
\]
But only numerical values can be represented in a matrix. So, what do we do about the features like Serving Type? We convert them to numerical values. Here, Serving Type has two types of values- Hot & Cold. So, we can encode these as Hot = 0 and Cold = 1. The matrix becomes-
\[
X =
\begin{bmatrix}
20 & 1 & 0 & 80 & 75 \\
20 & 1 & 0 & 45 & 42 \\
30 & 0 & 1 & 70 & 82\\
20 & 1 & 0 & 65 & 120 \\
30 & 0 & 0 & 55 & 65 \\
28 & 0 & 1 & 100 & 135 \\
\end{bmatrix}
\]
But why do machines “think” in vectors and matrices? This is because once real-world information is converted into numbers, vectors and matrices provide a structured way to store and manipulate that information. More importantly, mathematical operations such as multiplication, addition, and, transformations can be performed efficiently on these structures. This is what allows a machine learning model to take numerical representations of our data, combine them with learned parameters, identify relationships between features, and ultimately produce a prediction. Hence, what looks like the machine “finding patterns” is, basically, linear algebra operating on numbers.
How Linear Algebra Powers Learning
We know that machine learning models make predictions. But what actually happens when a model makes one? Or what IS a prediction? Think of a prediction as the model taking the input numbers and combining them with the weights it has learned. This combining happens by performing a dot product between the input vector and weights vector and adding a bias to the product. A weight tells the model how strongly a particular feature contributes to the prediction. When combine features and weights(like we have done below); in some contexts, it can also represent similarity/alignment.
\[
\widehat{y} = \mathbf{w} \cdot \mathbf{x} + b
\]
Suppose our input is:
\[
x =
\begin{bmatrix}
x_1 \\
x_2 \\
x_3 \\
\end{bmatrix}
\]
and the model has learned:
\[
w =
\begin{bmatrix}
w_1 \\
w_2 \\
w_3 \\
\end{bmatrix}
\]
The dot product multiplies the corresponding numbers and adds them:
\[
\mathbf{w} \cdot \mathbf{x} = w_1x_1 + w_2x_2 + w_3x_3
\]
Then we add the bias b:
\[
\widehat{y} = w_1x_1 + w_2x_2 + w_3x_3 + b
\]
That final number, \( \widehat{y} \) , is the prediction.
Now this prediction was only for one input. But a machine learning model usually has to make predictions for hundreds, thousands, or even millions of data points. So, instead of calculating each prediction separately, we can put all our inputs into a matrix:
\[
X =
\begin{bmatrix}
x_{11} & x_{12} & x_{13} \\
x_{21} & x_{22} & x_{23} \\
\vdots & \vdots & \vdots \\
x_{m1} & x_{m2} & x_{m3}
\end{bmatrix}
\]
Each row represents one data point, and each column represents a feature.
Now the same idea can be written as a matrix multiplication:
\[
\widehat{y} =wX + b
\]
Where did the dot product go? Actually, it is still there! Each row of \( X \) takes a dot product with the weight vector \( w \) to produce one prediction. Matrix multiplication helps in performing all those dot products together. This is one of the reasons matrix multiplication is so important in machine learning, it lets us perform many calculations at once.
Linear Algebra Across Machine Learning Algorithms
Till now, we saw how vectors, dot products, and matrix multiplication can transform input data into predictions. Now, let us see how linear algebra finds its way across various machine learning models:
Linear Regression
The example we just saw while learning how predictions are made is actually an example of Linear Regression. It is perhaps the simplest one. The model predicts an output by taking a weighted combination of the input features:
\[
\widehat{y} =wX + b
\]
The model learns the weight vector \( w \) so that its predictions are as close as possible to the actual values. So even the seemingly simple task of fitting a line is really a problem involving vectors, matrices, and optimization
Logistic Regression
Logistic regression starts from almost the same linear combination:
\[
z=wX + b
\]
Instead of using (z) directly as the prediction, it passes it through the sigmoid function:
\[
P(y=1\mid\mathbf{x})=\frac{1}{1+e^{-z}}
\]
The result is a value between 0 and 1 that can be interpreted as a probability. So although logistic regression is used for classification rather than predicting a continuous number, the underlying calculation still begins with a dot product between the input and the learned weights.
Principle Component Analysis(PCA)
Now, PCA approaches the problem from a completely different angle. Instead of predicting an output, it tries to find the directions in the data along which the data varies the most, because, the more the variation is, the information it carries. PCA basically looks at your data, finds the directions where most of the variation is happening, and keeps those important directions discarding the less useful ones.
This is where eigenvectors enter the picture. PCA works with the covariance matrix of the data, and its principal directions are given by the eigenvectors of that matrix:
\[
Cv=\lambda v
\]
Here, \( v \) is an eigenvector representing a direction in the data, while \(\lambda\) tells us how much variance lies along that direction.
The data can then be projected onto these important directions, allowing us to represent high-dimensional data using fewer dimensions.
Singular Value Decomposition
Another powerful linear algebra tool is Singular Value Decomposition (SVD). It takes one complicated matrix and breaks its information into three simpler matrices:
\[
X=U\Sigma V^T
\]
\( U \) : contains the left singular vectors, which describe important directions in the data.
\( \Sigma \) : contains the singular values, which tell us how much importance each direction carries.
\( V^T \) : contains the right singular vectors, which describe important directions in the original feature space.
One particularly interesting application is recommendation systems. Imagine a matrix where rows represent users, columns represent movies, and each entry represents a rating, as shown in a table below. SVD breaks this table into three smaller parts: one that shows people’s preferences(\( U \)), one that shows the importance of each movie(\( \Sigma \)), and one that shows how similar the movies are to each other(\( V^T \)). It can help reveal hidden patterns in this matrix, such as, a user’s preference for particular types of movies, even when many ratings are missing.
| Name | Mera Naam Joker | Mr. India |
|---|---|---|
| Saksham | 5 | 3 |
| Siddhi | 4 | 2 |
| Rishi | 2 | 5 |
K-Means
K-Means looks very different from regression and PCA. Instead of making predictions based on weights, here, the algorithm tries to group similar data points together.
In K-Means, key operation is distance. For two points,
\[
\mathbf{x}=
\begin{bmatrix}
x_1\
x_2
\end{bmatrix},
\qquad
\mathbf{c}=
\begin{bmatrix}
c_1\
c_2
\end{bmatrix}
\]
K-Means needs to answer a simple question: How close are they? It does this using the Euclidean norm of their difference:
\[
|\mathbf{x}-\mathbf{c}|_2
\]
So, subtracting one vector from another gives the direction between them, and the norm tells us how far apart they are. K-Means assigns each data point to the centroid with the smallest distance.
But the linear algebra does not stop there. Once the points have been assigned to clusters, K-Means updates each centroid by taking the mean of all the vectors belonging to that cluster. It then repeats this process until the clusters stop changing significantly. In fact, the objective K-Means is trying to minimise is the total squared distance between each point and its assigned centroid:
\[
\sum_{i=1}^{n} |\mathbf{x}_i-\mathbf{c}_i|_2^2
\]
So K-Means is not just “putting similar things together”. It is really a repeated application of vector subtraction, norms, vector averaging, and optimisation.
Word Embeddings and Transformers
The same ideas continue into modern language models. Words and tokens can be represented as vectors, called embeddings. Words with related meanings can occupy nearby regions of this high-dimensional space, and relationships between them can be explored using operations such as dot products.
Transformers take these ideas much further. At the heart of their attention mechanism are matrix multiplications and dot products. Given query, key, and value matrices \(Q\), \(K\), and \(V\), scaled dot-product attention is written as:
\[
softmax
\left(
\frac{QK^T}{\sqrt{d_k}}
\right)V
\]
The \(QK^T\) operation produces scores that describe how strongly different tokens should attend to one another. Those scores are then used to combine information from the value vectors.
So even in a Transformer containing billions of parameters, we repeatedly encounter the same familiar operations: vectors, dot products, matrix multiplication, and transformations.
Why Learning Linear Algebra Makes You Better at Machine Learning
If you are serious about machine learning, linear algebra is one of those things you can either keep treating as a prerequisite you once studied for an exam, or actually learn to use. Vectors, matrices, dot products, matrix multiplication, eigenvectors, and SVD are not just mathematical vocabulary, rather, they are the operations those algorithms are actually built from. Once you understand them, equations in research papers stop looking like ghosts. You can look at any equation and know what is happening, understand why a model is changing dimensions, or recognise what a matrix represents instead of blindly feeding it into a library.
You also don’t need to disappear into a linear algebra textbook for six months before touching another ML model. Start with vectors and matrices, then get comfortable with dot products, matrix multiplication, and geometric intuition. After that, move to linear transformations, eigenvectors/eigenvalues, and SVD, and finally connect them to things you already encounter in ML, like, linear regression, neural networks, PCA, embeddings, and dimensionality reduction. The point isn’t to become a mathematician before becoming an ML practitioner. It’s to reach the point where, when a model does something unexpected, you can look underneath the API and have some idea of why.
And that brings us to the end of this little journey through linear algebra and machine learning. I’d love to know what you thought of the post, did it make linear algebra feel a little less intimidating, and was it actually helpful? I’m learning and figuring things out along the way too, so if you spot something I could explain better, simplify, or correct, I genuinely welcome your suggestions. This is a learning journey we can take together. There’s plenty more coming on machine learning, mathematics, Python, and computational neuroscience, so if those are things you’re interested in, stay tuned for what’s next!
Also discover Why Your ML Model Performs Great on Training Data but Fails in Real Life