What is Broadcasting and how it helps?

Broadcasting refer to the rules of how different shaped arrays are treated during arithmetic operations. Code examples are given for demonstration.

How does it help?

Before diving in to the details of what broadcasting is, lets see what it can actually do. Below is the multiplication of 2 matrices, where shape of matrix1 is (5,784) and shape of matrix2 is (784,10).

Multiplication with nested loops:


Multiplication with Broadcasting:


Speed Difference:

As you just saw, matrix multiplication with nested loops took 1.85 seconds while the same matrices multiplied together with broadcasting took only 2ms, thats around a thousand times faster. So what exactly happened there? Lets find out.


Broadcasting refers to the rules of how arrays with different shapes are treated during arithmetic operations. Under certain constraints, the smaller array is 'broadcast' across the broader array so that their shapes are equal.
Things are better understood when looking at code, so lets move on to code.

Arithmetic operations between scalars and matrices:


When we multiply 2 with the matrix, what happens is that the '2' is broadcast to a matrix of size 3x3 and then an element wise multiplication is carried out. BUT the new 'broadcast matrix' is not actually stored in memory, which means we have a 3x3 matrix which is of the size of a single number in our memory.

To demonstrate this further, lets add a 1x3 matrix to a 3x3 matrix.


In line 67 (in the image) while adding the matrices, the 'x_matrix' is 'expanded' to the 3x3 'mat' matrix by adding 2 more rows of the same values to it. The 'expand_as' operation is demonstrated in line 69 in the image. In line 70, you can see that it only stores 3 numbers in the memory rather than storing the whole matrix, and the calculations are also done in C language rather than Python which speeds the operations up.

Where do we use it?

Broadcasting is used in many places, Numpy and pytorch does most of the matrix calculations by broadcasting. In Machine Learning, we do hundreds of thousands of Matrix Multiplications, without broadcasting, it wouldn't be possible.

One example of difference of speeds in Matrix Multiplication with nested loops and broadcasting was provided at the start of this blog, here's the code for both of those methods:

Matrix Multiplication with Nested Loops:

def matmul(mat1,mat2):
    a,b = mat1.shape
    c,d = mat2.shape
    res= torch.zeros(a,d)
    for i in range(0,a):
        for j in range(0,d):
            for k in range(0,b):
                res[i,j] += mat1[i,k] * mat2[k,j]
    return res


Matrix Multiplication with Broadcasting:

#Matmul with broadcasting
def matmul_broad(a,b):
    ar,ac = a.shape
    br,bc = b.shape
    assert ac==br
    c = torch.zeros(ar, bc)
    for i in range(ar):
        c[i]   = (a[i  ].unsqueeze(-1) * b).sum(dim=0)
    return c


Code Explanation:

I assume you understand how the nested loops work. So I will only explain how the code with Broadcasting works.

First you need to understand what 'unsqueeze' does. See the image for that:


torch.unsqueeze adds an additional dimension to the tensor. So let's say you have a tensor of shape (3), if you add a dimension at the 0 position, it will be of shape (1,3), which means 1 row and 3 columns. If you add at the 1 position, it will be (3,1), which means 3 rows and 1 column. If you have a 2D tensor of shape (2,2) add add an extra dimension at the 0 position, this will result of the tensor having a shape of (1,2,2), which means one channel, 2 rows and 2 columns.

In our code, inside the for loop, we take 1 row of the matrix, change it in to a column, multiply that column with each column of the other matrix (the one column is broadcast to 3 columns behind the scenes), then we sum each column of the resultant matrix.

This is equivalent to how we used to learn matrix multiplication in schools i.e "First row in to first column, first row in to second column, first row in to 3rd column, then second row in to first column, second row into 2nd column and so on".

Learn Django with us

Visit Youtube Channel
Latest articles