In the first article we waljed through really fast on why to use tensors instead of numpy arrays but let's dig in a little bit more in this topic
In the realm of scientific computing and machine learning, efficient data manipulation and numerical computations are paramount. Traditionally, NumPy arrays have been the go-to data structure for numerical operations in Python. However, with the advent of deep learning and the rise of frameworks like PyTorch, Tensors have emerged as a powerful alternative to NumPy arrays. Tensors offer several advantages that make them indispensable in modern machine learning workflows. Such as:
Tensors in frameworks like PyTorch are designed to work seamlessly with computational graphs and automatic differentiation. These graphs represent the flow of data and computations, allowing for efficient backpropagation during neural network training. This automatic differentiation feature is a key enabler of gradient-based optimization algorithms, such as stochastic gradient descent (SGD), which lie at the heart of deep learning.
Tensors can be easily moved onto Graphics Processing Units (GPUs), enabling accelerated computations. GPUs are highly parallel processors that excel at performing matrix operations in parallel. With PyTorch and similar frameworks, tensor operations can be automatically offloaded to GPUs, leading to significant speedups for large-scale computations commonly encountered in deep learning tasks. This GPU acceleration makes Tensors ideal for training complex neural networks and handling large datasets efficiently.
Tensors serve as the fundamental data structure in popular deep learning frameworks like PyTorch and TensorFlow. These frameworks provide a wide range of pre-built neural network layers, loss functions, optimization algorithms, and other utilities optimized for Tensor operations. By using Tensors, practitioners can seamlessly integrate their data and models with these frameworks, leveraging their extensive functionality and performance optimizations.
Tensors bridge the gap between deep learning frameworks and other scientific computing libraries. They offer seamless interoperability with NumPy arrays, enabling easy data exchange between different libraries and modules. This integration allows practitioners to benefit from the mature ecosystem of scientific computing tools, data processing pipelines, and visualization libraries built around NumPy.
Tensors are designed to optimize memory usage and facilitate parallel computations. They allow for efficient storage and manipulation of large multidimensional arrays, and their underlying memory layout enables optimized operations across various dimensions. Tensors also support parallel execution on multi-core CPUs and GPUs, unlocking the potential for parallelism and concurrency in numerical computations.
Specifically, the conversion from NumPy arrays to tensors has emerged as a
critical tool in this realm. By providing the means to bridge the divide
between these data structures, we unlock a world of possibilities and empower
practitioners to navigate the complexities of modern data manipulation and
numerical computations.
That's why PyTorch provides seamless integration with NumPy, allowing for easy
conversion between PyTorch Tensors and NumPy Arrays. These conversions enable
the sharing of underlying memory locations between the two, meaning that any
changes made to one object will be reflected in the other. This convenient
feature simplifies the interoperability between PyTorch and NumPy, making it
easier to leverage the strengths of both libraries.
We'll talk in features post about memory storage in python.
To convert a PyTorch Tensor to a NumPy Array, the numpy() method is used. This method returns a view of the original tensor as a NumPy array, meaning that the data is not copied but shared between the two objects. Let's see an example:
a = torch.ones(5)
print(a)
b = a.numpy()
print(b)
The output will be:
tensor([1., 1., 1., 1., 1.])
[1. 1. 1. 1. 1.]
As you can see, the NumPy array b reflects the values of the original PyTorch Tensor a. This shared memory allows for efficient data interchange between the two libraries.
Bare in mind that due to the memory optimization in python, if we modify the PyTorch Tensor a, the NumPy array b will also be affected:
a.add_(1)
print(a)
print(b)
The output will be:
tensor([2., 2., 2., 2., 2.])
[2. 2. 2. 2. 2.]
Conversely, we can convert a NumPy Array to a PyTorch Tensor using the from_numpy() function. This function creates a new PyTorch Tensor that shares the underlying memory with the NumPy Array. Any modifications made to the NumPy Array will automatically reflect in the PyTorch Tensor. Here's an example:
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)
The output will be:
[2. 2. 2. 2. 2.]
tensor([2., 2., 2., 2., 2.], dtype=torch.float64)
As shown, when we modify the NumPy Array a by adding 1 to each element, the
PyTorch Tensor b is automatically updated to reflect the changes.
Overall, the seamless conversion between PyTorch Tensors and NumPy Arrays
simplifies data interchange and allows for leveraging the strengths of both
libraries. It facilitates working with existing NumPy code and seamlessly
integrating it with PyTorch's powerful tensor operations and automatic
differentiation capabilities.
Now, let's apply the concepts of converting between PyTorch Tensors and NumPy Arrays:
a = np.ones((5, 2))
b = torch.Tensor(a)
In the code above, we first create a NumPy Array a of size (5, 2) initialized with ones. Then, we convert it to a PyTorch Tensor b using the torch.Tensor() function. This will create an array an a tensor with size (5,2) with all ones.
We can check if the tensor b is still linked to the array a
b += 1
print(a,b)
The output will be:
array([[1., 1.],
[1., 1.],
[1., 1.],
[1., 1.],
[1., 1.]]),
tensor([[2., 2.],
[2., 2.],
[2., 2.],
[2., 2.],
[2., 2.]])
The reason the array is no longer linked to the tensor in this case is due to the behavior of the torch.Tensor() function when converting a NumPy array.
When you create the tensor b using b = torch.Tensor(a), a new tensor object is created, and its values are initialized with the values from the NumPy array a. However, the new tensor b and the original NumPy array a are independent of each other.
Subsequently, when you modify the tensor b using the in-place addition operation b += 1, the values of b are incremented by 1. This operation does not affect the original NumPy array a.
Therefore, after performing the in-place addition, the NumPy array a remains unchanged with values of 1, while the tensor b has its values updated to 2. The lack of linkage between the modified tensor b and the original NumPy array a is due to the independent memory locations and separate data structures of the two objects.
Another way to create a tensor of ones will be:
torch.ones(5,2)
torch.ones(5,2)
tensor([[40., 40.],
[80., 80.]])
In the code above, we use the numpy() method on the PyTorch Tensor b to convert it back to a NumPy Array. The variable a1 now holds the NumPy Array. It's important to note that the NumPy Array a1 is a view of the original PyTorch Tensor b. This means that any modifications made to a1 will also affect b. Let's see an example:
To avoid the link between cell smemory, we can create a copy of the NumPy Array a1 using the copy() method. We then modify a2 by adding 1 to each element. As a result, the PyTorch Tensor b remains unchanged, while a2 reflects the modifications
a2 = b.numpy().copy()
a2 += 1
print(b)
print(a2)
The output will be:
tensor([[1., 1.],
[1., 1.],
[1., 1.],
[1., 1.],
[1., 1.]])
array([[2., 2.],
[2., 2.],
[2., 2.],
[2., 2.],
[2., 2.]], dtype=float32)
c1 = torch.from_numpy(a2)
c1 += 1
print(a2)
print(c1)
The output will be:
array([[2., 2.],
[2., 2.],
[2., 2.],
[2., 2.],
[2., 2.]], dtype=float32)
tensor([[3., 3.],
[3., 3.],
[3., 3.],
[3., 3.],
[3., 3.]])
In the code above, we convert the NumPy Array a2 back to a PyTorch Tensor c1 using the torch.from_numpy() function. Similar to the previous examples, modifying c1 will also affect a2 due to their shared memory.
PyTorch supports CUDA tensors for utilizing GPUs to accelerate computation. However, CUDA functionality requires a CUDA-enabled GPU and appropriate configurations.
We first check if CUDA is available. If it is, we can then move the tensors x and y to the GPU using the .cuda() function.
cuda_available = torch.cuda.is_available()
print(cuda_available)
The output will be:
False # If CUDA is not available
True # If CUDA is available
In case Cuda is avaliable, we'll have to transform the tensors to Cuda tensors.
if torch.cuda.is_available():
device = torch.device("cuda") # Create a CUDA device object
x = torch.tensor([1, 2, 3]).to(device) # Create a tensor and move it to the CUDA device
y = torch.tensor([4, 5, 6]).cuda() # Alternatively, you can use the `.cuda()` method
z = x + y # Perform operations on CUDA tensors
print(z)
The output will be:
tensor([5, 7, 9], device='cuda:0')
Please note that the above code will work only if you have a CUDA-enabled GPU and the appropriate CUDA configurations. If CUDA is not available, you will encounter an error.
Overall, the seamless conversion between PyTorch Tensors and NumPy Arrays enables easy interoperability between the two libraries and simplifies data manipulation and integration in machine learning workflows.