Recurrent Neural Network are very suitable to work with sequential data and make prediction using this kind of data. In this posts, we will Talk about RNNs and some of its variants, GRU and BiGRU and HAN architecture which uses self attention with BidirGRU. Also, we will see an application of HAN to classify IMDB movie reviews. Other interesting kind of RNN such as LSTM will be covered later in another post.

Recurrent Neural Network - RNN

An RNN takes as input a sequence \((x_{1}, x_{2}, …, x_{T})\) and outputs a sequence of hidden states (or representations) \((h_{1}, h_{2}, …, h_{T})\) which are often called annnotations. At each time step \(t\), the RNN unit takes as input \(x_{t}\) and the previous hidden state \(h_{t-1}\) to compute the hidden state as follows:

The hidden representations are used then to make a prediction. This makes RNNs relays only on past information and no future information are used:

RNNs can process input of any size and the model size remains independent of the input size as the weight are share across time. But they are computationally slow and they have difficulties to learn long term dependancies. So, they suffers from vanishing/exploding gradient because of multiplicative gradient that can be exponentially decreasing or increasing with respect to the number of layers.

Gated Recurrent Unit - GRU

GRU are an RNN that deals with vanishing gradient problem through using specific gates. In GRU we have two gates, reset (relevance) gate and update gate, defined as follows:

\begin{equation}r_{t} = \sigma (U_{r} x_{t} + W_{r} h_{t-1} + b_{r})\end{equation}

\begin{equation}z_{t} = \sigma (U_{z} x_{t} + W_{z} h_{t-1} + b_{z})\end{equation}

The reset gate determines how much information should be discarder from previous time steps stored in \(h_{t-1}\).

So we compute a candidate hidden state using this reset gates as follows :

This candidates hidden states is used along with the previous hidden states to obtain the final hidden state by linearly interpolating them using the update gates:

Bidirectional GRU

Bidirectional GRU is GRU variant that consider hidden state from previous and future steps to predict the current hidden state:

Bidirectional GRU can be implemented easily using Pytorch

import torch
gru = nn.GRU(
            input_size=d, # The size of the input
            hidden_size=n_units, # The size of the hidden state 
            num_layers=1,
            bias=True,
            batch_first=True,
            bidirectional=True,
        )

The bidirectional argument is set to True to specify a biderctional GRU. If it is set to False then we have normal GRU

Self Attention

The attention mechanism was developed in encoder-decoder architecture for NMT context and then used for other context. The attention also used in encoder only settings and it is called self or inner attention. Self attention is the core component of transformers.

The idea behind attention is to use for prediction a weighted sum, where all the weights are determined using trainable parameters, of the all hidden states \((h_{1}, h_{2}, …, h_{T})\) rather than using the last hidden state which is prone to information loss.

The hidden states are passed to a dense layer (eq 3). The alignment coefficient are computed by comparing the output of the dense layer with a trainable context vector u and normalized using Softmax (eq 4). The attentional vector is computed then using a weighted sum of hidden states with alignment coefficient as weights (eq 5).

\begin{equation}u_t = tanh(W h_t) \end{equation} \begin{equation}\alpha_t = \frac{exp(u_{t}^{T}u)}{\sum_{t’=1}^{T}exp(u_{t’}^{T}u)} \end{equation} \begin{equation}s = \sum_{t=1}^{T}\alpha_{t}h_{t}\end{equation}

The self attention can be implemented as follows using Pytorch

import torch
from torch import nn
from torch.utils.data import DataLoader


class AttentionWithContext(nn.Module):
    """
    Class implementing self attention mechanism:
    u_t = tanh(W . x_t)
    a_t = (u_t^T . u)/sum(u_i^T . u)
    s = sum(a_t * x_t)
    """
  
    def __init__(self, input_shape, return_coefficients=False, bias=True):
        super(AttentionWithContext, self).__init__()
        self.return_coefficients = return_coefficients

        # Dense Layer W
        self.W = nn.Linear(input_shape, input_shape, bias=bias)
        self.tanh = nn.Tanh()
        # Trainable context vector u 
        self.u = nn.Linear(input_shape, 1, bias=False)

        self.init_weights()

    def init_weights(self):
        initrange = 0.1
        self.W.weight.data.uniform_(-initrange, initrange)
        self.W.bias.data.uniform_(-initrange, initrange)
        self.u.weight.data.uniform_(-initrange, initrange)

   
    def forward(self, h):
        # compute uit = tanh(W . h)  where h are the hidden states
        uit = self.W(h)  
        uit = self.tanh(uit) #(N, L, d) --> (N, L, d)
        
        # compute the attention coefficient alphas : u_t^T . u
        ait = self.u(uit) #(N, L, 1) --> (N, L, 1)
        #Normalizing with softmax
        a = torch.exp(ait)
        # in some cases especially in the early stages of training the sum may be almost zero
        # and this results in NaN's. A workaround is to add a very small positive number ε to the sum.
        eps = 1e-9
        a = a / (torch.sum(a, axis=1, keepdim=True) + eps) #(N, L, 1) --> (N, L, 1)
        
        # Compute the attentional vector : s = sum(a_t * x_t)
        weighted_input = torch.sum(torch.mul(a, h), axis=1)
        
        if self.return_coefficients:
            return (
                weighted_input,
                a,
            )  ### [attentional vector, coefficients] ### use torch.sum to compute s
        else:
            return weighted_input  ### attentional vector only ###

The attention mechanism can provide similar summation weights for all the hidden states. A penalization term was proposed by [9] to encourage the diversity of summation weight vectors. The penalization term is as follows:

where F refer to the frobenius norm of a matrix \({\lVert A \rVert}_{F}^{2} = \sum_{i=1}^{n}\sum_{j=1}^{n} a_{ij}^{2}\)

def Frobenius(mat):
    size = mat.size()
    if len(size) == 3:  # batched matrix
        ret = torch.sum(torch.sum((mat ** 2), 2), 1).squeeze() + 1e-10
        return torch.sum(ret) / size[0]
    else:
        raise Exception('matrix for computing Frobenius norm should be with 3 dims')

# Create a batch of indentity matrices of size L where L represent tha length of the sequence.
I = Variable(torch.zeros(batch_size, L, L))
for i in range(batch_size):
    for j in range(L):
        I.data[i][j][j] = 1
#Computing the penalization term
A_T = torch.transpose(A, 1, 2).contiguous()
P = Frobenius(torch.bmm(A, A_T) - I[:A.size(0)]) 

Hierarchical Attention Network - HAN

Hierarchical Attention Network is an interesting architecture which use self attention and was proposed by [5]. The architecture contains many level and each level is RNNs followed by self attention layer. Which makes this architecture suitable for data that has a hierarchy e.g word —> sentence —> document. First, a sentence encoder produces an embedding for each sentence from word embeddings; then a document encoder produces the document embedding vector from the sentence embeddings previously produced. Each encoder is a Bidir GRU followed by a self attention.

HAN makes sense for two reason: First it matches the natural hierarchy of a document; second; it allows the model to first determine which words are important in each sentence and then which sentence are important overall.

By being able to re-weight the word attentional coefficients by the sentence attentional coefficients the model captures the fact that a word may be very important in a sentence but it’s less important in another sentence.

class AttentionBiGRU(nn.Module):
    def __init__(self, input_shape, n_units, index_to_word, dropout=0):
        super(AttentionBiGRU, self).__init__()
        self.embedding = nn.Embedding(
            len(index_to_word) + 2, # vocab size
            d,  # dimensionality of embedding space
            padding_idx=0,
        )
        self.dropout = nn.Dropout(drop_rate)
        self.gru = nn.GRU(
            input_size=d,
            hidden_size=n_units,
            num_layers=1, 
            bias=True,
            batch_first=True,
            bidirectional=True,
        )
        self.attention = AttentionWithContext(
            2 * n_units,  # the input shape for the attention layer
            return_coefficients=True,
        )

    def forward(self, sent_ints):
        sent_wv = self.embedding(sent_ints)
        sent_wv_dr = self.dropout(sent_wv)
        sent_wa, _ = self.gru(sent_wv_dr)  # GRU layer
        sent_att_vec, word_att_coeffs = self.attention(
            sent_wa
        )  # attentional vector for the sent
        sent_att_vec_dr = self.dropout(sent_att_vec)
        return sent_att_vec_dr, word_att_coeffs


class TimeDistributed(nn.Module):
    def __init__(self, module, batch_first=False):
        super(TimeDistributed, self).__init__()
        self.module = module
        self.batch_first = batch_first

    def forward(self, x):
        if len(x.size()) <= 2:
            return self.module(x)
        # Squash samples and timesteps into a single axis
        x_reshape = x.contiguous().view(
            -1, x.size(-1)
        )  # (samples * timesteps, input_size) (224, 30)
        sent_att_vec_dr, word_att_coeffs = self.module(x_reshape)
        # We have to reshape the output
        if self.batch_first:
            sent_att_vec_dr = sent_att_vec_dr.contiguous().view(
                x.size(0), -1, sent_att_vec_dr.size(-1)
            )  # (samples, timesteps, output_size)
            word_att_coeffs = word_att_coeffs.contiguous().view(
                x.size(0), -1, word_att_coeffs.size(-1)
            )  # (samples, timesteps, output_size)
        else:
            sent_att_vec_dr = sent_att_vec_dr.view(
                -1, x.size(1), sent_att_vec_dr.size(-1)
            )  # (timesteps, samples, output_size)
            word_att_coeffs = word_att_coeffs.view(
                -1, x.size(1), word_att_coeffs.size(-1)
            )  # (timesteps, samples, output_size)
        return sent_att_vec_dr, word_att_coeffs


class HAN(nn.Module):
    def __init__(self, input_shape, n_units, index_to_word, dropout=0):
        super(HAN, self).__init__()
        self.encoder = AttentionBiGRU(input_shape, n_units, index_to_word, dropout)
        self.timeDistributed = TimeDistributed(self.encoder, True)
        self.dropout = nn.Dropout(drop_rate)
        self.gru = nn.GRU(
            input_size=2 * n_units,  # the input shape of GRU layer
            hidden_size=n_units,
            num_layers=1,
            bias=True,
            batch_first=True,
            bidirectional=True,
        )
        self.attention = AttentionWithContext(
            2
            * n_units,  # the input shape of between-sentence attention layer
            return_coefficients=True,
        )
        self.lin_out = nn.Linear(
            2 * n_units, 1  # the input size of the last linear layer
        )
        self.preds = nn.Sigmoid()

    def forward(self, doc_ints):
        sent_att_vecs_dr, word_att_coeffs = self.timeDistributed(
            doc_ints
        )  # get sentence representation
        doc_sa, _ = self.gru(sent_att_vecs_dr)
        doc_att_vec, sent_att_coeffs = self.attention(doc_sa)
        doc_att_vec_dr = self.dropout(doc_att_vec)
        doc_att_vec_dr = self.lin_out(doc_att_vec_dr)
        return self.preds(doc_att_vec_dr), word_att_coeffs, sent_att_coeffs

While HAN is an interesting architecture, it has a major limitation. For a given sentence, the embedding is produced in isolation. This means, it ignores the other sentences. So, If redundant parts exist in the document; the model can spend the attention budget on them and neglects the others aspects.

Dataset

The dataset considered in this application is IMDB review dataset. It contains reviews about movies and their classes, positive or negative. The goal is to classify these reviews into positive or negative.

Each review is converted to an array of integers that has a size of \((1, doc_size, sent_size)\). the doc{_}size specifies the maximum number of allowed sentences per document and sent{_}size the specifies the maximum number of allowed words per sentence. Smaller sentences are padded by a special padding token and smaller documents are padded with sentences containing only a special padding token. Longer documents or sentences are truncated.

The mapping of a word to an integers is done by creating a vocabulary dictionary from the training set where each word has an integer value. The most frequent word has a value of 2. 0 and 1 are reserved for special token and out of vocabulary token.

An example of a sentence and its mapping to an array of integers:

Sentence:

"There 's a sign on The Lost Highway that says : OOV SPOILERS OOV ( but you already knew that , did n't you ? )"

Corresponding array:

array([  130,    14,     6,  1991,    28,    22,  2746, 17943,    13,
         564,    85,     1,  3225,     1,    25,    26,    29,   488,
         697,    13,     3,    84,    27,    29,    45,    24,     0,
           0,     0,     0])

We have some 0 in the end of the array as the choosen sent_size is 30 and the sentence needs to be padded because it contains only 26 words.

Training

The training consists of minimizing a loss function. As we are dealing with classification problem, the loss used is Binary Cross Entropy (BCE):

model = HAN(input_size, n_units, index_to_word).to(device)
model = model.double()
lr = 0.001  # learning rate
criterion = (
    nn.BCELoss()
)  # Binary cross entropy from torch.nn: https://pytorch.org/docs/stable/nn.html#loss-functions
optimizer = torch.optim.Adam(model.parameters(), lr=lr)  # Adam optimizer

def train(
    x_train=my_docs_array_train,
    y_train=my_labels_array_train,
    x_test=my_docs_array_test,
    y_test=my_labels_array_test,
    word_dict=index_to_word,
    batch_size=batch_size,
):

    train_data = get_loader(x_train, y_train, batch_size)
    test_data = get_loader(my_docs_array_test, my_labels_array_test, batch_size)

    best_validation_acc = 0.0
    p = 0  # patience

    for epoch in range(1, nb_epochs + 1):
        losses = []
        accuracies = []
        with tqdm(train_data, unit="batch") as tepoch:
            for idx, data in enumerate(tepoch):
                tepoch.set_description(f"Epoch {epoch}")
                model.train()
                optimizer.zero_grad()
                input = data["document"].to(device)
                label = data["label"].to(device)
                label = label.double()
                out = model.forward(input)
                output = out[0][:, -1]
                loss = criterion(output, label)    # compute the loss
                loss.backward()
                torch.nn.utils.clip_grad_norm_(
                    model.parameters(), 0.5
                )  # Clipping to prevent exploding gradient
                optimizer.step()

                losses.append(loss.item())
                accuracy = torch.sum(torch.round(output) == label).item() / batch_size
                accuracies.append(accuracy)
                tepoch.set_postfix(
                    loss=sum(losses) / len(losses),
                    accuracy=100.0 * sum(accuracies) / len(accuracies),
                )

        train_acc = evaluate_accuracy(train_data, False)
        test_acc = evaluate_accuracy(test_data, False)
        print(
            "===> Epoch {} Complete: Avg. Loss: {:.4f}, Validation Accuracy: {:3.2f}%".format(
                epoch, sum(losses) / len(losses), 100.0 * test_acc
            )
        )
        if test_acc >= best_validation_acc:
            best_validation_acc = test_acc
            print("Validation accuracy improved, saving model...")
            torch.save(model.state_dict(), "./best_model.pt")
            p = 0
            print()
        else:
            p += 1
            if p == my_patience:
                print(
                    "Validation accuracy did not improve for {} epochs, stopping training...".format(
                        my_patience
                    )
                )
    print("Loading best checkpoint...")
    model.load_state_dict(torch.load("./best_model.pt"))
    model.eval()
    print("done.")


train()

Analysis

Using the mean of the hidden state of GRU instead attention will lead to treating the hidden state in a sequence equally. This lead to the same contribution of irrelevant elements to the prediction with the same importance as relevant element. Which can be subject to poor performances. So, the advantage of the attention mechanism is to have weights releaving the importance of each hidden state; and so each element, in the sequence (words in case of sentence, and sentences in case of a document).

The following plots, shows the attention coefficient per words and per sentences of positive and negative reviews from the test set. The more the sentence or the word is red, the more the attention of this element is high. <|OOV|> means Out Of Vocabulary, is a token to replace words that are not in the training vocabulary.

From the plots, we can see that words and sentence that relevant and more linked to a positive or negative reviews have a high attention coefficient regarding the other words. For example in the positive review, we have words such as “Brilliant”, “Master Piece”, “Great” which are positive and have a high attention. Same remark for sentences such as “:) First of all, Mulholland Drive is downright brilliant.” or “A masterpiece”. In the negative review we have words or sentences with a negative tonality have a high attention; such as “confused” “suspenful”.

Depth estimation is a task consisting of estimating the distance of each pixel realtive to the camera to result in a depth map. Using the depth map with some camera parameters, explained later, we can map an image $ I \in\mathbb{R}^{M\times N \times 3}$ to its 3D point cloud $X \in\mathbb{R}^{M \times N \times 3}$. This means, we can map each pixel in the 2D image to its corresponding point in the 3D space.

A camera projects a 3D point in the real worl to a 2D point on the image. This transformation can be modeled using matrix multiplication using the intersic camera parameters and extrinsic camera parameters. The camera extrinsic matrix (parameters), denoted $E$, is used to map a 3D point in the real world reference to a 3D point cloud in the camera reference. The camera intrinsic matrix (parameters), denoted $K$ is, used to map a 3D camera centered point to a 2D point on the image plane.

The matrix $K$ and $E$ are defined as follows:

So, given the intrinsic matrix $K$ and the depth map $D$, and using the pinhole camera model, we can map each pixel $p_{i}=(x_{i}, y_{i})^{T}$ in the 2D image plane to its corresponding the 3D point in the camera referential $P_{c, i} = (X_{c, i}, Y_{c, i}, Z_{c, i})^{T}$ using the schema below:

We will be interested in only on mapping the image pixel to its 3D point in the camera referential.\

To map to the 3D real world space, we can use the relationshape between a point in the camera referential $P_{c}$ and its corresponding point in the real world referential $P_{r}$, which is:

The following python function implements the pinhole model to generate 3D point in the camera referential using depth map and the intrinsic matrix.

def generate_point_cloud(depth_map, intrinsic_matrix, depth_info, rgb_image=None):
    # Image dimensions
    H, W = depth_map.shape
    # Intrinsic parameters
    fx, fy = intrinsic_matrix[0, 0], intrinsic_matrix[1, 1]
    cx, cy = intrinsic_matrix[0, 2], intrinsic_matrix[1, 2]
    depth_max = depth_info[3] # If depth info are provided
    depth_min = depth_info[0]
    
    # Create a grid of pixel coordinates
    u, v = np.meshgrid(np.arange(W), np.arange(H))
    # Convert to normalized camera coordinates
    x_norm = (u - cx) / fx
    y_norm = (v - cy) / fy
    # Back-project to 3D camera coordinates
    z_cam = depth_map
    x_cam = z_cam * x_norm
    y_cam = z_cam * y_norm
    # Stack into a 3D array (camera coordinates)
    points_cam = np.stack((x_cam, z_cam, -1*y_cam), axis=-1).reshape(-1, 3)
    if rgb_image is not None:
      colors = rgb_image[v, u] / 255.0
    return points_cam, colors

NUY Depth V2 dataset

NUY Depth V2 is a large dataset that contains indoor scenes. The following presents some scene examples from the data and different views of the constructed 3D point of these scenes.

Approach and Architecure

The objective is to map the image $ I \in\mathbb{R}^{M\times N \times 3}$ to its 3D point cloud $X \in\mathbb{R}^{M\times N \times 3}$ . we can think about a naive or simple autoencoder decoder architecture such as UNET. Where the econder learn the image representation and the decoder map the representation to 3D point cloud. UNET architecture has been tested but it did not give satisfying results.

Our focus will be focus on the camera referential. If we consider the Pinhole camera model, we have the cordinaates $X$, $Y$ and $Z$ depend on the depth map as shown in equation (1).

The idea is instead to learn a mapping function of the image to the 3D point cloud, we will think of a design that estimates the depth map and the camera parameters.

We will follow an approach proposed by Yin et al [6] that has two stages:

1. The first stage is a training a neural network to learn estimating the depth map that will be used to reconstruct a 3D point cloud using standard camera parameters. Not using the correct parameters, precisely, the focal length will result in a distorded point cloud even the global shape is preserved.

2. The second stage then is composed from two networks each take as input a distorded point cloud and estimate either a focal scale or depth shift to restore the distorded 3D point.

First Stage - Depth Estimation

For depth map estimation eigen et al [1] introduced a CNN model levereging on multi scale network.This approach involves training a coarse scale network to predict the depth map at global level which is subsequently refined by a secondary network to refine the local regions. Using a multi-scale architecture proven to be effective, Xian et al [7] proposed a multi scale network that use a feature fusion module to fuse features from the encoder and decoder at different scales to obtain finer prediction. Additionnaly to using a multiscale architecture, in Big to Small model [2], the author proposed, under the assumption of locar planar, a local planar guidance module to guide features to the final depth. Unlike other methods which use only skip connection from encoder stage and upsampling to recover the final depth.

We will use BTS model for this task. BTS has an encoder-decoder architecture. The encoder outputs a dense feature map of $H/8$ resolution. The decoding phase consists of $4$ stages. Each stage $k$, (with $k \in {8, 4, 2, 1}$) takes a dense feature map of $H/k$ resolution and apply two operations:

1. We apply an up-convolution operation to have a feature map of resolution of $2H/k$.
2. We apply local planar guidance to produce a coarse depth map $\tilde{c}^{k\times k}$ of resolution $H$ which downsampled using linear interpolation to a resolution of $2H/k$ The output of these two operations are element-wise multiplied and passed through a convolution operation to have the input feature map of resolution $2H/k$ for the next stage.

To have the estimated depth map $d$, all the coersed depth map produced by the local planar guidance are used together as follows:

the LPG module: Given a feature map having a spatial resolution $H/k$, it estimates $4D$ plane coefficient for each spatial cell to reconstruct a coarse depth that fit a locally defined $k\times k$ patch on the full resolution. The LPG uses ray-plane intersection to convert each estimated 4D plane coefficient to $k\times k$ local depth cues on the full resolution:

where $n = (n_{1}, n_{2}, n_{3}, n_{4})$ are the estimated 4D parameters and $(u_{i}, v_{i})$ \are $k\times k$ patch wise normalized coordinate of pixel $i$

The $n$ parameters are the plane parameters where $(n_{1}, n_{2}, n_{3})$ is the normal vector and $n_{4}$ is the distance from the origine to the plane. To estimate these parameters, they use the fact that a normal vector can be computed using two angles, polars and azimuthal, using the the following formulas: (more details in the Appendix)

To estimates these three parameters at the scale $H/k$, the LPG takes as input the feature of the previous fusion module, i.e feature at scale $H/2k$, and pass them through a series of $1\times1$ convolution to reduce the number of channels by a factor of $2$ until it reaches $3$. So the final convolution layer of the LPG estimates $\theta$, $\phi$ and $d$. More details about the computation are in the implementation of this module. Thus using the LPG, we have an estimation of depth map at different scales. The lower scales learns the global shapes and the higher scales learns local details. The final depth is estimated through a convolution that takes all the depth map at each scale

focal length and depth shift estimation

The input distorded point cloud are created as follows:

1. for the first network, we shift the depth map by a value drawn from a uniform distribution. The shift will result in a distorded shape as it will affect non uniformly, the X, Y, and Z. So, the goal of the first network is to estimate the depth shift value.
2. For the second network, the distortion is created by scaling the focal length by a coefficient drawn from a uniform distribution. This scaling will affect only X and Y and will results in points far from each other or closer to each other.

to estimate the depth shift and focal length shift. As in [6] we have neural network based on PointNet to predict the depth shift or focal length shift given a input of distorted point cloud:

where \(\Delta^{*}_d\) is drawn from a uniform distribution \(\text{Uniform}(-0.25, 0.8)\) during training

where \(\alpha^{*}\) is drawn from a uniform distribution \(\text{Uniform}(0.6, 1.25)\) during training

The network used for focal scale estimation and depth shift estimation is a PoinNet [9] applied for regression task. The network takes as input a 3D point cloud (B, N, 3) and apply an input transformation block followed by a 1D conv layer to extract features. A feature transformation block is applied on the extracted and followed by a series of 1D conv operation to result in a vector of global feature using a max aggregation. The max pooling is used to have invariance w.t.r point cloud order. The input transformation block and feature transformation block contains a series of conv and linear operations and use relu as activation function and max pooling to aggregates information. The transormation block is used to transform the point cloud to a canonical form to have invariance w.r.t to transformation.

Loss and Metrics

The depth estimation, focal length scale estimation or depth shift estimation are regression task. For the loss will be based of L1 or MSE loss In depth estimation many losses function have been proposed in the literature such as Huber Loss, silog loss, ordinal regression loss. In our case, we will use scale invariance log which computes the error between the ground truth and the prediction without taking into account the scale discrepency.So, it consider only the relative error between the values.

for evaluation we consider the following metrics used in the literature:

• Accuracy under a threshold $\delta$ % of \(p : \delta = max(\frac{\hat{d}_p}{d_p}, \frac{d_p}{\hat{d}_p}) < threshold\)

• Abs. Rel.: Mean Absolute Value of the Relative Error. \(\frac{1}{T} \sum_{p \in T} \left| \frac{d_p - \hat{d}_p}{d_p} \right|\)

Data Preparation

The NUY depth dataset is handled respectively by NUYDepth class that stores samples of images, depth map for training depth map network and point cloud with depth shift or focal scale for PointNet networks. The class contains the method

def _generate_point_cloud():

which generate a point cloud using Pinhol camera model.

The image and depth map are resized to (256, 256) for computational reasons in the case we are training for depth map estimation. For depth shift or focal scale estimation, we keep the full resolution of the depth map to construct the 3D point cloud, but we use only a sample of 4096 points for for computational reasons.

The images are normalized using mean and std. We use data augmentation, flipping and brightness adjusting as we are using only a subset of 1300 image from the data The original data has 490GB.

Implementation

Here the implementation of the models, datasets and training loops. The results discussion are right after the following cells.

import torch
import torch.nn as nn
import torch.nn.functional as torch_nn_func
import math

from collections import namedtuple

# This sets the batch norm layers in pytorch as if {'is_training': False, 'scale': True} in tensorflow
def bn_init_as_tf(m):
    if isinstance(m, nn.BatchNorm2d):
        m.track_running_stats = True  # These two lines enable using stats (moving mean and var) loaded from pretrained model
        m.eval()                      # or zero mean and variance of one if the batch norm layer has no pretrained values
        m.affine = True
        m.requires_grad = True


def weights_init_xavier(m):
    if isinstance(m, nn.Conv2d):
        torch.nn.init.xavier_uniform_(m.weight)
        if m.bias is not None:
            torch.nn.init.zeros_(m.bias)
            

class silog_loss(nn.Module):
    def __init__(self, variance_focus):
        super(silog_loss, self).__init__()
        self.variance_focus = variance_focus

    def forward(self, depth_est, depth_gt, mask):
        d = torch.log(depth_est[mask]) - torch.log(depth_gt[mask])
        return torch.sqrt((d ** 2).mean() - self.variance_focus * (d.mean() ** 2)) * 10.0


class atrous_conv(nn.Sequential):
    def __init__(self, in_channels, out_channels, dilation, apply_bn_first=True):
        super(atrous_conv, self).__init__()
        self.atrous_conv = torch.nn.Sequential()
        if apply_bn_first:
            self.atrous_conv.add_module('first_bn', nn.BatchNorm2d(in_channels, momentum=0.01, affine=True, track_running_stats=True, eps=1.1e-5))
        
        self.atrous_conv.add_module('aconv_sequence', nn.Sequential(nn.ReLU(),
                                                                    nn.Conv2d(in_channels=in_channels, out_channels=out_channels*2, bias=False, kernel_size=1, stride=1, padding=0),
                                                                    nn.BatchNorm2d(out_channels*2, momentum=0.01, affine=True, track_running_stats=True),
                                                                    nn.ReLU(),
                                                                    nn.Conv2d(in_channels=out_channels * 2, out_channels=out_channels, bias=False, kernel_size=3, stride=1,
                                                                              padding=(dilation, dilation), dilation=dilation)))

    def forward(self, x):
        return self.atrous_conv.forward(x)
    

class upconv(nn.Module):
    def __init__(self, in_channels, out_channels, ratio=2):
        super(upconv, self).__init__()
        self.elu = nn.ELU()
        self.conv = nn.Conv2d(in_channels=in_channels, out_channels=out_channels, bias=False, kernel_size=3, stride=1, padding=1)
        self.ratio = ratio
        
    def forward(self, x):
        up_x = torch_nn_func.interpolate(x, scale_factor=self.ratio, mode='nearest')
        out = self.conv(up_x)
        out = self.elu(out)
        return out


class reduction_1x1(nn.Sequential):
    def __init__(self, num_in_filters, num_out_filters, max_depth, is_final=False):
        super(reduction_1x1, self).__init__()        
        self.max_depth = max_depth
        self.is_final = is_final
        self.sigmoid = nn.Sigmoid()
        self.reduc = torch.nn.Sequential()
        
        while num_out_filters >= 4:
            if num_out_filters < 8:
                if self.is_final:
                    self.reduc.add_module('final', torch.nn.Sequential(nn.Conv2d(num_in_filters, out_channels=1, bias=False,
                                                                                 kernel_size=1, stride=1, padding=0),
                                                                       nn.Sigmoid()))
                else:
                    self.reduc.add_module('plane_params', torch.nn.Conv2d(num_in_filters, out_channels=3, bias=False,
                                                                          kernel_size=1, stride=1, padding=0))
                break
            else:
                self.reduc.add_module('inter_{}_{}'.format(num_in_filters, num_out_filters),
                                      torch.nn.Sequential(nn.Conv2d(in_channels=num_in_filters, out_channels=num_out_filters,
                                                                    bias=False, kernel_size=1, stride=1, padding=0),
                                                          nn.ELU()))

            num_in_filters = num_out_filters
            num_out_filters = num_out_filters // 2
    
    def forward(self, net):
        net = self.reduc.forward(net)
        if not self.is_final:
            theta = self.sigmoid(net[:, 0, :, :]) * math.pi / 3
            phi = self.sigmoid(net[:, 1, :, :]) * math.pi * 2
            dist = self.sigmoid(net[:, 2, :, :]) * self.max_depth
            n1 = torch.mul(torch.sin(theta), torch.cos(phi)).unsqueeze(1)
            n2 = torch.mul(torch.sin(theta), torch.sin(phi)).unsqueeze(1)
            n3 = torch.cos(theta).unsqueeze(1)
            n4 = dist.unsqueeze(1)
            net = torch.cat([n1, n2, n3, n4], dim=1)
        
        return net

class local_planar_guidance(nn.Module):
    def __init__(self, upratio):
        super(local_planar_guidance, self).__init__()
        self.upratio = upratio
        self.u = torch.arange(self.upratio).reshape([1, 1, self.upratio]).float()
        self.v = torch.arange(int(self.upratio)).reshape([1, self.upratio, 1]).float()
        self.upratio = float(upratio)

    def forward(self, plane_eq, focal):
        plane_eq_expanded = torch.repeat_interleave(plane_eq, int(self.upratio), 2)
        plane_eq_expanded = torch.repeat_interleave(plane_eq_expanded, int(self.upratio), 3)
        n1 = plane_eq_expanded[:, 0, :, :]
        n2 = plane_eq_expanded[:, 1, :, :]
        n3 = plane_eq_expanded[:, 2, :, :]
        n4 = plane_eq_expanded[:, 3, :, :]
        
        u = self.u.repeat(plane_eq.size(0), plane_eq.size(2) * int(self.upratio), plane_eq.size(3))#.cuda()
        u = (u - (self.upratio - 1) * 0.5) / self.upratio
        
        v = self.v.repeat(plane_eq.size(0), plane_eq.size(2), plane_eq.size(3) * int(self.upratio))#.cuda()
        v = (v - (self.upratio - 1) * 0.5) / self.upratio

        return n4 / (n1 * u + n2 * v + n3)

class bts(nn.Module):
    def __init__(self, feat_out_channels, num_features=512):
        super(bts, self).__init__()
        self.max_depth = 1.
        self.upconv5    = upconv(feat_out_channels[4], num_features)
        self.bn5        = nn.BatchNorm2d(num_features, momentum=0.01, affine=True, eps=1.1e-5)
        
        self.conv5      = torch.nn.Sequential(nn.Conv2d(num_features + feat_out_channels[3], num_features, 3, 1, 1, bias=False),
                                              nn.ELU())
        self.upconv4    = upconv(num_features, num_features // 2)
        self.bn4        = nn.BatchNorm2d(num_features // 2, momentum=0.01, affine=True, eps=1.1e-5)
        self.conv4      = torch.nn.Sequential(nn.Conv2d(num_features // 2 + feat_out_channels[2], num_features // 2, 3, 1, 1, bias=False),
                                              nn.ELU())
        self.bn4_2      = nn.BatchNorm2d(num_features // 2, momentum=0.01, affine=True, eps=1.1e-5)
        
        self.daspp_3    = atrous_conv(num_features // 2, num_features // 4, 3, apply_bn_first=False)
        self.daspp_6    = atrous_conv(num_features // 2 + num_features // 4 + feat_out_channels[2], num_features // 4, 6)
        self.daspp_12   = atrous_conv(num_features + feat_out_channels[2], num_features // 4, 12)
        self.daspp_18   = atrous_conv(num_features + num_features // 4 + feat_out_channels[2], num_features // 4, 18)
        self.daspp_24   = atrous_conv(num_features + num_features // 2 + feat_out_channels[2], num_features // 4, 24)
        self.daspp_conv = torch.nn.Sequential(nn.Conv2d(num_features + num_features // 2 + num_features // 4, num_features // 4, 3, 1, 1, bias=False),
                                              nn.ELU())
        self.reduc8x8   = reduction_1x1(num_features // 4, num_features // 4, self.max_depth)
        self.lpg8x8     = local_planar_guidance(8)
        
        self.upconv3    = upconv(num_features // 4, num_features // 4)
        self.bn3        = nn.BatchNorm2d(num_features // 4, momentum=0.01, affine=True, eps=1.1e-5)
        self.conv3      = torch.nn.Sequential(nn.Conv2d(num_features // 4 + feat_out_channels[1] + 1, num_features // 4, 3, 1, 1, bias=False),
                                              nn.ELU())
        self.reduc4x4   = reduction_1x1(num_features // 4, num_features // 8, self.max_depth)
        self.lpg4x4     = local_planar_guidance(4)
        
        self.upconv2    = upconv(num_features // 4, num_features // 8)
        self.bn2        = nn.BatchNorm2d(num_features // 8, momentum=0.01, affine=True, eps=1.1e-5)
        self.conv2      = torch.nn.Sequential(nn.Conv2d(num_features // 8 + feat_out_channels[0] + 1, num_features // 8, 3, 1, 1, bias=False),
                                              nn.ELU())
        
        self.reduc2x2   = reduction_1x1(num_features // 8, num_features // 16, self.max_depth)
        self.lpg2x2     = local_planar_guidance(2)
        
        self.upconv1    = upconv(num_features // 8, num_features // 16)
        self.reduc1x1   = reduction_1x1(num_features // 16, num_features // 32, self.max_depth, is_final=True)
        self.conv1      = torch.nn.Sequential(nn.Conv2d(num_features // 16 + 4, num_features // 16, 3, 1, 1, bias=False),
                                              nn.ELU())
        self.get_depth  = torch.nn.Sequential(nn.Conv2d(num_features // 16, 1, 3, 1, 1, bias=False),
                                              nn.Sigmoid())

    def forward(self, features, focal):
        skip0, skip1, skip2, skip3 = features[0], features[1], features[2], features[3]
        dense_features = torch.nn.ReLU()(features[4])
        upconv5 = self.upconv5(dense_features) # H/16
        upconv5 = self.bn5(upconv5)
        concat5 = torch.cat([upconv5, skip3], dim=1)
        iconv5 = self.conv5(concat5)
        
        upconv4 = self.upconv4(iconv5) # H/8
        upconv4 = self.bn4(upconv4)
        concat4 = torch.cat([upconv4, skip2], dim=1)
        iconv4 = self.conv4(concat4)
        iconv4 = self.bn4_2(iconv4)
        
        daspp_3 = self.daspp_3(iconv4)
        concat4_2 = torch.cat([concat4, daspp_3], dim=1)
        daspp_6 = self.daspp_6(concat4_2)
        concat4_3 = torch.cat([concat4_2, daspp_6], dim=1)
        daspp_12 = self.daspp_12(concat4_3)
        concat4_4 = torch.cat([concat4_3, daspp_12], dim=1)
        daspp_18 = self.daspp_18(concat4_4)
        concat4_5 = torch.cat([concat4_4, daspp_18], dim=1)
        daspp_24 = self.daspp_24(concat4_5)
        concat4_daspp = torch.cat([iconv4, daspp_3, daspp_6, daspp_12, daspp_18, daspp_24], dim=1)
        daspp_feat = self.daspp_conv(concat4_daspp)
        
        reduc8x8 = self.reduc8x8(daspp_feat)
        plane_normal_8x8 = reduc8x8[:, :3, :, :]
        plane_normal_8x8 = torch_nn_func.normalize(plane_normal_8x8, 2, 1)
        plane_dist_8x8 = reduc8x8[:, 3, :, :]
        plane_eq_8x8 = torch.cat([plane_normal_8x8, plane_dist_8x8.unsqueeze(1)], 1)
        depth_8x8 = self.lpg8x8(plane_eq_8x8, focal)
        depth_8x8_scaled = depth_8x8.unsqueeze(1) / self.max_depth
        depth_8x8_scaled_ds = torch_nn_func.interpolate(depth_8x8_scaled, scale_factor=0.25, mode='nearest')
        
        upconv3 = self.upconv3(daspp_feat) # H/4
        upconv3 = self.bn3(upconv3)
        concat3 = torch.cat([upconv3, skip1, depth_8x8_scaled_ds], dim=1)
        iconv3 = self.conv3(concat3)
        
        reduc4x4 = self.reduc4x4(iconv3)
        plane_normal_4x4 = reduc4x4[:, :3, :, :]
        plane_normal_4x4 = torch_nn_func.normalize(plane_normal_4x4, 2, 1)
        plane_dist_4x4 = reduc4x4[:, 3, :, :]
        plane_eq_4x4 = torch.cat([plane_normal_4x4, plane_dist_4x4.unsqueeze(1)], 1)
        depth_4x4 = self.lpg4x4(plane_eq_4x4, focal)
        depth_4x4_scaled = depth_4x4.unsqueeze(1) / self.max_depth
        depth_4x4_scaled_ds = torch_nn_func.interpolate(depth_4x4_scaled, scale_factor=0.5, mode='nearest')
        
        upconv2 = self.upconv2(iconv3) # H/2
        upconv2 = self.bn2(upconv2)
        concat2 = torch.cat([upconv2, skip0, depth_4x4_scaled_ds], dim=1)
        iconv2 = self.conv2(concat2)
        
        reduc2x2 = self.reduc2x2(iconv2)
        plane_normal_2x2 = reduc2x2[:, :3, :, :]
        plane_normal_2x2 = torch_nn_func.normalize(plane_normal_2x2, 2, 1)
        plane_dist_2x2 = reduc2x2[:, 3, :, :]
        plane_eq_2x2 = torch.cat([plane_normal_2x2, plane_dist_2x2.unsqueeze(1)], 1)
        depth_2x2 = self.lpg2x2(plane_eq_2x2, focal)
        depth_2x2_scaled = depth_2x2.unsqueeze(1) / self.max_depth
        
        upconv1 = self.upconv1(iconv2)
        reduc1x1 = self.reduc1x1(upconv1)
        concat1 = torch.cat([upconv1, reduc1x1, depth_2x2_scaled, depth_4x4_scaled, depth_8x8_scaled], dim=1)
        iconv1 = self.conv1(concat1)
        final_depth = self.max_depth * self.get_depth(iconv1)

        
        return final_depth, depth_8x8_scaled, depth_4x4_scaled, depth_2x2_scaled, reduc1x1

class encoder(nn.Module):
    def __init__(self):
        super(encoder, self).__init__()
        import torchvision.models as models
      
        self.base_model = models.resnet50(pretrained=True)
        self.feat_names = ['relu', 'layer1', 'layer2', 'layer3', 'layer4']
        self.feat_out_channels = [64, 256, 512, 1024, 2048]

    def forward(self, x):
        feature = x
        skip_feat = []
        i = 1
        for k, v in self.base_model._modules.items():
            if 'fc' in k or 'avgpool' in k:
                continue
            feature = v(feature)
            if any(x in k for x in self.feat_names):
                skip_feat.append(feature)
            i = i + 1
        return skip_feat
    

class BtsModel(nn.Module):
    def __init__(self):
        super(BtsModel, self).__init__()
        self.encoder = encoder()
        self.decoder = bts(self.encoder.feat_out_channels, 512)

    def forward(self, x, focal=1):
        skip_feat = self.encoder(x)
        return self.decoder(skip_feat, focal)

Depth shift and focal scale network

    
class MEADSTD_TANH_NORM_Loss(nn.Module):
    """
    loss = MAE((d-u)/s - d') + MAE(tanh(0.01*(d-u)/s) - tanh(0.01*d'))
    """
    def __init__(self, valid_threshold=-1e-8, max_threshold=1e8):
        super(MEADSTD_TANH_NORM_Loss, self).__init__()
        self.valid_threshold = valid_threshold
        self.max_threshold = max_threshold
        #self.thres1 = 0.9
 
    def transform(self, gt):
        # Get mean and standard deviation
        data_mean = []
        data_std_dev = []
        for i in range(gt.shape[0]):
            gt_i = gt[i]
            mask = gt_i > 0
            depth_valid = gt_i[mask]
            if depth_valid.shape[0] < 10:
                data_mean.append(torch.tensor(0).cuda())
                data_std_dev.append(torch.tensor(1).cuda())
                continue
            size = depth_valid.shape[0]
            depth_valid_sort, _ = torch.sort(depth_valid, 0)
            depth_valid_mask = depth_valid_sort[int(size*0.1): -int(size*0.1)]
            data_mean.append(depth_valid_mask.mean())
            data_std_dev.append(depth_valid_mask.std())
        data_mean = torch.stack(data_mean, dim=0).cuda()
        data_std_dev = torch.stack(data_std_dev, dim=0).cuda()

        return data_mean, data_std_dev

    def forward(self, pred, gt):
        """
        Calculate loss.
        """
        #gt = torch.nn.functional.interpolate(gt,
        #                                    size=(
        #                                        pred.size()[2], pred.size()[3]),
        #                                    mode='nearest').to('cuda')
        mask = (gt > self.valid_threshold) & (gt < self.max_threshold)   # [b, c, h, w]
        mask_sum = torch.sum(mask, dim=(1, 2, 3))
        # mask invalid batches
        mask_batch = mask_sum > 100
        if True not in mask_batch:
            return torch.tensor(0.0, dtype=torch.float).cuda()
        mask_maskbatch = mask[mask_batch]
        pred_maskbatch = pred[mask_batch]
        gt_maskbatch = gt[mask_batch]

        gt_mean, gt_std = self.transform(gt_maskbatch)
        gt_trans = (gt_maskbatch - gt_mean[:, None, None, None]) / (gt_std[:, None, None, None] + 1e-8)

        B, C, H, W = gt_maskbatch.shape
        loss = 0
        loss_tanh = 0
        for i in range(B):
            mask_i = mask_maskbatch[i, ...]
            pred_depth_i = pred_maskbatch[i, ...][mask_i]
            gt_trans_i = gt_trans[i, ...][mask_i]

            depth_diff = torch.abs(gt_trans_i - pred_depth_i)
            loss += torch.mean(depth_diff)

            tanh_norm_gt = torch.tanh(0.01*gt_trans_i)
            tanh_norm_pred = torch.tanh(0.01*pred_depth_i)
            loss_tanh += torch.mean(torch.abs(tanh_norm_gt - tanh_norm_pred))
        loss_out = loss/B + loss_tanh/B
        return loss_out.float()

class Shift_Loss(nn.Module):
    def __init__(self):
        super(Shift_Loss, self).__init__()
    def forward(pred, gt):
        return torch.abs(pred - gt)


class MAE_error(nn.Module):
    def __init__(self):
        super(MAE_error, self).__init__()

    def forward(self, prediction, gt):
        # prediction = prediction[:, 0:1]
        abs_err = torch.abs(prediction - gt)
        mask = (gt > 1e-3).detach()
        mae_error = torch.mean(abs_err[mask])
        return mae_error

class RelMAE_error(nn.Module):
    def __init__(self):
        super(RelMAE_error, self).__init__()
    def forward(self, prediction, gt):
        # prediction = prediction[:, 0:1]
        #prediction = torch.clamp(prediction, min=1e-4)
        prediction = torch.abs(prediction)
        abs_err = torch.abs(prediction - gt)
        mask = (gt > 1e-3).detach()
        mae_error = torch.mean(abs_err[mask]/gt[mask])
        return mae_error
class Gamma_Metric(nn.Module):
    def __init__(self):
        super(Gamma_Metric, self).__init__()

    def forward(self, prediction, gt):
        # prediction = prediction[:, 0:1]
        mask = (gt > 1e-3).detach()
        prediction = torch.abs(prediction) #torch.clamp(prediction, min=1e-4)
        max_proportion = torch.max(prediction[mask]/gt[mask], gt[mask]/prediction[mask])
        gamma = torch.mean(1.*(max_proportion < 1.25))
        return gamma

Quantitative Results

We plot the evolution of the training and test loss for both models on both dataset. Both models converge on both dataset except for the model adapt on blended MVS were the test loss is fluctuating along a single value which does not show any decreasing trend and that could be a sign of the inability of the model on that dataset. This could be due to the fact that the model is to small to generalize on that data.

The table below resume the evaluation metrics on both dataset and we have the adapt model with the local planar shows slight increase in both metrics accuracy under threshold and relative MAE, for both datasets, comparing to the adapt modal alone. For the accuracy under threshold the highest the better and for relative MAE the lower the better.

Qualitative Results

The pipeline for inference starts by predicting the depth map, on which we apply a sigmoid function to have positive values. The depth map prediction model outputs depth maps with negative values. we Have test using sigmoid, ReLU or Softplus activation function to output the final depth map during training but it led to bad results. Then, we estimate the focal length and the depth shift. To do so, we create a 3D point cloud from a standard focal length and camera optical center. Then we predict refine the focal length by estimating a focal scale from the constructed 3D point. We use the new focal length to construct new 3D point cloud and to refine then the depth by estimating a depth shift. Again use the refined depth to refine another time the focal length. Then we have the final depth and the focal length that we use to generate the final 3D point cloud.

Qualitativly, we can see that the approach can capture the global shapes, but the point cloud are very distorded. For example if we compare the left views and top views between group truth and the prediction, we can see easily the huge difference which could be due to the depth prediction.

LLM architecture

Most of modern Large language models (LLMs) are based on decoder-only architecture. As illustrated in the figure below, every architecture use the same logic and workflow. They start with a tokenization step followed by an embedding layer, to produce a representation for each word in the input text. Then, they use a variant of Multi-Head Attention block (such as Masked Grouped Query attention in figure example, Multi-Head Latent Attention in models like DeepSeek V3/R1), Feed Forward block (containing Linear layers with activation function such as SiLU) and a Normalization Layer (e.g Layer Normalization, RMS Norm) to produce a general representation of the input text. This representation is finally passed through a linear layer to output a probability distribution over all the tokens in the vocabulary set of the LLM. This probability is used then to generate the next token. The generated token is added to the initial input text and the resulting in text is feeded to the LLM to generate the second next token until we have the full answer. This process makes LLMs autoregressive models which means to generate a token we need to process all the previous tokens.

To have an idea about each component of an LLM, we will explain the original transformers architecture, which is an encoder-decoder architecture. As all models are based on that architecture and only slight changes made to it.

Transformers

Released by google in their paper “Attention is all you need” in 2017, transformers are backbone of modern LLMs architectures. They overcome Recurrent Neural Networks limitations. Prior to transformers, RNNs were the standards for sequential data such as text (e.g $s=(w_1, w_2, …, w_n)$ ). An RNN generates a hidden state $h_t$ for each element $w_t$ in the sequence by utilizing the element’s representation and the preceeding hidden state $h_{t-1}$. While this allows the model to capture context, it introduces a critical dependency: the computation for step $t$ must wait for step $t-1$. This sequential dependency causes significat issues:

• Inefficiency: Training cannot be parallelized across GPUs
• Instability: RNNs suffer from vanishing/exploding gradients.
• Information Loss: The context vector degrades over long sequences, failing to capture long-range dependencies.

Transformers solve these limitation by using a Multi-head attention mechanism to process input tokens in parallel, preserving relationships regardless of distance as illustrated in the figure below.

Let’s dig into the architecture of a transformer.

Input Embedding and positional Encodding.

1. From text to Tokens: The input to an LLM is a sequence of tokens ids, denoted as $s=(t_1, t_2, …, t_L)$ ($L$ represents the length of the sequence). These tokens ids are generated by a tokenizer (a process we will cover later), which takes the sentence and splits it into words or sub-words called tokens and finally each token is mapped to its own id. Each LLM have its own tokenizer, for example tokenizer of GPT is different from LLama and so on. The figure below ilustrate this process on the sentence “The mind of man is capable of anything.” (taken from heart of darkness by Joseph Conrad)

1. The embedding Layer: Once tokenized, the sequence is passed through an embedding layer (it is learnable). This maps discrete tokens’ ids to dense vectors. If we assume the embedding dimension is $d_{model}$ (e.g $d_{model}=512$), this step outputs a tensor of shape $(L, d_{model})$. So, each token id $t_{i}$ is now represented by a dense vector of dimension $d_{model}=512$

We can use nn module of pytorch to code the embedding layer as follows.

embedding_layer = nn.Embedding(vocab_size, d_model)
x_embedding = embedding(x)

1. Positional Encodings: Here lies a critical challenge. Unlike RNNs, transformers process all tokens in parallel. This means the model has no inherent sense of order (it cannot distinguish between “the dog bit the man” and “The man bit the dog”). To solve this, we add Positional Embedding vector (not learnable) to each token’s embedding vector. This adds information about the token’s absolute or relative position in the sentence and help the model to understand the word function (whether it acts as noun, verb, subject etc). Without this, the Multi-Head Attention would view the sentence as a chaotic “bag of words” rather than a structured sequence.

#  Create a mtrix of shape (seq_len, d_model)
  pe = torch.zeros(seq_len, d_model)
  # Create a vector of shape (seq_len)
  position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1) # (seq_len, 1)
  div_term = torch.exp(torch.arange(0, d_model, 2).float()*(-math.log(10000.0) / d_model))

  ## Apply sin
  pe[:, 0::2] = torch.sin(position*div_term)
  pe[:, 1::2] = torch.cos(position*div_term)

  pe.unsqueeze_(0) # (1, seq_len, d_model)

Multi-Headed Attention

As illustrated in the transformer architecture above, the input embedding and positional encoding block produce an input tensor $X$ of size $(L, d_{model})$ which contains a dense vectors representing tokens in the text sequence. This input tensor is feeded then to Multi-Head Attention block to capture token’s interaction with other words. Multi-head Attention block is based on Scaled Dot-Product Attention, which we will explain first.

Scaled Dot-Product Attention The scaled Dot-Product attention takes as input three matrices of size $(L, d_{model})$: Queries $Q$, Keys $K$ and Values $V$. Those three matrices are just the input tensor (often via linear projection) where:

The mechanism then compute a new representation for each token, enriching it with contextual information from other tokens using The following formula and the figure below illustrate the mechanism:

1. Similarity: $QK^{T}$ calculates the similarity between every Query and every Key
2. Scaling: We divide by $\sqrt d_k$ to prevent gradients from vanishing in the softmax.
3. Weighting: The softmax turns the scaled similarities into attention scores (or probabilities) representing how relevant every other token is to the current token. Multiplying these scores by the V matrix produces a weighted sum of the value embeddings.

Properties:

• Permutation invariant: Attention mechanism is mainly matrix computation. It does not inherently respect the order of the sequence. Changing row order of input will output the same results. This is why positional embedding is required.
• Masking: The score matrix yielded by the Softmax operation has a size of $L \times L$. If we want to prevent interactions between some tokens (like hiding future tokens during training), we apply a mask - setting those scores to $-\infty$, before applying the softmax function, so their probabilities become zeros.

def attention(query, key, value, mask):
  d_k = query.shape[-1]
  attention_scores = query@key.transpose(-2, -1) / math.sqrt(d_k)
  if mask is not None:
      attention_scores.masked_fill_(mask==0, -1e9)
  attention_scores = attention_scores.softmax(dim=-1)
  output = attention_scores@value
  return output

Multi-head Attention Multi-HEad attention refines the mechanism and improves the model’s performance by allowing it to jointly attend to information from different representation subspaces at different positions.

Instead of performing single attention $d_{model}$-dimensional matrices, it is benificial to linearly project the Q, K, and V matrices $h$ times with different, learned linear projections to $d_k$, $d_k$, $d_v$, respectively.

A scaled dot-product attention is performed on each of the $h$ projected versions of Q, K, and V, yielding $d_v$-dimensional output values. These are concatenated and once again project to result in the final values. The following formula and figure illustrate the process:

W_q = nn.Linear(d_model, d_model)
W_k = nn.Linear(d_model, d_model)
W_v = nn.Linear(d_model, d_model)
W_o = nn.Linear(d_model, d_model)

query = W_q(q)
key = W_k(k)
value = W_v(v)

B, seq_len = query.shape[0], query.shape[1]

query = query.view(B, seq_len, h, d_k).transpose(1, 2)
key = key.view(B, seq_len, h, d_k).transpose(1, 2)
value = value.view(B, seq_len, h, d_k).transpose(1, 2)

x = attention(query, key, value, mask)
x = x.transpose(1, 2).contiguous().view(B, seq_len, d_k*d_h)
x = W_o(x)

Add & Norm Layer and Residual connection

The output of the multi-head attention block is a tensor $M$ of size $L \times d_{model}$ which is passed to Add & Norm layer. In this layer the input tensor $X$ is added to $M$ via a residual connection. This addition or residual connection prevents from vanishing gradient. Then the output of this addition is normalized.

Either in the original transformer paper the Norm layer is placed after the attention block or the feed forward layer, known as Post-Normalization, many LLMs architecture use the normalization layer before, known as Pre-Normalization. In 2020, Xiong et al, they showed thatPre-Normalization is more benificial by resulting in more well behaved gradients at initialization and works well without careful learning rate warm-up which is critical for Post-Normalization.

In some LLMs such as OLMo 2, they use Post-Normalization with RMSNorm instead of Layer Norm or Batch Norm as it results in smoother gradients and then more stability in the training.

Feed Forward Layer

Decoder

The decoder is similar to the encoder. It takes the outputs as input and pass them through embedding and position encoding block. The tokens’ tensor is passed as Queries, Keys and Values, through a Masked Multi-head Attention. The output of the Masked Multi-head Attention is passed as Queries to another Multi-Head Attention along with the encoder output which is considered as Keys and Values.

The goal of using Masked Multi-Head Attention is to make the model causal (i.e the output at a certain position can only depend on the words on the previous position). The model must not be able to have information about the future. This is achieved by using the masking property of Scaled Dot-Product Attention. The scores that correspond to the current token and its interaction with future tokens are set to $-\infty$ which will yield to 0 when applying softmax.

Linear and softmax

proj = nn.Linear(d_model, vocab_size)
out = torch.log_softmax(proj(x), dim=-1)

Training

Training an LLM is done through two steps: Pre-training step which consists of training the LLM on a large scale data (billions to trillions of tokens). The result of this step is a base model and it aims only to encode the world knowledge into the models. This means it can generate token in a way to have meaningful generation but not necessarly correct. For example if we ask the base model “when was the first LLM trained” and it answers “1439”, it gives a good output but it does not matter if it is true. This step is very expansive as it requires powerful resources that scale up with model’s number of parameters and the data.
The Second step is Post-training step which consists of training the base-model on high quality data to align its behavior imporve its capabilities and make it more helpful. For post-training, we can use mainly Supervised fine-tuning (SFT) or reinforcement learning (RL).
Pre-training and SFT consists of optimizing the same loss function and reiforcement learning use different loss function. But the training logic is mainly the same. We will details how we train an LLM in general and then we will give some examples of loss function for RL.

The goal of a language model is to predict the probability of the sequence of tokens. Let $(x_1, x_2, …, x_n)$ be our sequence. Its probability is:

When applying the log the sequence probability becomes:

So, predicting the next token, we mainly want a token having the highest probability as follows:

In SFT, the data is composed from input context $x = {x_1, x_2, …, x_n}$ and the desired output $y = {y_1, y_2, …, y_T}$. The SFT is a multiclass-classification problem, the loss function used is negative log-likelihood of producing the correct sequence $y$ given the input $x$. This is often implemented as cross-entropy between the predicted class by the model and the true class in the dataset.
At seqeunce step $t$, let \(y^{*}_{t}\) the ground truth token and let \(\log p_{\theta}(y^{*}_{t}|x_1, ..., x_n, y_1, ..., y_{t-1})\) denote the log-probability of producting the token $y^{*}_t$ by the model.

Then the model optimizes the following Loss function:

in Practice we use the following formulation:

Intuition: The following loss is trying to maximize always the probability of the ground truth token $y^{*}_{t}$ and push it to be 1. The log is applied on the probabilities which have values between $0$ and $1$, so the -log gives values between $+\infty$ and $0$. So minimizing this loss means pushing the -log to be 0 and then the probabilities to be 1.

Inference

At inference, we have a context $x=(x_1, x_2, …, x_n)$ and we want to generate a sequence $\hat y=(y_1, y_2, …, y_T)$ such that the following conditional probability is maximal:

So the generation process is as follows: at eacht time step $t$ we generate a token $y_t$ using its conditional probability $P (y_t|x_0,…, x_{n}, y_0, …, y_{t-1})$ . Add the generated token to the context which is use after to generate the next token $y_{t+1}$ . But how do we use the conditional probabilities to generate the next token?

Greedy Search Strategy: This strategy consists of selecting the token with the highest conditional probability at each time step $t$:

Selecting always at each time step $t$ the token with the highest probability does not entail that the the entire output sequence has the maximum joint probability. Maybe selecting the second probable token at time step $t$ could lead to have token with higher probability at timestep $t+1$ than a token yielded by selecting the first probable token at timestep $t$.

So, while this method is efficient as it sees at each timestep $t$ one token (the most probable token) comparing to other methods that take into account many tokens and keep track of them to generate the sequence (will be explained later). It can miss better tokens as explained above and lead to short-sighted output.

Beam Search: At each timestep $t$ we select $n$ first probable tokens ($n$ is the number of beam). This process is repeated until we reach the predifined maximum length $T$ or end-of-sequence token appears. Then, we will choose the sequence $y$ that have maximum joint probability $P(y|x)$

Top-k sampling strategy: It leverage the probability distribution generated by the language model to select a token randomly from the k most likely options. This method add an element of randomness in the generation process.

You might have heared that temperature controls the creativeness of the LLM. Higher temperature more creative model and lower temperature more deterministice model. Actually, the temperature $\tau$ add another sort of randomness in the generation process. It affects the shape of the probability distribution. The probability distribution is formulated as follows:

So, higher temperature encourages the probability distribution to be near to a uniform distribution. This is why higher temperature lead to hallucination. Lower temperature increases the probability of the most probable tokens and decreases the probability of unlikely tokens.

If we use top-k sampling strategy with lower temperature, we will likely sample the same sequence if we call the LLM many times.

Nucleus sampling strategy: known also as top-p sampling. At each timestep $t$, it consists of selecting top probable tokens in a descending way until their cumulitive probabilities exceeds a cutoff values $p$. Then, the next token is sampled from the selected tokens. Using this strategy, we have at each timestep $t$ a different number of tokens to select from. This encourages more diverse and creative output.

Quantization:

Large Language Models require powerful resources to generate a response in a reasonable time as they get bigger and have more parameters. Not only the number of parameters that influences the quantity of computational resources needed, but also the data type of the parameters affects and the efficiency of the model. Generally in deep learning, and in LLMs specifically, FP32, FP16 and BF16 are used as data types:

• FP32: Uses 32 bits to represent a number. One for the the sign, eight for the exponent, and the remaining for the significand. This data type provide a high degree of precision but it is computational high and memory footprint.
• FP16: Uses 16 bits to represent a number. One bit for the sign, five for the exponent and 10 for the significand. It is more memory efficient and less computational but it introduce numerical stability and potentially impact the model performance due to the reduced range and precision in representing the number.
• BF16: Uses also 16 bits to represent a number. One for the sign, seven for the exponent and 8 for the significand. It expands the representable range compared to FP16, thus decreasing underflow andoverflow risks. Despite a reduction in precision due to fewer significand bits, BF16 typically does not significantly impact the model performance.

As you can conclude, using 1B model with FP32 or FP16 will come with a tradoff between accuracy and memory footprint and latency. Depending on the use case, we can have the accuracy needed with data types that are computational lower than FP16, such as INT8 or INT4. The model parameters (weights) are generally in FP32 and converting them to a lower-precision data type such as INT8 or INT4 is called Quantization.

Generally, we have two types of Quantization:

• Post-Training Quantization (PTQ): It is straightforward technique where the weights of an already trained model are converted to a lower-precision data type.
• Quantization-aware training (QAT): incorporate the quantization process during training stage of finetuning. QAT is computational expensive and requires representative training data.

Let’s get an intuition by introducting some naive-quantization 8-bit techniques.

Absolute Maximum Quantization (absmax): It uses the following formula to convert an FP32/FP16 number to INT8 number:

This methods is symetric and it maps weights values to a range of $[-127, 127]$

This methods comes with a loss of precision due to rounding. When applying the dequantization formula to get the original weights, we will have a precision error.

Zero-Point Quantization: It applies the following formula to perform quantization.

This methods is asymetric and maps the weights value to a range of $[-128, 127]$

In practice, we apply vector-wise quantization which consider the variability of values in rows and columns of the same Tensor. Quantizing the entire Tensor in one pass will lead to a huge model degradation and quantizing individual values will yield to a big computation overhead.

The both methods discussed above are sensitive to outliers.

LLMs such as GPT family, Llama, Qwen or deepseek … are trained on large data that is gathered mainly by crawling the internet. This step of training on large data is called pretraining which is very costly. For example llama 3 with 1 billion parameters requires 314k GPU hour of H100 with 80GB. The knowledge of these LLMs is constrained by the website crawled (i.e data trained on) and time of crawling. This means when asking these LLMs about information that are resticted, for example companies internal documents or information that are occured after the crawling time, they will fail to answer correctly i.e they will hallucinates. Addressing these challenge is very crucial for LLMs to be more valuable, especially within companies.

We can think of updating LLMs with new information by fine-tuning them using full fine-tuning or techniques such as LoRA. But fine-tuning has also a cost. Imagine an Investor needs to have a summary of news everyday about differents topics e.g politics, finance, regulations and technology and within different countries. For this scenario when the data is updated on a regular basis, finetuning is not practical. To overcome this challenge, Retrieval Augmented Generation tend to be a suitable solution. Retrieval Augmented Generation take advantage of the ability of LLMs to understand the information provided within the prompt, called “context”, and mix it with retrieval techniques which provides information similar to the query from an external database.

RAG Framework

Basic RAG Workflow

layout: distill title: LLM From Scratch date: 2025-01-20 11:12:00-0400 description: tags: categories: Technical-post related_posts: false giscus_comments: true featured: true mermaid: enabled: true zoomable: true code_diff: true map: true chart: chartjs: true echarts: true vega_lite: true tikzjax: true typograms: true

authors:

• name: Rida Lefdali url: “www.linkedin.com/in/rlefdali/” affiliations: name: Independent toc:
• name: Math
• name: Probability, Statistics, Algebrea.

_styles: > .fake-img { background: #bbb; border: 1px solid rgba(0, 0, 0, 0.1); box-shadow: 0 0px 4px rgba(0, 0, 0, 0.1); margin-bottom: 12px; } .fake-img p { font-family: monospace; color: white; text-align: left; margin: 12px 0; text-align: center; font-size: 16px; }

In this post we will cover LSTM through time series forecasting. The aim is to explain LSTM and its advantage regarding RNNs. We will cover an application of Energy consumption and cover also some statistical algorithm for comparison.

As we explained previously RNN takes as input a sequence of elements and outputs a sequece of hidden states. Each unit in RNN take as input an element from sequence and the hidden state of the previous element from the sequence. Dealing with sequences makes RNN suitable for making prediction using time series as data. Time series is a sequence of elements, that could be real value, images or texts, that is ordered in a chronological order.

A first problem of RNN is its inability to perform well on tasks that require the use of information distant from the current point of procssing. The hidden states tends to be local and relevant only the most recent parts of the input sequence. The second problem is vanishing gradients, as during training the error needs to backpropagate through time and then we have repeated multiplications which results in that gradients are eventualy driven to zero.

To address this limitations, more complex network was designed. And LSTM is the most commonly used as an extension to RNN.

Long Short Term Memory LSTM

Long Short term memory solves the limitation of neglicting information distant from the current time step, encountered in RNN. It achieves this by adding a context layer to the architecture that output a context vector \(c_t\). And, adding, also, gates that allow either to remove information that are no longer needed or to add information to be needed for later decision making.

The first gate to consider is the forget gate \(f_t\). Its purpose is to delete information that are no longer needed and computed as follows:

The second gate to consider is the add gate which aims to select information to add to the current context and computed as follows:

where \(g_t\) is the actual information we need to extract from the previous hidden state and current input :

the output of the add gate and the forget gate are summed then to get the current context vector:

finally we have the output gate that decides what information is required for the current hidden state. Is is computed as follows:

We can remark that the gates have a similar design. They contain a feedforward layer, followed by a sigmoid function, and finally followed by an element-wise multiplication with the layer being gated.
The choice of sigmoid function arises from its tendency to push its output to either 0 or 1. With the use of the element-wise multiplication, we have the effect of a binary mask. This allows then the values in the layer being gated that align with values near to 1 to pass and the values that align vaues near 0 to be erased. This makes the intuition behind how the add gate or the forget gate select or delete information.

The LSTM can be implemented using Pytorch simply as follows:

import torch
lstm = nn.LSTM(
            input_size=d, # The size of the input
            hidden_size=n_units, # The size of the hidden state 
            num_layers=1,
            bias=True,
            batch_first=True,
            bidirectional=True,
        )

to be continued …