Introduction
A Bidirectional Long Short-Term Memory (Bi-LSTM) is a type of recurrent neural network (RNN) designed to process sequential data by capturing dependencies in both forward and backward temporal directions. Introduced as an extension of the Long Short-Term Memory (LSTM) architecture, Bi-LSTMs address the limitation of unidirectional models, which only consider past context. By integrating information from both preceding and succeeding elements in a sequence, Bi-LSTMs enhance modeling capabilities for tasks requiring comprehensive context understanding. The concept builds on bidirectional RNNs (Bi-RNNs), first proposed in 1997 by Schuster and Paliwal, and was later adapted for LSTMs to improve performance in domains like natural language processing (NLP), speech recognition, and bioinformatics.
Architecture and Functionality
The Bi-LSTM architecture consists of two separate LSTM layers:
- A forward LSTM, processing input sequences from the initial to the final element (e.g., $ t = 1 $ to $ t = T $).
- A backward LSTM, processing the same sequence in reverse order (e.g., $ t = T $ to $ t = 1 $).
At each time step $ t $, the forward LSTM computes a hidden state $ \overrightarrow{h_t} $, while the backward LSTM computes $ \overleftarrow{h_t} $. These states are concatenated to form a combined representation $ h_t = [\overrightarrow{h_t}; \overleftarrow{h_t}] $, where $ ; $ denotes concatenation. This dual-directional processing enables the model to capture dependencies from both temporal directions, effectively encoding contextual information from the entire sequence.
The mathematical formulation of each LSTM layer (forward and backward) follows the standard LSTM equations:
- Input gate: $ i_t = \sigma(W_i [x_t, h_{t-1}] + b_i) $
- Forget gate: $ f_t = \sigma(W_f [x_t, h_{t-1}] + b_f) $
- Candidate cell state: $ \tilde{C}_t = \tanh(W_C [x_t, h_{t-1}] + b_C) $
- Cell state: $ C_t = f_t \cdot C_{t-1} + i_t \cdot \tilde{C}_t $
- Output gate: $ o_t = \sigma(W_o [x_t, h_{t-1}] + b_o) $
- Hidden state: $ h_t = o_t \cdot \tanh(C_t) $
Here