Transformers are always at the center of attention, as they have proven to be effective in handling sequential data, such as text, images, or time series. Today, they are the backbone of many state-of-the-art AI models, and they continue to evolve constantly. Researchers, aiming to improve transformers, consistently develop new methods to increase their efficiency. They focus on various areas, from improving the attention mechanism to enhancing memory and long-context capabilities.

So here is a list of 10 novel approaches for improving Transformers’ efficiency:

  1. Differential Transformer (DIFF Transformer) uses a differential attention mechanism that calculates attention scores by subtracting two softmax maps. By taking their difference, this approach focuses attention on relevant information, reducing noise and hallucinations. → Read more

  2. The normalized Transformer (nGPT) normalizes all vectors, such as embeddings and hidden states, to unit length on a hypersphere, with each layer adjusting them toward the correct output. This design speeds up learning, reducing training steps by 4 to 20 times while maintaining accuracy. → Read more

  3. DART (Denoising Autoregressive Transformer) is a new model that overcomes diffusion models’ limitations caused by step-by-step (Markovian) process. It combines autoregressive and diffusion methods to denoise image patches without relying on image quantization. DART also handles both text and image data. → Read more

  4. Cottention approach replaces softmax with cosine attention, reducing memory usage and achieving linear memory complexity for longer sequences. It can be reformulated as an RNN for constant memory usage during inference, while maintaining performance similar to softmax attention. → Read more

  5. DnD-Transformer improves image generation by addressing information loss in vector-quantization (VQ) models. It introduces a 2D autoregression method to predict more image details through depth and sequence length. It produces higher-quality images with the same model size as traditional methods and can generate images with text and graphics. → Read more

  6. Retrieval-augmented decision transformer (RA-DT) uses external memory to store and retrieve only relevant past experiences, making in-context learning more efficient. It performs well in grid-worlds and robotics simulations, using shorter context lengths while outperforming existing methods. → Read more

  7. Transformer with selective attention: Selective attention improves transformer performance by limiting attention to unnecessary elements in the context. These transformers matching the performance of models with twice the parameters. This reduces memory, computation needs, and efficiency in tasks with long context. → Read more

  8. Graph Transformers are neural networks designed for graph-structured data, combining the strengths of transformers and graph learning. They implement graph attention mechanisms and are used in node, edge, and graph-level tasks. This survey classifies graph transformer, reviewing their progress and implementation. → Read more

  9. Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey: Reviews recent advancements in improving LLMs' ability to handle longer inputs, covering upgrades to transformer architecture, evaluation methods, and optimization tools. It discusses challenges and future directions for improving LLMs with long-context inputs. → Read more

  10. Non-stationary Transformers solve the problem of ability to predict sudden changes using two modules: Series Stationarization (for predictability) and De-stationary Attention (to capture key changes). This improves forecasting performance across various models. → Read more

Reply

Avatar

or to participate

Keep Reading