SubQ レビュー

NSA enhances long-context modeling in LLMs by seamlessly incorporating sparsity into both training and inference, utilizing hierarchical token modeling and a hardware-aligned design for real-world speedups.

Like SubQ, NSA focuses on sparse attention for long contexts and efficiency. NSA emphasizes hierarchical token modeling and hardware alignment for real-world speedups in both training and inference, aiming to maintain accuracy and contextual relevance.

DHSA is a data-driven framework that dynamically predicts attention sparsity online without retraining the base LLM, achieving significant prefill speedups and preserving near-dense accuracy.

DHSA directly competes with SubQ by offering a dynamic approach to sparse attention for long contexts, focusing on adaptability and significant prefill speedups on existing LLMs, even on resource-constrained devices. It aims to overcome limitations of static sparse methods by adapting to task or input variations.

SALE is a fine-grained sparse attention method that accelerates long-context LLM prefilling using 4-bit quantized query-key products and block-sparse attention, requiring no parameter training.

SALE offers a fine-grained sparse attention method specifically for accelerating the prefilling stage of long-context LLMs, similar to SubQ's efficiency goal, but with a focus on low-bit quantization and seamless integration into existing systems without retraining.

SparDA is a decoupled sparse attention architecture that introduces a 'Forecast' projection to predict and prefetch KV blocks, enabling lookahead selection and improving both prefill and decode speed.

SparDA provides a novel decoupled sparse attention architecture that aims for efficiency in both prefill and decode, directly addressing the same performance and long-context challenges as SubQ, but with a unique lookahead selection mechanism and minimal parameter overhead.