Transformer-Based Context Fusion For Long-Sequence Prediction In Sparse Data Environments

Abstract

Accurate and long-sequence time-series forecasting has been challenging in real-world systems due to observations are often sparse, irregularly sampled, and affected by missing sensor readings, or the transmission failures. The conventional recurrent, and the transformer-based models typically assume dense, and the uniformly sampled sequences, which leads to instability, and degraded performance when sparsity is high. This article presents a Transformer-Based Context Fusion (TCF) framework designed to explicitly model missingness, temporal gaps, and multi-scale temporal dependencies within a unified architecture. The method integrates sparse-aware embeddings, multi-context fusion attention, multi-scale dilated Transformer encoding, and a hybrid interpolation–attention reconstruction mechanism to maintain stable forecasting under severe data loss. Experimental validation on electricity load, clinical vital-sign, and the climate datasets demonstrates the consistent accuracy improvements over the state-of-the-art baselines, with up to 15–40% reduction in mean-squared-error under the sparsity levels of 60–80%, confirming robustness for the deployment in operational environments.

• Introduces the sparse-aware embedding, and multi-context attention architecture that jointly models the missing values, time gaps, periodicity, and contextual metadata.
• Developed the hybrid interpolation, and the attention-based reconstruction strategy integrated with the multi-scale dilated Transformer encoder for the long-sequence forecasting.
• Validates robustness on three real-world datasets under the controlled sparsity scenarios, which is showing significant accuracy gains, and the stable long-horizon prediction performance.