A HYBRID HOG–PCA AND VISION TRANSFORMER–BASED SEQUENCE-TO-SEQUENCE FRAMEWORK FOR LOW RESOURCE SENTENCE-LEVEL INDIAN SIGN LANGUAGE VIDEO-TO-TEXT TRANSLATION

Abstract

Automatic Indian Sign Language (ISL) video-to-text translation is still a problem because of the complex spatial-temporal relationships between gestures, signer variability and scarcity of annotated training material. A hybrid sequence-to sequence model with HOG–PCA and Vision Transformer (ViT) for low-resource sentence-level ISL translation is proposed in this study. The proposed architecture consists of concatenating handcrafted gradient descriptors with the contextual embeddings obtained from the ViT. An autoregressive decoder produces English sentences from the temporal Transformer encoder, which models the sequential dependencies of the gestures. Comparative experiments, low-resource evaluations, architectural ablations and PCA dimensionality analysis were conducted to evaluate translation performance. The proposed framework achieved BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores of 30.80, 22.60, 17.40, and 13.20, respectively, alongside ROUGE-L and chrF scores of 37.60 and 32.80. Over the conventional ViT Transformer and the parameter-matched baseline, BLEU-4 gained 4.70 and 3.50 points, respectively. The hybrid model achieved better performance than the ViT-only model in all the training-data percentages with 128 PCA components showing the best translation performance. Based on these findings, the complementary handcrafted and deep visual representations can be considered as a potential approach for improving ISL sentence-level translation with limited supervision.