HyTEN: A Hybrid Transformer Architecture for Computationally Efficient Intrusion Detection in 6G Vehicular Networks

We consider the problem of deploying flow-based Network Intrusion Detection Systems (NIDS) at the vehicular network edge, where roadside units and edge gateways must inspect high-throughput 6 G traffic under tight compute and latency budgets. We propose HyTEN (Hybrid Transformer for Edge NIDS), a family of four transformer variants - encoder-only, decoder-only, and two encoder-decoder hybrids - designed to operate on NetFlow-style features exported from vehicular and IoT domains and to raise attack alerts toward a cloud/SOC layer. Under a unified architectural template, we explore depth and hybridization, evaluating over 100 combinations of input encodings and classification heads to identify an edge-deployable configuration. We benchmark HyTEN on the NF-UNSW-NB15-V2 dataset across three stratifications (1k, 10k, 100k NetFlow records) against Random Forest, SVM, Gradient Boosting, XGBoost, and Logistic Regression. We further discuss extension to NF-ToN-IoT-v2, a 43-feature IoT NetFlow benchmark, to address dataset diversity while preserving the same input representation. Our best variant, HyTEN-D (2 encoders +2 decoders with record-level projected embeddings and a featurewise classification head), achieves 98.8% recall and 95.0% F1-score on the 100k stratification with only 1.28M parameters, while maintaining stable training-time scaling. Although tree-based baselines attain slightly higher F1-scores, HyTEN-D offers superior recall - critical for NIDS where missed attacks are more costly than false alarms-and a compact hybrid transformer architecture amenable to hierarchical extension and further optimization for vehicular edge deployment.