A Multimodal Framework for Vibration Signals via Knowledge-Guided Preprocessing (O-XSTFT) and Reconstruction-Contrastive Tokenization (ReCoFormer)

초록

This study formulates vibration-based multimodal diagnosis as a physically grounded signal-to-token interface problem and proposes an integrated framework that transforms rotating-machinery vibration signals into fixed-length tokens for large language model (LLM)-based diagnosis. The proposed method consists of order-cross short-time Fourier transform (O-XSTFT)-based preprocessing, a vision transformer-based vibration encoder, reconstruction-contrastive token compression through ReCoFormer, and LLM-based fault classification and structured diagnostic description generation. O-XSTFT preserves localized time-frequency structure while incorporating order-domain normalization and inter-axis relational information, providing a mechanically meaningful input representation under variable-speed conditions. The vibration encoder learns local token representations through self-supervised reconstruction, and ReCoFormer compresses reference-relative local tokens into fixed-length vibration prefix tokens suitable for LLM conditioning. Experiments on a unified benchmark constructed from four public rotating-machinery datasets show that the proposed 8B model with K = 64 achieves a Macro-F1 of 0.96 and a ground-truth-conditioned LLM-as-a-Judge score of 0.854. Comparisons with representative LLM-based diagnostic baselines further show that the proposed O-XSTFT and ReCoFormer-based prefix-token interface preserves diagnostically useful vibration evidence more effectively than text-based signal descriptions or highly compact vibration-token integration. Ablation studies also confirm the contribution of O-XSTFT, vibration prefix tokens, and reconstruction-contrastive tokenization.

키워드