Towards Audio Intelligence: Perception, Integration, and Alignment

Abstract:
This thesis investigates how models can perceive acoustic information, integrate it with language and reasoning, and align their decisions with the requirements of an application. It addresses these challenges through audio representation learning for perception, efficient integration of audio representations into large language models (LLMs), and alignment of audio-LLM decisions with editorial judgment.

For perception, the Multi Axis Audio Spectrogram Transformer (Max-AST) combines convolution, window attention, and grid attention to capture local and global context with lower computational and parameter costs than the Audio Spectrogram Transformer. The Decoupled Time-Frequency Audio Transformer (DTF-AT) investigates specialised branches for the distinct structures of time and frequency. To learn from unlabelled audio containing overlapping sources, Self-Supervised Learning from Audio Mixtures (SSLAM) combines partially mixed recordings with a Source Retention Loss. It improves polyphonic representations while maintaining strong transfer to general and predominantly monophonic audio tasks.

For integration, high-resolution audio tokens can be costly to process throughout an LLM, while compression can discard useful acoustic detail. Lightweight Audio LLM Integration (LAL) supplies audio representations as keys and values in selected attention layers without passing them through intervening feed-forward blocks, reducing memory use and increasing training throughput while retaining competitive performance. PAL extends LAL with compact audio summary tokens propagated through the LLM, improving the balance between efficiency and performance across speech, sound, and music understanding.

For alignment, the thesis considers media chapterization, where boundaries reflect creator-authored editorial judgment about meaningful transitions. AudioChaps uses Group Relative Policy Optimisation to align audio-LLM decisions with creator-authored chapter-boundary annotations. Rewarding agreement with these annotations substantially improves chapter-boundary judgments across diverse audio categories and different audio-LLM families.