MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model
Text-to-music alignment is usually tested with a single global similarity score that hides which specific musical attributes are missing or misaligned. This paper proposes MQAScore, a fine-grained evaluator that uses a large audio language model to verify each atomic concept—across genre, instrument, mood\/theme, vocal, and usage dimensions—via binary question answering. The work matters because it moves beyond opaque global scores toward interpretable, per-dimension diagnostics that can directly guide improvements in text-to-music generation systems.
MQAScore offers a conceptually clean and practically needed extension of QA-based alignment scoring to the music domain. By decomposing alignment into five interpretable clip-level dimensions and introducing the Sensitivity-Invariance Ratio $R_{S/I}$, the paper provides both a diagnostic metric and a way to audit whether an evaluator is reacting to audio content or mere wording changes. Empirical gains over CLAP and MuQ-MuLan are consistent, but the study is primarily a meta-evaluation of evaluators rather than a validation on generated music, and its verification pipeline leans heavily on LLM-generated labels.
The core argument that global similarity obscures which attributes fail to align is convincingly demonstrated, and the taxonomy grounded in CAL500 and MTG-Jamendo is well justified. The $R_{S/I}$ metric is a principled addition: it shows similarity-based models often respond as strongly to near-synonyms as to contradictions, whereas concept-level LALM scoring tracks audio semantics far more selectively. Results in Table 2 further show that LALM-based evaluators approach human-human concordance on expert pairwise annotations.
The paper relies on LLM-guided concept extraction and substitution for mood\/theme, vocal, and usage negatives, which risks circular evaluation when the tested LALM shares text priors with the model that generated the labels. The single-label GTZAN assumption for genre is unrealistic, and the expert pairwise set is small (147+119 pairs). The authors also acknowledge that results on generated music are relegated to the project page due to space limits, meaning the manuscript itself lacks practical validation on text-to-music systems. Finally, the high $R_{S/I}$ observed for caption-level AQAScore is partly an artifact of dilution in long captions, making cross-granularity comparisons of the ratio less meaningful.
The evidence supports the main claims through controlled ablations comparing caption-level versus concept-level inputs across both similarity and QA paradigms. The gains for LALMs over similarity metrics on abstract dimensions are large and consistent across AUC, $R_{S/I}$, and expert agreement. However, the paper omits confidence intervals or statistical tests in Tables 1 and 2, leaving significance unclear. It also does not benchmark against other fine-grained reference-free metrics such as ELSA, even though related work notes their emergence.
Reproducibility is hampered by missing implementation details. While the paper names models and datasets and states that scores derive from single-forward-pass next-token probabilities without sampling, exact prompt templates, hyperparameters, and temperature settings are largely absent; only one example template is provided in Section 3.2. There is no mention of code release in the manuscript, and the practical evaluation of state-of-the-art TTM systems is deferred to the project page. Independent reproduction would therefore require substantial guesswork about the scoring protocol and LLM inference configuration.
Text-to-music alignment is commonly evaluated by the global similarity between text and audio embeddings, which collapses all described attributes into a single score and obscures which ones are missing. We instead introduce MQAScore, which verifies each atomic musical concept along five clip-level dimensions (genre, instrument, mood/theme, vocal, and usage) by the probability that a large audio language model (LALM) answers affirmatively to a binary question. To assess evaluator reliability, we construct verification labels from presence/absence annotations and LLM-guided concept substitutions, and propose the Sensitivity-Invariance Ratio, which measures how selectively an evaluator responds to meaning-changing substitutions. Experiments show that concept-level scoring improves both question-answering and similarity-based evaluators, and MQAScore can be a more reliable and interpretable fine-grained complement to global similarity for evaluating text-to-music alignment.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.