Spectral Gap Rigidity And Unique Prime Decomposition