For the H.264/AVC video stream with covert data by collusion-irresistent steganography, a blind video steganalysis scheme is proposed based on the collusion sensitivity and noise classification. It exploits temporal frames weighted averaging (TFWA) for collusion to improve the capabilities of host approximation and watermark removal, instead of the traditional temporal frames averaging (TFA). To overcome the interferences by motion and illuminance variation, a content change factor (CCF) is defined to adaptively classify the noise existing in prediction error frames (PEF). For passive steganalysis, final decision is made by the center of mass (COM) feature of histogram characteristic function (HCF). Experimental results demonstrate that the proposed approach can cope with temporal-domain, transform-domain and even spread spectrum based steganographic algorithms. For a stego-video with embedding strength of 10%, it can achieve a probability of correct detection (PCD) about 99.82%.