Cognitive vector quantization for malware detection