Fortifying robustness: unveiling the intricacies of training and inference vulnerabilities in centralized and federated neural networks