In clinical environments, the transparency of artificial intelligence models is paramount, as understanding the rationale behind automated choices is vital. Although black-box AI frameworks exhibit remarkable efficacy, their deployment in healthcare remains impractical unless their underlying mechanisms can be clarified for medical professionals. Deep neural networks (DNNs) achieve state-of-the-art results, yet the justifications they provide are frequently incomprehensible to human observers. Conversely, traditional machine learning approaches and manually designed features extracted to capture distinct data characteristics offer greater transparency, though human constraints in feature engineering often limit their overall effectiveness. To bridge this gap, we present ExShall-CNN, a novel and transparent shallow convolutional neural network tailored for medical image analytics. This framework builds upon hand-crafted feature methodologies to preserve human interpretability, guaranteeing clear and verifiable decision-making. We introduce the explainable shallow convolutional neural network (ExShall-CNN), which merges the high performance of sophisticated deep architectures like U-Net with the inherent clarity of hand-crafted features for segmenting medical images. Grounded in contemporary machine learning progress, ExShall-CNN integrates widely utilized kernels while maintaining structural openness, allowing medical practitioners to visually inspect and comprehend its outputs. This balanced strategy delivers the precision expected of deep learning paradigms alongside the explainability required for clinical deployment.