The integration of deep learning (DL) algorithms offers significant potential to enhance healthcare applications. Specifically, DL has advanced medical imaging diagnostics, therapeutic interventions, and disease management. However, deploying deep learning algorithms on sensitive medical imagery introduces substantial challenges regarding data security and patient privacy. Optimizing medical imaging performance while simultaneously ensuring patient anonymity remains a complex task. Consequently, privacy-preserving techniques designed for deep learning model training and inference are increasingly gaining traction. Currently, these medical image sequences are interpreted using state-of-the-art computer-aided detection/diagnosis (CAD) methodologies. Yet, frameworks that require uploading medical photographs to external servers inherently create privacy vulnerabilities.To address this, this article introduces a convolutional Bi-LSTM network designed to evaluate fully homomorphic-encrypted (HE) time-series medical images. Within this framework, convolutional blocks extract selective spatial features from the secure image sequences, while analytical sequence layers based on Bi-LSTM encode the temporal data. To optimize efficiency and minimize misdiagnoses, a weighted unit and sequence voting layer incorporates spatial data using varying weights. The effectiveness of this framework is demonstrated using two rigorous public benchmarks: the CheXpert and BreaKHis datasets. The proposed technique surpasses several competing methods, achieving an accuracy exceeding 0.99 across both datasets. These findings indicate that the introduced methodology can successfully extract visual representations and sequential dynamics from encrypted medical image series, thereby safeguarding patient privacy while maintaining high performance in medical image analysis.