A Cloud-Integrated Threat Monitoring Pipeline Using AWS CloudWatch Logs and Prefect Orchestration: Design and Practical Evaluation

Abstract

This paper proposes a cloud-integrated threat monitoring pipeline combining AWS CloudWatch Logs with Prefect orchestration to enable automated, scalable, and low-latency security log analysis. The system extracts, processes, and stores CloudWatch log data in PostgreSQL through scheduled and parallel Prefect workflows. A practical evaluation using real datasets demonstrates improvements in processing latency, throughput, and reliability compared to traditional Lambda-based and ELK pipelines. Results show that the Prefect-orchestrated approach offers a flexible and efficient solution for continuous threat monitoring in cloud environments.

Keywords