ETL Pipeline on AWS

City Watch Logo
Diagram of VM ETL solution
Diagram of container ETL solution
Diagram of Lambda ETL solution

Project information

Serverless, Batch, or VMs

CityWatch has provided consent for sharing the following details and is comfortable with making a generalized code repository available.
City Watch commissioned the development of an ETL pipeline to ingest, transform, and analyze police department data. High-level architecture diagrams tailored to their use case are accessible for review.
In response, three distinct solutions were meticulously crafted and implemented to assess the most suitable architecture based on considerations of cost, user-friendliness, and scalability. The VM ETL architecture was ultimately selected for their ETL process, for dedicated host software licensing requirements. To optimize costs, the virtual machines (VMs) were strategically activated on a scheduled basis.