Project information
- Company: CityWatch
- Project date: October, 2023
- Project duration: 1 month
- Project repo: https://github.com/Benwagrez/scaling-aws-etl-pipeline
Serverless, Batch, or VMs
CityWatch has provided consent for sharing the following details and is comfortable with making a generalized code repository available.
City Watch commissioned the development of an ETL pipeline to ingest, transform, and analyze police department data. High-level architecture diagrams tailored to their use case are accessible for review.
In response, three distinct solutions were meticulously crafted and implemented to assess the most suitable architecture based on considerations of cost, user-friendliness, and scalability. The VM ETL architecture was ultimately selected for their ETL process, for dedicated host software licensing requirements. To optimize costs, the virtual machines (VMs) were strategically activated on a scheduled basis.