Think about understanding whether or not a film evaluate is a glowing endorsement or a scathing critique — earlier than even studying it. That’s the magic of Film Evaluation Sentiment Prediction, my newest challenge that blends machine studying, cloud infrastructure, and DevOps into one seamless journey.
Constructed with MLflow for experiment monitoring, AWS for scalability, and FastAPI for real-time predictions, this challenge takes you from uncooked information to a deployed mannequin with monitoring — all wrapped up in a neat CI/CD pipeline.
Able to dive in? Let’s discover the journey after which construct it step-by-step!
This isn’t simply one other ML mannequin — it’s a full ecosystem. We pull film evaluate information from AWS S3, course of it by means of a sturdy information pipeline (ingestion, preprocessing, characteristic engineering), and prepare a sentiment prediction mannequin tracked with MLflow and Dagshub.
Firstly we run totally different experiments on python notebooks and observe them on MLFlow. The mannequin’s efficiency is evaluated, registered, and staged in MLflow’s Mannequin Registry. Then, FastAPI serves up predictions utilizing the most recent mannequin, whereas Docker and AWS EKS deal with deployment. Add Prometheus and Grafana for monitoring, and GitHub Actions for CI/CD, and also you’ve acquired a production-ready ML answer. Excited? Let’s get began!
Kick issues off by making a GitHub repo and cloning it regionally. Arrange a digital atmosphere and scaffold your challenge with Cookiecutter for information science aptitude:
Your challenge construction is now primed for motion!
Head to Dagshub, create a repo, and join it to your GitHub challenge. Seize the experiment monitoring URL and code snippet from Dagshub:
- Set up dependencies: pip set up dagshub mlflow.
- Run your experiment notebooks, observe leads to MLflow, and push modifications to Git.
Now your experiments are logged remotely — fairly slick, proper?
Initialize DVC (dvc init) and arrange a neighborhood folder (local_s3) for momentary storage (Think about this state of affairs in testing) : dvc distant add -d mylocal local_s3. Add these recordsdata to src/:
- data_ingestion.py, data_preprocessing.py, feature_engineering.py.
- model_building.py, model_evaluation.py, register_model.py.
Outline your pipeline in dvc.yaml and params.yaml, then run dvc repro. Connect with AWS S3:
- Create an S3 bucket and IAM consumer, set up dvc[s3] & awscli, and configure credentials (aws configure).
- Hyperlink S3:
dvc distant add -d myremote s3://& push information with dvc push.
Your information pipeline is now cloud-ready!
In model_building.py, prepare your sentiment mannequin. Consider it in model_evaluation.py, logging metrics to MLflow. Register it in register_model.py and transition it to the “Staging” stage in MLflow’s Mannequin Registry. Your mannequin’s lifecycle is now tracked and reproducible!
Create a app/ listing, add your FastAPI app, pulling the most recent mannequin from MLflow for inference. Set up fastapi and check regionally. Freeze dependencies: pip freeze > necessities.txt.
Add a exams/ listing with:
- test_app.py: Checks FastAPI routes (/residence, /predict).
- test_model.py: Validates mannequin efficiency from MLflow.
- promote_model.py: Strikes the mannequin from Staging to Manufacturing.
High quality assurance? Examine!
Set up pipreqs, generate necessities (pipreqs flask_app/ — pressure), and create a Dockerfile. Construct your picture: docker construct -t capstone-app:newest .. Take a look at it regionally: docker run -p 8888:5000 -e CAPSTONE_TEST=. Push to AWS ECR after organising credentials and repository:
- Configure AWS CLI, create an ECR repo (capstone-proj), and push:
docker push..dkr.ecr.us-east-1.amazonaws.com/capstone-proj:newest
Add .github/workflows/ci.yaml to automate constructing and pushing to ECR. Generate a Dagshub token, retailer it in GitHub Secrets and techniques, and replace your CI file. Your pipeline’s now hands-free!
Arrange AWS CLI, kubectl, and eksctl (obtain by way of PowerShell — see directions). Create an EKS cluster:
- eksctl create cluster — identify flask-app-cluster — area us-east-1 — node-type t3.small — nodes 1.
- Replace kubectl config: aws eks update-kubeconfig — identify flask-app-cluster — area us-east-1.
Deploy your app with deployment.yaml and expose it by way of a LoadBalancer service. Edit safety teams for port 5000 entry. Seize the exterior IP: kubectl get svc flask-app-service. Take a look at it: curl http://
Launch two EC2 cases (Ubuntu, t3.medium):
- Prometheus: Set up (wget & extract), configure prometheus.yml along with your app’s exterior IP, and run:
/usr/native/bin/prometheus — config.file=/and so forth/prometheus/prometheus.yml. Entry at:9090. - Grafana: Set up (wget & apt set up), begin (systemctl begin grafana-server), and add Prometheus as a knowledge supply at
:9090. Construct dashboards at :3000.
Your app’s now monitored like a professional!
Keep away from AWS prices: Delete EKS cluster (eksctl delete cluster), deployment (kubectl delete), ECR/S3 artifacts, and confirm by way of AWS Console.
From information in S3 to predictions on EKS, this challenge showcases the facility of recent ML workflows. With MLflow, FastAPI, Docker, and AWS, plus monitoring and CI/CD, it’s a blueprint for taking ML from pocket book to manufacturing.
Strive it out, tweak it, and let me know the way it goes — completely satisfied coding!
My Portfolio
