Designing, Building & Deploying an AI Chat App from Scratch (Part 2) | by Joris Baan | Jan, 2025

Cloud Deployment and Scaling

Within the previous post, we constructed an AI-powered chat utility on our native laptop utilizing microservices. Our stack included FastAPI, Docker, Postgres, Nginx and llama.cpp. The aim of this submit is to study extra concerning the fundamentals of cloud deployment and scaling by deploying our app to Azure, making it accessible to actual customers. We’ll use Azure as a result of they provide a free education account, however the course of is comparable for different platforms like AWS and GCP.

You possibly can verify a reside demo of the app at chat.jorisbaan.nl. Now, clearly, this demo isn’t very large-scale, as a result of the prices ramp up in a short time. With the tight scaling limits I configured I reckon it may well deal with about 10–40 concurrent customers till I run out of Azure credit. Nevertheless, I do hope it demonstrates the ideas behind a scalable manufacturing system. We might simply configure it to scale to many extra customers with a better price range.

I give a whole breakdown of our infrastructure and the prices on the finish. The codebase is at https://github.com/jsbaan/ai-app-from-scratch.

1.1. Recap: native utility

Let’s recap how we constructed our native app: A consumer can begin or proceed a chat with a language mannequin by sending an HTTP request to http://localhost. An Nginx reverse proxy receives and forwards the request to a UI over a personal Docker community. The UI shops a session cookie to determine the consumer, and sends requests to the backend: the language mannequin API that generates textual content, and the database API that queries the database server.

1. Introduction
   1.1 Recap: native utility
2. Cloud structure
   2.1 Scaling
   2.2 Kubernetes Ideas
   2.3 Azure Container Apps
   2.4 Azure structure: placing all of it collectively
3. Deployment
   3.1 Organising
   3.2 PostgreSQL server deployment
   3.3 Azure Container App Setting deployment
   3.4 Azure Container Apps deployment
   3.5 Scaling our Container Apps
   3.6 Customized area title & HTTPS
4. Assets & prices overview
5. Roadmap
6. Closing ideas
   Acknowledgements
   AI utilization

Conceptually, our cloud structure is not going to be too completely different from our native utility: a bunch of containers in a personal community with a gateway to the skin world, our customers.

Nevertheless, as an alternative of operating containers on our native laptop with Docker Compose, we’ll deploy them to a computing setting that mechanically scales throughout digital or psychical machines to many concurrent customers.

Scaling is a central idea in cloud architectures. It means having the ability to dynamically deal with various numbers of customers (i.e., HTTP requests). Uvicorn, the online server operating our UI and database API, can already deal with about 40 concurrent requests. It’s even potential to make use of one other internet server known as Gunicorn as a course of supervisor that employs a number of Uvicorn staff in the identical container, additional rising concurrency.

Now, if we wish to help much more concurrent request, we might give every container extra assets, like CPUs or reminiscence (vertical scaling). Nevertheless, a extra dependable strategy is to dynamically create copies (replicas) of a container primarily based on the variety of incoming HTTP requests or reminiscence/CPU utilization, and distribute the incoming site visitors throughout replicas (horizontal scaling). Every reproduction container might be assigned an IP deal with, so we additionally want to consider networking: how you can centrally obtain all requests and distribute them over the container replicas.

This “prism” sample is necessary: requests arrive centrally in some server (a load balancer) and fan out for parallel processing to a number of different servers (e.g., a number of an identical UI containers).

Kubernetes is the business normal system for automating deployment, scaling and administration of containerized functions. Its core ideas are essential to know trendy cloud architectures, together with ours, so let’s rapidly assessment the fundamentals.

• Node: A bodily or digital machine to run containerized app or handle the cluster.
• Cluster: A set of Nodes managed by Kubernetes.
• Pod: The smallest deployable unit in Kubernetes. Runs one fundamental app container with non-compulsory secondary containers that share storage and networking.
• Deployment: An abstraction that manages the specified state of a set of Pod replicas by deploying, scaling and updating them.
• Service: An abstraction that manages a steady entrypoint (the service’s DNS title) to reveal a set of Pods by distributing incoming site visitors over the assorted dynamic Pod IP addresses. A Service has a number of sorts:
  – A ClusterIP Service exposes Pods throughout the Cluster
  – A LoadBalancer Service exposes Pods to outdoors the Cluster. It triggers the cloud supplier to provision an exterior public IP and cargo balancer outdoors the cluster that can be utilized to achieve the cluster. These exterior requests are then routed through the Service to particular person Pods.
• Ingress: An abstraction that defines extra complicated guidelines for a cluster’s entrypoint. It could route site visitors to a number of Providers; give Providers externally-reachable URLs; load stability site visitors; and deal with safe HTTPS.
• Ingress Controller: Implements the Ingress guidelines. For instance, an Nginx-based controller runs an Nginx server (like in our local app) beneath the hood that’s dynamically configured to route site visitors based on Ingress guidelines. To show the Ingress Controller itself to the skin world, you need to use a LoadBalancer Service. This structure is usually used.

Armed with these ideas, as an alternative of deploying our app with Kubernetes straight, I wished to experiment slightly through the use of Azure Container Apps (ACA). It is a serverless platform constructed on high of Kubernetes that abstracts away a few of its complexity.

With a single command, we will create a Container App Setting, which, beneath the hood, is an invisible Kubernetes Cluster managed by Azure. Inside this Setting, we will run a container as a Container App that Azure internally manages as Kubernetes Deployments, Providers, and Pods. See article 1 and article 2 for detailed comparisons.

A Container App Setting additionally auto-creates:

1. An invisible Envoy Ingress Controller that routes requests to inner Apps and handles HTTPS and App auto-scaling primarily based on request quantity.
2. An exterior Public IP deal with and Azure Load Balancer that routes exterior site visitors to the Ingress Controller that in flip routes it to Apps (sounds much like a Kubernetes LoadBalancer Service, eh?).
3. An Azure-generated URL for every Container App that’s publicly accessible over the web or inner, primarily based on its Ingress config.

This provides us all the pieces we have to run our containers at scale. The one factor lacking is a database. We’ll use an Azure-managed PostgreSQL server as an alternative of deploying our personal container, as a result of it’s simpler, extra dependable and scalable. Our native Nginx reverse proxy container can also be out of date as a result of ACA mechanically deploys an Envoy Ingress Controller.

It’s attention-grabbing to notice that we actually don’t have to vary a single line of code in our native utility, we will simply deal with it as a bunch of containers!

Here’s a diagram of the complete cloud structure for our chat utility that accommodates all our Azure assets. Let’s take a excessive stage take a look at how a consumer request flows by way of the system.

1. Consumer sends HTTPS request to chat.jorisbaan.nl.
2. A Public DNS server like Google DNS resolves this area title to an Azure Public IP deal with.
3. The Azure Load Balancer on this IP deal with routes the request to the (for us invisible) Envoy Ingress Controller.
4. The Ingress Controller routes the request to UI Container App, who routes it to one in all its Replicas the place a UI internet server is operating.
5. The UI internet server makes requests to the database API and language mannequin API Apps, who each route it to one in all their Replicas.
6. A database API reproduction queries the PostgreSQL server hostname. The Azure Personal DNS Zone resolves the hostname to the PostgreSQL server’s IP deal with.

So, how will we really create all this? Reasonably than clicking round within the Azure Portal, infrastructure-as-code instruments like Terraform are greatest to create and handle cloud assets. Nevertheless, for simplicity, I’ll as an alternative use the Azure CLI to create a bash script that deploys our complete utility step-by-step. You will discover the complete deployment script together with setting variables here 🤖. We’ll undergo it step-by-step now.

We want an Azure account (I’m utilizing a free education account), a clone of the https://github.com/jsbaan/ai-app-from-scratch repo, Docker to construct and push the container photographs, the downloaded model, and the Azure CLI to start out creating cloud assets.

We first create a useful resource group so our assets are simpler to search out, handle and delete. The --location parameter refers back to the bodily datacenter we’ll use to deploy our app’s infrastructure. Ideally, it’s near our customers. We then create a personal digital community with 256 IP addresses to isolate, safe and join our database server and Container Apps.

brew replace && brew set up azure-cli # for macos
echo "Create useful resource group"
az group create 
--name $RESOURCE_GROUP 
--location "$LOCATION"
echo "Create VNET with 256 IP addresses"
az community vnet create 
--resource-group $RESOURCE_GROUP 
--name $VNET 
--address-prefix 10.0.0.0/24 
--location $LOCATION

Relying on the {hardware}, an Azure-managed PostgreSQL database server prices about $13 to $7000 a month. To speak with Container Apps, we put the DB server throughout the similar non-public digital community however in its personal subnet. A subnet is a devoted vary of IP addresses that may have its personal safety and routing guidelines.

We create the Azure PostgreSQL Versatile Server with non-public entry. This implies solely assets throughout the similar digital community can attain it. Azure mechanically creates a Personal DNS Zone that manages a hostname for the database that resolves to its IP deal with. The database API will later use this hostname to connect with the database server.

We’ll randomly generate the database credentials and retailer them in a safe place: Azure KeyVault.

echo "Create subnet for DB with 128 IP addresses"
az community vnet subnet create 
--resource-group $RESOURCE_GROUP 
--name $DB_SUBNET 
--vnet-name $VNET 
--address-prefix 10.0.0.128/25
echo "Create a key vault to securely retailer and retrieve secrets and techniques, 
just like the db password"
az keyvault create 
--name $KEYVAULT 
--resource-group $RESOURCE_GROUP 
--location $LOCATION
echo "Give myself entry to the important thing vault so I can retailer and retrieve 
the db password"
az function project create 
--role "Key Vault Secrets and techniques Officer" 
--assignee $EMAIL 
--scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/suppliers/Microsoft.KeyVault/vaults/$KEYVAULT"
echo "Retailer random db username and password in the important thing vault"
az keyvault secret set 
--name postgres-username 
--vault-name $KEYVAULT  
--value $(openssl rand -base64 12 | tr -dc 'a-zA-Z' | head -c 12)
az keyvault secret set 
--name postgres-password 
--vault-name $KEYVAULT  
--value $(openssl rand -base64 16)
echo "Whereas we're at it, let's already retailer a secret session key for the UI"
az keyvault secret set 
--name session-key 
--vault-name $KEYVAULT  
--value $(openssl rand -base64 16)
echo "Create PostgreSQL versatile server in our VNET in its personal subnet. 
Auto-creates Personal DS Zone."
POSTGRES_USERNAME=$(az keyvault secret present --name postgres-username --vault-name $KEYVAULT --query "worth" --output tsv)
POSTGRES_PASSWORD=$(az keyvault secret present --name postgres-password --vault-name $KEYVAULT --query "worth" --output tsv)
az postgres flexible-server create 
--resource-group $RESOURCE_GROUP 
--name $DB_SERVER 
--vnet $VNET 
--subnet $DB_SUBNET 
--location $LOCATION 
--admin-user $POSTGRES_USERNAME 
--admin-password $POSTGRES_PASSWORD 
--sku-name Standard_B1ms 
--tier Burstable 
--storage-size 32 
--version 16 
--yes

With the community and database in place, let’s deploy the infrastructure to run containers — the Container App Setting (recall, it is a Kubernetes cluster beneath the hood).

We create one other subnet with 128 IP addresses and delegate its administration to the Container App Setting. The subnet ought to be large enough for every ten new replicas to get a new IP address within the subrange. We are able to then create the Setting. That is only a single command with out a lot configuration.

echo "Create subnet for ACA with 128 IP addresses."
az community vnet subnet create 
--resource-group $RESOURCE_GROUP 
--name $ACA_SUBNET 
--vnet-name $VNET 
--address-prefix 10.0.0.0/25
echo "Delegate the subnet to ACA"
az community vnet subnet replace 
--resource-group $RESOURCE_GROUP 
--vnet-name $VNET 
--name $ACA_SUBNET 
--delegations Microsoft.App/environments
echo "Receive the ID of our subnet"
ACA_SUBNET_ID=$(az community vnet subnet present 
--resource-group $RESOURCE_GROUP 
--name $ACA_SUBNET 
--vnet-name $VNET 
--query id --output tsv)
echo "Create Container Apps Setting in our customized subnet.
By default, it has a Workload profile with Consumption plan."
az containerapp env create 
--resource-group $RESOURCE_GROUP 
--name $ACA_ENVIRONMENT 
--infrastructure-subnet-resource-id $ACA_SUBNET_ID 
--location $LOCATION

Every Container App wants a Docker picture to run. Let’s first setup a Container Registry, after which construct all our photographs domestically and push them to the registry. Word that we merely copied the mannequin file into the language mannequin picture utilizing its Dockerfile, so we don’t must mount exterior storage like we did for local deployment in part 1.

echo "Create container registry (ACR)"
az acr create 
--resource-group $RESOURCE_GROUP 
--name $ACR 
--sku Normal 
--admin-enabled true
echo "Login to ACR and push native photographs"
az acr login --name $ACR
docker construct --tag $ACR.azurecr.io/$DB_API $DB_API
docker push $ACR.azurecr.io/$DB_API
docker construct --tag $ACR.azurecr.io/$LM_API $LM_API
docker push $ACR.azurecr.io/$LM_API
docker construct --tag $ACR.azurecr.io/$UI $UI
docker push $ACR.azurecr.io/$UI

Now, onto deployment. To create Container Apps we specify their Setting, container registry, picture, and the port they’ll hearken to for requests. The ingress parameter regulates whether or not Container Apps could be reached from the skin world. Our two APIs are inner and due to this fact utterly remoted, with no public URL and no site visitors ever routed from the Envoy Ingress Controller. The UI is exterior and has a public URL, however sends inner HTTP requests over the digital community to our APIs. We cross these inner hostnames and db credentials as setting variables.

echo "Deploy DB API on Container Apps with the db credentials from the important thing 
vault as env vars. Safer is to make use of a managed id that permits the 
container itself to retrieve them from the important thing vault. However for simplicity we 
merely fetch it ourselves utilizing the CLI."
POSTGRES_USERNAME=$(az keyvault secret present --name postgres-username --vault-name $KEYVAULT --query "worth" --output tsv)
POSTGRES_PASSWORD=$(az keyvault secret present --name postgres-password --vault-name $KEYVAULT --query "worth" --output tsv)
az containerapp create --name $DB_API 
--resource-group $RESOURCE_GROUP 
--environment $ACA_ENVIRONMENT 
--registry-server $ACR.azurecr.io 
--image $ACR.azurecr.io/$DB_API 
--target-port 80 
--ingress inner 
--env-vars "POSTGRES_HOST=$DB_SERVER.postgres.database.azure.com" "POSTGRES_USERNAME=$POSTGRES_USERNAME" "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" 
--min-replicas 1 
--max-replicas 5 
--cpu 0.5 
--memory 1
echo "Deploy UI on Container Apps, and retrieve the key random session 
key the UI makes use of to encrypt session cookies"
SESSION_KEY=$(az keyvault secret present --name session-key --vault-name $KEYVAULT --query "worth" --output tsv)
az containerapp create --name $UI 
--resource-group $RESOURCE_GROUP 
--environment $ACA_ENVIRONMENT 
--registry-server $ACR.azurecr.io 
--image $ACR.azurecr.io/$UI 
--target-port 80 
--ingress exterior 
--env-vars "db_api_url=http://$DB_API" "lm_api_url=http://$LM_API" "session_key=$SESSION_KEY" 
--min-replicas 1 
--max-replicas 5 
--cpu 0.5 
--memory 1 
echo "Deploy LM API on Container Apps"
az containerapp create --name $LM_API 
--resource-group $RESOURCE_GROUP 
--environment $ACA_ENVIRONMENT 
--registry-server $ACR.azurecr.io 
--image $ACR.azurecr.io/$LM_API 
--target-port 80 
--ingress inner 
--min-replicas 1 
--max-replicas 5 
--cpu 2 
--memory 4 
--scale-rule-name my-http-rule 
--scale-rule-http-concurrency 2

Let’s check out how our Container Apps they scale. Container Apps can scale to zero, which implies they’ve zero replicas and cease operating (and cease incurring prices). It is a function of the serverless paradigm, the place infrastructure is provisioned on demand. The invisible Envoy proxy handles scaling primarily based on triggers, like concurrent HTTP requests. Spawning new replicas might take a while, which is known as cold-start. We set the minimal variety of replicas to 1 to keep away from chilly begins and the ensuing timeout errors for first requests.

The default scaling rule creates a brand new reproduction each time an present reproduction receives 10 concurrent HTTP requests. This is applicable to the UI and the database API. To check whether or not this scaling rule is smart, we must carry out load testing to simulate actual consumer site visitors and see what every Container App reproduction can deal with individually. My guess is that they will deal with much more concurrent request than 10, and we might chill out the rule.

Even with our small, quantized language mannequin, inference requires rather more compute than a easy FastAPI app. The inference server handles incoming requests sequentially, and the default Container App assets of 0.5 digital CPU cores and 1GB reminiscence lead to very sluggish response occasions: as much as 30 seconds for producing 128 tokens with a context window of 1024 (these parameters are outlined within the LM API’s Dockerfile).

Rising vCPU to 2 and reminiscence to 4GB provides a lot better inference pace, and handles about 10 requests inside 30 seconds. I configured the http scaling rule very tightly at 2 concurrent requests, so each time 2 customers chat on the similar time, the LM API will scale out.

With 5 most replicas, I feel it will permit for roughly 10–40 concurrent customers, relying on the size of the chat histories. Now, clearly, this isn’t very large-scale, however with a better price range, we might improve vCPUs, reminiscence and the variety of replicas. Finally we would want to maneuver to GPU-based inference. Extra on that later.

The mechanically generated URL from the UI App seems to be like https://chat-ui.purplepebble-ac46ada4.germanywestcentral.azurecontainerapps.io/. This isn’t very memorable, so I wish to make our app accessible as subdomain on my web site: chat.jorisbaan.nl.

I merely add two DNS data on my area registrar portal (like GoDaddy). A CNAME document that hyperlinks my chat subdomain to the UI’s URL, and TXT document to show possession of the subdomain to Azure and acquire a TLS certificates.

# Receive UI URL and verification code
URL=$(az containerapp present -n $UI -g $RESOURCE_GROUP -o tsv --query "properties.configuration.ingress.fqdn")
VERIFICATION_CODE=$(az containerapp present -n $UI -g $RESOURCE_GROUP -o tsv --query "properties.customDomainVerificationId")
# Add a CNAME document with the URL and a TXT document with the verification code to area registrar
# (Do that manually)
# Add customized area title to UI App
az containerapp hostname add --hostname chat.jorisbaan.nl -g $RESOURCE_GROUP -n $UI
# Configure managed certificates for HTTPS
az containerapp hostname bind --hostname chat.jorisbaan.nl -g $RESOURCE_GROUP -n $UI --environment $ACA_ENVIRONMENT --validation-method CNAME

Container Apps manages a free TLS certificates for my subdomain so long as the CNAME document factors on to the container’s area title.

The general public URL for the UI adjustments each time I tear down and redeploy an Setting. We might use a fancier service like Azure Front Door or Application Gateway to get a steady URL and act as reverse proxy with extra safety, international availability, and edge caching.

Now that the app is deployed, let’s take a look at an summary of all of the Azure assets it app makes use of. We created most of them ourselves, however Azure additionally mechanically created a Load balancer, Public IP, Personal DNS Zone, Community Watcher and Log Analytics workspace.

Some assets are free, others are free as much as a sure time or compute price range, which is a part of the rationale I selected them. The next assets incur the very best prices:

• Load Balancer (normal Tier): free for 1 month, then $18/month.
• Container Registry (normal Tier): free for 12 months, then $19/month.
• PostgreSQL Versatile Server (Burstable B1MS Compute Tier): free for 12 months, then a minimum of $13/month.
• Container App: Free for 50 CPU hours/month or 2M requests/month, then $10/month for an App with a single reproduction, 0.5 vCPUs and 1GB reminiscence. The LM API with 2vCPUs, 4GB reminiscence prices about $50 per 30 days for a single reproduction.

You possibly can see that the prices of this small (however scalable) app can rapidly add as much as tons of of {dollars} per 30 days, even with out a GPU server to run a stronger language mannequin! That’s the rationale why the app in all probability received’t be up if you’re studying this.

It additionally turns into clear that Azure Container Apps is costlier then I initially thought: it requires a standard-Tier Load balancer for automated exterior ingress, HTTPS and auto-scaling. We might get round this by disabling exterior ingress and deploying a less expensive various — like a VM with a customized reverse proxy, or a basic-Tier Load balancer. Nonetheless, a standard-tier Kubernetes cluster would have value a minimum of $150/month, so ACA could be cheaper at small scale.

Now, earlier than we wrap up, let’s take a look at just some of the numerous instructions to enhance this deployment.

Steady Integration & Steady Deployment. I might arrange a CI/CD pipeline that runs unit and integration exams and redeploys the app upon code adjustments. It may be triggered by a brand new git commit or merged pull request. This will even make it simpler to see when a service isn’t deployed correctly. I might additionally arrange monitoring and alerting to concentrate on points rapidly (like a crashing Container App occasion).

Decrease latency: the language mannequin server. I might load check the entire app — simulating real-world consumer site visitors — with one thing like Locust or Azure Load Testing. Even with out load testing, we have now an apparent bottleneck: the LM server. Small and quantized as it’s, it may well nonetheless take up fairly some time for prolonged solutions, with no concurrency. For extra customers it could be quicker and extra environment friendly to run a GPU inference server with a batching mechanism that collects a number of technology requests in a queue — maybe with Kafka — and runs batch inference on chunks.

With much more customers, we would need a number of GPU-based LM servers that devour from the identical queue. For GPU infrastructure I’d look into Azure Digital Machines or one thing extra fancy like Azure Machine Studying.

The llama.cpp inference engine is sweet for single-user CPU-based inference. When transferring to a GPU-server, I might look into inference engines extra appropriate to batch inference, like vLLM or Huggingface TGI. And, clearly, a greater (larger) mannequin for elevated response high quality — relying on the use case.

I hope this undertaking provides a glimpse of what an AI-powered internet app in manufacturing might seem like. I attempted to stability lifelike engineering with slicing about each nook to maintain it easy, low cost, comprehensible, and restrict my time and compute price range. Sadly, I can’t preserve the app reside for lengthy since it could rapidly value tons of of {dollars} per 30 days. If somebody may help with Azure credit to maintain the app operating, let me know!

Some closing ideas about utilizing managed companies: Though Azure Container Apps abstracts away a number of the Kubernetes complexity, it’s nonetheless extraordinarily helpful to have an understanding of the lower-level Kubernetes ideas. The mechanically created invisible infrastructure like Public IPs, Load balancers and ingress controllers add unexpected prices and make it obscure what’s occurring. Additionally, ACA documentation is proscribed in comparison with Kubernetes. Nevertheless, if you recognize what you’re doing, you possibly can set one thing up in a short time.

I closely relied on the Azure docs, and the ACA docs particularly. Due to Dennis Ulmer for proofreading and Lucas de Haas for helpful dialogue.

I experimented a bit extra with AI instruments in comparison with half 1. I used Pycharm’s CoPilot plugin for code completion and had fairly some back-and-forth with ChatGPT to study concerning the Azure or Kubernetes ecosystem, and to spar about bugs. I double-checked all the pieces within the docs and a lot of the data was strong. Like half 1, I didn’t use AI to put in writing this submit, although I did use ChatGPT to paraphrase some bad-running sentences.