If me for some time, you most likely know I began my profession as a QA engineer earlier than transitioning into the world of knowledge analytics. I didn’t go to highschool for it, didn’t have a mentor, and didn’t land in a proper coaching program. All the things I do know in the present day—from SQL to modeling to storytelling with knowledge—is self-taught. And consider me, it’s been a journey of trial, error, studying, and re-learning.
The Dilemma That Modified My Profession
A couple of years in the past, I began fascinated with switching organizations. Like many individuals in fast-evolving tech roles, I confronted a surprisingly troublesome query:
What position am I really doing? Which roles ought to I apply for?
On paper, I used to be a Information Analyst. However in actuality, my position straddled a number of features: writing SQL pipelines, constructing dashboards, defining KPIs, and digging into product analytics. I wasn’t certain whether or not I needs to be making use of for Analyst roles, BI roles, or one thing solely completely different.
To make issues worse, again then, job titles had been obscure, and job descriptions had been bloated with buzzwords. You’d discover a posting titled “Information Analyst” that listed necessities like:
- Construct ML pipelines
- Write complicated ETL scripts
- Keep knowledge lakes
- Create dashboards
- Current executive-level insights
- And oh, by the way in which, be nice at stakeholder administration
It was overwhelming and complicated. And I do know I’m not alone on this.
Quick ahead to in the present day: fortunately, issues are evolving. There’s nonetheless overlap between roles, however organizations have began to outline them extra clearly. On this article, I wish to break down the actual variations between knowledge roles, by means of the lens of a real-world instance.
A Actual-World State of affairs: Meet Quikee
Let’s think about a fictional quick-commerce startup known as Quikee, launching throughout a number of Indian cities. Their worth proposition? Ship groceries and necessities inside 10 minutes.
Prospects place orders by means of the app or web site. Behind the scenes, there are micro-warehouses (additionally known as “darkish shops”) throughout cities, and a fleet of supply companions who make these lightning-fast deliveries.
Now, let’s stroll by means of the info wants of this firm—from the second an order is positioned, to the dashboards executives use of their Monday morning conferences.
Step 1: Capturing and Storing Uncooked Information
The second a buyer locations an order, transactional knowledge is generated:
- Timestamps
- Order ID
- Gadgets ordered
- Worth
- Low cost codes
- Buyer location
- Fee technique
- Assigned supply accomplice
Let’s assume Quikee makes use of Amazon Kinesis to stream this knowledge in actual time to an S3 knowledge lake. That stream is high-volume, time-sensitive, and essential for enterprise monitoring.
However right here’s the catch: uncooked knowledge is messy. You’ll be able to’t use it straight for decision-making.
So what occurs subsequent?
Step 2: Constructing Information Pipelines
Enter the Information Engineers.
They’re answerable for:
- Ingesting real-time knowledge
- Validating schema consistency
- Dealing with failures and retries
- Writing pipelines to maneuver knowledge from S3 into a knowledge warehouse (say, Snowflake or Redshift)
That is the place ETL (Extract, Rework, Load) or ELT pipelines come into play. Information engineers clear, format, and construction the info to make it queryable.
For instance, an order desk may get break up into:
- Orders → One row per order
- Order_Items → One row per merchandise in an order
- Funds → One row per fee try
At this stage, uncooked logs are was structured tables that analysts can work with.
Step 3: Dimensional Modeling & OLAP
As management begins asking strategic questions like:
- “Which metropolis brings in essentially the most income?”
- “Which retailer is underperforming?”
- “What’s our common supply time by zone?”
…it turns into clear that querying transactional knowledge straight received’t scale.
That’s the place dimensional modeling is available in.
As a substitute of flat, uncooked tables, knowledge is structured into Truth and Dimension Tables.
🔸 Truth Tables
- Massive, quantitative knowledge tables which comprise international keys together with measures and metrics (Properly, more often than not. There are factless truth tables as properly which do not need any measures).
- Examples:
fact_orders,fact_payments,fact_deliveries - Comprise metrics like income, order depend, supply time
🔹 Dimension Tables
- Smaller, descriptive tables that assist perceive the info in a truth desk
- Examples:
dim_store,dim_product,dim_customer,dim_delivery_agent - Assist filter, group, and be part of details for deeper insights
This construction allows OLAP—quick, analytical querying throughout a number of dimensions. For instance, now you can run queries like:
“Present me common supply time by retailer and hour of day, over the past 7 days.”
This step is completed by Information Engineers at a lot of the organisations however I did construct few Dim and Truth tables once I was working as a Business Intelligence Engineer at Amazon.
Step 4: Defining KPIs and Metrics
That is the place Analytics Engineers (or BI Engineers) shine.
They sit between the technical knowledge layer and enterprise customers. Their tasks usually embody:
- Defining KPIs (e.g., churn price, repeat buy %, time-to-fulfillment)
- Writing logic for complicated metrics (e.g., cohort retention, lively customers)
- Creating semantic fashions or metrics layers in instruments like dbt or Looker
- Guaranteeing constant definitions throughout the corporate
For instance, at Amazon, our group didn’t question uncooked knowledge to calculate income each time. As a substitute, we created pre-aggregated truth tables at each day, weekly, and month-to-month grains. That means, dashboards loaded quicker, and metrics stayed constant throughout groups.
Analytics Engineers act as translators between engineering and the enterprise—defining what we measure and how we measure it.
Step 5: Evaluation, Reporting & Storytelling
Now comes the position of the Data Analyst.
Armed with clear, modeled knowledge, they concentrate on answering actual enterprise questions like:
- “Why did retention drop in Bangalore final month?”
- “Which coupon codes drive essentially the most new customers?”
- “What are the highest merchandise reordered within the first 30 days?”
They construct dashboards in instruments like Tableau, Energy BI, or Looker. They run ad-hoc SQL queries. They dive into A/B take a look at outcomes, consumer habits traits, and marketing campaign effectiveness.
However above all, they inform tales with knowledge—making complicated numbers comprehensible and actionable for stakeholders.
Who’s Who?
TL;DR: The place Do You Match?
Right here’s how I give it some thought:
- Love constructing sturdy pipelines and fixing scalability issues? → You’re a Information Engineer
- Love defining enterprise metrics and organizing complicated datasets? → You’re an Analytics Engineer
- Love uncovering insights and storytelling with knowledge? → You’re a Information Analyst
After all, real-world roles usually mix these. Particularly at smaller corporations, you could put on a number of hats. And that’s okay.
The hot button is not the title—however the place you add essentially the most worth and what energizes you.
Ultimate Ideas
It took me a very long time to know what I really do—not simply what my job title says. And in the event you’ve ever felt that confusion, you’re not alone.
In the present day, I can clearly say I function on the intersection of knowledge modeling, enterprise logic, and storytelling—a candy spot between analytics and engineering. And I’ve discovered that the flexibility to attach the dots is extra essential than becoming into an ideal field.
In case you’ve walked an analogous path—or put on a number of hats in your position—I’d love to listen to your story.
Drop a remark 👇 or share this with somebody figuring it out too.
