Become a Better Data Scientist with These Prompt Engineering Tips and Tricks

I’m sharing with you my favourite prompts and immediate engineering ideas that assist me sort out Information Science and AI duties.

As Immediate Engineering is rising as a required talent in most job descriptions, I assumed it will be helpful to share with you some ideas and tips to enhance your Information Science workflows.

We’re speaking right here about particular prompts for cleansing knowledge, exploratory knowledge evaluation, and have engineering.

That is the first of a collection of 3 articles I’m going to put in writing about Immediate Engineering for Information Science:

• Half 1: Immediate Engineering for Planning, Cleansing, and EDA (this text)

• Half 2: Immediate Engineering for Options, Modeling, and Analysis
• Half 3: Immediate Engineering for Docs, DevOps, and Studying

👉All of the prompts on this article can be found on the finish of this text as a cheat sheet 😉

On this article:

1. Why Immediate Engineering Is a Superpower for DSs
2. The DS Lifecycle, Reimagined with LLMs
3. Immediate Engineering for Planning, Cleansing, and EDA

Why Immediate Engineering is a superpower for DSs

I do know, Immediate Engineering sounds similar to a trending buzzword nowadays. I used to assume that once I began listening to the time period.

I’d see it in all places and assume: it’s simply writing a immediate. Why are individuals so overhyped about it? What might be so tough about it?

After testing a number of prompts and watching a number of tutorials, I now perceive that it is among the most helpful (and in addition underestimated) expertise a knowledge scientist can purchase proper now.

It’s already widespread to see within the job descriptions that immediate engineering is among the required expertise for the job.

Mirror with me: how usually do you ask ChatGPT/Claude/your fav chatbot that will help you re-write code, clear knowledge, or simply brainstorm a venture or some concepts you could have? And the way usually do you get helpful and significant, non-generical solutions?

Immediate Engineering is the artwork (and science) of getting giant language fashions (LLMs) like GPT-4 or Claude to truly do what you need, if you need it, in a manner that is smart in your workflow.

As a result of right here’s the factor: LLMs are in all places now.
In your notebooks.
In your IDE.
In your BI dashboards.
In your code evaluate instruments.

They usually’re solely getting higher.

As knowledge science work will get extra complicated—extra instruments, extra expectations, extra pipelines—with the ability to speak to AI in a exact, structured manner turns into a critical benefit.

I see immediate engineering as a superpower. Not only for junior of us attempting to hurry issues up, however for knowledgeable knowledge scientists who need to work smarter.

On this collection, I’ll present you the way immediate engineering can assist you at each stage of the information science lifecycle—from brainstorming and cleansing, to modeling, analysis, documentation, and past.

The DS lifecycle, reimagined with LLMs

If you end up constructing a Information Science or Machine Studying venture, it actually looks like a complete journey.

From determining what drawback you’re fixing, all the best way to creating a stakeholder perceive why your mannequin issues (with out exhibiting them a single line of code).

Right here’s a typical DS lifecycle:

• You plan & brainstorm, to determining the appropriate inquiries to ask and what issues should be solved
• You collect knowledge, or knowledge is gathered for you.
• You clear knowledge and preprocess it – this the place you spend 80% of your time (and endurance!).
• The enjoyable begins: you begin making exploratory knowledge evaluation (EDA) – getting a really feel for the information, discovering tales in numbers.
• You begin constructing: function engineering and modeling begins.
• Then, you consider and validate if issues really do work.
• Lastly, you doc and report your findings, so others can perceive it too.

Now… think about having a useful assistant that:

• Writes strong starter code in seconds,
• Suggests higher methods to clear or visualize knowledge,
• Helps you clarify mannequin efficiency to non-tech individuals,
• Reminds you to examine for belongings you would possibly miss (like knowledge leakage or class imbalance),
• And is obtainable 24/7.

That’s what LLMs might be, in case you immediate them the appropriate manner!

They received’t substitute you, don’t worry. They aren’t capable of do it!

However they’ll and will certainly amplify you. You continue to have to know what you’re constructing and how (and why!), however now you could have an assistant that lets you do all of this in a better manner.

Now I’ll present you the way immediate engineering can amplify you as a knowledge scientist.

Immediate Engineering for planning, cleansing, and EDA

1. Planning & brainstorming: No extra clean pages

You’ve bought a dataset. You’ve bought a objective. Now what?

You possibly can immediate GPT-4 or Claude to listing steps for an end-to-end venture given a dataset description and objective.

This section is the place LLMs can already offer you a lift.

Instance: Planning an vitality consumption prediction venture

Right here’s an precise immediate I’ve used (with ChatGPT):

“You’re a senior knowledge scientist. I’ve an vitality consumption dataset (12,000 rows, hourly knowledge over 18 months) together with options like temperature, usage_kwh, area, and weekday.
Activity: Suggest a step-by-step venture plan to forecast future vitality consumption. Embody preprocessing steps, seasonality dealing with, function engineering concepts, and mannequin choices. We’ll be deploying a dashboard for inside stakeholders.”

This sort of structured immediate provides:

• Context (dataset dimension, variables, objective)
• Constraints (class imbalance)
• Hints at deployment

Notice: if you’re utilizing ChatGPT’s latest mannequin, o3-pro, make certain to offer it a lot of context. This new mannequin thrives if you feed it with full transcripts, docs, knowledge, and so forth.

An analogous Claude immediate would work, as Claude additionally favors express directions. Claude’s bigger context window even permits together with extra dataset schema particulars or examples if wanted, which may yield a extra tailor-made plan

I re-tested this immediate with o3-pro as I used to be curious to see the outcomes

The response from o3-pro was nothing lower than a full knowledge science venture plan, from cleansing and have engineering to mannequin choice and deployment, however extra importantly: with crucial choice factors, real looking timelines, and questions that problem our assumptions upfront.

Here’s a snapshot of the response:

Bonus technique: Make clear – Affirm – Full

When you want a extra complicated planning, there’s a trick referred to as “Make clear, Affirm, Full” that you should use earlier than the AI provides the ultimate plan.

You possibly can ask the mannequin to:

1. Make clear what it must know first
2. Affirm the appropriate strategy
3. Then full a full plan

For instance:

“I need to analyze late deliveries for our logistics community.
Earlier than giving an evaluation plan:

1. Make clear what knowledge or operational metrics could be related to supply delays
2. Affirm the perfect evaluation strategy for figuring out delay drivers
3. Then full an in depth venture plan (knowledge cleansing, function engineering, mannequin or evaluation strategies, and reporting steps).”

This strategy forces the LLM to first ask questions or state assumptions (e.g., about obtainable knowledge or metrics). This forces the mannequin to decelerate and assume, similar to we people do!

Information cleansing & preprocessing: Bye bye boilerplate

Now that the plan’s prepared, it’s time to roll up your sleeves. Cleansing knowledge is 80% of the job, and for positive not a enjoyable activity.

GPT-4 and Claude can each generate code snippets for widespread duties like dealing with lacking values or remodeling variables, given a very good immediate.

Instance: Write me some pandas code

Immediate:

“I’ve a DataFrame df with columns age, revenue, metropolis.
Some values are lacking, and there are revenue outliers.
Activity:

1. Drop rows the place metropolis is lacking
2. Fill lacking age with the median
3. Cap revenue outliers utilizing IQR technique
   Embody feedback within the code.”

Inside seconds, you get a code block with dropna(), fillna(), and the IQR logic, all with explanations.

Instance: Steerage on cleansing methods

You possibly can question conceptual recommendation as effectively.

Immediate:

“What are completely different approaches to deal with outliers in a monetary transactions dataset? Clarify when to make use of every and the professionals/cons.”

A immediate’s reply like this may output the a number of strategies particular to your area of interest, as a substitute of a one-size-fits-all answer.

This helps keep away from the simplistic and even deceptive recommendation one would possibly get from a too-general query (for instance, asking “greatest approach to deal with outliers” will in all probability output an oversimplified “take away all outliers” suggestion.

Strive few-shot prompting for Consistency

Want variable descriptions in a constant format?

Simply present the LLM how:

Immediate:

“Authentic: “Buyer age” → Standardized: “Age of buyer at time of transaction.”
Authentic: “purchase_amt” → Standardized: “Transaction quantity in USD.”

Now standardize:

• Authentic: “cust_tenure”
• Authentic: “item_ct” “

It follows the model completely. You should use this trick to standardize labels, outline options, and even describe mannequin steps later.

Exploratory knowledge evaluation (EDA): Ask higher questions

EDA is the place we begin asking, “What’s attention-grabbing right here?” and that is the place obscure prompts can actually damage.

A generic “analyze this dataset” will usually return… generic ideas.

Examples: EDA duties

“I’ve an e-commerce dataset with customer_id, product, date, and quantity.
I need to perceive:

1. Buy habits patterns
2. Merchandise usually purchased collectively
3. Adjustments in buying over time
   For every, recommend columns to research and Python strategies.”

The reply will in all probability embrace grouped stats, time tendencies, and even code snippets utilizing groupby(), seaborn, and market basket evaluation.

If you have already got synopsis statistics, you may even paste them and ask:

Immediate:

“Based mostly on these abstract stats, what stands out or what potential points ought to I look into?”.

GPT-4/Claude would possibly level out a excessive variance in a single function or a suspicious variety of lacking entries in one other. (Be cautious: the mannequin can solely infer from what you present; it might hallucinate patterns if requested to invest with out knowledge.)

Instance immediate: Guided EDA

“I’ve a dataset with 50 columns (mixture of numeric and categorical). Counsel an exploratory knowledge evaluation plan: listing 5 key analyses to carry out (e.g., distribution checks, correlations, and so forth.). For every, specify which particular columns or pairs of columns to take a look at, given I need to perceive gross sales efficiency drivers.”

This immediate is restricted in regards to the objective (gross sales drivers) so the AI would possibly advocate, say, analyzing gross sales vs marketing_spend scatter plot, a time collection plot if date is current, and so forth., custom-made to “efficiency drivers.” Apart from, the structured output (listing of 5 analyses) shall be simpler to comply with than a protracted paragraph.

Instance: Let the LLM clarify your plots

You possibly can even ask:

“What can a field plot of revenue by occupation inform me?”

It should clarify quartiles, IQR, and what outliers would possibly imply. That is extra useful when mentoring juniors or getting ready slides for reviews, displays, and so forth.

Pitfalls to watch out about

This early stage is the place most individuals misuse LLMs. Right here’s what to look at for:

Broad or obscure prompts

When you say: “What ought to I do with this dataset?”
You’ll get one thing like: “Clear the information, analyze it, construct a mannequin.”

As an alternative, at all times embrace:

• Context (knowledge kind, dimension, variables)
• Objectives (predict churn, analyze gross sales, and so forth.)
• Constraints (imbalanced knowledge, lacking values, area guidelines)

Blind belief within the output

Sure, LLMs write code quick. However check all the things.

I as soon as requested for code to impute lacking values. It used fillna() for all columns, together with the explicit ones. It didn’t examine knowledge sorts, and neither did I… the primary time. 😬

Privateness and leakage

When you’re working with actual firm knowledge, don’t paste uncooked rows into the immediate except you’re utilizing a personal/enterprise mannequin. Describe the information abstractly as a substitute. And even higher, seek the advice of your supervisor about this matter.

Thanks for studying!

👉 Seize the Immediate Engineering Cheat Sheet with all prompts of this text organized. I’ll ship it to you if you subscribe to Sara’s AI Automation Digest. You’ll additionally get entry to an AI device library and my free AI automation publication each week!

Thanks for studying! 😉

I supply mentorship on profession development and transition here.

If you wish to assist my work, you’ll be able to buy me my favorite coffee: a cappuccino. 😊

References

A Guide to Using ChatGPT For Data Science Projects | DataCamp

(29) Prompt Engineering for Document Analysis: What I Learned Moving from GPT-4 to Claude 4 🧠 | LinkedIn

Prompt Engineering for Data Professionals – Dataquest

Geeks for Geeks