How AI Is Rewriting the Day-to-Day of Data Scientists

articles, I’ve explored and in contrast many AI instruments, for instance, Google’s Data Science Agent, ChatGPT vs. Claude vs. Gemini for Data Science, DeepSeek V3, and so on. Nonetheless, that is solely a small subset of all of the AI instruments out there for Data Science. Simply to call a number of that I used at work:

• OpenAI API: I take advantage of it to categorize and summarize buyer suggestions and floor product ache factors (see my tutorial article).
• ChatGPT and Gemini: They assist me draft Slack messages and emails, write evaluation studies, and even efficiency opinions.
• Glean AI: I used Glean AI to seek out solutions throughout inner documentation and communications rapidly.
• Cursor and Copilot: I get pleasure from simply urgent tab-tab to auto-complete code and feedback.
• Hex Magic: I take advantage of Hex for collaborative knowledge notebooks at work. Additionally they provide a characteristic referred to as Hex Magic to put in writing code and repair bugs utilizing conversational AI.
• Snowflake Cortex: Cortex AI permits customers to name Llm endpoints, construct RAG and text-to-SQL companies utilizing knowledge in Snowflake.

I’m positive you’ll be able to add much more to this listing, and new AI instruments are being launched on daily basis. It’s virtually inconceivable to get a whole listing at this level. Subsequently, on this article, I wish to take one step again and give attention to a much bigger query: what do we actually want as knowledge professionals, and the way AI will help. 

Within the part under, I’ll give attention to two predominant instructions — eliminating low-value duties and accelerating high-value work. 

1. Eliminating Low-Worth Duties

I grew to become an information scientist as a result of I actually get pleasure from uncovering enterprise insights from advanced knowledge and driving enterprise selections. Nonetheless, having labored within the trade for over seven years now, I’ve to confess that not all of the work is as thrilling as I had hoped. Earlier than conducting superior analyses or constructing machine studying fashions, there are lots of low-value work streams which are unavoidable each day — and in lots of circumstances, it’s as a result of we don’t have the correct tooling to empower our stakeholders for self-serve analytics. Let’s have a look at the place we’re right now and the best state:

Present state: We work as knowledge interpreters and gatekeepers (generally “SQL monkeys”)

• Easy knowledge pull requests come to me and my crew on Slack each week asking, “What was the GMV final month?” “Are you able to pull the listing of shoppers who meet these standards?” “Are you able to assist me fill on this quantity on the deck that I must current tomorrow?” 
• BI instruments don’t assist self-service use circumstances nicely. We adopted BI instruments like Looker and Tableau so stakeholders can discover the info and monitor the metrics simply. However the actuality is that there’s at all times a trade-off between simplicity and self-servability. Generally we make the dashboards straightforward to grasp with a number of metrics, however they’ll solely fulfill a number of use circumstances. In the meantime, if we make the instrument very customizable with the potential to discover the metrics and underlying knowledge freely, stakeholders might discover the instrument complicated and lack the boldness to make use of it, and within the worst case, the info is pulled and interpreted within the improper approach.  
• Documentation is sparse or outdated. It is a widespread state of affairs, however may very well be brought on by completely different causes — perhaps we transfer quick and give attention to delivering outcomes, or there isn’t a nice knowledge documentation and governance insurance policies in place. In consequence, tribal information turns into the bottleneck for individuals outdoors of the info crew to make use of the info.

Ideally suited state: Empower stakeholders to self-serve so we are able to decrease low-value work

• Stakeholders can do easy knowledge pulls and reply fundamental knowledge questions simply and confidently.
• Information groups spend much less time on repetitive reporting or one-off fundamental queries.
• Dashboards are discoverable, interpretable, and actionable with out hand-holding.

So, to get nearer to the best state, what position can AI play right here? From what I’ve noticed, these are the widespread instructions AI instruments are going to shut the hole:

1. Question knowledge with pure language (Textual content-to-SQL): One method to decrease the technical barrier is to allow stakeholders to question the info with pure language. There are many Textual content-to-SQL efforts within the trade:
   • For instance, Snowflake is one firm that has made numerous progress in Text2SQL models and began integrating the potential into its product. 
   • Many corporations (together with mine) additionally explored in-house Text2SQL options. For instance, Uber shared their journey with Uber’s QueryGPT to make knowledge querying extra accessible for his or her Operations crew. This text defined intimately how Uber designed a multi-agent structure for question technology. In the meantime, it additionally surfaced main challenges on this space, together with precisely decoding consumer intent, dealing with massive desk schemas, and avoiding hallucinations and so on. 
   • Truthfully, to make Textual content-to-SQL work, there’s a very excessive bar as it’s important to make the question correct — even when the instrument fails simply as soon as, it might wreck the belief and ultimately stakeholders will come again to you to validate the queries (then you want to learn+rewrite the queries, which just about double the work 🙁). To date, I haven’t discovered a Textual content-to-SQL mannequin or instrument that works completely. I solely see it achievable when you’re querying from a really small subset of well-documented core datasets for particular and standardized use circumstances, however it is rather laborious to scale to all of the out there knowledge and completely different enterprise situations. 
   • However in fact, given the big quantity of funding on this space and fast improvement in AI, I’m positive we’ll get nearer and nearer to correct and scalable Textual content-to-SQL options. 
2. Chat-based BI assistant: One other widespread space to enhance stakeholders’ expertise with BI instruments is the chat-based BI assistant. This truly takes one step additional than Textual content-to-SQL — as a substitute of producing a SQL question primarily based on a consumer immediate, it responds within the format of a visualization plus textual content abstract. 
   • Gemini in Looker is an instance right here. Looker is owned by Google, so it is rather pure for them to combine with Gemini. One other benefit for Looker to construct their AI characteristic is that knowledge fields are already documented within the LookML semantic layer, with widespread joins outlined and well-liked metrics inbuilt dashboards. Subsequently, it has numerous nice knowledge to study from. Gemini permits customers to regulate Looker dashboards, ask questions concerning the knowledge, and even construct customized knowledge brokers for Conversational Analytics. Although primarily based on my restricted experimentation with the instrument, it instances out typically and fails to reply easy questions generally. Let me know when you’ve got a unique expertise and have made it work…
   • Tableau additionally launched the same characteristic, Tableau AI. I haven’t used it myself, however primarily based on the demo, it helps the info crew to arrange knowledge and make dashboards rapidly utilizing pure language, and summarise knowledge insights into “Tableau Pulse” for stakeholders to simply spot metric modifications and irregular tendencies.   
3. Information Catalog Instruments: AI can even assist with the problem of sparse or outdated knowledge documentation. 
   • Throughout one inner hackathon, I bear in mind one mission from our knowledge engineers was to make use of LLM to extend desk documentation protection. AI is ready to learn the codebase and describe the columns accordingly with excessive accuracy generally, so it might assist enhance documentation rapidly with restricted human validation and changes. 
   • Equally, when my crew creates new tables, now we have began to ask Cursor to put in writing the desk documentation YAML recordsdata to save lots of us time with high-quality output. 
   • There are additionally numerous knowledge catalogs and governance instruments which were built-in with AI. After I google “ai knowledge catalog”, I see the logos of knowledge catalog instruments like Atlan, Alation, Collibra, Informatica, and so on (disclaimer: I’ve used none of them..). That is clearly an trade pattern. 

2. Accelerating high-value work

Now that we’ve talked about how AI might assist with eliminating low-value duties, let’s talk about the way it can speed up high-value knowledge tasks. Right here, high-value work refers to knowledge tasks that mix technical excellence with enterprise context, and drive significant influence by way of cross-functional collaboration. For instance, a deep dive evaluation that understands product utilization patterns and results in product modifications, or a churn prediction mannequin to determine churn-risk clients and ends in churn-prevention initiatives. Let’s evaluate the present state and the best future:

Present state: Productivity bottlenecks exist in on a regular basis workflows 

• EDA is time-consuming. This step is vital to get an preliminary understanding of the info, nevertheless it might take a very long time to conduct all of the univariate and multivariate analyses.
• Time misplaced to coding and debugging. Let’s be sincere — nobody can bear in mind all of the numpy and pandas syntax and sklearn mannequin parameters. We continuously must lookup documentation whereas coding.
• Wealthy unstructured knowledge is just not absolutely utilized. Enterprise generates numerous textual content knowledge on daily basis from surveys, assist tickets, and opinions. However easy methods to extract insights scalably stays a problem.

Ideally suited state: Information scientists give attention to deep pondering, not syntax 

• Writing code feels quicker with out the interruption to lookup syntax.
• Analysts spend extra time decoding outcomes, much less time wrangling knowledge.
• Unstructured knowledge is not a blocker and may be rapidly analyzed.

Seeing the best state, I’m positive you have already got some AI instrument candidates in thoughts. Let’s see how AI can affect or is already making a distinction: 

1. AI coding and debugging assistants. I feel that is by far essentially the most adopted sort of AI instrument for anybody who codes. And we’re already seeing it iterating.
   • When LLM chatbots like ChatGPT and Claude got here out, engineers realized they might simply throw their syntax questions or error messages to the chatbot with high-accuracy solutions. That is nonetheless an interruption to the coding workflow, however significantly better than clicking by way of a dozen StackOverflow tabs — this already seems like final century. 
   • Later, we see an increasing number of built-in AI coding instruments popping up — GitHub Copilot and Cursor combine together with your code editor and might learn by way of your codebase to proactively counsel code completions and debug points inside your IDE. 
   • As I briefly talked about at first, knowledge instruments like Snowflake and Hex additionally began to embed AI coding assistants to assist knowledge analysts and knowledge scientists write code simply. 
2. AI for EDA and evaluation. That is considerably much like the Chat-based BI assistant instruments I discussed above, however their aim is extra formidable — they begin with the uncooked datasets and goal to automate the entire evaluation cycle of knowledge cleansing, pre-processing, exploratory evaluation, and generally even modeling. These are the instruments often marketed as “changing knowledge analysts” (however are they?).
   • Google Data Science Agent is a really spectacular new instrument that may generate an entire Jupyter Pocket book with a easy immediate. I just lately wrote an article exhibiting what it might do and what it can’t. Briefly, it might rapidly spin up a well-structured and functioning Jupyter Pocket book primarily based on a customizable execution plan. Nonetheless, it’s lacking the capabilities of modifying the Jupyter Pocket book primarily based on follow-up questions, nonetheless requires somebody with stable knowledge science information to audit the strategies and make guide iterations, and desires a transparent knowledge drawback assertion with clear and well-documented datasets. Subsequently, I view it as an amazing instrument to free us a while on starter code, as a substitute of threatening our jobs.
   • ChatGPT’s Data Analyst tool will also be categorized below this space. It permits customers to add a dataset and chat with it to get their evaluation performed, visualizations generated, and questions answered. You will discover my prior article discussing its capabilities here. It additionally faces related challenges and works higher as an EDA helper as a substitute of changing knowledge analysts.
3. Simple-to-use and scalable NLP capabilities. LLM is nice at conversations. Subsequently, NLP is made exponentially simpler with LLM right now.
   • My firm hosts an inner hackathon yearly. I bear in mind my hackathon mission three years in the past was to attempt BERT and different conventional matter modeling strategies to investigate NPS survey responses, which was enjoyable however truthfully very laborious to make it correct and significant for the enterprise. Then two years in the past, throughout the hackathon, we tried OpenAI API to categorize and summarise those self same suggestions knowledge — it labored like magic as you are able to do high-accuracy matter modeling, sentiment evaluation, suggestions categorization all simply in a single API name, and the outputs nicely match into our enterprise context primarily based on the system immediate. We later constructed an inner pipeline that scaled simply to textual content knowledge throughout survey responses, assist tickets, Gross sales calls, consumer analysis notes, and so on., and it has develop into the centralized buyer suggestions hub and knowledgeable our product roadmap. You will discover extra in this tech blog.
   • There are additionally numerous new corporations constructing packaged AI buyer suggestions evaluation instruments, product assessment evaluation instruments, customer support assistant instruments, and so on. The concepts are all the identical — using the benefit of how LLM can perceive textual content context and make conversations to create specialised AI brokers in textual content analytics. 

Conclusion

It’s straightforward to get caught up chasing the most recent AI instruments. However on the finish of the day, what issues most is utilizing AI to remove what slows us down and speed up what strikes us ahead. The bottom line is to remain pragmatic: undertake what works right now, keep inquisitive about what’s rising, and by no means lose sight of the core function of knowledge science—to drive higher selections by way of higher understanding.