Ivory Tower Notes: The Problem | Towards Data Science

months on a Machine Learning undertaking, solely to find you by no means outlined the “appropriate” downside initially? In that case, or even when not, and you might be solely beginning with the information science or AI area, welcome to my first Ivory Tower Word, the place I’ll handle this subject. 

The time period “Ivory Tower” is a metaphor for a state of affairs through which somebody is remoted from the sensible realities of on a regular basis life. In academia, the time period typically refers to researchers who have interaction deeply in theoretical pursuits and stay distant from the realities that practitioners face outdoors academia.

As a former researcher, I wrote a quick sequence of posts from my outdated Ivory Tower notes — the notes earlier than the LLM period.

Scary, I do know. I’m scripting this to handle expectations and the query, “Why ever did you do issues this fashion?” — “As a result of no LLM advised me the best way to do in any other case 10+ years in the past.”

That’s why my notes include “legacy” subjects corresponding to information mining, machine studying, multi-criteria decision-making, and (generally) human interactions, airplanes ✈️ and artwork.

Nonetheless, at any time when there is a chance, I’ll map my “outdated” data to generative AI advances and clarify how I utilized it to datasets past the Ivory Tower.

Welcome to publish #1…

How each Machine Studying and AI journey begins

 — It begins with an issue. 

For you, that is normally “the” downside as a result of it’s essential to dwell with it for months or, within the case of analysis, years. 

With “the” downside, I’m addressing the enterprise downside you don’t absolutely perceive or know the best way to resolve at first. 

A fair worse state of affairs is once you suppose you absolutely perceive and know the best way to resolve it shortly. This then creates solely extra issues which might be once more solely yours to resolve. However extra about this within the upcoming sections. 

So, what’s “the” downside about?

Causa: It’s principally about not managing or leveraging sources correctly —  workforce, gear, cash, or time. 

Ratio: It’s normally about producing enterprise worth, which might span from improved accuracy, elevated productiveness, value financial savings, income good points, quicker response, choice, planning, supply or turnaround instances. 

Veritas: It’s at all times about discovering an answer that depends and is hidden someplace within the present dataset. 

Or, a couple of dataset that somebody labelled as “the one”, and that’s ready so that you can resolve the downside. As a result of datasets comply with and are created from technical or enterprise course of logs, “there must be an answer mendacity someplace inside them.”

Ah, if solely it have been really easy.

Avoiding a unique chain of thought once more, the purpose is you will have to:

1 — Perceive the issue absolutely,
2 — If not given, discover the dataset “behind” it, and 
3 — Create a technique to get to the answer that may generate enterprise worth from it. 

On this path, you can be tracked and measured, and time won’t be in your aspect to ship the answer that may resolve “the universe equation.” 

That’s why you will have to method the issue methodologically, drill right down to smaller issues first, and focus fully on them as a result of they’re the basis reason behind the general downside. 

That’s why it’s good to learn to…

Think like a Data Scientist.

Returning to the issue itself, let’s think about that you’re a vacationer misplaced someplace within the large museum, and also you wish to work out the place you might be. What you do subsequent is stroll to the closest data map on the ground, which is able to present your present location. 

At this second, in entrance of you, you see one thing like this: 

The subsequent factor you may inform your self is, “I wish to get to Frida Kahlo’s portray.” (Word: These are the insights you wish to get.)

As a result of your objective is to see this one portray that introduced you miles away from your own home and now sits two flooring beneath, you head straight to the second flooring. Beforehand, you memorized the shortest path to succeed in your objective. (Word: That is the preliminary information assortment and discovery part.)

Nevertheless, alongside the best way, you encounter some obstacles — the elevator is shut down for renovation, so you need to use the steps. The museum work have been reordered simply two days in the past, and the information plans didn’t replicate the adjustments, so the trail you had in thoughts to get to the portray is just not correct. 

Then you end up wandering across the third flooring already, asking quietly once more, “How do I get out of this labyrinth and get to my portray quicker?”

When you don’t know the reply, you ask the museum employees on the third flooring that will help you out, and also you begin gathering the brand new information to get the proper path to your portray. (Word: This can be a new information assortment and discovery part.)

Nonetheless, when you get to the second flooring, you get misplaced once more, however what you do subsequent is begin noticing a sample in how the work have been ordered chronologically and thematically to group the artists whose types overlap, thus providing you with a sign of the place to go to search out your portray. (Word: This can be a modelling part overlapped with the enrichment part from the dataset you collected throughout college days — your artwork data.)

Lastly, after adapting the sample evaluation and recalling the collected inputs on the museum route, you arrive in entrance of the portray you had been planning to see since reserving your flight a number of months in the past. 

What I described now’s the way you method information science and, these days, generative AI issues. You at all times begin with the top objective in thoughts and ask your self:

“What’s the anticipated final result I need or must get from this?”

You then begin planning from this query backwards. The instance above began with requesting holidays, reserving flights, arranging lodging, touring to a vacation spot, shopping for museum tickets, wandering round in a museum, after which seeing the portray you’ve been studying about for ages. 

After all, there’s extra to it, and this course of ought to be approached in another way if it’s essential to resolve another person’s downside, which is a little more complicated than finding the portray within the museum. 

On this case, you need to…

Ask the “good” questions.

To do that, let’s define what a good question means [1]: 

A good information science query should be concrete, tractable, and answerable. Your query works nicely if it naturally factors to a possible method to your undertaking. In case your query is too obscure to recommend what information you want, it gained’t successfully information your work.

Formulating good questions retains you on observe so that you don’t get misplaced within the information that ought to be used to get to the precise downside resolution, otherwise you don’t find yourself fixing the unsuitable downside.

Going into extra element, good questions will assist determine gaps in reasoning, keep away from defective premises, and create different situations in case issues do go south (which just about at all times occurs)👇🏼.

From the above-presented diagram, you perceive how good questions, at first, must help concrete assumptions. This implies they must be formulated in a method that your premises are clear and guarantee they are often examined with out mixing up info with opinions.

Good questions produce solutions that transfer you nearer to your objective, whether or not by way of confirming hypotheses, offering new insights, or eliminating unsuitable paths. They’re measurable, and with this, they hook up with undertaking targets as a result of they’re formulated with consideration of what’s potential, useful, and environment friendly [2].

Good questions are answerable with accessible information, contemplating present information relevance and limitations. 

Final however not least, good questions anticipate obstacles. If one thing is definite in information science, that is the uncertainty, so having backup plans when issues don’t work as anticipated is necessary to provide outcomes to your undertaking.

Let’s exemplify this with one use case of an airline firm that has a problem with growing its fleet availability as a consequence of unplanned technical groundings (UTG).

These surprising upkeep occasions disrupt flights and value the corporate vital cash. Due to this, executives determined to react to the issue and name in a knowledge scientist (you) to assist them enhance plane availability.

Now, if this might be the primary information science activity you ever received, you’ll possibly begin an investigation by asking:

“How can we get rid of all unplanned upkeep occasions?”

You perceive how this query is an instance of the unsuitable or “poor” one as a result of:

• It isn’t real looking: It consists of each potential defect, each small and massive, into one unimaginable objective of “zero operational interruptions”.
• It doesn’t maintain a measure of success: There’s no concrete metric to indicate progress, and if you happen to’re not at zero, you’re at “failure.”
• It isn’t data-driven: The query didn’t cowl which information is recorded earlier than delays happen, and the way the plane unavailability is measured and reported from it.

So, as a substitute of this obscure query, you’ll in all probability ask a set of focused questions:

1. Which plane (sub)system is most crucial to flight disruptions?
   (Concrete, particular, answerable) This query narrows down your scope, specializing in just one or two particular (sub) methods affecting most delays.
2. What constitutes “crucial downtime” from an operational perspective?
   (Precious, ties to enterprise targets) If the airline (or regulatory physique) doesn’t outline what number of minutes of unscheduled downtime matter for schedule disruptions, you may waste effort fixing much less pressing points.
3. Which information sources seize the basis causes, and the way can we fuse them?
   (Manageable, narrows the scope of the undertaking additional) This clarifies which information sources one would want to search out the issue resolution.

With these sharper questions, you’ll drill right down to the actual downside:

• Not all delays weigh the identical in value or impression. The “appropriate” information science downside is to foretell crucial subsystem failures that result in operationally pricey interruptions so upkeep crews can prioritize them.

That’s why…

Defining the issue determines each step after. 

It’s the inspiration upon which your information, modelling, and analysis phases are constructed 👇🏼.

It means you might be clarifying the undertaking’s goals, constraints, and scope; it’s essential to articulate the final word objective first and, apart from asking “What’s the anticipated final result I need or must get from this?”, ask as nicely: 

What would success appear like and the way can we measure it?

From there, drill right down to (potential) next-level questions that you simply (I) have realized from the Ivory Tower days:
 — Historical past questions: “Has anybody tried to resolve this earlier than? What occurred? What continues to be lacking?”
 —  Context questions: “Who’s affected by this downside and the way? How are they partially resolving it now? Which sources, strategies, and instruments are they utilizing now, and might they nonetheless be reused within the new fashions?”
 — Influence Questions: “What occurs if we don’t resolve this? What adjustments if we do? Is there a worth we will create by default? How a lot will this method value?”
— Assumption Questions: “What are we taking as a right that may not be true (particularly in terms of information and stakeholders’ concepts)?”
 — ….

Then, do that within the loop and at all times “ask, ask once more, and don’t cease asking” questions so you’ll be able to drill down and perceive which information and evaluation are wanted and what the bottom downside is. 

That is the evergreen data you’ll be able to apply these days, too, when deciding in case your downside is of a predictive or generative nature. 

(Extra about this in another notice the place I’ll clarify how problematic it’s making an attempt to resolve the issue with the fashions which have by no means seen — or have by no means been skilled on — related issues earlier than.)

Now, going again to reminiscence lane…

I wish to add one necessary notice: I’ve realized from late nights within the Ivory Tower that no quantity of knowledge or information science data can prevent if you happen to’re fixing the unsuitable downside and making an attempt to get the answer (reply) from a query that was merely unsuitable and obscure. 

When you have got an issue readily available, don’t rush into assumptions or constructing the fashions with out understanding what it’s essential to do (Festina lente). 

As well as, put together your self for surprising conditions and do a correct investigation together with your stakeholders and area specialists as a result of their persistence might be restricted, too. 

With this, I wish to say that the “actual artwork” of being profitable in information initiatives is figuring out exactly what the issue is, determining if it may be solved within the first place, after which developing with the “how” half. 

You get there by studying to ask good questions.

Thanks for studying, and keep tuned for the following Ivory Tower notice.

For those who discovered this publish useful, be at liberty to share it together with your community. 👏

Join for extra tales on Medium ✍️ and LinkedIn 🖇️.

References: 

[1] DS4Humans, Backwards Design, accessed: April fifth 2025, https://ds4humans.com/40_in_practice/05_backwards_design.html#defining-a-good-question

[2] Godsey, B. (2017), Assume Like a Knowledge Scientist: Deal with the information science course of step-by-step, Manning Publications.