Evaluating Time Series Models for Real-World Forecasting: A Practical Comparison | by Karan_bhutani | May, 2025

Time sequence forecasting is each an artwork and a science. With a myriad of modeling approaches — from old-school statistical fashions to cutting-edge deep studying — it may be difficult to determine which one to make use of for a given downside. On this article, we’ll evaluate six standard time sequence fashions on a real-world dataset, highlighting their efficiency, velocity, and practicality. Our aim is to offer a virtually grounded comparability and proposals for when to make use of every mannequin. We’ll be taking a look at:

• ARIMA: A traditional statistical mannequin.
• LSTM: A neural community mannequin with long-term reminiscence.
• GRU: A simplified neural community much like LSTM.
• LightGBM: A gradient boosting tree methodology utilized to time sequence.
• Prophet: A decomposable additive mannequin (by Meta/Fb).
• Prophet with customized regressors: Prophet augmented with further options.

Every mannequin shall be launched with a brief rationale, a code snippet (utilizing Python libraries), and observations from an experiment forecasting the Jena Local weather dataset (a climate time sequence). After that, we’ll dive right into a cross-model comparability based mostly on metrics like error charges, coaching/inference time, residual patterns, interpretability, flexibility, and real-world deployment issues. Lastly, we’ll supply suggestions on which mannequin to decide on for various eventualities.

Let’s dive in and see how these fashions stack up!

When and Why to Use ARIMA:
ARIMA (AutoRegressive Built-in Transferring Common) is a venerable time sequence mannequin that captures a sequence’ previous values (AR phrases) and previous forecast errors (MA phrases) to foretell future factors. It’s most relevant when your time sequence is comparatively brief, has a single variable of curiosity, and displays patterns like autocorrelation or seasonality that may be captured with linear relations. ARIMA works finest on stationary knowledge (steady imply and variance over time), so we frequently differentiate the sequence (the “I” half) to take away tendencies or seasonal cycles. Use ARIMA as a fast baseline or if you want an interpretable mannequin (its parameters can let you know about development or seasonality power) with out requiring quite a lot of coaching knowledge. It’s additionally fairly quick for small datasets.

Code Snippet (Becoming ARIMA in Python): Utilizing statsmodels, we are able to match an ARIMA or seasonal ARIMA (SARIMA) mannequin. For instance, if we anticipate a every day seasonal cycle in hourly knowledge (seasonal interval = 24 hours), we would do:

import statsmodels.api as sm
# Assume y_train is a pandas Collection of coaching knowledge
mannequin = sm.tsa.SARIMAX(y_train, order=(2,1,2), seasonal_order=(1,0,1,24))
consequence = mannequin.match(disp=False)  # Match ARIMA mannequin
forecast = consequence.forecast(steps=len(y_test))  # Forecast out-of-sample

On this snippet, order=(2,1,2) means ARIMA with p=2 AR phrases, d=1 differencing, q=2 MA phrases. seasonal_order=(1,0,1,24) provides one seasonal AR and MA time period with a 24-hour seasonality. The mannequin is then fitted on the coaching knowledge, and we forecast the identical variety of steps because the take a look at set size.

Conduct within the Jena Local weather Experiment:
In our Jena local weather dataset (hourly climate readings), ARIMA served as a wise baseline however struggled with sure features. We fitted a seasonal ARIMA to seize the every day cycle. It did an okay job modeling the periodic ups and downs, nevertheless it didn’t anticipate a serious downward development within the take a look at interval. The ARIMA forecast basically leveled off (a virtually horizontal line into the longer term), whereas the precise temperatures within the take a look at knowledge dropped considerably. This resulted in massive errors — an RMSE of round 8 (within the unique items) on the take a look at set. The residuals (forecast errors) from ARIMA had a transparent sample: they began round zero and trended negatively because the forecast horizon went on, indicating the mannequin was systematically overshooting the precise values (it constantly predicted larger than actuality through the take a look at interval). Briefly, ARIMA underfit the altering development — a standard situation if a sudden change happens exterior the vary of historic patterns the mannequin realized. On the constructive facet, ARIMA was extraordinarily quick to coach (solely a few seconds) and nearly instantaneous to provide a forecast as soon as fitted. It’s a light-weight mannequin, however on this case its simplicity left important patterns unmodeled.

When and Why to Use LSTM:
LSTM (Lengthy Quick-Time period Reminiscence) networks are a kind of recurrent neural community (RNN) adept at studying long-range dependencies in sequence knowledge. Use an LSTM when your time sequence has complicated patterns that aren’t purely linear — for instance, if you suspect there are nonlinear relationships or when the impact of a previous occasion on the longer term would possibly span a protracted interval. LSTMs shine in dealing with sequences with seasonality, tendencies, and different context you probably have plenty of coaching knowledge. They’ll incorporate a number of variables simply (simply embody them as further enter options), which makes LSTMs helpful for multivariate time sequence or if you wish to embody exogenous alerts. Nevertheless, LSTMs are heavy-weight in comparison with ARIMA/Prophet — they require tuning of structure and take longer to coach. In addition they threat overfitting if the coaching set is small or if the mannequin is just too complicated.

Code Snippet (Becoming an LSTM): We will construct an LSTM mannequin utilizing libraries like Keras (TensorFlow). Sometimes, we first put together the information as sequences of a hard and fast window size (e.g., use the previous 24 hours to foretell the following hour). Right here’s a simplified instance:

from tensorflow.keras.fashions import Sequential
from tensorflow.keras.layers import LSTM, Dense
# Construct a easy LSTM mannequin
mannequin = Sequential()
mannequin.add(LSTM(50, input_shape=(window_size, n_features)))
mannequin.add(Dense(1))  # predict the following worth
mannequin.compile(optimizer='adam', loss='mse')
# X_train form: (samples, window_size, n_features), y_train form: (samples,)
mannequin.match(X_train, y_train, epochs=10, batch_size=32, verbose=0)
y_pred = mannequin.predict(X_test)  # predict on take a look at sequences

This defines an LSTM with 50 items (reminiscence cells) adopted by a Dense layer to output the forecast. We prepare for 10 epochs on the coaching sequences. In follow, one would tune the window_size (variety of previous time steps to feed in) and presumably use a number of LSTM layers or regularization to enhance efficiency.

Conduct within the Jena Local weather Experiment:
The LSTM mannequin’s efficiency was astonishingly good — at the very least, at face worth. It yielded an especially low error on the take a look at knowledge (nearly near-zero RMSE and MAE). Its forecasted values virtually completely overlapped the precise take a look at observations, tracing each wiggle and drop with uncanny accuracy. On paper, that’s a dream consequence, however in actuality it raises a pink flag: such perfection usually implies the mannequin might need successfully overfit or “leaked” info from the take a look at knowledge. In our setup, the LSTM was possible being fed the earlier precise knowledge level (or a sequence of latest actuals) to foretell the following one, a method often known as rolling one-step forecasting. This implies at every step the mannequin had the advantage of figuring out the latest floor fact earlier than predicting the following level — an affordable strategy in a deployment situation (the place you replace your forecast as new knowledge is available in), nevertheless it makes the analysis very simple for the mannequin. The LSTM principally simply needed to be taught to imitate the short-term dynamics, and it did so brilliantly — basically memorizing the sequence and yielding little to no error.

The draw back is that if we requested the LSTM to foretell many steps forward with out feeding it contemporary precise knowledge, it won’t do almost as nicely — the errors may compound shortly. In our take a look at, nevertheless, as a result of we evaluated it in a rolling vogue, the LSTM seemed like a star performer. It captured the every day cycle, the gradual seasonal transition, and even the sharp downward development in autumn, all virtually completely. Coaching the LSTM was essentially the most time-consuming half (a number of minutes of coaching), and the mannequin is a black field — we are able to’t simply interpret why it made a given prediction. However there’s little doubt it was highly effective: given sufficient knowledge and correct framing, the LSTM can mannequin very complicated time sequence patterns. One should be cautious to keep away from overfitting (e.g., use validation units, regularization, and so on.), as a result of an LSTM will fortunately be taught even random fluctuations if allowed.

When and Why to Use GRU:
GRU (Gated Recurrent Unit) networks are a sibling of LSTMs — a barely less complicated RNN structure that always trains sooner and requires fewer sources. GRUs use gating mechanisms like LSTMs however have fewer parameters (they lack a separate cell state). In follow, GRUs can carry out on par with LSTMs on many duties and could be a sensible choice if you would like among the sequence-learning energy of an LSTM however with a bit much less complexity. Use a GRU you probably have a fancy sequence with long-term dependencies however maybe wish to scale back mannequin measurement or coaching time. Very like LSTMs, GRUs want quantity of knowledge and cautious tuning. They’re appropriate if you don’t strictly want the complete expressive energy of an LSTM — as an example, some patterns could be captured nicely with a GRU at decrease computational price.

Code Snippet (Becoming a GRU): The method is analogous to LSTM, simply swapping the layer:

from tensorflow.keras.fashions import Sequential
from tensorflow.keras.layers import GRU, Dense
mannequin = Sequential()
mannequin.add(GRU(50, input_shape=(window_size, n_features)))
mannequin.add(Dense(1))
mannequin.compile(optimizer='adam', loss='mse')
mannequin.match(X_train, y_train, epochs=10, batch_size=32, verbose=0)
y_pred = mannequin.predict(X_test)

This defines a GRU with 50 items. The remaining is much like the LSTM instance. Primarily, you deal with it the identical means: put together sequences of knowledge, prepare, and predict.

Conduct within the Jena Local weather Experiment:
The GRU’s outcomes have been virtually a mirror of the LSTM. It achieved extraordinarily low error on the take a look at set, basically nailing the forecast with near-zero RMSE. The GRU’s predictions within the take a look at interval have been additionally on prime of the particular knowledge, matching each peak and valley. This isn’t too stunning — given the similarity of the fashions, our GRU possible realized the patterns simply as successfully because the LSTM. It might need even educated a contact sooner (we observed the GRU coaching time was barely decrease than the LSTM’s in our experiment, which is according to GRUs being a bit less complicated). Just like the LSTM, the GRU was evaluated with one-step-ahead forecasting utilizing true earlier values, so it too had a better job and doubtlessly overfit the situation. The take-away is that GRUs will be as highly effective as LSTMs for time sequence, reaching comparable accuracy. In our case, it shared the identical caveats: its efficiency seemed good because of how we fed it knowledge, and it stays uninterpretable to people. But when one is on the lookout for a quick RNN to implement, GRU might be a positive various to LSTM, particularly if you wish to scale back mannequin complexity with out sacrificing a lot accuracy.

When and Why to Use LightGBM:
LightGBM is a gradient boosting framework that’s often used for regression or classification, however it may be utilized to time sequence by reworking the issue right into a regression on lagged options. Primarily, as an alternative of a steady recurrence like an RNN, you create options comparable to “worth at time t-1”, “worth at time t-24” (a every day lag if hourly knowledge), or some other related indicators (day of week, hour of day, and so on.), after which prepare a boosted choice tree mannequin to foretell the following worth. LightGBM is quick, environment friendly, and may deal with plenty of options (even irrelevant ones) because of built-in regularization. It’s an important selection when you have got wealthy exterior regressors or a number of seasonalities encoded as options, and also you need a mannequin that’s fast to coach and simple to tweak. It’s additionally inherently capable of deal with lacking values (it treats them in a particular means throughout tree splitting) and non-linear relationships. Use LightGBM should you choose a extra easy machine studying strategy that advantages from characteristic engineering — for instance, forecasting gross sales the place you possibly can present options like final yr’s gross sales, promotions, holidays, and so on., all in a single mannequin.

Code Snippet (Becoming LightGBM for forecasting): We first want to organize a characteristic matrix X and goal y. Suppose we use the final 3 hours and final 24 hours as options:

import lightgbm as lgb
import numpy as np
# Put together options (instance: utilizing 3-hour and 24-hour lag)
def make_features(sequence, lag_hours=[1,2,3,24]):
X, y = [], []
for t in vary(max(lag_hours), len(sequence)):
X.append([series[t-h] for h in lag_hours])
y.append(sequence[t])
return np.array(X), np.array(y)
X_train, y_train = make_features(train_values, lag_hours=[1,2,3,24])
X_test,  y_test  = make_features(full_series, lag_hours=[1,2,3,24])[1]  # use final half for take a look at
# Practice LightGBM mannequin
mannequin = lgb.LGBMRegressor(n_estimators=100)
mannequin.match(X_train, y_train)
y_pred = mannequin.predict(X_test)

Right here we manually constructed options: the values 1, 2, 3 hours in the past, and 24 hours in the past, to foretell the present worth. We then prepare a LightGBM regressor on these options. In follow, one would possibly use a library like sktime or skforecast to streamline this creation of lag options. You can too embody time-of-day or different exogenous knowledge as further options in X_train.

Conduct within the Jena Local weather Experiment:
LightGBM turned out to be a powerful performer in our experiment. Regardless that it’s not a specialised time sequence mannequin, by feeding it the appropriate options it captured a lot of the construction of the information. We gave it lagged temperature values and in addition let it infer any non-linear relationships between these lags and the goal. The consequence: LightGBM’s forecast was fairly near the precise observations, and it did a greater job at predicting the downward development within the take a look at interval than ARIMA and Prophet did. Its error was low (not as almost-zero because the neural networks, however these had an unfair benefit, as mentioned). The RMSE for LightGBM on take a look at was on the order of ~0.5 — a really small error relative to the size of temperature variations — and much better than the ~8–10 of the normal fashions. The residuals for LightGBM have been centered close to zero with comparatively small unfold, indicating it didn’t have a big systematic bias.

When it comes to velocity, LightGBM was quick to coach and predict. Coaching took only some seconds (even with 100 bushes, due to the effectivity of the LightGBM algorithm), and forecasting was virtually instantaneous. One of many advantages we noticed was flexibility: we may simply lengthen the characteristic set if we wished the mannequin to account for, say, humidity or day-of-week results — it’s simply one other column in X. Nevertheless, LightGBM’s predictions should not as transparently interpretable as Prophet’s (we are able to’t immediately get a development or seasonality plot out of it with out further evaluation). We will, although, look at characteristic importances to see which lag mattered most — in our case, it possible gave heavy weight to the 24-hour lag (yesterday’s similar hour temperature) due to the sturdy every day cycle in climate knowledge. Total, LightGBM offered a pleasant steadiness: excessive accuracy, low computation, however requiring us to do some characteristic engineering and never giving express perception into “why” the forecast is what it’s.

When and Why to Use Prophet:
Prophet is a forecasting device open-sourced by Fb (now Meta) that turned standard for its ease of use and interpretability. Prophet is an additive mannequin: it assumes the time sequence consists of an total development, plus seasonal elements (every day, weekly, yearly, and so on.), plus results of holidays or occasions. It matches these elements utilizing sturdy statistical strategies. The massive enchantment of Prophet is that you just don’t have to be a time sequence knowledgeable to make use of it — it routinely handles quite a lot of issues (like seasonality and development adjustments) and produces cheap forecasts with confidence intervals. It’s nice when your knowledge has clear seasonal patterns and also you need a fast, interpretable mannequin. It additionally permits incorporating identified holidays or particular occasions which may have an effect on the sequence (e.g., promotions, and so on.) and can regulate the forecast for these. Prophet works finest for longer-term forecasts the place these seasonal patterns matter (e.g., every day knowledge over a number of years with yearly seasonality). It’s much less fitted to brief sequences or purely short-term dynamics with out clear recurring patterns. It’s additionally univariate by default — you mannequin one time sequence at a time (although you possibly can add further regressors, which we’ll focus on subsequent).

Code Snippet (Becoming Prophet): Prophet has an intuitive API. We put together a DataFrame with columns ds (date/time stamp) and y (worth), then:

from prophet import Prophet
# Put together knowledge body for Prophet
train_df = train_data.reset_index().rename(columns={'timestamp':'ds', 'temperature':'y'})
m = Prophet(daily_seasonality=True)  # allow every day seasonality for hourly knowledge
m.match(train_df)  # match the mannequin
# Make a DataFrame for future timestamps
future = m.make_future_dataframe(intervals=len(test_data), freq='H')
forecast = m.predict(future)

We set daily_seasonality=True as a result of our knowledge is hourly and we all know there’s a every day cycle (Prophet by default won’t embody hourly seasonality until specified). The forecast DataFrame returned by m.predict will include columns for development, seasonal elements, and the anticipated worth (yhat).

Conduct within the Jena Local weather Experiment:
Prophet was the worst performer when it comes to uncooked accuracy on this dataset. Its forecast for the take a look at interval was virtually a flat line (with a slight development), failing to anticipate the dramatic drop in temperatures that occurred. This led to an RMSE above 10, the best amongst all fashions we tried. What went fallacious? It appears Prophet’s strengths (capturing common seasonality and easy tendencies) turned a weak spot right here: the mannequin possible noticed a powerful every day cycle and possibly a delicate upward development in the summertime coaching knowledge and projected that ahead. It didn’t know concerning the coming seasonal transition (from summer season to fall) as a result of the coaching interval didn’t embody that sample, and we didn’t present any exterior indicator of season change. Prophet can deal with yearly seasonality if it has at the very least a yr of knowledge, however right here we solely gave it roughly 8 months of knowledge — not sufficient to be taught that winters are colder. So it basically assumed the established order would proceed, leading to a forecast that overshot the precise cooling development.

From a diagnostics perspective, Prophet’s residuals confirmed a transparent bias — the forecast was constantly too excessive within the take a look at interval (residuals have been largely destructive, precise minus forecast round -10 on common). This sort of systematic error means the mannequin missed an necessary part (on this case, a seasonal shift). Prophet’s power, nevertheless, is what it permits us to examine. We may crack open the forecast elements and see, as an example, what development it realized (possible a flat or slight upward development) and what every day sample it realized (most likely the day-night temperature oscillation). These elements are interpretable: we are able to plot “development” and see the way it thought the general stage would evolve, and plot “every day” to see the estimated common every day cycle of temperature. Coaching Prophet was comparatively fast (~20 seconds), and producing a forecast was additionally easy (a single name to predict, although it took just a few seconds to compute all these elements for every hour of the forecast horizon). Prophet additionally offers uncertainty intervals by default (the forecast comes with a decrease and higher sure), however in our case, even the decrease sure of Prophet’s forecast was above the precise values – the mannequin was essentially off, and no quantity of Prophet’s inside uncertainty may repair that. The lesson is that Prophet is straightforward and interpretable, nevertheless it’s not magic – in case your coaching knowledge doesn’t include a sample that the longer term holds (otherwise you don’t feed that information in), Prophet can’t guess it out of skinny air. In real-world phrases, it’s like a seasoned forecaster who has by no means seen a winter making an attempt to foretell December climate based mostly solely on spring and summer season knowledge!

When and Why to Use Prophet with Regressors:
One highly effective characteristic of Prophet is the flexibility so as to add customized regressors — principally further predictor columns that you just suppose will affect the forecast. For instance, if you’re forecasting power demand, you would possibly add temperature as a regressor; should you forecast gross sales, you would possibly add advertising and marketing spend or net visitors. Prophet will then match a coefficient to every regressor and embody their impact within the forecast. This strategy is helpful when you have got area information or exterior knowledge that may enhance the forecast. Use Prophet with regressors when the time sequence alone doesn’t inform the complete story, and you’ve got a number of explanatory variables which are identified for future dates (or will be forecasted individually). It retains Prophet’s interpretability (you possibly can see the impact of every regressor) whereas permitting a type of multivariate modeling.

Code Snippet (Prophet with Regressors): Suppose within the Jena local weather knowledge we suspect that atmospheric stress (p) and humidity (rh) would possibly assist predict temperature. We will add these as regressors:

m = Prophet(daily_seasonality=True)
m.add_regressor('p')   # add stress as regressor
m.add_regressor('rh')  # add humidity as regressor
m.match(train_df)        # train_df consists of columns 'p' and 'rh'
future = m.make_future_dataframe(intervals=len(test_data), freq='H')
# we should additionally present regressor values for future dates:
future['p'] = pd.concat([train_df['p'], test_df['p']])
future['rh'] = pd.concat([train_df['rh'], test_df['rh']])
forecast = m.predict(future)

We name add_regressor for every further variable. When forecasting, we prolonged our future DataFrame and appended the identified future values of these regressors (right here we assume we all know stress and humidity within the take a look at interval – which could be the case if we’re doing a historic backtest, or we have now forecasts for them). Prophet will output in forecast a column for every regressor’s contribution, so we are able to see how, say, stress was affecting the forecast.

Conduct within the Jena Local weather Experiment:
We tried Prophet with a few further inputs from the dataset (as an example, we included atmospheric stress and humidity as regressors alongside the date). Intuitively, one would possibly anticipate that as autumn comes, maybe stress adjustments or humidity adjustments may sign the temperature drop — giving Prophet a clue concerning the coming development. The consequence was considerably blended. Prophet with regressors did enhance the accuracy a bit in comparison with vanilla Prophet — the RMSE dropped from round 10 to about 8 — nevertheless it was nonetheless not almost pretty much as good as our different fashions. The forecast with regressors nonetheless largely missed the magnitude of the temperature drop, basically popping out as a barely adjusted flat line. Why didn’t the regressors save the day? Probably as a result of the connection between these regressors and the goal was not sturdy sufficient or not linear sufficient for Prophet’s easy additive mannequin to seize. It’s additionally doable we didn’t select the appropriate regressors — possibly together with one thing like an express “month” indicator or an extended historical past would have helped Prophet perceive the seasonality higher. In essence, Prophet with regressors is barely pretty much as good because the alerts you feed into it.

On the constructive facet, this strategy remained interpretable. We may take a look at the mannequin’s realized coefficients for every regressor to see the way it thinks stress or humidity correlate with temperature. If, say, stress had a powerful destructive coefficient, it means each time stress rises, temperature tends to drop (simply an instance). This sort of perception will be helpful for area understanding. The computation price with regressors was a bit larger: coaching took barely longer as a result of there are extra parameters to suit, and producing the longer term dataframe with regressors and making predictions had some overhead (our inference time for Prophet with regressors was the best amongst all fashions, round 7–8 seconds, possible as a result of further work of dealing with these regressors). Nonetheless, underneath the hood it’s the identical Prophet algorithm, and seven–8 seconds is hardly a priority for many use instances. The principle takeaway is that including regressors may also help Prophet if these regressors have predictive energy — nevertheless it’s not a silver bullet. In our real-world take a look at, the additional info offered solely marginal good points as a result of the core situation (lack of a yearly seasonal sample in coaching) wasn’t totally solved by the few regressors we tried.

Now that we’ve launched every mannequin and seen how they behaved, let’s evaluate them head-to-head throughout a number of necessary dimensions. We’ll take a look at forecast accuracy (RMSE/MAE), computational effectivity (coaching and inference instances), residual diagnostics, interpretability, flexibility, and suitability for deployment. This cross-model comparability will spotlight the trade-offs and assist us type suggestions for selecting a mannequin.

To make the comparability concrete, think about the summarized outcomes from our experiment. We educated all fashions on the identical coaching knowledge and evaluated on the identical take a look at interval. The chart beneath illustrates some key metrics for every mannequin:

Metrics abstract for every mannequin on the take a look at set. High: RMSE and MAE (decrease is healthier) for forecast error. Backside: Coaching time and inference time in seconds. ARIMA and LightGBM had average errors and have been extraordinarily quick; LSTM and GRU achieved the bottom errors however took longest to coach; Prophet (with and with out regressors) had larger errors and comparatively modest coaching instances however slowest prediction instances.

Wanting on the error metrics (RMSE and MAE within the prime row of the determine), we see a dramatic unfold: the neural networks (LSTM and GRU) achieved near-zero error on the take a look at set — basically an virtually good rating — whereas Prophet was the worst with RMSE above 10 and MAE round 8–9. LightGBM additionally did very nicely, with errors virtually as little as the RNN fashions. ARIMA and Prophet with regressors have been within the center (RMSE round 8). At first look, one would possibly conclude “LSTM/GRU are the perfect by far, and Prophet is horrible.” However as mentioned earlier, the analysis technique issues. The LSTM and GRU have been allowed to replace with precise latest knowledge at every step (one-step forward predictions), so their error being so low partly displays that benefit and doubtlessly some overfitting to short-term patterns. Prophet, against this, was making a real out-of-sample multi-step forecast with no updates, which is a tougher process (and it didn’t have the appropriate seasonal information). So, this isn’t a very apples-to-apples comparability. Nonetheless, it tells us that the RNNs and LightGBM have been capable of seize the short-term dynamics extraordinarily nicely, whereas Prophet and ARIMA missed one thing essential (a longer-term change).

The coaching instances (bottom-left of the determine) present one other facet of the story: LSTM and GRU took the longest to coach (on the order of three minutes for GRU and a bit much less for LSTM, on this setup), whereas the whole lot else was a lot sooner. ARIMA was virtually negligible in coaching time (just a few seconds at most). LightGBM was additionally very quick (underneath 10 seconds). Prophet and Prophet+regressors took on the order of tens of seconds (the plot reveals Prophet ~30s, Prophet+regressors ~15s — presumably as a result of our coaching set wasn’t big and we didn’t use a fancy seasonal mannequin past every day seasonality). The distinction in coaching time is necessary: if you have to retrain your mannequin regularly or prepare hundreds of fashions, a neural community could be too gradual, whereas ARIMA or LightGBM may deal with it. GRU being barely slower than LSTM in our case might be because of needing just a few extra epochs or simply variance in coaching — usually GRU has fewer parameters, however right here each have been fairly small fashions anyway.

For inference (forecasting) velocity (bottom-right of the determine), we additionally see attention-grabbing variations. ARIMA and LightGBM can produce forecasts virtually immediately — their bars are so low they’re barely seen. Prophet and Prophet with regressors have been the slowest at predicting, every taking round 7–8 seconds to generate the forecast for the take a look at interval. LSTM and GRU have been in between: just a few seconds (GRU barely slower than LSTM right here). Why is Prophet slower at inference regardless of being comparatively easy? Below the hood, Prophet’s .predict methodology is doing fairly a bit: computing a number of Fourier sequence elements for every seasonal impact for every time level, making use of development mannequin, and so on., for every timestamp sooner or later dataframe. It’s vectorized and in pure Python, which isn’t as optimized because the compiled code of LightGBM or the matrix ops of an RNN on TensorFlow. Furthermore, we included uncertainty intervals (by default Prophet simulates development uncertainty), which may add to computation. 7–8 seconds continues to be not unhealthy (it’s nice for every day or hourly forecasts in most purposes), nevertheless it was noticeably the slowest in our small benchmark. LightGBM’s near-zero inference time reveals how environment friendly tree fashions are at making predictions (principally simply traversing some if-else situations for every tree). The RNNs’ few seconds of inference have been possible as a result of we did a step-by-step forecast in a loop; if we used a vectorized strategy to get the whole sequence in a single go, it might need been even sooner.

A superb forecast mannequin mustn’t solely have low error but additionally produce residuals (errors) that appear to be noise — any construction in residuals means the mannequin is lacking one thing. We examined the residuals of every mannequin for the take a look at interval:

Residual evaluation for every mannequin on the take a look at interval. Left column: residuals over time (excellent residuals hover round 0 with out clear patterns). Proper column: distribution of residuals (histogram). The dimensions differs for every (word Prophet and ARIMA have wider ranges, LSTM/GRU have very slender residual distributions close to zero).

From the above residual plots, we noticed:

• Prophet (and Prophet with regressors) had residuals that weren’t centered round 0 — they have been largely destructive within the take a look at interval (indicating the forecasts have been too excessive). There was additionally a noticeable development: the residuals began considerably much less destructive and have become extra destructive as time went on, reflecting how Prophet’s forecast stayed excessive whereas actuals stored dropping. The distribution of Prophet residuals was large and shifted (an enormous mode round -10 or so). This can be a clear signal that Prophet missed a structural ingredient (on this case, the season change) — its errors have been biased and never merely random noise.
• ARIMA additionally confirmed a sample: its residuals over time trended upward in the direction of 0 by the top, however have been principally destructive for portion, indicating it too was typically over-forecasting. The ARIMA residual distribution was wider than most, with quite a lot of mass round -5 to -10. Once more, some construction remained (maybe ARIMA didn’t totally seize a slowly rising development within the knowledge).
• LightGBM residuals have been a lot tighter. Over time, they fluctuated round zero with no apparent development — generally a bit above, generally beneath. The histogram of LightGBM residuals was slender (principally between -2 and +2). There might need been a slight destructive bias (possibly a small mode just under 0), however nothing like Prophet’s big bias. This means LightGBM captured the primary sample and solely small random errors remained — signal.
• LSTM and GRU residuals have been extremely small and patternless. On the time plot, they hugged the zero line virtually flatly — it’s laborious to even discern the residual wiggles as a result of the mannequin was so correct. The histograms for LSTM/GRU residuals have been basically a good lump round 0 with little or no unfold (on the order of possibly ±1 or much less). This once more highlights how nicely (maybe too nicely) these fashions match the take a look at knowledge. The truth is, such minimal residuals generally is a double-edged sword: both the mannequin really captured the whole lot (which is feasible if the take a look at knowledge supplied no surprises and the mannequin may be very succesful), or it overfit and used some leakage. In our case, as mentioned, it’s possible the latter — the LSTM/GRU had successfully minimal forecasting problem on this one-step-ahead setup, so their residuals aren’t a real measure of how they’d do in a stricter situation. Nonetheless, from a pure residual standpoint, the deep studying fashions “gained” right here — nothing unexplained left within the errors.

In abstract, Prophet and ARIMA exhibited clear systematic errors (indicating underfitting or lacking elements), whereas LightGBM and particularly the RNNs had residuals that seemed far more like white noise. Residual diagnostics in follow would immediate us to refine Prophet or ARIMA (possibly add that lacking yearly seasonality or a change-point for the development) to enhance them. For LightGBM and RNN, the diagnostics recommend the fashions match very nicely — maybe too nicely, so one would possibly think about including regularization or checking robustness on one other holdout set.

The spectrum of interpretability throughout these fashions is kind of broad:

• Prophet is very interpretable. After becoming, we are able to plot the development part and see, for instance, that it thinks there’s a gentle enhance of X items monthly (if any), and we are able to plot seasonal elements (every day cycle, weekly sample, and so on.) to visually examine what the mannequin realized. It additionally offers these good uncertainty intervals. In our use case, interpretability means we may clearly clarify why Prophet made the forecast it did: “Prophet assumed a continuing development round 18°C and a ±5°C every day oscillation, and thus forecasted roughly that sample ahead.” We will even say “Prophet didn’t understand it ought to go down as a result of it had no yearly cycle information.” This readability is effective in enterprise settings the place explaining the forecast is as necessary because the forecast itself.
• Prophet with regressors provides one other layer of interpretability: the impact of every regressor. We may look at the fitted parameters and see, as an example, “for each 1 hPa enhance in stress, Prophet predicts temperature to drop by 0.1°C” (simply an illustrative instance). These linear coefficients (or the plotted part for every regressor) give a direct sense of how exterior elements affect the prediction. Once more, very helpful for insights — you’re successfully getting a easy mannequin of the relationships, not only a black field.
• ARIMA is reasonably interpretable to these with time sequence information. The ARIMA mannequin’s parameters (the AR and MA coefficients) can let you know how a lot affect previous values have. For instance, an AR(2) coefficient of 0.7 for the final hour would possibly let you know there’s excessive persistence, and a seasonal time period would possibly point out a powerful 24-hour lag impact. ARIMA additionally permits decomposition: you possibly can separate the forecast into development (if differencing was undone) and seasonal elements by trying on the mannequin construction. Nevertheless, ARIMA doesn’t offer you a clear “right here’s the every day sample plot” like Prophet does — it’s important to infer it from the coefficients or use the seasonal differencing strategy to know that it thought-about a seasonal part. Nonetheless, it’s linear, so you possibly can comparatively simply hint why it made a sure prediction (it’s a weighted sum of previous observations and errors).
• LightGBM is much less interpretable than the above, however not completely opaque. Being a tree ensemble, the direct prediction mechanism is complicated (a whole lot of choice bushes voting on an end result). You’ll be able to’t simply extract a neat equation or decomposition of development/season. Nevertheless, LightGBM can output characteristic importances, which rank which options (lags, time indicators, and so on.) have been most influential. In our situation, if we noticed that the “24-hour lag” characteristic had the best significance, we’d know the mannequin closely depends on yesterday’s temperature to forecast right now’s — confirming it’s leveraging the every day seasonality. We may additionally do partial dependence plots: e.g., see how the anticipated temperature adjustments as we range yesterday’s temperature whereas holding different options fixed. That may generally floor a relationship (like a close to 1:1 relationship if it’s principally utilizing yesterday’s temp as a baseline). These instruments make LightGBM considerably interpretable, although it’s a far cry from Prophet’s clear development/season outputs.
• LSTM and GRU are the least interpretable of the lot. They’re basically black packing containers with realized weights in a neural community. Whereas one can try strategies like SHAP values or analyze the community’s hidden state dynamics, it’s fairly complicated to elucidate their predictions in human phrases. We will’t simply say why the LSTM predicted a sure spike or drop — it’s all implicit within the educated weights. We do know the options we gave it (in our case, simply the previous values of the sequence), so at the very least we all know it was basing predictions on latest historical past. However in contrast to ARIMA which explicitly makes use of, say, the final worth instances a coefficient, the LSTM has no such express construction — it realized an inside illustration. For a lot of practitioners, this black-box nature is a disadvantage if the forecast must be defined to stakeholders. In our experiment, the LSTM/GRU principally “memorized” the sequence — we are able to’t extract a lot perception from that, besides to say the mannequin has sufficient capability to imitate the coaching knowledge extraordinarily nicely.

When it comes to uncertainty estimates, Prophet (and ARIMA to some extent) have a built-in solution to generate them. Prophet’s intervals are based mostly on the variability within the development and noise, ARIMA’s forecast intervals come from the variance of the mannequin’s error time period. These offer you a way of confidence within the prediction. For the opposite fashions:

• LightGBM doesn’t inherently present uncertainty, however one can prepare an ensemble of fashions or use strategies like quantile regression to get prediction intervals.
• LSTM/GRU additionally don’t present uncertainty out-of-the-box. You’d possible want to make use of Monte Carlo dropout or an ensemble of networks or some Bayesian strategy to quantify uncertainty. That is an additional step if figuring out confidence is necessary.

Totally different fashions have completely different necessities and suppleness in what they will deal with:

• Dealing with lacking knowledge: LightGBM can naturally deal with lacking values in options (it will possibly deal with lacking as a separate class in tree splitting). ARIMA/Prophet usually require a steady sequence — if knowledge is lacking, you’d often fill or interpolate it. Prophet can deal with gaps in dates gracefully (you possibly can have days with no knowledge and it simply treats it as no occasion, so long as you present the dates; but when timestamps are lacking utterly, it’s important to fill the time index). LSTM/GRU require an entire sequence (you would possibly fill lacking values or masks them out — however masking is superior and never generally accomplished for forecasting). Total, fashions like LightGBM and Prophet are a bit extra sturdy to lacking knowledge conditions (Prophet by advantage of having the ability to work with irregular timesteps to some extent, LightGBM by dealing with NAs), whereas ARIMA and RNNs often want a preprocessed, full sequence.
• Seasonality and development: Prophet is constructed to deal with a number of seasonalities and a piecewise linear or logistic development. Should you suspect yearly seasonality or weekly seasonality, Prophet can incorporate these simply (only a parameter swap or it does it routinely if knowledge helps it). ARIMA can deal with seasonality should you configure a seasonal ARIMA (however you have to know the interval and presumably distinction the sequence). LightGBM can deal with seasonality should you feed it seasonal indicators (e.g., sine/cosine of day-of-year or dummy variables for month, and so on.). LSTM/GRU can in concept be taught any seasonality on their very own if given sufficient knowledge — they’ll detect the periodic sample — however they may have to see a number of cycles and have sufficient capability to memorize it. In our case, every day seasonality was realized by LSTM/GRU simply from the information; Prophet we explicitly informed about every day seasonality; ARIMA we set seasonal order to 24h; LightGBM we included a 24h lag characteristic. Every mannequin wanted that seasonal information both explicitly or implicitly. For extra complicated seasonal patterns (say each every day and yearly), Prophet and LightGBM (with options) would make it simple, ARIMA may with cautious setup, LSTM/GRU may however would wish plenty of knowledge.
• Exogenous regressors: Prophet and LightGBM are each naturally suited to together with further regressors/options. As we noticed, Prophet’s add_regressor makes it easy you probably have one thing like one other sensor studying or a identified future enter (like a deliberate occasion). LightGBM, being a normal ML mannequin, handles any variety of options – it’s maybe essentially the most versatile on this regard (categorical, steady, and so on., simply throw them in after encoding). ARIMA can embody exogenous variables too (that will be an ARIMAX or SARIMAX mannequin), nevertheless it usually solely handles a small variety of regressors and assumes linear relationships with them. It’s extra guide to make use of (it’s important to guarantee these regressors align and stay out there for forecasting). LSTM/GRU may also take a number of enter sequences – you simply stack the extra regressors at every time step as further options within the enter vector. We didn’t try this in our easy experiment, nevertheless it’s generally accomplished (e.g., feed previous temperature and previous humidity to foretell future temperature). So all fashions can use regressors, however the ease varies: Prophet and LightGBM make it simple; ARIMA and neural nets require extra guide work and cautious thought.
• Robustness to anomalies/outliers: Prophet is designed to be sturdy to outliers (it makes use of a way akin to median/quantile regression for development and seasonal becoming), so just a few loopy spikes gained’t throw it off an excessive amount of. ARIMA and LightGBM will be delicate to outliers until you deal with them (ARIMA’s error distribution will be skewed by large shocks; LightGBM would possibly attempt to match them until you restrict tree depth or use loss capabilities much less delicate than MSE). LSTMs may also get thrown off by outliers until educated nicely (they may deal with an outlier as a sample to be taught if it seems in coaching). In follow, one would possibly do outlier elimination or transformation earlier than feeding to those fashions.

Total, LightGBM and Prophet emerged as fairly versatile in accommodating varied knowledge quirks (with Prophet excelling in built-in seasonal dealing with, LightGBM excelling in multi-feature integration). ARIMA is a bit inflexible (wants stationary knowledge, single sequence, guide tuning for seasonality) and LSTM/GRU are versatile in type (they will mannequin virtually something given sufficient knowledge) however require cautious preparation (constant sequences, scaling of knowledge, and so on.) and aren’t as forgiving should you don’t have quite a lot of knowledge or in case your coaching knowledge isn’t consultant of the longer term.

Which mannequin is best to deploy in a real-world situation? The reply will depend on constraints like what number of forecasts you have to produce, how regularly you retrain, and whether or not you want explainability:

• ARIMA is light-weight to deploy. A single ARIMA mannequin is only a small set of coefficients — making a prediction is a matter of some arithmetic operations. If it’s important to deploy hundreds of per-product or per-sensor ARIMA fashions, that’s possible (simply retailer their coefficients and do the recurrences). Nevertheless, sustaining ARIMA fashions will be labor-intensive if accomplished manually (checking for stationarity, re-fitting fashions as tendencies change, and so on.). There are auto-ARIMA instruments that may automate hyperparameter choice, but when your knowledge traits change, you would possibly have to replace the mannequin parameters periodically. In environments the place the information slowly evolves, one may even automate ARIMA refitting now and again. It’s not laborious to combine (pure Python and even in-database implementations exist), nevertheless it won’t scale nicely in case your forecasting downside turns into very complicated (a number of seasonalities, many regressors — at that time ARIMA’s simplicity won’t suffice).
• Prophet can be pretty simple to deploy for a average variety of sequence. It’s only a Python/R library name, and if you have to produce say just a few hundred forecasts every day, Prophet can deal with that on an honest server. The interpretability is a plus in deployment as a result of you possibly can connect explanations to your forecasts (like “the rise is because of seasonal results” and so on.). Prophet fashions are a bit heavier than ARIMA (since every mannequin has to retailer seasonal Fourier phrases, and so on., nevertheless it’s nonetheless not big). The place Prophet would possibly wrestle is should you want large scale (hundreds of sequence up to date in real-time) — it’s not as optimized for velocity as some machine-learning fashions. Additionally, Prophet doesn’t adapt on-line — you’d retrain it if new knowledge is available in and also you wish to replace the forecast. The excellent news is retraining Prophet isn’t terribly gradual for average knowledge, and it’s automated (no hyperparameters to tune besides maybe seasonalities). In real-world deployments, Prophet is commonly used for weekly or month-to-month forecasts the place an analyst is within the loop, moderately than high-frequency streaming predictions.
• Prophet with regressors in deployment requires that you’ve the longer term values of these regressors out there. This will complicate issues: for instance, should you embody “humidity forecast” as a regressor for temperature forecast, you now want a supply of humidity forecasts to enter. In case your regressors are deterministic (like a identified vacation schedule or a advertising and marketing plan), that’s nice. But it surely means the forecasting pipeline will get a bit extra complicated (you have to collect or predict the exogenous inputs earlier than forecasting the goal). The mannequin itself is much like Prophet in complexity — simple to run — simply guarantee your future DataFrame is ready with regressor values. So deployment continues to be okay, however with the caveat of needing these further inputs reliably.
• LightGBM is very deployable. It’s used broadly in business for a lot of prediction duties. A LightGBM mannequin is basically a bunch of choice bushes, and making use of them may be very quick (the compiled C++ code is environment friendly). It may also be utilized in embedded programs or low-latency environments. The principle consideration is that you have to have your characteristic era pipeline wherever you deploy. Which means should you forecast hourly and use the final 24 hours as options, your manufacturing system must at all times pull the final 24 hours of knowledge, compute no matter options (lags, transferring averages, day of week, and so on.) precisely as was accomplished in coaching, after which feed it to the mannequin. This characteristic engineering step should be stored in sync with the mannequin. However that’s true of any engineered strategy — and it’s manageable with good MLOps practices. LightGBM fashions are additionally small in reminiscence and quick to retrain if wanted. In case your utility wants steady updating, you may even periodically refit or replace the mannequin with new knowledge (although on-line studying isn’t built-in, you’d possible prepare from scratch or use heat begin with new knowledge appended). In abstract, LightGBM is production-friendly, however it’s important to handle the information pipeline for options. It doesn’t inherently give explanations to end-users, however you possibly can increase it with purpose codes (like “yesterday’s temperature was an enormous think about right now’s forecast”).
• LSTM/GRU neural networks are the heaviest to deploy. They require a runtime that may execute the neural web (TensorFlow, PyTorch, or an exported model like ONNX runtime). The mannequin measurement will be bigger (although our easy ones have been most likely small — tens of KBs — however many actual use instances have a lot bigger networks). Inference will be quick (just a few milliseconds per step on CPU, or sooner on GPU if parallel), however you probably have many fashions or a really lengthy sequence to foretell, it will possibly add up. You probably have one LSTM mannequin dealing with many sequence (for instance, a single mannequin that was educated on all of your time sequence by concatenating knowledge with a sequence ID characteristic), that might be environment friendly. However in our case we educated a separate LSTM for one sequence; scaling that to a whole lot of sequence could be painful (each coaching and serving a whole lot of neural nets). Moreover, sustaining neural nets means you have to monitor for drift and doubtlessly retrain with new knowledge periodically, which will be gradual and requires experience to make sure they don’t overfit or underfit. On the plus facet, frameworks do permit some optimizations (like quantization, and so on.) to hurry up and lighten fashions for deployment, and should you do want the best possible predictive accuracy and have the infrastructure, deploying an LSTM/GRU is doable — many firms do use deep studying in manufacturing for time sequence (particularly in instances like demand forecasting, the place a single mannequin could be realized for a lot of merchandise). But it surely’s a much bigger funding in tooling and information. Interpretability can be a hurdle — in a regulated business, deploying a black-box mannequin won’t be acceptable with out further clarification instruments.

To sum up the deployment facet: for fast, easy deployment, LightGBM and ARIMA are nice (quick, light-weight). Prophet can be fairly deployable and has the benefit of built-in interpretability, making it a sensible choice for reporting contexts. Neural networks (LSTM/GRU) will be deployed with fashionable ML infrastructure, however they require extra upkeep and oversight — you’d select them when the accuracy acquire is value it and you’ve got the sources to assist them.