Time-Series Forecast with Adaptive Feedback Controlled Predictor

This paper describes a novel approach to predicting time-series which blends techniques developed in the areas of observer design and numerical solvers for ODEs. The developed predictor is based on a novel feedback control architecture which leads to computationally efficient and a fairly accurate forecast even for volatile economic series. Application to series of various kinds shows that the developed forecaster possesses some basic properties of numerical solvers for ODE. In the same time it prediction horizon is favorably compared with a time step attaining in numerical simulations for the series with precisely known models whereas no knowledge of the series’ global model is assumed in our forecast. We demonstrate that for noisy series the accuracy of prediction reduces to the level of noise to signal ratio as well as that reduction of noise by smoothing the series comparably increases the accuracy of prediction. It is also shown that the developed approach provides practically valuable forecast in application to volatile economic series.


Introduction
Modeling and prediction of time series is an important problem for various fields of contemporary science and engineering. It frequently is considered in probabilistic setting and is based on utility of running models with successively updated parameters which leads concurrently to reduction of noise and prediction of subsequent series' values. For example, ARMA and ARIMA are popular stochastic forecasting techniques that are based on linear models [1].Some nonlinear models, as ARCH [2] and GARCH [3]and their different extensions, are developed for taking into account heteroscedasticity -a change in variance along time. More details for these models and current literature review in this area could be found in [4]. Representation of time-series in state-space form attracts application of Kalman and particle filters to linear and nonlinear problems in stochastic estimation and forecast; more details and references on these subjects could be found in [4,5] and [6]- [8]. Comparable techniques for stochastic data assimilation, which also are based on Bayesian inference methods, were recently presented in [9]. Note that in practice estimation of noise statistics involve some errors which could reduce performance of optimal stochastic estimators as Kalman filter and alike. This is especially apparent for non-stationary series.
Artificial intelligence [10] and soft computing [11] paradigms were used extensively for forecasting of series. These techniques mainly are based on application of artificial neural networks [12] - [15], support vector machine [16,17], fuzzy logic [18]- [ 20] and their various combinations to find of repeated patterns and forecast of series. It was reported in cited above papers that artificial neural networks and probabilistic techniques like ARIMA frequently provide quite similar forecast. Some papers integrate artificial intelligence and ARIMA forecast methodologies [21].
Wavelets decomposition has been used in forecasting of series in a number of publications [22,23] which report mixed results [22].Various smoothing techniques have been applied to series forecast as well [24].
Note that in a diverse and broad field of forecasting we quote above only some resent and known publications where more details and subsequent references could be found.
Most forecasting techniques frequently provide a lagging forecast. The lag normally extends for volatile and rapidly altering series which contribute to their relatively humble predictability.
The concept of feedback controlled observers/estimators, which have been primarily developed in control literature, presents an attractive approach to forecasting of deterministic/denoised series and sometimes their derivatives. In the area of control a concept of observer was developed by Luenberger [25]- [27] for linear systems; subsequent and more recent references could be found in [28]. Design of observers for nonlinear and discrete-time systems was suggested in [29] - [35]. In physics literature the methodology of observer design was recently used for estimating of parameters and states of nonlinear and chaotic 48 Time-Series Forecast with Adaptive Feedback Controlled Predictor systems, see [36,37]where additional references in physics literature on this subject can be found.
An observer for continuous systems can be written into a forecaster form by the utility of delayed observations while a discrete observer and forecaster formally can be presented by practically identical models. The design of observers for estimating directly immeasurable system states typically is based on stability control of zero-equilibrium solution to so known error-equations which implies decay of estimation error. However, this approach cannot be directly utilized for prediction of series for which the underlying models are practically unknown. Hence forecast of series is based normally on utility of artificial running models with sequentially estimated parameters.
This paper blends techniques from the areas of numerical analysis and control and develops approach for predicting of the next value of a time-series. Enlarging of sampling step leads to prediction of series on extended time horizons. For simplicity we adopt a polynomial running model controlling by a linear feedback. Various more complex models and control architectures could be naturally included into this approach. However, we notice in our simulations that a gain in model complexity could deprive the forecast accuracy which is especially apparent for volatile and noisy series.
We derive a discrete forecast model from its continuous analog by application of the implicit Euler's method and estimate control parameters of the running model by minimizing least square error norm combining delay-errors in both the state variable and discrete approximation of its derivative. The developed procedure is applied to various time series where it delivers a fairly accurate forecast. This paper is organized as follows. Section 1 presents a continuous prediction model which is written in section 2 in a discrete form adapting for forecasting of series that is common in applications. Section 3 tests the accuracy of the developed forecasters in application to various kinds of series. Sections 4 and 5 discuss and summarize this study.

Estimator/Forecaster for Continuous Series
While our objective is to develop a forecaster for discrete time-series, we begin with brief review of the methodology which can be used for estimating of the derivatives and forecasting of continues series and subsequently adapt it for forecasting of data sampling with a finite step. Since ) (t x represents a measured/denoised variable, we assume initially that its first derivative, ) (t x′ , could be derived directly from the data or measured independently. Using these observations we intend to estimate higher order derivatives of this series through utility of its polynomial model as follows: where control parameters const b a k k = , normally are chosen to ensure asymptotic stability of the equilibrium solution to the error equations which can be written as: control parameters for these estimators [28]. If only delay observations are available, (1) can be written as 1 1 whereτ is the delay. Last equations could be interpreted as delayed estimator or predictor equations.
Obviously, observation/forecast errors decay if zero equilibrium solution of (3) is stable. Various stability conditions for delay equations could be used to determine controller parameters for equation (3),see [38,39] and subsequent references therein. However application of this approach to forecasting of relatively complex series is limited since it adopts a global polynomial model; consequently control parameters k a and k b are constant in this setting. Note that utility of more complex models could improve forecast accuracy if they are naturally embraced by the given series. Otherwise a simpler model often turns out to be more robust and superior.

Adaptive Forecaster for Discrete Time-Series
This section derives the equation for a discrete forecaster as an approximation to it continuous analog. Approximation of derivatives in (3) using backward differences yields a system of difference equations which also can be obtained by application of implicit Euler's method: ∆ is a 2 nd -order approximation to first derivative of time-series which is computed as follows: Now we could shift i forward by one unit to write (4) formally as a forecaster equation; instead we keep (4) and assume that a current observation is adopted on 1 i − -step and prediction is delivered on i -step. Prediction on a few time-steps can be obtained from a slightly modified version of this equation which assumes thatτ equals to a positive integer. Note that prediction horizon is determined by sampling step h which normally should be equated to one in this setting. Yet, our simulations show that an appropriate choice of h-values in some cases enhances the accuracy of forecast providing by (4). Now we write (4) in the form suitable for numerical simulations: Thus the objective function for the corresponding optimization problem includes two components: the least-square errors for predicting values of the series and difference approximation of it first derivative

Simulation Results
Note that (6) implies a nonlinear optimization problem with multiple extremes which can sensitively depend upon initial values of adjustable parameters. To reduce this sensitivity and enhance the forecast accuracy we apply exponential smoothing to successive values of these parameters -a technique is commonly used in simulations to enhance convergence of iterations.
We will use two measures of forecast errors: where medium of a sequence of forecast values is computed on the entire interval where the data is available. Note that while these measures fairly are gaging relative accuracy of prediction, their absolute values could remain small for a lagging forecast which reduces their practical significance for series with frequently changing directions, for example, for some economic series. Thus to assess the forecast lag, it is also instructive to examine how accurately the system can predict local/major turning points of the series -the ultimate goal of an automatic forecast.

Forecast of Multifrequency Data Set
To evaluate how the prediction accuracy depends upon variability of data, we forecast the values of oscillatory series which is defined by sampling with a fixed step a periodic function with period 2π: Where a is an amplitude and K=6 is used in our simulations. Figures 1 and 2 compare actual and predicted series values when x is sampled with steps π /2.2 and π /3.  (7) sampling with three different step sizes (1st row).
Note that forecast accuracy is waned if x is sampled with a step larger than π /2.2. For smallest of these three steps the 50 Time-Series Forecast with Adaptive Feedback Controlled Predictor forecast yields superior accuracy and practically negligible lag while for the largest step it delivers a satisfactory accuracy as well. It could be seen from figure 1 that forecast error slightly decreases over time due to adaptation of the model to the data set. This effect is more apparent on initial time intervals where the forecast error rapidly decays. Note that the accuracy of this forecast practically exceeds one which can be delivered by standard numerical techniques in the cases when a model for the series is precisely known and thus can be simulated.

Forecast of Trending Multifrequency Data Set
Our simulations show that to retain the desired accuracy for steeply-trending series, the forecast horizon normally should be decreased. This is similar to acknowledged performance of numerical solvers for stiff ODEs. Yet, the trend of the series could be removed by application of such standard procedures as subtracting a long moving average from the data, subsequent differencing of the data, etc. Since the trend rarely can be removed completely, figure 3 compares actual and predicted values for slow-trending data that is formed by sampling the following function with step π /3: 1 ( sin( / )) 4 ln( 1)

Forecast of Multifrequency Noisy Data Set
To reveal the effect of noise, we predict upcoming values of series (7) which is augment by additive noise as follows: where ( ) N t is zero mean noise which is assumed to be either uncorrelated uniformly distributed in interval 0.2a ± or normal noise with standard deviation equals 0.25a . Figure 4 shows that addition of uniformly distributed noise noticeably increases prediction error which still remains close to noise to signal ratio. In fact, the error measures in this case are: 1 0.23 M = and 2 0.29 M = .Note that to enhance prediction accuracy in this case we significantly amplified the rate of exponential averaging which is applied to the running set of adjustable control parameters.

Figure 4.
Comparison of actual and predictive values for multifrequencydata set which is augment by additive uniformly distributed uncorrelated noise. The data is sampled with step π/3.  Comparison of actual and predictive values for multifrequencydata set which is augment by normal noise with standard deviation equals 0.25a . The data is sampled with step π/3. Application of slight exponential smoothing to this series reduces its noise level and subsequently decreases the error measures. Thus addition of noise increases roughness of the series and practically to the same degree increases prediction errors. In turn, smoothing noisy data enhances the accuracy of prediction delivering by our technique.

Forecasting of Chaotic Series
In this section we estimate the accuracy of forecasting of chaotic series that are developed by simulating the Lorenz equations [40] by forward Euler's method with a constant time-step. We chose a standard set of parameters for these equations that implies chaotic regimes [40]. It could be empirically determined that the forward Euler's method becomes unstable in simulating of these equations if it time-step exceeds 0.01. This value is used as a benchmark for comparison with the length of forecast horizon reachable by our forecast technique while some other numerical solvers would be able to run with larger step for this problem. . Figure 8 compares simulated chaotic solutions to Lorenz equations with prediction of the corresponding chaotic series sampling with step equals three which exceeds our benchmark in 300-times! It also shows gradual adaptation of our forecaster to the data set.

Forecast of Economic Series
Predicting prices of securities trading on exchanges even on short time-intervals is a notoriously challenging problem. Most of known techniques provide lagging forecast for these data which have questionable practical value. A more special problem in predicting only the price direction seems to be fairly intractable as well probably due to frequent turns in price direction. Indeed, application of our approach to predicting prices of etfs tracking the indices of major securities delivers feasible results. Here we assess the accuracy of predicting the next value for series comprising of daily-close prices of two etfs -spy and agg representing major stock and bond indexes. Note that in this case application of standard statistical error measures might be misleading. For example, a straight line fitting current and previousdata-entriesreturns simplest but lagging prediction with relatively small error measures for these series while a trading system following this forecast consistently lose money. Thus a practical value of this forecast could be evaluated by inspecting the plots of original and forecasted series as well as by evaluating the equity-line of a trading system which follows this forecast. Figure 9 presents forecast of agg-series which has been previously detrended by subtracting from the corresponding data a long exponential moving average.  The forecaster shows transient behavior on an initial short time-interval where the running error is gradually declined but some bursts of the error are present over the entire time-interval. Similar forecast is obtained for spy daily prices as well. To assess the practical value of this forecast, we simulated performance of a trading system that automatically buy/sell-short of a security due to gain/loss in it predicted value. Figure 10plots time-histories of equities accumulating by this system and buy-and-hold approach on ten-year interval for agg-price series. The trading system makes more than five times than its benchmark and it equity line remains above the benchmark for the entire time-period while the trading system remains practically as volatile as the benchmark especially during recent stock market crash.
Fairly comparable results were found in application of this trading system to etfspy, see Figure 11. In this case the trading system also earns a few times more than buy-andholds benchmark on the chosen time-interval and remains slightly more volatile than the benchmark as well. Figure 10. Comparison of equity lines for simple trading system utilizing the developed forecast ( blue-line)and buy-and-hold approach (red-line) on ten-year time interval for agg-daily-close prices. Figure 11. Comparison of equity lines for a simple trading system utilizing the developed forecast ( blue-line) and buy-and-hold approach (red-line) for spy-daily-close prices.
We would mention that application of ARIMA for predicting next values for these two datasets drastically amplifies the running-time in our simulations and delivers the trading systems that are inferior to buy-and-hold benchmark on corresponding time-intervals.

Discussion
This paper developed and tested a new approach to predicting the next value of series which adopts some techniques developed in the areas of observers' design and numerical solvers for ODEs. A numerical solver approximates next solution value using some of its previous values and the knowledge of the governing equations. Since typically the underlying models for series are unknown, their forecasters are naturally defined as running systems with sequentially adjusted parameters. Our system formally emulates a linear observer for a polynomial model. A typical design of observer requires knowledge of the underlying model which could include some uncertainties. This allows determining control parameters which stabilize zero solution to error-equations and consequently ensure decay of observation errors for all time values. In turn, in series forecast the parameters of feedback control of the running model are sequentially adjusted to minimize the prediction error computing on some former time-intervals. A corresponding nonlinear optimization problem is solved numerically. To minimize the running time for this program, we assume that next optimal control values are seeded by the current ones.
The objective function for the corresponding optimization problem includes two components which are weighted by a sequentially adjusted parameter. These components are the least-square errors for predicting values of the series and difference approximation of it first derivative. For later, more accurate second order approximation has been chosen. To our knowledge this type of control of observation error has not been used in practice probably since it was assumed that differencing of data amplifies it noise component. Yet, we found that application of this type of control decreases prediction error even for the most volatile economic series. But our attempt to advance this control architecture by including error components in second derivative was not fruitful.
In some cases the prediction accuracy could be substantially elevated by increasing the number of adjustable parameters which possibly let to escape trapping in a local minimum. This was particular apparent in forecasting of volatile economic series where the time-step, h, was added to adjustable parameters. To enhance convergence of numerical minimizer, we smooth running sequences of adjustable control parameters using exponential moving averages. The degrees of averaging were tuned to each particular series and we found out that the optimal degrees of averaging normally would increase for more volatile series.
Our running system implements a model of third degree polynomial controlling by a linear feedback. More complex models and nonlinear control architectures turns out to be inferior in our simulations especially ones that were applied to forecasting of economic series probably due to their high noise to signal ratios. However this does not preclude favorable utility of more complex models and advance control schemes in other set ups. Prediction accuracy of this forecaster reduces for more volatile and steeply-trending series. Moreover, this forecaster becomes unstable if the volatility of a series measuring, for example by its standard deviation, exceeds a certain threshold. For ODEs the corresponding behavior is known as stiffness and normally would require both application of special solvers and reduction in step-size. This motivated us to implement different stiff-solvers schemes fordiscretizing of a continuous observer model to only find out that implicit Euler's method delivers the best performance across a considerable range of data. In the same time we found that reduction in sampling step, attaining by data interpolation, decreases prediction error which matches performance of ODEs solvers. However, such augment of data could be impractical in some applications.
In our simulations the data were detrended by subtracting its long exponential moving average. Yet, reduction of volatility requires application of some smoothing procedures which usually introduce time-delay and decrease the accuracy of forecast. To reveal more details of this behavior, we smooth the entire data-set using resistant to outliers median and rloess-filters which are available in matlab.
Smoothing the entire data set voids time-delay for all entries which are distant from the end-data entries at least by half length of smoothing window since for every such entry the smoothing utilizes subset of data symmetrically locating about this point. Note that this type of smoothing cannot be used for actual prediction since it uses unavailable future data and is considered here for illustration purpose. Minor smoothing of this kind develops data sets which cut off the sharpest picks of the data but otherwise leave the modified data fairly close to the original one. We were surprise to find that predicting next values for such slightly less volatile series turns out to be significantly more accurate than forecasting of the original data set. To illustrate this behavior, we predict next value for such smoothed series consisting of agg-daily-close-prices. Figure 12 compares actual and forecasted values for this data set and displays that after a short transition period the forecaster delivers superior prediction precision.
To better assess the prediction accuracy of such smoothed data; we simulated automatic trading of the corresponding security due to our projections in a way that was already described above. We experimented with data sampling with different time-steps across various securities and found out that these automatic trading systems delivers superior results even for most volatile data sets. Figure 13 plots the equity line of such system trading slightly smoothed data of 18-yearsdaily-close-prices for Apple-stock (appl) and shows that the developed predictor offers superior profitability for such volatile synthetic series. Figure 13. Comparison of buy-and-hold (red-line) and trading-system (blue-line) performance for smoothed daily-close data for Apple-stock, (appl). Note that the system follows synthetic data set that was developed by application of median-filter to the original data set.

Conclusion
This paper presents a novel approach to the design of series predictors which blends some techniques that are developed in the area of observers' design and numerical solvers for ODEs. To assess it accuracy, we apply this predictor to various kinds of data including synthetic multifrequency series which were also corrupted with different kind of noises, data produced by simulating of chaotic Lorenz system and various volatile data sets comprising of historical prices for different securities trading on stock exchanges. We found out that this technique delivers rapid and fairly accurate forecast which could be used in various online applications. In the same time, the accuracy of this forecast declines for steeply trending and volatile series which qualitatively matches the corresponding behavior of ODEs solvers. Indeed, for multifrequency and chaotic data our forecast horizon exceeds the maximal time step that typically could be used in numerical simulations of comparable ODE-solutions while we did not assume any knowledge of the underlined models for these data in our simulations.
We also demonstrated that adding substantial uniformly distributed or normal noises to deterministic data increases forecast errors up to the levels comparable to noise to signal ratios for these series. Next we show that smoothing the data by application of exponential moving average proportionally reduces forecast error measures for these noisy data. These observations embrace splitting the forecast of noisy data into two successive steps -denoising/smoothing and subsequent forecasting of deterministic series. The first step could be accomplished by application of various techniques that will best utilize the information available for the noise component in the data.
Application of our technique to forecast of the most volatile series comprising of stocks and etfs-prices produce some encouraging results as well. Indeed, a trading system which generates buys/sells signals due to this forecast consistently beats by a large margin a buy-and-hold approach on long time intervals while remains practically as volatile as the underlying series. In turn, forecasting these series using ARIMA significantly amplifies the running-time in our simulations and delivers inferior prediction on corresponding time-intervals.