MARSS - MARSS 3.11.7 (GitHub)

Note: See also notes from 3.11.6.

This release is focused on moving the MARSS User Guide and derivation files into a inst/userguide and inst/derivations along with Makefiles to build. This will facilitate updating MARSS and converting the User Guide to an eBook.

• Added pkgdown. Note this required special files for MathJax in the pkgdown folder.
• Removed packages in Suggests: that were associated with the User Guide and tests and not used in examples.
• Converted Quick_Start.Rnw to Rmd so it works better with pkgdown and renders to html for CRAN.
• Updated DESCRIPTION and README to refer to marssTMB.
• Added marssTMB to Suggests with Additional_repositories: https://atsa-es.r-universe.dev

BUGS

• In predict_marssMLE(), the newdata model.tsp was not being set.
• In plot.marssMLE(), the CIs for the state residuals did not plot correctly because there is always a NA at the end.
• In `autoplot.marssMLE(), two plots would always be plotted if one plot type was specified.

DOCUMENTATION

• Added vignettes/Learning_MARSS.Rmd
• Fixed various equation formatting typos (unbolded x's and x's that should have been X's in expectations).
• Added MARSS.dfa() to the manual built by CRAN by removing keyword internal.
• Added inst/DEVELOPER_NOTES.md
• Minor change to make the v and w in EMDerivation.Rnw in slant font as they are random variables.

- R
Published by eeholmes about 3 years ago

MARSS - MARSS 3.11.6 (GitHub)

This release is focused on adding new method method="TMB" which uses the package {marssTMB}.

• Added methods "TMB", "BFGSTMB" and "nlminbTMB" and changed MARSS() to recognize these.
• Added the generic function MARSSfit() and methods for "kem" and "BFGS".
• Added the defaults to R/onLoad.R and made it easier to specify new methods and constraints on methods
• Made changes to checkMARSSinputs.R, is_marssMLE.R and is.validvarcov() so they are not so specific to method="BFGS" but work for any methods with similar constraints.
• Export MARSSvectorizeparam(). Need when writing methods for MARSSfit() generic.
• Added {generics} as a dependency and removed {forecast} and {broom} from Suggests. Cleaned up examples and text throughout to remove references to these packages.

BUGS

• The iterations count for optim() was using number of function calls not number of gradient calls.

OTHER

• Allow toLatex() to output the raw tex so that it can directly go into RMarkdown or Quarto.
• Use match.arg() for form and method.
• Fix typo in DLM chapter re y_1^{t-1}

- R
Published by eeholmes about 3 years ago

MARSS - Version 3.11.4

This is a CRAN release also. This release is focused on improving the plotting functions for marssMLE, marssResiduals and marssPredict objects. The website links also needed to be updated to the new GitHub organization home for MARSS (and the other ATSA material): atsa-es.

ENHANCEMENTS

• Added a variety of new plots to autoplot.marssMLE and plot.marssMLE: all types of residuals with all possible standardizations versus time, plus ACF and QQPlots for the same, and all possible fitted y and x plots. Cleaned-up the plots in various ways (e.g. missing CIs). Added notes (that can be turned off) to the bottom of the autoplot plots to explain what the plot is and guide the user to the more standard plots.
• Revamped residuals plots made by autoplot.marssMLE() and plot.marssMLE() to allow a full range of residuals plots but to only show a subset for a specific set of residuals diagnostics plots by default.
• Created autoplot.marssResiduals() and plot.marssResiduals() for marssResiduals objects. Simplifies standard residuals plots. This needs to be separate from plot.marssMLE() (i.e. cannot be called from plot.marssMLE) since it is designed to plot whatever happens to be in the marssResiduals object passed to plot.marssResiduals(). plot.marssMLE() runs residuals() to create a specific set of residuals diagnostics plots.
• Improved autoplot.marssPredict() with better titles and notes below the plots. Removed pi.int as an argument for autoplot.marssPredict() and plot.marssPredict() as this was extraneous. The PI/CI info is pulled in from the marssPredict object.
• Added utility function match.arg.exact() which does exact argument matching. The base R match.arg() uses pmatch() and does partial matching. This is a problem for many functions where "xtt1" is different than "xtt". This function implements exact matching.
• Added row and column names to the output from coef.marssMLE() when type="matrix".
• Added more text to help file for MARSSresiduals() to explain variance and correlation for standardized residuals.
• Added var.ytt1 and var.Eytt1 to the output from MARSShatyt() when only.kem=FALSE. This is for convenience for the plot functions.
• Added str_to_sentence utility function for the notes in autoplot.marssMLE() plots.
• Added interval.type to the marssPredict objects otherwise the type of interval in the object (prediction or confidence) cannot be known.
• Removed the loess lines from the residuals plots.

BUGS

• plot.predictMARSS() was not showing the forecasts and the state predictions when h=0 was garbling the CIs and PIs when a short newdata was passed in (short = shorter than original data).
• autoplot.marssPredict() was not using the time info from ts object, so x-axis was showing 1, 2, 3 etc instead of the years, for example.
• MARSS.dfa() used if form="dfa" allowed Z to be passed in. This form is a helper function that forms a default DFA model with a user specified number of trends (m). If the user needs a custom Z, they should not use form="dfa" but use the default MARSS() (form="marxss"). MARSS.dfa() was changed to not allow Z to be passed into the model argument.
• coef.marssMLE() was not properly showing a time-varying A or U when type="matrix", form="marss" and D or C estimated.
• plot.marssMLE() was not resetting par() when done thus affecting users plot environment.
• residuals.marssMLE() value column for states was wrong when there were more than one state because c($.x[2:TT], NA) was used. Changed to not offsetting either .x or .fitted columns and instead added clarification to the documentation for residuals.marssMLE(). The value column was used for examples in help file for residuals.marssMLE() and for the coefficient of determination reported by glance.marssMLE().
• MARSS() was not setting convergence to an error code when Kalman filter function throws and error before model is fit.

DOCUMENTATION

• Fixed some references to MARSSsettings(). This was replaced with pkg_globals in the package environment via .onLoad().
• More clean-up of the equations. Just being more consistent about using \eqn{} when R, Q etc refer to the matrices in the MARSS equation versus code.
• residuals_marssMLE.Rd had a few typos. Main one was that name column was called .type column.
• Fixed a typo in Residuals.Rnw. LaTeX definition for Vtt1 was not working. Also fixed a couple misspellings in EMDerivation.Rnw.
• Did a thorough re-read of the User Guide and did a lot of clean-up.

- R
Published by eeholmes over 4 years ago

MARSS - Version 3.11.3

This is a copy of the CRAN 3.11.3. release. This is an update based on version 3.11.2 and 3.11.2-beta (GitHub releases). It is mainly focused on providing graceful exiting for models that report errors due to ill-conditioned variance matrices and for models with fixed parameters. The testing and output (plot, residuals, tsSmooth, fitted) was made less reliant on MARSSkfss(), which involves an inversion of Vtt1 and which can become ill-conditioned and report an error. The update also fixes a bug in the log-likelihood calculation due to not specifying the tol=0 in SSModel() call. This bug would come up only for variance matrices with extremely high condition numbers fit with method=BFGS. Data and covariates can now be a ts object and the time information will be used for plotting.

ENHANCEMENTS

• Stop all MARSSkfss() calls when trace=-1. MARSSkfss() is used for error checks (because it has verbose information to indicate model problems) but because it uses matrix inversions, it will stop models from being fit just because they cannot be run through MARSSkfss() even if they run fine with MARSSkfas(), which doesn't use these matrix inversions.
• Allow data, covariates and newdata to be a ts object. model.tsp attribute added to model and marss elements of marssMLE object and this information used for plotting and for t column in fitted, residuals and tsSmooth output.
• Add xtt and Vtt to MARSSkfas() to avoid MARSSkfss() calls when unnecessary.
• Added graceful exiting for various functions for marssMLE objects without a par element. This occurs when MARSS() was run with fit=FALSE.
• Added graceful exiting for MARSSparamCIs() when model is fixed and thus no parameters estimated.
• Add alert when there are negative values on the diagonal of VtT which sometimes happens for MARSSkfas(). Give user helpful suggestions for switching the Kalman filter/smoother function.
• Addition of a KFAS chapter which is mainly a testing chapter. All variants of model and state residuals computed by the KFAS package are compared to the residuals produced by MARSS. Similarly the StructTS chapter is mainly a testing chapter which compares output for StructTS objects to the equivalent output for marssMLE objects. The tests in these chapters are repeated in the tests directory (which is on GitHub).

BUGS

• In KFS(), a tolerance correction affected the log-likelihood value when R was below square root of machine tolerance or condition number was very high (if R non-diagonal). This created a large (incorrect) jump in the log-likelihood. This would be reported with a warning that the log-likelihood dropped if using the EM algorithm. Solution was to set the tolerance to 0 in the KFAS model in MARSSkfas(). Note this did not happen for all cases of small R and a warning would have been generated alerting the user to a problem.
• MARSSkfas() did not recognize if H was time-varying.
• If model in MARSS() call was a marssMLE or marssMODEL object, the tinitx and diffuse elements were not being passed in, only the parameter matrices.
• No ACF should be plotted for state smoothation residuals (smoothation residuals are not temporally uncorrelated). Fix applied to plot.marssMLE() and autoplot.marssMLE().
• marssMLE$fun.kf was not always being passed to MARSShatyt() so it didn't necessarily use the function requested by the user.
• coef.marssMLE() did not change what to, say, par.se if type passed in so coef(fit, what="par.lowCI", type="Z") returned coef(fit, type="Z").
• print.marssMLE() and coef.marssMLE() would fail ungracefully if all the parameters were fixed or MARSS() was run with fit=FALSE.

DOCUMENTATION

• Minor fixes to the derivations table in EMDerivation.Rnw and added some information on the initial conditions for the Kalman filter in the expectations section. Typo in eqn 29-31. u^T should have been u. Added information on EM algorithm when parameter set is updated by parts.
• Added notation definitions to the Kalman smoother algorithm section for xtt and xtt1 etc.
• Added information on how to get CIs on rotated loadings to DFA chapter.
• Cleaned up MARSSkf.Rd sections on initial conditions and cleaned up equation formatting so looks better in pdf format.
• Added section on normalization calculations to Residuals.Rnw.
• Cleaned up notation for conditional expectations in the man files.
• Fixed bug in the Covariates chapter which was not showing the B estimates for kem.plank.4.

OTHER

• If tinitx=1, then Vtt1T[,,1] does not exist. Replaced Vtt1T[,,1] with NA instead of 0 in this case. Note Vtt1T[,,1] would never be used in this case as V10T is used instead however a value of 0 is not correct. The value is does not exist so NA is the correct value.

- R
Published by eeholmes over 5 years ago

MARSS - Version 3.11.2-beta

This is a further improvement on release 3.11.2 earlier in September. This is an update to make the testing and output (plots, residuals, tsSmooth, fitted) less reliant on MARSSkfss(), which involves an inversion of Vtt1 which can become ill-conditioned thus preventing inversion. The update also fixes a bug in the log-likelihood calculation due to not specifying the tol=0 in SSModel() call. This bug would come up only for certain variance matrices with very high condition numbers fit with method=BFGS. Data and covariates can now be a ts object and the time information will be used for plotting.

ENHANCEMENTS

• Stop all MARSSkfss() calls when trace=-1. MARSSkfss() is used for error checks (because it has verbose information to indicate model problems) but because it uses matrix inversions, it will stop models from being fit just because they cannot be run through MARSSkfss() even if they run fine with MARSSkfas(), which doesn't use these matrix inversions.
• Allow data, covariates and newdata to be a ts object. model.tsp attribute added to model and marss elements of marssMLE object and this information used for plotting and for t column in fitted, residuals and tsSmooth output.
• Add xtt and Vtt to MARSSkfas() to avoid MARSSkfss() calls when unnecessary.

BUGS

• In KFS(), a tolerance correction affected the log-likelihood value when R was below square root of machine tolerance or condition number was very high (if R non-diagonal). This created a large (incorrect) jump in the log-likelihood. This would be reported with a warning that the log-likelihood dropped if using the EM algorithm. Solution was to set the tolerance to 0 in the KFAS model in MARSSkfas(). Note this did not happen for all cases of small R and a warning would have been generated alerting the user to a problem.
• MARSSkfas() did not recognize if H was time-varying.
• No ACF should be plotted for state smoothation residuals. Fix to plot.marssMLE() and autoplot.marssMLE().
• marssMLE$fun.kf was not always being passed to MARSShatyt() so it didn't necessarily use the function requested by the user.
• coef.marssMLE() would not allow you to change type to, say, par.se.

DOCUMENTATION * Minor fixes to the derivations table in EMDerivation.Rnw and added some information on the initial conditions for the Kalman filter in the expectations section. Typo in eqn 29-31. u^T should have been u. Added information on EM algorithm when parameter set is updated by parts. * Added information on how to get CIs on rotated loadings to DFA chapter. * Cleaned up MARSSkf.Rd sections on initial conditions. * Added section on normalization calculations to Residuals.Rnw.

- R
Published by eeholmes over 5 years ago

MARSS - Version 3.11.2

This is a minor update to stop all MARSSkfss() calls when trace=-1. MARSSkfss() is used for error checks (because it has verbose information to indicate model problems) but because it uses matrix inversions, it will stop models from being fit just because they cannot be run through MARSSkfss() even if they run fine with MARSSkfas(), which doesn't use these matrix inversions.

To install: install.packages("devtools") library(devtools) install_github("nwfsc-timeseries/MARSS@v3.11.2")

- R
Published by eeholmes over 5 years ago

MARSS - Version 3.11.1

Version 3.11.1 is focused on addition of the predict, forecast, fitted and residuals functions along with plotting functions for the output. Documentation for these functions along with background literature and the derivation of the residuals algorithms have been updated. Residuals in state-space models are complex as there are two processes (observation and state), three types of conditioning (data to t-1, t or T), and four types of standardization used in the literature (none, marginal, Cholesky on the full variance matrix, and Cholesky on only model or state residual variance). The MARSS package computes all the variants of residuals. Many of the predict changes are listed below for 3.10.13 release on GitHub. New chapters illustrating structural equation models using MARSS versus StructTS and the KFAS package were added. The KFAS chapter compares the KFAS residuals functions to the MARSS residuals functions. The two packages use different algorithms and different semantics to compute the same residuals.

ENHANCEMENTS

• ldiag() convenience function added to make list diagonal matrices. This replaces having to do code like a <- matrix(list(0),2,2); diag(a) <- list(2,"a"). Now you can call ldiag(list(2,"a")).
• Added accurancy.marssMLE() and accuracy.marssPredict() which returns accuracy metrics sensu the forecast package.
• Added is.unitcircle() utility function and added tol so that it does not fail if abs(eigenvalue) is above 1 by machine tolerance.
• Added ACF plots for model and state innovation residuals to plot.marssMLE() and autoplot.marssMLE().
• Revamped residuals.marssMLE(). Got rid of augment.marssMLE() and renamed it residuals.marssMLE(). The old residuals.marssMLE() became MARSSresiduals(). There was too much duplication between residuals.marssMLE() and augment.marssMLE() and between augment.marssMLE() and fitted.marssMLE(). Also I want to minimize dependency on other packages and augment is a class in the broom package. This required changes to the glance.marssMLE(), plot.marssMLE() and autoplot.marssMLE() code.
• Revamped tidy.marssMLE. tsSmooth.marssMLE now returns the estimates from the Kalman filter or smoother which tidy.marssMLE had returned. tidy.marssMLE only returns a data frame for the parameter estimates.
• V0T was computed with an inverse of Vtt1[,,1]. This led to unstable numerics when V00 was like matrix(big, m, m). Changed to use solve(t(Vtt1[,,1]), B%*%V00) which should be faster and seems to have lower numerical error.
• predict.marssMLE updated to return ytt1, ytt, ytt1.
• Added state innovations and contemporaneous residuals to MARSSresiduals.tt1 and MARSSresiduals.tt but not returned by residuals.marssMLE() (unless clean=FALSE). Only returned by MARSSresiduals().
• Added Block Cholesky standardized residuals.

BUGS

• This bug affected residuals() in cases where R=0. In v 3.10.12, I introduced a bug into MARSSkfss() for cases where R has 0s on diagonal. History: To limit propagation of numerical errors when R=0, the row/col of Vtt for the fully determined x need to be set to 0. In v 3.10.11 and earlier, my algorithm for finding these x was not robust and zero-d out Vtt row/cols when it should not have if Z was under-determined. This bug (in < 3.10.12) only affected underdetermined models (such as models with a stochastic trend and AR-1 errors). To fix I added a utility function fully.spec.x(). This returns the x that are fully determined by the data. There was a bug in these corrections which made MARSSkfss()$xtT wrong whenever there were 0s on diagonal of R. This would show up in residuals() since that was using MARSSkfss() (in order to get some output that MARSSkfas() doesn't provide.) The problem was fully.spec.x(). It did not recognize when Z.R0 (the Z for the R=0) was all 0 for an x and thus was not (could not be) fully specified by the data. Fix was simple check that colSums of Z.R0 was not all 0.
• When computing the Cholesky standardized residuals, the lower triangle of the Cholesky decomposition should be used so that the residuals are standardized to a variance of 1. base::chol() returns the upper triangle. Thus the lines in MARSSresiduals.tT() and MARSSresiduals.tt1() that applied the standardization need t(chol()).
• trace=1 would fail because MARSSapplynames() did not recognize that kf$xtt and kf$Innov were a message instead of a matrix. I had changed the MARSSkfas() behavior to not return these due to some questions about the values returned by the KFAS function.
• is.validvarcov() used eigenvalues >= 0 as passing positive-definite test. Should be strictly positive so > 0.
• MARSSkfas() had bug on the line where V0T was computed when tinitx=0. It was using * instead of %*% for the last J0 multiplication. It would affect models with a non-zero V0 under certain B matrices, such as structural models fit by StructTS().
• The following bug was in MARSSresiduals.tT() and MARSSresiduals.tt1() but was in the old residuals.marssMLE() also. If MLE object had the kf element, kf was not assigned in code since there was no kf <- MLEobj$kf line for that case. Normally MLE objects do not have the kf element, but it could be added or is added for some settings of control$trace. This caused residuals() to fail if trace=2.

DOCUMENTATION and MAN FILES

• Added covariates and example to MARSS_dfa.Rd
• Added chapter on Structural time series models which compares StructTS() to MARSS() output.
• Removed all mention of augment() from documentation and manuals. Replaced with residuals().
• predict.marssMLE.Rd (help page) had bug in the examples. remove Q=Q from the model list in the first example.
• Cleaned-up the man pages for predict() and predict.marssMLE().
• In the chapter on structural breaks and outliers, Koopman et al (1998) use the marginal residuals in their example rather than the Cholesky standardized residuals. Changed to use marginal residuals to follow their example.
• In Covariates.Rnw I show the acf of residuals. This should use innovations instead of smoothations. Only the former are temporally independent.
• Added derivation for joint variance-covariance matrices for innovations model and state residuals.

OTHER

• Changed fitted.marssMLE to have column with .x which is conceptually the same as the y column for observations. It is the left-side of the x equation (with error term) while fitted is the right-side without the error term.
• Changed the x0 estimation behavior for predict.marssMLE() when no data passed in.
• Added x0 argument to predict.marssMLE() so that user can specify x0 if needed.
• Removed the tibble class from the data frames returned by residuals.marssMLE(). The data frames are still in tibble form. Removed all reference to tibbles in the documentation.
• When trace = -1 some tests were still being done. I added a test for trace = -1 to a few more test lines in MARSS.R and MARSS_marxss.R.
• Changed the default behavior of residuals.marssMLE() to return innovations instead of smoothations.
• Columns of model estimated values in fitted.marssMLE, residuals.marssMLE, and tsSmooth.marssMLE have a leading ".".

- R
Published by eeholmes almost 6 years ago

MARSS - Version 3.10.13

Version 3.10.13 mainly has to do with the predict() and forecast() functions along with plotting and printing methods.

This version is still fairly beta, and I am fixing minor bugs and posting to the master branch: * predict.marssMLE.Rd (help page) has a bug in the examples. remove Q=Q from the model list in the first example. * futzing with the man pages for predict() and predict.marssMLE * changed the x0 estimation behavior for predict.marssMLE() when no data passed in. * probably will change the arguments for plot.marssMLE()

3.10.13 ENHANCEMENTS

• predict.marssMLE() Shows the prediction or confidence intervals for data or states. Forecasts can be done by passing in h. newdata can be passed in also and fitted model will be used to fit these data and show the intervals. Output is a tibble. Returns a list of class marssPredict.
• forecast.marssMLE() This does the foreward forecasting past the end of the data. Intended to be called by predict.marssMLE. I did not write a marssMLE method for the forecast generic in the forecast package since that would require that the forecast package be required for the MARSS package.
• plot.marssPredict() plot method for the new marssPredict object. This is designed to look like the plot.forecast() function in the forecast package.
• print.marssPredict() print method for marssPredict objects.
• MARSSkfss() and MARSSkfas() Add rownames to the x elements of the list.
• MARSSkf() Added newdata to allow user to pass in a new dataset to fit with the fitted model.

- R
Published by eeholmes over 6 years ago

MARSS - Version 3.10.12

Version 3.10.12 update mainly has to do with the tidy(), fitted() and augment() enhancements which clarify ytT, xtT and residual intervals for MARSS models. This was a major update though probably users won't notice much as it only affects residuals output. A few minor bugs were fixed which caused errors to be thrown in some rare time-varying cases. One bug that affected bootstrap confidence intervals was fixed. The documentation got a major clean-up. The Residuals report has been heavily edited to improve precision and clarity (with added verbosity). The help files and automated manual from the help files were cleaned-up and some of the internal functions moved out of the manual.

BUGS

• MARSSsimulate() missing values were placed in the wrong positions in simulated data. This would affect all simulated data with missing values and thus any function that used MARSSboot(), for example bootstrap confidence intervals for a data set with missing values. Default is to use Hessian so the user would not normally have encountered this bug and it had little effect on the CIs.
• fitted.marssMLE() Fixed bug in fitted.marssMLE for states when one.step.ahead=TRUE. It was using xtt1[,t-1] instead of xtt[,t-1]. The former meant it only used data up to t-2.
• degen.test() in MARSSkem() was not catching when R or Q was time-varying (and thus degeneracy should not be allowed). Changed to test the 3D of model.dims == 1 or not.
• residuals.marssMLE(..., Harvey=TRUE) would fail if Q, B, or G was time-varying because parmat() called with t+1. Changed to only call parmat() when t<TT. Q, B, and G do not appear in the recursion when t=TT so parmat() with t=t+1 is never needed.
• MARSS_dfa() used form="dfa" in MARSS.call list. Just info. Never used.
• Default A matrix ("scaling") was throwing an error for manually set up DLM models. Problem was call to check that Z was a design matrix in MARSS_marxss.R. It was not catching that Z was time-varying before running is.design().
• toLatex.marssMODEL() Fixed some old bugs in toLatexmarssMODEL.R. Added S3 class declaration in NAMESPACE for toLatex. fixed equation attribute in MARSSmarxss. G{t} was used instead of G{t}. Only affected toLatexmarssMODEL(). Had extra line in build.mat.tex() that removed last line of matrices. This function was not exported so users would never have run into these bugs.
• MARSSkfss() To limit propogation of numerical errors when R=0, the row/col of Vtt for the fully determined x (determined from data) need to be set to 0. My algorithm for finding these x was not robust and zero-d out Vtt row/cols when it should not have if Z was under-determined. MARSSkfss() is not used for fitting and only affected underdetermined models (such as models with a stochastic trend and AR-1 errors). To fix I added a function fully.det.x() to the utility functions. This returns the x that are fully determined by the data. Note, MARSkfss() is the classic Kalman filter/smoother. The MARSS algorithm does not use this normally. Normally MARSSkfas(), build off the Koopman et al algorithm which avoids unneeded matrix inverses, is used. MARSSkfas() uses the Kalman filter/smoother in the KFAS package.

ENHANCEMENTS

• MARSShatyt() Added ytt, ytt1, Ott, Ott1 to MARSShatyt() so that tidy.marssMLE() can more easily return the one-step-ahead preditions for Y(t). Also added var.ytT and var.EytT so you can easily get the estimates, CI and prediction intervals for missing data. Added only.kem to MARSShatyt() so that only values conditioned on 1:T as needed by MARSS kem are returned. This makes the Ey part of a MARSS object smaller and speeds up MARSShatyt() a little.
• tidy.marssMLE() Changed type for tidy() to xtT, ytT and fitted.ytT. tidy() exclusively gives estimates of things (parameters, X, Y, fitted Y) conditioned on all the data.
• fitted.marssMLE() Added interval=c("none", "confidence", "prediction") to fitted() and returns a list with se's (or sd's if prediction) and intervals. Also added conditioning argument to fitted.marssMLE which gives fitted values with different conditioning. Changed default output to tibble.
• augment.marssMLE() Changed standard errors output for augment() to .se.fit for std error of fitted y and .sigma to std error of residuals. This matches what augment.lm outputs.
• plot.marssMLE() and autoplot.marssMLE(). Added plot.par to plot.marssMLE and autoplot.marssMLE so that the plots can be customized. Added plot of ytT to both functions. Changed the residuals plots to use the CIs for the residuals not the loess CIs.
• MARSSinfo() Added "AZR0" to MARSSinfo() to give info if user gets error that A cannot be estimated with R=0. Added more informative message to MARSSkemcheck() for that case.
• Made all if statements checking class of object robust to the class returning more than one class (so vector of length > 1). Due to change in R 4.0.0 where matrix has class c("matrix","array")
• Updated all code to tidyverse style
• Changed residuals.marssMLE(). This is now a helper function which calls MARSSresiduals.tT() or MARSSresiduals.tt1(). The former is smoothation residuals and the latter is innovations (one-step-ahead) residuals.

DOCUMENTATION and MAN FILES

• Added derivation of variance of Y conditioned on y and X to EMDerivation.Rnw. Needed for CI on the missing values estimate.
• Major update to Residuals report. No changes to equations but much editting to improve precision and clarity (with much more verbosity). Reposted to Arxiv. Added innovations residuals.
• tidy, augment and fitted man files got major update.
• internal functions given \keyword{internal} so they don't appear in the documentation, but will appear if you use ? or help.search.
• Rd files extensively cleaned to improve linking and move more of the internal functions out of view of the normal user. Equations cleaned up (though not completely).

- R
Published by eeholmes over 6 years ago

MARSS - Version 3.10.11

Version 3.10.11 has some edits to speed up the code by minimizing calls to expensive checking functions and fixes a bug in MARSSharveyobsFI() that appeared if a parameter was fixed and time-varying and MARSSparamCIs() was called.

1. is.validvarcov is expensive. minimize calls to it. If called with a diagonal matrix, it should automatically pass so added a check to is.validvarcov() to see if matrix is diagonal.
2. is.marssMLE is expensive. Replace with a call to class().
3. MARSSkfss had a bug * used in place of %*% with J0. Would never show up unless V0 estimated.
4. Add S3 methods for broom functions if broom is loaded.
5. Add autoplot function and updated plot documentation to cover autoplot functions.
6. Fixed typos in Case Study 4 and Derivation eq 143b.
7. Bug in MARSSharveyobsFI() which arose if a parameter was fixed and time-varying.

- R
Published by eeholmes almost 7 years ago

MARSS - Version 3.10.9

See version 3.10.5 information. Version 3.10.9 has some minor fixes and added plot() method. Note 3.10.9 is the same as 3.10.8 submitted to CRAN on 3-30-18.

1. The vignettes were using the geometry LaTeX library which CRAN does not have. I removed the dependency on the package and specified geometry with native LaTeX.
2. The augment and tidy functions were not respecting the conf.level. They used conf.level=0.95 always.
3. The DFA chapter had an error in the function that computed the fitted values (observations) under rotation. The fitted values do not change, only the estimated loadings and the trends.
4. The tidy function checks if the user is requesting parameters and rotation if form="dfa". Outputs an error since getting the CIs on a rotated Z is non-trivial.
5. Added a plot.marssMLE() function.

- R
Published by eeholmes about 8 years ago

MARSS - Version 3.10.7-alpha

Pre-release while working on a fix for DFA models. Was an error in documentation re the fitted values when the model is rotated.

Should be:

\begin{equation} \begin{gathered} \yyt = \ZZ\HH^{-1}\xx^\daggert+\aa+\vvt\ \xx^\daggert = \xx^\dagger{t-1}+\ww^\daggert\ \xx^\daggert=\HH\xxt; \ww^\daggert=\wwt \end{gathered}
\label{eq:dfa.rotated} \end{equation} $\xx^\dagger$ are the rotated trends.

- R
Published by eeholmes about 8 years ago

MARSS - Version 3.10.5

CHANGES IN MARSS 3.10.5

3.10.4 and 3.10.5 are the same with a few added functions: augment, tidy and glance functions (sensu broom R package and tidyverse) so that model output works nicely with ggplot().

CHANGES IN MARSS 3.10

The 3.10 update was mainly concerned with the residuals and confidence intervals calculations. Many improvements were also made.

BUGS

residuals.marssMLE() * Erroneous standardized residuals when Z was non-diagonal (and thus also > 1 row). Also changed residuals.MARSSMLE so it returns residuals (which would be equal to 0) when there are 0s on the diagonal of Q or R * If t=1 and x0 was fixed, residuals were not returned. * The wrong residuals were returned if there were any missing values. Fix involved implementing the missing values modifications for the Kalman filter described in Shumway and Stoffer.

inits functions * in MARSSinitsmarxss() function would give error if U, A, C, or D fixed and user passed in inits. inits ignored in this case so should not throw error. * alldefaults could be updated by form. A few functions were neglecting to (re)load alldefaults or to ressign alldefaults when updated: ismarssMLE(), MARSSinits.marxss(), MARSSinits(). The variables in the pkg_globals environment should be (and only be) loaded when needed by a function and only loaded into the function environment.

kalman filter functions * MARSSkf() was not passing optional function args to MARSSkfas(). * MARSSkfss() mis-counting num data points when R=0, V0=0, and tinitx=1. When Ft[,,1]=0 (e.g. when R=0, V0=0, and tinitx=1), MARSSkfss was including the y[1] associated with Ft[,,1]=0 in the # number of data points. These should be excluded since they don't affect x10.

Confidence intervals and std error for R and Q MARSSparamCIs() gave the wrong s.e. for variances and covariances when method="hessian". It also gave the wrong CIs for variances and covariances when var-cov matrix was non-diagonal. There were a series of issues related to back-transforming from the hessian of a chol-transformed var-cov matrix ( Sigma=chol%*%t(chol) ).

MARSSparamCIs(), vrs 3.9 was getting the hessian numerically using a var-cov matrix that had been transformed with a Cholesky decomposition to ensure it stays positive-definite. The upper and lower CIs were computed from the s.e.'s. I back-transformed the Hessian to the original (non-chol) scale the same way I back transformed a var-covariance matrix. But the variance of s^2 is not the var(s)^2, which is what I was doing, essentially. So the s.e. for R and Q were wrong in all cases. Note, using a Hessian to estimate CIs for variance-covariance matrices is generally a bad idea anyhow however.

For non-diagonal matrices. There was a bug in MARSShessian() in subscripting the d matrix when doing the chol transform. Caused NAs for cases with non-diagonal matrices. However, had the se been returned, they would be wrong for non-diagonal matrices because the elements of the chol transformed matrices do not correspond one-to-one to the non-transformed matrices. E.g. the untransformed [2,2]^2 is chol transformed [1,2]^2+[2,2]^2. The hessian used was for the chol-transformation and the curvature of the LL surface for the chol-transformed values is different than the curvature for the untransformed var-cov elements.

Fix: I completely abandoned working with the chol transformed variance-covariance matrices for the Hessian calculation. The chol transformation was not necessary for computing the Hessian since the Hessian is computed at the MLEs and localized.

1 Created a new function MARSSharveyobsFI() which uses the Harvey (1989) recursion to analyticallycompute the observed Fisher Information matrix. This is the Hessian for the untransformed var-cov parameters. So CIs on variances can be negative since the variance of the MLE is being approximated by a MVN (which can lead to negative lower CIs).

2 Harvey1989 is now the default function when method='hessian'.

3 The user can also select method='hessian' and hessian.fun='fdHess' or hessian.fun='optim'. This will compute the Hessian (of the log-LL function at the MLEs) numerically using these functions. The variance-covariance matrices are NOT chol transformed. These are numerically estimated Hessians of the untransformed variance-covariance matrices.

4 Added MARSSinfo(26) which discusses the reason for NAs in the Hessian.

Misc minor bugs * MARSShatyt() was setting ytt1 (expected value of y(t) conditioned on data up to t-1) to y(t), which is incorrect. Expected value of y(t) conditioned on y up to t-1 is Z xtt1 + a * CSEGriskfigure() panel 2 was wrong when mu>0 (increasing population). * CSEGriskfigure() panel 2 CIs were wrong when CI method=hessian since Q had not been back transformed (so was using sqrt(Q)). * MARSSkemcheck()'s test that fixed B is in unit circle failed when B was time-varying and some fixed and others estimated. Also when the eigenvalues were complex, it should test the real part only. * coef.marssMLE() was not stopping when illegal "what: arg passed in. man page did not say what happens when type=parameter. * passing in method not in allowed.methods was causing errors since model conversion and testing happening before checkMARSSinputs and model testing is algorithm dependent (some forms not allowed by BFGS). Added check for method at top of MARSS function.

IMPROVEMENTS * w(t) and v(t) can be specified as G(t)w(t) and H(t)v(t) where G and H are fixed matrices (not estimated). In version 3.10, G and H are restricted to being 0 or identity, however the code is in place for other values. - changes to MARSS.marxss and MARSS.marss to allow G, H, and L passed in - change to MARSSkem to specifiy star lists with G, H, and L (mathbb(elem) in EM Derivation) - changed MARSSkss to use Q=G Q t(G), R=H R t(H) and V0*=L V0 t(L) * added Multivariate linear regression chapter * added chapter on estimating a Leslie matrix from stage time series using a MARSS model. * removed the function MARSSmcinits and added chapter on searching over the initial conditions into the User Guide. As the MARSS models that MARSS() can fit expanded, MARSSmcinits was increasing obsolete and it was impossible to come up with good searching distributions. Because MARSSmcinits was removed, control$MCInits list item was removed also from defaults and from accepted input. * added default inits for c and d in marxss form so that user can pass in inits using coef(fit); was balking because this includes d and c which didn't have defaults. Removed msg referring to need that model be in marss form for inits (not true). * changed MARSS.marxss() to allow c and d to be 3D arrays. This allows one to use inits=fit to set inits and not get a d (or c) must be 2D error. * adding info to MARSSinfo(4) regarding errors about R=0 and x0 not fixed. Added info to the error warnings to direct user to MARSSinfo(). * changed order of MARSS args to be MARSS(y, model= , inits=, ...) * added pchol and psolve functions to return the chol or inverse via solve when there are 0s on the diagonal * added information to print.marssMODEL on summary.marssMODEL. Added silent argument to summary.marssMODEL to block printing to the console. * print.marssMLE(x, what="par") returned a vector of estimated values instead of the list of par. Changed to return the list. * added E[y(t), x(t+1)] to MARSShatyt output. Needed for residuals.marssMLE(). * added code to is.validvarcov() so that it returns an error if the user specifies a structurally illegal variance-covariance matrix. Added info to MARSSinfo(25). * added a model.frame method for marssMODEL and marssMLE * added broom augment, tidy and glance functions for marssMLE * added logLik method for marssMLE objects * added fitted method for marssMLE objects to return Z xtT + u (model fitted value of y)

MISC * moved info in MARSSsettings.R to .onLoad function. * added suppressWarnings() wrapper to KFAS call when R=0 in MARSSkfas since update to KFAS package produces warning messages when R=0. * Typo in Eqn 124 of EMDerivation.pdf. \beta should have been ^{-1}. Typo in Eqns 133 and 134. vec parentheses should have been in front of R in second summation. R in first line of eqn 133, was not referring to R (the var-cov matrix). It should have had a new symbol. Switched to T. Eqn 134 was not R but this 'T'. * added safe to control list in man file MARSS.Rd Left off accidentally. * Small change to DLM chapter to clarify that rotation matrix only exists if Z has more than 2 columns. * All subfunctions for a function moved into the main functions so they are hidden to the rest of the functions. * The logLik function was using the logLik, samp.size and df attributes from the MLE object, but this is prone to creating errors. The user may have changed the model structure or data in a MLE object and is trying to get the new logLik. Changed to recompute the logLik. * removed use of stringr package; did not need * Poor name choice. y.se was not the standardard error of ytT since sqrt(OtT) not sqrt(OtT-ytT^2) was being returned. Changed name of y.se to ytT.se

- R
Published by eeholmes about 8 years ago

MARSS - Version 3.10.4

CHANGES IN MARSS 3.10.4

The main changes for versions 3.10.x have to do with residuals and confidence intervals calculations.

BUGS

residuals.marssMLE() * Erroneous standardized residuals when Z was non-diagonal (and thus also > 1 row). Also changed residuals.MARSSMLE so it returns residuals (which would be equal to 0) when there are 0s on the diagonal of Q or R * If t=1 and x0 was fixed, residuals were not returned. * The wrong residuals were returned if there were any missing values. Fix involved implementing the missing values modifications for the Kalman filter described in Shumway and Stoffer.

inits functions * in MARSSinitsmarxss() function would give error if U, A, C, or D fixed and user passed in inits. inits ignored in this case so should not throw error. * alldefaults could be updated by form. A few functions were neglecting to (re)load alldefaults or to ressign alldefaults when updated: ismarssMLE(), MARSSinits.marxss(), MARSSinits(). The variables in the pkg_globals environment should be (and only be) loaded when needed by a function and only loaded into the function environment.

kalman filter functions * MARSSkf() was not passing optional function args to MARSSkfas(). * MARSSkfss() mis-counting num data points when R=0, V0=0, and tinitx=1. When Ft[,,1]=0 (e.g. when R=0, V0=0, and tinitx=1), MARSSkfss was including the y[1] associated with Ft[,,1]=0 in the # number of data points. These should be excluded since they don't affect x10.

Confidence intervals and std error for R and Q MARSSparamCIs() gave the wrong s.e. for variances and covariances when method="hessian". It also gave the wrong CIs for variances and covariances when var-cov matrix was non-diagonal. There were a series of issues related to back-transforming from the hessian of a chol-transformed var-cov matrix ( Sigma=chol%*%t(chol) ). * MARSSparamCIs(), vrs 3.9 was getting the hessian numerically using a var-cov matrix that had been transformed with a Cholesky decomposition to ensure it stays positive-definite. The upper and lower CIs were computed from the s.e.'s. I back-transformed the Hessian to the original (non-chol) scale the same way I back transformed a var-covariance matrix. But the variance of s^2 is not the var(s)^2, which is what I was doing, essentially. So the s.e. for R and Q were wrong in all cases. Note, using a Hessian to estimate CIs for variance-covariance matrices is generally a bad idea anyhow however. * For non-diagonal matrices. There was a bug in MARSShessian() in subscripting the d matrix when doing the chol transform. Caused NAs for cases with non-diagonal matrices. However, had the se been returned, they would be wrong for non-diagonal matrices because the elements of the chol transformed matrices do not correspond one-to-one to the non-transformed matrices. E.g. the untransformed [2,2]^2 is chol transformed [1,2]^2+[2,2]^2. The hessian used was for the chol-transformation and the curvature of the LL surface for the chol-transformed values is different than the curvature for the untransformed var-cov elements.

Fix: I completely abandoned working with the chol transformed variance-covariance matrices for the Hessian calculation. The chol transformation was not necessary for computing the Hessian since the Hessian is computed at the MLEs and localized.
* Created a new function MARSSharveyobsFI() which uses the Harvey (1989) recursion to analyticallycompute the observed Fisher Information matrix. This is the Hessian for the untransformed var-cov parameters. So CIs on variances can be negative since the variance of the MLE is being approximated by a MVN (which can lead to negative lower CIs). * Harvey1989 is now the default function when method='hessian'. * The user can also select method='hessian' and hessian.fun='fdHess' or hessian.fun='optim'. This will compute the Hessian (of the log-LL function at the MLEs) numerically using these functions. The variance-covariance matrices are NOT chol transformed. These are numerically estimated Hessians of the untransformed variance-covariance matrices. * Added MARSSinfo(26) which discusses the reason for NAs in the Hessian.

Misc minor bugs * MARSShatyt() was setting ytt1 (expected value of y(t) conditioned on data up to t-1) to y(t), which is incorrect. Expected value of y(t) conditioned on y up to t-1 is Z xtt1 + a * CSEGriskfigure() panel 2 was wrong when mu>0 (increasing population). * CSEGriskfigure() panel 2 CIs were wrong when CI method=hessian since Q had not been back transformed (so was using sqrt(Q)). * MARSSkemcheck()'s test that fixed B is in unit circle failed when B was time-varying and some fixed and others estimated. Also when the eigenvalues were complex, it should test the real part only. * coef.marssMLE() was not stopping when illegal "what: arg passed in. man page did not say what happens when type=parameter. * passing in method not in allowed.methods was causing errors since model conversion and testing happening before checkMARSSinputs and model testing is algorithm dependent (some forms not allowed by BFGS). Added check for method at top of MARSS function.

IMPROVEMENTS * w(t) and v(t) can be specified as G(t)w(t) and H(t)v(t) where G and H are fixed matrices (not estimated). In version 3.10, G and H are restricted to being 0 or identity, however the code is in place for other values. - changes to MARSS.marxss and MARSS.marss to allow G, H, and L passed in - change to MARSSkem to specifiy star lists with G, H, and L (mathbb(elem) in EM Derivation) - changed MARSSkss to use Q=G Q t(G), R=H R t(H) and V0*=L V0 t(L) * added Multivariate linear regression chapter * added chapter on estimating a Leslie matrix from stage time series using a MARSS model. * removed the function MARSSmcinits and added chapter on searching over the initial conditions into the User Guide. As the MARSS models that MARSS() can fit expanded, MARSSmcinits was increasing obsolete and it was impossible to come up with good searching distributions. Because MARSSmcinits was removed, control$MCInits list item was removed also from defaults and from accepted input. * added default inits for c and d in marxss form so that user can pass in inits using coef(fit); was balking because this includes d and c which didn't have defaults. Removed msg referring to need that model be in marss form for inits (not true). * changed MARSS.marxss() to allow c and d to be 3D arrays. This allows one to use inits=fit to set inits and not get a d (or c) must be 2D error. * adding info to MARSSinfo(4) regarding errors about R=0 and x0 not fixed. Added info to the error warnings to direct user to MARSSinfo(). * changed order of MARSS args to be MARSS(y, model= , inits=, ...) * added pchol and psolve functions to return the chol or inverse via solve when there are 0s on the diagonal * added information to print.marssMODEL on summary.marssMODEL. Added silent argument to summary.marssMODEL to block printing to the console. * print.marssMLE(x, what="par") returned a vector of estimated values instead of the list of par. Changed to return the list. * added logLik method for marssMLE objects * added fitted method for marssMLE objects to return Z xtT + u (model fitted value of y) * added E[y(t), x(t+1)] to MARSShatyt output. Needed for residuals.marssMLE(). * added code to is.validvarcov() so that it returns an error if the user specifies a structurally illegal variance-covariance matrix. Added info to MARSSinfo(25).

MISC * moved info in MARSSsettings.R to .onLoad function. * added suppressWarnings() wrapper to KFAS call when R=0 in MARSSkfas since update to KFAS package produces warning messages when R=0. * Typo in Eqn 124 of EMDerivation.pdf. \beta should have been ^{-1}. Typo in Eqns 133 and 134. vec parentheses should have been in front of R in second summation. R in first line of eqn 133, was not referring to R (the var-cov matrix). It should have had a new symbol. Switched to T. Eqn 134 was not R but this 'T'. * added safe to control list in man file MARSS.Rd Left off accidentally. * Small change to DLM chapter to clarify that rotation matrix only exists if Z has more than 2 columns. * All subfunctions for a function moved into the main functions so they are hidden to the rest of the functions. * The logLik function was using the logLik, samp.size and df attributes from the MLE object, but this is prone to creating errors. The user may have changed the model structure or data in a MLE object and is trying to get the new logLik. Changed to recompute the logLik. * removed use of stringr package; only needed 1 or 2 functions * Poor name choice. y.se was not the standardard error of ytT since sqrt(OtT) not sqrt(OtT-ytT^2) was being returned. Changed name of y.se to ytT.se

- R
Published by eeholmes over 8 years ago

MARSS - Version 3.9

CHANGES IN MARSS 3.9

none. resubmission due to missing file

CHANGES IN MARSS 3.8

• added check for fun.kf value in checkMARSSinputs()
• added check to print.marssMLE to make sure models are class marssMODEL. Added info to MARSSinfo() to give the user some code to convert pre-3.5 marssMLE object to the 3.5+ form.
• changed summary.marssMODEL to return the list matrix instead of the marssMODEL passed in. Added tinitx to the returned (and printed) list.
• removed is.blockunconst and is.blockequaltri functions. Not really used or useful and were buggy.
• much of their code (assoc with identifying blocks) incorporated into a better is.validvarcov function to test for many more illegal constraints on a variance-covariance matrix. This will catch most but not all illegal constraints on Q, R and V0. It has a method argument, so method=BFGS can be passed in to check that all blocks are diagonal or unconstrained as needed by the chol transformation used in the MARSSoption() code to ensure varcov matrices stay postitive-definite.
• in MARSS().
  -- Switched to use MARSSkf() to return kf (so use what user requested), but set Innov, Sigma, Kt etc with MARSSkfss. -- Added row names to states.se and y.se.
• in MARSSkem(). Removed adding of kf and Ey when trace>0. This happens in MARSS().
• Changed summary.marssMODEL to use marssMODEL attributes for par.names and model.dims, so it works on non-marss form marssMODEL objects.
• added ability to handle time-varying var-cov matrices in MARSShessian()
• added check that Hessian CIs are only computed for models with diagonal var-cov matrices
• added ability to deal with NAs in Hessian in MARSShessian()
• The OR and CA years for the harborSeal dataset were off. Fixed and added references to the man file for harborSeal. Removed the harborSealnomiss dataset as that is no longer used in the User Guide.

• Rewrote the Seal Population Structure chapter and MAR(p) chapter.

• bugs

• in MARSSkfss. When Z was not square (num rows > num cols), OmgRVtt was not getting set. OmgRVtt sets Vtt diagonals (and corresponding cols and row) to zero when R had 0s on the diagonal.

• in MARSSkfas. Was returning $Innov and $Sigma using $v and $F, but as detailed in the KFS help page (KFAS package), the ones returned by KFS are not the same as the standard innovations and Sigma for multivariate data. Now, MARSSkfas() returns a text message to use MARSSkfss to get these.

• residuals.marssMLE & MARSSinnovationsboot that was not running MARSSkfss to get Innov, Kt, and Sigma when R was not diagonal. Problem occurred after I changed MARSSkfss() to return text error instead of NULL for these.

• bug introduced in 3.6 that printed no absol convergence when convergence=10. Should have printed abstol convergence only.

• bug in MARSSoptim (method=BFGS) that lead to only diagonal var-cov matrices when anything other than a diagonal var-cov matrix was selected.

• same bug affected attempt to compute CIs for non-diagonal var-cov matrices with Hessian.

• bug in MARSSoptim (method=BFGS) that allowed user to specify time-varying Q and R models, which code does not allow because cannot backsolve for par in that case.

• bug in MARSSoptim (method=BFGS) that allowed Q, R, and V0 structures that can't be handled by the chol transformation in that code. The transformation requires that Q, R, and V0 matrices be block unconstrained. Blocks can be identical or unique or some identical and others unique but each must be unconstrained. Note, in the context of a "block" matrix, a diagonal matrix is composed of n 1x1 blocks where n=nrows. Thus by definition, a diagonal matrix (with shared or unshared elements on the diagonal) is always block unconstrained. Dealt with with new is.validvarcov() function.

• bug in convert.model.mat() when user used names like "2" or "1" and had fixed values of the same (e.g. 1,2). This is because, inexplicably, R considers 1=="1" to be TRUE (and 2=="2", etc). Replaced with sapply and identical() embedded within.

• There is a check in MARSSkfss() that any 0s on the diagonal of Vtt1 have a corresponding 0 on the diagonal of Q. Was this line: Q0s=identical(which(diag.Q==0),which(diag.Vtt1==0)). But that forced a more stringent requirement, that all 0s on diag of Q were identical to 0s on diag of vtt1 rather than that 0s on diag of Vtt1 had 0 on diag on Q, but not the converse. Changed to Q0s=all(which(diag.Vtt1==0)%in%which(diag.Q==0)) so that requirement is one-way.

• X names were not getting applied to states in MARSS(); default X.names would be odd for non-design Z matrices. MARSSmarss() and MARSSmarxss().

- R
Published by eeholmes about 9 years ago