| Title: | Fit Hidden Markov Models using Template Model Builder |
|---|---|
| Description: | Fitting hidden Markov models using automatic differentiation and Laplace approximation, allowing for fast inference and flexible covariate effects (including random effects and smoothing splines) on model parameters. The package is described by Michelot (2025) <doi:10.18637/jss.v114.i05>. |
| Authors: | Theo Michelot [aut, cre], Richard Glennie [aut], Jan-Ole Fischer [ctb] (SPDE smooths), Duncan Cameron-Steinke [ctb] (Forecast class), Pauline Rocu [ctb] |
| Maintainer: | Theo Michelot <[email protected]> |
| License: | GPL-3 |
| Version: | 1.1.2 |
| Built: | 2026-11-09 21:53:48 UTC |
| Source: | https://github.com/theomichelot/hmmtmb |
Read formula with as.character without splitting
as_character_formula(x, ...)as_character_formula(x, ...)
x |
R formula |
... |
Unused |
Citation: this function was taken from the R package formula.tools: Christopher Brown (2018). formula.tools: Programmatic Utilities for Manipulating Formulas, Expressions, Calls, Assignments and Other R Objects. R package version 1.7.1. https://CRAN.R-project.org/package=formula.tools
Transforms matrix to dgTMatrix
as_sparse(x)as_sparse(x)
x |
Matrix or vector. If this is a vector, it is formatted into a single-column matrix. |
Sparse matrix of class dgTMatrix
This version of bdiag checks whether the matrices passed as arguments are NULL. This avoids errors that would arise if using bdiag directly.
bdiag_check(...)bdiag_check(...)
... |
Matrix or list of matrices (only the first argument is used) |
Block diagonal matrix
Check values in vector are contiguous
check_contiguous(x)check_contiguous(x)
x |
Vector of values (can be numeric, character, factor) |
Logical: are values contiguous?
Currently, this raises an error message if tensor product smooths are used (te, ti, t2).
check_smooths(form)check_smooths(form)
form |
Model formula |
Grid of covariates
cov_grid(var, data = NULL, obj = NULL, covs = NULL, formulas, n_grid = 1000)cov_grid(var, data = NULL, obj = NULL, covs = NULL, formulas, n_grid = 1000)
var |
Name of variable |
data |
Data frame containing the covariates. If not provided, data are extracted from obj |
obj |
HMM model object containing data and formulas |
covs |
Optional named list for values of covariates (other than 'var') that should be used in the plot (or dataframe with single row). If this is not specified, the mean value is used for numeric variables, and the first level for factor variables. |
formulas |
List of formulas used in the model |
n_grid |
Grid size (number of points). Default: 1000. |
Data frame of covariates, with 'var' defined over a grid, and other covariates fixed to their mean (numeric) or first level (factor).
The two-dimensional counterpart of cov_grid, used by
HMM$plot_2d(). Both var and var2 are varied over a
regular n_grid by n_grid lattice spanning their ranges in the
data, and every other covariate is held at one value, as in
cov_grid: the one given in covs, otherwise the mean of
a numeric covariate or the first level of a factor.
cov_grid_2d( var, var2, data = NULL, obj = NULL, covs = NULL, formulas, n_grid = 40, padding = 0 )cov_grid_2d( var, var2, data = NULL, obj = NULL, covs = NULL, formulas, n_grid = 40, padding = 0 )
var |
Name of the first covariate, on the x axis |
var2 |
Name of the second covariate, on the y axis |
data |
Data frame of covariate values |
obj |
HMM model object, used to find the covariates if |
covs |
Optional named list of values for the other covariates |
formulas |
List of model formulas |
n_grid |
Number of points along each axis, so the grid has
|
padding |
Numeric vector of length 1 or 2 giving the amount to pad the 'var' and 'var2' ranges beyond the data. A single value is recycled to both axes, and a length-2 vector is interpreted as (padding in var, padding in var2). Defaults to 0. |
The lattice is regular because that is what ggplot2::geom_raster()
needs, and var varies fastest, so that the result can be reshaped
into a matrix column by column.
Data frame with n_grid^2 rows and one column for each
covariate
all.vars() on a model formula returns every symbol in it, including
objects a smooth refers to through its xt argument – the mesh of an
SPDE field, say – which are not covariates and are not columns of the data.
Those have to be dropped before the data are indexed by name.
cov_names_in_data(var_names, data)cov_names_in_data(var_names, data)
var_names |
Character vector of names found in the formulas |
data |
Data frame |
A covariate that is genuinely missing from the data is dropped here too,
rather than raising an error. mgcv::gam() reports it when the design
matrices are built, which is a better place to say so.
The subset of var_names naming columns of data
Contains the probability density/mass function, and the link and inverse link functions for a probability distribution.
Dist$new()Create a Dist object
Dist$new( name, pdf, rng, cdf = NULL, link, invlink, npar, parnames, parapprox = NULL, fixed = NULL, name_long = name, par_alt = NULL )
nameName of distribution
pdfProbability density/mass function of the distribution
(e.g. dnorm for normal distribution).
rngRandom generator function of the distribution (e.g.
rnorm for normal distribution).
cdfCumulative distribution function of the distribution
(e.g., pnorm for normal distribution). This is used to compute
pseudo-residuals.
linkNamed list of link functions for distribution parameters
invlinkNamed list of inverse link functions for distribution parameters
nparNumber of parameters of the distribution
parnamesCharacter vector with name of each parameter
parapproxFunction that takes a sample and produces approximate values for the unknown parameters
fixedVector with element for each parameter which is TRUE if parameter is fixed
name_longLong version of the name of the distribution, possibly more user-readable than name.
par_altFunction that takes a vector of the distribution parameters as input and returns them in a different format. Only relevant for some distributions (e.g., MVN, where the SDs and correlations can be reformatted into a covariance matrix)
A new Dist object
Dist$name()Return name of Dist object
Dist$name()
Dist$pdf()Return pdf of Dist object
Dist$pdf()
Dist$cdf()Return cdf of Dist object
Dist$cdf()
Dist$rng()Return random generator function of Dist object
Dist$rng()
Dist$link()Return link function of Dist object
Dist$link()
Dist$invlink()Return inverse link function of Dist object
Dist$invlink()
Dist$npar()Return number of parameters of Dist object
Dist$npar()
Dist$parnames()Return names of parameters
Dist$parnames()
Dist$parapprox()Return function that approximates parameters
Dist$parapprox()
Dist$fixed()Return which parameters are fixed
Dist$fixed()
Dist$code()Return code of Dist object
Dist$code()
Dist$name_long()Human-readable name of Dist object
Dist$name_long()
Dist$set_npar()Set number of parameters this distribution has
Dist$set_npar(new_npar)
new_nparNumber of parameters
Dist$set_parnames()Set parameter names
Dist$set_parnames(new_parnames)
new_parnamesParameter names
Dist$set_code()Set distribution code
Dist$set_code(new_code)
new_codeDistribution code
Dist$pdf_apply()Evaluate probability density/mass function
This method is used in the Dist$obs_probs() method. It is a wrapper around Dist$pdf(), which prepares the parameters and passes them to the function.
Dist$pdf_apply(x, par, log = FALSE)
xValue at which the function should be evaluated
parVector of parameters. The entries should be named if they are not in the same order as expected by the R function. (E.g. shape/scale rather than shape/rate for gamma distribution.)
logLogical. If TRUE, the log-density is returned. Default: FALSE.
Probability density/mass function evaluated at x for parameters par
Dist$rng_apply()Random number generator
This method is a wrapper around Dist$rng(), which prepares the parameters and passes them to the function.
Dist$rng_apply(n, par)
nNumber of realisations to generate
parVector of parameters. The entries should be named if they are not in the same order as expected by the R function. (E.g. shape/scale rather than shape/rate for gamma distribution.)
Vector of n realisations of this distribution
Dist$par_alt()Alternative parameter formatting
Dist$par_alt(par)
parVector of distribution parameters
Formatted parameters
Dist$n2w()Natural to working parameter transformation
This method transforms parameters from the natural scale (i.e., their domain of definition) to the "working" or "linear predictor" scale (i.e., the real line). It is a wrapper for Dist$link().
Dist$n2w(par)
parList of parameters
Vector of parameters on the working scale
Dist$w2n()Working to natural parameter transformation
This method transforms parameters from the "working" or "linear predictor" scale (i.e., the real line) to the natural scale (i.e., their domain of definition). It is a wrapper for Dist$invlink().
Dist$w2n(wpar, as_matrix = FALSE)
wparVector of working parameters
as_matrixLogical. If TRUE, the natural parameters are returned as a matrix with one row for each state and one column for each parameter. If FALSE, the natural parameters are returned as a list (default).
List or matrix of parameters on natural scale
Dist$clone()The objects of this class are cloneable with this method.
Dist$clone(deep = FALSE)
deepWhether to make a deep clone.
Density function of von Mises distribution
dvm(x, mu, kappa, log = FALSE)dvm(x, mu, kappa, log = FALSE)
x |
Angle |
mu |
Mean parameter |
kappa |
Concentration parameter |
log |
Should log-density be returned? |
Von Mises density
Density function of wrapped Cauchy distribution
dwrpcauchy(x, mu, rho, log = FALSE)dwrpcauchy(x, mu, rho, log = FALSE)
x |
Angle |
mu |
Mean parameter |
rho |
Concentration parameter |
log |
Should log-density be returned? |
Wrapped Cauchy density
This function is used to identify the variables "x" which are included as s(x, bs = "re") in the formula, in particular to check that they are factors.
find_re(form)find_re(form)
form |
Model formula |
Vector of names of variables for which a random effect term is included in the model.
Creates a forecast object from a fitted HMM model, projecting future state probabilities and unconditional observation densities for a user-defined horizon using a covariate data-frame if applicable. Forecasting occurs in the constructor; results are accessible via public methods.
Forecast$new()Construct a forecast object.
Forecast$new( hmm = NULL, n = NULL, forecast_data = NULL, preset_eval_range = NULL, starting_state_distribution = "last" )
hmmA fitted HMM object.
nForecast horizon: a whole number greater than zero. Ignored if forecast_data is provided.
forecast_datadata.frame of future covariates / IDs. Must include all covariates in fitted hmm.
preset_eval_rangeNamed list: elements are numeric vectors (univariate) or matrices (multivariate, rows as dimension ex. mu1, mu2, columns as points). For any observation variable without an entry, a default grid of 100 equally spaced points is used, spanning the range of the training data for that variable with each endpoint expanded outward by 10% of its magnitude. A warning is issued whenever this default is applied, as the resulting grid may not suit the distribution's support.
starting_state_distributionDistribution over states at the first forecast step. Either a numeric vector of length nstates() that is non-negative and sums to 1, or one of "last" (state probabilities at the final training row, propagated one step) or "stationary" (the model's stationary distribution).
Algorithm overview
Validate the arguments.
Build or accept the future design matrix (forecast_data).
Obtain time-varying transition matrices (tpm_forecast) and
observation parameters (obs_par_forecast) via HMM$predict().
Forward-propagate the hidden-state distribution
(hidden_state_forecast).
Marginalise over states to obtain unconditional pdfs on the
evaluation grid (forecast_dists).
A Forecast object. Results are read back with the accessor
methods below, each invoked with ():
Matrix of nstates rows by n forecast steps.
Array of time-varying observation parameters.
Array of time-varying transition matrices.
List of unconditional pdf matrices, one per observation variable.
List of grids on which each pdf was evaluated.
The data.frame of future covariates used for prediction.
The evaluation grid can be changed after construction with
update_eval_range(), which recomputes forecast_dists.
data(faithful)
# Create training and forecast data frames
training_fraction <- 0.8
n_training <- floor(nrow(faithful) * training_fraction)
training_df <- data.frame(
ID = rep(1, n_training),
waiting = faithful$waiting[1:n_training],
eruptions = faithful$eruptions[1:n_training]
)
forecast_df <- data.frame(
ID = rep(1, nrow(faithful) - n_training),
waiting = faithful$waiting[(n_training + 1):nrow(faithful)],
eruptions = faithful$eruptions[(n_training + 1):nrow(faithful)]
)
# Create 2-state model with non-linear effect of waiting on all transitions
hid_model <- MarkovChain$new(
data = training_df,
n_states = 2,
formula = ~ s(waiting, k = 10, bs = "cs")
)
# Create observation model with normal distribution for duration
obs_model <- Observation$new(
data = training_df,
n_states = 2,
dists = list(eruptions = "norm"),
par = list(eruptions = list(
mean = c(0, 0),
sd = c(1, 1)
))
)
# Update model parameters to suggested
obs_model$update_par(par = obs_model$suggest_initial())
# Create HMM model
hmm <- HMM$new(
hid = hid_model,
obs = obs_model
)
hmm$fit(silent = TRUE)
# Create forecast object. Setting the evaluation grid explicitly is
# recommended; omitting it falls back to a default grid and warns.
forecast <- Forecast$new(
hmm = hmm,
forecast_data = forecast_df,
preset_eval_range = list(eruptions = seq(1, 6, length.out = 100)),
starting_state_distribution = "last"
)
# Unconditional predictive densities: grid points x forecast steps
dim(forecast$forecast_dists()$eruptions)
Forecast$obs_par_forecast()Get predicted observation parameters.
Forecast$obs_par_forecast()
Forecast$tpm_forecast()Get predicted transition matrices.
Forecast$tpm_forecast()
Forecast$eval_range()Get the evaluation grid for each response variable.
Forecast$eval_range()
Forecast$forecast_data()Get the forecast data used for predictions.
Forecast$forecast_data()
Forecast$update_eval_range()Update the evaluation grid for forecast pdfs. This method reconfigures the evaluation range used to compute unconditional predictive pdfs. It validates the new eval_range, updates internal state, reinitializes forecast densities, and returns the object invisibly for chaining.
Forecast$update_eval_range(eval_range)
eval_rangeA named list specifying the new evaluation grid for each observation variable.
The Forecast object (invisibly), allowing method chaining.
Forecast$hidden_state_forecast()Get the forward state probabilities.
Forecast$hidden_state_forecast()
Forecast$forecast_dists()Get the unconditional predictive pdfs.
Forecast$forecast_dists()
Forecast$quantile()Quantiles of forecast distribution
Forecast$quantile(var, probs)
varName of observation variable, as character string
probsNumeric vector of probabilities
Data frame with one column for each quantile probability and one row for each time step in the forecast
Forecast$mean()Mean of forecast distribution
Forecast$mean(var)
varName of observation variable, as character string
Numeric vector of means, with one element for each time step in the forecast
Forecast$plot()Plot forecast
Forecast$plot(var, show_data = FALSE)
varName of observation variable, as character string
show_dataLogical: should data be included in the plot?
The plot shows the mean of the forecast distribution over time (red line), as well as bands of decreasing opacity for the 50 90
A ggplot object
Forecast$clone()The objects of this class are cloneable with this method.
Forecast$clone(deep = FALSE)
deepWhether to make a deep clone.
HMM for the fitted model that is forecast from,
and MarkovChain and Observation for the
hidden state and observation components it is built from.
## ------------------------------------------------ ## Method `Forecast$new()` ## ------------------------------------------------ data(faithful) # Create training and forecast data frames training_fraction <- 0.8 n_training <- floor(nrow(faithful) * training_fraction) training_df <- data.frame( ID = rep(1, n_training), waiting = faithful$waiting[1:n_training], eruptions = faithful$eruptions[1:n_training] ) forecast_df <- data.frame( ID = rep(1, nrow(faithful) - n_training), waiting = faithful$waiting[(n_training + 1):nrow(faithful)], eruptions = faithful$eruptions[(n_training + 1):nrow(faithful)] ) # Create 2-state model with non-linear effect of waiting on all transitions hid_model <- MarkovChain$new( data = training_df, n_states = 2, formula = ~ s(waiting, k = 10, bs = "cs") ) # Create observation model with normal distribution for duration obs_model <- Observation$new( data = training_df, n_states = 2, dists = list(eruptions = "norm"), par = list(eruptions = list( mean = c(0, 0), sd = c(1, 1) )) ) # Update model parameters to suggested obs_model$update_par(par = obs_model$suggest_initial()) # Create HMM model hmm <- HMM$new( hid = hid_model, obs = obs_model ) hmm$fit(silent = TRUE) # Create forecast object. Setting the evaluation grid explicitly is # recommended; omitting it falls back to a default grid and warns. forecast <- Forecast$new( hmm = hmm, forecast_data = forecast_df, preset_eval_range = list(eruptions = seq(1, 6, length.out = 100)), starting_state_distribution = "last" ) # Unconditional predictive densities: grid points x forecast steps dim(forecast$forecast_dists()$eruptions)## ------------------------------------------------ ## Method `Forecast$new()` ## ------------------------------------------------ data(faithful) # Create training and forecast data frames training_fraction <- 0.8 n_training <- floor(nrow(faithful) * training_fraction) training_df <- data.frame( ID = rep(1, n_training), waiting = faithful$waiting[1:n_training], eruptions = faithful$eruptions[1:n_training] ) forecast_df <- data.frame( ID = rep(1, nrow(faithful) - n_training), waiting = faithful$waiting[(n_training + 1):nrow(faithful)], eruptions = faithful$eruptions[(n_training + 1):nrow(faithful)] ) # Create 2-state model with non-linear effect of waiting on all transitions hid_model <- MarkovChain$new( data = training_df, n_states = 2, formula = ~ s(waiting, k = 10, bs = "cs") ) # Create observation model with normal distribution for duration obs_model <- Observation$new( data = training_df, n_states = 2, dists = list(eruptions = "norm"), par = list(eruptions = list( mean = c(0, 0), sd = c(1, 1) )) ) # Update model parameters to suggested obs_model$update_par(par = obs_model$suggest_initial()) # Create HMM model hmm <- HMM$new( hid = hid_model, obs = obs_model ) hmm$fit(silent = TRUE) # Create forecast object. Setting the evaluation grid explicitly is # recommended; omitting it falls back to a default grid and warns. forecast <- Forecast$new( hmm = hmm, forecast_data = forecast_df, preset_eval_range = list(eruptions = seq(1, 6, length.out = 100)), starting_state_distribution = "last" ) # Unconditional predictive densities: grid points x forecast steps dim(forecast$forecast_dists()$eruptions)
mgcv::predict.gam() with type = "lpmatrix" uses only the model
frame, the terms objects, the factor levels and contrasts, and the smooth
objects – all of which mgcv::gam(fit = FALSE) already produces. This
wraps that unfitted setup in something predict.gam() accepts, so that
a prediction matrix can be built without fitting anything.
gam_shell(G)gam_shell(G)
G |
Output of |
The coefficients are set to zero. predict.gam() requires the slot to
exist and to have the right length, but for type = "lpmatrix" it
returns the design matrix itself and never multiplies by them.
An object of class "gam", only usable for
predict(type = "lpmatrix")
Generalized determinant = product of non-zero eigenvalues (see e.g., Wood 2017). Used for (log)determinant of penalty matrices, required in log-likelihood function.
gdeterminant(x, eps = 1e-10, log = TRUE)gdeterminant(x, eps = 1e-10, log = TRUE)
x |
Numeric matrix |
eps |
Threshold below which eigenvalues are ignored (default: 1e-10) |
log |
Logical: should the log-determinant be returned? |
Generalized determinant of input matrix
Encapsulates the observation and hidden state models for a hidden Markov model.
HMM$new()Create new HMM object
HMM$new( obs = NULL, hid = NULL, file = NULL, init = NULL, fixpar = NULL, bw = NULL )
obsObservation object, created with Observation$new().
This contains the formulation for the observation model.
hidMarkovChain object, created with MarkovChain$new().
This contains the formulation for the state process model.
filePath to specification file for HMM. If this argument is
used, then obs and hid are unnecessary.
initHMM object, used to initialise the parameters for this model.
If init is passed, then all parameters that are included in init
and in the present model are copied. This may be useful when fitting
increasingly complex models: start from a simple model, then pass it as
init to create a more complex model, and so on.
fixparNamed list, with optional elements: 'hid', 'obs', 'delta0',
'lambda_obs', and 'lambda_hid'. Each element is a named vector of
parameters in coeff_fe that should either be fixed (if the corresponding
element is set to NA) or estimated to a common value (using integers or
factor levels).don See examples in the vignettes, and check the TMB
documentation to understand the inner workings (argument map
of TMB::MakeADFun()).
bwBandwidth of the banded forward algorithm; see
HMM$update_bw(). Defaults to NULL, i.e. 15 if the model
contains a Gaussian field and 0 (the exact algorithm) otherwise.
A new HMM object
# Load data set (included with R)
data(nottem)
data <- data.frame(temp = as.vector(t(nottem)))
# Create hidden state and observation models
hid <- MarkovChain$new(data = data, n_states = 2)
par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5)))
obs <- Observation$new(data = data, n_states = 2,
dists = list(temp = "norm"),
par = par0)
# Create HMM
hmm <- HMM$new(hid = hid, obs = obs)
HMM$obs()Observation object for this model
HMM$obs()
HMM$hid()MarkovChain object for this model
HMM$hid()
HMM$out()Output of optimiser after model fitting
HMM$out()
HMM$tmb_obj()Model object created by TMB. This is the output of
the TMB function MakeADFun, and it is a list including elements
fnObjective function
grGradient function of fn
parVector of initial parameters on working scale
HMM$tmb_obj()
HMM$tmb_obj_joint()Model object created by TMB for the joint likelihood of
the fixed and random effects. This is the output of the TMB function
MakeADFun, and it is a list including elements
fnObjective function
grGradient function of fn
parVector of initial parameters on working scale
HMM$tmb_obj_joint()
HMM$tmb_rep()Output of the TMB function sdreport, which includes
estimates and standard errors for all model parameters.
HMM$tmb_rep()
HMM$states()Vector of estimated states, after viterbi has
been run
HMM$states()
HMM$coeff_fe()Coefficients for fixed effect parameters
HMM$coeff_fe()
HMM$coeff_re()Coefficients for random effect parameters
HMM$coeff_re()
HMM$coeff_list()List of all model coefficients
These are the parameters estimated by the model, including fixed and random effect parameters for the observation parameters and the transition probabilities, (transformed) initial probabilities, and smoothness parameters.
HMM$coeff_list()
HMM$fixpar()Fixed parameters
HMM$fixpar(all = FALSE)
allLogical. If FALSE, only user-specified fixed parameters are returned, but not parameters that are fixed by definition (e.g., size of binomial distribution).
HMM$coeff_array()Array of working parameters
HMM$coeff_array()
HMM$lambda()Smoothness parameters
HMM$lambda()
HMM$bw()Bandwidth of the forward algorithm (0 = exact)
HMM$bw()
HMM$update_bw()Update bandwidth of the forward algorithm
The exact forward algorithm accumulates log-likelihood contributions
that are each conditional on the entire process history, so the Hessian
with respect to latent variables spread over time is dense. The banded
algorithm of Fischer (2026) splits the time series into blocks of length
bw and truncates that conditioning at the previous block, which
makes the Hessian banded. It costs about twice as much per evaluation,
and its error decays geometrically in bw, because an ergodic
Markov chain forgets its initial condition exponentially fast.
A model containing a Gaussian field is banded by default, with
bw = 15, because the dense Hessian of the exact algorithm
defeats the sparsity such a field exists to provide. That default is a
starting point, not an answer: the bandwidth needed depends on how fast
the chain forgets its initial condition, so a persistent chain needs
more. Every other model is exact by default. Use
HMM$check_bw() to choose a bandwidth before fitting, and
afterwards check that raising it leaves the estimates unchanged.
HMM$update_bw(bw = NULL)
bwNon-negative integer (0 = exact algorithm), or NULL to
choose automatically. This is TMB data, so changing it discards any
existing setup, which is rebuilt on the next setup() or
fit().
HMM$check_bw()Profile the log-likelihood against the bandwidth
Evaluates the joint log-likelihood at the model's current parameter values for a range of bandwidths. A sensible bandwidth is the smallest one beyond which the value no longer changes appreciably. This is evaluated at fixed parameters, so it is cheap and does not require the model to be re-fitted.
HMM$check_bw(bws = seq(5, 40, by = 5), silent = TRUE)
bwsVector of bandwidths (each at least 2)
silentLogical. If TRUE (default), TMB tracing output is hidden.
Data frame with columns bw and llk, including a
row with bw = Inf for the exact forward algorithm.
HMM$update_par()Update parameters stored inside model object
HMM$update_par(par_list = NULL, iter = NULL)
par_listList with elements for coeff_fe_obs, coeff_fe_hid, coeff_re_obs, coeff_re_hid, log_delta0, log_lambda_hid, and log_lambda_obs
iterOptional argument to update model parameters based on MCMC iterations (if using rstan). Either the index of the iteration to use, or "mean" if the posterior mean should be used.
HMM$sd_re()Standard deviation of smooth terms (or random effects)
This function transforms the smoothness parameter of each smooth term into a standard deviation, given by SD = 1/sqrt(lambda). It is particularly helpful to get the standard deviations of independent normal random effects.
HMM$sd_re()
List of standard deviations for observation model and hidden state model.
HMM$par()Model parameters
HMM$par(t = 1)
treturns parameters at time t, default is t = 1
A list with elements:
obsparParameters of observation model
tpmTransition probability matrix of hidden state model
HMM$set_priors()Set priors for coefficients
HMM$set_priors(new_priors = NULL)
new_priorsis a named list of matrices with optional elements coeff_fe_obs, coeff_fe_hid, log_lambda_obs, andlog_lambda_hid. Each matrix has two columns (first col = mean, second col = sd) specifying parameters for normal priors.
HMM$priors()Extract stored priors
HMM$priors()
HMM$iters()Iterations from stan MCMC fit
HMM$iters(type = "response")
typeEither "response" for parameters on the response (natural) scale, or "raw" for parameters on the linear predictor scale.
see output of as.matrix in stan
HMM$out_stan()fitted stan object from MCMC fit
HMM$out_stan()
the stanfit object
HMM$llk()Log-likelihood at current parameters
HMM$llk()
Log-likelihood
HMM$edf()Effective degrees of freedom
HMM$edf()
Number of effective degrees of freedom (accounting for flexibility in non-parametric terms implied by smoothing)
HMM$suggest_initial()Suggest initial parameter values
Uses K-means clustering to split the data into naive "states", and estimates observation parameters within each of these states. This is meant to help with selecting initial parameter values before model fitting, but users should still think about the values carefully, and try multiple set of initial parameter values to ensure convergence to the global maximum of the likelihood function.
HMM$suggest_initial()
List of initial parameters
HMM$setup()TMB setup
This creates an attribute tmb_obj, which can be used to
evaluate the negative log-likelihood function.
HMM$setup(silent = TRUE)
silentLogical. If TRUE, all tracing outputs are hidden (default).
HMM$fit_stan()Fit model using tmbstan
Consult documentation of the tmbstan package for more information.
After this method has been called, the Stan output can be accessed
using the method out_stan(). This Stan output can for example
be visualised using functions from the rstan package. The parameters
stored in this HMM object are automatically updated to the mean
posterior estimate, although this can be changed using
update_par().
HMM$fit_stan(..., silent = FALSE)
...Arguments passed to tmbstan
silentLogical. If FALSE, all tracing outputs are shown (default).
HMM$fit()Model fitting
The negative log-likelihood of the model is minimised using the
function nlminb(). TMB uses the Laplace approximation to integrate
the random effects out of the likelihood.
After the model has been fitted, the output of nlminb() can be
accessed using the method out(). The estimated parameters can
be accessed using the methods par() (for the HMM parameters,
possibly dependent on covariates), predict() (for uncertainty
quantification and prediction of the HMM parameters for new covariate
values), coeff_fe() (for estimated fixed effect coefficients on
the linear predictor scale), and coeff_re() (for estimated random
effect coefficients on the linear predictor scale).
HMM$fit(silent = FALSE, ...)
silentLogical. If FALSE, all tracing outputs are shown (default).
...Other arguments to nlminb which is used to optimise the
likelihood. This currently only supports the additional argument
control, which is a list of control parameters such as
eval.max and iter.max (see ?nlminb)
# Load data set (included with R)
data(nottem)
data <- data.frame(temp = as.vector(t(nottem)))
# Create hidden state and observation models
hid <- MarkovChain$new(data = data, n_states = 2)
par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5)))
obs <- Observation$new(data = data, n_states = 2,
dists = list(temp = "norm"),
par = par0)
# Create HMM
hmm <- HMM$new(hid = hid, obs = obs)
# Fit HMM
hmm$fit(silent = TRUE)
HMM$mle()Get maximum likelihood estimates once model fitted
HMM$mle()
list of maximum likelihood estimates as described as input for the function update_par()
HMM$forward_backward()Forward-backward algorithm
The forward probability for time step t and state j
is the joint pdf/pmf of observations up to time t and of being in
state j at time t, p(Z[1], Z[2], ..., Z[t], S[t] = j).
The backward probability for time t and state j is the
conditional pdf/pmf of observations between time t + 1 and n,
given state j at time t, p(Z[t+1], Z[t+2], ..., Z[n] | S[t] = j).
This function returns their logarithm, for use in other methods
state_probs, and sample_states.
HMM$forward_backward()
Log-forward and log-backward probabilities
HMM$pseudores()Pseudo-residuals
Compute pseudo-residuals for the fitted model. If the fitted model is the "true" model, the pseudo-residuals follow a standard normal distribution. Deviations from normality suggest lack of fit.
HMM$pseudores()
List (of length the number of variables), where each element is a vector of pseudo-residuals (of length the number of data points)
HMM$viterbi()Viterbi algorithm
HMM$viterbi()
Most likely state sequence
HMM$sample_states()Sample posterior state sequences using forward-filtering backward-sampling
The forward-filtering backward-sampling algorithm returns a sequence of states, similarly to the Viterbi algorithm, but it generates it from the posterior distribution of state sequences, i.e., accounting for uncertainty in the state classification. Multiple generated sequences will therefore generally not be the same.
HMM$sample_states(nsamp = 1, full = FALSE)
nsampNumber of samples to produce
fullIf TRUE and model fit by fit_stan then parameter estimates are sampled from the posterior samples before simulating each sequence
Matrix where each column is a different sample of state sequences, and each row is a time of observation
HMM$state_probs()Compute posterior probability of being in each state
HMM$state_probs()
matrix with a row for each observation and a column for each state
HMM$post_coeff()Posterior sampling for model coefficients
HMM$post_coeff(n_post)
n_postNumber of posterior samples
Matrix with one column for each coefficient and one row for each posterior draw
HMM$post_linpred()Posterior sampling for linear predictor
HMM$post_linpred(n_post)
n_postNumber of posterior samples
List with elements obs and hid, where each is a matrix with one column for each predictor and one row for each posterior draw
HMM$post_fn()Create posterior simulations of a function of a model component
HMM$post_fn(fn, n_post, comp = NULL, ..., level = 0, return_post = FALSE)
fnFunction which takes a vector of linear predictors as input and produces either a scalar or vector output
n_postNumber of posterior simulations
compEither "obs" for observation model linear predictor, or "hid" for hidden model linear predictor
...Arguments passed to fn
levelConfidence interval level if required (e.g., 0.95 for 95 confidence intervals). Default is 0, i.e., confidence intervals are not returned.
return_postLogical indicating whether to return the posterior samples. If FALSE (default), only mean estimates and confidence intervals are returned
A list with elements:
If return_post = TRUE, this is a vector (for scalar outputs of fn) or matrix (for vector outputs) with a column for each simulation
Mean over posterior samples
Lower confidence interval bound (if level !=0)
Upper confidence interval bound (if level !=0)
HMM$predict()Predict estimates from a fitted model
By default, this returns point estimates of the HMM parameters for a new data frame of covariates. See the argument 'n_post' to also get confidence intervals.
HMM$predict( what, t = 1, newdata = NULL, n_post = 0, level = 0.95, return_post = FALSE, as_list = TRUE )
whatWhich estimates to predict? Options include transition probability matrices "tpm", stationary distributions "delta", or observation distribution parameters "obspar"
tTime points to predict at
newdataNew dataframe to use for prediction
n_postIf greater than zero then n_post posterior samples are produced, and used to create confidence intervals.
levelLevel of the confidence intervals, e.g. CI = 0.95 will produce 95% confidence intervals (default)
return_postLogical. If TRUE, a list of posterior samples is returned.
as_listLogical. If confidence intervals are required for the
transition probabilities or observation parameters, this
argument determines whether the MLE, lower confidence limit and upper
confidence limit are returned as separate elements in a list (if
TRUE; default), or whether they are combined into a single array (if
FALSE). Ignored if what = "delta" or if n_post = 0.
...Other arguments to the respective functions for hid$tpm, hid$delta, obs$par
Maximum likelihood estimates (mle) of predictions,
and confidence limits (lcl and ucl) if requested. The
format of the output depends on whether confidence intervals are
required (specified through n_post), and on the argument
as_list.
# Load data set (included with R)
data(nottem)
data <- data.frame(temp = as.vector(t(nottem)))
# Create hidden state and observation models
hid <- MarkovChain$new(data = data, n_states = 2)
par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5)))
obs <- Observation$new(data = data, n_states = 2,
dists = list(temp = "norm"),
par = par0)
# Create HMM
hmm <- HMM$new(hid = hid, obs = obs)
# Fit HMM
hmm$fit(silent = TRUE)
# Get transition probability matrix with confidence intervals
hmm$predict(what = "tpm", n_post = 1000)
HMM$confint()Confidence intervals for working parameters
This function computes standard errors for all fixed effect model parameters based on the diagonal of the inverse of the Hessian matrix, and then derives Wald-type confidence intervals.
HMM$confint(level = 0.95)
levelLevel of confidence intervals. Defaults to 0.95, i.e., 95% confidence intervals.
List of matrices with three columns: mle (maximum likelihood estimate), lcl (lower confidence limit), ucl (upper confidence limit), and se (standard error). One such matrix is produced for the working parameters of the observation model, the working parameters of the hidden state model, the smoothness parameters of the observation model, and the smoothness parameters of the hidden state model.
HMM$simulate()Simulate from hidden Markov model
HMM$simulate(n, data = NULL, silent = FALSE)
nNumber of time steps to simulate
dataOptional data frame including covariates
silentif TRUE then no messages are printed
Data frame including columns of data (if provided), and simulated data variables
HMM$check()Compute goodness-of-fit statistics using simulation
Many time series are simulated from the fitted model, and the statistic(s) of interest are calculated for each. A histogram of those values can for example be used to compare to the observed value of the statistic. An observation far in the tails of the distribution of simulated statistics suggests lack of fit.
HMM$check(check_fn, nsims = 100, full = FALSE, silent = FALSE)
check_fnGoodness-of-fit function which accepts "data" as input and returns a statistic (either a vector or a single number) to be compared between observed data and simulations.
nsimsNumber of simulations to perform
fullIf model fitted with 'fit_stan', then full = TRUE will sample from posterior for each simulation
silentLogical. If FALSE, simulation progress is shown. (Default: TRUE)
List with elements:
Vector of values of goodness-of-fit statistics for the observed data
Matrix of values of goodness-of-fit statistics for the simulated data sets (one row for each statistic, and one column for each simulation)
ggplot object
HMM$plot_ts()Time series plot coloured by states
Creates a plot of the data coloured by the most likely state sequence, as estimated by the Viterbi algorithm. If one variable name is passed as input, it is plotted against time. If two variables are passed, they are plotted against each other.
HMM$plot_ts(var, var2 = NULL, line = TRUE)
varName of the variable to plot.
var2Optional name of a second variable, for 2-d plot.
lineLogical. If TRUE (default), lines are drawn between successive data points. Can be set to FALSE if another geom is needed (e.g., geom_point).
A ggplot object
HMM$plot_dist()Plot observation distributions weighted by frequency in Viterbi
This is a wrapper around Observation$plot_dist, where the distribution for each state is weighted by the proportion of time spent in that state (according to the Viterbi state sequence).
HMM$plot_dist(var = NULL)
varName of data variable. If NULL, a list of plots are returned (one for each observation variable)
Plot of distributions with data histogram
HMM$plot()Plot a model component
HMM$plot( what, var = NULL, covs = NULL, i = NULL, j = NULL, n_grid = 50, n_post = 1000 )
whatName of model component to plot: should be one of "tpm" (transition probabilities), "delta" (stationary state probabilities), or "obspar" (state-dependent observation parameters)
varName of covariate to plot on x-axis
covsOptional named list for values of covariates (other than 'var') that should be used in the plot (or dataframe with single row). If this is not specified, the mean value is used for numeric variables, and the first level for factor variables.
iIf plotting tpm then rows of tpm; if plotting delta then indices of states to plot; if plotting obspar then full names of parameters to plot (e.g., obsvar.mean)
jIf plotting tpm then columnss of tpm to plot; if plotting delta then this is ignored,; if plotting obspar then indices of states to plot
n_gridNumber of points in grid over x-axis (default: 50)
n_postNumber of posterior simulations to use when computing
confidence intervals; default: 1000. See predict function for
more detail.
A ggplot object
HMM$plot_2d()Plot a model component over two covariates
The two-dimensional counterpart of HMM$plot(), for a model
containing a bivariate smooth such as s(x, y) or a Gaussian
field. HMM$plot() varies one covariate and holds the rest fixed,
which for a bivariate term shows a single slice through the surface;
this varies both and draws the surface itself.
Nothing here is specific to any basis. The surface is built from
HMM$predict() on a lattice of covariate values, so it works for
any smoother hmmTMB accepts on two covariates – a thin plate spline,
bs = "gp", an SPDE field – and for two covariates that enter the
model separately, where it shows their combined effect.
HMM$plot_2d( what, var, var2, covs = NULL, i = NULL, j = NULL, n_grid = 50, n_post = 0, level = 0.95, show = "mle", too_far = 0.1, contour = TRUE, padding = 0 )
whatName of model component to plot: one of "tpm" (transition probabilities), "delta" (stationary state probabilities), or "obspar" (state-dependent observation parameters)
varName of covariate on the x-axis
var2Name of covariate on the y-axis
covsOptional named list for values of covariates other than
var and var2. If not specified, the mean is used for
numeric covariates and the first level for factors.
iIf plotting tpm then rows of tpm; if plotting delta then indices of states; if plotting obspar then full names of parameters
jIf plotting tpm then columns of tpm; if plotting delta then ignored; if plotting obspar then indices of states
n_gridNumber of points along each axis, so the surface is
evaluated at n_grid^2 points (default: 50). At the default
n_post = 0 this costs one prediction and is cheap even for a
fine grid; with posterior simulation the cost grows with
n_grid^2, so it is worth lowering there.
n_postNumber of posterior simulations used for the confidence
interval. Defaults to 0, i.e. the maximum likelihood surface only,
because a surface needs many more evaluation points than a curve does.
Required to be positive for show = "ci".
levelConfidence level, when n_post > 0 (default: 0.95)
showWhat the fill shows: "mle" for the estimate itself, or "ci" for the width of the confidence interval, which says where the surface is well determined and where it is not.
too_farCells whose distance from the nearest observation exceeds
this, as a proportion of the plot's diagonal, are left blank. This is
mgcv::exclude.too.far(), and it matters for a bivariate smooth:
the corners of the lattice are usually extrapolation. Set to 0 to show
the whole rectangle (default: 0.1).
contourAdd contour lines over the surface (default: TRUE)
paddingNumeric vector of length 1 or 2 giving the amount to pad the 'var' and 'var2' ranges beyond the data. A single value is recycled to both axes, and a length-2 vector is interpreted as (padding in var, padding in var2). Defaults to 0.
A ggplot object
HMM$AIC_marginal()Marginal Akaike Information Criterion
The marginal AIC is for example defined by Wood (2017), as AIC = - 2L + 2k where L is the maximum marginal log-likelihood (of fixed effects), and k is the number of degrees of freedom of the fixed effect component of the model
HMM$AIC_marginal()
Marginal AIC
HMM$AIC_conditional()Conditional Akaike Information Criterion
The conditional AIC is for example defined by Wood (2017), as AIC = - 2L + 2k where L is the maximum joint log-likelihood (of fixed and random effects), and k is the number of effective degrees of freedom of the model (accounting for flexibility in non-parametric terms implied by smoothing)
HMM$AIC_conditional()
Conditional AIC
HMM$print_obspar()Print observation parameters at t = 1
HMM$print_obspar()
HMM$print_tpm()Print observation parameters at t = 1
HMM$print_tpm()
HMM$formulation()Print model formulation
HMM$formulation()
HMM$print()Print HMM object
HMM$print()
HMM$clone()The objects of this class are cloneable with this method.
HMM$clone(deep = FALSE)
deepWhether to make a deep clone.
## ------------------------------------------------ ## Method `HMM$new()` ## ------------------------------------------------ # Load data set (included with R) data(nottem) data <- data.frame(temp = as.vector(t(nottem))) # Create hidden state and observation models hid <- MarkovChain$new(data = data, n_states = 2) par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5))) obs <- Observation$new(data = data, n_states = 2, dists = list(temp = "norm"), par = par0) # Create HMM hmm <- HMM$new(hid = hid, obs = obs) ## ------------------------------------------------ ## Method `HMM$fit()` ## ------------------------------------------------ # Load data set (included with R) data(nottem) data <- data.frame(temp = as.vector(t(nottem))) # Create hidden state and observation models hid <- MarkovChain$new(data = data, n_states = 2) par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5))) obs <- Observation$new(data = data, n_states = 2, dists = list(temp = "norm"), par = par0) # Create HMM hmm <- HMM$new(hid = hid, obs = obs) # Fit HMM hmm$fit(silent = TRUE) ## ------------------------------------------------ ## Method `HMM$predict()` ## ------------------------------------------------ # Load data set (included with R) data(nottem) data <- data.frame(temp = as.vector(t(nottem))) # Create hidden state and observation models hid <- MarkovChain$new(data = data, n_states = 2) par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5))) obs <- Observation$new(data = data, n_states = 2, dists = list(temp = "norm"), par = par0) # Create HMM hmm <- HMM$new(hid = hid, obs = obs) # Fit HMM hmm$fit(silent = TRUE) # Get transition probability matrix with confidence intervals hmm$predict(what = "tpm", n_post = 1000)## ------------------------------------------------ ## Method `HMM$new()` ## ------------------------------------------------ # Load data set (included with R) data(nottem) data <- data.frame(temp = as.vector(t(nottem))) # Create hidden state and observation models hid <- MarkovChain$new(data = data, n_states = 2) par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5))) obs <- Observation$new(data = data, n_states = 2, dists = list(temp = "norm"), par = par0) # Create HMM hmm <- HMM$new(hid = hid, obs = obs) ## ------------------------------------------------ ## Method `HMM$fit()` ## ------------------------------------------------ # Load data set (included with R) data(nottem) data <- data.frame(temp = as.vector(t(nottem))) # Create hidden state and observation models hid <- MarkovChain$new(data = data, n_states = 2) par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5))) obs <- Observation$new(data = data, n_states = 2, dists = list(temp = "norm"), par = par0) # Create HMM hmm <- HMM$new(hid = hid, obs = obs) # Fit HMM hmm$fit(silent = TRUE) ## ------------------------------------------------ ## Method `HMM$predict()` ## ------------------------------------------------ # Load data set (included with R) data(nottem) data <- data.frame(temp = as.vector(t(nottem))) # Create hidden state and observation models hid <- MarkovChain$new(data = data, n_states = 2) par0 <- list(temp = list(mean = c(40, 60), sd = c(5, 5))) obs <- Observation$new(data = data, n_states = 2, dists = list(temp = "norm"), par = par0) # Create HMM hmm <- HMM$new(hid = hid, obs = obs) # Fit HMM hmm$fit(silent = TRUE) # Get transition probability matrix with confidence intervals hmm$predict(what = "tpm", n_post = 1000)
Multivarite inverse logit function
invmlogit(x)invmlogit(x)
x |
Numeric vector |
Check if number of whole number
is_whole_number(x, tol = 1e-10)is_whole_number(x, tol = 1e-10)
x |
number to check or vector of numbers |
tol |
how far away from whole number is ok? |
TRUE if it is a whole number within tolerance
This function makes it possible to call generic R methods such as AIC and BIC on HMM objects. It is based on the number of degrees of freedom of the *conditional* AIC (rather than marginal AIC), i.e., including degrees of freedom from the smooth/random effect components of the model.
## S3 method for class 'HMM' logLik(object, ...)## S3 method for class 'HMM' logLik(object, ...)
object |
HMM model object |
... |
For compatibility with S3 method |
Maximum log-likelihood value for the model, with attributes
df (degrees of freedom) and nobs (number of observations)
Log of sum of exponentials
logsumexp(x)logsumexp(x)
x |
Numeric vector |
Make covariance matrix from standard deviations and correlations
make_cov(sds, corr)make_cov(sds, corr)
sds |
Vector of standard deviations |
corr |
Vector of correlations (must be of length m*(m-1)/2 if sds is of length m) |
An m by m covariance matrix
Process formulas and store in nested list
make_formulas(input_forms, var_names, par_names, n_states)make_formulas(input_forms, var_names, par_names, n_states)
input_forms |
Nested list of formulas, with two levels: observed variable, and parameter of the observation distribution. The formulas can contain state-specific terms, e.g. "~ state1(x1) + x2". |
var_names |
character vector name of each observation variable |
par_names |
list with element for each observation variable that contains character vector of name of each parameter in its distribution |
n_states |
Number of states |
Formulas for the observation parameters can be different for the different states, using special functions of the form "state1", "state2", etc. This method processes the list of formulas passed by the user to extract the state-specific formulas. Missing formulas are assumed to be intercept-only ~1.
Nested list of formulas, with three levels: observed variable, parameter of the observation distribution, and state.
input_forms <- list(step = list(shape = ~ state1(x1) + x2, scale = ~ x1), count = list(lambda = ~ state1(x1) + state2(s(x2, bs = "cs")))) make_formulas(input_forms = input_forms, var_names = names(input_forms), par_names = lapply(input_forms, names), n_states = 2)input_forms <- list(step = list(shape = ~ state1(x1) + x2, scale = ~ x1), count = list(lambda = ~ state1(x1) + state2(s(x2, bs = "cs")))) make_formulas(input_forms = input_forms, var_names = names(input_forms), par_names = lapply(input_forms, names), n_states = 2)
Create model matrices
make_matrices(formulas, data, new_data = NULL, gam_args = NULL)make_matrices(formulas, data, new_data = NULL, gam_args = NULL)
formulas |
List of formulas (possibly nested, e.g. for use within Observation) |
data |
Data frame including covariates |
new_data |
Optional new data set, including covariates for which
the design matrices should be created. This needs to be passed in addition
to the argument ' |
gam_args |
Named list of additional arguments for |
Note the two indices: ncol_re, log_det_S and gmrf have
one entry per penalty, while L, sp_names,
sp_gmrf and theta_start have one per smoothing
parameter. The two coincide for an ordinary smooth, which has one of each,
but an SPDE field has three penalties and two parameters.
A list of
X_fe Design matrix for fixed effects
X_re Design matrix for random effects
S Smoothness matrix
log_det_S Vector of log-determinants of smoothness matrices, one for each penalty
ncol_fe Number of columns of X_fe for each parameter
ncol_re Number of columns of X_re and S for each random effect
L Matrix with log(lambda) = L * theta, mapping the smoothing
parameters of each smooth to the weights of its penalties (mgcv's
convention). It is the identity unless a smooth combines several
penalties through fewer parameters. See ?mgcv::smooth.construct
for details.
gmrf For each penalty, 1 if its log-determinant depends on the parameters and so has to be computed inside the likelihood, and 0 if the penalty is a fixed matrix scaled by exp(theta)
sp_names Name of each smoothing parameter
theta_start Starting value of each log smoothing parameter
Contains the parameters and model formulas for the hidden process model.
MarkovChain$new()Create new MarkovChain object
MarkovChain$new( data = NULL, formula = NULL, n_states, tpm = NULL, initial_state = "estimated", fixpar = NULL, ref = 1:n_states, gam_args = NULL )
dataData frame, needed to create model matrices, and to identify the number of time series (which each have a separate initial distribution)
formulaEither (1) R formula, used for all transition probabilities,
or (2) matrix of character strings giving the formula for each transition
probability, with "." along the diagonal (or for reference elements; see
ref argument). (Default: no covariate dependence.)
n_statesNumber of states. If not specified, then formula
needs to be provided as a matrix, and n_states is deduced from its dimensions.
tpmOptional transition probability matrix, to initialise the model parameters (intercepts in model with covariates). If not provided, the default is a matrix with 0.9 on the diagonal.
initial_stateSpecify model for initial state distribution. There are five different options:
"estimated": a separate initial distribution is estimated for each ID (default)
"stationary": the initial distribution is fixed to the stationary distribution of the transition probability matrix for the first time point of each ID
"shared": a common initial distribution is estimated for all IDs
integer value between 1 and n_states: used as the known initial state for all IDs
vector of integers between 1 and n_states (of length the number of IDs): each element is used as the known initial state for the corresponding ID
fixparList with optional elements "hid" (fixed parameters for transition probabilities), "lambda_hid" (fixed smoothness parameters), and "delta0" (fixed parameters for initial distribution). Each element is a named vector of coefficients that should either be fixed (if the corresponding element is set to NA) or estimated to a common value (using integers or factor levels).
refVector of indices for reference transition probabilities,
of length n_states. The i-th element is the index for the
reference in the i-th row of the transition probability matrix. For
example, ref = c(1, 1) means that the first element of the first row
Pr(1>1) and the first element of the second row Pr(2>1) are used as
reference elements and are not estimated. If this is not provided,
the diagonal transition probabilities are used as references.
gam_argsNamed list of arguments passed to mgcv::gam() in
MarkovChain$make_mat(), e.g., "knots". Use at your own risk.
A new MarkovChain object
# Load data set from MSwM package
data(energy, package = "MSwM")
# Create 2-state covariate-free model and initialise transition
# probability matrix
hid <- MarkovChain$new(data = energy, n_states = 2,
tpm = matrix(c(0.8, 0.3, 0.2, 0.7), 2, 2))
# Create 2-state model with non-linear effect of Oil on all transition
# probabilities
hid <- MarkovChain$new(data = energy, n_states = 2,
formula = ~ s(Oil, k = 5, bs = "cs"))
# Create 2-state model with quadratic effect of Oil on Pr(1 > 2)
structure <- matrix(c(".", "~poly(Oil, 2)",
"~1", "."),
ncol = 2, byrow = TRUE)
hid <- MarkovChain$new(data = energy, n_states = 2,
formula = structure)
MarkovChain$formula()Formula of MarkovChain model
MarkovChain$formula()
MarkovChain$formulas()List of formulas for MarkovChain model
MarkovChain$formulas()
MarkovChain$tpm()Get transition probability matrices
MarkovChain$tpm(t = 1, linpred = NULL)
tTime index or vector of time indices; default = 1. If t = "all" then all transition probability matrices are returned.
linpredOptional custom linear predictor
Array with one slice for each transition probability matrix
MarkovChain$ref()Indices of reference elements in transition probability matrix
MarkovChain$ref()
MarkovChain$ref_mat()Matrix of reference elements in transition probability matrix
MarkovChain$ref_mat()
MarkovChain$ref_delta0()Indices of reference elements in initial distribution
MarkovChain$ref_delta0()
MarkovChain$coeff_fe()Current parameter estimates (fixed effects)
MarkovChain$coeff_fe()
MarkovChain$delta()Stationary distribution
MarkovChain$delta(t = NULL, linpred = NULL)
tTime point(s) for which stationary distribution should be returned. If t = "all", all deltas are returned; else this should be a vector of time indices. If NULL (default), the stationary distribution for the first time step is returned.
linpredOptional custom linear predictor
Matrix of stationary distributions. Each row corresponds to a row of the design matrices, and each column corresponds to a state.
MarkovChain$delta0()Initial distribution
MarkovChain$delta0(log = FALSE, as_matrix = TRUE)
logLogical indicating whether to return the log of the initial probabilities (default: FALSE). If TRUE, then the last element is excluded, as it is not estimated.
as_matrixLogical indicating whether the output should be formatted as a matrix (default). If as_matrix is FALSE and log is TRUE, the result is formatted as a column vector.
Matrix with one row for each time series ID, and one column for each state. For each ID, the i-th element of the corresponding row is the probability Pr(S[1] = i)
MarkovChain$stationary()Use stationary distribution as initial distribution?
MarkovChain$stationary()
MarkovChain$fixpar()Fixed parameters
MarkovChain$fixpar(all = FALSE)
allLogical. If FALSE, only user-specified fixed parameters are returned, but not parameters that are fixed for some other reason (e.g., from '.' in formula)
MarkovChain$coeff_re()Current parameter estimates (random effects)
MarkovChain$coeff_re()
MarkovChain$X_fe()Fixed effect design matrix
MarkovChain$X_fe()
MarkovChain$X_re()Random effect design matrix
MarkovChain$X_re()
MarkovChain$lambda()Smoothness parameters
MarkovChain$lambda()
MarkovChain$sd_re()Standard deviation of smooth terms
This function transforms the smoothness parameter of
each smooth term into a standard deviation, given by
SD = 1/sqrt(lambda). It is particularly helpful to get the
standard deviations of independent normal random effects.
A smooth with its own parameterisation rather than a single smoothness
parameter – a Gaussian field, whose parameters are already a marginal
standard deviation and a range – has no such transformation and
returns NA; read those off lambda() directly.
MarkovChain$sd_re()
MarkovChain$nstates()Number of states
MarkovChain$nstates()
MarkovChain$terms()Terms of model formulas
MarkovChain$terms()
MarkovChain$unique_ID()Number of time series
MarkovChain$unique_ID()
MarkovChain$initial_state()Initial state (see constructor argument)
MarkovChain$initial_state()
MarkovChain$empty()Empty model? (for simulation only)
MarkovChain$empty()
MarkovChain$gam_args()Extra arguments for mgcv::gam (passed to make_matrices)
MarkovChain$gam_args()
MarkovChain$update_tpm()Update transition probability matrix
MarkovChain$update_tpm(tpm)
tpmNew transition probability matrix
MarkovChain$update_coeff_fe()Update coefficients for fixed effect parameters
MarkovChain$update_coeff_fe(coeff_fe)
coeff_feVector of coefficients for fixed effect parameters
MarkovChain$update_coeff_re()Update coefficients for random effect parameters
MarkovChain$update_coeff_re(coeff_re)
coeff_reVector of coefficients for random effect parameters
MarkovChain$update_X_fe()Update design matrix for fixed effects
MarkovChain$update_X_fe(X_fe)
X_fenew design matrix for fixed effects
MarkovChain$update_X_re()Update design matrix for random effects
MarkovChain$update_X_re(X_re)
X_renew design matrix for random effects
MarkovChain$update_delta0()Update initial distribution
MarkovChain$update_delta0(delta0)
delta0Either a matrix where the i-th row is the initial distribution for the i-th time series in the data, or a vector which is then used for all time series. Entries of each row of delta0 should sum to one.
MarkovChain$update_lambda()Update smoothness parameters
MarkovChain$update_lambda(lambda)
lambdaNew smoothness parameter vector
MarkovChain$update_fixpar()Update information about fixed parameters
MarkovChain$update_fixpar(fixpar)
fixparNew list of fixed parameters, in the same format expected by MarkovChain$new()
MarkovChain$make_mat()Make model matrices
MarkovChain$make_mat(data, new_data = NULL)
dataData frame containing all needed covariates
new_dataOptional new data set, including covariates for which
the design matrices should be created. This needs to be passed in addition
to the argument 'data', for cases where smooth terms or factor
covariates are included, and the original data set is needed to determine
the full range of covariate values.
A list with elements:
Design matrix for fixed effects
Design matrix for random effects
Smoothness matrix for random effects
Number of columns of X_fe for each parameter
Number of columns of X_re and S for each random effect
MarkovChain$make_mat_grid()Design matrices for grid of covariates
Used in plotting functions such as HMM$plot_tpm and HMM$plot_stat_dist
MarkovChain$make_mat_grid(var, data, covs = NULL, n_grid = 1000)
varName of variable
dataData frame containing the covariates
covsOptional named list for values of covariates (other than 'var') that should be used in the plot (or dataframe with single row). If this is not specified, the mean value is used for numeric variables, and the first level for factor variables.
n_gridGrid size (number of points). Default: 1000.
A list with the same elements as the output of make_mat, plus a data frame of covariates values.
MarkovChain$tpm2par()Transform transition probabilities to working scale
Apply the multinomial logit link function to get the corresponding parameters on the working scale (i.e., linear predictor scale).
MarkovChain$tpm2par(tpm)
tpmTransition probability matrix
Vector of parameters on linear predictor scale
MarkovChain$par2tpm()Transform working parameters to transition probabilities
Apply the inverse multinomial logit link function to transform the parameters on the working scale (i.e., linear predictor scale) into the transition probabilities.
MarkovChain$par2tpm(par)
parVector of parameters on working scale
Transition probability matrix
MarkovChain$linpred()Linear predictor for transition probabilities
MarkovChain$linpred()
MarkovChain$simulate()Simulate from Markov chain
MarkovChain$simulate(n, data = NULL, new_data = NULL, silent = FALSE)
nNumber of time steps to simulate
dataOptional data frame containing all needed covariates
new_dataOptional new data set, including covariates for which
the design matrices should be created. This needs to be passed in addition
to the argument 'data', for cases where smooth terms or factor
covariates are included, and the original data set is needed to determine
the full range of covariate values.
silentif TRUE then no messages are printed
Sequence of states of simulated chain
MarkovChain$formulation()Print model formulation
MarkovChain$formulation()
MarkovChain$print()Print MarkovChain object
MarkovChain$print()
MarkovChain$clone()The objects of this class are cloneable with this method.
MarkovChain$clone(deep = FALSE)
deepWhether to make a deep clone.
## ------------------------------------------------ ## Method `MarkovChain$new()` ## ------------------------------------------------ # Load data set from MSwM package data(energy, package = "MSwM") # Create 2-state covariate-free model and initialise transition # probability matrix hid <- MarkovChain$new(data = energy, n_states = 2, tpm = matrix(c(0.8, 0.3, 0.2, 0.7), 2, 2)) # Create 2-state model with non-linear effect of Oil on all transition # probabilities hid <- MarkovChain$new(data = energy, n_states = 2, formula = ~ s(Oil, k = 5, bs = "cs")) # Create 2-state model with quadratic effect of Oil on Pr(1 > 2) structure <- matrix(c(".", "~poly(Oil, 2)", "~1", "."), ncol = 2, byrow = TRUE) hid <- MarkovChain$new(data = energy, n_states = 2, formula = structure)## ------------------------------------------------ ## Method `MarkovChain$new()` ## ------------------------------------------------ # Load data set from MSwM package data(energy, package = "MSwM") # Create 2-state covariate-free model and initialise transition # probability matrix hid <- MarkovChain$new(data = energy, n_states = 2, tpm = matrix(c(0.8, 0.3, 0.2, 0.7), 2, 2)) # Create 2-state model with non-linear effect of Oil on all transition # probabilities hid <- MarkovChain$new(data = energy, n_states = 2, formula = ~ s(Oil, k = 5, bs = "cs")) # Create 2-state model with quadratic effect of Oil on Pr(1 > 2) structure <- matrix(c(".", "~poly(Oil, 2)", "~1", "."), ncol = 2, byrow = TRUE) hid <- MarkovChain$new(data = energy, n_states = 2, formula = structure)
Multivariate logit function
mlogit(x)mlogit(x)
x |
Numeric vector |
Multivariate Normal inverse link function
mvnorm_invlink(x)mvnorm_invlink(x)
x |
Vector of parameters on linear predictor scale (in the order: means, SDs, correlations) |
Multivariate Normal link function
mvnorm_link(x)mvnorm_link(x)
x |
Vector of parameters on natural scale (in the order: means, SDs, correlations) |
Replace NA entries in a vector by the last non-NA value. If the first entry of the vector is NA, it is replaced by the first non-NA value. If the vector passed as input doesn't contain NAs, it is returned as is.
na_fill(x)na_fill(x)
x |
Vector in which NAs should be removed |
Copy of x in which NAs have been replaced by nearest available value.
Contains the data, distributions, parameters, and formulas for the observation model from a hidden Markov model.
Observation$new()Create new Observation object
Observation$new( data = NULL, dists, formulas = NULL, n_states = NULL, par, fixpar = NULL, gam_args = NULL )
dataData frame containing response variables (named in dists and par) and covariates (named in formulas). Note that "state" is a reserved column name, used to pass known states to the model (semi- supervised learning).
distsNamed list of distribution names for each data stream, with the following options: beta, binom, cat, dir, exp, foldednorm, gamma, gamma2, lnorm, mvnorm, nbinom, norm, pois, t, truncnorm, tweedie, vm, weibull, wrpcauchy, zibinom, zigamma, zigamma2, zinbinom, zipois, zoibeta, ztnbinom, ztpois. See vignette about list of distributions for more details, e.g., list of parameters for each distribution.
formulasList of formulas for observation parameters. This should be a nested list, where the outer list has one element for each observed variable, and the inner lists have one element for each parameter. Any parameter that is not included is assumed to have the formula ~1. By default, all parameters have the formula ~1 (i.e., no covariate effects).
n_statesNumber of states (optional). If not provided, the number
of states is derived from the length of entries of par.
parList of initial observation parameters. This should
be a nested list, where the outer list has one element for each
observed variable, and the inner lists have one element for each
parameter. The choice of good initial values can be important, especially
for complex models; the package vignettes discuss approaches to selecting
them (e.g., see Observation$suggest_initial()).
fixparList with optional elements "obs" (fixed coefficients for observation parameters), and "lambda_obs" (fixed smoothness parameters), Each element is a named vector of coefficients that should either be fixed (if the corresponding element is set to NA) or estimated to a common value (using integers or factor levels).
gam_argsNamed list of arguments passed to mgcv::gam() in
Observation$make_mat(), e.g., "knots". Use at your own risk.
A new Observation object
# Load data set from MSwM package
data(energy, package = "MSwM")
# Initial observation parameters
par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2)))
# Model "energy" with normal distributions
obs <- Observation$new(data = energy,
dists = list(Price = "norm"),
par = par0)
# Model "energy" with gamma distributions
obs <- Observation$new(data = energy,
dists = list(Price = "gamma2"),
par = par0)
# Model with non-linear effect of EurDol on mean price
f <- list(Price = list(mean = ~ s(EurDol, k = 5, bs = "cs")))
obs <- Observation$new(data = energy,
dists = list(Price = "norm"),
par = par0,
formula = f)
Observation$data()Data frame
Observation$data()
Observation$dists()List of distributions
Observation$dists()
Observation$nstates()Number of states
Observation$nstates()
Observation$par()Parameters on natural scale
Observation$par(t = 1, full_names = TRUE, linpred = NULL, as_list = FALSE)
tTime index or vector of time indices; default t = 1. If t = "all", then return observation parameters for all time points.
full_namesLogical. If TRUE, the rows of the output are named in the format "variable.parameter" (default). If FALSE, the rows are names in the format "parameter". The latter is used in various internal functions, when the parameters need to be passed on to an R function.
linpredOptional custom linear predictor.
as_listLogical. If TRUE, the output is a nested list with three levels: (1) time step, (2) observed variable, (3) observation parameter. If FALSE (default), the output is an array with one row for each observation parameter, one column for each state, and one slice for each time step.
Array of parameters with one row for each observation parameter, one column for each state, and one slice for each time step. (See as_list argument for alternative output format.)
# Load data set from MSwM package
data(energy, package = "MSwM")
# Initial observation parameters
par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2)))
# Model with linear effect of EurDol on mean price
f <- list(Price = list(mean = ~ EurDol))
obs <- Observation$new(data = energy,
dists = list(Price = "norm"),
par = par0,
n_states = 2,
formula = f)
# Set slope coefficients
obs$update_coeff_fe(coeff_fe = c(3, 2, 6, -2, log(2), log(2)))
# Observation parameter values for given data rows
obs$par(t = c(1, 10, 20))
Observation$par_alt()Alternative parameter output
This function is only useful for the categorical and multivariate normal distributions, and it formats the parameters in a slightly nicer way.
Observation$par_alt(var = NULL, t = 1)
varName of observation variable for which parameters are required. By default, the first variable in 'dists' is used.
tTime index for covariate values. Only one value should be provided.
List of distribution parameters, with one element for each state
Observation$inipar()Return initial parameter values supplied
Observation$inipar()
Observation$coeff_fe()Fixed effect parameters on working scale
Observation$coeff_fe()
Observation$coeff_re()Random effect parameters
Observation$coeff_re()
Observation$X_fe()Fixed effect design matrix
Observation$X_fe()
Observation$X_re()Random effect design matrix
Observation$X_re()
Observation$lambda()Smoothness parameters
Observation$lambda()
Observation$sd_re()Standard deviation of smooth terms
This function transforms the smoothness parameter of
each smooth term into a standard deviation, given by
SD = 1/sqrt(lambda). It is particularly helpful to get the
standard deviations of independent normal random effects.
A smooth with its own parameterisation rather than a single smoothness
parameter – a Gaussian field, whose parameters are already a marginal
standard deviation and a range – has no such transformation and
returns NA; read those off lambda() directly.
Observation$sd_re()
Observation$formulas()List of model formulas for observation model
Observation$formulas(raw = FALSE)
rawLogical. If FALSE, returns the nested list created by make_formulas (default). If TRUE, returns formulas passed as input.
Observation$terms()Terms of model formulas
Observation$terms()
Observation$obs_var()Data frame of response variables
Observation$obs_var(expand = FALSE)
expandIf TRUE, then multivariate variables in observations are expanded to be univariate, creating extra columns.
Data frame of observation variables
Observation$known_states()Vector of known states
Observation$known_states(mat = TRUE)
matLogical.
Observation$fixpar()Fixed parameters
Observation$fixpar(all = FALSE)
allLogical. If FALSE, only user-specified fixed parameters are returned, but not parameters that are fixed for some other reason (e.g., size of binomial distribution)
Observation$empty()Empty model? (for simulation only)
Observation$empty()
Observation$gam_args()Extra arguments for mgcv::gam (passed to make_matrices)
Observation$gam_args()
Observation$update_par()Update parameters
Updates the 'par' attribute to the list passed as input, and updates the intercept elements of 'coeff_fe' using the list passed as input
Observation$update_par(par)
parNew list of parameters
Observation$update_coeff_fe()Update coefficients for fixed effect parameters
Observation$update_coeff_fe(coeff_fe)
coeff_feNew vector of coefficients for fixed effect parameters
Observation$update_coeff_re()Update random effect parameters
Observation$update_coeff_re(coeff_re)
coeff_reNew vector of coefficients for random effect parameters
Observation$update_X_fe()Update fixed effect design matrix
Observation$update_X_fe(X_fe)
X_feNew fixed effect design matrix
Observation$update_X_re()Update random effect design matrix
Observation$update_X_re(X_re)
X_reNew random effect design matrix
Observation$update_lambda()Update smoothness parameters
Observation$update_lambda(lambda)
lambdaNew smoothness parameter vector
Observation$update_data()Update data
Observation$update_data(data)
dataNew data frame
Observation$update_fixpar()Update information about fixed parameters
Observation$update_fixpar(fixpar)
fixparNew list of fixed parameters, in the same format expected by Observation$new()
Observation$make_mat()Make model matrices
Observation$make_mat(new_data = NULL)
new_dataOptional new data set, including covariates for which the design matrices should be created. If this argument is not specified, the design matrices are based on the original data frame.
A list with elements:
Design matrix for fixed effects
Design matrix for random effects
Smoothness matrix for random effects
Number of columns of X_fe for each parameter
Number of columns of X_re and S for each random effect
Observation$make_newdata_grid()Design matrices for grid of covariates
Observation$make_newdata_grid(var, covs = NULL, n_grid = 1000)
varName of variable
covsOptional named list for values of covariates (other than 'var') that should be used in the plot (or dataframe with single row). If this is not specified, the mean value is used for numeric variables, and the first level for factor variables.
n_gridGrid size (number of points). Default: 1000.
A list with the same elements as the output of make_mat, plus a data frame of covariates values.
Observation$n2w()Natural to working parameter transformation
This function applies the link functions of the distribution parameters, to transform parameters from their natural scale to the working scale (i.e., linear predictor scale)
Observation$n2w(par)
parList of parameters on natural scale
Vector of parameters on working scale
Observation$w2n()Working to natural parameter transformation
This function applies the inverse link functions of the distribution parameters, to transform parameters from the working scale (i.e., linear predictor scale) to their natural scale.
Observation$w2n(wpar)
wparVector of parameters on working scale
List of parameters on natural scale
Observation$linpred()Compute linear predictor
Observation$linpred()
Observation$obs_probs()Observation likelihoods
Observation$obs_probs(data = NULL)
dataOptional dataframe to include in form of obs_var() output
Matrix of likelihoods of observations, with one row for each time step, and one column for each state.
Observation$cdf()Cumulative probabilities of observations
Observation$cdf()
List of cumulative probabilities, with one element for each observed variable. Matrix rows correspond to time steps, and columns correspond to states.
Observation$suggest_initial()Suggest initial observation parameters
The K-means algorithm is used to define clusters of observations
(supposed to approximate the HMM states). Then, for each cluster,
the parapprox function of the relevant Dist object
is used to obtain parameter values.
Observation$suggest_initial()
List of initial parameters for each observation variable
# Load data set from MSwM package
data(energy, package = "MSwM")
# Initial observation parameters
par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2)))
# Model "energy" with normal distributions
obs <- Observation$new(data = energy,
dists = list(Price = "norm"),
par = par0,
n_states = 2)
# Print observation parameters
obs$par()
# Suggest initial parameters
par0_new <- obs$suggest_initial()
par0_new
# Update model parameters to suggested
obs$update_par(par = par0_new)
obs$par()
Observation$plot_dist()Plot histogram of data and pdfs
Plot histogram of observations for the variable specified by the argument name, overlaid with the pdf of the specified distribution for that data stream. Helpful to select initial parameter values for model fitting, or to visualise fitted state-dependent distributions.
Observation$plot_dist(var = NULL, weights = NULL, t = 1)
varName of response variable for which the histogram and pdfs should be plotted.
weightsOptional vector of length the number of pdfs that are plotted. Useful to visualise a mixture of distributions weighted by the proportion of time spent in the different states.
tIndex of time step to use for covariates (default: 1).
A ggplot object
Observation$formulation()Print model formulation
Observation$formulation()
Observation$print()Print Observation object Check constructor arguments
Observation$print()
Observation$clone()The objects of this class are cloneable with this method.
Observation$clone(deep = FALSE)
deepWhether to make a deep clone.
## ------------------------------------------------ ## Method `Observation$new()` ## ------------------------------------------------ # Load data set from MSwM package data(energy, package = "MSwM") # Initial observation parameters par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2))) # Model "energy" with normal distributions obs <- Observation$new(data = energy, dists = list(Price = "norm"), par = par0) # Model "energy" with gamma distributions obs <- Observation$new(data = energy, dists = list(Price = "gamma2"), par = par0) # Model with non-linear effect of EurDol on mean price f <- list(Price = list(mean = ~ s(EurDol, k = 5, bs = "cs"))) obs <- Observation$new(data = energy, dists = list(Price = "norm"), par = par0, formula = f) ## ------------------------------------------------ ## Method `Observation$par()` ## ------------------------------------------------ # Load data set from MSwM package data(energy, package = "MSwM") # Initial observation parameters par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2))) # Model with linear effect of EurDol on mean price f <- list(Price = list(mean = ~ EurDol)) obs <- Observation$new(data = energy, dists = list(Price = "norm"), par = par0, n_states = 2, formula = f) # Set slope coefficients obs$update_coeff_fe(coeff_fe = c(3, 2, 6, -2, log(2), log(2))) # Observation parameter values for given data rows obs$par(t = c(1, 10, 20)) ## ------------------------------------------------ ## Method `Observation$suggest_initial()` ## ------------------------------------------------ # Load data set from MSwM package data(energy, package = "MSwM") # Initial observation parameters par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2))) # Model "energy" with normal distributions obs <- Observation$new(data = energy, dists = list(Price = "norm"), par = par0, n_states = 2) # Print observation parameters obs$par() # Suggest initial parameters par0_new <- obs$suggest_initial() par0_new # Update model parameters to suggested obs$update_par(par = par0_new) obs$par()## ------------------------------------------------ ## Method `Observation$new()` ## ------------------------------------------------ # Load data set from MSwM package data(energy, package = "MSwM") # Initial observation parameters par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2))) # Model "energy" with normal distributions obs <- Observation$new(data = energy, dists = list(Price = "norm"), par = par0) # Model "energy" with gamma distributions obs <- Observation$new(data = energy, dists = list(Price = "gamma2"), par = par0) # Model with non-linear effect of EurDol on mean price f <- list(Price = list(mean = ~ s(EurDol, k = 5, bs = "cs"))) obs <- Observation$new(data = energy, dists = list(Price = "norm"), par = par0, formula = f) ## ------------------------------------------------ ## Method `Observation$par()` ## ------------------------------------------------ # Load data set from MSwM package data(energy, package = "MSwM") # Initial observation parameters par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2))) # Model with linear effect of EurDol on mean price f <- list(Price = list(mean = ~ EurDol)) obs <- Observation$new(data = energy, dists = list(Price = "norm"), par = par0, n_states = 2, formula = f) # Set slope coefficients obs$update_coeff_fe(coeff_fe = c(3, 2, 6, -2, log(2), log(2))) # Observation parameter values for given data rows obs$par(t = c(1, 10, 20)) ## ------------------------------------------------ ## Method `Observation$suggest_initial()` ## ------------------------------------------------ # Load data set from MSwM package data(energy, package = "MSwM") # Initial observation parameters par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 2))) # Model "energy" with normal distributions obs <- Observation$new(data = energy, dists = list(Price = "norm"), par = par0, n_states = 2) # Print observation parameters obs$par() # Suggest initial parameters par0_new <- obs$suggest_initial() par0_new # Update model parameters to suggested obs$update_par(par = par0_new) obs$par()
The covariance matrix is the inverse of the precision matrix. By default,
the function solve is used for inversion. If it fails (e.g.,
singular system), then MASS::ginv is used instead, and returns the
Moore-Penrose generalised inverse of the precision matrix.
prec_to_cov(prec_mat)prec_to_cov(prec_mat)
prec_mat |
Precision matrix (either of 'matrix' type or sparse matrix on which as.matrix can be used) |
Precision matrix
Solve for positive root of quadratic ax^2 + bx + c = 0 when it exists
quad_pos_solve(a, b, c)quad_pos_solve(a, b, c)
a |
coefficient of x^2 |
b |
coefficient of x |
c |
scalar coefficient |
real positive root if it exists
Draws from N(mu, Q^-1) through a sparse Cholesky factorisation of the precision, Q = P' L L' P, as x = mu + P' L^-T z with z standard normal. Nothing is inverted, and the Cholesky factor keeps the sparsity of Q, so the cost is that of the factorisation rather than of a dense inverse followed by a dense n-by-n Cholesky of it. A dense precision is simply a sparse matrix with no zeros, and costs no more this way than the old route did.
rmvn_prec(n, mu, prec_mat)rmvn_prec(n, mu, prec_mat)
n |
Number of samples |
mu |
Mean vector |
prec_mat |
Precision matrix, sparse or dense |
If prec_mat is not positive definite – a fit that has not converged,
or a singular Hessian – there is no Cholesky factor to be had, and this
falls back on prec_to_cov, which uses a generalised inverse
and warns.
Matrix with one row for each sample and one column for each element
of mu
Sample from von Mises distribution
rvm(n, mu, kappa)rvm(n, mu, kappa)
n |
Number of samples |
mu |
Mean parameter |
kappa |
Concentration parameter |
Uses basic rejection sampling, based on dvm(), which might be inefficient for large kappa. Could be improved following Best & Fisher (1979), Efficient simulation of the von Mises distribution, JRSSC, 28(2), 152-157.
Vector of n samples from vm(mu, kappa)
Sample from wrapped Cauchy distribution
rwrpcauchy(n, mu, rho)rwrpcauchy(n, mu, rho)
n |
Number of samples |
mu |
Mean parameter |
rho |
Concentration parameter |
Uses basic rejection sampling, based on dwrpcauchy(), which might be inefficient for large rho.
Vector of n samples from wrpcauchy(mu, rho)
A smooth term for a spatially (or temporally) continuous Gaussian random
field, usable as s(x, y, bs = "spde", xt = list(mesh = mesh)) in any
hmmTMB formula, and so on any transition probability or any parameter of any
observation distribution.
## S3 method for class 'spde.smooth.spec' smooth.construct(object, data, knots) ## S3 method for class 'spde.smooth' Predict.matrix(object, data)## S3 method for class 'spde.smooth.spec' smooth.construct(object, data, knots) ## S3 method for class 'spde.smooth' Predict.matrix(object, data)
object, data, knots
|
As |
A smoothCon object of class spde.smooth, with sparse
X and sparse penalties.
A one-dimensional field needs no mesh: s(x, bs = "spde") builds one
by default, as k quadratic B-spline basis functions on evenly spaced
knots, so
f <- ~ s(x, bs = "spde") # and s(x, bs = "spde", k = 40) for a finer one
is the whole of it. The knots are spread over the range of the covariate widened by a fifth on each side, which is the one-dimensional version of the outer extension of a two-dimensional mesh: with the knots stopping at the data, the boundary condition inflates the field's variance at the two ends by a third or more.
k is a resolution, not a smoothing parameter, and this is the one
place where a field does not behave like the smooths around it. The range is
estimated and is what does the smoothing, so k is not a cap on
wiggliness that the penalty then pulls back from: it only has to be large
enough to resolve the range the data ask for, knots about a third of a range
apart as in two dimensions. That is why the default is k = 15 rather
than mgcv's 10 – with the widening, 15 knots put the knot spacing at
about half the range the smoother starts from, which is a fifth of the
spread of the covariate.
Too small a k does not simply oversmooth. Once the knots are further
apart than the range the data want, the weights are near enough independent
that the likelihood cannot tell one short range from another, and the range
collapses towards zero while the standard deviation grows to compensate. The
fitted curve can still look right when that happens, because the basis is
then doing the smoothing on its own, but the two parameters are meaningless.
So read lambda() before reading the curve: a range far below the knot
spacing, or a standard deviation orders of magnitude above the scale of the
linear predictor, means raise k and refit.
Passing your own fmesher::fm_mesh_1d() overrides all of this, and is
worth doing when the covariate is unevenly spread and you want the knots
placed accordingly.
In two dimensions the mesh is yours to build and pass in, because a good one
depends on the domain, the data and the range you expect.
fmesher::fm_mesh_2d() takes locations, a maximum triangle edge length
inside the domain and in the outer extension, and a cutoff below which
nearby points are merged:
mesh <- fmesher::fm_mesh_2d(loc = cbind(data$x, data$y),
max.edge = c(0.1, 0.3),
cutoff = 0.05, offset = c(0.1, 0.3))
f <- ~ s(x, y, bs = "spde", xt = list(mesh = mesh))
A rule of thumb is max.edge no larger than a third of the range you
expect, and an offset about equal to the range; without the outer extension
the field's variance is inflated at the boundary.
The field is evaluated at every time step, so missing covariates are replaced by the last non-missing value as elsewhere in hmmTMB. If that is too crude for locations, interpolate them before fitting.
The precision is
with , , the finite element matrices from
fmesher::fm_fem(). alpha = 2 throughout, so the Matern
smoothness is : in two dimensions and
in one. The range is and
the marginal variance is
,
which is in two dimensions and
in one. Both constants depend on , so the
one- and two-dimensional smoothers do not share them.
The two estimated parameters are and , not
and , and appear as sd and
range in lambda(). The change is exact – a linear map of the
log parameters, absorbed into the constants in front of the three matrices
– and worth making: the parameters mean something on their own, and the
likelihood's long ridge along constant marginal variance runs diagonally in
but along an axis in , which the
optimiser finds much easier.
Both are estimated freely, with no prior. A single realisation identifies them only loosely, and a field whose range approaches the size of the domain is close to improper: the range then runs off and the standard deviation follows it, with the fitted surface barely changing. The fit is still usable if that happens, but the two parameters are not, and a finer mesh will not help – it is the data that are uninformative.
The field is proper for any positive , so no identifiability
constraint is imposed, as in INLA. It is not centred, and its level is only
weakly separated from the intercept when the range is large.
A model containing this smooth is fitted with the banded forward algorithm,
because the exact one makes the Hessian with respect to the field weights
dense and so defeats the sparsity the SPDE representation exists to provide.
See HMM$update_bw() and HMM$check_bw().
Lindgren, F., Rue, H. and Lindstrom, J. (2011). An explicit link between Gaussian fields and Gaussian Markov random fields. JRSS-B 73, 423-498.
Miller, D. L., Glennie, R., & Seaton, A. E. (2020). Understanding the stochastic partial differential equation approach to smoothing. Journal of Agricultural, Biological and Environmental Statistics, 25(1), 1-16.
Fischer, J.-O. (2026). Fast and scalable inference in hidden Markov models with Gaussian fields.
# A one-dimensional field over a covariate, with a default mesh if(requireNamespace("fmesher", quietly = TRUE)) { d <- data.frame(x = runif(100)) sm <- mgcv::smoothCon(mgcv::s(x, bs = "spde"), data = d)[[1]] c(k = ncol(sm$X), sm$theta.names) }# A one-dimensional field over a covariate, with a default mesh if(requireNamespace("fmesher", quietly = TRUE)) { d <- data.frame(x = runif(100)) sm <- mgcv::smoothCon(mgcv::s(x, bs = "spde"), data = d)[[1]] c(k = ncol(sm$X), sm$theta.names) }
The larger side of the bounding box of the mesh nodes that carry basis weight, which is the region the data occupy rather than the outer extension. A one-dimensional mesh stores its knots in 'loc' and the midpoints of its basis functions in 'mid', and only the latter are one per column of X, so 'mid' is the one to use where there is a choice.
spde_extent(mesh, X)spde_extent(mesh, X)
mesh |
An fmesher mesh |
X |
The smooth's design matrix |
A positive number
Observation locations for an SPDE smooth
spde_loc(term, data)spde_loc(term, data)
term |
Character vector of one or two covariate names |
data |
Data frame containing those covariates |
Numeric vector (1d) or two-column matrix (2d) of locations
k quadratic B-spline basis functions on evenly spaced knots, over the
range of the covariate widened by a fifth on each side. The widening is the
one-dimensional counterpart of the outer extension of a two-dimensional
mesh: fmesher's default boundary condition is Neumann, under which a field
whose knots stop at the data has its variance inflated by a third or more at
the two ends. A fifth of the spread of the covariate is about one range at
the starting value used in smooth.construct.spde.smooth.spec(), which
is the same rule of thumb used for the offset in two dimensions.
spde_mesh_1d(x, k)spde_mesh_1d(x, k)
x |
The covariate |
k |
Basis dimension, as passed to |
A degree-2 mesh has one basis function fewer than it has knots, so k
knots would not give k of them; k keeps its mgcv meaning of a
basis dimension here, and k + 1 knots are laid down.
The default of 15 is above mgcv's 10, because k is a resolution here
and not a cap on wiggliness: with the widening, 15 knots put the knot
spacing at about half the range the smoother starts from, in the spirit of
max.edge in two dimensions. Ten of them leave the knots further apart
than that range, and it collapses towards zero as a result.
An fm_mesh_1d with k basis functions
Strip comments marked with a hash from a character vector
strip_comments(str)strip_comments(str)
str |
the character vector |
character vector with comments removed (and lines with only comments completely removed)
Update a model to a new model by changing one formula
## S3 method for class 'HMM' update(object, type, i, j, change, fit = TRUE, silent = FALSE, ...)## S3 method for class 'HMM' update(object, type, i, j, change, fit = TRUE, silent = FALSE, ...)
object |
HMM model object |
type |
Character string for the part of the model that is updated (either "hid" or "obs") |
i |
If type = "hid" then i is the row of the formula containing the change. If type = "obs" then i is the observation variable name. |
j |
If type = "hid" then j is the column of the formula containing the change. If type = "obs" then j is the parameter whose formula is to be changed. |
change |
The change to make to the formula, see ?update.formula for details. |
fit |
If FALSE then change is made but model is not re-fit. |
silent |
If TRUE then no model fitting output is given |
... |
Additional arguments are ignored (for compatibility with generic S3 method) |
# Load data set from MSwM package data(energy, package = "MSwM") # Create hidden state and observation models hid <- MarkovChain$new(data = energy, n_states = 2) par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 3))) obs <- Observation$new(data = energy, n_states = 2, dists = list(Price = "norm"), par = par0) # Create HMM (no covariate effects) hmm <- HMM$new(hid = hid, obs = obs) hmm$hid()$formula() hmm$obs()$formulas() # Update transition probability formulas (one at a time) hmm <- update(hmm, type = "hid", i = 1, j = 2, change = ~ . + Oil, fit = FALSE) hmm <- update(hmm, type = "hid", i = 2, j = 1, change = ~ . + Gas + Coal, fit = FALSE) hmm$hid()$formula() # Update observation parameter formulas (one at a time) hmm <- update(hmm, type = "obs", i = "Price", j = "mean", change = ~ . + EurDol, fit = FALSE) hmm$obs()$formulas()# Load data set from MSwM package data(energy, package = "MSwM") # Create hidden state and observation models hid <- MarkovChain$new(data = energy, n_states = 2) par0 <- list(Price = list(mean = c(3, 6), sd = c(2, 3))) obs <- Observation$new(data = energy, n_states = 2, dists = list(Price = "norm"), par = par0) # Create HMM (no covariate effects) hmm <- HMM$new(hid = hid, obs = obs) hmm$hid()$formula() hmm$obs()$formulas() # Update transition probability formulas (one at a time) hmm <- update(hmm, type = "hid", i = 1, j = 2, change = ~ . + Oil, fit = FALSE) hmm <- update(hmm, type = "hid", i = 2, j = 1, change = ~ . + Gas + Coal, fit = FALSE) hmm$hid()$formula() # Update observation parameter formulas (one at a time) hmm <- update(hmm, type = "obs", i = "Price", j = "mean", change = ~ . + EurDol, fit = FALSE) hmm$obs()$formulas()