Version 0.0-1 --> 0.0-2 ======================= The changes are lost in the mists of time. Version 0.0-2 --> 0.0-3 ======================= The changes are lost in the mists of time. Version 0.0-3 --> 0.0-4 ======================= The changes may be recoverable, but haven't yet been recovered. Version 0.0-4 --> 0.0-5 ======================= The changes may be recoverable, but haven't yet been recovered. Version 0.0-5 --> 0.0-6 ======================= The changes may be recoverable, but haven't yet been recovered. Version 0.0-6 --> 0.0-7 ======================= The changes may be recoverable, but haven't yet been recovered. Version 0.0-7 --> 0.0-8 ======================= The changes may be recoverable, but haven't yet been recovered. Version 0.0-8 --> 0.0-9 ======================= 26 February 2009: Fixed bug in revise.tpm.R; if "mixture" was set to TRUE in hmm(), this function returned a one-column (!!!) matrix, not a square one. So the package was not (ever?) working with "mixture" set to be TRUE. Thanks to Mohammad Amin Jamalzadeh for pointing out this bug. Version 0.0-9 --> 0.1-0 ======================== 23 September 2009: Changed the input format of the y values. Now y should be either a single vector (of observations) or a list of such vectors. If there is more than one vector of observations, these should ***not*** be presented as the columns of a matrix. (This is still allowed, but is deprecated. Allowing the presentation of y as a matrix will be discontinued in some future release of this package.) Changed all functions that take observation sequences as an argument to expect multiple observation sequences to be presented as components of a list rather than as columns of a matrix. Thus multiple observation sequences are not constrained to be of the same length. Revised hmm() so that it does not insist that the underlying Markov chain be stationary at time 0. An argument ``stationary'' which defaults to TRUE (giving the old behaviour) has been introduced. The value of ``stationary'' is saved in the list returned by hmm(). Revised hmm() so that the observations y are (optionally; default TRUE) saved in the list returned by hmm(). Revised hmm() so that the set of values of the observations can be any finite set. Changed the treatment/usage of the yval argument of hmm() somewhat. This argument can be any superset of the unique values of the observations y. Any values in yval which are not matched by any actually observed values will correspond to rows of the estimate Rho all of whose entries are 0. Gave the object returned by hmm() a class --- hmm.discnp --- assigned, so that the fitted values may be calculated by a method for fitted(). Created a method for fitted(), fitted.hmm.discnp. Changed the treatment of the ``y'' argument of mps(); if this argument is missing, mps() attempts to extract it from the ``object'' argument. Removed ``yval'' from the argument lists of mps() and viterbi. Revised sim.hmm() to allow the initial state probability distribution, ispd, to be specified, rather than forcing it to be equal to the stationary distribution determined by the transition probability matrix. Added a ``yvals'' argument to the argument list of sim.hmm() so that the simulated observations can be from any specified set of cardinality nrow(Rho). I.e. these observations are no longer restricted to being in the sequence of integers from 1 to nrow(Rho). Changed the argument nsim to be a *vector*. Its length specifies the number of independent sequences generated; its entries specify the length of each sequence. Deleted the argument ``nrep''. Added a function pr() which calculates the probability of a state sequence s given an observation sequence y. Added a function sp() which returns the probabilities of each state, at each time point, given the model parameters and an observation sequence (or sequences). (Added at the request of Murat Ahmed.) Fixed a bug in viterbi() in respect of the way the values of the observation vector were matched up with the rows of Rho. Thanks to Graham Wood for pointing out this bug. Change tolerance below which probabilities are deemed to be zero from 10*.Machine$double.eps to the more orthodox sqrt(.Machine$double.eps). Version 0.1-0 --> 0.1-1 ======================= 30 November 2009: Bug fix: The function hmm() threw an error when the observations were numeric and contained 0 or did not conform to 1:max(observations) (and argument ``yvals'' was not specified). Fixed. (Thanks to Alon Ben Ari for pointing out this bug.) Bug fix: In revise.rho(), if there were observations missing from the sequence 1:max(observations), the estimate of Rho would turn into all NAs and the algorithm would converge instantly. (The consequence was that all of the f(y) values became NA and were consequently turned into 1 --- which is *supposed* to happen when y is NA, but not otherwise. Psigh!) Fixed. Changed the .First.lib() function to say ``PLEASE NOTE'' rather than ``WARNING''. Version 0.1-1 --> 0.1-2 ======================= 21 July 2010: Thanks to Thomas Lotze for the pointing out the problems requiring the following fixes: Bug fix: The length of the k-th replicate sequence (equal to lns(k)) was not passed to the afun fortran subroutine. Fixed. Bug fix: If "stationary" was TRUE and then ispd remained NULL, causing a fortran error. Now ispd is initially set to revise.ispd(tpm) irrespective of the value of "stationary". The help page for hmm() was not revised in respect of the way that the argument "y" should be supplied. It retained the old specification requiring y to be a matrix, with unequal length replicates adjusted for by padding with NAs. This has now been remedied. Version 0.1-2 --> 0.1-3 ======================= 23 July 2010 Fixed a glitch in the help page for hmm() in respect of the description of the ispd component of the returned value. Fixed another glitch --- multiple ``Warning'' sections. Fixed a glitch in the documentation of ``mps'' (relating to the matrix/list dichotomy). Made a change to pr() allowing "y" to be single observation sequence when "s" is a list of several state sequences; otherwise the length of the list "y" must (still) match that of the list "s". Changed the help file accordingly. Version 0.1-3 --> 0.1-4 ======================= 4 September 2011 Added a function logLikHmm() to calculate the log likelihood of a set of observations given a fixed set of parameters for the hidden Markov model. Thanks to Subhajit Dutta for the suggestion. Fixed a few minor typos in the documentation. Version 0.1-4 --> 0.1-5 ======================= 4 November 2011 Added a namespace. Version 0.1-5 --> 0.1-6 ======================= 9 February 2012 Fixed a bug which was drawn to my attention by Martin Ellis. I had omitted coercing "y" to character in ffun(). This coersion was there in version 0.1-3 but disappeared in version 0.1-4. This was effected to accommodate the use of the function logLikHmm() --- but it had repercussions that I didn't twig to. Have made adjustments so that now "everything" works. Also made an adjustment to check.yval() to allow for a mismatch in mode between the y-values and the row names of Rho. Version 0.1-6 --> 0.1-7 ======================= 9 February 2012 Fixed up some infelicities in this ChangeLog. (Note that version 0.1-6 was never installed on CRAN.) Version 0.1-7 --> 0.1-8 ======================= 2 March 2012 Changed the function sim.hmm() so that storage to hold the simulated values is completely pre-allocated ***as a vector***. For simulation of a long (e.g. length = 600000) single data sequence, this speeds up the simulation process dramatically. (E.g. from c. 5250 seconds of ellapsed time to c. 30 seconds of ellapsed time!) Thanks to Martin Ellis for drawing the problem of slowness of the simulation procedure to my attention, and indicating the solution of the problem. Version 0.1-8 --> 0.1-9 ======================= 4 April 2013 Fixed up the dimensioning of Fortran arrays so that they no longer use dummy 1's as the final dimension. 17 April 2013 Uploaded to CRAN Version 0.1-9 --> 0.2-0 ======================= 30 September 2013 Added the capability to use a different ispd for each (independent) sequence in the data set. Uploaded to CRAN Version 0.2-0 --> 0.2-1 Changed the way non-constant initial state probability distribution is handled by recurse(); now non-constancy is detected from the shape of ispd. The logical argument "cis" is not needed. Fixed the viterbi() function to avoid ridiculously small probability values and consequent underflow. Thanks to an inquiry from Owen Marshall. Either uses logarithms or normalizes "delta" to sum to 1 (the latter being the default procedure). Various other small infelicities fixed. Uploaded to CRAN 31 January 2014. Version 0.2-1 --> 0.2-2 Fixed typo (an omitted parenthesis) in fitted.hmm.discnp.Rd. (02/02/2014) Augmented the details in fitted.hmm.discnp.Rd to explain the fitted value concept. (02/02/2014) Revised code of fitted.hmm.discnp() to (a) make sure that "y" is a list and to check that all components of the list are numeric. (02/02/2014) Fixed up the description of "ispd" in logLikeHmm.Rd and in mps.Rd to cover the case where ispd is a matrix. (02/02/2014) Fixed bug which occurred when the observations contained missing values causing Rho[x,1:ncol(Rho)] to throw a "subscript out of bounds error. (18/05/2014) Revised code to make sure that Rho *always* has row names and that the observations *always* get treated as *character*. (18/05/2014) Added some example data sets: The Sydney coliform count data and the Albert et al. multiple sclerosis lesion count data. (18/05/2014) Changed mat2list() to charList() which does the conversion from matrix to list (and the whinge-producing) and coerces the y-values to character. (19/05/2014) Changed check.yval() to return an emission probability matrix Rho if the given Rho doesn't have row names. The returned Rho is equipped with appropriate row names if possible. Otherwise an error is thrown. (19/05/2014) Uploaded to CRAN 19/05/2014. Version 0.2-2 |--> 0.2-3 Made the word "missing" a reserved word (in the function charList()). (20/05/2014) Changed the way starting values for ispd are set when "stationary" is FALSE. Now set ispd[i] = 1/K for all i (irrespective of value of "cis") whereas previous set ispd[1] = 1 and the rest 0 when cis was FALSE and set ispd = stationary distribution when cis was TRUE. (4/08/2014) Adjusted the return mechanism so that when the number of observation sequences is 1, the value of "ispd" is a *vector* irrespective of the value of argument "cis". (4/08/2014) Uploaded to CRAN 05/08/2014. Version 0.2-3 |--> 0.2-4 Fixed bug in respect of the naming of the rows of "Rho" (the "emissions" probability matrix). Thanks to Harshita Patak, an enquiry from whom revealed the bug. (7/04/2016) Fixed up some other deficiencies in respect of the row and column names of tpm and Rho; made sure the values of Rho and tpm that are returned have appropriate row and column names and that the returned value of ispd is a named vector. (7/04/2016) Fixed several problems that occurred if the sorted row names of Rho were not the same as the original row names. (8/04/2016) Changed error trapping to allow the set of unique y values to be a proper subset of the row names of Rho. (8/04/2016) Uploaded to CRAN 08/04/2016 Version 0.2-4 |--> 0.3-0 Major renovation; introduced *bivariate* emissions. Distributed the actual work amongst three functions: hmmUV() (univariate), hmmBI() (bivariate independent), and hmmBD() (bivariate dependent). Initial stage of renovation completed 03/07/2016. Renovations continued; adjusted logLikHmm(), check.yval(), sp(), pr(), mps(), sim.hmm(), and viterbi(). Revised the help files. Finished this pass 13/07/2016. Changed revise.Rho() to make a correct estimate of Rho in the bivariate dependent variables setting. Uses optim(). Not yet tested. 20/07/2016 Checks out OK 22/07/2016 Removed the function sim.hmm() and replaced it by generic and methods --- rhmm(), rhmm.default() and rhmm.hmm.discnp() Wrote the appropriate help file. 13/10/2016. Adjusted the argument list of logLikHmm() (got rid of "par", and effectively replaced it by "model". Changed the name of argument "object" to "model" in mps() pr(), sp() and viterbi(). Adjusted help files accordingly. 13/10/2016. Built, checked and locally installed 13/10/2016. Adjusted logLikHmm so that it works if "y" is not supplied, but "model" is. (Simply by extracting "log.like" from "model".) 17/10/2016 Fixed a glitch in check.yval.R (used lapply(as.character,yval) in the type==1 setting). Made adjustment to logLikHmm.R --- changed "charList" to "tidyList", and applied mat2charmat. Same for pr.R, mps.R, sp.R, viterbi.R. Created function mat2charmat(); changed charList() to tidyList() (to keep numeric values as numeric). Adjusted the hmm*.R files; made "keep.y" the responsibility of hmm(); improved the consistency (between functions) of checking the validity of arguments; eliminated repetitive code by introduing for loops. (25/10/2016) Made various adjustements to the rhmm simulation functions; fixed error in rhmm.default() --- incorrect treatment of the dimensions of Rho; allowed for yval to be passed from rhmm.hmm.discnp to rhmm.default. Fixed up problem with using length() in settings where nrow() should have been used. Created functions() misstify() to insert missing values into simulated data and nafracCalc() to determine the fraction(s) of missing values in existing data sets. Added a "drop=" argument to the extraction of subarrays of Rho in ffun() to handle the K=1 case properly. (26/10/2016) Re-wrote the function revise.rho() to catch various numerical screw-ups in respect of getting infinite values in log(Rho) and "G". Fixed some glitches in hmmUV(). Version 0.3-0 never installed on CRAN Version 0.3-0 |--> 0.3.1 Added "npar" (the number of fitted parameters) to the object returned by hmm(), to facilitate doing an automated likelihood ratio test. Wrote an anova() method to effect the aforesaid likelihood ratio test. (01/11/2016) Messed about debugging; fixed a major screw up in revise.rho() that was causing anomalous *decreases* in the achieved log likelihood with increasing K. Still doesn't work right; getting crashes, with wll() returning -Inf. (05/11/2016) Version 0.3-1 never installed on CRAN Version 0.3-1 |--> 0.3-2 Major reworking of revise.rho() to overcome the -Inf problems. Finally started doing the calculations *right* thanks to a tip from Chuck Berry on R-help. Much messing around and debugging, getting revision to check with a known correct answer. Fixed another silly mistake in revise.rho(). Got everything to run with K=4. (08/11/2016) Added AIC to the list returned by hmm(). Adjusted the help file for hmm() accordingly. Added "cis" to the list returned by hmm() so as to accord with the help file. (08/11/2016) Adjusted names of a couple of utility functions in the environment of revise.rho(). Corrected slight bug in the returned value when K=1; especially in hmmUV(). Adjusted tolerance in the utility function r2zVfun() (in the environment of revise.rho()). Now using x > sqrt(.Machine$double.eps) rather than the simplistic x > 0. Done to prevent singularity of matrix in the set of equations to be solved for the zetas; seems to work, so far. (???) (09/11/2016) Version 0.3-2 never installed on CRAN Added BIC value to the list returned by hmm(). Changed the K=1 type=3 estimator to do the correct maximum likelihood estimator of Rho, rather than just mindlessly using the crosstabs estimator (i.e. allowing for (x_t,NA) and (NA,y_t) entries in the data. (10/11/2016) Fixed several glitches in the handling of the K=1 case in hmmBD(). Fixed a screw up involving summing over the wrong index in ffun(). Similar screw up fixed in ell(). Fixed problem with K=1 case in hmmBD(), hmmBI(), hmmUV(); needed a call to mat2charmat(). (11/11/2016) Added the argument "warn" to logLikHmm(), mps(), pr(), sp() and viterbi(). This argument is passed on the utility function check.yval() which actually issues the warning if warn=TRUE. (13/11/2016) Fixed a bug in hmmBI(), in the K=1 case. (Wrong in divisor used in forming the Rho[[i]] from the tabulation of the respective columns of the y-matrices.) Added a componet par0 (the starting values used by the EM algorithm) to the list returned by hmm(). (14/11/2016) Version 0.3-3 never installed on CRAN Version 0.3-3 |--> Version 1.0-0 Started working on allowing dependence of emission probabilities on auxiliary predictors. (10/03/2017) Added the foregoing capacity --- but only in the univariate setting. In the bivariate settings it is less obvious how to deal with modelling the dependence of the emission probabilities on the predictors. Using a multinomial model becomes much less intuitively appealing. Dealing with missing values of the emissions appears to be highly problematic. As indicated above, in the univariate case a multinomial model (using multinom() from the nnet package) is used to handle the dependence on the predictors. A different representation/parameterisation of Rho (as a data frame) is required to accommodate this dependence. If no predictors are supplied the previous matrix representation of Rho is still permitted. Did a lot of adjusting of hmm() code, eliminating redundancies and adding helper functions to do repeated tasks. Added the capacity to deal with predictors to functions that are called a posteriori: sp(), predict(), fitted(), logLikHmm(). Changed the default for keep.y from TRUE to FALSE. (21/04/2017) Added a "direct method", using nlm() (so far applicable only in the univariate setting). (04/05/2017, ..., 08/05/2017) Struggled with an anomalous phenomenon: Fitting the K=4 model to the monocytes data only (univariate) using the direct method ran (slowly! But it got there.) on my laptop. Tried it on the Nesi cluster; it crashed on step 73. Used the printed out parameters from step 72 to restart. It ran through to completion. Put in try() calls in an attempt to catch the parameters actually causing the problem. It ran to completion. Put in a different try() and a stop() call. It again crashed on step 73. Figured out what was causing the crash, and how to guard against it. Re-jigged the code with this guard, and set the function running again. We shall see .... (08/05/2017) It ran, but returned a "code" value of *3*. (09/05/2017) Made some adjustements to the help files, amongst other things making them consistent with the (new) possibility of using the "direct" maximisation method. (09/05/2017) More tweaking of help files. Fixed a glitch in revise.ispd(). (10--14/05/2017). Corrected the value of "npar" to allow for the non-stationary and non-cis settings. (15/05/2017). Modified hmmDirect() to return the covariance matrix of the parameter estimates if hessian=TRUE was set in the call. Added example data sets obtained from the "HYDAT" data base (and discretised to deciles). 04/01/2018 Modified hmmDirect() to use optim() as well as nlm(). Modified rhmm.default() to allow for Rho being a data frame. Added example data sets obtained from Prof. Christian Weiß (from his new book "An Introduction to Discrete-Valued Time Series"). 05/01/2018 Set version number to 1.0-0. 06/01/2018 Version 1.0-0 |--> 1.0-1 Added function scovmat() to estimate covariance matrix of parameter estimates via simulation. Made modifications to the simulation functions rhmm.default() and rhmm.hmm.discnp(), including adding the "inMiss" argument to the latter and the "nafrac", "p2" and "drop" arguments to the former. Changed the meaning of the "nsim" argument, introduced the "ylengths" argument to serve its old role; made appropriate modifications throughout. Changed name of "verb" argument to "verbose". Modified misstify() appropriately. Added "parity" for the data structure "y" returned by tidyList() and adjusted hmm() to make use of it. Changed stop message in respect of "newstyle" in hmm(). Modified the returned value of hmm() and of the hmm*() functions that it calls. This included adding "parity", "hessian" and "optimiser" to the list, where appropriate. Added function (method) update.hmm.discnp(); modified hmm() to return values needed by update() and by rhmm(), and to return values in a better organised way. Modified message produced in the functions called in "First.R". Wrote documentation for new functions and adjusted old documentation appropriately. 07/01/2018 Added function squantCI(). Incremented the version number. 08/01/2018 Version 1.0-1 |--> 2.0-0 Started on adding cappacity for using Levinberg-Marquardt algorithm, 08/02/2018. Finished (???) adding LM capacity to the package 27/03/2018. (*SUBSTANTIAL* modifications effected.) Tidied up and checked, 28/06/2018. Incremented the version number. 28/06/2018 Version 2.0-0 |--> 2.0-1 Fixed up the Sydney coliform count data; now including *only* the discretised version, and *not* the pseudo-Poisson count version. Put in the *correct* discretised version (having fixed a fuck-up in the hglmm "data preparation". Made some adjustments to nafracCalc.Rd. Incremented the version number. 29/06/2018 Version 2.0-1 |--> 2.0-2 In ffun(), in the type=4 setting, changed the way that calculations are done when the y value(s) is/are missing. Changed tidyList() to coerce the y-values into *character* mode. This fixed a serious bug whereby the dimnames of Rho were not properly matching (or not matching at all) the y-values when these were numeric. Made adjustment so the stopping criterion defaults to "PCLL" for the EM algorithm and to "ABSGRD" for the LM algorithm. Made adjustment so that tolerance defaults to 1e-4 for the EM algorithm and to sqrt(.Machine$double.eps) for the LM algorithm. Fixed things up so that "digits" gets set (properly) in the subsidiary hmm* functions (rather than in hmm()). Changed the "converged" component of the returned value to be *always* TRUE or FALSE (even when the "bf" method is used). Added an attribute to "converged", equal to the value of the convergence criterion returned by the optimiser, when the "bf" method is used. Adjusted the documentation of hmm() to refer appropriately to optim() as well as to nlm(). Made other adjustments to the documentation. Added a component "numeric" to the returned list, TRUE if the original data were numeric. (Basically for the use of the "fitted.hmm.discnp()" method. In the documentation of hmm() fixed up the description of the value returned. Added comments in tidyList() clarifying what is going on with the "numeric" and "character" modes. Incremented version number. 01/07/2018 Version 2.0-2 |--> 2.0-3 Fixed bug in hmmUV(); had not set "crit" for hmmLM() (when crit is NULL). 02/07/2018. Revised way crit is set (obviating the foregoing change) --- went back to the old way; also "PCLL" is the default used in all non-"bf" settings, including "LM". Revised the way tolerance is set --- went back to the old way; now 1e-4 is the default used in all circumstances. Fixed up the names of "theta" and the dimnames of the hessian which weren't getting set properly in hmmLM(). Also fixed up the names of grad and added grad to the returned value (for hmmLM() and for hmmNumOpt(); the latter being possible only when the optimiser is *not* optim(), which does not return the gradient --- only the hessian. Go figure.) Made the calculations of the stopping criteria in hmmBD(), hmmBI(), and hmmUV() (when EM is used) the same as those in hmmLM() --- i.e. making them all "relative", with tolerance added to the denominators. Added the final values of the stopping criteria ("stopCrit") to the returned value (except of course for the "bf" method, wherein these criteria are internal to nlm() or optim() and are obscure). Re-ordered the components of the returned lists a bit. Incremented the version number. 04/07/2018. Version 2.0-3 |--> 2.0-4 Revised the way an error in revise.ispd() (multiple eigenvalues of t(tpm) equal to 1) is treated. Added a new method "steepest descent" ("SD"), allowing this method to stand on its own rather than just being used as an escape hatch when "LM" screws up. Fixed up a number of anomalies in the documentation of hmm() (inappropriate references to "EM"). Added an "lmc" argument to hmm() --- to be passed on to hmmUV() and thence to hmmLM(). Got rid of the unnecessary/erroneously used function newstep(). Incremented the version number. 07/07/2018 Version 2.0-4 |--> 2.0-5 Fixed bug in revise.ispd() --- had "eee$vectors" in error message; should be "eee$values". Change conditional browsers() in ffun(), logLikHmm() and msRho() (in type4stuff.R) to depend on interactive(). Incremented the version number. 10/07/2018 Version 2.0-5 |--> 2.0-6 Adjusted hmmNumOpt() to do a restart from the penultimate parameter values when it hits a problem with ispd. Stores the penultimate value in a specially created environment. Not at all clear why this should "work" but it seems to have an impact. Added some comments to the functions in directMaxStuff.R, to clarify their roles. Incremented the version number. 11/07/2018 Version 2.0-6 |--> 2.0-7 Did some more commenting. Re-indexed the sub-functions if ffun() to be f1, f2, f3 and f4 (rather than the confusing f0, ..., f3). Changed the argument "Dat" of f1, f2, f3 and f4 to "Y" to avoid confusion with list "Dat". (Note that "Y" is one of the *entries* of "Dat".) Incremented the version number. 18/07/2018 Version 2.0-7 |--> 2.0-8 Added the capability for hmmNumOpt() to tell nlm() or optim() to avail themselves of an analytically calculated gradient vector produced by get.gl(). Whether to do so is specified by an argument useAnalGrad, which is passed to hmmNumOpt() via the ... argument. Adjusted the documentation in hmm.discnp-internal.Rd and hmm.Rd accordingly. Incremented the version number. 20/07/2018 Version 2.0-8 |--> 2.0-9 Corrected a typo in hydroDat.Rd. 04/08/2018 Fixed a major fuck-up in the calculation of the hessian. An important term had been omitted. 09/08/2018. Incremented the version number. (To 2.0-9.) 09/08/2018 Fixed init.c w.r.t. the call to gethgl. 11/08/2018 Fixed a bug that had been lurking in get.hgl.R and getghl.r (and in get.gl.R and gethl.r) for God knows how long. An undefined value "nphi" was being passed as an argument to the Fortran code. This fix eliminated a seg fault that had been occuring. 13/08/2018. Later got email from Peter Dalgaard saying that this should have triggered an "object nphi not found" error --- which it didn't! I put the code back (???) to the way it was before, and then of course got that error! Re-righted the code. Seg fault still "eliminated", so I'll just carry on. Incremented the version number. (To 2.0-10.) 16/08/2018 Fixed a bug in tidylist() (the string "colnms" had been written as "colnames" whence the *function* was found, yielding a bewildering error. 03/10/2018 Fixed a bug in logLikHmm() --- didn't handle the K=1 case properly. 03/10/2018 Revised update.hmm.discnp() --- now has a new argment Kplus1=TRUE which allows fitting a model with K+1 states starting from a fit to a model with K states. (Thus an improvement of the log likelihood is guaranteed when K is increased by 1.) 03/10/2018 Changed the EM algorithm procedure so that if an increase of the log likelihood occurs, the function switches to the "LM" method rather than to the "bf" method. 03/10/2018 Incremented the version number. (To 2.0-11.) 03/10/2018 Adjusted update.hmm.discnp() to have arguments "tpm2" and "verbose". 04/10/2018. (No change to version number.) Changed "decrease in log likelihood" to "increase in log likelihood" in hmmLM() and hmmSD(). (Duhhh!) Adjusted hmmLM() and hmmSD() to calculate initial log likelhood (rather than using -Inf) and to print out the initial value when verbose=TRUE. Adjusted hmmUV() to output a blank line after printing the initial log likelihood value. 22/10/2018. (No change to version number.) Fixed bug in fitted.hmm.discnp(); object$y produced the object "ylengths" rather than "y" (and always produced a result, even if "keep.y" had been set to FALSE) because of partial matching. Changed this to object[["y"]]. Fixed bug in update.hmm.discnp(); it was effectively applying tidyList() to the data twice, resulting in the loss of any "numeric" characteristic (since tidyList() coerces its arguments to "character" after discerning whether they have "numeric" mode. Adjusted "tidyList()" so that its returned value is of class "tidyList". Adjusted hmm() so that if its "y" argument is (already) of class "tidyList", then the tidyList() function is *not* applied to it. Incremented version number. (To 2.0-12.) 23/10/2018. Adjusted sp() to make use of addIntercept argument in the appropriate manner. Adjusted logLikHmm() to make use of addIntercept argument in the appropriate manner. Adjusted makeDat() to make use of of prednames and addIntercept arguments in the appropriate manner. Got rid of the old shaganappi system for dealing with adding an intercept column to the predictor matrices. Added "list" to the classes from which the value of makeDat() inherits. Added a function checkConstColumns() to deal with problems arising when adding an intercept and checking for certain anomalies in the predictor matrices. Did a major revision of tidyList(); now tidyList() handles adding an intercept column to the predictor matrices. It is now better organised in respect of determining whether "addIntercept=TRUE" actually makes sense. The "prednames" attribute now has "Intercept" as its first entry whenever the predictors indeed have an intercept. The "ncol" attribute now counts the intercept column. Added "list" to the classes from which the value of tidyList() inherits. Changed hmmUV() so that if there is a predictor matrix (which LM can't handle --- yet???) it switches to the "bf" method when the EM algorith produces a decrease in the log likelihood. Put in error-throws if hmmUV() is called with method = LM or SD, when there is a predictor "X". Major revision to updata.hmm.discnp() so that it deals properly with auxilliary predictor variables "X". It now handles the K=1 case properly, even when there are predictor variables. Adjusted init.all() to conform properly to the new structure of the output of tidyList(). It now uses "prednames" to construct the column names of the Rho dataframe. The "ncX" argument (number of columns of X) is no longer used, and neither is "addIntercept". Incremented the version number. (To 2.0-13.) 29/10/2018 Adjusted revise.rho() to use the "prednames" attribute of "Dat" to form the column names of Rho. Fixed bug (w.r.t. augmenting "prednames" with "Intercept") in checkConstColumns(). Did major revision/bug-fix of tidyList() to handle the "prednames" structure properly. In particular corrected the way "prednames" is formed if entries of "X" have no column names. Adjusted makeDat() to provide a "prednames" attribute when "X" is NULL. Amended the hmm() help to specify the restrictions/conditions that the "X" argument (if present) must satisfy. Incremented the version number. (To 2.0-14.) 31/10/2018 Fixed bugs in misstify(). (Wrong order of application of fix1() and fix2().) Fixed bugs in rhmm.default() (was using x[j] rather that x[[j]], where "x" is a list, in some instances). Revised the argument structure of misstify() (and hence of rhmm.*()). Replace the shaganappi "p2" by "fep" (first entry present). Provided sensible default values of the entries of fep. Revised the help for rhmm() and misstify(). Added the data set "ccprSim" which consists of data simulated from a model fitted to a real data set which has to be kept confidential due to ethical constraints. Incremented the version number. (To 2.0-15.) 01/11/2018 Fixed bug in makeDat (assigned attribute "rhovals" to the returned structure when type is 2). Re-did hmmNumOpt so that if nlm() falls over, the function switches to using optim() rather the previous shaganappi procedure of retrying nlm() from a previous iterate of the parameter vecot. (This latter procedure should never work anyway!) Introduced the argument "hessian" indicating whether a hessian matrix should be returned by nlm() or optim(). Defaults to FALSE which saves a huge amount of frustrating time at the end (between completion of iterations and actually returning a value). It appears that this final calculation of the hessian is *very* time consuming. Incremented the version number. (To 2.0-16.) 04/11/2018 Improved the logical flow, instigated by an error thrown when the get.{hgl,gl,l}() functions could not find the "rhovals" (the unique values of the emissions) due the object returned by makeDat() being incorrectly structered. If it was right for some circumstances it was wrong for others, under the old (shaganappi) logial flow. * Reorganised tidyList(). * Reorganised makeDat(). * Reorganised get.l(). * Reorganised get.gl(). * Reorganised get.hgl(). * Reorganised ffun(). * Adjusted hmmUV(). * Adjusted hmmBI(). * Adjusted logLikHmm(). * Adjusted sp(). In particular, the object returned by tidyList() now always has a "lvls" (formerly "uval") attribute so that makeDat() can get the levels of the factor in the "y" column from that attribute and they no longer have to be passed as an argument. The makeDat() function now just returns data frames whose nature depends only the parity (which is an attribute of the "y" argument) and not on "type" so type no longer has to be passed as an argument. Incremented the version number. (To 2.1-1.) 06/11/2018 Fixed some bugs in get.*.R involving a missnaming of the "rhovals" attribute of a "madeDat" object. Changed name of the "rhovals" attribute in the makeDat() function to "lvls" to make it consistent with tidyList(). Added an acknowledgement of Prof. Kurt Brännäs in the documentation of the data from Prof. Weiß's book. Incremented the version number. (To 2.1-2.) 07/11/2018 Fixed a bug in ffun(), type 3. The columns of Y must be passed to f2() as Y[,1,drop=FALSE] and Y[,2,drop=FALSE], since f2() is expected data frames, not factors. Fixed a bug in init.all() --- need to have the ncol argument of matrix() equal to nval[i], *not* nval. (If the vector nval is passed, the *first* entry of nval is used, equivalent to always using nval[1].) Fixed hmmBI() to resolve conflict/inconsistency between the "yval" argument and rownames(Rho[[i]]) if Rho is given in par0. The row names of the components of Rho now take precedence. Fixed hmmBD(), in a manner similar to that with which hmmBI() was treated above, to resolve conflict/inconsistency between the "yval" argument and dimnames(Rho) if Rho is given in par0. The dimnames attribute of Rho now takes precedence. Put in a check on the adequacy of the data (must have at least *one* completely non-missing combination of the values of variable 1 and variable 2) in hmmBD(). Found a really bad error in hmmBI() --- the non-redundant parameters from Rho were being extracted completely incorrectly! Fixed. Incremented the version number. (To 2.1-3.) 10/11/2018 Fixed a numerical instability in derivp() that was causing errors to be thrown. Fixed a similar instability in derivf() (although this had not --- yet --- triggered any errors). Incremented the version number. (To 2.1-4.) 19/11/2018 Fixed a bug in hmmUV(); wrong "yval"/"lvls" value were being used if the unique value of the observations did not "fill out" all of the rownames of par0$Rho (or all of the levels of par0$Rho$y if the "newstyle" structure was being used. As for hmmBI() and hmmBD() above, made the charcteristics of par0$Rho take precedence. Fixed a bug in hmmUV() which occurred when newstyle==TRUE. In the call to multinom() problems arose if not all of the "lvls" values actually appeared in the data. Suppressed the warnings coming from multinom() and arranged the values of "lvls" corresponding to absent data to get supplied with large negative values (making the corresponding probabilities equal to zero --- as they should be.) Fixed a bug in hmmUV() w.r.t. the switch to method="LM" if there is a *decrease* in the log likelihood. The data "Dat" were being (now completely unnecessarily) being restructured, which was mucking up the call to get.l(). Eliminated the restructuring. Incremented the version number (to 2.1-5) and submitted to CRAN 26/11/2018. Found an egregious error in derivp.R. Fixed it. (Strangely the corresponding error was *not* made in derivf.R.) Incremented the version number (to 2.1-6). 29/11/2018 Re-visited derivf.R --- turns out there were errors therein even more egregious than in derivp.R. Corrected these errors. Incremented the version number (to 2.1-7). 26/01/2019 Fixed bug in viterbi(). Thanks to Jeroen Donkers for pointing out (and isolating) the bug. Fixed corresponding bugs in pr() and mps(); made some related changes in sp(). Fixed a bug in fitted.hmm.discnp(). Changed check.yval() --- realised that there is no need to pass all of the data set y, only to extract the "lvls" attribute!! So made the first argument yval, rather than y. Now the calls to this function are made in the form check.yval(attr(y,"lvls"),...). Incremented the version number (to 2.1-8). 02/02/2019 Adjusted the treatment of a specified "yval" slightly; if it is identical with the rowname of a supplied par0$Rho, then no warning is issued. Changed code so that if there is no predictor matrix "X" then revise.rho() and fun() get called with type=2 (with appropriate adjustments being made to "Rho"), which speeds things up by a factor of 5+. Incremented the version number (to 2.1-9). 25/02/2019 Change/adjusted sp(), fitted.hmm.discnp() and predict.hmm.discnp() so that "fitted/predicted" values (in a certain sense) are returned when the data are categorical. Reorganised and streamlined (???) code. Incremented the version number (to 2.1-10). 09/03/2019 Added the S3 class "multipleHmmDataSets"; arranged for the objects returned by rhmm() to have this class. Added an extractor method for this class. Added the function predictEngineHmmDiscnp() to be called by fitted.hmm.discnp() and predict.hmm.disnp(), thereby tidying up the code 14/03/2019 Fixed a major glitch in ffun.R. I had neglected to allow for the "numerical gotcha" resulting from doing exp(v)/sum(exp(v)) when there are large (positive) values in v. In such cases one gets "Inf/Inf" whence "Nan". The fix is of course to subtract the maximum of v from each entry of v. Done Incremented the version number (to 2.1-11). 01/06/2019 Fixed up some glitches in the documentation. Added as suppressWarnings() to a call to as.numeric() in check.yval() to avoid a disconcerting (misleading and bascically spurious) warning that can occur. Adjusted a comment (in f1()) in ffun.R. 07/08/2019 Adjusted code to accommodate the fact that R now assigns the class of a matrix as c("matrix","array"). Incremented the version number (to 2.1-12). Uploaded to CRAN. 05/12/2019 01/01/2020 Commenced a major (???) revision. In particular got rid of the "newstyle" argument. The "Rho" parameter matrix will now be accepted in either old (matrix) or new (dataframe) style, but if presented in the old style it will be immediately and automatically converted to the new style. 11/01/2020 Got things to work and check. Some problems remain in respect of rhmm() and mps(). Set the version number to 3.0-1. 04/02/2020 Added in the "ipopt" method of fitting models; this depends on the (rather arcane) Ipopt package. 13/02/2020 Fixed up rhmm() to return *numeric* data where appropriate. Fixed up mps() to always return "categorical" states. Also fixed it to handle objects of class "multipleHmmDataSets". Checked on Viterbi; apparently it never handled missing values in the observations. Adjusted it so that now it does; arranged things so that it calls upon ffun() to do the calculations of probabilities. Checked that the new version gave the same answer as the old one in an example where there were no missing values. Fixed a couple of serious glitches in pr(). Adjusted pr() to work if "y" is of class "multipleHmmDataSets" (as returned by rhmm() if "nsim" is greater than 1. Incremented the version number (to 3.9-2). 14/02/2020 Adjusted hmm() to return the value of the "stationary" argument (so that reparam() can make use of it when it is applied to an object of class "hmm.discnp". Adjusted reparam() so that it can do the "raw" (probability) parameters as well as the "exponential" form of the parameters. (Choice controlled by the logical argument "expForm".) Adjusted scovmat() and squantCI() accordingly. Adjusted the help files for: * logLikHmm() * misstify() * nafracCalc() * predict.hmm.discnp() * pr() * scovmat() * sp() * squantCI() * mps() * viterbi() * hmm() (In particular got the help consistent with the capacity of functions to handle objects of class "multipleHmmDataSets".) Fixed a glitch in init.all() (did not give dimnames to Rho matrices or array in the bivariate setting). Fixed a glitch in hmmBD() --- the value of "lns" was set in the wrong place (after its initial use!!!). Made a wee adjustment to checkStartVal() --- made sure "stnms" was of mode character. Incremented the version number (to 3.9-3). 12/04/2020 Fixed bug in tidyList() --- made sure that the "lvls" attribute of the returned value (or each of the components thereof in the bivariate case) is of mode "character". 13/04/2020 Fixed buglet w.r.t. "method"; need to apply match.arg() to "method" before comparing method with "ipoptr". 14/04/2020 Fixed bug in cnvrtRho(): needed to use expForm2p() when converting from "data frame form" to "matrix" form. 23/06/2020 Backed out the introduction of the "ipopt" method; doesn't seem to work worth a damn and is not actually available on CRAN anyhow. Incremented the version number (to 3.0-4). Uploaded to CRAN 26/06/2020 Started searching for the bug revealed by an experiment with the toy data that came from Noric Couderc. (That I *should* have caught before uploading to CRAN, Grrrrrr!!!) Fixed a mess-up w.r.t where "rhovals" gets defined in hmmUV(). Fixed the bug that was fucking everything up for the LM method; turned out to be "m <- nrow(Rho)" which assumed that Rho was a *matrix* of emission probabilities, whereas it is (now) a data frame providing the "new style" representation. This resulted in the Fortran code writing to unallocated memory, and all hell broke loose. Adjusted init.all() so that rand.start is allowed to be a single logical scalar (and not necessarily a list of two such). If it is a logical scalar, it is converted into a list, both components of which are equal to that scalar. Changed the name of "method" in get.hgl() to "hglmethod"; took out the match.arg() construction, which does not belong there. Put the match.arg() construction into hmm() where it does belong. Got rid of the procedure of automatically switching methods (to "bf" or "LM") if the EM algorithm produces a decrease in the log likelihood. An error is now thrown and the user is instructed to change the method explicitly, starting from scratch. Changed the name of the file "type4stuff.R" to "bivDepStuff.R". (There is no longer a "type = 4" case!!!) Incremented the version number (to 3.0-5). Uploaded to CRAN. 27/06/2020 Realized there was *STILL* a fucking bug in the package!!! The same error appears in get.gl() as in get.hgl(). I had fixed the latter but not the former. It is mysterious why the bug was not triggered by Noric Couderc's toy data. Get a slightly different answer when using the correct code than when using the buggy code, but only slightly different. And no crashes. Go figure. Have now fixed get.gl(). Incremented the version number (to 3.0-6). Uploaded to CRAN. 09/02/2022 Changed the response, to a decrease in the log likelihood when the EM algorithm is used, from throwing an error to returning the model corresponding to the last "valid" EM step, with a message pointing out that a decrease had occurred and suggesting the use of the returned value to provided starting values for the "bf" or "LM" algorithms. Adjusted the help file for hmm() accordingly. Augmented the examples accordingly. Incremented the version number (to 3.0-7). Uploaded to CRAN. 15/02/2022 Fixed bug --- fix.tpm() gave wrong answers when K was greater than 2; the byrow=TRUE specification was incorrect. But fix.tpm() was redundant anyway; should have been using getTpm() from directMaxStuff.R. Adjust code replacing calls to fix.tpm() by calls to getTpm(). Removed the file fix.tpm.R. 16/02/2022 Amended the message produced when the EM algorithm encounters a decrease in the log likelihood. Add the ordinal() function (cribbed from spatstat; code in suffices.R) to get the suffix appended to "em.step" correct. Adjusted the help file a bit. 19/02/2022 Fixed some glitches in the arguments of functions called by steepest() (caused by copying and pasting from hmmRT, where the argument lists are different). 20/02/2022 Adjusted the f1() function in ffun() to allow for the possibility that Y has a *single* column, which it will if f1() is called from f2(). 03/05/2022 Incremented the version number (to 3.0-8). 27/08/2022 Fixed a massive nest of bugs relating to the handling of additional (i.e. in addition to "state") predictors. I must not have tested out this facility. Problem pointed out to me by Leah Walker, 25/08/2022. 28/08/2022 Added an example involving additional predictors to the help for hmm(). 26/09/2022 Added material to the help file for hmm(), discussing the interpretation of the coefficients in the model that is fitted. Incremented the version number (to 3.0-9).