TOCPOs are more complicated than FOCPOs, since they potentially need
to operate on data at three different points: During initial training,
during the re-transformation for new prediction data, and during the
inversion of predictions made by a model trained on transformed data.
Similarly to makeCPO(), makeCPOTargetOp()
splits these operations up into functions that create
“control” objects, and functions that do the actual
transformation. makeCPOExtendedTargetOp(), on the other
hand, gives the user more flexibility at the price of the user having to
make sure that transformation and retransformation perform the same
operation–similarly to makeCPOExtendedTrafo() for
FOCPOs.
makeCPOTargetOp()
makeCPOTargetOp() has a cpo.train and
cpo.retrafo function parameter that work similarly to the
ones of makeCPO(). In contrast to makeCPO(),
however, cpo.retrafo must return the target data
instead of the feature data. The data and
target parameters of cpo.retrafo get the same
data as they get in a FOCPO created with makeCPO(), with
the exception that if dataformat is "task" or
"df.all", the target parameter will receive
the whole input data in form of a Task or
data.frame (while the data argument, as in a
FOCPO, will receive only the feature data.frame). The
return value of cpo.retrafo for a TOCPO must always be in
the same format as the input target value: a
data.frame with the manipulated target values when
dataformat is anything besides "task" or
"df.all", or a Task or data.frame
of all data (with non-target columns unmodified) otherwise.
Inversion of predictions is performed using the functions
cpo.train.invert and cpo.invert.
cpo.train.invert takes a data and a
control argument, and any arguments declared in the
par.set. It is called whenever new data is fed into the
CPO or its retrafo CPOTrained, and creates a
CPOTrained state that is used to invert the prediction done
on this new data. The control argument takes the value
returned by the cpo.train function upon initial training,
and the data argument is the new data for which to prepare
the CPOTrained inverter. It has the form dictated by
dataformat, with the exception that "task" and
"df.all" dataformat are handled as
"df.feature"; this is necessary since the new data could be
a data.frame of data with unknown target.
The following is an example of a TOCPO that trains a classification
Learner on a binary classification Task and
changes it to a Task of whether or not the
Learner predicted the truth for a given data line
correctly. (Real-world applications would probably need to take some
precautions against overfitting.) In its cpo.train step,
the given Learner is trained on the incoming data and the
resulting WrappedModel object is returned as the
“control” object. This is given to the
cpo.retrafo function, which performs prediction and creates
a new classification Task with the match / mismatch between
model prediction and ground truth as target. When an external
Learner is trained on data that was preprocessed like this,
its prediction will be whether the CPO-internal
Learner can be trusted to predict a given data row. To
“invert” this, i.e. to get the actual prediction, the
cpo.invert function needs to have the internal
Learner’s prediction as well as the prediction made by the
external Learner. The former is provided by
cpo.train.invert, which uses the WrappedModel
to make a prediction on the new data, and given as
control.invert to cpo.invert. The latter is
the target data given to cpo.invert. This
example CPO supports inverting both "response"
and "prob" predict.type predictions, as
declared in the predict.type.map argument. The actual
predict.type to invert is given to cpo.invert
as an argument.
xmpMetaLearn = makeCPOTargetOp("xmp.meta", # nolint
pSS(lrn: untyped),
dataformat = "task",
properties.target = c("classif", "twoclass"),
predict.type.map = c(response = "response", prob = "prob"),
cpo.train = function(data, target, lrn) {
cat("*** cpo.train ***\n")
lrn = setPredictType(lrn, "prob")
train(lrn, data)
},
cpo.retrafo = function(data, target, control, lrn) {
cat("*** cpo.retrafo ***\n")
prediction = predict(control, target)
tname = getTaskTargetNames(target)
tdata = getTaskData(target)
tdata[[tname]] = factor(prediction$data$response == prediction$data$truth)
makeClassifTask(getTaskId(target), tdata, tname, positive = "TRUE",
fixup.data = "no", check.data = FALSE)
},
cpo.train.invert = function(data, control, lrn) {
cat("*** cpo.train.invert ***\n")
predict(control, newdata = data)$data
},
cpo.invert = function(target, control.invert, predict.type, lrn) {
cat("*** cpo.invert ***\n")
if (predict.type == "prob") {
outmat = as.matrix(control.invert[grep("^prob\\.", names(control.invert))])
revmat = outmat[, c(2, 1)]
outmat * target[, "prob.TRUE", drop = TRUE] +
revmat * target[, "prob.FALSE", drop = TRUE]
} else {
stopifnot(levels(target) == c("FALSE", "TRUE"))
numeric.prediction = as.numeric(control.invert$response)
numeric.res = ifelse(target == "TRUE",
numeric.prediction,
3 - numeric.prediction)
factor(levels(control.invert$response)[numeric.res],
levels(control.invert$response))
}
})
cpo = xmpMetaLearn(makeLearner("classif.logreg"))
To show the inner workings of this CPO, the following
example data is used.
set.seed(12)
split = makeResampleInstance(hout, pid.task)
train.task = subsetTask(pid.task, split$train.inds[[1]])
test.task = subsetTask(pid.task, split$predict.inds[[1]])
It can be instructive to watch the cat() output of this
CPO to see which function gets called at what point in the
lifecycle. The cpo.train function is called first to create
the control object. The Task is transformed in
cpo.retrafo. Also cpo.train.invert is called,
since an inverter attribute is attached to the returned
trafo.
trafd = train.task %>>% cpo
#> *** cpo.train ***
#> *** cpo.train.invert ***
#> *** cpo.retrafo ***
attributes(trafd)
#> $names
#> [1] "type" "env" "weights" "blocking" "coordinates"
#> [6] "task.desc"
#>
#> $class
#> [1] "ClassifTask" "SupervisedTask" "Task"
#>
#> $retrafo
#> CPO Retrafo chain {type:classif}
#> [RETRAFO xmp.meta(lrn = <classif.log...)]
#>
#> $inverter
#> CPO Inverter chain {type:classif} (able to predict 'response', 'prob')
#> [INVERTER xmp.meta(lrn = <classif.log...){type:classif}]
The values of the target column (“diabetes”) of the result can be
compared with the prediction of a "classif.logreg"
Learner on the same data:
head(getTaskData(trafd))
#> pregnant glucose pressure triceps insulin mass pedigree age diabetes
#> 450 0 120 74 18 63 30.5 0.285 26 TRUE
#> 346 8 126 88 36 108 38.5 0.349 49 FALSE
#> 336 0 165 76 43 255 47.9 0.259 26 FALSE
#> 247 10 122 68 0 0 31.2 0.258 41 TRUE
#> 174 1 79 60 42 48 43.5 0.678 23 TRUE
#> 453 0 91 68 32 210 39.9 0.381 25 TRUE
model = train(makeLearner("classif.logreg", predict.type = "prob"), train.task)
head(predict(model, train.task)$data[c("truth", "response")])
#> truth response
#> 450 neg neg
#> 346 neg pos
#> 336 neg pos
#> 247 neg neg
#> 174 neg neg
#> 453 neg neg
When new data is transformed using the retrafo
CPOTrained, another inverter attribute is
created, and hence cpo.train.invert is called again. Since
the target column of the test.task in the following example
is also transformed, the cpo.retrafo function is
called.
retr = test.task %>>% retrafo(trafd)
#> *** cpo.retrafo ***
#> *** cpo.train.invert ***
attributes(retr)
#> $names
#> [1] "type" "env" "weights" "blocking" "coordinates"
#> [6] "task.desc"
#>
#> $class
#> [1] "ClassifTask" "SupervisedTask" "Task"
#>
#> $inverter
#> CPO Inverter chain {type:classif} (able to predict 'response', 'prob')
#> [INVERTER xmp.meta(lrn = <classif.log...){type:classif}]
In a real world application, it would be possible for the new
incoming data to have unknown target values. In that case, no target
column would need to be changed, and cpo.retrafo is
not called. The resulting data, retr.df, equals
the input data with a retrafo attribute added.
retr.df = getTaskData(test.task, target.extra = TRUE)$data %>>% retrafo(trafd)
#> *** cpo.train.invert ***
names(attributes(retr.df))
#> [1] "names" "class" "row.names" "inverter"
The invert functionality can be demonstrated by making a prediction
with an external model.
ext.model = train("classif.svm", trafd)
ext.pred = predict(ext.model, retr)
newpred = invert(inverter(retr), ext.pred)
#> *** cpo.invert ***
performance(newpred)
#> mmce
#> 0.2200521
It may also be instructive to attach the xmpMetaLearn
CPO to a Learner to see which functions get
called during training and prediction of a TOCPO-Learner.
Since the Learner does not do inversion of the training
data, a CPOTrained for inversion is not created during
training, and cpo.train.invert is hence not called. Only
cpo.train (for control object creation) and
cpo.retrafo (target value change) are called. During
prediction, the input data is used to create an (internally used)
inversion CPOTrained which promptly gets used by the
prediction made by "classif.svm". Hence both
cpo.train.invert and cpo.invert are called in
succession.
cpo.learner = cpo %>>% makeLearner("classif.svm")
cpo.model = train(cpo.learner, train.task)
#> *** cpo.train ***
#> *** cpo.retrafo ***
lrnpred = predict(cpo.model, test.task)
#> *** cpo.train.invert ***
#> *** cpo.invert ***
performance(lrnpred)
#> mmce
#> 0.2200521
See Postscriptum for an evaluation of
xmpMeatLearn’s performance.
Functional TOCPO
Just like for FOCPOs, it is possible to create functional TOCPOs. In
the case of makeCPOTargetOp(), it is possible to have
cpo.train create cpo.retrafo and
cpo.train.invert, instead of giving them to
makeCPOTargetOp() directly. Just as in
makeCPO, these functions can then access the state of their
environment in the cpo.train call and hence have neither a
control argument, nor any arguments for the
par.set parameters. Since cpo.train must in
this case create two functions, these functions only need to be defined
within cpo.train, the return value is ignored.
Note that cpo.retrafo and cpo.train.invert
must either be both functional or both object based.
It is furthermore possible to return a cpo.invert
function by cpo.train.invert, instead of giving it to
makeCPOTargetOp(). As above, the returned function should
not have any parameters for the ones given in par.set, and
should not have a control.invert. cpo.invert
can be functional or not, independently of whether
cpo.retrafo and cpo.train.invert are
functional.
As in makeCPO(), all functions that are given
functionally must be explicitly set to NULL in the
makeCPOTargetOp() call.
The xmpMetaLearn example above with functional
cpo.retrafo, cpo.train.invert and
cpo.invert would look like the following:
xmpMetaLearn = makeCPOTargetOp("xmp.meta.fnc", # nolint
pSS(lrn: untyped),
dataformat = "task",
properties.target = c("classif", "twoclass"),
predict.type.map = c(response = "response", prob = "prob"),
# set the cpo.* parameters not needed to NULL:
cpo.retrafo = NULL, cpo.train.invert = NULL, cpo.invert = NULL,
cpo.train = function(data, target, lrn) {
cat("*** cpo.train ***\n")
lrn = setPredictType(lrn, "prob")
model = train(lrn, data)
cpo.retrafo = function(data, target) {
cat("*** cpo.retrafo ***\n")
prediction = predict(model, target)
tname = getTaskTargetNames(target)
tdata = getTaskData(target)
tdata[[tname]] = factor(prediction$data$response == prediction$data$truth)
makeClassifTask(getTaskId(target), tdata, tname, positive = "TRUE",
fixup.data = "no", check.data = FALSE)
}
cpo.train.invert = function(data) {
cat("*** cpo.train.invert ***\n")
prediction = predict(model, newdata = data)$data
function(target, predict.type) { # this is returned as cpo.invert
cat("*** cpo.invert ***\n")
if (predict.type == "prob") {
outmat = as.matrix(prediction[grep("^prob\\.", names(prediction))])
revmat = outmat[, c(2, 1)]
outmat * target[, "prob.TRUE", drop = TRUE] +
revmat * target[, "prob.FALSE", drop = TRUE]
} else {
stopifnot(levels(target) == c("FALSE", "TRUE"))
numeric.prediction = as.numeric(prediction$response)
numeric.res = ifelse(target == "TRUE",
numeric.prediction,
3 - numeric.prediction)
factor(levels(prediction$response)[numeric.res],
levels(prediction$response))
}
}
}
})
Constant Invert TOCPOs
The example given above is a relatively elaborate TOCPO which needs
information from the prediction data to perform inversion. Many simpler
applications of target transformation do not need this information if
their inversion step is independent of this data. It is possible to
declare such a TOCPO using the constant.invert flag in
makeCPOTargetOp(). If constant.invert is set
to TRUE, the cpo.train.invert argument must be
explicitly set to NULL. cpo.train still needs
to have a control.invert argument; it is set to the value
returned by cpo.train.
The following example is a TOCPO for regression Tasks
that centers target values during training. After prediction, the data
is inverted by adding the original mean of the training data to the
predictions. This inversion operation does not need any information
about the prediction data going in, so the TOCPO can be declared
constant.invert.
The cpo.retrafo function is also called when new
prediction data with a target column is transformed (as during
model validation). In that case, the mean of the training data
column is subtracted. Therefore the mean generated by
cpo.train needs to be used in cpo.retrafo
(i.e. the control value), not the mean of the
target data present.
xmpRegCenter = makeCPOTargetOp("xmp.center", # nolint
constant.invert = TRUE,
cpo.train.invert = NULL, # necessary for constant.invert = TRUE
dataformat = "df.feature",
properties.target = "regr",
cpo.train = function(data, target) {
# control value is just the mean of the target column
mean(target[[1]])
},
cpo.retrafo = function(data, target, control) {
# subtract mean from target column in retrafo
target[[1]] = target[[1]] - control
target
},
cpo.invert = function(target, predict.type, control.invert) {
target + control.invert
})
cpo = xmpRegCenter()
To illustrate this CPO, the following data is used:
train.task = subsetTask(bh.task, 150:155)
getTaskTargets(train.task)
#> [1] 15.4 21.5 19.6 15.3 19.4 17.0
predict.task = subsetTask(bh.task, 156:160)
getTaskTargets(predict.task)
#> [1] 15.6 13.1 41.3 24.3 23.3
The target column of the task after transformation has a mean of
0.
trafd = train.task %>>% cpo
getTaskTargets(trafd)
#> [1] -2.633333 3.466667 1.566667 -2.733333 1.366667 -1.033333
When applying the retrafo CPOTrained to a new task, the
mean of the training task target column is subtracted.
getTaskTargets(predict.task)
#> [1] 15.6 13.1 41.3 24.3 23.3
retr = retrafo(trafd)
predict.traf = predict.task %>>% retr
getTaskTargets(predict.traf)
#> [1] -2.433333 -4.933333 23.266667 6.266667 5.266667
When inverting a regression prediction, the mean of the training data
target column is added to the prediction.
model = train("regr.lm", trafd)
pred = predict(model, predict.traf)
#> Warning in predict.lm(.model$learner.model, newdata = .newdata, se.fit = FALSE,
#> : prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
pred
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 -2.433333 -2.854142
#> 157 2 -4.933333 5.802064
#> 158 3 23.266667 14.442103
#> 159 4 6.266667 8.302782
#> 160 5 5.266667 8.633308
invert(inverter(predict.traf), pred)
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 15.6 15.17919
#> 157 2 13.1 23.83540
#> 158 3 41.3 32.47544
#> 159 4 24.3 26.33611
#> 160 5 23.3 26.66664
Since "regr.lm" is translation invariant and
deterministic, the prediction equals the prediction made without
centering the target:
model = train("regr.lm", train.task)
predict(model, predict.task)
#> Warning in predict.lm(.model$learner.model, newdata = .newdata, se.fit = FALSE,
#> : prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 15.6 15.17919
#> 157 2 13.1 23.83540
#> 158 3 41.3 32.47544
#> 159 4 24.3 26.33611
#> 160 5 23.3 26.66664
A special property of constant.invert TOCPOs is that
their retrafo CPOTrained can also be used for inversion.
This is the case since the tight coupling of inversion operation to the
data used to create the prediction is not necessary when the inversion
is actually independent of this data. This is indicated by
getCPOTrainedCapability() returning a vector with the
"invert" capability set to 1. However, when
using the retrafo CPOTrained for inversion, the “truth”
column is absent from the inverted prediction.
getCPOTrainedCapability(retr)
#> retrafo invert
#> 1 1
invert(retr, pred)
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id response
#> 156 1 15.17919
#> 157 2 23.83540
#> 158 3 32.47544
#> 159 4 26.33611
#> 160 5 26.66664
Functional Constant Invert TOCPO
Just as above, constant.invert TOCPOs can be
functional. For this, the cpo.train function must
declare both a cpo.retrafo and a
cpo.invert variable which perform the requested operations.
These functions have no control or
control.invert parameter, and no parameters pertaining to
par.set.
Stateless TOCPO
Very simple target column operations that operate on a row-by-row
basis without needing information e.g. from training data, can be
declared as “stateless”. Similarly to makeCPO(), when
cpo.train parameter is set to NULL, no control
object is created for a CPOTrained. Furthermore, a
stateless TOCPO must always have constant.invert set as
well. Therefore, only cpo.retrafo and
cpo.invert are given as functions, both without a
control or control.invert argument. One
example is a TOCPO that log-transforms the target column of a regression
task, and exponentiates the predictions made from this during inversion.
(A better inversion would take the "se" prediction into
account, see cpoLogTrafoRegr.)
xmpLogRegr = makeCPOTargetOp("log.regr", # nolint
constant.invert = TRUE,
properties.target = "regr",
cpo.train = NULL, cpo.train.invert = NULL,
cpo.retrafo = function(data, target) {
target[[1]] = log(target[[1]])
target
},
cpo.invert = function(target, predict.type) {
exp(target)
})
cpo = xmpLogRegr()
The CPO takes the logarithm of the task target column
both during training and when using the retrafo
CPOTrained.
trafd = train.task %>>% cpo
getTaskTargets(trafd)
#> [1] 2.734368 3.068053 2.975530 2.727853 2.965273 2.833213
retr = retrafo(trafd)
predict.traf = predict.task %>>% retr
getTaskTargets(predict.traf)
#> [1] 2.747271 2.572612 3.720862 3.190476 3.148453
model = train("regr.lm", trafd)
pred = predict(model, predict.traf)
#> Warning in predict.lm(.model$learner.model, newdata = .newdata, se.fit = FALSE,
#> : prediction from rank-deficient fit; attr(*, "non-estim") has doubtful cases
pred
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 2.747271 2.726065
#> 157 2 2.572612 3.224557
#> 158 3 3.720862 3.685047
#> 159 4 3.190476 3.353014
#> 160 5 3.148453 3.365374
Note that both the inverter and the retrafo
CPOTrained can be used for inversion, since a stateless
TOCPO also has constant.invert set. As above, when using
the retrafo CPOTrained, the truth column is absent from the
result.
invert(inverter(predict.traf), pred)
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 15.6 15.27267
#> 157 2 13.1 25.14244
#> 158 3 41.3 39.84699
#> 159 4 24.3 28.58876
#> 160 5 23.3 28.94433
invert(retr, pred)
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id response
#> 156 1 15.27267
#> 157 2 25.14244
#> 158 3 39.84699
#> 159 4 28.58876
#> 160 5 28.94433
makeCPOExtendedTargetOp()
Just as for FOCPOs, it is possible to declare a TOCPO while having
more direct control over what happens at which stage of training,
re-transformation, or inversion. In a TOCPO defined with
makeCPOTargetOp(), the cpo.retrafo and
cpo.train.invert functions are called automatically when
necessary during training and re-transformation.
makeCPOExtendedTargetOp() instead has a
cpo.trafo and a cpo.retrafo parameter, which
get called during the respective operation.
cpo.trafo must be a function taking the same parameters
as cpo.train in makeCPOTargetOp(). Instead of
returning a control object, it must define a variable named
“control”, and a variable named
“control.invert”. The former is used as the
control argument of cpo.retrafo, the latter is
used as control.invert for cpo.invert when
using the inverter CPOTrained created during training. The
return value of cpo.trafo must be similar to the value
returned by cpo.retrafo in makeCPOTargetOp():
it must be the modified data set or target, depending on
dataformat.
cpo.retrafo must take the same parameters as in
makeCPOTargetOp(). It must declare a
control.invert variable that will be given to
cpo.retrafo when using the inverter CPOTrained
created during retransformation. Since cpo.retrafo is
always called during retrafo CPOTrained application, a
“target” column may or may not be present. If a target column is not
present, the target parameter of cpo.retrafo
is NULL and the return value of cpo.retrafo is
ignored; otherwise it must be the transformed target value
(which, as in makeCPOTargetOp(), can be a Task
or data.frame of all data if
dataformat is "task" or
"df.all").
cpo.invert works just as in
makeCPOTargetOp().
The following is a nonsensical, synthetic example that adds
1 to the target column of a regression Task
during initial training, subtracts 1 during retrafo
re-application and is a no-op during inversion.
xmpSynCPO = makeCPOExtendedTargetOp("syn.cpo", # nolint
properties.target = "regr",
cpo.trafo = function(data, target) {
cat("*** cpo.trafo ***\n")
target[[1]] = target[[1]] + 1
control = "control created in cpo.trafo"
control.invert = "control.invert created in cpo.trafo"
target
},
cpo.retrafo = function(data, target, control) {
cat("*** cpo.retrafo ***", "control is:", deparse(control), sep = "\n")
control.invert = "control.invert created in cpo.retrafo"
if (!is.null(target)) {
cat("target is non-NULL, performing transformation\n")
target[[1]] = target[[1]] - 1
return(target)
} else {
cat("target is NULL, no transformation (but control.invert was created)\n")
return(NULL) # is ignored.
}
},
cpo.invert = function(target, control.invert, predict.type) {
cat("*** invert ***", "control.invert is:", deparse(control.invert),
sep = "\n")
target
})
cpo = xmpSynCPO()
For an “extended” TOCPO, only one of the transformation functions is
called in each invocation. Initial transformation calls
cpo.trafo and adds 1 to the targets; using the
CPOTrained for re-transformation calls
cpo.retrafo and subtracts 1.
trafd = train.task %>>% cpo
#> *** cpo.trafo ***
getTaskTargets(trafd)
#> [1] 16.4 22.5 20.6 16.3 20.4 18.0
retrafd = train.task %>>% retrafo(trafd)
#> *** cpo.retrafo ***
#> control is:
#> "control created in cpo.trafo"
#> target is non-NULL, performing transformation
getTaskTargets(retrafd)
#> [1] 14.4 20.5 18.6 14.3 18.4 16.0
It is also possible to perform re-transformation with a
data.frame that does not include the target column. In that
case the target value given to cpo.retrafo
will be NULL, as reported by that function in this
example:
retrafd = getTaskData(train.task, target.extra = TRUE)$data %>>% retrafo(trafd)
#> *** cpo.retrafo ***
#> control is:
#> "control created in cpo.trafo"
#> target is NULL, no transformation (but control.invert was created)
The trafd object has an inverter CPOTrained
attribute that was created by cpo.trafo, the
retrafd object has an inverter CPOTrained
attribute created by cpo.retrafo (necessarily). This is
made visible by the given example inverter function:
inv = invert(inverter(trafd), 1:6)
#> *** invert ***
#> control.invert is:
#> "control.invert created in cpo.trafo"
inv = invert(inverter(retrafd), 1:6)
#> *** invert ***
#> control.invert is:
#> "control.invert created in cpo.retrafo"
