adanet.ensemble¶
Defines built-in ensemble methods and interfaces for custom ensembles.
Ensembles¶
Interfaces and containers for defining ensembles.
Ensemble¶
-
class
adanet.ensemble.
Ensemble
[source]¶ An abstract ensemble of subnetworks.
-
logits
¶ Ensemble logits
tf.Tensor
.
-
predictions
¶ Optional dict of Ensemble predictions to be merged in EstimatorSpec.
These will be additional (over the default included by the head) predictions which will be included in the EstimatorSpec in predictions and export_outputs (wrapped as PredictOutput).
-
subnetworks
¶ Returns an ordered
Iterable
of the ensemble’s subnetworks.
-
ComplexityRegularized¶
-
class
adanet.ensemble.
ComplexityRegularized
[source]¶ An AdaNet ensemble where subnetworks are regularized by model complexity.
Hence an ensemble is a collection of subnetworks which forms a neural network through the weighted sum of their outputs:
\[F(x) = \sum_{i=1}^{N}w_ih_i(x) + b\]Parameters: - weighted_subnetworks – List of
adanet.ensemble.WeightedSubnetwork
instances that form this ensemble. Ordered from first to most recent. - bias – Bias term
tf.Tensor
or dict of string to bias termtf.Tensor
(for multi-head) for the ensemble’s logits. - logits – Logits
tf.Tensor
or dict of string to logitstf.Tensor
(for multi-head). The result of the function f as defined in Section 5.1 which is the sum of the logits of alladanet.WeightedSubnetwork
instances in ensemble. - subnetworks – List of
adanet.subnetwork.Subnetwork
instances that form this ensemble. This is kept together with weighted_subnetworks for legacy reasons. - complexity_regularization – Regularization to be added in the Adanet loss.
Returns: An
adanet.ensemble.Weighted
instance.- weighted_subnetworks – List of
MeanEnsemble¶
-
class
adanet.ensemble.
MeanEnsemble
[source]¶ Mean ensemble.
-
logits
¶ Logits
tf.Tensor
or dict of string to logitstf.Tensor
(for multi-head).
-
subnetworks
¶ List of
adanet.subnetwork.Subnetwork
instances that form this ensemble.
-
predictions
¶ Optional dict mapping prediction keys to Tensors. MeanEnsembler can export mean_last_layer if the subnetworks have the last_layer of the same dimension.
-
MixtureWeightType¶
-
class
adanet.ensemble.
MixtureWeightType
[source]¶ Mixture weight types available for learning subnetwork contributions.
The following mixture weight types are defined:
- SCALAR: Produces a rank 0 Tensor mixture weight.
- VECTOR: Produces a rank 1 Tensor mixture weight.
- MATRIX: Produces a rank 2 Tensor mixture weight.
WeightedSubnetwork¶
-
class
adanet.ensemble.
WeightedSubnetwork
[source]¶ An AdaNet weighted subnetwork.
A weighted subnetwork is a weight applied to a subnetwork’s last layer or logits (depending on the mixture weights type).
Parameters: - name – String name of
subnetwork
as defined by itsadanet.subnetwork.Builder
. - iteration_number – Integer iteration when the subnetwork was created.
- weight – The weight
tf.Tensor
or dict of string to weighttf.Tensor
(for multi-head) to apply to this subnetwork. The AdaNet paper refers to this weight as \(w\) in Equations (4), (5), and (6). - logits – The output
tf.Tensor
or dict of string to weighttf.Tensor
(for multi-head) after the matrix multiplication ofweight
and the subnetwork’slast_layer
. The output’s shape is [batch_size, logits_dimension]. It is equivalent to a linear logits layer in a neural network. - subnetwork – The
adanet.subnetwork.Subnetwork
to weight.
Returns: An
adanet.ensemble.WeightedSubnetwork
object.- name – String name of
Ensemblers¶
Ensemble learning definitions.
Ensembler¶
-
class
adanet.ensemble.
Ensembler
[source]¶ An abstract ensembler.
-
build_ensemble
(subnetworks, previous_ensemble_subnetworks, features, labels, logits_dimension, training, iteration_step, summary, previous_ensemble, previous_iteration_checkpoint)[source]¶ Builds an ensemble of subnetworks.
Accessing the global step via
tf.train.get_or_create_global_step()
ortf.train.get_global_step()
within this scope will return an incrementable iteration step since the beginning of the iteration.Parameters: - subnetworks – Ordered iterable of
adanet.subnetwork.Subnetwork
instances to ensemble. Must have at least one element. - previous_ensemble_subnetworks – Ordered iterable of
adanet.subnetwork.Subnetwork
instances present in previous ensemble to be used. The subnetworks from previous_ensemble not included in this list should be pruned. Can be set to None or empty. - features – Input
dict
oftf.Tensor
objects. - labels – Labels
tf.Tensor
or a dictionary of string label name totf.Tensor
(for multi-head). Can beNone
. - logits_dimension – Size of the last dimension of the logits
tf.Tensor
. Typically, logits have for shape [batch_size, logits_dimension]. - training – A python boolean indicating whether the graph is in training mode or prediction mode.
- iteration_step – Integer
tf.Tensor
representing the step since the beginning of the current iteration, as opposed to the global step. - summary – An
adanet.Summary
for scoping summaries to individual ensembles in Tensorboard. Usingtf.summary()
within this scope will use thisadanet.Summary
under the hood. - previous_ensemble – The best
adanet.Ensemble
from iteration t-1. The created subnetwork will extend the previous ensemble to form theadanet.Ensemble
at iteration t. - previous_iteration_checkpoint – The tf.train.Checkpoint object associated with the previous iteration.
Returns: An
adanet.ensemble.Ensemble
subclass instance.- subnetworks – Ordered iterable of
-
build_train_op
(ensemble, loss, var_list, labels, iteration_step, summary, previous_ensemble)[source]¶ Returns an op for training an ensemble.
Accessing the global step via
tf.train.get_or_create_global_step()
ortf.train.get_global_step()
within this scope will return an incrementable iteration step since the beginning of the iteration.Parameters: - ensemble – The
adanet.ensemble.Ensemble
subclass instance returned by this instance’sbuild_ensemble()
. - loss – A
tf.Tensor
containing the ensemble’s loss to minimize. - var_list – List of ensemble
tf.Variable
parameters to update as part of the training operation. - labels – Labels
tf.Tensor
or a dictionary of string label name totf.Tensor
(for multi-head). - iteration_step – Integer
tf.Tensor
representing the step since the beginning of the current iteration, as opposed to the global step. - summary – An
adanet.Summary
for scoping summaries to individual ensembles in Tensorboard. Usingtf.summary
within this scope will use thisadanet.Summary
under the hood. - previous_ensemble – The best
adanet.ensemble.Ensemble
from the previous iteration.
Returns: Either a train op or an
adanet.ensemble.TrainOpSpec
.- ensemble – The
-
name
¶ This ensembler’s unique string name.
-
ComplexityRegularizedEnsembler¶
-
class
adanet.ensemble.
ComplexityRegularizedEnsembler
(optimizer=None, mixture_weight_type='scalar', mixture_weight_initializer=None, warm_start_mixture_weights=False, model_dir=None, adanet_lambda=0.0, adanet_beta=0.0, use_bias=False, name=None)[source]¶ The AdaNet algorithm implemented as an
adanet.ensemble.Ensembler
.The AdaNet algorithm was introduced in the [Cortes et al. ICML 2017] paper: https://arxiv.org/abs/1607.01097.
The AdaNet algorithm uses a weak learning algorithm to iteratively generate a set of candidate subnetworks that attempt to minimize the loss function defined in Equation (4) as part of an ensemble. At the end of each iteration, the best candidate is chosen based on its ensemble’s complexity-regularized train loss. New subnetworks are allowed to use any subnetwork weights within the previous iteration’s ensemble in order to improve upon them. If the complexity-regularized loss of the new ensemble, as defined in Equation (4), is less than that of the previous iteration’s ensemble, the AdaNet algorithm continues onto the next iteration.
AdaNet attempts to minimize the following loss function to learn the mixture weights \(w\) of each subnetwork \(h\) in the ensemble with differentiable convex non-increasing surrogate loss function \(\Phi\):
Equation (4):
\[F(w) = \frac{1}{m} \sum_{i=1}^{m} \Phi \left(\sum_{j=1}^{N}w_jh_j(x_i), y_i \right) + \sum_{j=1}^{N} \left(\lambda r(h_j) + \beta \right) |w_j|\]with \(\lambda >= 0\) and \(\beta >= 0\).
Parameters: - optimizer – String,
tf.train.Optimizer
object, or callable that creates the optimizer to use for training the ensemble weights. If left asNone
,tf.no_op()
is used instead. - mixture_weight_type – The
adanet.ensemble.MixtureWeightType
defining which mixture weight type to learn on top of the subnetworks’ logits. - mixture_weight_initializer –
The initializer for mixture_weights. When
None
, the default is different according tomixture_weight_type
:SCALAR
initializes to \(1/N\) where \(N\) is the number of subnetworks in the ensemble giving a uniform average.VECTOR
initializes each entry to \(1/N\) where \(N\) is the number of subnetworks in the ensemble giving a uniform average.MATRIX
usestf.zeros_initializer()
.
- warm_start_mixture_weights – Whether, at the beginning of an iteration, to
initialize the mixture weights of the subnetworks from the previous
ensemble to their learned value at the previous iteration, as opposed to
retraining them from scratch. Takes precedence over the value for
mixture_weight_initializer
for subnetworks from previous iterations. - model_dir – The model dir to use for warm-starting mixture weights and bias
at the logit layer. Ignored if
warm_start_mixture_weights
isFalse
. - adanet_lambda – Float multiplier \(\lambda\) for applying \(L1\) regularization to subnetworks’ mixture weights \(w\) in the ensemble proportional to their complexity. See Equation (4) in the AdaNet paper.
- adanet_beta – Float \(L1\) regularization multiplier \(\beta\) to apply equally to all subnetworks’ weights \(w\) in the ensemble regardless of their complexity. See Equation (4) in the AdaNet paper.
- use_bias – Whether to add a bias term to the ensemble’s logits.
- name – Optional name for the ensembler. Defaults to ‘complexity_regularized’.
Returns: An adanet.ensemble.ComplexityRegularizedEnsembler instance.
Raises: ValueError
– ifwarm_start_mixture_weights
isTrue
butmodel_dir
isNone
.
-
build_ensemble
(subnetworks, previous_ensemble_subnetworks, features, labels, logits_dimension, training, iteration_step, summary, previous_ensemble, previous_iteration_checkpoint=None)[source]¶ Builds an ensemble of subnetworks.
Accessing the global step via
tf.train.get_or_create_global_step()
ortf.train.get_global_step()
within this scope will return an incrementable iteration step since the beginning of the iteration.Parameters: - subnetworks – Ordered iterable of
adanet.subnetwork.Subnetwork
instances to ensemble. Must have at least one element. - previous_ensemble_subnetworks – Ordered iterable of
adanet.subnetwork.Subnetwork
instances present in previous ensemble to be used. The subnetworks from previous_ensemble not included in this list should be pruned. Can be set to None or empty. - features – Input
dict
oftf.Tensor
objects. - labels – Labels
tf.Tensor
or a dictionary of string label name totf.Tensor
(for multi-head). Can beNone
. - logits_dimension – Size of the last dimension of the logits
tf.Tensor
. Typically, logits have for shape [batch_size, logits_dimension]. - training – A python boolean indicating whether the graph is in training mode or prediction mode.
- iteration_step – Integer
tf.Tensor
representing the step since the beginning of the current iteration, as opposed to the global step. - summary – An
adanet.Summary
for scoping summaries to individual ensembles in Tensorboard. Usingtf.summary()
within this scope will use thisadanet.Summary
under the hood. - previous_ensemble – The best
adanet.Ensemble
from iteration t-1. The created subnetwork will extend the previous ensemble to form theadanet.Ensemble
at iteration t. - previous_iteration_checkpoint – The tf.train.Checkpoint object associated with the previous iteration.
Returns: An
adanet.ensemble.Ensemble
subclass instance.- subnetworks – Ordered iterable of
-
build_train_op
(ensemble, loss, var_list, labels, iteration_step, summary, previous_ensemble)[source]¶ Returns an op for training an ensemble.
Accessing the global step via
tf.train.get_or_create_global_step()
ortf.train.get_global_step()
within this scope will return an incrementable iteration step since the beginning of the iteration.Parameters: - ensemble – The
adanet.ensemble.Ensemble
subclass instance returned by this instance’sbuild_ensemble()
. - loss – A
tf.Tensor
containing the ensemble’s loss to minimize. - var_list – List of ensemble
tf.Variable
parameters to update as part of the training operation. - labels – Labels
tf.Tensor
or a dictionary of string label name totf.Tensor
(for multi-head). - iteration_step – Integer
tf.Tensor
representing the step since the beginning of the current iteration, as opposed to the global step. - summary – An
adanet.Summary
for scoping summaries to individual ensembles in Tensorboard. Usingtf.summary
within this scope will use thisadanet.Summary
under the hood. - previous_ensemble – The best
adanet.ensemble.Ensemble
from the previous iteration.
Returns: Either a train op or an
adanet.ensemble.TrainOpSpec
.- ensemble – The
-
name
¶ This ensembler’s unique string name.
- optimizer – String,
MeanEnsembler¶
-
class
adanet.ensemble.
MeanEnsembler
(name=None, add_mean_last_layer_predictions=False)[source]¶ Ensembler that takes the mean of logits returned by its subnetworks.
-
name
¶ Optional name for the ensembler. Defaults to ‘complexity_regularized’.
-
add_mean_last_layer_predictions
¶ Set to True to add mean of last_layer in subnetworks in estimator’s predictions and export outputs.
-
build_ensemble
(subnetworks, previous_ensemble_subnetworks, features, labels, logits_dimension, training, iteration_step, summary, previous_ensemble, previous_iteration_checkpoint)[source]¶ Builds an ensemble of subnetworks.
Accessing the global step via
tf.train.get_or_create_global_step()
ortf.train.get_global_step()
within this scope will return an incrementable iteration step since the beginning of the iteration.Parameters: - subnetworks – Ordered iterable of
adanet.subnetwork.Subnetwork
instances to ensemble. Must have at least one element. - previous_ensemble_subnetworks – Ordered iterable of
adanet.subnetwork.Subnetwork
instances present in previous ensemble to be used. The subnetworks from previous_ensemble not included in this list should be pruned. Can be set to None or empty. - features – Input
dict
oftf.Tensor
objects. - labels – Labels
tf.Tensor
or a dictionary of string label name totf.Tensor
(for multi-head). Can beNone
. - logits_dimension – Size of the last dimension of the logits
tf.Tensor
. Typically, logits have for shape [batch_size, logits_dimension]. - training – A python boolean indicating whether the graph is in training mode or prediction mode.
- iteration_step – Integer
tf.Tensor
representing the step since the beginning of the current iteration, as opposed to the global step. - summary – An
adanet.Summary
for scoping summaries to individual ensembles in Tensorboard. Usingtf.summary()
within this scope will use thisadanet.Summary
under the hood. - previous_ensemble – The best
adanet.Ensemble
from iteration t-1. The created subnetwork will extend the previous ensemble to form theadanet.Ensemble
at iteration t. - previous_iteration_checkpoint – The tf.train.Checkpoint object associated with the previous iteration.
Returns: An
adanet.ensemble.Ensemble
subclass instance.- subnetworks – Ordered iterable of
-
build_train_op
(ensemble, loss, var_list, labels, iteration_step, summary, previous_ensemble)[source]¶ Returns an op for training an ensemble.
Accessing the global step via
tf.train.get_or_create_global_step()
ortf.train.get_global_step()
within this scope will return an incrementable iteration step since the beginning of the iteration.Parameters: - ensemble – The
adanet.ensemble.Ensemble
subclass instance returned by this instance’sbuild_ensemble()
. - loss – A
tf.Tensor
containing the ensemble’s loss to minimize. - var_list – List of ensemble
tf.Variable
parameters to update as part of the training operation. - labels – Labels
tf.Tensor
or a dictionary of string label name totf.Tensor
(for multi-head). - iteration_step – Integer
tf.Tensor
representing the step since the beginning of the current iteration, as opposed to the global step. - summary – An
adanet.Summary
for scoping summaries to individual ensembles in Tensorboard. Usingtf.summary
within this scope will use thisadanet.Summary
under the hood. - previous_ensemble – The best
adanet.ensemble.Ensemble
from the previous iteration.
Returns: Either a train op or an
adanet.ensemble.TrainOpSpec
.- ensemble – The
-
name
This ensembler’s unique string name.
-
TrainOpSpec¶
-
class
adanet.ensemble.
TrainOpSpec
[source]¶ A data structure for specifying ensembler training operations.
Parameters: - train_op – Op for the training step.
- chief_hooks – Iterable of
tf.train.SessionRunHook
objects to run on the chief worker during training. - hooks – Iterable of
tf.train.SessionRunHook
objects to run on all workers during training.
Returns: An
adanet.ensemble.TrainOpSpec
object.
Strategies¶
Ensemble strategies for grouping subnetworks.
Strategy¶
-
class
adanet.ensemble.
Strategy
[source]¶ An abstract ensemble strategy.
-
generate_ensemble_candidates
(subnetwork_builders, previous_ensemble_subnetwork_builders)[source]¶ Generates ensemble candidates to search over this iteration.
Parameters: - subnetwork_builders – Candidate
adanet.subnetwork.Builder
instances for this iteration. - previous_ensemble_subnetwork_builders –
adanet.subnetwork.Builder
instances from the previous ensemble. Including only a subset of these in a returnedadanet.ensemble.Candidate
is equivalent to pruning the previous ensemble.
Returns: An iterable of
adanet.ensemble.Candidate
instances to train and consider this iteration.- subnetwork_builders – Candidate
-
SoloStrategy¶
-
class
adanet.ensemble.
SoloStrategy
[source]¶ Produces a model composed of a single subnetwork.
An ensemble of one.
This is effectively the same as pruning all previous ensemble subnetworks, and only adding one subnetwork candidate to the ensemble.
-
generate_ensemble_candidates
(subnetwork_builders, previous_ensemble_subnetwork_builders)[source]¶ Generates ensemble candidates to search over this iteration.
Parameters: - subnetwork_builders – Candidate
adanet.subnetwork.Builder
instances for this iteration. - previous_ensemble_subnetwork_builders –
adanet.subnetwork.Builder
instances from the previous ensemble. Including only a subset of these in a returnedadanet.ensemble.Candidate
is equivalent to pruning the previous ensemble.
Returns: An iterable of
adanet.ensemble.Candidate
instances to train and consider this iteration.- subnetwork_builders – Candidate
-
GrowStrategy¶
-
class
adanet.ensemble.
GrowStrategy
[source]¶ Greedily grows an ensemble, one subnetwork at a time.
-
generate_ensemble_candidates
(subnetwork_builders, previous_ensemble_subnetwork_builders)[source]¶ Generates ensemble candidates to search over this iteration.
Parameters: - subnetwork_builders – Candidate
adanet.subnetwork.Builder
instances for this iteration. - previous_ensemble_subnetwork_builders –
adanet.subnetwork.Builder
instances from the previous ensemble. Including only a subset of these in a returnedadanet.ensemble.Candidate
is equivalent to pruning the previous ensemble.
Returns: An iterable of
adanet.ensemble.Candidate
instances to train and consider this iteration.- subnetwork_builders – Candidate
-
AllStrategy¶
-
class
adanet.ensemble.
AllStrategy
[source]¶ Ensembles all subnetworks from the current iteration.
-
generate_ensemble_candidates
(subnetwork_builders, previous_ensemble_subnetwork_builders)[source]¶ Generates ensemble candidates to search over this iteration.
Parameters: - subnetwork_builders – Candidate
adanet.subnetwork.Builder
instances for this iteration. - previous_ensemble_subnetwork_builders –
adanet.subnetwork.Builder
instances from the previous ensemble. Including only a subset of these in a returnedadanet.ensemble.Candidate
is equivalent to pruning the previous ensemble.
Returns: An iterable of
adanet.ensemble.Candidate
instances to train and consider this iteration.- subnetwork_builders – Candidate
-
Candidate¶
-
class
adanet.ensemble.
Candidate
[source]¶ An ensemble candidate found during the search phase.
Parameters: - name – String name of this ensemble candidate.
- subnetwork_builders – Candidate
adanet.subnetwork.Builder
instances to include in the ensemble. - previous_ensemble_subnetwork_builders –
adanet.subnetwork.Builder
instances to include from the previous ensemble.