adanet.ensemble

Defines built-in ensemble methods and interfaces for custom ensembles.

Ensembles

Interfaces and containers for defining ensembles.

Ensemble

class adanet.ensemble.Ensemble[source]

An abstract ensemble of subnetworks.

logits

Ensemble logits tf.Tensor.

predictions

Optional dict of Ensemble predictions to be merged in EstimatorSpec.

These will be additional (over the default included by the head) predictions which will be included in the EstimatorSpec in predictions and export_outputs (wrapped as PredictOutput).

subnetworks

Returns an ordered Iterable of the ensemble’s subnetworks.

ComplexityRegularized

class adanet.ensemble.ComplexityRegularized[source]

An AdaNet ensemble where subnetworks are regularized by model complexity.

Hence an ensemble is a collection of subnetworks which forms a neural network through the weighted sum of their outputs:

\[F(x) = \sum_{i=1}^{N}w_ih_i(x) + b\]
Parameters:
  • weighted_subnetworks – List of adanet.ensemble.WeightedSubnetwork instances that form this ensemble. Ordered from first to most recent.
  • bias – Bias term tf.Tensor or dict of string to bias term tf.Tensor (for multi-head) for the ensemble’s logits.
  • logits – Logits tf.Tensor or dict of string to logits tf.Tensor (for multi-head). The result of the function f as defined in Section 5.1 which is the sum of the logits of all adanet.WeightedSubnetwork instances in ensemble.
  • subnetworks – List of adanet.subnetwork.Subnetwork instances that form this ensemble. This is kept together with weighted_subnetworks for legacy reasons.
  • complexity_regularization – Regularization to be added in the Adanet loss.
Returns:

An adanet.ensemble.Weighted instance.

MeanEnsemble

class adanet.ensemble.MeanEnsemble[source]

Mean ensemble.

logits

Logits tf.Tensor or dict of string to logits tf.Tensor (for multi-head).

subnetworks

List of adanet.subnetwork.Subnetwork instances that form this ensemble.

predictions

Optional dict mapping prediction keys to Tensors. MeanEnsembler can export mean_last_layer if the subnetworks have the last_layer of the same dimension.

MixtureWeightType

class adanet.ensemble.MixtureWeightType[source]

Mixture weight types available for learning subnetwork contributions.

The following mixture weight types are defined:

  • SCALAR: Produces a rank 0 Tensor mixture weight.
  • VECTOR: Produces a rank 1 Tensor mixture weight.
  • MATRIX: Produces a rank 2 Tensor mixture weight.

WeightedSubnetwork

class adanet.ensemble.WeightedSubnetwork[source]

An AdaNet weighted subnetwork.

A weighted subnetwork is a weight applied to a subnetwork’s last layer or logits (depending on the mixture weights type).

Parameters:
  • name – String name of subnetwork as defined by its adanet.subnetwork.Builder.
  • iteration_number – Integer iteration when the subnetwork was created.
  • weight – The weight tf.Tensor or dict of string to weight tf.Tensor (for multi-head) to apply to this subnetwork. The AdaNet paper refers to this weight as \(w\) in Equations (4), (5), and (6).
  • logits – The output tf.Tensor or dict of string to weight tf.Tensor (for multi-head) after the matrix multiplication of weight and the subnetwork’s last_layer. The output’s shape is [batch_size, logits_dimension]. It is equivalent to a linear logits layer in a neural network.
  • subnetwork – The adanet.subnetwork.Subnetwork to weight.
Returns:

An adanet.ensemble.WeightedSubnetwork object.

Ensemblers

Ensemble learning definitions.

Ensembler

class adanet.ensemble.Ensembler[source]

An abstract ensembler.

build_ensemble(subnetworks, previous_ensemble_subnetworks, features, labels, logits_dimension, training, iteration_step, summary, previous_ensemble, previous_iteration_checkpoint)[source]

Builds an ensemble of subnetworks.

Accessing the global step via tf.train.get_or_create_global_step() or tf.train.get_global_step() within this scope will return an incrementable iteration step since the beginning of the iteration.

Parameters:
  • subnetworks – Ordered iterable of adanet.subnetwork.Subnetwork instances to ensemble. Must have at least one element.
  • previous_ensemble_subnetworks – Ordered iterable of adanet.subnetwork.Subnetwork instances present in previous ensemble to be used. The subnetworks from previous_ensemble not included in this list should be pruned. Can be set to None or empty.
  • features – Input dict of tf.Tensor objects.
  • labels – Labels tf.Tensor or a dictionary of string label name to tf.Tensor (for multi-head). Can be None.
  • logits_dimension – Size of the last dimension of the logits tf.Tensor. Typically, logits have for shape [batch_size, logits_dimension].
  • training – A python boolean indicating whether the graph is in training mode or prediction mode.
  • iteration_step – Integer tf.Tensor representing the step since the beginning of the current iteration, as opposed to the global step.
  • summary – An adanet.Summary for scoping summaries to individual ensembles in Tensorboard. Using tf.summary() within this scope will use this adanet.Summary under the hood.
  • previous_ensemble – The best adanet.Ensemble from iteration t-1. The created subnetwork will extend the previous ensemble to form the adanet.Ensemble at iteration t.
  • previous_iteration_checkpoint – The tf.train.Checkpoint object associated with the previous iteration.
Returns:

An adanet.ensemble.Ensemble subclass instance.

build_train_op(ensemble, loss, var_list, labels, iteration_step, summary, previous_ensemble)[source]

Returns an op for training an ensemble.

Accessing the global step via tf.train.get_or_create_global_step() or tf.train.get_global_step() within this scope will return an incrementable iteration step since the beginning of the iteration.

Parameters:
  • ensemble – The adanet.ensemble.Ensemble subclass instance returned by this instance’s build_ensemble().
  • loss – A tf.Tensor containing the ensemble’s loss to minimize.
  • var_list – List of ensemble tf.Variable parameters to update as part of the training operation.
  • labels – Labels tf.Tensor or a dictionary of string label name to tf.Tensor (for multi-head).
  • iteration_step – Integer tf.Tensor representing the step since the beginning of the current iteration, as opposed to the global step.
  • summary – An adanet.Summary for scoping summaries to individual ensembles in Tensorboard. Using tf.summary within this scope will use this adanet.Summary under the hood.
  • previous_ensemble – The best adanet.ensemble.Ensemble from the previous iteration.
Returns:

Either a train op or an adanet.ensemble.TrainOpSpec.

name

This ensembler’s unique string name.

ComplexityRegularizedEnsembler

class adanet.ensemble.ComplexityRegularizedEnsembler(optimizer=None, mixture_weight_type='scalar', mixture_weight_initializer=None, warm_start_mixture_weights=False, model_dir=None, adanet_lambda=0.0, adanet_beta=0.0, use_bias=False, name=None)[source]

The AdaNet algorithm implemented as an adanet.ensemble.Ensembler.

The AdaNet algorithm was introduced in the [Cortes et al. ICML 2017] paper: https://arxiv.org/abs/1607.01097.

The AdaNet algorithm uses a weak learning algorithm to iteratively generate a set of candidate subnetworks that attempt to minimize the loss function defined in Equation (4) as part of an ensemble. At the end of each iteration, the best candidate is chosen based on its ensemble’s complexity-regularized train loss. New subnetworks are allowed to use any subnetwork weights within the previous iteration’s ensemble in order to improve upon them. If the complexity-regularized loss of the new ensemble, as defined in Equation (4), is less than that of the previous iteration’s ensemble, the AdaNet algorithm continues onto the next iteration.

AdaNet attempts to minimize the following loss function to learn the mixture weights \(w\) of each subnetwork \(h\) in the ensemble with differentiable convex non-increasing surrogate loss function \(\Phi\):

Equation (4):

\[F(w) = \frac{1}{m} \sum_{i=1}^{m} \Phi \left(\sum_{j=1}^{N}w_jh_j(x_i), y_i \right) + \sum_{j=1}^{N} \left(\lambda r(h_j) + \beta \right) |w_j|\]

with \(\lambda >= 0\) and \(\beta >= 0\).

Parameters:
  • optimizer – String, tf.train.Optimizer object, or callable that creates the optimizer to use for training the ensemble weights. If left as None, tf.no_op() is used instead.
  • mixture_weight_type – The adanet.ensemble.MixtureWeightType defining which mixture weight type to learn on top of the subnetworks’ logits.
  • mixture_weight_initializer

    The initializer for mixture_weights. When None, the default is different according to mixture_weight_type:

    • SCALAR initializes to \(1/N\) where \(N\) is the number of subnetworks in the ensemble giving a uniform average.
    • VECTOR initializes each entry to \(1/N\) where \(N\) is the number of subnetworks in the ensemble giving a uniform average.
    • MATRIX uses tf.zeros_initializer().
  • warm_start_mixture_weights – Whether, at the beginning of an iteration, to initialize the mixture weights of the subnetworks from the previous ensemble to their learned value at the previous iteration, as opposed to retraining them from scratch. Takes precedence over the value for mixture_weight_initializer for subnetworks from previous iterations.
  • model_dir – The model dir to use for warm-starting mixture weights and bias at the logit layer. Ignored if warm_start_mixture_weights is False.
  • adanet_lambda – Float multiplier \(\lambda\) for applying \(L1\) regularization to subnetworks’ mixture weights \(w\) in the ensemble proportional to their complexity. See Equation (4) in the AdaNet paper.
  • adanet_beta – Float \(L1\) regularization multiplier \(\beta\) to apply equally to all subnetworks’ weights \(w\) in the ensemble regardless of their complexity. See Equation (4) in the AdaNet paper.
  • use_bias – Whether to add a bias term to the ensemble’s logits.
  • name – Optional name for the ensembler. Defaults to ‘complexity_regularized’.
Returns:

An adanet.ensemble.ComplexityRegularizedEnsembler instance.

Raises:
  • ValueError – if warm_start_mixture_weights is True but
  • model_dir is None.
build_ensemble(subnetworks, previous_ensemble_subnetworks, features, labels, logits_dimension, training, iteration_step, summary, previous_ensemble, previous_iteration_checkpoint=None)[source]

Builds an ensemble of subnetworks.

Accessing the global step via tf.train.get_or_create_global_step() or tf.train.get_global_step() within this scope will return an incrementable iteration step since the beginning of the iteration.

Parameters:
  • subnetworks – Ordered iterable of adanet.subnetwork.Subnetwork instances to ensemble. Must have at least one element.
  • previous_ensemble_subnetworks – Ordered iterable of adanet.subnetwork.Subnetwork instances present in previous ensemble to be used. The subnetworks from previous_ensemble not included in this list should be pruned. Can be set to None or empty.
  • features – Input dict of tf.Tensor objects.
  • labels – Labels tf.Tensor or a dictionary of string label name to tf.Tensor (for multi-head). Can be None.
  • logits_dimension – Size of the last dimension of the logits tf.Tensor. Typically, logits have for shape [batch_size, logits_dimension].
  • training – A python boolean indicating whether the graph is in training mode or prediction mode.
  • iteration_step – Integer tf.Tensor representing the step since the beginning of the current iteration, as opposed to the global step.
  • summary – An adanet.Summary for scoping summaries to individual ensembles in Tensorboard. Using tf.summary() within this scope will use this adanet.Summary under the hood.
  • previous_ensemble – The best adanet.Ensemble from iteration t-1. The created subnetwork will extend the previous ensemble to form the adanet.Ensemble at iteration t.
  • previous_iteration_checkpoint – The tf.train.Checkpoint object associated with the previous iteration.
Returns:

An adanet.ensemble.Ensemble subclass instance.

build_train_op(ensemble, loss, var_list, labels, iteration_step, summary, previous_ensemble)[source]

Returns an op for training an ensemble.

Accessing the global step via tf.train.get_or_create_global_step() or tf.train.get_global_step() within this scope will return an incrementable iteration step since the beginning of the iteration.

Parameters:
  • ensemble – The adanet.ensemble.Ensemble subclass instance returned by this instance’s build_ensemble().
  • loss – A tf.Tensor containing the ensemble’s loss to minimize.
  • var_list – List of ensemble tf.Variable parameters to update as part of the training operation.
  • labels – Labels tf.Tensor or a dictionary of string label name to tf.Tensor (for multi-head).
  • iteration_step – Integer tf.Tensor representing the step since the beginning of the current iteration, as opposed to the global step.
  • summary – An adanet.Summary for scoping summaries to individual ensembles in Tensorboard. Using tf.summary within this scope will use this adanet.Summary under the hood.
  • previous_ensemble – The best adanet.ensemble.Ensemble from the previous iteration.
Returns:

Either a train op or an adanet.ensemble.TrainOpSpec.

name

This ensembler’s unique string name.

MeanEnsembler

class adanet.ensemble.MeanEnsembler(name=None, add_mean_last_layer_predictions=False)[source]

Ensembler that takes the mean of logits returned by its subnetworks.

name

Optional name for the ensembler. Defaults to ‘complexity_regularized’.

add_mean_last_layer_predictions

Set to True to add mean of last_layer in subnetworks in estimator’s predictions and export outputs.

build_ensemble(subnetworks, previous_ensemble_subnetworks, features, labels, logits_dimension, training, iteration_step, summary, previous_ensemble, previous_iteration_checkpoint)[source]

Builds an ensemble of subnetworks.

Accessing the global step via tf.train.get_or_create_global_step() or tf.train.get_global_step() within this scope will return an incrementable iteration step since the beginning of the iteration.

Parameters:
  • subnetworks – Ordered iterable of adanet.subnetwork.Subnetwork instances to ensemble. Must have at least one element.
  • previous_ensemble_subnetworks – Ordered iterable of adanet.subnetwork.Subnetwork instances present in previous ensemble to be used. The subnetworks from previous_ensemble not included in this list should be pruned. Can be set to None or empty.
  • features – Input dict of tf.Tensor objects.
  • labels – Labels tf.Tensor or a dictionary of string label name to tf.Tensor (for multi-head). Can be None.
  • logits_dimension – Size of the last dimension of the logits tf.Tensor. Typically, logits have for shape [batch_size, logits_dimension].
  • training – A python boolean indicating whether the graph is in training mode or prediction mode.
  • iteration_step – Integer tf.Tensor representing the step since the beginning of the current iteration, as opposed to the global step.
  • summary – An adanet.Summary for scoping summaries to individual ensembles in Tensorboard. Using tf.summary() within this scope will use this adanet.Summary under the hood.
  • previous_ensemble – The best adanet.Ensemble from iteration t-1. The created subnetwork will extend the previous ensemble to form the adanet.Ensemble at iteration t.
  • previous_iteration_checkpoint – The tf.train.Checkpoint object associated with the previous iteration.
Returns:

An adanet.ensemble.Ensemble subclass instance.

build_train_op(ensemble, loss, var_list, labels, iteration_step, summary, previous_ensemble)[source]

Returns an op for training an ensemble.

Accessing the global step via tf.train.get_or_create_global_step() or tf.train.get_global_step() within this scope will return an incrementable iteration step since the beginning of the iteration.

Parameters:
  • ensemble – The adanet.ensemble.Ensemble subclass instance returned by this instance’s build_ensemble().
  • loss – A tf.Tensor containing the ensemble’s loss to minimize.
  • var_list – List of ensemble tf.Variable parameters to update as part of the training operation.
  • labels – Labels tf.Tensor or a dictionary of string label name to tf.Tensor (for multi-head).
  • iteration_step – Integer tf.Tensor representing the step since the beginning of the current iteration, as opposed to the global step.
  • summary – An adanet.Summary for scoping summaries to individual ensembles in Tensorboard. Using tf.summary within this scope will use this adanet.Summary under the hood.
  • previous_ensemble – The best adanet.ensemble.Ensemble from the previous iteration.
Returns:

Either a train op or an adanet.ensemble.TrainOpSpec.

name

This ensembler’s unique string name.

TrainOpSpec

class adanet.ensemble.TrainOpSpec[source]

A data structure for specifying ensembler training operations.

Parameters:
  • train_op – Op for the training step.
  • chief_hooks – Iterable of tf.train.SessionRunHook objects to run on the chief worker during training.
  • hooks – Iterable of tf.train.SessionRunHook objects to run on all workers during training.
Returns:

An adanet.ensemble.TrainOpSpec object.

Strategies

Ensemble strategies for grouping subnetworks.

Strategy

class adanet.ensemble.Strategy[source]

An abstract ensemble strategy.

generate_ensemble_candidates(subnetwork_builders, previous_ensemble_subnetwork_builders)[source]

Generates ensemble candidates to search over this iteration.

Parameters:
Returns:

An iterable of adanet.ensemble.Candidate instances to train and consider this iteration.

SoloStrategy

class adanet.ensemble.SoloStrategy[source]

Produces a model composed of a single subnetwork.

An ensemble of one.

This is effectively the same as pruning all previous ensemble subnetworks, and only adding one subnetwork candidate to the ensemble.

generate_ensemble_candidates(subnetwork_builders, previous_ensemble_subnetwork_builders)[source]

Generates ensemble candidates to search over this iteration.

Parameters:
Returns:

An iterable of adanet.ensemble.Candidate instances to train and consider this iteration.

GrowStrategy

class adanet.ensemble.GrowStrategy[source]

Greedily grows an ensemble, one subnetwork at a time.

generate_ensemble_candidates(subnetwork_builders, previous_ensemble_subnetwork_builders)[source]

Generates ensemble candidates to search over this iteration.

Parameters:
Returns:

An iterable of adanet.ensemble.Candidate instances to train and consider this iteration.

AllStrategy

class adanet.ensemble.AllStrategy[source]

Ensembles all subnetworks from the current iteration.

generate_ensemble_candidates(subnetwork_builders, previous_ensemble_subnetwork_builders)[source]

Generates ensemble candidates to search over this iteration.

Parameters:
Returns:

An iterable of adanet.ensemble.Candidate instances to train and consider this iteration.

Candidate

class adanet.ensemble.Candidate[source]

An ensemble candidate found during the search phase.

Parameters:
  • name – String name of this ensemble candidate.
  • subnetwork_builders – Candidate adanet.subnetwork.Builder instances to include in the ensemble.
  • previous_ensemble_subnetwork_buildersadanet.subnetwork.Builder instances to include from the previous ensemble.