adanet.subnetwork

Low-level APIs for defining custom subnetworks and search spaces.

Generators

Interfaces and containers for defining subnetworks, search spaces, and search algorithms.

Subnetwork

class adanet.subnetwork.Subnetwork[source]

An AdaNet subnetwork.

In the AdaNet paper, an adanet.subnetwork.Subnetwork is are called a subnetwork, and indicated by h. A collection of weighted subnetworks form an AdaNet ensemble.

Parameters:
  • last_layer – tf.Tensor output or dict of string to tf.Tensor outputs (for multi-head) of the last layer of the subnetwork, i.e the layer before the logits layer. When the mixture weight type is MATRIX, the AdaNet algorithm takes care of computing ensemble mixture weights matrices (one per subnetwork) that multiply the various last layers of the ensemble’s subnetworks, and regularize them using their subnetwork’s complexity. This field is represented by h in the AdaNet paper.
  • logits – tf.Tensor logits or dict of string to tf.Tensor logits (for multi-head) for training the subnetwork. These logits are not used in the ensemble’s outputs if the mixture weight type is MATRIX, instead AdaNet learns its own logits (mixture weights) from the subnetwork’s last_layers with complexity regularization. The logits are used in the ensemble only when the mixture weights type is SCALAR or VECTOR. Even though the logits are not used in the ensemble in some cases, they should always be supplied as adanet uses the logits to train the subnetworks.
  • complexity – A scalar tf.Tensor representing the complexity of the subnetwork’s architecture. It is used for choosing the best subnetwork at each iteration, and for regularizing the weighted outputs of more complex subnetworks.
  • persisted_tensors – DEPRECATED. See shared. Optional nested dictionary of string to tf.Tensor to persist across iterations. At the end of an iteration, the tf.Tensor instances will be available to subnetworks in the next iterations, whereas others that are not part of the Subnetwork will be pruned. This allows later adanet.subnetwork.Subnetwork instances to dynamically build upon arbitrary tf.Tensors from previous adanet.subnetwork.Subnetwork instances.
  • shared – Optional Python object(s), primitive(s), or function(s) to share with subnetworks within the same iteration or in future iterations.
  • local_init_ops – Iterable of tf.Operation objects to run to initialize local variables.
Returns:

A validated adanet.subnetwork.Subnetwork object.
Raises:
  • ValueError – If last_layer is None.
  • ValueError – If logits is None.
  • ValueError – If logits is a dict but last_layer is not.
  • ValueError – If last_layer is a dict but logits is not.
  • ValueError – If complexity is None.
  • ValueError – If persisted_tensors is present but not a dictionary.
  • ValueError – If persisted_tensors contains an empty nested dictionary.
deprecation = <module 'tensorflow.python.util.deprecation' from '/home/docs/checkouts/readthedocs.org/user_builds/adanet/envs/latest/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py'>

TrainOpSpec

class adanet.subnetwork.TrainOpSpec[source]

A data structure for specifying training operations.

Parameters:
  • train_op – Op for the training step.
  • chief_hooks – Iterable of tf.train.SessionRunHook objects to run on the chief worker during training.
  • hooks – Iterable of tf.train.SessionRunHook objects to run on all workers during training.
Returns:

A adanet.subnetwork.TrainOpSpec object.

Builder

class adanet.subnetwork.Builder[source]

Bases: object

Interface for a subnetwork builder.

Given features, labels, and the best ensemble of subnetworks at iteration t-1, a Builder creates a Subnetwork to add to a candidate ensemble at iteration t. These candidate ensembles are evaluated against one another at the end of the iteration, and the best one is selected based on its complexity-regularized loss.

build_subnetwork(features, labels, logits_dimension, training, iteration_step, summary, previous_ensemble=None)[source]

Returns the candidate Subnetwork to add to the ensemble.

This method will be called only once before build_subnetwork_train_op(). This method should construct the candidate subnetwork’s graph operations and variables.

Accessing the global step via tf.train.get_or_create_global_step() or tf.train.get_global_step() within this scope will return an incrementable iteration step since the beginning of the iteration.

Parameters:
  • features – Input dict of tf.Tensor objects.
  • labels – Labels tf.Tensor or a dictionary of string label name to tf.Tensor (for multi-head). Can be None.
  • logits_dimension – Size of the last dimension of the logits tf.Tensor. Typically, logits have for shape [batch_size, logits_dimension].
  • training – A python boolean indicating whether the graph is in training mode or prediction mode.
  • iteration_step – Integer tf.Tensor representing the step since the beginning of the current iteration, as opposed to the global step.
  • summary – An adanet.Summary for scoping summaries to individual subnetworks in Tensorboard. Using tf.summary() within this scope will use this adanet.Summary under the hood.
  • previous_ensemble – The best adanet.Ensemble from iteration t-1. The created subnetwork will extend the previous ensemble to form the adanet.Ensemble at iteration t.
Returns:

An adanet.subnetwork.Subnetwork instance.
build_subnetwork_report()[source]

Returns a subnetwork.Report to materialize and record.

This method will be called once after build_subnetwork(). Do NOT depend on variables created in build_subnetwork_train_op(), because they are not called before build_subnetwork_report() is called.

If it returns None, AdaNet records the name and standard eval metrics.

build_subnetwork_train_op(subnetwork, loss, var_list, labels, iteration_step, summary, previous_ensemble)[source]

Returns an op for training a new subnetwork.

This method will be called once after build_subnetwork().

Accessing the global step via tf.train.get_or_create_global_step() or tf.train.get_global_step() within this scope will return an incrementable iteration step since the beginning of the iteration.

Parameters:
  • subnetwork – Newest subnetwork, that is not part of the previous_ensemble.
  • loss – A tf.Tensor containing the subnetwork’s loss to minimize.
  • var_list – List of subnetwork tf.Variable parameters to update as part of the training operation.
  • labels – Labels tf.Tensor or a dictionary of string label name to tf.Tensor (for multi-head).
  • iteration_step – Integer tf.Tensor representing the step since the beginning of the current iteration, as opposed to the global step.
  • summary – An adanet.Summary for scoping summaries to individual subnetworks in Tensorboard. Using tf.summary within this scope will use this adanet.Summary under the hood.
  • previous_ensemble – The best Ensemble from iteration t-1. The created subnetwork will extend the previous ensemble to form the Ensemble at iteration t. Is None for iteration 0.
Returns:

Either a train op or an adanet.subnetwork.TrainOpSpec.
name

Returns the unique name of this subnetwork within an iteration.

Returns:String name of this subnetwork.

Generator

class adanet.subnetwork.Generator[source]

Bases: object

Interface for a candidate subnetwork generator.

Given the ensemble of subnetworks at iteration t-1, this object is responsible for generating the set of candidate subnetworks for iteration t that minimize the objective as part of an ensemble.

generate_candidates(previous_ensemble, iteration_number, previous_ensemble_reports, all_reports, config)[source]

Generates adanet.subnetwork.Builder instances for an iteration.

NOTE: Every call to generate_candidates() must be deterministic for the given arguments.

Parameters:
  • previous_ensemble – The best adanet.Ensemble from iteration t-1. DEPRECATED. We are transitioning away from the use of previous_ensemble in generate_candidates. New Generators should not use previous_ensemble in their implementation of generate_candidates – please only use iteration_number, previous_ensemble_reports and all_reports.
  • iteration_number – Python integer AdaNet iteration t, starting from 0.
  • previous_ensemble_reports – List of adanet.subnetwork.MaterializedReport instances corresponding to the Builders composing adanet.Ensemble from iteration t-1. The first element in the list corresponds to the Builder added in the first iteration. If a adanet.subnetwork.MaterializedReport is not supplied to the estimator, previous_ensemble_report is None.
  • all_reports – List of adanet.subnetwork.MaterializedReport instances. If an adanet.subnetwork.ReportMaterializer is not supplied to the estimator, all_reports is None. If adanet.subnetwork.ReportMaterializer is supplied to the estimator and t=0, all_reports is an empty List. Otherwise, all_reports is a sequence of Lists. Each element of the sequence is a List containing all the adanet.subnetwork.MaterializedReport instances in an AdaNet iteration, starting from iteration 0, and ending at iteration t-1.
  • config – The current tf.estimator.RunConfig object to configure the runtime settings.
Returns:

A list of adanet.subnetwork.Builder instances.

Reports

Containers for metadata about trained subnetworks.

Report

class adanet.subnetwork.Report[source]

A container for data to be collected about a Subnetwork.

Parameters:
  • hparams – A dict mapping strings to python strings, ints, bools, or floats. It is meant to contain the constants that define the adanet.subnetwork.Builder, such as dropout, number of layers, or initial learning rate.
  • attributes – A dict mapping strings to rank 0 Tensors of dtype string, int32, or float32. It is meant to contain properties that may or may not change over the course of training the adanet.subnetwork.Subnetwork, such as the number of parameters, the Lipschitz constant, the \(L2\) norm of the weights, or learning rate at materialization time.
  • metrics – Dict of metric results keyed by name. The values of the dict are the results of calling a metric function, namely a (metric_tensor, update_op) tuple. metric_tensor should be evaluated without any impact on state (typically is a pure computation results based on variables.). For example, it should not trigger the update_op or requires any input fetching. This is meant to contain metrics of interest, such as the training loss, complexity regularized loss, or standard deviation of the last layer outputs.
Returns:

A validated adanet.subnetwork.Report object.
Raises:

ValueError – If validation fails.

MaterializedReport

class adanet.subnetwork.MaterializedReport[source]

Data collected about a adanet.subnetwork.Subnetwork.

Parameters:
  • iteration_number – A python integer for the AdaNet iteration number, starting from 0.
  • name – A string, which is either the name of the corresponding Builder, or “previous_ensemble” if it refers to the previous_ensemble.
  • hparams – A dict mapping strings to python strings, ints, or floats. These are constants passed from the author of the adanet.subnetwork.Builder that was used to construct this adanet.subnetwork.Subnetwork. It is meant to contain the arguments that defined the adanet.subnetwork.Builder, such as dropout, number of layers, or initial learning rate.
  • attributes – A dict mapping strings to python strings, ints, bools, or floats. These are python primitives that come from materialized Tensors; these Tensors were defined by the author of the adanet.subnetwork.Builder that was used to construct this adanet.subnetwork.Subnetwork. It is meant to contain properties that may or may not change over the course of training the adanet.subnetwork.Subnetwork, such as the number of parameters, the Lipschitz constant, or the \(L2\) norm of the weights.
  • metrics – A dict mapping strings to python strings, ints, or floats. These are python primitives that come from metrics that were evaluated on the trained adanet.subnetwork.Subnetwork over some dataset; these metrics were defined by the author of the adanet.subnetwork.Builder that was used to construct this adanet.subnetwork.Subnetwork. It is meant to contain performance metrics or measures that could predict generalization, such as the training loss, complexity regularized loss, or standard deviation of the last layer outputs.
  • included_in_final_ensemble – A boolean denoting whether the associated adanet.subnetwork.Subnetwork was included in the ensemble at the end of the AdaNet iteration.
Returns:

An adanet.subnetwork.MaterializedReport object.