adanet.subnetwork¶
Low-level APIs for defining custom subnetworks and search spaces.
Generators¶
Interfaces and containers for defining subnetworks, search spaces, and search algorithms.
Subnetwork¶
-
class
adanet.subnetwork.
Subnetwork
[source]¶ An AdaNet subnetwork.
In the AdaNet paper, an
adanet.subnetwork.Subnetwork
is are called a subnetwork, and indicated by h. A collection of weighted subnetworks form an AdaNet ensemble.Parameters: - last_layer –
tf.Tensor
output or dict of string totf.Tensor
outputs (for multi-head) of the last layer of the subnetwork, i.e the layer before the logits layer. When the mixture weight type isMATRIX
, the AdaNet algorithm takes care of computing ensemble mixture weights matrices (one per subnetwork) that multiply the various last layers of the ensemble’s subnetworks, and regularize them using their subnetwork’s complexity. This field is represented by h in the AdaNet paper. - logits –
tf.Tensor
logits or dict of string totf.Tensor
logits (for multi-head) for training the subnetwork. These logits are not used in the ensemble’s outputs if the mixture weight type isMATRIX
, instead AdaNet learns its own logits (mixture weights) from the subnetwork’s last_layers with complexity regularization. The logits are used in the ensemble only when the mixture weights type isSCALAR
orVECTOR
. Even though the logits are not used in the ensemble in some cases, they should always be supplied as adanet uses the logits to train the subnetworks. - complexity – A scalar
tf.Tensor
representing the complexity of the subnetwork’s architecture. It is used for choosing the best subnetwork at each iteration, and for regularizing the weighted outputs of more complex subnetworks. - persisted_tensors – DEPRECATED. See shared. Optional nested dictionary of
string to
tf.Tensor
to persist across iterations. At the end of an iteration, thetf.Tensor
instances will be available to subnetworks in the next iterations, whereas others that are not part of the Subnetwork will be pruned. This allows lateradanet.subnetwork.Subnetwork
instances to dynamically build upon arbitrarytf.Tensors
from previousadanet.subnetwork.Subnetwork
instances. - shared – Optional Python object(s), primitive(s), or function(s) to share with subnetworks within the same iteration or in future iterations.
- local_init_ops – Iterable of
tf.Operation
objects to run to initialize local variables.
Returns: A validated
adanet.subnetwork.Subnetwork
object.Raises: ValueError
– If last_layer is None.ValueError
– If logits is None.ValueError
– If logits is a dict but last_layer is not.ValueError
– If last_layer is a dict but logits is not.ValueError
– If complexity is None.ValueError
– If persisted_tensors is present but not a dictionary.ValueError
– If persisted_tensors contains an empty nested dictionary.
-
deprecation
= <module 'tensorflow.python.util.deprecation' from '/home/docs/checkouts/readthedocs.org/user_builds/adanet/envs/latest/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py'>¶
- last_layer –
TrainOpSpec¶
-
class
adanet.subnetwork.
TrainOpSpec
[source]¶ A data structure for specifying training operations.
Parameters: - train_op – Op for the training step.
- chief_hooks – Iterable of
tf.train.SessionRunHook
objects to run on the chief worker during training. - hooks – Iterable of
tf.train.SessionRunHook
objects to run on all workers during training.
Returns: A
adanet.subnetwork.TrainOpSpec
object.
Builder¶
-
class
adanet.subnetwork.
Builder
[source]¶ Bases:
object
Interface for a subnetwork builder.
Given features, labels, and the best ensemble of subnetworks at iteration t-1, a Builder creates a Subnetwork to add to a candidate ensemble at iteration t. These candidate ensembles are evaluated against one another at the end of the iteration, and the best one is selected based on its complexity-regularized loss.
-
build_subnetwork
(features, labels, logits_dimension, training, iteration_step, summary, previous_ensemble=None)[source]¶ Returns the candidate Subnetwork to add to the ensemble.
This method will be called only once before
build_subnetwork_train_op()
. This method should construct the candidate subnetwork’s graph operations and variables.Accessing the global step via
tf.train.get_or_create_global_step()
ortf.train.get_global_step()
within this scope will return an incrementable iteration step since the beginning of the iteration.Parameters: - features – Input dict of
tf.Tensor
objects. - labels – Labels
tf.Tensor
or a dictionary of string label name totf.Tensor
(for multi-head). Can be None. - logits_dimension – Size of the last dimension of the logits
tf.Tensor
. Typically, logits have for shape [batch_size, logits_dimension]. - training – A python boolean indicating whether the graph is in training mode or prediction mode.
- iteration_step – Integer
tf.Tensor
representing the step since the beginning of the current iteration, as opposed to the global step. - summary – An
adanet.Summary
for scoping summaries to individual subnetworks in Tensorboard. Usingtf.summary()
within this scope will use thisadanet.Summary
under the hood. - previous_ensemble – The best
adanet.Ensemble
from iteration t-1. The created subnetwork will extend the previous ensemble to form theadanet.Ensemble
at iteration t.
Returns: An
adanet.subnetwork.Subnetwork
instance.- features – Input dict of
-
build_subnetwork_report
()[source]¶ Returns a subnetwork.Report to materialize and record.
This method will be called once after
build_subnetwork()
. Do NOT depend on variables created inbuild_subnetwork_train_op()
, because they are not called beforebuild_subnetwork_report()
is called.If it returns None, AdaNet records the name and standard eval metrics.
-
build_subnetwork_train_op
(subnetwork, loss, var_list, labels, iteration_step, summary, previous_ensemble)[source]¶ Returns an op for training a new subnetwork.
This method will be called once after
build_subnetwork()
.Accessing the global step via
tf.train.get_or_create_global_step()
ortf.train.get_global_step()
within this scope will return an incrementable iteration step since the beginning of the iteration.Parameters: - subnetwork – Newest subnetwork, that is not part of the previous_ensemble.
- loss – A
tf.Tensor
containing the subnetwork’s loss to minimize. - var_list – List of subnetwork
tf.Variable
parameters to update as part of the training operation. - labels – Labels
tf.Tensor
or a dictionary of string label name totf.Tensor
(for multi-head). - iteration_step – Integer
tf.Tensor
representing the step since the beginning of the current iteration, as opposed to the global step. - summary – An
adanet.Summary
for scoping summaries to individual subnetworks in Tensorboard. Using tf.summary within this scope will use thisadanet.Summary
under the hood. - previous_ensemble – The best Ensemble from iteration t-1. The created subnetwork will extend the previous ensemble to form the Ensemble at iteration t. Is None for iteration 0.
Returns: Either a train op or an
adanet.subnetwork.TrainOpSpec
.
-
name
¶ Returns the unique name of this subnetwork within an iteration.
Returns: String name of this subnetwork.
-
Generator¶
-
class
adanet.subnetwork.
Generator
[source]¶ Bases:
object
Interface for a candidate subnetwork generator.
Given the ensemble of subnetworks at iteration t-1, this object is responsible for generating the set of candidate subnetworks for iteration t that minimize the objective as part of an ensemble.
-
generate_candidates
(previous_ensemble, iteration_number, previous_ensemble_reports, all_reports, config)[source]¶ Generates
adanet.subnetwork.Builder
instances for an iteration.NOTE: Every call to
generate_candidates()
must be deterministic for the given arguments.Parameters: - previous_ensemble – The best
adanet.Ensemble
from iteration t-1. DEPRECATED. We are transitioning away from the use of previous_ensemble in generate_candidates. New Generators should not use previous_ensemble in their implementation of generate_candidates – please only use iteration_number, previous_ensemble_reports and all_reports. - iteration_number – Python integer AdaNet iteration t, starting from 0.
- previous_ensemble_reports – List of
adanet.subnetwork.MaterializedReport
instances corresponding to the Builders composingadanet.Ensemble
from iteration t-1. The first element in the list corresponds to the Builder added in the first iteration. If aadanet.subnetwork.MaterializedReport
is not supplied to the estimator, previous_ensemble_report is None. - all_reports – List of
adanet.subnetwork.MaterializedReport
instances. If anadanet.subnetwork.ReportMaterializer
is not supplied to the estimator, all_reports is None. Ifadanet.subnetwork.ReportMaterializer
is supplied to the estimator and t=0, all_reports is an empty List. Otherwise, all_reports is a sequence of Lists. Each element of the sequence is a List containing all theadanet.subnetwork.MaterializedReport
instances in an AdaNet iteration, starting from iteration 0, and ending at iteration t-1. - config – The current
tf.estimator.RunConfig
object to configure the runtime settings.
Returns: A list of
adanet.subnetwork.Builder
instances.- previous_ensemble – The best
-
Reports¶
Containers for metadata about trained subnetworks.
Report¶
-
class
adanet.subnetwork.
Report
[source]¶ A container for data to be collected about a
Subnetwork
.Parameters: - hparams – A dict mapping strings to python strings, ints, bools, or floats.
It is meant to contain the constants that define the
adanet.subnetwork.Builder
, such as dropout, number of layers, or initial learning rate. - attributes – A dict mapping strings to rank 0 Tensors of dtype string, int32,
or float32. It is meant to contain properties that may or may not change
over the course of training the
adanet.subnetwork.Subnetwork
, such as the number of parameters, the Lipschitz constant, the \(L2\) norm of the weights, or learning rate at materialization time. - metrics – Dict of metric results keyed by name. The values of the dict are
the results of calling a metric function, namely a (metric_tensor,
update_op) tuple. metric_tensor should be evaluated without any impact
on state (typically is a pure computation results based on variables.).
For example, it should not trigger the
update_op
or requires any input fetching. This is meant to contain metrics of interest, such as the training loss, complexity regularized loss, or standard deviation of the last layer outputs.
Returns: A validated
adanet.subnetwork.Report
object.Raises: ValueError
– If validation fails.- hparams – A dict mapping strings to python strings, ints, bools, or floats.
It is meant to contain the constants that define the
MaterializedReport¶
-
class
adanet.subnetwork.
MaterializedReport
[source]¶ Data collected about a
adanet.subnetwork.Subnetwork
.Parameters: - iteration_number – A python integer for the AdaNet iteration number, starting from 0.
- name – A string, which is either the name of the corresponding Builder, or “previous_ensemble” if it refers to the previous_ensemble.
- hparams – A dict mapping strings to python strings, ints, or floats. These
are constants passed from the author of the
adanet.subnetwork.Builder
that was used to construct thisadanet.subnetwork.Subnetwork
. It is meant to contain the arguments that defined theadanet.subnetwork.Builder
, such as dropout, number of layers, or initial learning rate. - attributes – A dict mapping strings to python strings, ints, bools, or
floats. These are python primitives that come from materialized Tensors;
these Tensors were defined by the author of the
adanet.subnetwork.Builder
that was used to construct thisadanet.subnetwork.Subnetwork
. It is meant to contain properties that may or may not change over the course of training theadanet.subnetwork.Subnetwork
, such as the number of parameters, the Lipschitz constant, or the \(L2\) norm of the weights. - metrics – A dict mapping strings to python strings, ints, or floats. These
are python primitives that come from metrics that were evaluated on the
trained
adanet.subnetwork.Subnetwork
over some dataset; these metrics were defined by the author of theadanet.subnetwork.Builder
that was used to construct thisadanet.subnetwork.Subnetwork
. It is meant to contain performance metrics or measures that could predict generalization, such as the training loss, complexity regularized loss, or standard deviation of the last layer outputs. - included_in_final_ensemble – A boolean denoting whether the associated
adanet.subnetwork.Subnetwork
was included in the ensemble at the end of the AdaNet iteration.
Returns: An
adanet.subnetwork.MaterializedReport
object.