delve package

Submodules

delve.kerascallback module

Note

Keras implementation is in development and not tested.

delve.metrics module

delve.metrics.compute_intrinsic_dimensionality(cov: torch.Tensor, thresh: float = 0.99) → int

Compute the intrinsic dimensionality based on the covariance matrix :param cov: the covariance matrix as a torch tensor :param thresh: delta value; the explained variance of the covariance matrix :return: The intrinsic dimensionality; an integer value greater than zero

delve.metrics.compute_saturation(cov: torch.Tensor, thresh: float = 0.99) → float

Computes the saturation :param cov: the covariance matrix as a torch tensor :param thresh: delta value; the explained variance of the covariance matrix :return: a value between 0 and 1

delve.metrics.compute_cov_determinant(cov: torch.Tensor) → float

Computes the determinant of the covariance matrix (also known as generalized variance) :param cov: the covariannce matrix as torch tensor :return: the determinant

delve.metrics.compute_cov_trace(cov: torch.Tensor) → float

Computes the trace of the covariance matrix (also known as total variance) :param cov: the covariannce matrix as torch tensor :return: the trace

delve.metrics.compute_diag_trace(cov: torch.Tensor) → float

Computes the trace of the covariance matrix diagonal matrix :param cov: the covariannce matrix as torch tensor :return: the trace

delve.pca_layers module

delve.pca_layers.rvs(dim=3) → numpy.ndarray

Create random orthonormal matrix of size dim.

Note

Yanked from hpaulj’s implementation of SciPy’s scipy.stats.special_ortho_group() in Numpy at https://stackoverflow.com/questions/38426349/how-to-create-random-orthonormal-matrix-in-python-numpy which is from the paper:

Stewart, G.W., “The efficient generation of random orthogonal matrices with an application to condition estimators”, SIAM Journal on Numerical Analysis, 17(3), pp. 403-409, 1980.

delve.pca_layers.change_all_pca_layer_thresholds_and_inject_random_directions(threshold: float, network: torch.nn.modules.module.Module, verbose: bool = False, device='cpu', include_names: bool = False) → Tuple[list, list, list]

delve.pca_layers.change_all_pca_layer_thresholds(threshold: float, network: torch.nn.modules.module.Module, verbose: bool = False)

delve.pca_layers.change_all_pca_layer_centering(centering: bool, network: torch.nn.modules.module.Module, verbose: bool = False, downsampling=None)

class delve.pca_layers.LinearPCALayer(in_features: int, threshold: float = 0.99, keepdim: bool = True, verbose: bool = False, gradient_epoch_start: int = 20, centering: bool = True)

Bases: torch.nn.modules.module.Module

Eigenspace of the covariance matrix generated in TorchCovarianceMatrix with equation (1).

num = 0

is_floating_point()

property threshold: float

property centering

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class delve.pca_layers.Conv2DPCALayer(in_filters, threshold: float = 0.99, verbose: bool = True, gradient_epoch_start: int = 20, centering: bool = False, downsampling: Optional[int] = None)

Bases: delve.pca_layers.LinearPCALayer

Compute PCA of Conv2D layer

training: bool

keepdim: bool

verbose: bool

pca_computed: bool

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

delve.tools module

delve.tools.reconstruct_csv_from_npy_data(npywriter_out_path: str, savefile: Optional[str] = None, thresh: Optional[float] = None, stats: List[str] = ['lsat', 'idim']) → pandas.core.frame.DataFrame

This function allows the user to reconstruct the csv as constructed by the csv-writer. It further allows computing metrics that were initially not computed during training, if the covariance matrix is stored during training using the npy-writer. It also allows to recompute stats using different thresholds than originally used during training. :param npywriter_out_path: The path were the npy-writer has written the output :param savefile: the target-file to store the csv into, if not set, the data is not saved :param thresh: the threshold for computing the intrinsic dimensionality and saturation. If not set, the function tries to find stored version of all metrics. If set to any value, the stats will be computed from the stored covariance matrix. If not covariance matrix is given and threshold is set an error is raised. :param stats: the statistic to compute or read. Valid values are ‘sat’ and ‘idim’. All other metrics that were recorded during training and properly stored as npy-files will be read as well. :return:

delve.torch_utils module

class delve.torch_utils.TorchCovarianceMatrix(bias: bool = False, device: str = 'cuda:0', save_data: bool = False)

Bases: object

Computes covariance matrix of features as described in https://arxiv.org/pdf/2006.08679.pdf:

\begin{eqnarray} Q(Z_l, Z_l) = \frac{\sum^{B}_{b=0}A_{l,b}^T A_{l,b}}{n} -(\bar{A}_l \bigotimes \bar{A}_l) \end{eqnarray}

for \(B\) batches of layer output matrix \(A_l\) and \(n\) number of samples.

Note

Method enforces float-64 precision, which may cause numerical instability in some cases.

_init_internals(x: torch.Tensor)

Init the internal structures.

The reason this is not done in the constructor is that we want to be able to derive the input dimension and the dtype directly from the data this class receives.

update(x: torch.Tensor, vae: bool)

Update internal structures given a batch of data

fix(center=True)

Returns the Covariance matrix

delve.torchcallback module

class delve.torchcallback.CheckLayerSat(savefile: str, save_to: Union[str, delve.writers.AbstractWriter], modules: torch.nn.modules.module.Module, writer_args: Optional[Dict[str, Any]] = None, log_interval=1, max_samples=None, stats: list = ['lsat'], layerwise_sat: bool = True, reset_covariance: bool = True, average_sat: bool = False, ignore_layer_names: List[str] = [], include_conv: bool = True, conv_method: str = 'channelwise', timeseries_method: str = 'last_timestep', sat_threshold: str = 0.99, nosave=False, verbose: bool = False, device='cuda:0', initial_epoch: int = 0, interpolation_strategy: Optional[str] = None, interpolation_downsampling: int = 32)

Bases: object

Takes PyTorch module and records layer saturation,

intrinsic dimensionality and other scalars.

Parameters

• savefile (str) – destination for summaries

• (str (save_to) –

  Specify one or multiple save strategies.

  You can use preimplemented save strategies or inherit from the AbstractWriter in order to implement your own preferred saving strategy.

  pre-existing saving strategies are:

  csvstores all stats in a csv-file with one

  row for each epoch.

  plotproduces plots from intrinsic dimensionality

  and / or layer saturation

  tensorboard : saves all stats to tensorboard print : print all metrics on console

  as soon as they are logged

  npycreates a folder-structure with npy-files

  containing the logged values. This is the only save strategy that can save the full covariance matrix. This strategy is useful if you want to reproduce intrinsic dimensionality and saturation values with other thresholds without re-evaluating model checkpoints.

• List[Union[str –

  Specify one or multiple save strategies.

  You can use preimplemented save strategies or inherit from the AbstractWriter in order to implement your own preferred saving strategy.

  pre-existing saving strategies are:

  csvstores all stats in a csv-file with one

  row for each epoch.

  plotproduces plots from intrinsic dimensionality

  and / or layer saturation

  tensorboard : saves all stats to tensorboard print : print all metrics on console

  as soon as they are logged

  npycreates a folder-structure with npy-files

  containing the logged values. This is the only save strategy that can save the full covariance matrix. This strategy is useful if you want to reproduce intrinsic dimensionality and saturation values with other thresholds without re-evaluating model checkpoints.

• delve.writers.AbstractWriter]] –

  Specify one or multiple save strategies.

  You can use preimplemented save strategies or inherit from the AbstractWriter in order to implement your own preferred saving strategy.

  pre-existing saving strategies are:

  csvstores all stats in a csv-file with one

  row for each epoch.

  plotproduces plots from intrinsic dimensionality

  and / or layer saturation

  tensorboard : saves all stats to tensorboard print : print all metrics on console

  as soon as they are logged

  npycreates a folder-structure with npy-files

  containing the logged values. This is the only save strategy that can save the full covariance matrix. This strategy is useful if you want to reproduce intrinsic dimensionality and saturation values with other thresholds without re-evaluating model checkpoints.

• modules (torch modules or list of modules) – layer-containing object. Per default, only Conv2D, Linear and LSTM-Cells are recorded

• writers_args (dict) – contains additional arguments passed over to the writers. This is only used, when a writer is initialized through a string-key.

• log_interval (int) – distances between two batches used for updating the covariance matrix. Default value is 1, which means that all data is used for computing intrinsic dimensionality and saturation. Increasing the log interval is usefull on very large datasets to reduce numeric instability.

• max_samples (int) – (optional) the covariance matrix in each layer will halt updating itself when max_samples are reached. Usecase is similar to log-interval, when datasets are very large.

• stats (list of str) –

  list of stats to compute

  supported stats are:

  idim : intrinsic dimensionality lsat : layer saturation (intrinsic dimensionality divided by feature space dimensionality) cov : the covariance-matrix (only saveable using the ‘npy’ save strategy) det : the determinant of the covariance matrix (also known as generalized variance) trc : the trace of the covariance matrix, generally a more useful metric than det for determining

  the total variance of the data than the determinant. However note that this does not take the correlation between features into account. On the other hand, in most cases the determinent will be zero, since there will be very strongly correlated features, so trace might be the better option.

  dtrc : the trace of the diagonalmatrix, another way of measuring the dispersion of the data. lsat : layer saturation (intrinsic dimensionality

  divided by feature space dimensionality)

  embed : samples embedded in the eigenspace of dimension 2

• layerwise_sat (bool) – whether or not to include layerwise saturation when saving

• reset_covariance (bool) – True by default, resets the covariance every time the stats are computed. Disabling this option will strongly bias covariance since the gradient will influence the model. We recommend computing saturation at the end of training and testing.

• include_conv – setting to False includes only linear layers

• conv_method (str) –

  how to subsample convolutional layers. Default is

  channelwise, which means that the each position of the filter tensor is considered a datapoint, effectivly yielding a data matrix of shape (height*width*batch_size, num_filters)

  supported methods are:

  channelwisetreats every depth vector of the tensor as a

  datapoint, effectivly reshaping the data tensor from shape (batch_size, height, width, channel) into (batch_size*height*width, channel).

  meanapplies global average pooling on

  each feature map

  maxapplies global max pooling on

  each feature map

  medianapplies global median pooling on

  each feature map

  flattenflattenes the entire feature map to a vector,

  reshaping the data tensor into a data matrix of shape (batch_size, height*width*channel). This strategy for dealing with convolutions is extremly memory intensive and will likely cause memory and performance problems for any non toy-problem

• timeseries_method (str) –

  how to subsample timeseries methods. Default

  is last_timestep.

  supported methods are:

  timestepwise : stacks each sample timestep-by-timestep last_timestep : selects the last timestep’s output

• nosave (bool) – If True, disables saving artifacts (images), default is False

• verbose (bool) – print saturation for every layer during training

• sat_threshold (float) – threshold used to determine the number of eigendirections belonging to the latent space. In effect, this is the threshold determining the the intrinsic dimensionality. Default value is 0.99 (99% of the explained variance), which is a compromise between a good and interpretable approximation. From experience the threshold should be between 0.97 and 0.9995 for meaningfull results.

• verbose – Change verbosity level (default is 0)

• device (str) – Device to do the computations on. Default is cuda:0. Generally it is recommended to do the computations on the gpu in order to get maximum performance. Using the cpu is generally slower but it lets delve use regular RAM instead of the generally more limited VRAM of the GPU. Not having delve run on the same device as the network causes slight performance decrease due to copying memory between devices during each forward pass. Delve can handle models distributed on multiple GPUs, however delve itself will always run on a single device.

• initial_epoch (int) – The initial epoch to start with. Default is 0, which corresponds to a new run. If initial_epoch != 0 the writers will look for save states that they can resume. If set to zero, all existing states will be overwritten. If set to a lower epoch than actually recorded the behavior of the writers is undefined and may result in crashes, loss of data or corrupted data.

• interpolation_strategy (str) – Default is None (disabled). If set to a string key accepted by the model-argument of torch.nn.functional.interpolate, the feature map will be resized to match the interpolated size. This is useful if you work with large resolutions and want to save up on computation time. is done if the resolution is smaller.

• interpolation_downsampling (int) – Default is 32. The target resolution if downsampling is enabled.

is_recording() → bool

stop()

resume()

close()

User endpoint to close writer and progress bars.

get_layer_from_submodule(submodule: torch.nn.modules.module.Module, layers: dict, name_prefix: str = '')

get_layers_recursive(modules: Union[list, torch.nn.modules.module.Module])

_get_writer(save_to, writers_args) → delve.writers.AbstractWriter

Create a writer to log history to writer_dir.

register_forward_hooks(layer: torch.nn.modules.module.Module, stats: list)

Register hook to show stats in layer.

add_saturations(save=True)

Computes saturation and saves all stats :return:

save()

delve.writers module

This file contains alternative file writers

class delve.writers.AbstractWriter

Bases: abc.ABC

_check_savestate_ok(savepath: str) → bool

Checks if a savestate from a writer is okay; raises a warning if not :param savepath: the path to the savestate :return:

abstract resume_from_saved_state(initial_epoch: int)

abstract add_scalar(name, value, **kwargs)

abstract add_scalars(prefix, value_dict, global_step, **kwargs)

abstract save()

abstract close()

class delve.writers.CompositWriter(writers: List[delve.writers.AbstractWriter])

Bases: delve.writers.AbstractWriter

__init__(writers: List[delve.writers.AbstractWriter])

This writer combines multiple writers. :param writers: List of writers. Each writer is called when the CompositeWriter is invoked.

resume_from_saved_state(initial_epoch: int)

add_scalar(name, value, **kwargs)

add_scalars(prefix, value_dict, **kwargs)

save()

close()

class delve.writers.CSVWriter(savepath: str, **kwargs)

Bases: delve.writers.AbstractWriter

This writer produces a csv file with all saturation values. The csv-file is overwritten with an updated version every time save() is called. :param savepath: CSV file path

resume_from_saved_state(initial_epoch: int)

add_scalar(name, value, **kwargs)

add_scalars(prefix, value_dict, **kwargs)

save()

close()

class delve.writers.NPYWriter(savepath: str, zip: bool = False, **kwargs)

Bases: delve.writers.AbstractWriter

__init__(savepath: str, zip: bool = False, **kwargs)

The NPYWriter creates a folder containing one subfolder for each stat. Each subfolder contains a npy-file with the saturation value for each epoch. This writer saves non-scalar values and can thus be used to save the covariance-matrix. :param savepath: The root folder to save the folder structure to :param zip: Whether to zip the output folder after every invocation

resume_from_saved_state(initial_epoch: int)

add_scalar(name, value, **kwargs)

add_scalars(prefix, value_dict, **kwargs)

save()

close()

class delve.writers.PrintWriter(**kwargs)

Bases: delve.writers.AbstractWriter

__init__(**kwargs)

Prints output to the console

resume_from_saved_state(initial_epoch: int)

add_scalar(name, value, **kwargs)

add_scalars(prefix, value_dict, **kwargs)

save()

close()

class delve.writers.TensorBoardWriter(savepath: str, **kwargs)

Bases: delve.writers.AbstractWriter

Writes output to tensorflow logs :param savepath: the path for result logging

resume_from_saved_state(initial_epoch: int)

add_scalar(name, value, **kwargs)

add_scalars(prefix, value_dict, **kwargs)

save()

close()

class delve.writers.CSVandPlottingWriter(savepath: str, plot_manipulation_func: Optional[Callable[[matplotlib.axes._axes.Axes], matplotlib.axes._axes.Axes]] = None, **kwargs)

Bases: delve.writers.CSVWriter

This writer produces CSV files and plots. :param savepath: Path to store plots and CSV files :param plot_manipulation_func: A function mapping an axis object to an axis object by

using pyplot code.

Parameters

kwargs –

resume_from_saved_state(initial_epoch: int)

add_scalar(name, value, **kwargs)

add_sample_scalar(name, value, **kwargs)

add_scalars(prefix, value_dict, sample_value_dict, **kwargs)

save()

close()

delve.writers.extract_layer_stat(df, epoch=19, primary_metric=None, stat='saturation', state_mode='train') → Tuple[pandas.core.frame.DataFrame, float]

Extracts a specific statistic for a single epoch from a result dataframe as produced by the CSV-writer :param df: The dataframe produced by a CSVWriter :param epoch: Epoch to filter by :param primary_metric: Primary metric for performance evaluation (optional) :param stat: The statistic to match. Must be a substring matching all columns belonging to stat statistic like “saturation” :return: A dataframe with a single row, corresponding to the epoch containing only the columns that contain the substring described in the stat-parameter in their name. Second return value is the primary metric value

delve.writers.plot_stat(df, stat, pm=- 1, savepath='run.png', epoch=0, primary_metric=None, fontsize=16, figsize=None, line=True, scatter=True, ylim=(0, 1.0), alpha_line=0.6, alpha_scatter=1.0, color_line=None, color_scatter=None, primary_metric_loc=(0.7, 0.8), show_col_label_x=True, show_col_label_y=True, show_grid=True, save=True, samples=False, stat_mode='train') → matplotlib.axes._axes.Axes

Plot statistics

Parameters

• df –

• stat –

• pm –

• savepath –

• epoch –

• primary_metric –

• fontsize –

• figsize –

• line –

• scatter –

• ylim –

• alpha_line –

• alpha_scatter –

• color_line –

• color_scatter –

• primary_metric_loc –

• show_col_label_x –

• show_col_label_y –

• show_grid –

• save –

Returns

delve.writers.plot_stat_level_from_results(savepath, epoch, stat, primary_metric=None, fontsize=16, figsize=None, line=True, scatter=True, ylim=(0, 1.0), alpha_line=0.6, alpha_scatter=1.0, color_line=None, color_scatter=None, primary_metric_loc=(0.7, 0.8), show_col_label_x=True, show_col_label_y=True, show_grid=True, save=True, stat_mode='train')

delve.writers.plot_scatter_from_results(savepath, epoch, stat, df)

Module contents