Kernel Module

Introduction

This module contains many different types of kernels. Each kernel is created based on some hyperparameters and a sample dataset.

General

If no sample dataset is provided, a random one will be initialized. This dataset can always be reinitialized (init_sample) or the alue of the datapoints can be updated updated (update_sample). In the latter case, the dimensions have to be matching. Furthermore, the sample dataset can also work in a stochastic manner, of which the indices can be controlled through the reset method.

Both the value of the sample datapoints as the hyperparameters are compatible with gradient graphs of PyTorch. If such a graph is to be computed, this has to be specifically specified during constructions.

All kernels can be centered, either implicitly using statistics on the kernel matrix of the sample dataset, either explicitly using a statistic on the explicit feature map. In the former case, this cannot be extended to fully out-of-sample computations.

At last, a Nystrom kernel is also implemented, which created an explicit feature map based on any kernel (possibly implicit), using eigendocomposition. Among other things, this can serve as a solution for centering fully out-of-sample kernel matrices of implicitly defined kernels.

The general structure of the module is based around an abstract kernel class base, of which kerch.kernle.implicit and explicit inherit. All other kernels inherit of one of these two at the exception of polynomial which directly inherits base as it has a primal formulation and a dual formulation which can be computed otherwise than with an inner product of the explicit feature map.

Kernel Factory

kerch.kernel.factory(kernel_type='rbf', **kwargs) → Kernel

Creates a kernel based on the specified name with the specified arguments. This is the same as calling kerch.kernel.name(*args, **kwargs) (if name is not a string here). This allows for the creation of kernel where the name of kernel is passed as a string.

Parameters:

• kernel_type (str, optional) – Type of kernel chosen. For the possible choices, please refer to the Factory Type column of the Kernel Module documentation. Defaults to kerch.DEFAULT_KERNEL_TYPE.

• **kwargs (dict, optional) – Arguments to be passed to the kernel constructor, such as sample or sigma. If an argument is passed that does not exist (e.g. sigma to a linear kernel), it will just be neglected. For the default values, please refer to the default values of the requested kernel.

Returns:

An instantiation of the specified kernel.

Examples

import kerch
import numpy as np
from matplotlib import pyplot as plt

sample = np.sin(np.arange(0,15) / np.pi) + .1
oos = np.sin(np.arange(15,30) / np.pi) + .1

k = kerch.kernel.factory(type="polynomial", sample=sample, center=True, normalize=True)

fig, axs = plt.subplots(2,2)

axs[0,0].imshow(k.K, vmin=-1, vmax=1)
axs[0,0].set_title("Sample - Sample")

axs[0,1].imshow(k.k(y=oos), vmin=-1, vmax=1)
axs[0,1].set_title("Sample - OOS")

axs[1,0].imshow(k.k(x=oos), vmin=-1, vmax=1)
axs[1,0].set_title("OOS - Sample")

im = axs[1,1].imshow(k.k(x=oos, y=oos), vmin=-1, vmax=1)
axs[1,1].set_title("OOS - OOS")

for ax in axs.flat:
    ax.set_xticks([])
    ax.set_yticks([])

fig.colorbar(im, ax=axs.ravel().tolist())

(Source code, png, hires.png, pdf)

Different Kernels

Generic Kernels

Class	Explicit Feature Map	Kernel	Parameters	Factory Type
kerch.kernel.Linear	\(\phi(x) = x\)	\(k(x,y)=\phi(x)^\top\phi(y)\)	None	’linear’
kerch.kernel.RBF		\(k(x,y)=\exp\left(-\frac{\left\lVert x-y\right\rVert^2_2}{2\sigma^2}\right)\)	sigma	’rbf’
kerch.kernel.Laplacian		\(k(x,y)=\exp\left(-\frac{\left\lVert x-y\right\rVert_2}{2\sigma^2}\right)\)	sigma	’laplacian’
kerch.kernel.Polynomial	See documentation	\(k(x,y) = \left(x^\top y + \beta\right)^\alpha\)	alpha, beta	’polynomial’
kerch.kernel.Cosine	\(\phi(x) = \frac{x}{\left\lVert x\right\rVert_2}\)	\(k(x,y)=\phi(x)^\top\phi(y)\)	None	’cosine’
kerch.kernel.Sigmoid		\(k(x,y) = \sigma\left( a (x^\top y) + b \right)\)	a, b	’sigmoid’
kerch.kernel.RFF	See documentation	\(k(x,y)=\phi(x)^\top\phi(y)\)	num_weights, sigma	’rff’
kerch.kernel.Nystrom	See documentation	See documentation	dim	’nystrom’

Network-Based Kernels

Class	Explicit Feature Map	Kernel	Parameters	Factory Type
kerch.kernel.ExplicitNN	\(\phi(x) = NN\left(x\right)\)	\(k(x,y)=\phi(x)^\top\phi(y)\)	None	’explicit_nn’
kerch.kernel.ImplicitNN		\(k(x,y) = NN\left( [x, y] \right).\)	None	’implicit_nn’

Time Kernels

The idea behind time kernels is that time has the same local effect at all time, or in other words that the kernels are translational invariant. We typically consider the following kernels:

Class	Explicit Feature Map	Kernel	Parameters	Factory Type
kerch.kernel.Indicator		See documentation	lag, gamma	’indicator’
kerch.kernel.RBF		\(k(x,y)=\exp\left(-\frac{\left\lVert x-y\right\rVert^2_2}{2\sigma^2}\right)\)	sigma	’rbf’
kerch.kernel.Hat		See documentation	lag	’hat’

Statistical Kernels

Class	Explicit Feature Map	Kernel	Parameters	Factory Type
kerch.kernel.Uniform		\(k(x,y)=1 \text{ for } \frac{d(x,y)}{\sigma} \leq 1\)	sigma, distance	’uniform’
kerch.kernel.Triangular		\(k(x,y)=1 - \frac{d(x,y)}{\sigma} \text{ for } \frac{d(x,y)}{\sigma} \leq 1\)	sigma, distance	’triangular’
kerch.kernel.Epanechnikov		\(k(x,y)=1 - \left(\frac{d(x,y)}{\sigma}\right)^2 \text{ for } \frac{d(x,y)}{\sigma} \leq 1\)	sigma, distance	’epanechnikov’
kerch.kernel.Quartic		\(k(x,y)= \left(1 - \left(\frac{d(x,y)}{\sigma}\right)^2\right)^2 \text{ for } \frac{d(x,y)}{\sigma} \leq 1\)	sigma, distance	’quartic’
kerch.kernel.Triweight		\(k(x,y)= \left(1 - \left(\frac{d(x,y)}{\sigma}\right)^2\right)^3 \text{ for } \frac{d(x,y)}{\sigma} \leq 1\)	sigma, distance	’triweight’
kerch.kernel.Tricube		\(k(x,y)= \left(1 - \left|\frac{d(x,y)}{\sigma}\right|^3\right)^3 \text{ for } \frac{d(x,y)}{\sigma} \leq 1\)	sigma, distance	’tricube’
kerch.kernel.Exponential		\(k(x,y)= \exp\left(-\frac{d(x,y)}{\sigma}\right)\)	sigma, distance, squared	’exponential’
kerch.kernel.Logistic		\(k(x,y)= \frac{4}{\exp\left(d(x,y)/\sigma\right) + 2 + \exp\left(-d(x,y)/\sigma\right)}\)	sigma, distance	’logistic’
kerch.kernel.Silverman		\(k(x,y)= \exp\left(-\frac{d(x,y)}{\sqrt{2}\sigma}\right)\sin\left(\frac{d(x,y)}{\sqrt{2}\sigma} + \frac{\pi}{4}\right)\)	sigma, distance	’silverman’

Vision Kernels

Class	Explicit Feature Map	Kernel	Parameters	Factory Type
kerch.kernel.AdditiveChi2		\(k(x,y) = \sum_i \frac{2x_i y_i}{x_i + y_i}\)	None	’additive_chi_2’
kerch.kernel.SkewedChi2		\(k(x,y) = \prod_i \frac{2\sqrt{x_i+p} \sqrt{y_i+p}}{x_i + y_i + 2}\)	p	’skewed_chi_2’

Abstract Kernels

Class	Explicit Feature Map	Kernel	Parameters	Factory Type
Exponential		_dist()	sigma	N/A
Explicit	_explicit()	\(k(x,y)=\phi(x)^\top\phi(y)\)	None	N/A
Implicit		_implicit()	None	N/A
Kernel	_explicit()	_implicit()	None	N/A

Creating Kernels

It is also possible to create other kernels than the ones already implemented and still taking benefit of all features common to all kernel such as automatic transformation of the input of the kernel (centering, normalization…), all attributes (K, Phi…), cache management, stochasicity etc. We refer to the examples in the documentation of the abstract classes kerch.kernel.Exponential, kerch.kernel.Explicit, kerch.kernel.Implicit and kerch.kernel.Kernel.