A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. It actually provides a nice way to parallelized training: Pretrain e.g.

It's a simple idea but the loss formulation is nice, you usually want your loss terms to be a measure of information. They also state that this hardening often happens automatically though. Then they add a new loss term that encourages hard decisions by minimising the entropy of the Bernoulli distribution, making the 2 weights converge to 0 and 1, at which point only one branch needs to be taken at inference. They only do 'proper processing' using the leaf nodes. They gradually harden the branches so they become hard branches at the end of training.Ī branch is computed as branch(input, N), with a neural network N computing a scalar c=N(input), then using a sigmoid to do a soft branch by returning the weighted sum of the recursive call s(c)*branch(input, N_left) + (1-s(c)) * branch(input, N_right) (the two weights s(c) and 1-s(c) sum to 1). An attempt at a summary: They use a sigmoid function to make differentiable 'soft' branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost.

Something hidden deep within einsein