Discrete illustration studying with VQ-VAE and stress circulate likelihood

About two weeks in the past, Launched Chance of Tensorflow (TFP)exhibiting how you can create and take a look at from Distributions and put them in a variational self -coexist (VAE) that learns your earlier one. At this time we go to a distinct specimen within the VAE mannequin zoo: the quantified variational self -chire of the vector (VQ-VAE) described in Neural discrete illustration studying (Oord, Vinyals and Kavukcuoglu 2017). This mannequin differs from most of them by which its approximate posterior shouldn’t be steady, however discreet, therefore the “quantified” within the title of the article. We are going to shortly see what this implies, after which we’ll immerse immediately within the code, combining layers of keras, anxious execution and TFP.

Many phenomena are thought of and modeled higher, as discreet. That is legitimate for phonemes and lexemes in language, increased degree buildings in photos (consider objects as an alternative of pixels) and duties that require reasoning and planning. Nonetheless, the latent code utilized in most VAE is steady; It’s usually a multivariate Gaussian. They’ve discovered that the continual area may be very profitable within the reconstruction of their contribution, however they typically undergo one thing referred to as posterior collapse: The decoder is so highly effective that it could create a sensible exit given alone any enter. Which means there isn’t a incentive to be taught an expressive latent area.

Nonetheless, in VQ-VAE, every entry pattern is mape deterministic to one in all a set of EMPRUSTING VECTORS. Collectively, these embedding vectors represent the earlier one for latent area. As such, an ink vector incorporates way more info than a median and a variance, and due to this fact, it’s way more troublesome to disregard by the decoder.

The query is then: the place is that magical hat, in order that we take out important inlays?

From the earlier conceptual description, we now have two inquiries to reply. First, why will we assign enter samples (which handed by the encoder) to the suitable embedding vectors? And second: How can we be taught to embed vectors which can be actually helpful representations, which once they feed on a decoder, will outcome that perceived entities belonged to the identical species?

As for the allocation, a tensioner issued from the encoder merely assigns its closest neighbor within the embedding area, utilizing the Euclidean distance. Incrustation vectors are up to date utilizing exponential cell averages. As we’ll see quickly, because of this they don’t seem to be actually discovered utilizing gradient descent, a attribute that’s value pointing, since we don’t discover it on daily basis in deep studying.

Particularly, how ought to the loss operate and the coaching course of be? This can in all probability be simpler within the code.

The entire code for this instance, together with the utilities for the storage of the mannequin and the visualization of photos, is Accessible in Github As a part of the examples of keras. The order of presentation right here might differ from the actual execution order for expository functions, so to execute the code, think about using the github instance.

As in all our earlier publications in VAE, we use an anxious execution, which presupposes the Keras tensorflow implementation.

As in our earlier publication about doing VAE with TFP, we’ll use Kuzushiji-Mnist(Clanuwat et al. 2018) as entrance. Now could be the time to look What we find yourself producing that point And place your guess: how will that be in contrast with the discreet latent area of VQ-VAE?

np <- import("numpy")
 
kuzushiji <- np$load("kmnist-train-imgs.npz")
kuzushiji <- kuzushiji$get("arr_0")

train_images <- kuzushiji %>%
  k_expand_dims() %>%
  k_cast(dtype = "float32")

train_images <- train_images %>% `/`(255)

buffer_size <- 60000
batch_size <- 64
num_examples_to_generate <- batch_size

batches_per_epoch <- buffer_size / batch_size

train_dataset <- tensor_slices_dataset(train_images) %>%
  dataset_shuffle(buffer_size) %>%
  dataset_batch(batch_size, drop_remainder = TRUE)

Hyperparameters

Along with the “regular” hyperparameters that we’ve in deep studying, the VQ-VAE infrastructure presents some particular mannequin. First, the embedding area is dimensionality Variety of embedding vectors instances Incrustation vector measurement:

# variety of embedding vectors
num_codes <- 64L
# dimensionality of the embedding vectors
code_size <- 16L

The latent area in our instance will likely be measurement one, that’s, we’ve just one vector of incrustation that represents the latent code for every enter pattern. This will likely be high quality for our knowledge set, but it surely needs to be taken into consideration that Van Den Oord et al. He used latent excessive dimension areas of better dimension in EG Imagenet and CIFFAR-10.

Encoder mannequin

The encoder makes use of convolutionary layers to extract picture traits. Your departure is a 3 -dimensional tensioner tons by tons * 1 * Code_Size.

activation <- "elu"
# modularizing the code just a bit bit
default_conv <- set_defaults(layer_conv_2d, record(padding = "identical", activation = activation))

base_depth <- 32

encoder_model <- operate(title = NULL,
                          code_size) {
  
  keras_model_custom(title = title, operate(self) {
    
    self$conv1 <- default_conv(filters = base_depth, kernel_size = 5)
    self$conv2 <- default_conv(filters = base_depth, kernel_size = 5, strides = 2)
    self$conv3 <- default_conv(filters = 2 * base_depth, kernel_size = 5)
    self$conv4 <- default_conv(filters = 2 * base_depth, kernel_size = 5, strides = 2)
    self$conv5 <- default_conv(filters = 4 * latent_size, kernel_size = 7, padding = "legitimate")
    self$flatten <- layer_flatten()
    self$dense <- layer_dense(models = latent_size * code_size)
    self$reshape <- layer_reshape(target_shape = c(latent_size, code_size))
    
    operate (x, masks = NULL) {
      x %>% 
        # output form:  7 28 28 32 
        self$conv1() %>% 
        # output form:  7 14 14 32 
        self$conv2() %>% 
        # output form:  7 14 14 64 
        self$conv3() %>% 
        # output form:  7 7 7 64 
        self$conv4() %>% 
        # output form:  7 1 1 4 
        self$conv5() %>% 
        # output form:  7 4 
        self$flatten() %>% 
        # output form:  7 16 
        self$dense() %>% 
        # output form:  7 1 16
        self$reshape()
    }
  })
}

As at all times, let’s use the truth that we’re utilizing anxious execution and see some instance outputs.

iter <- make_iterator_one_shot(train_dataset)
batch <-  iterator_get_next(iter)

encoder <- encoder_model(code_size = code_size)
encoded  <- encoder(batch)
encoded

tf.Tensor(
((( 0.00516277 -0.00746826  0.0268365  ... -0.012577   -0.07752544
   -0.02947626))
...

 ((-0.04757921 -0.07282603 -0.06814402 ... -0.10861694 -0.01237121
    0.11455103))), form=(64, 1, 16), dtype=float32)

Now, every of those 16D vectors must be assigned to the incrusting vector to which it’s nearer. This mapping is handled by one other mannequin: vector_quantizer.

Vector quantizing mannequin

That is how we’ll prompt the vector quantizer:

vector_quantizer <- vector_quantizer_model(num_codes = num_codes, code_size = code_size)

This mannequin has two functions: first, it acts as a retailer for incrusting vectors. Secondly, it coincides with the encoder output with obtainable inlays.

Right here, the present state of the incrustations is saved in codebook. ema_means and ema_count They’re just for accounting functions (have in mind how they’re established in order to not be trainable). We are going to see them in use shortly.

vector_quantizer_model <- operate(title = NULL, num_codes, code_size) {
  
    keras_model_custom(title = title, operate(self) {
      
      self$num_codes <- num_codes
      self$code_size <- code_size
      self$codebook <- tf$get_variable(
        "codebook",
        form = c(num_codes, code_size), 
        dtype = tf$float32
        )
      self$ema_count <- tf$get_variable(
        title = "ema_count", form = c(num_codes),
        initializer = tf$constant_initializer(0),
        trainable = FALSE
        )
      self$ema_means = tf$get_variable(
        title = "ema_means",
        initializer = self$codebook$initialized_value(),
        trainable = FALSE
        )
      
      operate (x, masks = NULL) { 
        
        # to be crammed in shortly ...
        
      }
    })
}

Along with actual inlays, of their name methodology vector_quantizer It incorporates the allocation logic. First, we calculate the Euclidean distance of every coding to the vectors within the code guide (tf$norm). We assign every coding to the closest as for that incrusting distance (tf$argmin) and a smelling perspective duties (tf$one_hot). Lastly, we isolate the corresponding vector masking everybody else and summarizing what stays (multiplication adopted by tf$reduce_sum).

With respect to axis argument used with many tensioning circulate features, needless to say, in distinction to its k_* brothers, uncooked tensor circulate (tf$*) Capabilities anticipate the axis numbering to be primarily based on 0. We even have so as to add the LAfter the numbers to satisfy the necessities of the tensorflow knowledge sort.

vector_quantizer_model <- operate(title = NULL, num_codes, code_size) {
  
    keras_model_custom(title = title, operate(self) {
      
      # right here we've the above occasion fields
      
      operate (x, masks = NULL) {
    
        # form: bs * 1 * num_codes
         distances <- tf$norm(
          tf$expand_dims(x, axis = 2L) -
            tf$reshape(self$codebook, 
                       c(1L, 1L, self$num_codes, self$code_size)),
                       axis = 3L 
        )
        
        # bs * 1
        assignments <- tf$argmin(distances, axis = 2L)
        
        # bs * 1 * num_codes
        one_hot_assignments <- tf$one_hot(assignments, depth = self$num_codes)
        
        # bs * 1 * code_size
        nearest_codebook_entries <- tf$reduce_sum(
          tf$expand_dims(
            one_hot_assignments, -1L) * 
            tf$reshape(self$codebook, c(1L, 1L, self$num_codes, self$code_size)),
                       axis = 2L 
                       )
        record(nearest_codebook_entries, one_hot_assignments)
      }
    })
  }

Now that we’ve seen how codes are saved, we add performance to replace them. As we stated earlier than, they don’t seem to be discovered by gradient descent. As an alternative, they’re exponential cell averages, repeatedly up to date by any new “class member” assigned to them.

So right here there’s a operate update_ema that can care for this.

update_ema Use tensorflow MOVING_AVERAGES to

• First, monitor the variety of samples at present assigned by code (updated_ema_count), and
• Second, calculate and assign the present exponential cell common (updated_ema_means).

moving_averages <- tf$python$coaching$moving_averages

# decay to make use of in computing exponential transferring common
decay <- 0.99

update_ema <- operate(
  vector_quantizer,
  one_hot_assignments,
  codes,
  decay) {
 
  updated_ema_count <- moving_averages$assign_moving_average(
    vector_quantizer$ema_count,
    tf$reduce_sum(one_hot_assignments, axis = c(0L, 1L)),
    decay,
    zero_debias = FALSE
  )

  updated_ema_means <- moving_averages$assign_moving_average(
    vector_quantizer$ema_means,
    # selects all assigned values (masking out the others) and sums them up over the batch
    # (will likely be divided by rely later, so we get a median)
    tf$reduce_sum(
      tf$expand_dims(codes, 2L) *
        tf$expand_dims(one_hot_assignments, 3L), axis = c(0L, 1L)),
    decay,
    zero_debias = FALSE
  )

  updated_ema_count <- updated_ema_count + 1e-5
  updated_ema_means <-  updated_ema_means / tf$expand_dims(updated_ema_count, axis = -1L)
  
  tf$assign(vector_quantizer$codebook, updated_ema_means)
}

Earlier than trying on the coaching loop, let’s shortly full the scene by including to the final actor, the decoder.

Decoding mannequin

The decoder is kind of commonplace, making a collection of deconvolutions and at last, returning a likelihood for every picture pixel.

default_deconv <- set_defaults(
  layer_conv_2d_transpose,
  record(padding = "identical", activation = activation)
)

decoder_model <- operate(title = NULL,
                          input_size,
                          output_shape) {
  
  keras_model_custom(title = title, operate(self) {
    
    self$reshape1 <- layer_reshape(target_shape = c(1, 1, input_size))
    self$deconv1 <-
      default_deconv(
        filters = 2 * base_depth,
        kernel_size = 7,
        padding = "legitimate"
      )
    self$deconv2 <-
      default_deconv(filters = 2 * base_depth, kernel_size = 5)
    self$deconv3 <-
      default_deconv(
        filters = 2 * base_depth,
        kernel_size = 5,
        strides = 2
      )
    self$deconv4 <-
      default_deconv(filters = base_depth, kernel_size = 5)
    self$deconv5 <-
      default_deconv(filters = base_depth,
                     kernel_size = 5,
                     strides = 2)
    self$deconv6 <-
      default_deconv(filters = base_depth, kernel_size = 5)
    self$conv1 <-
      default_conv(filters = output_shape(3),
                   kernel_size = 5,
                   activation = "linear")
    
    operate (x, masks = NULL) {
      
      x <- x %>%
        # output form:  7 1 1 16
        self$reshape1() %>%
        # output form:  7 7 7 64
        self$deconv1() %>%
        # output form:  7 7 7 64
        self$deconv2() %>%
        # output form:  7 14 14 64
        self$deconv3() %>%
        # output form:  7 14 14 32
        self$deconv4() %>%
        # output form:  7 28 28 32
        self$deconv5() %>%
        # output form:  7 28 28 32
        self$deconv6() %>%
        # output form:  7 28 28 1
        self$conv1()
      
      tfd$Impartial(tfd$Bernoulli(logits = x),
                      reinterpreted_batch_ndims = size(output_shape))
    }
  })
}

input_shape <- c(28, 28, 1)
decoder <- decoder_model(input_size = latent_size * code_size,
                         output_shape = input_shape)

Now we’re prepared to coach. One factor we’ve probably not talked about is the associated fee operate: given the variations in structure (in comparison with the usual VAE), will losses stay anticipated (the standard sum of the lack of reconstruction and the divergence of KL)? We are going to see that in a second.

Coaching loop

Right here is the optimizer we’ll use. The losses will likely be calculated on-line.

optimizer <- tf$practice$AdamOptimizer(learning_rate = learning_rate)

The coaching loop, as regular, is a loop on instances, the place every iteration is loads on tons obtained from the information set. For every lot, we’ve a ahead move, recorded by a gradientTapeprimarily based on which we calculate the loss. The tape will decide the gradients of all skilled weights all through the mannequin, and the optimizer will use these gradients to replace the weights.

Till now, all this suits a scheme that we’ve typically seen earlier than. Nonetheless, some extent to contemplate: on this identical loop, we additionally name update_ema To recalculate cell averages, since they don’t seem to be operated in the course of the backprop. Right here is the important performance:

num_epochs <- 20

for (epoch in seq_len(num_epochs)) {
  
  iter <- make_iterator_one_shot(train_dataset)
  
  until_out_of_range({
    
    x <-  iterator_get_next(iter)
    with(tf$GradientTape(persistent = TRUE) %as% tape, {
      
      # do ahead move
      # calculate losses
      
    })
    
    encoder_gradients <- tape$gradient(loss, encoder$variables)
    decoder_gradients <- tape$gradient(loss, decoder$variables)
    
    optimizer$apply_gradients(purrr::transpose(record(
      encoder_gradients, encoder$variables
    )),
    global_step = tf$practice$get_or_create_global_step())
    
    optimizer$apply_gradients(purrr::transpose(record(
      decoder_gradients, decoder$variables
    )),
    global_step = tf$practice$get_or_create_global_step())
    
    update_ema(vector_quantizer,
               one_hot_assignments,
               codes,
               decay)

    # periodically show some generated photos
    # see code on github 
    # visualize_images("kuzushiji", epoch, reconstructed_images, random_images)
  })
}

Now, for actual motion. Throughout the context of the gradient tape, we first decide which coded enter pattern is assigned to which embedding vector.

codes <- encoder(x)
c(nearest_codebook_entries, one_hot_assignments) %<-% vector_quantizer(codes)

Now, for this project operation there isn’t a gradient. Then again, what we will do is move the gradients of the decoder entrance on to the encoler exit. Right here tf$stop_gradient exempt nearest_codebook_entries of the gradient chain, so the encoder and decoder are linked by codes:

codes_straight_through <- codes + tf$stop_gradient(nearest_codebook_entries - codes)
decoder_distribution <- decoder(codes_straight_through)

In abstract, Backprop will likely be accountable for the weights of the decoder and the encoder, whereas latent integrities are up to date utilizing cell averages, as we’ve already seen.

Now we’re prepared to deal with losses. There are three elements:

• First, the lack of reconstruction, which is barely the likelihood of registration of the actual entry underneath the distribution discovered by the decoder.

reconstruction_loss <- -tf$reduce_mean(decoder_distribution$log_prob(x))

• Second, we’ve the lack of dedicationOutlined as the typical sq. deviation of the coded enter samples of the neighbors closest to these which were assigned: We would like the community to “compromise” with a concise set of latent codes!

commitment_loss <- tf$reduce_mean(tf$sq.(codes - tf$stop_gradient(nearest_codebook_entries)))

• Lastly, we’ve the standard KL divergent to a earlier one. As, a priori, all duties are equally possible, this part of the loss is fixed and might typically dispense. We’re including it primarily for illustrative functions.

prior_dist <- tfd$Multinomial(
  total_count = 1,
  logits = tf$zeros(c(latent_size, num_codes))
  )
prior_loss <- -tf$reduce_mean(
  tf$reduce_sum(prior_dist$log_prob(one_hot_assignments), 1L)
  )

Summarizing the three elements, we attain the final loss:

beta <- 0.25
loss <- reconstruction_loss + beta * commitment_loss + prior_loss

Earlier than trying on the outcomes, let’s examine what occurs inside gradientTape At one look:

with(tf$GradientTape(persistent = TRUE) %as% tape, {
      
  codes <- encoder(x)
  c(nearest_codebook_entries, one_hot_assignments) %<-% vector_quantizer(codes)
  codes_straight_through <- codes + tf$stop_gradient(nearest_codebook_entries - codes)
  decoder_distribution <- decoder(codes_straight_through)
      
  reconstruction_loss <- -tf$reduce_mean(decoder_distribution$log_prob(x))
  commitment_loss <- tf$reduce_mean(tf$sq.(codes - tf$stop_gradient(nearest_codebook_entries)))
  prior_dist <- tfd$Multinomial(
    total_count = 1,
    logits = tf$zeros(c(latent_size, num_codes))
  )
  prior_loss <- -tf$reduce_mean(tf$reduce_sum(prior_dist$log_prob(one_hot_assignments), 1L))
  
  loss <- reconstruction_loss + beta * commitment_loss + prior_loss
})

Outcomes

And right here we go. This time, we can not have the “transformation view” 2D that usually likes to point out with VAE (there’s merely no latent area in 2D). Then again, the 2 photos under are (1) letters generated from random entrance and (2) rebuilt present Letters, every saved after coaching for 9 instances.

Two issues leap to the attention: first, the generated letters are considerably extra clear than their earlier steady counterparts (from the earlier publication). And second, would you might have been in a position to say the random picture of the reconstruction picture?

At this level, we hope we’ve satisfied it of the ability and effectiveness of this strategy to discrete latent. Nonetheless, I might have anticipated this to extra complicated knowledge, akin to speech components that we point out within the introduction or photos of upper decision akin to in Imagenet.

The reality is that there’s steady compensation between the quantity of recent and thrilling strategies that we will present, and the time we will spend in iterations to efficiently apply these strategies to complicated knowledge units. In the long run, it’s you, our readers, who will put these strategies to a major use within the related knowledge of the actual world.