Classification of Duplicate Questions of Quora with keras

Introduction

On this publication we are going to use keras to categorise the duplicate questions of Quora. The info set appeared for the primary time within the Kaggy competitors Quora Query {Couples} And it consists of roughly 400,000 pairs of questions along with a column that signifies whether or not the pair of questions is taken into account a replica.

Our implementation is impressed by the Siamese recurring structurewith modifications to the measure of similarity and the layers of inlays (the unique doc makes use of beforehand skilled phrases). The usage of the sort of structure dates again to 2005 with Le cun et al and is beneficial for verification duties. The thought is to study a operate that maps the doorway patterns in an goal area so {that a} measure of similarity within the goal area approaches the “semantic” distance within the entrance area.

After the competitors, Quora additionally described his method to this downside on this Weblog.

Cargo information

The info may be downloaded from the Kaggy Knowledge set web site
or quora’s Knowledge set launch:

library(keras)
quora_data <- get_file(
  "quora_duplicate_questions.tsv",
  "https://qim.ec.quoracdn.internet/quora_duplicate_questions.tsv"
)

We’re utilizing the keras get_file() Operate for the file downloading to be in cache.

Studying and preprocessing

First we are going to load the info in R and make some preprocesses to make it simpler to incorporate within the mannequin. After downloading the info, you’ll be able to learn it utilizing the readr read_tsv() operate.

We are going to create a keras tokenizer To rework every phrase into a whole token. We may even specify a hyperparameter of our mannequin: the dimensions of the vocabulary. For now, let’s use the 50,000 commonest phrases (we are going to tune this parameter later). The Tokenizer will alter utilizing all of the distinctive questions of the info set.

tokenizer <- text_tokenizer(num_words = 50000)
tokenizer %>% fit_text_tokenizer(distinctive(c(df$question1, df$question2)))

Let’s save the tokenizer on the disc to make use of it for inference later.

save_text_tokenizer(tokenizer, "tokenizer-question-pairs")

Now we are going to use the textual content tokenizer to remodel every query into an inventory of integers.

question1 <- texts_to_sequences(tokenizer, df$question1)
question2 <- texts_to_sequences(tokenizer, df$question2)

Let’s check out the quantity of phrases in every query. It will assist us to determine the size of the filling, one other hyperpartparameter of our mannequin. The filling of the sequences normalizes them to the identical measurement in order that we are able to feed them to the keras mannequin.

80% 90% 95% 99% 
 14  18  23  31 

We will see that 99% of the questions have a most of 31, so we are going to select a fill size between 15 and 30. Let’s begin with 20 (we may even tune this parameter later). The default filling worth is 0, however we’re already utilizing this worth for phrases that don’t seem inside the 50,000 most frequent, so we are going to use 50,001 as a substitute.

question1_padded <- pad_sequences(question1, maxlen = 20, worth = 50000 + 1)
question2_padded <- pad_sequences(question2, maxlen = 20, worth = 50000 + 1)

Now we’ve completed the preprocessing steps. Now we are going to execute a easy reference mannequin earlier than shifting on to the keras mannequin.

Easy reference level

Earlier than creating a sophisticated mannequin, let’s take a easy method. We consider two predictors: share of words1 that seem in question2 and vice versa. Then we are going to use a logistics regression to foretell if the questions are duplicated.

perc_words_question1 <- map2_dbl(question1, question2, ~imply(.x %in% .y))
perc_words_question2 <- map2_dbl(question2, question1, ~imply(.x %in% .y))

df_model <- information.body(
  perc_words_question1 = perc_words_question1,
  perc_words_question2 = perc_words_question2,
  is_duplicate = df$is_duplicate
) %>%
  na.omit()

Now that we’ve our predictors, we consider the logistics mannequin. We are going to take a small pattern for validation.

val_sample <- pattern.int(nrow(df_model), 0.1*nrow(df_model))
logistic_regression <- glm(
  is_duplicate ~ perc_words_question1 + perc_words_question2, 
  household = "binomial",
  information = df_model(-val_sample,)
)
abstract(logistic_regression)

Name:
glm(system = is_duplicate ~ perc_words_question1 + perc_words_question2, 
    household = "binomial", information = df_model(-val_sample, ))

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.5938  -0.9097  -0.6106   1.1452   2.0292  

Coefficients:
                      Estimate Std. Error z worth Pr(>|z|)    
(Intercept)          -2.259007   0.009668 -233.66   <2e-16 ***
perc_words_question1  1.517990   0.023038   65.89   <2e-16 ***
perc_words_question2  1.681410   0.022795   73.76   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial household taken to be 1)

    Null deviance: 479158  on 363843  levels of freedom
Residual deviance: 431627  on 363841  levels of freedom
  (17 observations deleted attributable to missingness)
AIC: 431633

Variety of Fisher Scoring iterations: 3

Let’s calculate precision in our validation set.

pred <- predict(logistic_regression, df_model(val_sample,), kind = "response")
pred <- pred > imply(df_model$is_duplicate(-val_sample))
accuracy <- desk(pred, df_model$is_duplicate(val_sample)) %>% 
  prop.desk() %>% 
  diag() %>% 
  sum()
accuracy

(1) 0.6573577

We’ve got an accuracy of 65.7%. It’s not significantly better than random divination. Now we consider our mannequin in keras.

Mannequin definition

We are going to use a Siamese community to foretell whether or not the pairs are duplicated or not. The thought is to create a mannequin that may embed the questions (sequence of phrases) in a vector. Then we are able to evaluate the vectors for every query utilizing a measure of similarity and know if the questions are duplicated or not.

First we outline the entries for the mannequin.

input1 <- layer_input(form = c(20), identify = "input_question1")
input2 <- layer_input(form = c(20), identify = "input_question2")

Then, let’s outline the a part of the mannequin that can embed the questions in a vector.

word_embedder <- layer_embedding( 
  input_dim = 50000 + 2, # vocab measurement + UNK token + padding worth
  output_dim = 128,      # hyperparameter - embedding measurement
  input_length = 20,     # padding measurement,
  embeddings_regularizer = regularizer_l2(0.0001) # hyperparameter - regularization 
)

seq_embedder <- layer_lstm(
  items = 128, # hyperparameter -- sequence embedding measurement
  kernel_regularizer = regularizer_l2(0.0001) # hyperparameter - regularization 
)

Now we are going to outline the connection between the enter vectors and the layers of inlays. Needless to say we use the identical layers and weights in each entries. That’s the reason that is referred to as a Siamese community. It is sensible, as a result of we don’t need to have completely different exits if the question1 is modified with question2.

vector1 <- input1 %>% word_embedder() %>% seq_embedder()
vector2 <- input2 %>% word_embedder() %>% seq_embedder()

Then we outline the measure of similarity that we need to optimize. We would like duplicate inquiries to have larger values ​​of similarity. On this instance, we are going to use the similarity of cosine, however any similarity measure might be used. Do not forget that cosena similarity is the normalized product product of vectors, however for coaching it’s not essential to normalize the outcomes.

cosine_similarity <- layer_dot(checklist(vector1, vector2), axes = 1)

Subsequent, we outline a ultimate sigmoid layer to generate the likelihood that each questions be doubled.

output <- cosine_similarity %>% 
  layer_dense(items = 1, activation = "sigmoid")

Now that we outline the keras mannequin when it comes to its inputs and exits and compile it. Within the compilation part we outline our loss and optimizer operate. As within the Kaggy Problem, we are going to reduce the logloss (equal to minimizing binary intenty). We are going to use the Adam Optimizer.

mannequin <- keras_model(checklist(input1, input2), output)
mannequin %>% compile(
  optimizer = "adam", 
  metrics = checklist(acc = metric_binary_accuracy), 
  loss = "binary_crossentropy"
)

Then we are able to check out the mannequin with the abstract operate.

_______________________________________________________________________________________
Layer (kind)                Output Form       Param #    Linked to                 
=======================================================================================
input_question1 (InputLayer (None, 20)         0                                       
_______________________________________________________________________________________
input_question2 (InputLayer (None, 20)         0                                       
_______________________________________________________________________________________
embedding_1 (Embedding)     (None, 20, 128)    6400256    input_question1(0)(0)        
                                                          input_question2(0)(0)        
_______________________________________________________________________________________
lstm_1 (LSTM)               (None, 128)        131584     embedding_1(0)(0)            
                                                          embedding_1(1)(0)            
_______________________________________________________________________________________
dot_1 (Dot)                 (None, 1)          0          lstm_1(0)(0)                 
                                                          lstm_1(1)(0)                 
_______________________________________________________________________________________
dense_1 (Dense)             (None, 1)          2          dot_1(0)(0)                  
=======================================================================================
Whole params: 6,531,842
Trainable params: 6,531,842
Non-trainable params: 0
_______________________________________________________________________________________

Mannequin adjustment

Now we are going to match and tune in our mannequin. Nonetheless, earlier than persevering with, let’s take a pattern for validation.

set.seed(1817328)
val_sample <- pattern.int(nrow(question1_padded), measurement = 0.1*nrow(question1_padded))

train_question1_padded <- question1_padded(-val_sample,)
train_question2_padded <- question2_padded(-val_sample,)
train_is_duplicate <- df$is_duplicate(-val_sample)

val_question1_padded <- question1_padded(val_sample,)
val_question2_padded <- question2_padded(val_sample,)
val_is_duplicate <- df$is_duplicate(val_sample)

Now we use the match() Operate to coach the mannequin:

mannequin %>% match(
  checklist(train_question1_padded, train_question2_padded),
  train_is_duplicate, 
  batch_size = 64, 
  epochs = 10, 
  validation_data = checklist(
    checklist(val_question1_padded, val_question2_padded), 
    val_is_duplicate
  )
)

Practice on 363861 samples, validate on 40429 samples
Epoch 1/10
363861/363861 (==============================) - 89s 245us/step - loss: 0.5860 - acc: 0.7248 - val_loss: 0.5590 - val_acc: 0.7449
Epoch 2/10
363861/363861 (==============================) - 88s 243us/step - loss: 0.5528 - acc: 0.7461 - val_loss: 0.5472 - val_acc: 0.7510
Epoch 3/10
363861/363861 (==============================) - 88s 242us/step - loss: 0.5428 - acc: 0.7536 - val_loss: 0.5439 - val_acc: 0.7515
Epoch 4/10
363861/363861 (==============================) - 88s 242us/step - loss: 0.5353 - acc: 0.7595 - val_loss: 0.5358 - val_acc: 0.7590
Epoch 5/10
363861/363861 (==============================) - 88s 242us/step - loss: 0.5299 - acc: 0.7633 - val_loss: 0.5358 - val_acc: 0.7592
Epoch 6/10
363861/363861 (==============================) - 88s 242us/step - loss: 0.5256 - acc: 0.7662 - val_loss: 0.5309 - val_acc: 0.7631
Epoch 7/10
363861/363861 (==============================) - 88s 242us/step - loss: 0.5211 - acc: 0.7701 - val_loss: 0.5349 - val_acc: 0.7586
Epoch 8/10
363861/363861 (==============================) - 88s 242us/step - loss: 0.5173 - acc: 0.7733 - val_loss: 0.5278 - val_acc: 0.7667
Epoch 9/10
363861/363861 (==============================) - 88s 242us/step - loss: 0.5138 - acc: 0.7762 - val_loss: 0.5292 - val_acc: 0.7667
Epoch 10/10
363861/363861 (==============================) - 88s 242us/step - loss: 0.5092 - acc: 0.7794 - val_loss: 0.5313 - val_acc: 0.7654

After the coaching is accomplished, we are able to save our mannequin for inference with the save_model_hdf5()
operate.

save_model_hdf5(mannequin, "model-question-pairs.hdf5")

Mannequin adjustment

Now that we’ve an affordable mannequin, let’s tune the hyperparameters utilizing the
Tfruns package deal. We are going to begin by including FLAGS Declarations to our script for all of the hyperparameters that we need to tune in (FLAGS Allow us to fluctuate hyperparparma of the hyperparma with out altering our supply code):

FLAGS <- flags(
  flag_integer("vocab_size", 50000),
  flag_integer("max_len_padding", 20),
  flag_integer("embedding_size", 256),
  flag_numeric("regularization", 0.0001),
  flag_integer("seq_embedding_size", 512)
)

With this FLAGS Definition We will now write our code when it comes to the flags. For instance:

input1 <- layer_input(form = c(FLAGS$max_len_padding))
input2 <- layer_input(form = c(FLAGS$max_len_padding))

embedding <- layer_embedding(
  input_dim = FLAGS$vocab_size + 2, 
  output_dim = FLAGS$embedding_size, 
  input_length = FLAGS$max_len_padding, 
  embeddings_regularizer = regularizer_l2(l = FLAGS$regularization)
)

The whole supply code with FLAGS It may be discovered right here.

As well as, we add a return of early arrest name within the coaching step to cease coaching if the lack of validation doesn’t lower for five occasions in a row. With luck, this can scale back coaching time for dangerous fashions. We additionally add a studying price reducer to scale back the training price by an element of 10 when the loss doesn’t lower for 3 occasions (this method usually will increase the precision of the mannequin).

mannequin %>% match(
  ...,
  callbacks = checklist(
    callback_early_stopping(endurance = 5),
    callback_reduce_lr_on_plateau(endurance = 3)
  )
)

Now we are able to execute an adjustment execution to probe for the optimum mixture of hyperparameters. We name al tuning_run() operate, passing an inventory with the potential values ​​for every indicator. He tuning_run() The operate might be accountable for executing the script for all hyperparameters combos. We additionally specify the pattern Parameter to coach the mannequin just for a random pattern of all combos (considerably lowering coaching time).

library(tfruns)

runs <- tuning_run(
  "question-pairs.R", 
  flags = checklist(
    vocab_size = c(30000, 40000, 50000, 60000),
    max_len_padding = c(15, 20, 25),
    embedding_size = c(64, 128, 256),
    regularization = c(0.00001, 0.0001, 0.001),
    seq_embedding_size = c(128, 256, 512)
  ), 
  runs_dir = "tuning", 
  pattern = 0.2
)

The adjustment profession will return a information.body with outcomes for all executions. We found that one of the best execution reached an 84.9% accuracy utilizing the mix of hyperparameters proven under, so we modify our coaching script to make use of these values ​​equivalent to default values:

FLAGS <- flags(
  flag_integer("vocab_size", 50000),
  flag_integer("max_len_padding", 20),
  flag_integer("embedding_size", 256),
  flag_numeric("regularization", 1e-4),
  flag_integer("seq_embedding_size", 512)
)

Making predictions

Now that we’ve skilled and tuned our mannequin, we are able to begin making predictions. On the time of prediction we are going to load each the textual content tokenizer and the mannequin we carry on the disc earlier than.

library(keras)
mannequin <- load_model_hdf5("model-question-pairs.hdf5", compile = FALSE)
tokenizer <- load_text_tokenizer("tokenizer-question-pairs")

As we is not going to proceed coaching the mannequin, we specify the compile = FALSE argument.

Now let’s outline a operate to create predictions. On this operate, we prepay the entry information in the identical method that we prept the coaching information:

predict_question_pairs <- operate(mannequin, tokenizer, q1, q2) {
  q1 <- texts_to_sequences(tokenizer, checklist(q1))
  q2 <- texts_to_sequences(tokenizer, checklist(q2))
  
  q1 <- pad_sequences(q1, 20)
  q2 <- pad_sequences(q2, 20)
  
  as.numeric(predict(mannequin, checklist(q1, q2)))
}

Now we are able to name it with new pairs of questions, for instance:

predict_question_pairs(
  mannequin,
  tokenizer,
  "What's R programming?",
  "What's R in programming?"
)

(1) 0.9784008

The prediction is kind of quick (~ 40 milliseconds).

Implement the mannequin

To reveal the implementation of the skilled mannequin, we create a easy Brilliant Software, the place you’ll be able to paste 2 Quora Questions and discover the likelihood of doubleing. Attempt to change the questions under or enter two fully completely different questions.

The sensible utility may be present in https://jjallaire.shinyapps.io/shiny-quora/ And it’s the supply code in https://github.com/dfalbel/shiny-quora-question Pat.

Needless to say by implementing a KERAS mannequin, you simply have to load the beforehand saved mannequin and tokenizer file (coaching information or mannequin coaching steps aren’t required).

Conclude

• We practice a Siamese LSTM that provides us affordable precision (84%). Quora’s cutting-edge is 87%.
• We will enhance our mannequin through the use of phrases beforehand skilled in bigger information units. For instance, attempt to use what’s described in This instance. Quora makes use of its personal full corpus to coach the phrases inlays.
• After coaching, we implement our mannequin as a superb utility that provides two quora questions calculates the likelihood that your duplicates.