Spelling Error Correction System

This blog discusses the detailed explanation of the end-to-end creation and deployment of a spelling error correction system which can be implemented using any text file as a dataset and achieved a BLEU score of 0.8 on the test dataset.

Table of Contents:

  1. Introduction
  2. Dataset
  3. Exploratory Data Analysis(EDA)
  4. Pre-Processing
  5. Modeling
  6. Post-Processing
  7. Results and Deployment
  8. Future work
  9. Profile
  10. References

1. Introduction :

Spelling error correction is the task of automatically correcting spelling errors in text; e.g. [I followed his advice -> I followed his advice]. It can be used not only to help language learners improve their writing errors, but also alert native speakers to accidental mistakes or typos. The system can be implemented using various different LSTM architectures.

In this task, we have to build a spelling error correction system which can predict correctly spelled sentence given a sentence.

Business Problem: By building an automated spelling error correction system. We can create automated tools for writing English scientific texts, filtering out sentences that need spelling improvements, evaluating articles, etc. Currently, the mentioned task is done manually so, by automating this process we can save both time and money for the company.

ML Formulation: Building n-grams models using architectures like Encoder-Decoder architecture, Bidirectional LSTM with attention mechanism, etc. can be used.

Performance Metric: The performance metric for the task is the Bilingual Evaluation Understudy(BLEU) score. The metric compares machine translated sentence to one or more reference sequences. The metric ranges on a scale of 0 to 1, in an attempt to measure the adequacy and fluency of the machine translation(MT) output. The more overlap there is with their human reference translations and thus, the better the translation.

The BLEU is a programming task to compare n-grams of the translated sentences with the reference sentence and count the number of matches. The more the matches the better the performance.

2. Dataset :

The dataset for the task can be created by using any text file. For creating a dataset we need to read a text file and create an n-gram dataset of text files. These generated n-words need to be passed to a function so that we can add noise in the n-gram sentence which can be corrected by the model.

Consider an example of the text file that has the sentence: “I will be there tomorrow evening.” Suppose we are applying a 3-gram model. This text file has 3-grams like “I will be”, “will be there”, be there tomorrow”, and “there tomorrow evening”. This 3-gram word is passed through a function so that we can introduce noise in the 3-grams which will help the model to train. After introducing noise in 3-grams the input and output will be as follows:

Dataset before(INPUT) and after(OUTPUT) noise introduction.

The dataset size can be increased by combining n number of text files so that we could get high performance.

The noise in the words can be introduced using the mentioned code. The code is implemented using 4 methods as follows:

  1. Replace a character with a random character
  2. Delete a character.
  3. Add a random character.
  4. Transpose 2 characters.
def add_speling_errors(token, error_rate, VOCAB):
"""Add some artificial spelling mistakes."""
assert(0.0 <= error_rate < 1.0)
if len(token) < 3:
return token
rand = np.random.rand()
# Here are 4 different ways spelling mistakes can occur,
# each of which has equal chance.
prob = error_rate / 4.0
if rand < prob:
# Replace a character with a random character.
random_char_index = np.random.randint(len(token))
token = token[:random_char_index] + np.random.choice(VOCAB)\
+ token[random_char_index + 1:]
elif prob < rand < prob * 2:
# Delete a character.
random_char_index = np.random.randint(len(token))
token = token[:random_char_index]+token[random_char_inde+1:]
elif prob * 2 < rand < prob * 3:
# Add a random character.
random_char_index = np.random.randint(len(token))
token = token[:random_char_index] + np.random.choice(VOCAB)\
+ token[random_char_index:]
elif prob * 3 < rand < prob * 4:
# Transpose 2 characters.
random_char_index = np.random.randint(len(token) - 1)
token = token[:random_char_index]+token[random_char_inde+1]\
+token[random_char_index]+token[random_char_index+2:]
else:
# No spelling errors.
pass
return token

We have created a dataset by using 1,2, and 3-gram words by combining 4 text files which are available here.

3. Exploratory Data Analysis (EDA) :

In this section, we will try to analyze the created dataset and will answer the questions like

  1. What is the length of the n-gram word input?
  2. What is the vocab and vocab size that used in the n-grams?
  3. Null values in the dataset?

We have calculated the length 1,2, and 3-gram word so that we can choose the maximum length of the input.

PDF for length n-gram words

we have chosen 20, 24, and 32 as the maximum length for unigram, bigram, and trigram words as there are 99.9% of n-words that have a length less than mentioned lengths.

The vocab and vocab size is the same for bigram and trigram but different for unigram. The only difference between both the vocabs is the space character as unigram word has no space and other grams words has space between them.

UNIGRAM_VOCAB = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '<SOW>', '<EOW>']NGRAM_VOCAB = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', ' ', '<SOW>', '<EOW>']

There are no null values in the dataset.

4. Pre-Processing :

In this section, we will preprocess the text dataset by transforming it from text to numerical values so that it can be fitted to the model. Before transforming the text into numerical values we will add <SOW> and <EOW> tokens in n-gram word here SOW stands for the start of word and EOW stands for the end of the word.

Now, we will create a TensorFlow data generator for the efficient pipeline. In this pipeline, we will take text as input and convert the text into numerical values, and also pad the text to max length. For this conversion, I will try to create tf.data.Dataset pipeline for converting the text to numerical values.

unigram_vec = TextVectorization(output_sequence_length= unigram_maxlen+2, standardize = None, split='whitespace', max_tokens = len(UNIGRAM_VOCAB)+2, output_mode='int')unigram_vec.adapt(UNIGRAM_VOCAB)unigram_index_to_word = {idx: word for idx, word in enumerate(unigram_vec.get_vocabulary())}
unigram_word_to_index = {word: idx for idx, word in enumerate(unigram_vec.get_vocabulary())}
def unigram_mapping(x):
enc_inp = unigram_vec(x[:, 2])
dec_inp = unigram_vec(x[:, 3])
dec_out = unigram_vec(x[:, 4])
return (enc_inp, dec_inp), dec_out
unigram_train_dataset = tf.data.Dataset.from_tensor_slices(unigram_train.values).repeat().batch(batch_size).map(unigram_mapping).prefetch(1)unigram_val_dataset = tf.data.Dataset.from_tensor_slices(unigram_val.values).repeat().batch(batch_size).map(unigram_mapping).prefetch(1)

The above-mentioned code creates a TextVectorization object and attaches the vocabulary used. The code also creates tf.data pipeline which maps the text using the unigram_mapping() function. The code is efficient and has less latency as per TensorFlow documentation for more information visit here.

5. Modeling :

In this section, we will discuss the basics related to the model. We will implement a character-based lstm model. In which we give a unique id of character as input to lstm cell. For each model, we try the unigram, bigram, and trigram word dataset. For modeling, we tried different models like:

i. Seq2Seq LSTM Model

ii. Seq2Seq LSTM with Attention Mechanism Model.

iii. Seq2Seq Bidirectional LSTM with Attention Mechanism Model.

Seq2Seq LSTM Model is the simple encoder-decoder LSTM architecture in which the incorrectly spelled is given as input to the encoder and the cell state generated by the encoder is given as the initial state to the decoder. The output of each cell of the decoder is given as input to the next cell of the decoder. The output of each lstm cell is captured and applied to a further dense layer of vocab size to treat the problem as a classification problem. The complete code is mentioned below:

class Encoder(tf.keras.Model):
def __init__(self,inp_vocab_size,embedding_size,lstm_size,input_length):
super().__init__()
self.lstm_size = lstm_size
#Initialize Embedding layer
self.enc_embed = Embedding(input_dim = inp_vocab_size, output_dim = embedding_size, input_length= input_length)
#Intialize Encoder LSTM layer
self.enc_lstm = LSTM(lstm_size, return_sequences = True, return_state = True, dropout = 0.4)
def call(self,input_sequence,states):
embedding = self.enc_embed(input_sequence)
output_state, enc_h, enc_c = self.enc_lstm(embedding, initial_state = states)
return output_state, enc_h, enc_c

def initialize_states(self,batch_size):
return [tf.zeros((batch_size, self.lstm_size)), tf.zeros((batch_size, self.lstm_size))]
class Decoder(tf.keras.Model):
def __init__(self,out_vocab_size,embedding_size,lstm_size,input_length):
super().__init__()
#Initialize Embedding layer
self.dec_embed = Embedding(input_dim = out_vocab_size, output_dim = embedding_size, input_length = input_length)
#Intialize Decoder LSTM layer
self.dec_lstm = LSTM(lstm_size, return_sequences = True, return_state = True, dropout = 0.4)

def call(self,input_sequence, initial_states):
embedding = self.dec_embed(input_sequence)
output_state, dec_h, dec_c = self.dec_lstm(embedding, initial_state = initial_states)
return output_state, dec_h, dec_c
class Encoder_decoder(tf.keras.Model):
def __init__(self,*params):
super().__init__()
#Create encoder object
self.encoder = Encoder(inp_vocab_size = params[0], embedding_size = params[2], lstm_size = params[3], input_length = params[4])
#Create decoder object
self.decoder = Decoder(out_vocab_size = params[1], embedding_size = params[2], lstm_size = params[3], input_length = params[5])
#Intialize Dense layer(out_vocab_size) with activation='softmax'
self.dense = Dense(params[1], activation='softmax')

def call(self, params, training = True):
enc_inp, dec_inp = params[0], params[1]
# print(enc_inp, dec_inp)
initial_state = self.encoder.initialize_states(batch_size)
output_state, enc_h, enc_c = self.encoder(enc_inp, initial_state)
output, _, _ = self.decoder(dec_inp ,[enc_h, enc_c])
output = Dropout(0.5)(output)
return self.dense(output)
class pred_Encoder_decoder(tf.keras.Model):
def __init__(self,*params):
super().__init__()
#Create encoder object
self.encoder = Encoder(inp_vocab_size = params[0], embedding_size = params[2], lstm_size = params[3], input_length = params[4])
#Create decoder object
self.decoder = Decoder(out_vocab_size = params[1], embedding_size = params[2], lstm_size = params[3], input_length = params[5])
#Intialize Dense layer(out_vocab_size) with activation='softmax'
self.dense = Dense(params[1], activation='softmax')
self.word_to_index = params[6]

def call(self, params):
enc_inp = params[0]
initial_state = self.encoder.initialize_states(1)
output_state, enc_h, enc_c = self.encoder(enc_inp, initial_state)
pred = tf.expand_dims([self.word_to_index['<SOW>']], 0)
dec_h = enc_h
dec_c = enc_c
all_pred = []
for t in range(max_len):
pred, dec_h,dec_c = self.decoder(pred, [dec_h, dec_c])
pred = self.dense(pred)
pred = tf.argmax(pred, axis = -1)
all_pred.append(pred)
return all_pred

Seq2Seq LSTM with Attention Mechanism Model is same as seq2seq lstm model, the only difference is that output generated by the cells of the decoder is and the output of previous lstm cells(encoder as well as a decoder) are passed to attention layer and the output of attention layer is passed as input to next cell of the decoder. The mechanism helps the model to pay attention towards the importance of other cell's output so that it can help the model to increase the length of the input. The complete code is mentioned below:

class Encoder(tf.keras.layers.Layer):
def __init__(self,inp_vocab_size,embedding_size,lstm_size,input_length):
super(Encoder, self).__init__()
self.lstm_size = lstm_size
#Initialize Embedding layer
self.enc_embed = Embedding(input_dim = inp_vocab_size, output_dim = embedding_size)
#Intialize Encoder LSTM layer
self.enc_lstm = LSTM(lstm_size, return_sequences = True, return_state = True, dropout = 0.4)

def call(self,input_sequence,states):
embedding = self.enc_embed(input_sequence)
output_state, enc_h, enc_c = self.enc_lstm(embedding, initial_state = states)
return output_state, enc_h, enc_c

def initialize_states(self,batch_size):
return [tf.zeros((batch_size, self.lstm_size)), tf.zeros((batch_size, self.lstm_size))]
class Attention(tf.keras.layers.Layer):def __init__(self,scoring_function, att_units):
super(Attention, self).__init__()
self.scoring_function = scoring_function
if scoring_function == 'dot':
self.dot = Dot(axes = (1, 2))
elif scoring_function == 'general':
# Intialize variables needed for General score function here
self.W = Dense(att_units)
self.dot = Dot(axes = (1, 2))
elif scoring_function == 'concat':
# Intialize variables needed for Concat score function here
self.W1 = Dense(att_units)
self.W2 = Dense(att_units)
self.V = Dense(1)
def call(self,decoder_hidden_state,encoder_output):

decoder_hidden_state = tf.expand_dims(decoder_hidden_state, 1)

if self.scoring_function == 'dot':
# Implement Dot score function here
score = tf.transpose(self.dot([tf.transpose(decoder_hidden_state, (0, 2, 1)), encoder_output]), (0, 2,1))

elif self.scoring_function == 'general':
# Implement General score function here
mul = self.W(encoder_output)
score = tf.transpose(self.dot([tf.transpose(decoder_hidden_state, (0, 2, 1)), mul]), (0, 2,1))

elif self.scoring_function == 'concat':
# Implement General score function here
inter = self.W1(decoder_hidden_state) + self.W2(encoder_output)
tan = tf.nn.tanh(inter)
score = self.V(tan)
attention_weights = tf.nn.softmax(score, axis =1)
context_vector = attention_weights * encoder_output
context_vector = tf.reduce_sum(context_vector, axis=1)
return context_vector, attention_weights
class OneStepDecoder(tf.keras.layers.Layer):
def __init__(self,tar_vocab_size, embedding_dim, input_length, dec_units ,score_fun ,att_units):
super(OneStepDecoder, self).__init__()
# Initialize decoder embedding layer, LSTM and any other objects needed
self.embed_dec = Embedding(input_dim = tar_vocab_size, output_dim = embedding_dim)
self.lstm = LSTM(dec_units, return_sequences = True, return_state = True, dropout = 0.4)
self.attention = Attention(scoring_function = score_fun, att_units = att_units)
self.fc = Dense(tar_vocab_size)

def call(self,input_to_decoder, encoder_output, state_h,state_c):
embed = self.embed_dec(input_to_decoder)
context_vect, attention_weights = self.attention(state_h, encoder_output)
final_inp = tf.concat([tf.expand_dims(context_vect, 1), embed], axis = -1)
out, dec_h, dec_c = self.lstm(final_inp, [state_h, state_c])
out = tf.reshape(out, (-1, out.shape[2]))
output = self.fc(out)
output = Dropout(0.5)(output)
return output, dec_h, dec_c, attention_weights, context_vect
class encoder_decoder(tf.keras.Model):
def __init__(self, inp_vocab_size, out_vocab_size, embedding_dim, enc_units, dec_units, max_len_inp, max_len_out, score_fun, att_units, batch_size):
#Intialize objects from encoder decoder
super(encoder_decoder, self).__init__()
self.encoder = Encoder(inp_vocab_size, embedding_dim, enc_units, max_len_inp)
self.one_step_decoder = OneStepDecoder(out_vocab_size, embedding_dim, max_len_out, dec_units ,score_fun ,att_units)
self.batch_size = batch_size

def call(self, data):
enc_inp, dec_inp = data[0], data[1]
initial_state = self.encoder.initialize_states(self.batch_size)
enc_output, enc_h, enc_c = self.encoder(enc_inp, initial_state)
all_outputs = tf.TensorArray(dtype = tf.float32, size= max_len)

dec_h = enc_h
dec_c = enc_c
for timestep in range(max_len):
# Call onestepdecoder for each token in decoder_input
output, dec_h, dec_c, _, _ = self.one_step_decoder(dec_inp[:, timestep:timestep+1],
enc_output,
dec_h,
dec_c)
# Store the output in tensorarray
all_outputs = all_outputs.write(timestep, output)
# Return the tensor array
all_outputs = tf.transpose(all_outputs.stack(), (1, 0, 2))
# return the decoder output
return all_outputs
class pred_Encoder_decoder(tf.keras.Model):
def __init__(self, inp_vocab_size, out_vocab_size, embedding_dim, enc_units, dec_units, max_len_ita, max_len_eng, score_fun, att_units, word_to_index):
#Intialize objects from encoder decoder
super(pred_Encoder_decoder, self).__init__()
self.encoder = Encoder(inp_vocab_size, embedding_dim, enc_units, max_len_ita)
self.one_step_decoder = OneStepDecoder(out_vocab_size, embedding_dim, max_len_eng, dec_units ,score_fun ,att_units)
self.batch_size = batch_size
self.word_to_index = word_to_index
def call(self, params):
enc_inp = params[0]
initial_state = self.encoder.initialize_states(1)
output_state, enc_h, enc_c = self.encoder(enc_inp, initial_state)
pred = tf.expand_dims([self.word_to_index['<SOW>']], 0)
dec_h = enc_h
dec_c = enc_c
all_pred = []
all_attention = []
for t in range(max_len):
output, dec_h,dec_c, attention, _ = self.one_step_decoder(pred, output_state, dec_h, dec_c)
pred = tf.argmax(output, axis = -1)
all_pred.append(pred)
pred = tf.expand_dims(pred, 0)
all_attention.append(attention)
return all_pred, all_attention

Seq2Seq Bidirectional LSTM with Attention Mechanism Model is same as seq2seq lstm with attention mechanism model the only difference is, in that model, we are using unidirectional lstm model which doesn't help in providing the context for the character but when we use Bidirectional lstm it helps the model in providing the context related to previous and later occurring characters. The complete code is mentioned below:

class Encoder(tf.keras.layers.Layer):
def __init__(self, vocab_size, embedding_size, lstm_size, input_length):
super(Encoder, self).__init__()
self.lstm_size = lstm_size
self.enc_embed = Embedding(input_dim = vocab_size, output_dim = embedding_size)
self.enc_lstm = Bidirectional(LSTM(lstm_size, return_sequences = True, return_state = True, dropout = 0.4))

def call(self, input_sequence, states):
embedding = self.enc_embed(input_sequence)
output_state, enc_frwd_h, enc_frwd_c, enc_bkwd_h, enc_bkwd_c = self.enc_lstm(embedding, initial_state = states)
return output_state, enc_frwd_h, enc_frwd_c, enc_bkwd_h, enc_bkwd_c

def initialize_states(self, batch_size):
return [tf.zeros((batch_size, self.lstm_size)), tf.zeros((batch_size, self.lstm_size)),
tf.zeros((batch_size, self.lstm_size)), tf.zeros((batch_size, self.lstm_size))]
class Attention(tf.keras.layers.Layer):
def __init__(self,scoring_function, att_units):
super(Attention, self).__init__()
self.scoring_function = scoring_function
if scoring_function == 'dot':
self.dot = Dot(axes = (1, 2))
elif scoring_function == 'general':
self.W = Dense(att_units)
self.dot = Dot(axes = (1, 2))
elif scoring_function == 'concat':
self.W1 = Dense(att_units)
self.W2 = Dense(att_units)
self.W3 = Dense(att_units)
self.V = Dense(1)

def call(self, dec_frwd_state, dec_bkwd_state, encoder_output):
dec_frwd_state = tf.expand_dims(dec_frwd_state, 1)
dec_bkwd_state = tf.expand_dims(dec_bkwd_state, 1)
#
if self.scoring_function == 'dot':
score = tf.transpose(self.dot([tf.transpose(decoder_hidden_state, (0, 2, 1)), encoder_output]), (0, 2,1))
elif self.scoring_function == 'general':
mul = self.W(encoder_output)
score = tf.transpose(self.dot([tf.transpose(decoder_hidden_state, (0, 2, 1)), mul]), (0, 2,1))
elif self.scoring_function == 'concat':
inter = self.W1(dec_frwd_state) + self.W2(dec_bkwd_state) + self.W3(encoder_output)
tan = tf.nn.tanh(inter)
score = self.V(tan)
attention_weights = tf.nn.softmax(score, axis =1)
context_vector = attention_weights * encoder_output
context_vector = tf.reduce_sum(context_vector, axis=1)
return context_vector, attention_weights
class OneStepDecoder(tf.keras.layers.Layer):
def __init__(self, vocab_size, embedding_dim, input_length, dec_units ,score_fun ,att_units):
super(OneStepDecoder, self).__init__()
# Initialize decoder embedding layer, LSTM and any other objects needed
self.embed_dec = Embedding(input_dim = vocab_size, output_dim = embedding_dim)
self.lstm = Bidirectional(LSTM(dec_units, return_sequences = True, return_state = True, dropout = 0.4))
self.attention = Attention(scoring_function = score_fun, att_units = att_units)
self.fc = Dense(vocab_size)

def call(self,input_to_decoder, encoder_output, state_frwd_h, state_frwd_c, state_bkwd_h, state_bkwd_c):
embed = self.embed_dec(input_to_decoder)
context_vect, attention_weights = self.attention(state_frwd_h, state_bkwd_h, encoder_output)
final_inp = tf.concat([tf.expand_dims(context_vect, 1), embed], axis = -1)
out, dec_frwd_h, dec_frwd_c, dec_bkwd_h, dec_bkwd_c = self.lstm(final_inp, [state_frwd_h, state_frwd_c, state_bkwd_h, state_bkwd_c])
out = tf.reshape(out, (-1, out.shape[2]))
out = Dropout(0.5)(out)
output = self.fc(out)
return output, dec_frwd_h, dec_frwd_c, dec_bkwd_h, dec_bkwd_c, attention_weights, context_vect
class encoder_decoder(tf.keras.Model):
def __init__(self, vocab_size, embedding_dim, enc_units, dec_units, max_len, score_fun, att_units, batch_size):
#Intialize objects from encoder decoder
super(encoder_decoder, self).__init__()
self.encoder = Encoder(vocab_size, embedding_dim, enc_units, max_len)
self.one_step_decoder = OneStepDecoder(vocab_size, embedding_dim, max_len, dec_units ,score_fun ,att_units)
self.batch_size = batch_size

def call(self, data):
enc_inp, dec_inp = data[0], data[1]
initial_state = self.encoder.initialize_states(self.batch_size)
enc_output, enc_frwd_h, enc_frwd_c, enc_bkwd_h, enc_bkwd_c = self.encoder(enc_inp, initial_state)
all_outputs = tf.TensorArray(dtype = tf.float32, size= max_len)

dec_frwd_h = enc_frwd_h
dec_frwd_c = enc_frwd_c
dec_bkwd_h = enc_bkwd_h
dec_bkwd_c = enc_bkwd_c
for timestep in range(max_len):
# Call onestepdecoder for each token in decoder_input
output, dec_frwd_h, dec_frwd_c, dec_bkwd_h, dec_bkwd_c, _, _ = self.one_step_decoder(dec_inp[:, timestep:timestep+1], enc_output, dec_frwd_h, dec_frwd_c, dec_bkwd_h, dec_bkwd_c)
# Store the output in tensorarray
all_outputs = all_outputs.write(timestep, output)
# Return the tensor array
all_outputs = tf.transpose(all_outputs.stack(), (1, 0, 2))
# return the decoder output
return all_outputs

class pred_Encoder_decoder(tf.keras.Model):
def __init__(self, inp_vocab_size, out_vocab_size, embedding_dim, enc_units, dec_units, max_len_ita, max_len_eng, score_fun, att_units):
#Intialize objects from encoder decoder
super(pred_Encoder_decoder, self).__init__()
self.encoder = Encoder(inp_vocab_size, embedding_dim, enc_units, max_len_ita)
self.one_step_decoder = OneStepDecoder(out_vocab_size, embedding_dim, max_len_eng, dec_units, score_fun, att_units)

def call(self, params):
enc_inp = params[0]
initial_state = self.encoder.initialize_states(1)
enc_output, enc_frwd_h, enc_frwd_c, enc_bkwd_h, enc_bkwd_c = self.encoder(enc_inp, initial_state)
pred = tf.expand_dims([word_to_index['<SOW>']], 0)
all_pred = []
all_attention = []
dec_frwd_h = enc_frwd_h
dec_frwd_c = enc_frwd_c
dec_bkwd_h = enc_bkwd_h
dec_bkwd_c = enc_bkwd_c
for timestep in range(max_len):
# Call onestepdecoder for each token in decoder_input
output, dec_frwd_h, dec_frwd_c, dec_bkwd_h, dec_bkwd_c, attention, _ = self.one_step_decoder(pred, enc_output, dec_frwd_h, dec_frwd_c, dec_bkwd_h, dec_bkwd_c)
pred = tf.argmax(output, axis = -1)
all_pred.append(pred)
pred = tf.expand_dims(pred, 0)
all_attention.append(attention)
return all_pred, all_attention

6. Post-Processing :

In this section, we will try to apply quantization so, that we can reduce our model size and latency so that it can be made compatible with edge devices like mobile, drone, etc.

We also tried to apply quantization but the model size was increased 31MB to 39MB in our case and documentation also says that the size of the model slightly increases by decreasing latency. But as we were more concerned with the size we dropped the idea of quantization. The code used is as mentioned below :

converter = tf.lite.TFLiteConverter.from_keras_model(pred_model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
quant_model = converter.convert()
filename = 'quant_model.tflite'
with open(filename, 'wb') as file:
file.write(quant_model)

7. Results and Deployment :

After training 9 models we have achieved the best BLEU score with 3-gram Seq2Seq Bidirectional LSTM with Attention Mechanism Model. The results for all the models are available here:

BLEU Score for all the trained models.

The model is deployed using the flask framework and also been dockerized for avoiding portability issues. The complete code is available here:

8. Future work :

Further results can be improved by adding LSTM layers in the current architecture and also we can use different architectures like a transformer and for training and BERT for word embedding.

9. Profile :

For complete code visit ipynb notebook :

Stay connected with me on LinkedIn:

10. References :

  1. https://arxiv.org/pdf/1902.07178.pdf
  2. https://arxiv.org/pdf/2010.11085v1.pdf
  3. http://www.realworldnlpbook.com/blog/unreasonable-effectiveness-oftransformer-spell-checker.html
  4. https://machinelearnings.co/deep-spelling-9ffef96a24f6
  5. https://github.com/vuptran/deep-spell-checkr/tree/92344d58ea
  6. https://www.aclweb.org/anthology/P15-2097.pdf
  7. https://www.tensorflow.org/lite/performance/post_training_quantization
  8. AppliedAiCourse

I hope this blog is helpful.

AI and ML Enthusiast

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store