In this project, you'll classify images from the CIFAR-10 dataset. The dataset consists of airplanes, dogs, cats, and other objects. You'll preprocess the images, then train a convolutional neural network on all the samples. The images need to be normalized and the labels need to be one-hot encoded. You'll get to apply what you learned and build a convolutional, max pooling, dropout, and fully connected layers. At the end, you'll get to see your neural network's predictions on the sample images.
Run the following cell to download the CIFAR-10 dataset for python.
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import problem_unittests as tests
import tarfile
cifar10_dataset_folder_path = 'cifar-10-batches-py'
class DLProgress(tqdm):
last_block = 0
def hook(self, block_num=1, block_size=1, total_size=None):
self.total = total_size
self.update((block_num - self.last_block) * block_size)
self.last_block = block_num
if not isfile('cifar-10-python.tar.gz'):
with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:
urlretrieve(
'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',
'cifar-10-python.tar.gz',
pbar.hook)
if not isdir(cifar10_dataset_folder_path):
with tarfile.open('cifar-10-python.tar.gz') as tar:
tar.extractall()
tar.close()
tests.test_folder_path(cifar10_dataset_folder_path)
The dataset is broken into batches to prevent your machine from running out of memory. The CIFAR-10 dataset consists of 5 batches, named data_batch_1
, data_batch_2
, etc.. Each batch contains the labels and images that are one of the following:
Understanding a dataset is part of making predictions on the data. Play around with the code cell below by changing the batch_id
and sample_id
. The batch_id
is the id for a batch (1-5). The sample_id
is the id for a image and label pair in the batch.
Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?". Answers to questions like these will help you preprocess the data and end up with better predictions.
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import helper
import numpy as np
from matplotlib import pyplot as plt
# Explore the dataset
for i in range(1,6):
batch_id = i
sample_id = 5
plt.figure()
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)
print("All possible labels: ", helper._load_label_names())
def analyze_image_data(batch_id):
features, labels = helper.load_cfar10_batch(cifar10_dataset_folder_path, batch_id)
sample_id = np.random.randint(0, len(features))
sample_image = features[sample_id]
print("Sample image shape:", sample_image.shape)
print("Features min %d, max %d" % (features.min(), features.max()))
plt.figure()
plt.axis('off')
plt.title('Sample Image')
plt.imshow(sample_image)
colors = ['red', 'green', 'blue']
for color in range(3):
image = np.copy(sample_image)
selector = [x for x in range(3) if x != color]
image[:,:,[selector]] = 0
plt.figure()
plt.axis('off')
plt.title('Image using only ' + colors[color])
plt.imshow(image)
for i in range(1,6):
analyze_image_data(i)
Answer: Based on the result above, I would infer that the images are 28x28 pixels and 3-channel RGB (0 to 255 on each channel). It could be any 3 color space, but since RGB is the most common and I can eyeball a slightly difference between tones on the single colored images, I'm guessing it is RGB.
def analyze_label_order(batch_id):
features, labels = helper.load_cfar10_batch(cifar10_dataset_folder_path, batch_id)
print("First 20 labels on batch", batch_id, np.array(helper._load_label_names())[[labels[:20]]])
for i in range(1,6):
analyze_label_order(i)
Answer: The labels are not ordered.
def normalize(x):
"""
Normalize a list of sample image data in the range of 0 to 1
: x: List of image data. The image shape is (32, 32, 3)
: return: Numpy array of normalize data
"""
return x / 255
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)
Just like the previous code cell, you'll be implementing a function for preprocessing. This time, you'll implement the one_hot_encode
function. The input, x
, are a list of labels. Implement the function to return the list of labels as One-Hot encoded Numpy array. The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode
. Make sure to save the map of encodings outside the function.
Hint:
Look into LabelBinarizer in the preprocessing module of sklearn.
from sklearn.preprocessing import LabelBinarizer
def one_hot_encode(x):
"""
One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
: x: List of sample Labels
: return: Numpy array of one-hot encoded labels
"""
encoder = LabelBinarizer()
encoder.fit(range(10))
return encoder.transform(x)
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)
As you saw from exploring the data above, the order of the samples are randomized. It doesn't hurt to randomize it again, but you don't need to for this dataset.
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)
This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper
# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))
For the neural network, you'll build each layer into a function. Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function. This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.
Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section. TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.
However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the
conv2d
class, tf.layers.conv2d, you would want to use the TF Neural Network version ofconv2d
, tf.nn.conv2d.
Let's begin!
The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions
neural_net_image_input
image_shape
with batch size set to None
.name
parameter in the TF Placeholder.neural_net_label_input
n_classes
with batch size set to None
.name
parameter in the TF Placeholder.neural_net_keep_prob_input
name
parameter in the TF Placeholder.These names will be used at the end of the project to load your saved model.
Note: None
for shapes in TensorFlow allow for a dynamic size.
import tensorflow as tf
import numpy as np
def neural_net_image_input(image_shape):
"""
Return a Tensor for a batch of image input
: image_shape: Shape of the images
: return: Tensor for image input.
"""
return tf.placeholder(tf.float32, shape=(None,) + image_shape, name="x")
def neural_net_label_input(n_classes):
"""
Return a Tensor for a batch of label input
: n_classes: Number of classes
: return: Tensor for label input.
"""
return tf.placeholder(tf.int32, shape=(None, n_classes), name="y")
def neural_net_keep_prob_input():
"""
Return a Tensor for keep probability
: return: Tensor for keep probability.
"""
return tf.placeholder(tf.float32, name="keep_prob")
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)
Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool
to apply convolution then max pooling:
conv_ksize
, conv_num_outputs
and the shape of x_tensor
.x_tensor
using weight and conv_strides
.pool_ksize
and pool_strides
.Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer, but you can still use TensorFlow's Neural Network package. You may still use the shortcut option for all the other layers.
Hint:
When unpacking values as an argument in Python, look into the unpacking operator.
def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
"""
Apply convolution then max pooling to x_tensor
:param x_tensor: TensorFlow Tensor
:param conv_num_outputs: Number of outputs for the convolutional layer
:param conv_ksize: kernal size 2-D Tuple for the convolutional layer
:param conv_strides: Stride 2-D Tuple for convolution
:param pool_ksize: kernal size 2-D Tuple for pool
:param pool_strides: Stride 2-D Tuple for pool
: return: A tensor that represents convolution and max pooling of x_tensor
"""
in_shape = x_tensor.shape.as_list()[-1]
# Weights and biases
conv_weights = tf.Variable(tf.truncated_normal([*conv_ksize, in_shape, conv_num_outputs],
mean=0.0, stddev=1.0 / in_shape, dtype=tf.float32))
conv_biases = tf.Variable(tf.zeros([conv_num_outputs]))
# Convolution and adding biases
conv = tf.nn.conv2d(x_tensor, conv_weights, [1, *conv_strides, 1], "SAME")
pre_activation = tf.nn.bias_add(conv, conv_biases)
# Activation
activation = tf.nn.relu(pre_activation)
pool = tf.nn.max_pool(activation, [1, *pool_ksize, 1], [1, *pool_strides, 1], "SAME")
return pool
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)
Implement the flatten
function to change the dimension of x_tensor
from a 4-D tensor to a 2-D tensor. The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
def flatten(x_tensor):
"""
Flatten x_tensor to (Batch Size, Flattened Image Size)
: x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
: return: A tensor of size (Batch Size, Flattened Image Size).
"""
s = [-1 if x == None else x for x in x_tensor.get_shape().as_list()]
return tf.reshape(x_tensor, [s[0], s[1] * s[2] * s[3]])
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)
Implement the fully_conn
function to apply a fully connected layer to x_tensor
with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
def fully_conn(x_tensor, num_outputs):
"""
Apply a fully connected layer to x_tensor using weight and bias
: x_tensor: A 2-D tensor where the first dimension is batch size.
: num_outputs: The number of output that the new tensor should be.
: return: A 2-D tensor where the second dimension is num_outputs.
"""
in_shape = x_tensor.shape.as_list()[-1]
w = tf.Variable(tf.truncated_normal([in_shape, num_outputs],
mean=0.0, stddev=1.0 / in_shape, dtype=tf.float32))
b = tf.Variable(tf.zeros([num_outputs]))
pre_activation = tf.matmul(x_tensor, w) + b
return tf.nn.relu(pre_activation)
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)
Implement the output
function to apply a fully connected layer to x_tensor
with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Note: Activation, softmax, or cross entropy should not be applied to this.
def output(x_tensor, num_outputs):
"""
Apply a output layer to x_tensor using weight and bias
: x_tensor: A 2-D tensor where the first dimension is batch size.
: num_outputs: The number of output that the new tensor should be.
: return: A 2-D tensor where the second dimension is num_outputs.
"""
in_shape = x_tensor.shape.as_list()[-1]
w = tf.Variable(tf.truncated_normal([in_shape, num_outputs],
mean=0.0, stddev=1.0 / in_shape, dtype=tf.float32))
b = tf.Variable(tf.zeros([num_outputs]))
return tf.matmul(x_tensor, w) + b
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_output(output)
Implement the function conv_net
to create a convolutional neural network model. The function takes in a batch of images, x
, and outputs logits. Use the layers you created above to create this model:
keep_prob
. NUM_CLASSES=10
def karpathy_conv_net(x, keep_prob):
# https://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html
# ~51% accuracy on 1 batch
# ~57% accuracy on 5 batch
# ~57% accuracy on test
x = conv2d_maxpool(x, 3, (5, 5), (1, 1), (2, 2), (2, 2))
x = conv2d_maxpool(x, 16, (5, 5), (1, 1), (2, 2), (2, 2))
x = conv2d_maxpool(x, 20, (5, 5), (1, 1), (2, 2), (2, 2))
x = flatten(x)
x = output(x, 10)
return x
def eric_conv_net(x, keep_prob):
# https://discussions.udacity.com/t/image-classification-not-working/231425/3
# ~51% accuracy on 1 batch
# ~60% accuracy on 5 batch
# ~58% accuracy on test
x = conv2d_maxpool(x, 96, (3,3), (1,1), (2,2), (2,2))
x = flatten(x)
x = fully_conn(x, 512)
x = output(x, 10)
return x
def gopal_conv_net(x, keep_prob):
# https://discussions.udacity.com/t/network-does-not-learn-no-2/231391/29
# ~52% accuracy on 1 batch
# ~43% accuracy on 5 batch
# ~42% accuracy on test
x = conv2d_maxpool(x, 16, (4, 4), (1, 1), (2, 2), (2, 2))
x = conv2d_maxpool(x, 32, (4, 4), (1, 1), (2, 2), (2, 2))
x = conv2d_maxpool(x, 64, (4, 4), (1, 1), (2, 2), (2, 2))
x = flatten(x)
x = fully_conn(x, 10)
x = tf.nn.dropout(x, keep_prob)
x = fully_conn(x, 10)
x = tf.nn.dropout(x, keep_prob)
x = output(x, 10)
return x
def my_conv_net(x, keep_prob):
# A mix of karpathy's and gopal's architectures
# ~53% accuracy on 1 batch
# ~51% accuracy on 5 batch
# ~51% accuracy on test
x = conv2d_maxpool(x, 8, (5, 5), (1, 1), (2, 2), (2, 2))
x = conv2d_maxpool(x, 16, (5, 5), (1, 1), (2, 2), (2, 2))
x = conv2d_maxpool(x, 32, (5, 5), (1, 1), (2, 2), (2, 2))
x = flatten(x)
x = fully_conn(x, 512)
x = tf.nn.dropout(x, keep_prob)
x = output(x, 10)
return x
def conv_net(x, keep_prob):
return my_conv_net(x, keep_prob)
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
##############################
## Build the Neural Network ##
##############################
# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()
# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()
# Model
logits = conv_net(x, keep_prob)
# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')
# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)
tf.train.AdadeltaOptimizer()
# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')
tests.test_conv_net(conv_net)
Implement the function train_neural_network
to do a single optimization. The optimization should use optimizer
to optimize in session
with a feed_dict
of the following:
x
for image inputy
for labelskeep_prob
for keep probability for dropoutThis function will be called for each batch, so tf.global_variables_initializer()
has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
"""
Optimize the session on a batch of images and labels
: session: Current TensorFlow session
: optimizer: TensorFlow optimizer function
: keep_probability: keep probability
: feature_batch: Batch of Numpy image data
: label_batch: Batch of Numpy label data
"""
feed_dict = {x : feature_batch, y : label_batch, keep_prob : keep_probability}
session.run([optimizer], feed_dict=feed_dict)
pass
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)
Implement the function print_stats
to print loss and validation accuracy. Use the global variables valid_features
and valid_labels
to calculate validation accuracy. Use a keep probability of 1.0
to calculate the loss and validation accuracy.
def print_stats(session, feature_batch, label_batch, cost, accuracy):
"""
Print information about loss and validation accuracy
: session: Current TensorFlow session
: feature_batch: Batch of Numpy image data
: label_batch: Batch of Numpy label data
: cost: TensorFlow cost function
: accuracy: TensorFlow accuracy function
"""
feed_dict = { x: feature_batch, y: label_batch, keep_prob: 1.0 }
c = session.run([cost], feed_dict=feed_dict)
feed_dict = { x: valid_features, y: valid_labels, keep_prob: 1.0 }
a = session.run([accuracy], feed_dict=feed_dict)
print("Cost: {:.3f}, Accuracy: {:.3f}%".format(c[0], 100*a[0]))
pass
Tune the following parameters:
epochs
to the number of iterations until the network stops learning or start overfittingbatch_size
to the highest number that your machine has memory for. Most people set them to common sizes of memory:keep_probability
to the probability of keeping a node using dropoutepochs = 50
batch_size = 256
keep_probability = 0.9
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy. Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
# Initializing the variables
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(epochs):
batch_i = 1
for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
print('Epoch {:>2}, CIFAR-10 Batch {}: '.format(epoch + 1, batch_i), end='')
print_stats(sess, batch_features, batch_labels, cost, accuracy)
Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'
print('Training...')
with tf.Session() as sess:
# Initializing the variables
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(epochs):
# Loop over all batches
n_batches = 5
for batch_i in range(1, n_batches + 1):
for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
print('Epoch {:>2}, CIFAR-10 Batch {}: '.format(epoch + 1, batch_i), end='')
print_stats(sess, batch_features, batch_labels, cost, accuracy)
# Save Model
saver = tf.train.Saver()
save_path = saver.save(sess, save_model_path)
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import tensorflow as tf
import pickle
import helper
import random
# Set batch size if not already set
try:
if batch_size:
pass
except NameError:
batch_size = 64
save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3
def test_model():
"""
Test the saved model against the test dataset
"""
test_features, test_labels = pickle.load(open('preprocess_training.p', mode='rb'))
loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
# Load model
loader = tf.train.import_meta_graph(save_model_path + '.meta')
loader.restore(sess, save_model_path)
# Get Tensors from loaded model
loaded_x = loaded_graph.get_tensor_by_name('x:0')
loaded_y = loaded_graph.get_tensor_by_name('y:0')
loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')
# Get accuracy in batches for memory limitations
test_batch_acc_total = 0
test_batch_count = 0
for train_feature_batch, train_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
test_batch_acc_total += sess.run(
loaded_acc,
feed_dict={loaded_x: train_feature_batch, loaded_y: train_label_batch, loaded_keep_prob: 1.0})
test_batch_count += 1
print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))
# Print Random Samples
random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
random_test_predictions = sess.run(
tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)
test_model()
You might be wondering why you can't get an accuracy any higher. First things first, 50% isn't bad for a simple CNN. Pure guessing would get you 10% accuracy. That's because there are many more techniques that can be applied to your model and we recemmond that once you are done with this project, you explore!
When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "image_classification.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.