基本的TF functions, 覆盖了NN的基本的流程，initialization, activation function, regularization, 并以经典的MNIST 为例走一遍。先define model，不变的比如input，labels 设为tf.placeholder(), 需要update的比如weights和bias设为tf.Variables()， 然后定义calculation，最后用tf.Session().run()计算，赋值需要用到feed_dict

Basics of TensorFlow

Ran operations intf.Session

Create a constant tensor tf.constant()

Get input tf.placeholder() 用于函数run里定义and feed_dict用于tf.Session.run()赋值

定义变量 tf.Variable()

Basic math tf.add(), tf.subtract(), tf.multiply(), tf.divide()

Convert data type tf.cast()

Linear Function in TF

Inputs, matrix of the weights and biases

$$ y = xW+b$$

Weights $W$ and bias $b$ need to be a Tensor that can be modified-> tf.Variable

Matrix multiplication tf.matmul(), orders matter!

Initialization

TensorFlow variables must be initialized before they have values.

tf.global_variables_initializer() will initialize all TF variables.

1
2
3
init = tf.global_variables_initializer()
with tf.Session() as sess:
	sess.run(init)

Initialize the random weight from a normal distributiontf.truncated_normal() 

1
2
3
n_features = 120
n_labels = 5
weights = tf.Variable(tf.truncated_normal([n_features, n_labels]))

Initialize the bias to all zeros

1
bias=tf.Variable(tf.zeros(n_labels))

Softmax

The softmax function squashes it’s inputs, typically called logits or logit scores, to be between 0 and 1 and also normalizes the outputs such that they all sum to 1.

1
2
3
4
5
6
7
8
9
def run():
	output = None
    logit_data = [2.0, 1.0, 0.1]
  	logits = tf.placeholder(tf.float32)
  	softmax = tf.nn.softmax(logits)
    
    with tf.Session() as sess:
        output = sess.run(softmax, feed_dict={logis: logit_data})
    return output

Cross Entropy

cross entropy loss function

Sum functiontf.reduce_sum() and log function tf.log()

1
crossentropy = -tf.reduce_sum(tf.multiply(tf.log(softmax), one_hot))

Mini-batch

Training on subsets of the dataset instead of all the data at one time.

  1. Divide the data into batches

    1
    2
    3
    # Features and Labels
    features = tf.placeholder(tf.float32, [None, n_input])
    labels = tf.placeholder(tf.float32, [None, n_classes])

    none is a placeholder for the batch size. Tensorflow will accept any batch size greater than 0.

  2. 1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    import math
    def batches(batch_size, features, labels):
        """
        Create batches of features and labels
        :param batch_size: The batch size
        :param features: List of features
        :param labels: List of labels
        :return: Batches of (Features, Labels)
        """
        assert len(features) == len(labels)
        outout_batches = []
        
        sample_size = len(features)
        for start_i in range(0, sample_size, batch_size):
            end_i = start_i + batch_size
            batch = [features[start_i:end_i], 		  				 			 labels[start_i:end_i]]
            outout_batches.append(batch)
            
        return outout_batches

Calculate Accuracy

1
2
correct_prediction = tf.equal(tf.argmax(prediction,1), tf.argmax(labels,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

Activation

tf.nn.relu()

adding non-linearity

MNIST example

TF example for MNIST data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
# Import data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets(".", one_hot=True, reshape=False)
import tensorflow as tf
# Set parameters
learning_rate = 0.001
training_epochs = 20
batch_size = 128  # Decrease batch size if you don't have enough memory
display_step = 1
# Model size
n_input = 784  # MNIST data input (img shape: 28*28)
n_classes = 10  # MNIST total classes (0-9 digits)
n_hidden_layer = 256 # layer number of features
# Variables: Weights and Bias
weights = {
    'hidden_layer': tf.Variable(tf.random_normal([n_input, n_hidden_layer])),
    'out': tf.Variable(tf.random_normal([n_hidden_layer, n_classes]))
}
biases = {
    'hidden_layer': tf.Variable(tf.random_normal([n_hidden_layer])),
    'out': tf.Variable(tf.random_normal([n_classes]))
}
# Input: placeholder
x = tf.placeholder("float", [None, 28, 28, 1])
y = tf.placeholder("float", [None, n_classes])
x_flat = tf.reshape(x, [-1, n_input])
# Computation(linear&non-linear) for hidden layer and output layer
# Hidden layer with RELU activation
layer_1 = tf.add(tf.matmul(x_flat, weights['hidden_layer']),\
    			 biases['hidden_layer'])
layer_1 = tf.nn.relu(layer_1)
# Output layer with linear activation
logits = tf.add(tf.matmul(layer_1, weights['out']), biases['out'])
#Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits\			  						(logits=logits, labels=y))
optimizer= tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\
					.minimize(cost)
# Initializing the variables
init = tf.global_variables_initializer()
# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    # Training cycle
    for epoch in range(training_epochs):
        total_batch = int(mnist.train.num_examples/batch_size)
        # Loop over all batches
        for i in range(total_batch):
            batch_x, batch_y = mnist.train.next_batch(batch_size)
            # Run optimization and cost
            sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
        # Display logs per epoch step
        if epoch % display_step == 0:
            c = sess.run(cost, feed_dict={x: batch_x, y: batch_y})
            print("Epoch:", '%04d' % (epoch+1), "cost=", \
                "{:.9f}".format(c))
        print("Optimization Finished!")
        
        # Test model
        correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
    	# Calculate accuracy
    	accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

Save and Restore TensorFlow Models

Training a deep learning model from scratch on a large dataset is expensive computationally. Save the models once finishing training.

Useful tutorial

Save a Trained Model

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Remove previous Tensors and Operations
tf.reset_default_graph()
# start with a model
## 省略 data, placeholder, Variable, operation, loss and optimizer, accuracy
# The file path to save the data
save_file = './train_model.ckpt' # checkpoint
saver = tf.train.Saver()
with tf.Session() as sess:
    # 省略一堆 sess.run() init, optimizer, accuracy
# Save the model
    saver.save(sess, save_file)
    print('Trained Model Saved.')

Load a Trained Model

1
2
3
4
5
6
7
8
9
10
saver = tf.train.Saver()
with tf.Session() as sess:
    # Load the model
    saver.restore(sess, save_file)
    
    test_accuracy = sess.run(accuracy,
        feed_dict={features: mnist.test.images, labels: mnist.test.labels})
print('Test Accuracy: {}'.format(test_accuracy))

Since tf.train.Saver.restore() sets all the TensorFlow Variables, you don’t need to call tf.global_variables_initializer()

Fine Tuning

Loading saved Variables directly into a modified model can generate errors.

具体什么意思呢？

1
2
3
4
5
6
7
8
9
10
11
12
13
# suppose we have saved trained model
# Two Tensor Variables: weights and bias
weights = tf.Variable(tf.truncated_normal([2, 3])) #name set by TF "Variable:0"
bias = tf.Variable(tf.truncated_normal([3]))# "Variable_1:0"
saver.save(sess, save_file)
# Remove the previous setting
tf.reset_default_graph()
# set new Variables 
bias = tf.Variable(tf.truncated_normal([3])) # "Variable:0"
weights = tf.Variable(tf.truncated_normal([2, 3]))#"Variable_1:0"
# notice that the Variables order changed, the default Variable.name 根据顺序来的
saver.restore(sess, save_file)

Set the name manually for models. 

1
2
3
# Two Tensor Variables: weights and bias
weights = tf.Variable(tf.truncated_normal([2, 3]), name='weights')
bias = tf.Variable(tf.truncated_normal([3]), name='bias')

Regularization

tf.nn.dropout()

Dropout after Relu

1
2
3
4
5
keep_prob = tf.placeholder(tf.float32) # probability to keep units
									   
hidden_layer = tf.add(tf.matmul(features, weights[0]), biases[0])
hidden_layer = tf.nn.relu(hidden_layer)
hidden_layer = tf.nn.dropout(hidden_layer, keep_prob)

During training, a good starting value for keep_prob is 0.5

During testing, keep_prob = 1 to keep all units and maximize the power of model.

TensorFlow vs Numpy

Source from TensorFlow Tutorial

TensorFlow computations define a computation graph that has no numerical value until evaluated!

1
2
ta = tf.zeros((2,2))
print(ta.eval())