{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "TensorFlow 2.x Reference Guide.ipynb", "provenance": [], "collapsed_sections": [] }, "kernelspec": { "name": "python3", "display_name": "Python 3" } }, "cells": [ { "cell_type": "code", "metadata": { "id": "aB18AO4eOuJ5" }, "source": [ "import tensorflow as tf\r\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "hzSfiCu0OQAQ" }, "source": [ "# Introduction to Tensors" ] }, { "cell_type": "markdown", "metadata": { "id": "IcXv1d_TOVoL" }, "source": [ "In TensorFlow, tensors are classified into **constant tensors** and **variable tensors**.\r\n", "* A defined **constant tensor** has an unchangeable value and dimension, and a defined variable tensor has a changeable value and an unchangeable dimension.\r\n", "* In neural networks, **variable tensors** are generally used as matrices for storing weights and other information, and are a type of trainable data. Constant tensors can be used for storing hyperparameters or other structured data.\r\n", "\r\n", "\r\n" ] }, { "cell_type": "markdown", "metadata": { "id": "4wDHyqfIOjQn" }, "source": [ "## Tensor Creation" ] }, { "cell_type": "markdown", "metadata": { "id": "MEccyRpBOnUq" }, "source": [ "### Creating a Constant Tensor" ] }, { "cell_type": "markdown", "metadata": { "id": "Z6r-6we1OuxT" }, "source": [ "Common methods for creating a constant tensor include:\r\n", "\r\n", "\r\n", "* tf.constant(): creates a constant tensor.\r\n", "* tf.zeros(), tf.zeros_like(), tf.ones(), and tf.ones_like(): create an all-zero or all-one constant tensor.\r\n", "* \ttf.fill(): creates a tensor with a user-defined value.\r\n", "* \ttf.random: creates a tensor with a known distribution.\r\n", "* \tCreating a list object by using NumPy, and then converting the list object into a tensor by using tf.convert_to_tensor.\r\n" ] }, { "cell_type": "markdown", "metadata": { "id": "3Z69okLaO7-v" }, "source": [ "**tf.constant()**\r\n", "\r\n", "tf.constant(value, dtype=None, shape=None, name='Const'):\r\n", "*\tvalue: \tA constant value (or list) of output type dtype.\r\n", "*\tdtype: The type of the elements of the resulting tensor.\r\n", "*\tshape: Optional dimensions of resulting tensor.\r\n", "*\tname: Optional name for the tensor.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "f7T5qSQ9N7Z3", "outputId": "10e29690-2166-446f-f50f-83c842e79503" }, "source": [ "const_a = tf.constant([[1, 2, 3, 4]],shape=[2,2], dtype=tf.float32) # Create a 2x2 matrix with values 1, 2, 3, and 4.\r\n", "const_a" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 2 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "2C5gx467PLnX", "outputId": "78ec9db6-1c32-406f-ee63-8d19382a6e1e" }, "source": [ "#View common attributes.\r\n", "print(\"value of the constant const_a:\", const_a.numpy())\r\n", "print(\"data type of the constant const_a:\", const_a.dtype)\r\n", "print(\"shape of the constant const_a:\", const_a.shape)\r\n", "print(\"name of the device that is to generate the constant const_a:\", const_a.device)\r\n" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "value of the constant const_a: [[1. 2.]\n", " [3. 4.]]\n", "data type of the constant const_a: \n", "shape of the constant const_a: (2, 2)\n", "name of the device that is to generate the constant const_a: /job:localhost/replica:0/task:0/device:CPU:0\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "BVpo9m_jPSlo" }, "source": [ "**tf.zeros(), tf.zeros_like(), tf.ones(), and tf.ones_like()**\r\n", "\r\n", "Usages of tf.ones() and tf.ones_like() are similar to those of tf.zeros() and tf.zeros_like(). Therefore, the following describes only the usages of tf.ones() and tf.ones_like().\r\n", "\r\n", "Create a constant with the value 0.\r\n", "\r\n", "tf.zeros(shape, dtype=tf.float32, name=None):\r\n", "*\tshape: A list of integers, a tuple of integers, or a 1-D Tensor of type int32.t\r\n", "*\tdtype: The DType of an element in the resulting Tensor.\r\n", "*name: Optional string. A name for the operation.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0pLqZv7rPdMK", "outputId": "466a2546-13cb-49bc-ee9d-71b3edb8c1f2" }, "source": [ "zeros_b = tf.zeros(shape=[2, 3], dtype=tf.int32) # Create a 2x3 matrix with all values being 0.\r\n", "zeros_b" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 5 } ] }, { "cell_type": "markdown", "metadata": { "id": "mo1JOJ6MPsmA" }, "source": [ "Create a tensor whose value is 0 based on the input tensor, with its shape being the same as that of the input tensor. \r\n", "\r\n", "tf.zeros_like(input, dtype=None, name=None):\r\n", "*\tinput_tensor: A Tensor or array-like object.\r\n", "*\tdtype: A type for the returned Tensor. Must be float16, float32, float64, int8, uint8, int16, uint16, int32, int64, complex64, complex128, bool or string (optional).\r\n", "*\tname: A name for the operation (optional).\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "qwiPedQIPxXt", "outputId": "63a76f70-a45f-40cb-fb26-492a7ba9c383" }, "source": [ "zeros_like_c = tf.zeros_like(const_a)\r\n", "#View generated data.\r\n", "zeros_like_c.numpy()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array([[0., 0.],\n", " [0., 0.]], dtype=float32)" ] }, "metadata": { "tags": [] }, "execution_count": 6 } ] }, { "cell_type": "markdown", "metadata": { "id": "shB2KhlJP0L4" }, "source": [ "**tf.fill()**\r\n", "\r\n", "Create a tensor and fill it with a scalar value. \r\n", "\r\n", "tf.fill(dims, value, name=None):\r\n", "*\tdims: A 1-D sequence of non-negative numbers. Represents the shape of the output tf.Tensor. Entries should be of type: int32, int64.\r\n", "*\tvalue: A value to fill the returned tf.Tensor.\r\n", "*\tname: Optional string. The name of the output tf.Tensor.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "vhT94TcGP4_C", "outputId": "fcecd542-91ff-4244-af7e-23cf2b498eb6" }, "source": [ "fill_d = tf.fill([3,3], 8) # Create a 2x3 matrix with all values being 8.\r\n", "#View data.\r\n", "fill_d.numpy()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array([[8, 8, 8],\n", " [8, 8, 8],\n", " [8, 8, 8]], dtype=int32)" ] }, "metadata": { "tags": [] }, "execution_count": 7 } ] }, { "cell_type": "markdown", "metadata": { "id": "-xxSHiCeP87M" }, "source": [ "**tf.random**\r\n", "\r\n", "This module is used to generate a tensor with a specific distribution. Common methods in this module include tf.random.uniform(), tf.random.normal(), and tf.random.shuffle(). The following describes how to use tf.random.normal().\r\n", "Create a tensor that conforms to a normal distribution. \r\n", "\r\n", "tf.random.normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32,seed=None, name=None):\r\n", "*\tshape: A 1-D integer Tensor or Python array. The shape of the output tensor.\r\n", "*\tmean: A Tensor or Python value of type dtype, broadcastable with stddev. The mean of the normal distribution.\r\n", "*\tstddev: A Tensor or Python value of type dtype, broadcastable with mean. The standard deviation of the normal distribution.\r\n", "*\tdtype: The type of the output.\r\n", "*\tseed: \tA Python integer. Used to create a random seed for the distribution. See tf.random.set_seed for behavior.\r\n", "*\tname: A name for the operation (optional).\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "t_wiEcamQCQ7", "outputId": "9043f4d9-073d-42fa-841f-839da2c0af6a" }, "source": [ "random_e = tf.random.normal([5,5],mean=0,stddev=1.0, seed = 1)\r\n", "#View the created data.\r\n", "random_e.numpy()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array([[-0.8113182 , 1.4845988 , 0.06532937, -2.4427042 , 0.0992484 ],\n", " [ 0.5912243 , 0.59282297, -2.1229296 , -0.72289723, -0.05627038],\n", " [ 0.6435448 , -0.26432407, 1.8566332 , 0.5678417 , -0.3828359 ],\n", " [-1.4853433 , 1.2617711 , -0.02530608, -0.2646297 , 1.5328138 ],\n", " [-1.7429771 , -0.43789294, -0.56601 , 0.32066926, 1.132831 ]],\n", " dtype=float32)" ] }, "metadata": { "tags": [] }, "execution_count": 8 } ] }, { "cell_type": "markdown", "metadata": { "id": "rYdo3-_mQGbL" }, "source": [ "**Step 5\tCreate a list object by using NumPy, and then convert the list object into a tensor by using tf.convert_to_tensor.**\r\n", "\r\n", "This method can convert a given value into a tensor. tf.convert_to_tensor can be used to convert a Python data type into a tensor data type available to TensorFlow.\r\n", "\r\n", "tf.convert_to_tensor(value,dtype=None,dtype_hint=None,name=None):\r\n", "*\tvalue: An object whose type has a registered Tensor conversion function.\r\n", "*\tdtype: Optional element type for the returned tensor. If missing, the type is inferred from the type of value.\r\n", "*\tdtype_hint: Optional element type for the returned tensor, used when dtype is None. In some cases, a caller may not have a dtype in mind when converting to a tensor, so dtype_hint can be used as a soft preference. If the conversion to dtype_hint is not possible, this argument has no effect.\r\n", "*\tName:\tOptional name to use if a new Tensor is created.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-F_JLgO0QKxu", "outputId": "72911118-7e17-4582-96b4-4747f73136cf" }, "source": [ "#Create a list.\r\n", "list_f = [1,2,3,4,5,6]\r\n", "#View the data type.\r\n", "type(list_f)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "list" ] }, "metadata": { "tags": [] }, "execution_count": 9 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WP7fhUGEQPcW", "outputId": "e0fcda1b-580e-4388-b5d7-7fdf79d34215" }, "source": [ "tensor_f = tf.convert_to_tensor(list_f, dtype=tf.float32)\r\n", "tensor_f" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 10 } ] }, { "cell_type": "markdown", "metadata": { "id": "k8KOS5nAQUQO" }, "source": [ "### Creating a Variable Tensor" ] }, { "cell_type": "markdown", "metadata": { "id": "km7PWUPhQZPa" }, "source": [ "In TensorFlow, variables are operated using the tf.Variable class. tf.Variable indicates a tensor. The value of tf.Variable can be changed by running an arithmetic operation on tf.Variable. Variable values can be read and changed." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "DaoxEBfIQca4", "outputId": "1c6f9016-f6b2-4dd9-cf39-1975ced60ae5" }, "source": [ "#Create a variable. Only the initial value needs to be provided.\r\n", "var_1 = tf.Variable(tf.ones([2,3]))\r\n", "var_1" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 11 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "zGK8j_cbQfFi", "outputId": "f9dad3a4-0fa2-43e8-baea-3ac4112ff80a" }, "source": [ "#Read the variable value.\r\n", "print(\"Value of the variable var_1:\",var_1.read_value())\r\n", "#Assign a variable value.\r\n", "var_value_1=[[1,2,3],[4,5,6]] \r\n", "var_1.assign(var_value_1)\r\n", "print(\"Value of the variable var_1 after the assignment:\",var_1.read_value())" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Value of the variable var_1: tf.Tensor(\n", "[[1. 2. 3.]\n", " [4. 5. 6.]], shape=(2, 3), dtype=float32)\n", "Value of the variable var_1 after the assignment: tf.Tensor(\n", "[[1. 2. 3.]\n", " [4. 5. 6.]], shape=(2, 3), dtype=float32)\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wtA4-sTKQoDh", "outputId": "783a4b2d-37e2-4a25-b1bc-f87e6c18914f" }, "source": [ "#Variable addition\r\n", "var_1.assign_add(tf.ones([2,3]))\r\n", "var_1" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 14 } ] }, { "cell_type": "markdown", "metadata": { "id": "z8lrG-WkQs_g" }, "source": [ "## Tensor Slicing and Indexing" ] }, { "cell_type": "markdown", "metadata": { "id": "c30I5mNNQuyW" }, "source": [ "### Slicing" ] }, { "cell_type": "markdown", "metadata": { "id": "ggCD-7J3Q0pv" }, "source": [ "Tensor slicing methods include:\r\n", "*\t[start: end]: extracts a data slice from the start position to the end position of the tensor.\r\n", "*\t[start:end:step] or [::step]: extracts a data slice at an interval of step from the start position to the end position of the tensor.\r\n", "* [::-1]: slices data from the last element.\r\n", "*\t'...': indicates a data slice of any length.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WKwUTxfzQ4oj", "outputId": "9fdcbea6-f22d-42fa-8d56-38f6b976d359" }, "source": [ "#Create a 4-dimensional tensor. The tensor contains four images. The size of each image is 100 x 100 x 3.\r\n", "tensor_h = tf.random.normal([4,100,100,3])\r\n", "tensor_h" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 15 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "CuoFmuSQQ8tQ", "outputId": "2e074d59-270b-408a-e5ed-767688e24a06" }, "source": [ "#Extract the first image.\r\n", "tensor_h[0,:,:,:]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 16 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "OMw6AQLjRBIM", "outputId": "79446736-7ca1-4a84-e12e-ccfe792dc43c" }, "source": [ "#Extract one slice at an interval of two images.\r\n", "tensor_h[::2,...]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 17 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "SXOMArFWRDjb", "outputId": "a622593b-1cdc-42dd-af6c-bb9ed26322b8" }, "source": [ "#Slice data from the last element.\r\n", "tensor_h[::-1]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 18 } ] }, { "cell_type": "markdown", "metadata": { "id": "HWC4A0IERGJS" }, "source": [ "### Indexing" ] }, { "cell_type": "markdown", "metadata": { "id": "-qer0f9FRHkr" }, "source": [ "The basic format of an index is a[d1][d2][d3]." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wejgHCi4RJbP", "outputId": "3b463293-1a10-48ba-d18d-a55e6a14bfc0" }, "source": [ "#Obtain the pixel in the position [20,40] in the second channel of the first image.\r\n", "tensor_h[0][19][39][1]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 19 } ] }, { "cell_type": "markdown", "metadata": { "id": "Dc8laIH1RQOP" }, "source": [ "If the indexes of data to be extracted are nonconsecutive, tf.gather and tf.gather_nd are commonly used for data extraction in TensorFlow.\r\n", "To extract data from a particular dimension: \r\n", "tf.gather(params, indices,axis=None):\r\n", "*\tparams: input tensor\r\n", "*\tindices: index of the data to be extracted\r\n", "*\taxis: dimension of the data to be extracted\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "zyZ0XVcBRUPD", "outputId": "3dd9d348-70cc-489f-9db5-6d852b390fb1" }, "source": [ "#Extract the first, second, and fourth images from tensor_h ([4,100,100,3]).\r\n", "indices = [0,1,3]\r\n", "tf.gather(tensor_h,axis=0,indices=indices)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 20 } ] }, { "cell_type": "markdown", "metadata": { "id": "uvwDCpqxRYFM" }, "source": [ "tf.gather_nd allows data extraction from multiple dimensions.\r\n", "\r\n", "tf.gather_nd(params,indices, batch_dims=0, name=None):\r\n", "*\tparams: \tA Tensor. The tensor from which to gather values.\r\n", "*\tindices: A Tensor. Must be one of the following types: int32, int64. Index tensor.\r\n", "*\tName:\tA name for the operation (optional).\r\n", "*\tbatch_dims: An integer or a scalar 'Tensor'. The number of batch dimensions.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "b0w0lBj6RcbX", "outputId": "1271d41a-6912-4226-9525-c1f5ddc50d30" }, "source": [ "#Extract the pixel in [1,1] from the first dimension of the first image and the pixel in [2,2] from the first dimension of the second image in tensot_h ([4,100,100,3]).\r\n", "indices = [[0,1,1,0],[1,2,2,0]]\r\n", "tf.gather_nd(tensor_h,indices=indices)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 21 } ] }, { "cell_type": "markdown", "metadata": { "id": "ZXRCK70zRgq6" }, "source": [ "## Tensor Dimension Modification" ] }, { "cell_type": "markdown", "metadata": { "id": "MPrEfn0BRleK" }, "source": [ "### Dimension Display" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5wpEqmTpRmc9", "outputId": "aa361113-1bac-4102-83a9-c2abdddf2d27" }, "source": [ "const_d_1 = tf.constant([[1, 2, 3, 4]],shape=[2,2], dtype=tf.float32)\r\n", "#Three common methods for displaying a dimension:\r\n", "print(const_d_1.shape)\r\n", "print(const_d_1.get_shape())\r\n", "print(tf.shape(const_d_1))#The output is a tensor. The value of the tensor indicates the size of the tensor dimension to be displayed." ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "(2, 2)\n", "(2, 2)\n", "tf.Tensor([2 2], shape=(2,), dtype=int32)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "jlTpod2PRt1b" }, "source": [ "As described above, .shape and .get_shape() return TensorShape objects, and tf.shape(x) returns Tensor objects." ] }, { "cell_type": "markdown", "metadata": { "id": "MPtDnMadRwOc" }, "source": [ "### Dimension Reshaping" ] }, { "cell_type": "markdown", "metadata": { "id": "j30_fTm6R0Gp" }, "source": [ "tf.reshape(tensor,shape,name=None):\r\n", "*\ttensor: input tensor\r\n", "*\tshape: dimension of the reshaped tensor" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "7greydBYRwax", "outputId": "8e30a85e-ab9c-49b2-8795-97db9996bd09" }, "source": [ "reshape_1 = tf.constant([[1,2,3],[4,5,6]])\r\n", "print(reshape_1)\r\n", "tf.reshape(reshape_1, (3,2))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "tf.Tensor(\n", "[[1 2 3]\n", " [4 5 6]], shape=(2, 3), dtype=int32)\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 23 } ] }, { "cell_type": "markdown", "metadata": { "id": "XOy5O6KaR75h" }, "source": [ "### Dimension Expansion" ] }, { "cell_type": "markdown", "metadata": { "id": "h4yL7kp7R-nx" }, "source": [ "tf.expand_dims(input,axis,name=None):\r\n", "*\tinput: input tensor\r\n", "*\taxis: adds a dimension after the axis dimension. When the number of dimensions of the input data is D, the axis must fall in the range of [–(D + 1), D] (included). A negative value indicates adding a dimension in reverse order.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "yOKmRXa3SFkc", "outputId": "eae92425-ae35-46a9-dd1d-3582e82f5e5d" }, "source": [ "#Generate a 100 x 100 x 3 tensor to represent a 100 x 100 three-channel color image.\r\n", "expand_sample_1 = tf.random.normal([100,100,3], seed=1)\r\n", "print(\"size of the original data:\",expand_sample_1.shape)\r\n", "print(\"add a dimension before the first dimension (axis = 0): \",tf.expand_dims(expand_sample_1, axis=0).shape)\r\n", "print(\"add a dimension before the second dimension (axis = 1): \",tf.expand_dims(expand_sample_1, axis=1).shape)\r\n", "print(\"add a dimension after the last dimension (axis = –1): \",tf.expand_dims(expand_sample_1, axis=-1).shape)\r\n" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "size of the original data: (100, 100, 3)\n", "add a dimension before the first dimension (axis = 0): (1, 100, 100, 3)\n", "add a dimension before the second dimension (axis = 1): (100, 1, 100, 3)\n", "add a dimension after the last dimension (axis = –1): (100, 100, 3, 1)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "zHpyOv0JShVr" }, "source": [ "### Dimension Squeezing" ] }, { "cell_type": "markdown", "metadata": { "id": "_C6sCjY6Sjv-" }, "source": [ "tf.squeeze(input,axis=None,name=None):\r\n", "*\tinput: input tensor\r\n", "*\taxis: If axis is set to 1, dimension 1 needs to be deleted.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "brsLmwjESoI5", "outputId": "fcca810a-f898-4869-adba-33994808dbb0" }, "source": [ "#Generate a 100 x 100 x 3 tensor to represent a 100 x 100 three-channel color image.\r\n", "squeeze_sample_1 = tf.random.normal([1,100,100,3])\r\n", "print(\"size of the original data:\",squeeze_sample_1.shape)\r\n", "squeezed_sample_1 = tf.squeeze(expand_sample_1)\r\n", "print(\"data size after dimension squeezing:\",squeezed_sample_1.shape)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "size of the original data: (1, 100, 100, 3)\n", "data size after dimension squeezing: (100, 100, 3)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "dZqIp6TBSrbI" }, "source": [ "### Transpose" ] }, { "cell_type": "markdown", "metadata": { "id": "ivikN39ISsmT" }, "source": [ "tf.transpose(a,perm=None,conjugate=False,name='transpose'):\r\n", "*\ta: input tensor\r\n", "*\tperm: tensor size sequence, generally used to transpose high-dimensional arrays\r\n", "*\tconjugate: indicates complex number transpose.\r\n", "*\tname: tensor name\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "V3UnXhNWSxP6", "outputId": "d129d856-e836-4180-f5c9-59233c0affbb" }, "source": [ "#Input the tensor to be transposed, and call tf.transpose.\r\n", "trans_sample_1 = tf.constant([1,2,3,4,5,6],shape=[2,3])\r\n", "print(\"size of the original data:\",trans_sample_1.shape)\r\n", "transposed_sample_1 = tf.transpose(trans_sample_1)\r\n", "print(\"size of transposed data:\",transposed_sample_1.shape)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "size of the original data: (2, 3)\n", "size of transposed data: (3, 2)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "1eRFqGEJS068" }, "source": [ "perm is required for high-dimensional data transpose, and indicates the dimension sequence of the input tensor.\r\n", "\r\n", "The original dimension sequence of a three-dimensional tensor is [0, 1, 2] (perm), indicating the length, width, and height of high-dimensional data, respectively.\r\n", "\r\n", "Data dimensions can be transposed by changing the sequence of values in perm.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "PO9hBTrxS5SZ", "outputId": "0d879b04-a443-43e7-b38c-f863f45a8b38" }, "source": [ "#Generate a 4 x 100 x 200 x 3 tensor to represent four 100 x 200 three-channel color images.\r\n", "trans_sample_2 = tf.random.normal([4,100,200,3])\r\n", "print(\"size of the original data:\",trans_sample_2.shape)\r\n", "#Exchange the length and width for the four images: The original perm value is [0,1,2,3], and the new perm value is [0,2,1,3].\r\n", "transposed_sample_2 = tf.transpose(trans_sample_2,[0,2,1,3])\r\n", "print(\"size of transposed data:\",transposed_sample_2.shape)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "size of the original data: (4, 100, 200, 3)\n", "size of transposed data: (4, 200, 100, 3)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "jSs8sE9hTGcx" }, "source": [ "### Broadcast (broadcast_to)" ] }, { "cell_type": "markdown", "metadata": { "id": "9uInixiZTHiz" }, "source": [ "broadcast_to is used to broadcast data from a low dimension to a high dimension.\r\n", "tf.broadcast_to(input,shape,name=None):\r\n", "*\tinput: input tensor\r\n", "*\tshape: size of the output tensor\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WtrBnMyfTLoo", "outputId": "01a734de-3c3f-4985-e614-534934967f94" }, "source": [ "broadcast_sample_1 = tf.constant([1,2,3,4,5,6])\r\n", "print(\"original data:\",broadcast_sample_1.numpy())\r\n", "broadcasted_sample_1 = tf.broadcast_to(broadcast_sample_1,shape=[4,6])\r\n", "print(\"broadcasted data:\",broadcasted_sample_1.numpy())" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "original data: [1 2 3 4 5 6]\n", "broadcasted data: [[1 2 3 4 5 6]\n", " [1 2 3 4 5 6]\n", " [1 2 3 4 5 6]\n", " [1 2 3 4 5 6]]\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "TKzPHqDJTLrM", "outputId": "50ea66a0-930f-4d1d-debf-e23714ca2343" }, "source": [ "#During the operation, if two arrays have different shapes, TensorFlow automatically triggers the broadcast mechanism as NumPy does.\r\n", "a = tf.constant([[ 0, 0, 0],\r\n", " [10,10,10],\r\n", " [20,20,20],\r\n", " [30,30,30]])\r\n", "b = tf.constant([1,2,3])\r\n", "print(a + b)\r\n" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "tf.Tensor(\n", "[[ 1 2 3]\n", " [11 12 13]\n", " [21 22 23]\n", " [31 32 33]], shape=(4, 3), dtype=int32)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "0TCD8w0wTUU1" }, "source": [ "## Arithmetic Operations on Tensors" ] }, { "cell_type": "markdown", "metadata": { "id": "TfbWcumJUFwJ" }, "source": [ "### Arithmetic Operators" ] }, { "cell_type": "markdown", "metadata": { "id": "WQQlpSYmUJT8" }, "source": [ "Main arithmetic operations include addition (tf.add), subtraction (tf.subtract), multiplication (tf.multiply), division (tf.divide), logarithm (tf.math.log), and powers (tf.pow). The following describes only one addition example." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "IqacWcfaULLk", "outputId": "e9f3c67b-c053-4720-baba-3e39e75d00ad" }, "source": [ "a = tf.constant([[3, 5], [4, 8]])\r\n", "b = tf.constant([[1, 6], [2, 9]])\r\n", "print(tf.add(a, b))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "tf.Tensor(\n", "[[ 4 11]\n", " [ 6 17]], shape=(2, 2), dtype=int32)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "CLYxWCjuUNDm" }, "source": [ "### Matrix Multiplication" ] }, { "cell_type": "markdown", "metadata": { "id": "WVa3poyPUP2e" }, "source": [ "Matrix multiplication is implemented by calling tf.matmul." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "AC7kK9zwUQQX", "outputId": "94948b7d-229c-4358-9171-3005af8a2e8a" }, "source": [ "tf.matmul(a,b)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 31 } ] }, { "cell_type": "markdown", "metadata": { "id": "fQFiKLPhUS4Z" }, "source": [ "### Tensor Statistics Collection" ] }, { "cell_type": "markdown", "metadata": { "id": "A6Cl9eoNUWYc" }, "source": [ "Methods for collecting tensor statistics include:\r\n", "*\ttf.reduce_min/max/mean(): calculates the minimum, maximum, and mean values.\r\n", "*\ttf.argmax()/tf.argmin(): calculates the positions of the maximum and minimum values.\r\n", "*\ttf.equal(): checks whether two tensors are equal by element.\r\n", "*tf.unique(): removes duplicate elements from tensors.\r\n", "*\ttf.nn.in_top_k(prediction, target, K): calculates whether the predicted value is equal to the actual value, and returns a Boolean tensor.\r\n", "\r\n", "The following describes how to use **tf.argmax()**:\r\n", "\r\n", "Return the position of the maximum value.\r\n", "\r\n", "tf.argmax(input,axis):\r\n", "*\tinput: input tensor\r\n", "*\taxis: maximum output value in the axis dimension\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "MxXciqbXUdVQ", "outputId": "b33f9d44-3c1e-4ba8-dca5-a91c9627a34b" }, "source": [ "argmax_sample_1 = tf.constant([[1,3,2],[2,5,8],[7,5,9]])\r\n", "print(\"input tensor:\",argmax_sample_1.numpy())\r\n", "max_sample_1 = tf.argmax(argmax_sample_1, axis=0)\r\n", "max_sample_2 = tf.argmax(argmax_sample_1, axis=1)\r\n", "print(\"locate the maximum value by column:\",max_sample_1.numpy())\r\n", "print(\"locate the maximum value by row:\",max_sample_2.numpy())\r\n" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "input tensor: [[1 3 2]\n", " [2 5 8]\n", " [7 5 9]]\n", "locate the maximum value by column: [2 1 2]\n", "locate the maximum value by row: [1 2 2]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "J9ZuMjY4UmrO" }, "source": [ "### Dimension-based Arithmetic Operations" ] }, { "cell_type": "markdown", "metadata": { "id": "ciYbGiWRUnzo" }, "source": [ "In TensorFlow, a series of operations of tf.reduce_* reduce tensor dimensions. The series of operations can be performed on dimensional elements of a tensor, for example, calculating the mean value by row and calculating a product of all elements in the tensor.\r\n", "\r\n", "Common operations include tf.reduce_sum (addition), tf.reduce_prod (multiplication), tf.reduce_min (minimum), tf.reduce_max (maximum), tf.reduce_mean (mean value), tf.reduce_all (logical AND), tf.reduce_any (logical OR), and tf.reduce_logsumexp (log(sum(exp))).\r\n", "\r\n", "The methods for using these operations are similar. The following describes how to use tf.reduce_sum.\r\n", "\r\n", "Calculate the sum of elements in all dimensions of a tensor.\r\n", "\r\n", "tf.reduce_sum(input_tensor, axis=None, keepdims=False,name=None):\r\n", "*\tinput_tensor: The tensor to reduce. Should have numeric type.\r\n", "*\taxis: The dimensions to reduce. If None (the default), reduces all dimensions. Must be in the range [-rank(input_tensor),rank(input_tensor)].\r\n", "*\tkeepdims: If true, retains reduced dimensions with length 1.\r\n", "*\tname: A name for the operation (optional).\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "KYq3cpVRUwBi", "outputId": "933fdf96-5934-42b8-dd6d-ed2147c1c831" }, "source": [ "reduce_sample_1 = tf.constant([1,2,3,4,5,6],shape=[2,3])\r\n", "print(\"original data\",reduce_sample_1.numpy())\r\n", "print(\"calculate the sum of all elements in the tensor (axis = None): \",tf.reduce_sum(reduce_sample_1,axis=None).numpy())\r\n", "print(\"calculate the sum of elements in each column by column (axis = 0): \",tf.reduce_sum(reduce_sample_1,axis=0).numpy())\r\n", "print(\"calculate the sum of elements in each column by row (axis = 1): \",tf.reduce_sum(reduce_sample_1,axis=1).numpy())" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "original data [[1 2 3]\n", " [4 5 6]]\n", "calculate the sum of all elements in the tensor (axis = None): 21\n", "calculate the sum of elements in each column by column (axis = 0): [5 7 9]\n", "calculate the sum of elements in each column by row (axis = 1): [ 6 15]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "jL4b3lAAU5PR" }, "source": [ "## Tensor Concatenation and Splitting" ] }, { "cell_type": "markdown", "metadata": { "id": "ucJSyTV0U9IY" }, "source": [ "### Tensor Concatenation" ] }, { "cell_type": "markdown", "metadata": { "id": "yFFXv3TrVBAJ" }, "source": [ "In TensorFlow, tensor concatenation operations include:\r\n", "*\ttf.contact(): concatenates vectors based on the specified dimension, while keeping other dimensions unchanged.\r\n", "*\ttf.stack(): changes a group of R dimensional tensors to R+1 dimensional tensors, with the dimensions changed after the concatenation.\r\n", "*\ttf.concat(values, axis, name='concat'):\r\n", "*\tvalues: input tensor\r\n", "*\taxis: dimension to concatenate\r\n", "*\tname: operation name\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "jvkRDOZVVEuY", "outputId": "a531ce10-4c8a-4d0c-d655-8132cdaba50b" }, "source": [ "concat_sample_1 = tf.random.normal([4,100,100,3])\r\n", "concat_sample_2 = tf.random.normal([40,100,100,3])\r\n", "print(\"sizes of the original data:\",concat_sample_1.shape,concat_sample_2.shape)\r\n", "concated_sample_1 = tf.concat([concat_sample_1,concat_sample_2],axis=0)\r\n", "print(\"size of the concatenated data:\",concated_sample_1.shape)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "sizes of the original data: (4, 100, 100, 3) (40, 100, 100, 3)\n", "size of the concatenated data: (44, 100, 100, 3)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "kKGQi78bVJ9C" }, "source": [ "A dimension can be added to an original matrix in the same way. axis determines the position of the dimension.\r\n", "\r\n", "tf.stack(values, axis=0, name='stack'):\r\n", "*\tvalues: A list of Tensor objects with the same shape and type.\r\n", "*\taxis: An int. The axis to stack along. Defaults to the first dimension. Negative values wrap around, so the valid range is [-(R+1), R+1).\r\n", "*\tname: \tA name for this operation (optional).\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "itlvWXN4VOoj", "outputId": "361abe77-69d1-4d81-d081-3448697c652b" }, "source": [ "stack_sample_1 = tf.random.normal([100,100,3])\r\n", "stack_sample_2 = tf.random.normal([100,100,3])\r\n", "print(\"sizes of the original data: \",stack_sample_1.shape, stack_sample_2.shape)\r\n", "#Dimensions increase after the concatenation. If axis is set to 0, a dimension is added before the first dimension.\r\n", "stacked_sample_1 = tf.stack([stack_sample_1, stack_sample_2],axis=0)\r\n", "print(\"size of the concatenated data:\",stacked_sample_1.shape)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "sizes of the original data: (100, 100, 3) (100, 100, 3)\n", "size of the concatenated data: (2, 100, 100, 3)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "0pugCjYKVUjx" }, "source": [ "### Tensor Splitting" ] }, { "cell_type": "markdown", "metadata": { "id": "eYpwk-E6VXEA" }, "source": [ "In TensorFlow, tensor splitting operations include:\r\n", "*\ttf.unstack(): splits a tensor by a specific dimension.\r\n", "*\ttf.split(): splits a tensor into a specified number of sub tensors based on a specific dimension.\r\n", "*\ttf.split() is more flexible than tf.unstack().\r\n", "*\ttf.unstack(value,num=None,axis=0,name='unstack'):\r\n", "*\tvalue: input tensor\r\n", "*\tnum: indicates that a list containing num elements is output. The value of num must be the same as the number of elements in the specified dimension. This parameter can generally be ignored.\r\n", "*\taxis: specifies the dimension based on which the tensor is split.\r\n", "*\tname: operation name\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UeUh_sXkVbAE", "outputId": "9d8d50c8-cd5b-44fb-9c8d-6097a08c5cdd" }, "source": [ "#Split data based on the first dimension and output the split data in a list.\r\n", "tf.unstack(stacked_sample_1,axis=0)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[,\n", " ]" ] }, "metadata": { "tags": [] }, "execution_count": 37 } ] }, { "cell_type": "markdown", "metadata": { "id": "zM8I8-x0VilR" }, "source": [ "tf.split(value, num_or_size_splits, axis=0, num=None, name='split'):\r\n", "*\tvalue: The Tensor to split.\r\n", "*\tnum_or_size_splits:\tEither an integer indicating the number of splits along axis or a 1-D integer Tensor or Python list containing the sizes of each output tensor along axis. If a scalar, then it must evenly divide value.shape[axis]; otherwise the sum of sizes along the split axis must match that of the value.\r\n", "*\taxis: An integer or scalar int32 Tensor. The dimension along which to split. Must be in the range [-rank(value), rank(value)). Defaults to 0.\r\n", "*\tnum: Optional, used to specify the number of outputs when it cannot be inferred from the shape of size_splits.\r\n", "*\tname:\tA name for the operation (optional).\r\n", "\r\n", "tf.split() splits a tensor in either of the following ways:\r\n", "*\tIf the value of num_or_size_splits is an integer, the tensor is evenly split into sub tensors in the specified dimension (axis = D).\r\n", "*\tIf the value of num_or_size_splits is a vector, the tensor is split into sub tensors based on the element value of the vector in the specified dimension (axis = D).\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "94K3wIWnVlrL", "outputId": "60f953e4-62f1-4a42-8b1e-342aaaddb7b1" }, "source": [ "import numpy as np\r\n", "split_sample_1 = tf.random.normal([10,100,100,3])\r\n", "print(\"size of the original data:\",split_sample_1.shape)\r\n", "splited_sample_1 = tf.split(split_sample_1, num_or_size_splits=5,axis=0)\r\n", "print(\"size of the split data when m_or_size_splits is set to 10: \",np.shape(splited_sample_1))\r\n", "splited_sample_2 = tf.split(split_sample_1, num_or_size_splits=[3,5,2],axis=0)\r\n", "print(\"sizes of the split data when num_or_size_splits is set to [3,5,2]:\",\r\n", " np.shape(splited_sample_2[0]),\r\n", " np.shape(splited_sample_2[1]),\r\n", " np.shape(splited_sample_2[2]))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "size of the original data: (10, 100, 100, 3)\n", "size of the split data when m_or_size_splits is set to 10: (5, 2, 100, 100, 3)\n", "sizes of the split data when num_or_size_splits is set to [3,5,2]: (3, 100, 100, 3) (5, 100, 100, 3) (2, 100, 100, 3)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "hrt-n5htVqf6" }, "source": [ "## Tensor Sorting" ] }, { "cell_type": "markdown", "metadata": { "id": "a81f_H1LVtjP" }, "source": [ "In TensorFlow, tensor sorting operations include:\r\n", "*\ttf.sort(): sorts tensors in ascending or descending order and returns the sorted tensors.\r\n", "*\ttf.argsort(): sorts tensors in ascending or descending order, and returns tensor indexes.\r\n", "\r\n", "tf.nn.top_k(): returns the first k maximum values.\r\n", "\r\n", "*\ttf.sort/argsort(input, direction, axis):\r\n", "*\tinput: input tensor\r\n", "*\tdirection: sorting order, which can be set to DESCENDING (descending order) or ASCENDING (ascending order). The default value is ASCENDING.\r\n", "*\taxis: sorting by the dimension specified by axis. The default value of axis is –1, indicating the last dimension.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "SGILutMlV1Fr", "outputId": "2097c524-729a-4dcb-884f-bd6b9aa72e6a" }, "source": [ "sort_sample_1 = tf.random.shuffle(tf.range(10))\r\n", "print(\"input tensor:\",sort_sample_1.numpy())\r\n", "sorted_sample_1 = tf.sort(sort_sample_1, direction=\"ASCENDING\")\r\n", "print(\"tensor sorted in ascending order:\",sorted_sample_1.numpy())\r\n", "sorted_sample_2 = tf.argsort(sort_sample_1,direction=\"ASCENDING\")\r\n", "print(\"indexes of elements in ascending order:\",sorted_sample_2.numpy())" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "input tensor: [4 2 7 3 8 9 0 1 6 5]\n", "tensor sorted in ascending order: [0 1 2 3 4 5 6 7 8 9]\n", "indexes of elements in ascending order: [6 7 1 3 0 9 8 2 4 5]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "xv6qCBHtV4cH" }, "source": [ "tf.nn.top_k(input,k=1,sorted=True,name=None):\r\n", "*\tinput: 1-D or higher Tensor with last dimension at least k.\r\n", "*\tK: 0-D int32 Tensor. Number of top elements to look for along the last dimension (along each row for matrices).\r\n", "*\tsorted: If true the resulting k elements will be sorted by the values in descending order.\r\n", "*\tname:\tOptional name for the operation.\r\n", "\r\n", "Return two tensors:\r\n", "*\tvalues: k maximum values in each row\r\n", "*\tindices: positions of elements in the last dimension of the input tensor\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9faG7ZDrV7o_", "outputId": "04952138-8d81-4cfc-fb58-3e7e94fdd9c9" }, "source": [ "values, index = tf.nn.top_k(sort_sample_1,5)\r\n", "print(\"input tensor:\",sort_sample_1.numpy())\r\n", "print(\"first five values in ascending order:\", values.numpy())\r\n", "print(\"indexes of the first five values in ascending order:\", index.numpy())" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "input tensor: [4 2 7 3 8 9 0 1 6 5]\n", "first five values in ascending order: [9 8 7 6 5]\n", "indexes of the first five values in ascending order: [5 4 2 8 9]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "j_OFOtsLWBDY" }, "source": [ "# Common Modules of TensorFlow 2.x" ] }, { "cell_type": "markdown", "metadata": { "id": "lSFy6w5lWnDK" }, "source": [ "## Model Building" ] }, { "cell_type": "markdown", "metadata": { "id": "Zxa3U1UcW9WU" }, "source": [ "### Stacking a Model (tf.keras.Sequential)" ] }, { "cell_type": "markdown", "metadata": { "id": "35H8-xLaXAaY" }, "source": [ "The most common way to build a model is to stack layers by using tf.keras.Sequential." ] }, { "cell_type": "code", "metadata": { "id": "Lybz84VwXCgf" }, "source": [ "import tensorflow.keras.layers as layers\r\n", "model = tf.keras.Sequential()\r\n", "model.add(layers.Dense(32, activation='relu'))\r\n", "model.add(layers.Dense(32, activation='relu'))\r\n", "model.add(layers.Dense(10, activation='softmax'))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "8jm5y3JFXFdV" }, "source": [ "### Building a Functional Model" ] }, { "cell_type": "markdown", "metadata": { "id": "SOpPaeKSXJSJ" }, "source": [ "Functional models are mainly built by using tf.keras.Input and tf.keras.Model, which are more complex than tf.keras.Sequential but have a good effect. Variables can be input at the same time or in different phases, and data can be output in different phases. Functional models are preferred if more than one model output is needed.\r\n", "Stacked model (.Sequential) vs. functional model (.Model):\r\n", "The tf.keras.Sequential model is a simple stack of layers that cannot represent arbitrary models. You can use the Keras functional API to build complex model topologies such as:\r\n", "* Multi-input models\r\n", "*\tMulti-output models\r\n", "*\tModels with shared layers\r\n", "*\tModels with non-sequential data flows (for example, residual connections)\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wcOqChFqXMpg", "outputId": "b2aa28a6-4086-4171-94b8-f3e00014926e" }, "source": [ "#Use the output of the previous layer as the input of the next layer.\r\n", "x = tf.keras.Input(shape=(32,))\r\n", "h1 = layers.Dense(32, activation='relu')(x)\r\n", "h2 = layers.Dense(32, activation='relu')(h1)\r\n", "y = layers.Dense(10, activation='softmax')(h2)\r\n", "model_sample_2 = tf.keras.models.Model(x, y)\r\n", "\r\n", "#Print model information.\r\n", "model_sample_2.summary()" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Model: \"model\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input_1 (InputLayer) [(None, 32)] 0 \n", "_________________________________________________________________\n", "dense_3 (Dense) (None, 32) 1056 \n", "_________________________________________________________________\n", "dense_4 (Dense) (None, 32) 1056 \n", "_________________________________________________________________\n", "dense_5 (Dense) (None, 10) 330 \n", "=================================================================\n", "Total params: 2,442\n", "Trainable params: 2,442\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "_EjKm3zIXP6h" }, "source": [ "### Building a Network Layer (tf.keras.layers)" ] }, { "cell_type": "markdown", "metadata": { "id": "NTfNtmikXWJH" }, "source": [ "The tf.keras.layers module is used to configure neural network layers. Common classes include:\r\n", "*\ttf.keras.layers.Dense: builds a fully connected layer.\r\n", "*\ttf.keras.layers.Conv2D: builds a two-dimensional convolutional layer.\r\n", "*\ttf.keras.layers.MaxPooling2D/AveragePooling2D: builds a maximum/average pooling layer.\r\n", "*\ttf.keras.layers.RNN: builds a recurrent neural network layer.\r\n", "*\ttf.keras.layers.LSTM/tf.keras.layers.LSTMCell: builds an LSTM network layer/LSTM unit.\r\n", "*\ttf.keras.layers.GRU/tf.keras.layers.GRUCell: builds a GRU unit/GRU network layer. (GRU: Gated Reccurrent Unit)\r\n", "*\ttf.keras.layers.Embedding: converts a positive integer (subscript) into a vector of a fixed size, for example, converts [[4], [20]] into [[0.25, 0.1], [0.6, –0.2]]. The embedding layer can be used only as the first model layer.\r\n", "*\ttf.keras.layers.Dropout: builds the dropout layer.\r\n", "The following describes tf.keras.layers.Dense, tf.keras.layers.Conv2D, tf.keras.layers.MaxPooling2D/AveragePooling2D, and tf.keras.layers.LSTM/tf.keras.layers.LSTMCell.\r\n", "\r\n", "Main network configuration parameters in **tf.keras.layers** include:\r\n", "*\tactivation: sets the activation function for the layer. By default, the system applies no activation function.\r\n", "*\tkernel_initializer and bias_initializer: initialization schemes that create the layer's weights (kernel and bias). This defaults to the Glorot uniform initializer.\r\n", "*\tkernel_regularizer and bias_regularizer: regularization schemes that apply to the layer's weights (kernel and bias), such as L1 or L2 regularization. By default, the system applies no regularization function.\r\n" ] }, { "cell_type": "markdown", "metadata": { "id": "zSNngcGhXey7" }, "source": [ "Main configuration parameters in **tf.keras.layers.Dense** include:\r\n", "*\tunits: number of neurons\r\n", "*\tactivation: sets the activation function.\r\n", "*\tuse_bias: indicates whether to use bias terms. Bias terms are used by default.\r\n", "*\tkernel_initializer: initialization scheme that creates the layer's weight (kernel)\r\n", "*\tbias_initializer: initialization scheme that creates the layer's weight (bias)\r\n", "*\tkernel_regularizer: regularization scheme that applies to the layer's weight (kernel)\r\n", "*\tbias_regularizer: regularization scheme that applies to the layer's weight (bias)\r\n", "*\tactivity_regularizer: regular item applied to the output, a regularizer object\r\n", "*\tkernel_constraint: a constraint applied to a weight\r\n", "*\tbias_constraint: a constraint applied to a weight\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "YFR_bNZGXmY0", "outputId": "3208d8be-3425-465e-c329-c8684e23e14d" }, "source": [ "#Create a fully connected layer that contains 32 neurons. Set the activation function to sigmoid.\r\n", "#The activation parameter can be set to a function name string, for example, sigmoid or a function object, for example, tf.sigmoid.\r\n", "layers.Dense(32, activation='sigmoid')\r\n", "layers.Dense(32, activation=tf.sigmoid)\r\n", "\r\n", "#Set kernel_initializer.\r\n", "layers.Dense(32, kernel_initializer=tf.keras.initializers.he_normal)\r\n", "#Set kernel_regularizer to L2 regularization.\r\n", "layers.Dense(32, kernel_regularizer=tf.keras.regularizers.l2(0.01))" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 43 } ] }, { "cell_type": "markdown", "metadata": { "id": "lqebg2AjXpzf" }, "source": [ "Main configuration parameters in **tf.keras.layers.Conv2D** include:\r\n", "*\tfilters: number of convolution kernels (output dimensions)\r\n", "*\tkernel_size: width and length of a convolution kernel\r\n", "*\tstrides: convolution step\r\n", "*\tpadding: zero padding policy\r\n", "*\tWhen padding is set to valid, only valid convolution is performed, that is, boundary data is not processed. When padding is set to same, the convolution result at the boundary is reserved, and consequently, the output shape is usually the same as the input shape.\r\n", "*\tactivation: sets the activation function.\r\n", "*\tdata_format: data format, set to channels_first or channels_last. For example, for a 128 x 128 RGB image, data is organized as (3, 128, 128) if the value is channels_first, and (128, 128, 3) if the value is channels_last. The default value of this parameter is the value specified in ~/.keras/keras.json. If this parameter has never been set, the default value is channels_last.\r\n", "Other parameters include use_bias, kernel_initializer, bias_initializer, kernel_regularizer, bias_regularizer, activity_regularizer, kernel_constraints, and bias_constraints.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "b80FSeT3Xtzp", "outputId": "afde7fa8-c4ea-4b7d-d1cf-c09461a0b8df" }, "source": [ "layers.Conv2D(64,[1,1],2,padding='same',activation=\"relu\")" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 44 } ] }, { "cell_type": "markdown", "metadata": { "id": "1-aKaRDwX3M7" }, "source": [ "Main configuration parameters in **tf.keras.layers.MaxPool2D/AveragePool2D** include :\r\n", "*\tpool_size: size of the pooled kernel. For example, if the matrix (2, 2) is used, the picture becomes half of the original length in both dimensions. If this parameter is set to an integer, the integer is the values of all dimensions.\r\n", "* strides: Integer, tuple of 2 integers, or None. Strides values. Specifies how far the pooling window moves for each pooling step. If None, it will default to pool_size.\r\n", "*\tpadding: One of \"valid\" or \"same\" (case-insensitive). \"valid\" adds no zero padding. \"same\" adds padding such that if the stride is 1, the output shape is the same as input shape.\r\n", "*\tdata_format: A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be \"channels_last\".\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "cwrpm-P7X8Qd", "outputId": "ba549925-2ec3-4de0-8a59-0145a28eb4e5" }, "source": [ "layers.MaxPool2D(pool_size=(2,2),strides=(2,1))" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 46 } ] }, { "cell_type": "markdown", "metadata": { "id": "6CNLSVhrYBQO" }, "source": [ "Main configuration parameters in **tf.keras.layers.LSTM/tf.keras.layers.LSTMCell** include:\r\n", "*\tunits: output dimension\r\n", "*\tactivation: sets the activation function.\r\n", "*\trecurrent_activation: activation function to use for the recurrent step\r\n", "*\treturn_sequences: If the value is True, the system returns the full sequence. If the value is False, the system returns the output in the last cell of the output sequence.\r\n", "* return_state: Boolean value, indicating whether to return the last state in addition to the output.\r\n", "*\tdropout: float between 0 and 1, fraction of the neurons to drop for the linear transformation of the inputs.\r\n", "*\trecurrent_dropout: float between 0 and 1, fraction of the neurons to drop for the linear transformation of the recurrent state. \r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0HGdLP7lYIWL", "outputId": "88934b08-4360-48dd-978f-03a396d40873" }, "source": [ "import numpy as np\r\n", "inputs = tf.keras.Input(shape=(3, 1))\r\n", "lstm = layers.LSTM(1, return_sequences=True)(inputs)\r\n", "model_lstm_1 = tf.keras.models.Model(inputs=inputs, outputs=lstm)\r\n", "\r\n", "inputs = tf.keras.Input(shape=(3, 1))\r\n", "lstm = layers.LSTM(1, return_sequences=False)(inputs)\r\n", "model_lstm_2 = tf.keras.models.Model(inputs=inputs, outputs=lstm)\r\n", "\r\n", "#Sequences t1, t2, and t3\r\n", "data = [[[0.1],\r\n", " [0.2],\r\n", " [0.3]]]\r\n", "print(data)\r\n", "print(\"output when return_sequences is set to True\",model_lstm_1.predict(data))\r\n", "print(\"output when return_sequences is set to False\",model_lstm_2.predict(data))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "[[[0.1], [0.2], [0.3]]]\n", "output when return_sequences is set to True [[[0.02775694]\n", " [0.08219695]\n", " [0.15732154]]]\n", "output when return_sequences is set to False [[0.0585606]]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "2YGCcKRIYLaD" }, "source": [ "LSTMcell is the implementation unit of the LSTM layer.\r\n", "*\tLSTM is an LSTM network layer.\r\n", "*\tLSTMcell is a single-step computing unit, that is, an LSTM unit.\r\n" ] }, { "cell_type": "code", "metadata": { "id": "2yiYcyPIYOkJ" }, "source": [ "#LSTM\r\n", "tf.keras.layers.LSTM(16, return_sequences=True)\r\n", "\r\n", "#LSTMCell\r\n", "x = tf.keras.Input((None, 3))\r\n", "y = layers.RNN(layers.LSTMCell(16))(x)\r\n", "model_lstm_3= tf.keras.Model(x, y)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "iwC5l2grWnMP" }, "source": [ "## Training and Evaluation" ] }, { "cell_type": "markdown", "metadata": { "id": "92crhov9YT8Z" }, "source": [ "### Model Compilation" ] }, { "cell_type": "markdown", "metadata": { "id": "TcrxYrUJYbeY" }, "source": [ "After a model is built, you can call compile to configure the learning process of the model:\r\n", "\r\n", "compile(optimizer='rmsprop', loss=None, metrics=None, loss_weights=None,\r\n", " weighted_metrics=None, run_eagerly=None, **kwargs):\r\n", "\r\n", "*\toptimizer: String (name of optimizer) or optimizer instance. \r\n", "*\tloss: loss function, cross entropy for binary tasks and MSE for regression tasks\r\n", "*\tmetrics: model evaluation criteria during training and testing For example, metrics can be set to ['accuracy']. To specify multiple evaluation criteria, set a dictionary, for example, set metrics to {'output_a':'accuracy'}.\r\n", "*\tloss_weights: If the model has multiple task outputs, you need to specify a weight for each output when optimizing the global loss.\r\n", "*\tweighted_metrics: List of metrics to be evaluated and weighted by sample_weight or class_weight during training and testing.\r\n", "*\trun_eagerly: Bool. Defaults to False. If True, this Model's logic will not be wrapped in a tf.function. Recommended to leave this as None unless your Model cannot be run inside a tf.function.\r\n", "*\t**kwargs: Any additional arguments. Supported arguments:\r\n", "experimental_steps_per_execution: Int. The number of batches to run during each tf.function call. Running multiple batches inside a single tf.function call can greatly improve performance on TPUs or small models with a large Python overhead. Note that if this value is set to N, Callback.on_batch methods will only be called every N batches. This currently defaults to 1. At most, one full epoch will be run each execution. If a number larger than the size of the epoch is passed, the execution will be truncated to the size of the epoch.\r\n", "\r\n", "sample_weight_mode for backward compatibility.\r\n" ] }, { "cell_type": "code", "metadata": { "id": "okFCxXePYgGJ" }, "source": [ "model = tf.keras.Sequential()\r\n", "model.add(layers.Dense(10, activation='softmax'))\r\n", "#Determine the optimizer (optimizer), loss function (loss), and model evaluation method (metrics).\r\n", "model.compile(optimizer=tf.keras.optimizers.Adam(0.001),\r\n", " loss=tf.keras.losses.categorical_crossentropy,\r\n", " metrics=[tf.keras.metrics.categorical_accuracy])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "YXPzpdVjYjHU" }, "source": [ "### Model Training" ] }, { "cell_type": "markdown", "metadata": { "id": "4TP0N_qXYmTx" }, "source": [ "fit(x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None,\r\n", " validation_split=0.0, validation_data=None, shuffle=True, class_weight=None,\r\n", " sample_weight=None, initial_epoch=0, steps_per_epoch=None,\r\n", " validation_steps=None, validation_batch_size=None, validation_freq=1,\r\n", " max_queue_size=10, workers=1, use_multiprocessing=False):\r\n", "\r\n", "*\tx: input training data\r\n", "*\ty: target (labeled) data\r\n", "*\tbatch_size: number of samples for each gradient update The default value is 32.\r\n", "*\tepochs: number of iteration rounds of the training model\r\n", "*\tverbose: log display mode, set to 0, 1, or 2. \r\n", "**\t0: no display\r\n", "**\t1: progress bar\r\n", "**\t2: one line for each round\r\n", "*\tcallbacks: callback function used during training\r\n", "*\tvalidation_split: fraction of the training data to be used as validation data\r\n", "*\tvalidation_data: validation set. This parameter will overwrite validation_split.\r\n", "*\tshuffle: indicates whether to shuffle data before each round of iteration. This parameter is invalid when steps_per_epoch is not None. \r\n", "*\tinitial_epoch: epoch at which to start training (useful for resuming a previous training weight)\r\n", "*\tsteps_per_epoch: set to the dataset size or batch_size\r\n", "*\tvalidation_steps: Total number of steps (batches of samples) to validate before stopping. This parameter is valid only when steps_per_epoch is specified.\r\n", "*\tvalidation_batch_size: Integer or None. Number of samples per validation batch\r\n", "*\tvalidation_freq:\tOnly relevant if validation data is provided. Integer or collections_abc.\r\n", "*\tmax_queue_size\tInteger. Used for generator or keras.utils.Sequence input only. Maximum size for the generator queue. If unspecified, max_queue_size will default to 10.\r\n", "*\tworkers: Integer. Used for generator or keras.utils.Sequence input only. Maximum number of processes to spin up when using process-based threading. If unspecified, workers will default to 1. If 0, will execute the generator on the main thread.\r\n", "*\tuse_multiprocessing:\tBoolean. Used for generator or keras.utils.Sequence input only. If True, use process-based threading. If unspecified, use_multiprocessing will default to False. Note that because this implementation relies on multiprocessing, you should not pass non-picklable arguments to the generator as they can't be passed easily to children processes.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "qmnnvn1DY4RR", "outputId": "5aa68950-b0c0-461d-ec35-6fdeddf38cf1" }, "source": [ "import numpy as np\r\n", "train_x = np.random.random((1000, 36))\r\n", "train_y = np.random.random((1000, 10))\r\n", "val_x = np.random.random((200, 36))\r\n", "val_y = np.random.random((200, 10))\r\n", "model.fit(train_x, train_y, epochs=10, batch_size=100,\r\n", " validation_data=(val_x, val_y))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Epoch 1/10\n", "10/10 [==============================] - 1s 32ms/step - loss: 12.6468 - categorical_accuracy: 0.1093 - val_loss: 12.7299 - val_categorical_accuracy: 0.0550\n", "Epoch 2/10\n", "10/10 [==============================] - 0s 7ms/step - loss: 12.6935 - categorical_accuracy: 0.1218 - val_loss: 12.7295 - val_categorical_accuracy: 0.0600\n", "Epoch 3/10\n", "10/10 [==============================] - 0s 6ms/step - loss: 12.6134 - categorical_accuracy: 0.1075 - val_loss: 12.7284 - val_categorical_accuracy: 0.0600\n", "Epoch 4/10\n", "10/10 [==============================] - 0s 6ms/step - loss: 12.6448 - categorical_accuracy: 0.1258 - val_loss: 12.7272 - val_categorical_accuracy: 0.0600\n", "Epoch 5/10\n", "10/10 [==============================] - 0s 8ms/step - loss: 12.7151 - categorical_accuracy: 0.1115 - val_loss: 12.7270 - val_categorical_accuracy: 0.0600\n", "Epoch 6/10\n", "10/10 [==============================] - 0s 6ms/step - loss: 12.6250 - categorical_accuracy: 0.1083 - val_loss: 12.7268 - val_categorical_accuracy: 0.0600\n", "Epoch 7/10\n", "10/10 [==============================] - 0s 6ms/step - loss: 12.6509 - categorical_accuracy: 0.1063 - val_loss: 12.7249 - val_categorical_accuracy: 0.0600\n", "Epoch 8/10\n", "10/10 [==============================] - 0s 6ms/step - loss: 12.6965 - categorical_accuracy: 0.1156 - val_loss: 12.7248 - val_categorical_accuracy: 0.0600\n", "Epoch 9/10\n", "10/10 [==============================] - 0s 6ms/step - loss: 12.5985 - categorical_accuracy: 0.1154 - val_loss: 12.7242 - val_categorical_accuracy: 0.0600\n", "Epoch 10/10\n", "10/10 [==============================] - 0s 6ms/step - loss: 12.6299 - categorical_accuracy: 0.1066 - val_loss: 12.7246 - val_categorical_accuracy: 0.0600\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 51 } ] }, { "cell_type": "markdown", "metadata": { "id": "3soPvV0zY9Er" }, "source": [ "You can use **tf.data** to build training input pipelines for large datasets." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "CVNZSMp_Y_Zj", "outputId": "a594eef2-f999-436e-966d-ca78a2266d44" }, "source": [ "dataset = tf.data.Dataset.from_tensor_slices((train_x, train_y))\r\n", "dataset = dataset.batch(32)\r\n", "dataset = dataset.repeat()\r\n", "val_dataset = tf.data.Dataset.from_tensor_slices((val_x, val_y))\r\n", "val_dataset = val_dataset.batch(32)\r\n", "val_dataset = val_dataset.repeat()\r\n", "\r\n", "model.fit(dataset, epochs=10, steps_per_epoch=30,\r\n", " validation_data=val_dataset, validation_steps=3)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Epoch 1/10\n", "30/30 [==============================] - 0s 7ms/step - loss: 12.6768 - categorical_accuracy: 0.1146 - val_loss: 12.7037 - val_categorical_accuracy: 0.0521\n", "Epoch 2/10\n", "30/30 [==============================] - 0s 2ms/step - loss: 12.6483 - categorical_accuracy: 0.1154 - val_loss: 12.7033 - val_categorical_accuracy: 0.0521\n", "Epoch 3/10\n", "30/30 [==============================] - 0s 2ms/step - loss: 12.6606 - categorical_accuracy: 0.1079 - val_loss: 12.7028 - val_categorical_accuracy: 0.0521\n", "Epoch 4/10\n", "30/30 [==============================] - 0s 2ms/step - loss: 12.6244 - categorical_accuracy: 0.1154 - val_loss: 12.7014 - val_categorical_accuracy: 0.0521\n", "Epoch 5/10\n", "30/30 [==============================] - 0s 7ms/step - loss: 12.6703 - categorical_accuracy: 0.1090 - val_loss: 12.6994 - val_categorical_accuracy: 0.0521\n", "Epoch 6/10\n", "30/30 [==============================] - 0s 3ms/step - loss: 12.6715 - categorical_accuracy: 0.1132 - val_loss: 12.6970 - val_categorical_accuracy: 0.0521\n", "Epoch 7/10\n", "30/30 [==============================] - 0s 2ms/step - loss: 12.6313 - categorical_accuracy: 0.1090 - val_loss: 12.6952 - val_categorical_accuracy: 0.0521\n", "Epoch 8/10\n", "30/30 [==============================] - 0s 2ms/step - loss: 12.6373 - categorical_accuracy: 0.1175 - val_loss: 12.6930 - val_categorical_accuracy: 0.0521\n", "Epoch 9/10\n", "30/30 [==============================] - 0s 3ms/step - loss: 12.6467 - categorical_accuracy: 0.1122 - val_loss: 12.6902 - val_categorical_accuracy: 0.0521\n", "Epoch 10/10\n", "30/30 [==============================] - 0s 3ms/step - loss: 12.6450 - categorical_accuracy: 0.1090 - val_loss: 12.6877 - val_categorical_accuracy: 0.0521\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 52 } ] }, { "cell_type": "markdown", "metadata": { "id": "j2WwnchsZCfr" }, "source": [ "### Callback Functions" ] }, { "cell_type": "markdown", "metadata": { "id": "bwLgAn0pZFtP" }, "source": [ "A callback function is an object passed to the model to customize and extend the model's behavior during training. You can customize callback functions or use embedded functions in tf.keras.callbacks. Common embedded callback functions include:\r\n", "*\ttf.keras.callbacks.ModelCheckpoint: periodically saves models.\r\n", "* tf.keras.callbacks.LearningRateScheduler: dynamically changes the learning rate.\r\n", "*\ttf.keras.callbacks.EarlyStopping: stops the training in advance.\r\n", "*\ttf.keras.callbacks.TensorBoard: exports and visualizes the training progress and results with TensorBoard.\r\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "OelcqAPMZJwK", "outputId": "0c4c89d3-1d67-4236-d798-3682654926a7" }, "source": [ "#Set hyperparameters.\r\n", "Epochs = 10\r\n", "\r\n", "#Define a function for dynamically setting the learning rate.\r\n", "def lr_Scheduler(epoch):\r\n", " if epoch > 0.9 * Epochs:\r\n", " lr = 0.0001\r\n", " elif epoch > 0.5 * Epochs:\r\n", " lr = 0.001\r\n", " elif epoch > 0.25 * Epochs:\r\n", " lr = 0.01\r\n", " else:\r\n", " lr = 0.1\r\n", " \r\n", " print(lr)\r\n", " return lr\r\n", " \r\n", "\r\n", "callbacks = [\r\n", " #Early stopping:\r\n", " tf.keras.callbacks.EarlyStopping(\r\n", " #Metric for determining whether the model performance has no further improvement\r\n", " monitor='val_loss',\r\n", " #Threshold for determining whether the model performance has no further improvement\r\n", " min_delta=1e-2,\r\n", " #Number of epochs in which the model performance has no further improvement\r\n", " patience=2),\r\n", " \r\n", " #Periodically save models.\r\n", " tf.keras.callbacks.ModelCheckpoint(\r\n", " #Model path\r\n", " filepath='testmodel_{epoch}.h5',\r\n", " #Whether to save the optimal model.\r\n", " save_best_only=True,\r\n", " #Monitored metric\r\n", " monitor='val_loss'),\r\n", " \r\n", " #Dynamically change the learning rate.\r\n", " tf.keras.callbacks.LearningRateScheduler(lr_Scheduler),\r\n", " \r\n", " #Use TensorBoard.\r\n", " tf.keras.callbacks.TensorBoard(log_dir='./logs')\r\n", "]\r\n", "\r\n", "model.fit(train_x, train_y, batch_size=16, epochs=Epochs,\r\n", " callbacks=callbacks, validation_data=(val_x, val_y))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Epoch 1/10\n", "0.1\n", "63/63 [==============================] - 1s 7ms/step - loss: 12.8365 - categorical_accuracy: 0.0900 - val_loss: 12.9831 - val_categorical_accuracy: 0.0700\n", "Epoch 2/10\n", "0.1\n", "63/63 [==============================] - 0s 3ms/step - loss: 12.8342 - categorical_accuracy: 0.1020 - val_loss: 12.7327 - val_categorical_accuracy: 0.0700\n", "Epoch 3/10\n", "0.1\n", "63/63 [==============================] - 0s 2ms/step - loss: 12.8377 - categorical_accuracy: 0.0970 - val_loss: 12.8636 - val_categorical_accuracy: 0.0700\n", "Epoch 4/10\n", "0.01\n", "63/63 [==============================] - 0s 3ms/step - loss: 12.6808 - categorical_accuracy: 0.1020 - val_loss: 12.6276 - val_categorical_accuracy: 0.0750\n", "Epoch 5/10\n", "0.01\n", "63/63 [==============================] - 0s 2ms/step - loss: 12.5388 - categorical_accuracy: 0.1060 - val_loss: 12.6729 - val_categorical_accuracy: 0.0650\n", "Epoch 6/10\n", "0.01\n", "63/63 [==============================] - 0s 2ms/step - loss: 12.6101 - categorical_accuracy: 0.1070 - val_loss: 12.7205 - val_categorical_accuracy: 0.0700\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 53 } ] }, { "cell_type": "markdown", "metadata": { "id": "9JjoWxT3ZNlE" }, "source": [ "To load tensorboard" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 357 }, "id": "G9FFZ1SaZOsw", "outputId": "05352241-a59c-492f-f7de-063697c734f0" }, "source": [ "%load_ext tensorboard\r\n", "import datetime, os\r\n", "\r\n", "%tensorboard --logdir logs" ], "execution_count": 1, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "Launching TensorBoard..." ] }, "metadata": { "tags": [] } }, { "output_type": "error", "ename": "KeyboardInterrupt", "evalue": "ignored", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mdatetime\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmagic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'tensorboard --logdir logs'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mmagic\u001b[0;34m(self, arg_s)\u001b[0m\n\u001b[1;32m 2158\u001b[0m \u001b[0mmagic_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmagic_arg_s\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0marg_s\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpartition\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m' '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2159\u001b[0m \u001b[0mmagic_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmagic_name\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlstrip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefilter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mESC_MAGIC\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2160\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_line_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmagic_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2161\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2162\u001b[0m \u001b[0;31m#-------------------------------------------------------------------------\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_line_magic\u001b[0;34m(self, magic_name, line)\u001b[0m\n\u001b[1;32m 2079\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'local_ns'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_getframe\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstack_depth\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf_locals\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2080\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2081\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2082\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2083\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorboard/notebook.py\u001b[0m in \u001b[0;36m_start_magic\u001b[0;34m(line)\u001b[0m\n\u001b[1;32m 126\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_start_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 127\u001b[0m \u001b[0;34m\"\"\"Implementation of the `%tensorboard` line magic.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 128\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 129\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 130\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorboard/notebook.py\u001b[0m in \u001b[0;36mstart\u001b[0;34m(args_string)\u001b[0m\n\u001b[1;32m 160\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 161\u001b[0m \u001b[0mparsed_args\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mshlex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs_string\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcomments\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mposix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 162\u001b[0;31m \u001b[0mstart_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmanager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 163\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 164\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_result\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmanager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mStartLaunched\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorboard/manager.py\u001b[0m in \u001b[0;36mstart\u001b[0;34m(arguments, timeout)\u001b[0m\n\u001b[1;32m 423\u001b[0m \u001b[0mend_time_seconds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstart_time_seconds\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtotal_seconds\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 424\u001b[0m \u001b[0;32mwhile\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0mend_time_seconds\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 425\u001b[0;31m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpoll_interval_seconds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 426\u001b[0m \u001b[0msubprocess_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpoll\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 427\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0msubprocess_result\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mKeyboardInterrupt\u001b[0m: " ] } ] }, { "cell_type": "markdown", "metadata": { "id": "lt2VovR_Zl27" }, "source": [ "### Evaluation and Prediction" ] }, { "cell_type": "markdown", "metadata": { "id": "FIiI4u2GZocs" }, "source": [ "Evaluation and prediction functions: **tf.keras.Model.evaluate** and **tf.keras.Model.predict**." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wYqYhRPgZtf_", "outputId": "725e5a41-71e4-404a-9f0a-92599ef2b75d" }, "source": [ "#Model evaluation\r\n", "test_x = np.random.random((1000, 36))\r\n", "test_y = np.random.random((1000, 10))\r\n", "model.evaluate(test_x, test_y, batch_size=32)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "32/32 [==============================] - 0s 2ms/step - loss: 12.5588 - categorical_accuracy: 0.0920\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/plain": [ "[12.558799743652344, 0.09200000017881393]" ] }, "metadata": { "tags": [] }, "execution_count": 59 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5ytugTm1Zvx9", "outputId": "d0bace5d-35c0-465e-ed88-7c6856139936" }, "source": [ "#Model prediction\r\n", "pre_x = np.random.random((10, 36))\r\n", "result = model.predict(test_x,)\r\n", "print(result)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "[[0.08501494 0.13683964 0.04604724 ... 0.0897471 0.03132875 0.1954582 ]\n", " [0.0702811 0.19352007 0.0443555 ... 0.13622904 0.05826329 0.12528194]\n", " [0.06081135 0.0902251 0.05563699 ... 0.1920235 0.08421462 0.17420028]\n", " ...\n", " [0.05497359 0.14996825 0.03822347 ... 0.09183615 0.04417118 0.09440211]\n", " [0.06502375 0.0930286 0.1297896 ... 0.09527831 0.07760978 0.08561047]\n", " [0.0884625 0.10331904 0.03337023 ... 0.13911161 0.0956712 0.1463583 ]]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "EJ0_5UMUWnUq" }, "source": [ "## Model Saving and Restoration" ] }, { "cell_type": "markdown", "metadata": { "id": "fDui84L5Wp0x" }, "source": [ "### Saving and Restoring an Entire Model" ] }, { "cell_type": "code", "metadata": { "id": "g9NxYzYEWtoo" }, "source": [ "import numpy as np\r\n", "import os\r\n", "# create the file\r\n", "if not os.path.exists('./model/'):\r\n", " os.mkdir('./model/')\r\n", "#Save models.\r\n", "model.save('./model/the_save_model.h5')\r\n", "#Import models.\r\n", "new_model = tf.keras.models.load_model('./model/the_save_model.h5')\r\n", "new_prediction = new_model.predict(test_x)\r\n", "#np.testing.assert_allclose: determines whether the similarity between two objects exceeds the specified tolerance. If yes, the system displays an exception.\r\n", "#atol: specified tolerance\r\n", "np.testing.assert_allclose(result, new_prediction, atol=1e-6) # Prediction results are the same." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "0qYKvZ6yWve6" }, "source": [ "After a model is saved, you can find the corresponding weight file in the corresponding folder." ] }, { "cell_type": "markdown", "metadata": { "id": "F5V7SewTWyH-" }, "source": [ "### Saving and Loading Network Weights Only" ] }, { "cell_type": "markdown", "metadata": { "id": "sX-iGF-0W0wC" }, "source": [ "If the weight name is suffixed with .h5 or .keras, save the weight as an HDF5 file, or otherwise, save the weight as a TensorFlow checkpoint file by default." ] }, { "cell_type": "code", "metadata": { "id": "qTX7BwOjW2BM" }, "source": [ "model.save_weights('./model/model_weights')\r\n", "model.save_weights('./model/model_weights.h5')\r\n", "#Load the weights.\r\n", "model.load_weights('./model/model_weights')\r\n", "model.load_weights('./model/model_weights.h5')" ], "execution_count": null, "outputs": [] } ] }