Skip to content

Commit 3c1cd3e

Browse files
Lukasz Kaisercopybara-github
Lukasz Kaiser
authored and
copybara-github
committedAug 3, 2020
Corrected JSON in the notebook.
PiperOrigin-RevId: 324548649
1 parent 9846583 commit 3c1cd3e

File tree

2 files changed

+1182
-0
lines changed

2 files changed

+1182
-0
lines changed
 

‎docs/source/notebooks/tf_numpy_and_keras.ipynb

+591
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,591 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"colab_type": "text",
7+
"id": "7yuytuIllsv1"
8+
},
9+
"source": [
10+
"# Using Trax with TensorFlow NumPy and Keras\n",
11+
"\n",
12+
"This notebook ([run it in colab](https://colab.research.google.com/github/google/trax/blob/master/trax/tf_numpy_and_keras.ipynb)) shows how you can run [Trax](https://trax-ml.readthedocs.io/en/latest/) directly with [TensorFlow NumPy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy). You will also see how to use Trax layers and models inside [Keras](https://keras.io/) so you can use Trax in production, e.g., with [TensorFlow.js](https://www.tensorflow.org/js/) or [TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving).\n",
13+
"\n",
14+
" 1. **Trax with TensorFlow NumPy**: use Trax with [TensorFlow NumPy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy) without any code changes\n",
15+
" 1. **Convert Trax to Keras**: how to get a [Keras](https://keras.io/) layer for your Trax model and use it\n",
16+
" 1. **Exporting Trax Models for Deployment**: how to export Trax models to [TensorFlow SavedModel](https://www.tensorflow.org/guide/saved_model)\n",
17+
" \n"
18+
]
19+
},
20+
{
21+
"cell_type": "markdown",
22+
"metadata": {
23+
"colab_type": "text",
24+
"id": "-LQ89rFFsEdk"
25+
},
26+
"source": [
27+
"## 1. Trax with TensorFlow NumPy\n",
28+
"\n",
29+
"In Trax, all computations rely on accelerated math operations happening in the `fastmath` module. This module can use different backends for acceleration. One of them is [TensorFlow NumPy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy) which uses [TensorFlow 2](https://www.tensorflow.org/) to accelerate the computations.\n",
30+
"\n",
31+
"The backend can be set using a call to `trax.fastmath.set_backend` as you'll see below. Currently available backends are `jax` (default), `tensorflow-numpy` and `numpy` (for debugging). The `tensorflow-numpy` backend uses [TensorFlow Numpy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy) for executing `fastmath` functions on TensorFlow, while the `jax` backend calls [JAX](https://github.com/google/jax) which lowers to TensorFlow XLA.\n",
32+
"\n",
33+
"You may see that `tensorflow-numpy` and `jax` backends show different speed and memory characteristics. You may also see different error messages when debugging since it might expose you to the internals of the backends. However for the most part, users can choose a backend and not worry about the internal details of these backends.\n",
34+
"\n",
35+
"Let's train the sentiment analysis model from the [Trax intro](https://colab.research.google.com/github/google/trax/blob/master/trax/intro.ipynb) using TensorFlow NumPy to see how it works."
36+
]
37+
},
38+
{
39+
"cell_type": "markdown",
40+
"metadata": {
41+
"colab_type": "text",
42+
"id": "BIl27504La0G"
43+
},
44+
"source": [
45+
"**General Setup**\n",
46+
"\n",
47+
"Execute the following few cells (once) before running any of the code samples."
48+
]
49+
},
50+
{
51+
"cell_type": "code",
52+
"execution_count": null,
53+
"metadata": {
54+
"cellView": "form",
55+
"colab": {},
56+
"colab_type": "code",
57+
"id": "oILRLCWN_16u"
58+
},
59+
"outputs": [],
60+
"source": [
61+
"#@title\n",
62+
"# Copyright 2020 Google LLC.\n",
63+
"\n",
64+
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
65+
"# you may not use this file except in compliance with the License.\n",
66+
"# You may obtain a copy of the License at\n",
67+
"\n",
68+
"# https://www.apache.org/licenses/LICENSE-2.0\n",
69+
"\n",
70+
"# Unless required by applicable law or agreed to in writing, software\n",
71+
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
72+
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
73+
"# See the License for the specific language governing permissions and\n",
74+
"# limitations under the License.\n",
75+
"\n",
76+
"\n"
77+
]
78+
},
79+
{
80+
"cell_type": "code",
81+
"execution_count": null,
82+
"metadata": {
83+
"cellView": "both",
84+
"colab": {},
85+
"colab_type": "code",
86+
"id": "vlGjGoGMTt-D"
87+
},
88+
"outputs": [],
89+
"source": [
90+
"# Install and import Trax\n",
91+
"!pip install -q -U git+https://github.com/google/trax@master\n",
92+
"\n",
93+
"import os\n",
94+
"import numpy as np\n",
95+
"import trax"
96+
]
97+
},
98+
{
99+
"cell_type": "markdown",
100+
"metadata": {
101+
"colab_type": "text",
102+
"id": "O_3JcfZaT5oP"
103+
},
104+
"source": [
105+
"Here is how you can set the fastmath backend to `tensorflow-numpy` and verify that it's been set."
106+
]
107+
},
108+
{
109+
"cell_type": "code",
110+
"execution_count": null,
111+
"metadata": {
112+
"colab": {
113+
"height": 34
114+
},
115+
"colab_type": "code",
116+
"executionInfo": {
117+
"elapsed": 492,
118+
"status": "ok",
119+
"timestamp": 1596262541312,
120+
"user": {
121+
"displayName": "",
122+
"photoUrl": "",
123+
"userId": ""
124+
},
125+
"user_tz": 420
126+
},
127+
"id": "djTiSLcaNFGa",
128+
"outputId": "a639c273-0bb2-481f-82aa-1264ac70c045"
129+
},
130+
"outputs": [
131+
{
132+
"name": "stdout",
133+
"output_type": "stream",
134+
"text": [
135+
"tensorflow-numpy\n"
136+
]
137+
}
138+
],
139+
"source": [
140+
"# Use the tensorflow-numpy backend.\n",
141+
"trax.fastmath.set_backend('tensorflow-numpy')\n",
142+
"print(trax.fastmath.backend_name())"
143+
]
144+
},
145+
{
146+
"cell_type": "code",
147+
"execution_count": null,
148+
"metadata": {
149+
"colab": {
150+
"height": 34
151+
},
152+
"colab_type": "code",
153+
"executionInfo": {
154+
"elapsed": 19481,
155+
"status": "ok",
156+
"timestamp": 1596262587458,
157+
"user": {
158+
"displayName": "",
159+
"photoUrl": "",
160+
"userId": ""
161+
},
162+
"user_tz": 420
163+
},
164+
"id": "AV5wrgjZ10yU",
165+
"outputId": "bef80331-f121-4d85-9d74-2ab76f077658"
166+
},
167+
"outputs": [
168+
{
169+
"name": "stdout",
170+
"output_type": "stream",
171+
"text": [
172+
"batch shapes = [(8, 2048), (8,), (8,)]\n"
173+
]
174+
}
175+
],
176+
"source": [
177+
"# Create data streams.\n",
178+
"train_stream = trax.data.TFDS('imdb_reviews', keys=('text', 'label'), train=True)()\n",
179+
"eval_stream = trax.data.TFDS('imdb_reviews', keys=('text', 'label'), train=False)()\n",
180+
"\n",
181+
"data_pipeline = trax.data.Serial(\n",
182+
" trax.data.Tokenize(vocab_file='en_8k.subword', keys=[0]),\n",
183+
" trax.data.Shuffle(),\n",
184+
" trax.data.FilterByLength(max_length=2048, length_keys=[0]),\n",
185+
" trax.data.BucketByLength(boundaries=[ 32, 128, 512, 2048],\n",
186+
" batch_sizes=[512, 128, 32, 8, 1],\n",
187+
" length_keys=[0]),\n",
188+
" trax.data.AddLossWeights()\n",
189+
" )\n",
190+
"train_batches_stream = data_pipeline(train_stream)\n",
191+
"eval_batches_stream = data_pipeline(eval_stream)\n",
192+
"\n",
193+
"# Print example shapes.\n",
194+
"example_batch = next(train_batches_stream)\n",
195+
"print(f'batch shapes = {[x.shape for x in example_batch]}')"
196+
]
197+
},
198+
{
199+
"cell_type": "code",
200+
"execution_count": null,
201+
"metadata": {
202+
"colab": {
203+
"height": 119
204+
},
205+
"colab_type": "code",
206+
"executionInfo": {
207+
"elapsed": 551,
208+
"status": "ok",
209+
"timestamp": 1596262595652,
210+
"user": {
211+
"displayName": "",
212+
"photoUrl": "",
213+
"userId": ""
214+
},
215+
"user_tz": 420
216+
},
217+
"id": "WoSz5plIyXOU",
218+
"outputId": "70e9aca4-b59d-44af-9996-63b923869f25"
219+
},
220+
"outputs": [
221+
{
222+
"name": "stdout",
223+
"output_type": "stream",
224+
"text": [
225+
"Serial[\n",
226+
" Embedding_8192_256\n",
227+
" Mean\n",
228+
" Dense_2\n",
229+
" LogSoftmax\n",
230+
"]\n"
231+
]
232+
}
233+
],
234+
"source": [
235+
"# Create the model.\n",
236+
"from trax import layers as tl\n",
237+
"\n",
238+
"model = tl.Serial(\n",
239+
" tl.Embedding(vocab_size=8192, d_feature=256),\n",
240+
" tl.Mean(axis=1), # Average on axis 1 (length of sentence).\n",
241+
" tl.Dense(2), # Classify 2 classes.\n",
242+
" tl.LogSoftmax() # Produce log-probabilities.\n",
243+
")\n",
244+
"\n",
245+
"# You can print model structure.\n",
246+
"print(model)"
247+
]
248+
},
249+
{
250+
"cell_type": "code",
251+
"execution_count": null,
252+
"metadata": {
253+
"colab": {
254+
"height": 442
255+
},
256+
"colab_type": "code",
257+
"executionInfo": {
258+
"elapsed": 48938,
259+
"status": "ok",
260+
"timestamp": 1596262646955,
261+
"user": {
262+
"displayName": "",
263+
"photoUrl": "",
264+
"userId": ""
265+
},
266+
"user_tz": 420
267+
},
268+
"id": "d6bIKUO-3Cw8",
269+
"outputId": "81a6774a-306e-40df-c56b-e662c80779bb"
270+
},
271+
"outputs": [
272+
{
273+
"name": "stdout",
274+
"output_type": "stream",
275+
"text": [
276+
"\n",
277+
"Step 1: Ran 1 train steps in 1.40 secs\n",
278+
"Step 1: train CrossEntropyLoss | 0.70895958\n",
279+
"Step 1: eval CrossEntropyLoss | 1.13437700\n",
280+
"Step 1: eval Accuracy | 0.43281250\n",
281+
"\n",
282+
"Step 500: Ran 499 train steps in 11.97 secs\n",
283+
"Step 500: train CrossEntropyLoss | 0.56862390\n",
284+
"Step 500: eval CrossEntropyLoss | 0.44110590\n",
285+
"Step 500: eval Accuracy | 0.78125000\n",
286+
"\n",
287+
"Step 1000: Ran 500 train steps in 10.11 secs\n",
288+
"Step 1000: train CrossEntropyLoss | 0.39702696\n",
289+
"Step 1000: eval CrossEntropyLoss | 0.36471220\n",
290+
"Step 1000: eval Accuracy | 0.85000000\n",
291+
"\n",
292+
"Step 1500: Ran 500 train steps in 9.40 secs\n",
293+
"Step 1500: train CrossEntropyLoss | 0.37494999\n",
294+
"Step 1500: eval CrossEntropyLoss | 0.33786806\n",
295+
"Step 1500: eval Accuracy | 0.88437500\n",
296+
"\n",
297+
"Step 2000: Ran 500 train steps in 9.65 secs\n",
298+
"Step 2000: train CrossEntropyLoss | 0.29281971\n",
299+
"Step 2000: eval CrossEntropyLoss | 0.38205401\n",
300+
"Step 2000: eval Accuracy | 0.87656250\n"
301+
]
302+
}
303+
],
304+
"source": [
305+
"# Train the model.\n",
306+
"from trax.supervised import training\n",
307+
"\n",
308+
"# Training task.\n",
309+
"train_task = training.TrainTask(\n",
310+
" labeled_data=train_batches_stream,\n",
311+
" loss_layer=tl.CrossEntropyLoss(),\n",
312+
" optimizer=trax.optimizers.Adam(0.01),\n",
313+
" n_steps_per_checkpoint=500,\n",
314+
")\n",
315+
"\n",
316+
"# Evaluaton task.\n",
317+
"eval_task = training.EvalTask(\n",
318+
" labeled_data=eval_batches_stream,\n",
319+
" metrics=[tl.CrossEntropyLoss(), tl.Accuracy()],\n",
320+
" n_eval_batches=20 # For less variance in eval numbers.\n",
321+
")\n",
322+
"\n",
323+
"# Training loop saves checkpoints to output_dir.\n",
324+
"output_dir = os.path.expanduser('~/output_dir/')\n",
325+
"training_loop = training.Loop(model,\n",
326+
" train_task,\n",
327+
" eval_tasks=[eval_task],\n",
328+
" output_dir=output_dir)\n",
329+
"\n",
330+
"# Run 2000 steps (batches).\n",
331+
"training_loop.run(2000)"
332+
]
333+
},
334+
{
335+
"cell_type": "code",
336+
"execution_count": null,
337+
"metadata": {
338+
"colab": {
339+
"height": 71
340+
},
341+
"colab_type": "code",
342+
"executionInfo": {
343+
"elapsed": 2088,
344+
"status": "ok",
345+
"timestamp": 1596262654971,
346+
"user": {
347+
"displayName": "",
348+
"photoUrl": "",
349+
"userId": ""
350+
},
351+
"user_tz": 420
352+
},
353+
"id": "yuPu37Lp7GST",
354+
"outputId": "cd08204f-21be-477a-914e-f4cc88b3875a"
355+
},
356+
"outputs": [
357+
{
358+
"name": "stdout",
359+
"output_type": "stream",
360+
"text": [
361+
"example input_str: Harlan Banks is thief at the top of his game, but, after a successful career, he has decided to settle down with his woman and retire. However, he decides to take one last routine job in Las Vegas. All he has to do is drive the car and it seems simple. Unfortunately someone tips the police and after a hectic car chase he winds up in the slam only to escape and take revenge on those who betrayed and got him there in the first place. A typical action-fest ensues.\u003cbr /\u003e\u003cbr /\u003eSteven Seagal plays himself (surprise!) wearing a trench-coat and sporting his beloved Colt 1911 along with his usual bone-breaking aikido. The Colt and aikido have always been with him, but the first I recall him with the trench-coat is in 'The Foreigner.' It isn't particularly impressive, but it does add a little notch to Seagal's lethal arsenal of badassness. Or it covers up those extra pounds he is packing. Look at it however way you choose. His main buddy throughout the film is played by Treach (another new thing, a rapper in an action movie) and they both uncover a little conspiracy of bad-guys, on both sides of the law, and give each evil-doer his due.\u003cbr /\u003e\u003cbr /\u003eThe film's main problem is that is it painfully, and I truly mean painfully, unoriginal. Seagal just follows a clockwork plot throughout the movie and even that manages to get more and more dull as the film progresses. Then it goes from dull to utterly ridiculous in the final scene as people who seemed to be dead on killing each other suddenly, for no reason, start to talk. Groan inducing in every sense of the word. The only real positive thing here is the decent opening - a car chase in Las Vegas complete with flipping police cars and generally entertaining mayhem, but after that brief highlight you've seen it all before. 3/10\u003cbr /\u003e\u003cbr /\u003eRated R: constant violence and profanity\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\n",
362+
"Model returned sentiment probabilities: [[0.721052 0.27894798]]\n"
363+
]
364+
}
365+
],
366+
"source": [
367+
"# Run on an example.\n",
368+
"example_input = next(eval_batches_stream)[0][0]\n",
369+
"example_input_str = trax.data.detokenize(example_input, vocab_file='en_8k.subword')\n",
370+
"print(f'example input_str: {example_input_str}')\n",
371+
"sentiment_log_probs = model(example_input[None, :]) # Add batch dimension.\n",
372+
"print(f'Model returned sentiment probabilities: {np.exp(sentiment_log_probs)}')"
373+
]
374+
},
375+
{
376+
"cell_type": "markdown",
377+
"metadata": {
378+
"colab_type": "text",
379+
"id": "8wgfJyhdihfR"
380+
},
381+
"source": [
382+
"## 2. Convert Trax to Keras\n",
383+
"\n",
384+
"Thanks to [TensorFlow NumPy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy) you can convert the model you just trained into a [Keras](https://keras.io/) layer using `trax.AsKeras`. This allows you to:\n",
385+
"\n",
386+
"* use Trax layers inside Keras models\n",
387+
"* run Trax models with existing Keras input pipelines\n",
388+
"* export Trax models to [TensorFlow SavedModel](https://www.tensorflow.org/guide/saved_model)\n",
389+
"\n",
390+
"When creating a Keras layer from a Trax one, the Keras layer weights will get initialized to the ones the Trax layer had at the moment of creation. In this way, you can create Keras layers from pre-trained Trax models and save them as SavedModel as shown below."
391+
]
392+
},
393+
{
394+
"cell_type": "code",
395+
"execution_count": null,
396+
"metadata": {
397+
"colab": {
398+
"height": 51
399+
},
400+
"colab_type": "code",
401+
"executionInfo": {
402+
"elapsed": 792,
403+
"status": "ok",
404+
"timestamp": 1596262678494,
405+
"user": {
406+
"displayName": "",
407+
"photoUrl": "",
408+
"userId": ""
409+
},
410+
"user_tz": 420
411+
},
412+
"id": "bxSLRyjftuxH",
413+
"outputId": "b881ec5c-00ba-44ef-a91f-12c033206d46"
414+
},
415+
"outputs": [
416+
{
417+
"name": "stdout",
418+
"output_type": "stream",
419+
"text": [
420+
"\u003ctrax.trax2keras.AsKeras object at 0x7f4b22bfad30\u003e\n",
421+
"Keras returned sentiment probabilities: [[0.72105217 0.27894783]]\n"
422+
]
423+
}
424+
],
425+
"source": [
426+
"# Convert the model into a Keras layer, use the weights from model.\n",
427+
"keras_layer = trax.AsKeras(model)\n",
428+
"print(keras_layer)\n",
429+
"\n",
430+
"# Run the Keras layer to verify it returns the same result.\n",
431+
"sentiment_log_probs = keras_layer(example_input[None, :])\n",
432+
"print(f'Keras returned sentiment probabilities: {np.exp(sentiment_log_probs)}')"
433+
]
434+
},
435+
{
436+
"cell_type": "code",
437+
"execution_count": null,
438+
"metadata": {
439+
"colab": {
440+
"height": 51
441+
},
442+
"colab_type": "code",
443+
"executionInfo": {
444+
"elapsed": 624,
445+
"status": "ok",
446+
"timestamp": 1596262716695,
447+
"user": {
448+
"displayName": "",
449+
"photoUrl": "",
450+
"userId": ""
451+
},
452+
"user_tz": 420
453+
},
454+
"id": "r8C-FoFGxGE1",
455+
"outputId": "79e9663d-2b2e-4e4b-faca-e9a3950b06e3"
456+
},
457+
"outputs": [
458+
{
459+
"name": "stdout",
460+
"output_type": "stream",
461+
"text": [
462+
"\u003ctensorflow.python.keras.engine.functional.Functional object at 0x7f4b7cd05b38\u003e\n",
463+
"Keras returned sentiment probabilities: [[0.72105217 0.27894783]]\n"
464+
]
465+
}
466+
],
467+
"source": [
468+
"import tensorflow as tf\n",
469+
"\n",
470+
"# Create a full Keras model using the layer from Trax.\n",
471+
"inputs = tf.keras.Input(shape=(None,), dtype='int32')\n",
472+
"hidden = keras_layer(inputs) \n",
473+
"# You can add other Keras layers here operating on hidden.\n",
474+
"outputs = hidden\n",
475+
"keras_model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
476+
"print(keras_model)\n",
477+
"\n",
478+
"# Run the Keras model to verify it returns the same result.\n",
479+
"sentiment_log_probs = keras_model(example_input[None, :])\n",
480+
"print(f'Keras returned sentiment probabilities: {np.exp(sentiment_log_probs)}')"
481+
]
482+
},
483+
{
484+
"cell_type": "markdown",
485+
"metadata": {
486+
"colab_type": "text",
487+
"id": "EQH1bvXwy5fE"
488+
},
489+
"source": [
490+
"## 3. Exporting Trax Models for Deployment\n",
491+
"\n",
492+
"You can export the Keras model to disk as [TensorFlow SavedModel](https://www.tensorflow.org/guide/saved_model). It's as simple as calling `keras_model.save` and allows you to use models with TF tools [TensorFlow.js](https://www.tensorflow.org/js/), [TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving) and [TensorFlow Lite](https://www.tensorflow.org/lite)."
493+
]
494+
},
495+
{
496+
"cell_type": "code",
497+
"execution_count": null,
498+
"metadata": {
499+
"colab": {
500+
"height": 34
501+
},
502+
"colab_type": "code",
503+
"executionInfo": {
504+
"elapsed": 2161,
505+
"status": "ok",
506+
"timestamp": 1596262724732,
507+
"user": {
508+
"displayName": "",
509+
"photoUrl": "",
510+
"userId": ""
511+
},
512+
"user_tz": 420
513+
},
514+
"id": "nQIJrOUgxRfK",
515+
"outputId": "3cbcc0f3-4d81-497d-9e63-a98696bce7cf"
516+
},
517+
"outputs": [
518+
{
519+
"name": "stdout",
520+
"output_type": "stream",
521+
"text": [
522+
"Keras returned sentiment probabilities: [[0.72105217 0.27894783]]\n"
523+
]
524+
}
525+
],
526+
"source": [
527+
"# Save the Keras model to output_dir.\n",
528+
"model_file = os.path.join(output_dir, \"model_checkpoint\")\n",
529+
"keras_model.save(model_file)\n",
530+
"\n",
531+
"# Load the model from SavedModel.\n",
532+
"loaded_model = tf.keras.models.load_model(model_file)\n",
533+
"\n",
534+
"# Run the loaded model to verify it returns the same result.\n",
535+
"sentiment_log_probs = loaded_model(example_input[None, :])\n",
536+
"print(f'Keras returned sentiment probabilities: {np.exp(sentiment_log_probs)}')"
537+
]
538+
}
539+
],
540+
"metadata": {
541+
"accelerator": "GPU",
542+
"colab": {
543+
"collapsed_sections": [],
544+
"last_runtime": {
545+
"build_target": "//learning/deepmind/public/tools/ml_python:ml_notebook",
546+
"kind": "private"
547+
},
548+
"name": "Using Trax with Keras",
549+
"provenance": [
550+
{
551+
"file_id": "1RNbQoOuzKsp_FoDqOFQX4mA--Wzt5ofq",
552+
"timestamp": 1596181556972
553+
},
554+
{
555+
"file_id": "https://github.com/google/trax/blob/master/trax/intro.ipynb",
556+
"timestamp": 1596178511100
557+
},
558+
{
559+
"file_id": "trax/intro.ipynb",
560+
"timestamp": 1595931762204
561+
},
562+
{
563+
"file_id": "1v1GvTkEFjMH_1c-bdS7JzNS70u9RUEHV",
564+
"timestamp": 1578964243645
565+
},
566+
{
567+
"file_id": "1SplqILjJr_ZqXcIUkNIk0tSbthfhYm07",
568+
"timestamp": 1572044421118
569+
},
570+
{
571+
"file_id": "intro.ipynb",
572+
"timestamp": 1571858674399
573+
},
574+
{
575+
"file_id": "1sF8QbqJ19ZU6oy5z4GUTt4lgUCjqO6kt",
576+
"timestamp": 1569980697572
577+
},
578+
{
579+
"file_id": "1EH76AWQ_pvT4i8ZXfkv-SCV4MrmllEl5",
580+
"timestamp": 1563927451951
581+
}
582+
]
583+
},
584+
"kernelspec": {
585+
"display_name": "Python 3",
586+
"name": "python3"
587+
}
588+
},
589+
"nbformat": 4,
590+
"nbformat_minor": 0
591+
}

‎trax/tf_numpy_and_keras.ipynb

+591
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,591 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"colab_type": "text",
7+
"id": "7yuytuIllsv1"
8+
},
9+
"source": [
10+
"# Using Trax with TensorFlow NumPy and Keras\n",
11+
"\n",
12+
"This notebook ([run it in colab](https://colab.research.google.com/github/google/trax/blob/master/trax/tf_numpy_and_keras.ipynb)) shows how you can run [Trax](https://trax-ml.readthedocs.io/en/latest/) directly with [TensorFlow NumPy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy). You will also see how to use Trax layers and models inside [Keras](https://keras.io/) so you can use Trax in production, e.g., with [TensorFlow.js](https://www.tensorflow.org/js/) or [TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving).\n",
13+
"\n",
14+
" 1. **Trax with TensorFlow NumPy**: use Trax with [TensorFlow NumPy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy) without any code changes\n",
15+
" 1. **Convert Trax to Keras**: how to get a [Keras](https://keras.io/) layer for your Trax model and use it\n",
16+
" 1. **Exporting Trax Models for Deployment**: how to export Trax models to [TensorFlow SavedModel](https://www.tensorflow.org/guide/saved_model)\n",
17+
" \n"
18+
]
19+
},
20+
{
21+
"cell_type": "markdown",
22+
"metadata": {
23+
"colab_type": "text",
24+
"id": "-LQ89rFFsEdk"
25+
},
26+
"source": [
27+
"## 1. Trax with TensorFlow NumPy\n",
28+
"\n",
29+
"In Trax, all computations rely on accelerated math operations happening in the `fastmath` module. This module can use different backends for acceleration. One of them is [TensorFlow NumPy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy) which uses [TensorFlow 2](https://www.tensorflow.org/) to accelerate the computations.\n",
30+
"\n",
31+
"The backend can be set using a call to `trax.fastmath.set_backend` as you'll see below. Currently available backends are `jax` (default), `tensorflow-numpy` and `numpy` (for debugging). The `tensorflow-numpy` backend uses [TensorFlow Numpy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy) for executing `fastmath` functions on TensorFlow, while the `jax` backend calls [JAX](https://github.com/google/jax) which lowers to TensorFlow XLA.\n",
32+
"\n",
33+
"You may see that `tensorflow-numpy` and `jax` backends show different speed and memory characteristics. You may also see different error messages when debugging since it might expose you to the internals of the backends. However for the most part, users can choose a backend and not worry about the internal details of these backends.\n",
34+
"\n",
35+
"Let's train the sentiment analysis model from the [Trax intro](https://colab.research.google.com/github/google/trax/blob/master/trax/intro.ipynb) using TensorFlow NumPy to see how it works."
36+
]
37+
},
38+
{
39+
"cell_type": "markdown",
40+
"metadata": {
41+
"colab_type": "text",
42+
"id": "BIl27504La0G"
43+
},
44+
"source": [
45+
"**General Setup**\n",
46+
"\n",
47+
"Execute the following few cells (once) before running any of the code samples."
48+
]
49+
},
50+
{
51+
"cell_type": "code",
52+
"execution_count": null,
53+
"metadata": {
54+
"cellView": "form",
55+
"colab": {},
56+
"colab_type": "code",
57+
"id": "oILRLCWN_16u"
58+
},
59+
"outputs": [],
60+
"source": [
61+
"#@title\n",
62+
"# Copyright 2020 Google LLC.\n",
63+
"\n",
64+
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
65+
"# you may not use this file except in compliance with the License.\n",
66+
"# You may obtain a copy of the License at\n",
67+
"\n",
68+
"# https://www.apache.org/licenses/LICENSE-2.0\n",
69+
"\n",
70+
"# Unless required by applicable law or agreed to in writing, software\n",
71+
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
72+
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
73+
"# See the License for the specific language governing permissions and\n",
74+
"# limitations under the License.\n",
75+
"\n",
76+
"\n"
77+
]
78+
},
79+
{
80+
"cell_type": "code",
81+
"execution_count": null,
82+
"metadata": {
83+
"cellView": "both",
84+
"colab": {},
85+
"colab_type": "code",
86+
"id": "vlGjGoGMTt-D"
87+
},
88+
"outputs": [],
89+
"source": [
90+
"# Install and import Trax\n",
91+
"!pip install -q -U git+https://github.com/google/trax@master\n",
92+
"\n",
93+
"import os\n",
94+
"import numpy as np\n",
95+
"import trax"
96+
]
97+
},
98+
{
99+
"cell_type": "markdown",
100+
"metadata": {
101+
"colab_type": "text",
102+
"id": "O_3JcfZaT5oP"
103+
},
104+
"source": [
105+
"Here is how you can set the fastmath backend to `tensorflow-numpy` and verify that it's been set."
106+
]
107+
},
108+
{
109+
"cell_type": "code",
110+
"execution_count": null,
111+
"metadata": {
112+
"colab": {
113+
"height": 34
114+
},
115+
"colab_type": "code",
116+
"executionInfo": {
117+
"elapsed": 492,
118+
"status": "ok",
119+
"timestamp": 1596262541312,
120+
"user": {
121+
"displayName": "",
122+
"photoUrl": "",
123+
"userId": ""
124+
},
125+
"user_tz": 420
126+
},
127+
"id": "djTiSLcaNFGa",
128+
"outputId": "a639c273-0bb2-481f-82aa-1264ac70c045"
129+
},
130+
"outputs": [
131+
{
132+
"name": "stdout",
133+
"output_type": "stream",
134+
"text": [
135+
"tensorflow-numpy\n"
136+
]
137+
}
138+
],
139+
"source": [
140+
"# Use the tensorflow-numpy backend.\n",
141+
"trax.fastmath.set_backend('tensorflow-numpy')\n",
142+
"print(trax.fastmath.backend_name())"
143+
]
144+
},
145+
{
146+
"cell_type": "code",
147+
"execution_count": null,
148+
"metadata": {
149+
"colab": {
150+
"height": 34
151+
},
152+
"colab_type": "code",
153+
"executionInfo": {
154+
"elapsed": 19481,
155+
"status": "ok",
156+
"timestamp": 1596262587458,
157+
"user": {
158+
"displayName": "",
159+
"photoUrl": "",
160+
"userId": ""
161+
},
162+
"user_tz": 420
163+
},
164+
"id": "AV5wrgjZ10yU",
165+
"outputId": "bef80331-f121-4d85-9d74-2ab76f077658"
166+
},
167+
"outputs": [
168+
{
169+
"name": "stdout",
170+
"output_type": "stream",
171+
"text": [
172+
"batch shapes = [(8, 2048), (8,), (8,)]\n"
173+
]
174+
}
175+
],
176+
"source": [
177+
"# Create data streams.\n",
178+
"train_stream = trax.data.TFDS('imdb_reviews', keys=('text', 'label'), train=True)()\n",
179+
"eval_stream = trax.data.TFDS('imdb_reviews', keys=('text', 'label'), train=False)()\n",
180+
"\n",
181+
"data_pipeline = trax.data.Serial(\n",
182+
" trax.data.Tokenize(vocab_file='en_8k.subword', keys=[0]),\n",
183+
" trax.data.Shuffle(),\n",
184+
" trax.data.FilterByLength(max_length=2048, length_keys=[0]),\n",
185+
" trax.data.BucketByLength(boundaries=[ 32, 128, 512, 2048],\n",
186+
" batch_sizes=[512, 128, 32, 8, 1],\n",
187+
" length_keys=[0]),\n",
188+
" trax.data.AddLossWeights()\n",
189+
" )\n",
190+
"train_batches_stream = data_pipeline(train_stream)\n",
191+
"eval_batches_stream = data_pipeline(eval_stream)\n",
192+
"\n",
193+
"# Print example shapes.\n",
194+
"example_batch = next(train_batches_stream)\n",
195+
"print(f'batch shapes = {[x.shape for x in example_batch]}')"
196+
]
197+
},
198+
{
199+
"cell_type": "code",
200+
"execution_count": null,
201+
"metadata": {
202+
"colab": {
203+
"height": 119
204+
},
205+
"colab_type": "code",
206+
"executionInfo": {
207+
"elapsed": 551,
208+
"status": "ok",
209+
"timestamp": 1596262595652,
210+
"user": {
211+
"displayName": "",
212+
"photoUrl": "",
213+
"userId": ""
214+
},
215+
"user_tz": 420
216+
},
217+
"id": "WoSz5plIyXOU",
218+
"outputId": "70e9aca4-b59d-44af-9996-63b923869f25"
219+
},
220+
"outputs": [
221+
{
222+
"name": "stdout",
223+
"output_type": "stream",
224+
"text": [
225+
"Serial[\n",
226+
" Embedding_8192_256\n",
227+
" Mean\n",
228+
" Dense_2\n",
229+
" LogSoftmax\n",
230+
"]\n"
231+
]
232+
}
233+
],
234+
"source": [
235+
"# Create the model.\n",
236+
"from trax import layers as tl\n",
237+
"\n",
238+
"model = tl.Serial(\n",
239+
" tl.Embedding(vocab_size=8192, d_feature=256),\n",
240+
" tl.Mean(axis=1), # Average on axis 1 (length of sentence).\n",
241+
" tl.Dense(2), # Classify 2 classes.\n",
242+
" tl.LogSoftmax() # Produce log-probabilities.\n",
243+
")\n",
244+
"\n",
245+
"# You can print model structure.\n",
246+
"print(model)"
247+
]
248+
},
249+
{
250+
"cell_type": "code",
251+
"execution_count": null,
252+
"metadata": {
253+
"colab": {
254+
"height": 442
255+
},
256+
"colab_type": "code",
257+
"executionInfo": {
258+
"elapsed": 48938,
259+
"status": "ok",
260+
"timestamp": 1596262646955,
261+
"user": {
262+
"displayName": "",
263+
"photoUrl": "",
264+
"userId": ""
265+
},
266+
"user_tz": 420
267+
},
268+
"id": "d6bIKUO-3Cw8",
269+
"outputId": "81a6774a-306e-40df-c56b-e662c80779bb"
270+
},
271+
"outputs": [
272+
{
273+
"name": "stdout",
274+
"output_type": "stream",
275+
"text": [
276+
"\n",
277+
"Step 1: Ran 1 train steps in 1.40 secs\n",
278+
"Step 1: train CrossEntropyLoss | 0.70895958\n",
279+
"Step 1: eval CrossEntropyLoss | 1.13437700\n",
280+
"Step 1: eval Accuracy | 0.43281250\n",
281+
"\n",
282+
"Step 500: Ran 499 train steps in 11.97 secs\n",
283+
"Step 500: train CrossEntropyLoss | 0.56862390\n",
284+
"Step 500: eval CrossEntropyLoss | 0.44110590\n",
285+
"Step 500: eval Accuracy | 0.78125000\n",
286+
"\n",
287+
"Step 1000: Ran 500 train steps in 10.11 secs\n",
288+
"Step 1000: train CrossEntropyLoss | 0.39702696\n",
289+
"Step 1000: eval CrossEntropyLoss | 0.36471220\n",
290+
"Step 1000: eval Accuracy | 0.85000000\n",
291+
"\n",
292+
"Step 1500: Ran 500 train steps in 9.40 secs\n",
293+
"Step 1500: train CrossEntropyLoss | 0.37494999\n",
294+
"Step 1500: eval CrossEntropyLoss | 0.33786806\n",
295+
"Step 1500: eval Accuracy | 0.88437500\n",
296+
"\n",
297+
"Step 2000: Ran 500 train steps in 9.65 secs\n",
298+
"Step 2000: train CrossEntropyLoss | 0.29281971\n",
299+
"Step 2000: eval CrossEntropyLoss | 0.38205401\n",
300+
"Step 2000: eval Accuracy | 0.87656250\n"
301+
]
302+
}
303+
],
304+
"source": [
305+
"# Train the model.\n",
306+
"from trax.supervised import training\n",
307+
"\n",
308+
"# Training task.\n",
309+
"train_task = training.TrainTask(\n",
310+
" labeled_data=train_batches_stream,\n",
311+
" loss_layer=tl.CrossEntropyLoss(),\n",
312+
" optimizer=trax.optimizers.Adam(0.01),\n",
313+
" n_steps_per_checkpoint=500,\n",
314+
")\n",
315+
"\n",
316+
"# Evaluaton task.\n",
317+
"eval_task = training.EvalTask(\n",
318+
" labeled_data=eval_batches_stream,\n",
319+
" metrics=[tl.CrossEntropyLoss(), tl.Accuracy()],\n",
320+
" n_eval_batches=20 # For less variance in eval numbers.\n",
321+
")\n",
322+
"\n",
323+
"# Training loop saves checkpoints to output_dir.\n",
324+
"output_dir = os.path.expanduser('~/output_dir/')\n",
325+
"training_loop = training.Loop(model,\n",
326+
" train_task,\n",
327+
" eval_tasks=[eval_task],\n",
328+
" output_dir=output_dir)\n",
329+
"\n",
330+
"# Run 2000 steps (batches).\n",
331+
"training_loop.run(2000)"
332+
]
333+
},
334+
{
335+
"cell_type": "code",
336+
"execution_count": null,
337+
"metadata": {
338+
"colab": {
339+
"height": 71
340+
},
341+
"colab_type": "code",
342+
"executionInfo": {
343+
"elapsed": 2088,
344+
"status": "ok",
345+
"timestamp": 1596262654971,
346+
"user": {
347+
"displayName": "",
348+
"photoUrl": "",
349+
"userId": ""
350+
},
351+
"user_tz": 420
352+
},
353+
"id": "yuPu37Lp7GST",
354+
"outputId": "cd08204f-21be-477a-914e-f4cc88b3875a"
355+
},
356+
"outputs": [
357+
{
358+
"name": "stdout",
359+
"output_type": "stream",
360+
"text": [
361+
"example input_str: Harlan Banks is thief at the top of his game, but, after a successful career, he has decided to settle down with his woman and retire. However, he decides to take one last routine job in Las Vegas. All he has to do is drive the car and it seems simple. Unfortunately someone tips the police and after a hectic car chase he winds up in the slam only to escape and take revenge on those who betrayed and got him there in the first place. A typical action-fest ensues.\u003cbr /\u003e\u003cbr /\u003eSteven Seagal plays himself (surprise!) wearing a trench-coat and sporting his beloved Colt 1911 along with his usual bone-breaking aikido. The Colt and aikido have always been with him, but the first I recall him with the trench-coat is in 'The Foreigner.' It isn't particularly impressive, but it does add a little notch to Seagal's lethal arsenal of badassness. Or it covers up those extra pounds he is packing. Look at it however way you choose. His main buddy throughout the film is played by Treach (another new thing, a rapper in an action movie) and they both uncover a little conspiracy of bad-guys, on both sides of the law, and give each evil-doer his due.\u003cbr /\u003e\u003cbr /\u003eThe film's main problem is that is it painfully, and I truly mean painfully, unoriginal. Seagal just follows a clockwork plot throughout the movie and even that manages to get more and more dull as the film progresses. Then it goes from dull to utterly ridiculous in the final scene as people who seemed to be dead on killing each other suddenly, for no reason, start to talk. Groan inducing in every sense of the word. The only real positive thing here is the decent opening - a car chase in Las Vegas complete with flipping police cars and generally entertaining mayhem, but after that brief highlight you've seen it all before. 3/10\u003cbr /\u003e\u003cbr /\u003eRated R: constant violence and profanity\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\u003cpad\u003e\n",
362+
"Model returned sentiment probabilities: [[0.721052 0.27894798]]\n"
363+
]
364+
}
365+
],
366+
"source": [
367+
"# Run on an example.\n",
368+
"example_input = next(eval_batches_stream)[0][0]\n",
369+
"example_input_str = trax.data.detokenize(example_input, vocab_file='en_8k.subword')\n",
370+
"print(f'example input_str: {example_input_str}')\n",
371+
"sentiment_log_probs = model(example_input[None, :]) # Add batch dimension.\n",
372+
"print(f'Model returned sentiment probabilities: {np.exp(sentiment_log_probs)}')"
373+
]
374+
},
375+
{
376+
"cell_type": "markdown",
377+
"metadata": {
378+
"colab_type": "text",
379+
"id": "8wgfJyhdihfR"
380+
},
381+
"source": [
382+
"## 2. Convert Trax to Keras\n",
383+
"\n",
384+
"Thanks to [TensorFlow NumPy](https://www.tensorflow.org/api_docs/python/tf/experimental/numpy) you can convert the model you just trained into a [Keras](https://keras.io/) layer using `trax.AsKeras`. This allows you to:\n",
385+
"\n",
386+
"* use Trax layers inside Keras models\n",
387+
"* run Trax models with existing Keras input pipelines\n",
388+
"* export Trax models to [TensorFlow SavedModel](https://www.tensorflow.org/guide/saved_model)\n",
389+
"\n",
390+
"When creating a Keras layer from a Trax one, the Keras layer weights will get initialized to the ones the Trax layer had at the moment of creation. In this way, you can create Keras layers from pre-trained Trax models and save them as SavedModel as shown below."
391+
]
392+
},
393+
{
394+
"cell_type": "code",
395+
"execution_count": null,
396+
"metadata": {
397+
"colab": {
398+
"height": 51
399+
},
400+
"colab_type": "code",
401+
"executionInfo": {
402+
"elapsed": 792,
403+
"status": "ok",
404+
"timestamp": 1596262678494,
405+
"user": {
406+
"displayName": "",
407+
"photoUrl": "",
408+
"userId": ""
409+
},
410+
"user_tz": 420
411+
},
412+
"id": "bxSLRyjftuxH",
413+
"outputId": "b881ec5c-00ba-44ef-a91f-12c033206d46"
414+
},
415+
"outputs": [
416+
{
417+
"name": "stdout",
418+
"output_type": "stream",
419+
"text": [
420+
"\u003ctrax.trax2keras.AsKeras object at 0x7f4b22bfad30\u003e\n",
421+
"Keras returned sentiment probabilities: [[0.72105217 0.27894783]]\n"
422+
]
423+
}
424+
],
425+
"source": [
426+
"# Convert the model into a Keras layer, use the weights from model.\n",
427+
"keras_layer = trax.AsKeras(model)\n",
428+
"print(keras_layer)\n",
429+
"\n",
430+
"# Run the Keras layer to verify it returns the same result.\n",
431+
"sentiment_log_probs = keras_layer(example_input[None, :])\n",
432+
"print(f'Keras returned sentiment probabilities: {np.exp(sentiment_log_probs)}')"
433+
]
434+
},
435+
{
436+
"cell_type": "code",
437+
"execution_count": null,
438+
"metadata": {
439+
"colab": {
440+
"height": 51
441+
},
442+
"colab_type": "code",
443+
"executionInfo": {
444+
"elapsed": 624,
445+
"status": "ok",
446+
"timestamp": 1596262716695,
447+
"user": {
448+
"displayName": "",
449+
"photoUrl": "",
450+
"userId": ""
451+
},
452+
"user_tz": 420
453+
},
454+
"id": "r8C-FoFGxGE1",
455+
"outputId": "79e9663d-2b2e-4e4b-faca-e9a3950b06e3"
456+
},
457+
"outputs": [
458+
{
459+
"name": "stdout",
460+
"output_type": "stream",
461+
"text": [
462+
"\u003ctensorflow.python.keras.engine.functional.Functional object at 0x7f4b7cd05b38\u003e\n",
463+
"Keras returned sentiment probabilities: [[0.72105217 0.27894783]]\n"
464+
]
465+
}
466+
],
467+
"source": [
468+
"import tensorflow as tf\n",
469+
"\n",
470+
"# Create a full Keras model using the layer from Trax.\n",
471+
"inputs = tf.keras.Input(shape=(None,), dtype='int32')\n",
472+
"hidden = keras_layer(inputs) \n",
473+
"# You can add other Keras layers here operating on hidden.\n",
474+
"outputs = hidden\n",
475+
"keras_model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
476+
"print(keras_model)\n",
477+
"\n",
478+
"# Run the Keras model to verify it returns the same result.\n",
479+
"sentiment_log_probs = keras_model(example_input[None, :])\n",
480+
"print(f'Keras returned sentiment probabilities: {np.exp(sentiment_log_probs)}')"
481+
]
482+
},
483+
{
484+
"cell_type": "markdown",
485+
"metadata": {
486+
"colab_type": "text",
487+
"id": "EQH1bvXwy5fE"
488+
},
489+
"source": [
490+
"## 3. Exporting Trax Models for Deployment\n",
491+
"\n",
492+
"You can export the Keras model to disk as [TensorFlow SavedModel](https://www.tensorflow.org/guide/saved_model). It's as simple as calling `keras_model.save` and allows you to use models with TF tools [TensorFlow.js](https://www.tensorflow.org/js/), [TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving) and [TensorFlow Lite](https://www.tensorflow.org/lite)."
493+
]
494+
},
495+
{
496+
"cell_type": "code",
497+
"execution_count": null,
498+
"metadata": {
499+
"colab": {
500+
"height": 34
501+
},
502+
"colab_type": "code",
503+
"executionInfo": {
504+
"elapsed": 2161,
505+
"status": "ok",
506+
"timestamp": 1596262724732,
507+
"user": {
508+
"displayName": "",
509+
"photoUrl": "",
510+
"userId": ""
511+
},
512+
"user_tz": 420
513+
},
514+
"id": "nQIJrOUgxRfK",
515+
"outputId": "3cbcc0f3-4d81-497d-9e63-a98696bce7cf"
516+
},
517+
"outputs": [
518+
{
519+
"name": "stdout",
520+
"output_type": "stream",
521+
"text": [
522+
"Keras returned sentiment probabilities: [[0.72105217 0.27894783]]\n"
523+
]
524+
}
525+
],
526+
"source": [
527+
"# Save the Keras model to output_dir.\n",
528+
"model_file = os.path.join(output_dir, \"model_checkpoint\")\n",
529+
"keras_model.save(model_file)\n",
530+
"\n",
531+
"# Load the model from SavedModel.\n",
532+
"loaded_model = tf.keras.models.load_model(model_file)\n",
533+
"\n",
534+
"# Run the loaded model to verify it returns the same result.\n",
535+
"sentiment_log_probs = loaded_model(example_input[None, :])\n",
536+
"print(f'Keras returned sentiment probabilities: {np.exp(sentiment_log_probs)}')"
537+
]
538+
}
539+
],
540+
"metadata": {
541+
"accelerator": "GPU",
542+
"colab": {
543+
"collapsed_sections": [],
544+
"last_runtime": {
545+
"build_target": "//learning/deepmind/public/tools/ml_python:ml_notebook",
546+
"kind": "private"
547+
},
548+
"name": "Using Trax with Keras",
549+
"provenance": [
550+
{
551+
"file_id": "1RNbQoOuzKsp_FoDqOFQX4mA--Wzt5ofq",
552+
"timestamp": 1596181556972
553+
},
554+
{
555+
"file_id": "https://github.com/google/trax/blob/master/trax/intro.ipynb",
556+
"timestamp": 1596178511100
557+
},
558+
{
559+
"file_id": "trax/intro.ipynb",
560+
"timestamp": 1595931762204
561+
},
562+
{
563+
"file_id": "1v1GvTkEFjMH_1c-bdS7JzNS70u9RUEHV",
564+
"timestamp": 1578964243645
565+
},
566+
{
567+
"file_id": "1SplqILjJr_ZqXcIUkNIk0tSbthfhYm07",
568+
"timestamp": 1572044421118
569+
},
570+
{
571+
"file_id": "intro.ipynb",
572+
"timestamp": 1571858674399
573+
},
574+
{
575+
"file_id": "1sF8QbqJ19ZU6oy5z4GUTt4lgUCjqO6kt",
576+
"timestamp": 1569980697572
577+
},
578+
{
579+
"file_id": "1EH76AWQ_pvT4i8ZXfkv-SCV4MrmllEl5",
580+
"timestamp": 1563927451951
581+
}
582+
]
583+
},
584+
"kernelspec": {
585+
"display_name": "Python 3",
586+
"name": "python3"
587+
}
588+
},
589+
"nbformat": 4,
590+
"nbformat_minor": 0
591+
}

0 commit comments

Comments
 (0)
Please sign in to comment.