Transfer Learning with TensorFlow Hub Module

Growth on the machine learning side is lagging behind the software engineering side by 15-20 years.This creates a really interesting opportunity.We can look at some of the things that happened at software engineering and think about what kind of impact they may have on machine learning. Looking at software engineering there’s something that’s so fundamental and this idea of sharing code.

Shared Machine Learning


sharing code
Shared code repositories make us immediately more productive.We can search for code, download it, use it, it has these really powerful effects.This actually changes the way we write code.We refactor our code, put it in libraries, share those libraries and this really makes people even more productive.

sharing machine learning code
It’s the same dynamic that TensorFlow creates for machine learning with TensorFlow Hub.With TensorFlow Hub, you can build, share and use pieces of machine learning.

Why shared Machine Learning


Elements of Machine Learning
Anyone who’s done machine learning from scratch knows you need a lot to do, you need an algorithm, data, computer power and expertise and if you’re missing any of these you’re out of luck.

TensorFlow Hub lets you distill all these things down into a reusable package.We call a module.Those modules go into TensorFlow Hub where people can discover them and then easily use them.

TensorFlow Hub
Notice that it’s module instead of a model.A model is big for sharing.If you have a model you can use that model if you have the exact inputs it wants and you expect the exact outputs it provides.If there are any little differences you’re kind of out of luck. Modules are a smaller piece.If you think of a model like a binary and think of a module like a library.

Modules Contain


Module ContainModules contain pre-trained weights and graphs.The module is actually a saved model.It lets us package up the algorithm in the form of a graph.This is package up the weights. You can do things like initializing use assets.Modules are composable, reusable,re-trainable.

TensorFlow libraries make it(Models) very easy to instantiate in your TensorFlow code.So you can compose these in interesting ways.This makes things very reusable.You can produce one of these and share it. These are also retrainable.Once you patch it into your program you can backpropagate through it just like normal.This is really powerful because if you have enough data you can customize the TensorFlow hub module for your own application.

Image Retraining


Let’s look at a specific example of using a TensorFlow Hub for image retraining.Let’s say that we’re gonna make an app that can classify cat from the image. The problem is we have only a couple hundred examples, probably not enough to train an entire image classification model from scratch, but what we could do is start from an existing general-purpose image classification model.

Image Retraining

Most of the high-performance model is trained on millions of examples.They can easily classify thousands of categories.We want to reuse the architecture and the trained weights of that model without the classification layers and in that way we can add our own cat classifier on top.We can train it on our cat examples and keep the reused weights fixed.

Image Modules

You can find a list of all of the newly released the state-of-the-art, well-known image modules. some of them include the classification layers and some of them remove them just providing a feature vector as output.We’ll choose one of the feature vectors, for this NASNet which is a state-of-the-art image module that was created by a neural architecture.

So you just paste the URL of a module and TensorFlow Hub takes care of downloading the graph and all of its weights and importing it into your model in one line.You’re ready to use the module like any function.

Here, We just provide a batch of inputs and we get back our feature vectors.We add a classification layer on top and we output our predictions.But in that one line, you get a huge amount of value in this particular case more than 62,000 hours of GPU time.

Available Image Modules

NASNet is available in a large size as well as a mobile size module and there’s also the new progressive PNASNet .A number of new MobileNet modules for doing on-device image classification as well as some industry standard ones like Inception, and ResNet that complete list is available here.All those modules are pre-trained using the TF-slim checkpoints and already used for classification or as feature vector inputs to your own model.

Text Classification


We’d like to know whether a movie review is a positive or negative sentiment.One of the great things about TensorFlow Hub is that it packages graph with the date.

Restaurant ReviewText and embedding modules contain all of the pre-processing and included a thing like normalizing and tokenizing operations.We can use a pre-trained sentence embedding module to map a full sentence to an embedding vector.So If we want to classify some movie reviews then we just take one of those sentence embedding modules.we add our own classification layer on top and then we train with our review.We keep the sentence modules weights fixed.

TensorFlow Hub has a number of different text modules.It has neural network language models that are trained on Google news for English, Japanese, German, and Spanish. It has  word2Vec model trained on Wikipedia as well a new module called ELMO that models the characteristics of word use.

Universal Sentence Encoder

Universal sentence encoder it’s a sentence level embedding module.It is trained on a variety of tasks and it enables a variety of tasks in other words universal. So some of the things that are good for our semantic similarity doing custom text classification clustering and semantic search but the best part about is It requires little training data.

We just past that URL and TensorFlow Hub downloads the module and inserts it into your graph.This time we’re using the text embedding column that way we can feed it into an estimator to do the classification part.Like with the image retraining example this module can be fine-tuned with your model by setting trainable to true.You have to lower the learning rate.So that you don’t ruin any existing weights that are in there but it’s something that’s worth exploring if you have enough data.

Module URL


Module is a program So make sure what you’re executing is from a location that you trust. In this case, the module is from tfhub.dev.It provided modules like NASNet and a universal sentence encoder. TensorFlow create a place where you can publish the modules that you create.In this case, Google is the publisher and universal sentence encoder is the name of the module and the version number is 1.So Tensorflow Hub considers modules to be immutable that way you don’t have to worry about the weight changing between training sessions.So All of the module URLs on tfhub.dev include a version number and you can take that URL and past it into the browser and see that complete documentation for any module that’s hosted on tfhub.dve.

Conclusion

TensorFlow also has modules for other domains besides text classification and image retraining like generative image module. TensorFlow Hub added another module that was based on deep local features network which can identify the key points of landmark images both of those have great colab notebooks.

 

References

TensorFlow Hub (TensorFlow Dev Summit 2018)

 

Eager Execution:Pythonic way of using TensorFlow

When you enable Eager execution in TensorFlow operations are executed immediately as they are called from Python.

Eager Execution is an imperative object-oriented pythonic way of using TensorFlow. TensorFlow is graph execution engine for machine learning.

Why Graph Execution?


A really good reason is your computation represented as a platform independent.The graph is that once you have that it’s very easy to Automatic differentiation that graph.

If you have a platform independent abstract representation of your computation you can just go and deploy it to pretty much anything you want. You can run it on the TPU you can run on a GPU you can put it on a phone, Raspberry PI like all sorts of cool deployment scenarios.It’s really valuable to have this kind of platform independent view

The compilers work with data for graphs internally and they know how to do all sorts of nice optimizations that rely on having a global view of computation like constant folding common subexpression elimination and data laying thing like that.

A lot of these optimizations are really like deep learning specific.We can choose how to properly layout your channels and your height and width, so your convolutions are faster.

A key reason that’s very important is once you have a platform independent representation of your computation, You can just deploy it and distribute it across hundreds of machines or an TPU.

Why Eager Execution


These graphs are so good what made us to think that now it’s a good idea to move beyond them and let you do Eager Execution.

You can just build up a trace as you go and then walk back the trace to compute gradients.

You can iterate a lot more quickly you can play with your model as you build it.

You can inspect it you can poke and prod in it and this can let you just be more productive when you’re like making all these changes.

You can run your model for debuggers and profilers and add all sorts of like analysis tools to them to just really understand how they’re doing what they’re doing.

If you don’t force you to represent you computation in a separate way then the host programming language you’re using you can just use ultimate like machinery of your host programming language to do control flow and data flow complicated data structures which for some models is key to being able to make your model working at all.

Enable Eager Execution


You import tensorflow and you call tf.enable_eager_execution() and once you do that what happens is anytime you run a TensorFlow operation like in this case it runs immediately instead of building a graph. That later runs when executed is going to run that matrix multiplication. TensorFlow immediately runs that matrix multiplication for you and give you the result and you can print it you can slice it dice it you can do whatever you want with it.

Control Flow

Because thing happening immediately you can have highly dynamic control flow that depends on the actual values of the computation you’re executing and here is just simple like if conditions example.It doesn’t matter it just matters it has like while loops that depend on like complicated values are computed based on the computation and this runs just fine on whatever device you have.

TensorFlow also brings you a few symbols that make it easier for you to write code that’s going to work with both when building graphs you know executing eagerly.

Gradients

Different operations can occur during each call, TensorFlow record all forward operations to a tape, which is then played backward when computing gradients. After it computed the gradients, it discards the tape

The gradients_function call takes a Python function square() as an argument and returns a Python callable that computes the partial derivatives of square() with respect to its inputs. So, to get the derivative of square() at 10.0, invoke grad(10.), which is 20.

Loops

Also writing loops in Eager is very easy and straightforward you can just use a Python for loop to iterate over your datasets and datasets work in eager just fine and they work the same high performance you get in the graph execution engine then you can just do your predictions compute your gradient supply your gradients all the things you’re to doing.

Debugging

When Eager Execution is enabled you can just take any model code add notes to like drop into the Python debugger anywhere you want. Once you’re in the Python debugger you have the full power of debugging available you can print the value of anything.You can change the value of any tensor you can run any operation you want on any tensor and this will hopefully empower you to really understand what’s going on in your models and really be able to fix any problems you have.You can also take Eager Execution code and profile it using whatever profiling tool.

Variable is Object


A big change when programming with Eager from the graph that variables intensive though is usually a complicated thing to think about but when eager execution is enabled it’s much simpler.The TensorFlow variable is just the python object.You create one you have it, you can write, you can change its value, you can read this value when the last reference to it goes away, you get your memory back even if it’s a GPU memory. So if you want to share variables you just reuse those object you don’t worry about variable scopes or any other complicated structure and because TensorFlow has this like object-oriented approach to variables it can look at some of the API is intensively flowing like rethink them in a way that’s a little more.

Object-oriented Saving


TensorFlow also giving you a way to do object-oriented saving of TensorFlow models. If you’ve tried looking at TensorFlow checkpoints you know that they depend on variable names and variable names depend not just on a name you, but on all other which are present in your graph.This can make it a little hard for you to save and load subsets of your model and really control what’s in your checkpoint. TensorFlow introducing a completely object-oriented python object based saving API.Any variable that’s reachable from your model gets saved on your model.You can save any subset of your model, you can load any subset of your model, you can even use this.

Conclusion

With Eager TensorFlow bringing you a lot of new that make it easier for you to built TensorFlow graph and to execute models.These are compatible with both eager execution and graph building.

 

 

Machine Intelligence Library For Browser

Python has been one of the mainstream programming languages for Machine Learning and it’s been like that for a while and there’s a lot of tools and libraries around Python.JavaScript and Browser have a lot to offer.TensorFlow playground is a great example of that.It is an in-browser visualization of a small neural network and it shows in real time all the internals of the network as it training.This was a lot of fun to make.It has a huge educational success.
TensorFlow Playground

Tensorflow Playground

TensorFlow playground is built by Google.It is powered by a small neural network 300 lines of vanilla javascript code that Google wrote as a one-off library.It doesn’t scale.It is just simple for loops and it wasn’t engineered to be reusable.

Why Machine Learning in Browser?


The browser is the unique platform where the things you build, you can share with anyone with just a link.Those people that open your app don’t have to install any drivers or any software.

The browser is highly interactive and so the user is going to be engaged with whatever you’re building.

Another big thing is that browsers have access to sensors data.All of these sensors are standardize API’s that work on all browsers.
Mobile Sensors DataThe data that comes from these sensors don’t ever have to leave the client you don’t have to upload anything to the server which preserves privacy.

deeplearn.js


Google released deeplearn.js as a JavaScript library that is GPU accelerated and it does that via WebGL which is a standard in the browser.That allows you to render 3D graphics.It allows you to both run inference in the browser and training entirely in the browser.The community took the deeplearn.js and took existing models from python and build interactive things with it.

One example is the style transfer.

Style Transfer

Another is ported the character RNN and then build a novel interface that allows you to explore all the different possible endings of a sentence all generated by the model in real time.

character RNN

Another example is a phone generative model allowed users to explore the interesting dimensions in the embedding space and you can see how they relate to boldness and slanted the font.

phone generative model

There was an educational examples teachable machine that builds this fun little game. That taught people how computer vision models work.So people could interact directly with the webcam.

interact directly with the webcam

All above examples use deeplearn.js.

deeplearn.js has become TensorFlow.js

TensorFlow.js


Google releasing a new ecosystem of libraries and tools for machine learning with JavaScript call tensorflow.js.

Use cases

One use case is you can write models directly in the browser and this has huge educational implications think of the playground.

The second use case is a major, you can take a pre-existing model(pre-trained model in python) and you can import it into the browser to do inference.

The third related use case is the same model that you take to do inference you can retrain it potentially with private data that comes from those sensors of the browser in the browser itself.

Architecture


TensorFow.js architecture

The browser utilizes WebGL to do fast linear algebra, on top of it. tensorflow.js has two sets of API’s the Ops API(eager) which used to be deeplearn.js.It is powered by an automatic differentiation library that is built analogously to eager mode.On top of that, it has high-level API that allows you to use best practices and high-level building blocks to write models.

Google release tools that can take an existing TensorFlow save model or keras model and port it automatically for execution in the browser.