Google Cloud Speech API in Android APP

Brijesh Thumar / July 27, 2017April 28, 2018 / Machine Learning APIs

Many of you have used the “Ok Google” functionality on your phone.What the Speech API does is it lets developers integrate that functionality into their own applications, lets you do speech to text transcription in over 80 languages.

It also works in streaming mode.If you want to send it a continuous stream of audio, you can send it to a stream of audio and it returns transcriptions as that audio is coming in.In this tutorial, we are going to learn Streaming Recognition to perform speech recognition.

So the best way to see how the speech API works is through a demo.

Prerequisites

Google Cloud Platform account(You can use 12 months free trial)

1.Acquiring an API Key

To Use Google Cloud Speech API services in the app, you need a Service account keys. You can get one by creating a new project in the Google Cloud Platform console.

Once the project has been created, go to API Manager > Dashboard and press the Enable API button.

Enable Google Cloud Speech API

2.Creating a New Android Project

Google provides client libraries in a to simplify the process of building and sending requests and receiving and parsing responses.

Add the following compile dependencies to the app build.gradle:

apply plugin: 'com.google.protobuf'

ext {
    grpcVersion = '1.4.0'
}
android { 
   ...

    configurations.all {
        resolutionStrategy.force 'com.google.code.findbugs:jsr305:3.0.2'
    }
}
protobuf {
    protoc {
        artifact = 'com.google.protobuf:protoc:3.3.0'
    }
    plugins {
        javalite {
            artifact = "com.google.protobuf:protoc-gen-javalite:3.0.0"
        }
        grpc {
            artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
        }
    }
    generateProtoTasks {
        all().each { task ->
            task.plugins {
                javalite {}
                grpc {
                    // Options added to --grpc_out
                    option 'lite'
                }
            }
        }
    }
}


dependencies {
   ....
   // gRPC
    compile "io.grpc:grpc-okhttp:$grpcVersion"
    compile "io.grpc:grpc-protobuf-lite:$grpcVersion"
    compile "io.grpc:grpc-stub:$grpcVersion"
    compile 'javax.annotation:javax.annotation-api:1.2'
    protobuf 'com.google.protobuf:protobuf-java:3.3.1'

    compile group: 'com.google.api.grpc', name: 'grpc-google-cloud-speech-v1', version: '0.1.13'

    // OAuth2 for Google API
    compile('com.google.auth:google-auth-library-oauth2-http:0.7.0') {
        exclude module: 'httpclient'
    }

    compile 'com.android.support:multidex:1.0.0'
}

apply plugin: 'com.google.protobuf'

ext {

grpcVersion = '1.4.0'

}

android {

...

configurations.all {

resolutionStrategy.force 'com.google.code.findbugs:jsr305:3.0.2'

}

protobuf {

protoc {

artifact = 'com.google.protobuf:protoc:3.3.0'

}

plugins {

javalite {

artifact = "com.google.protobuf:protoc-gen-javalite:3.0.0"

}

grpc {

artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"

}

generateProtoTasks {

all().each { task ->

task.plugins {

javalite {}

grpc {

// Options added to --grpc_out

option 'lite'

}

dependencies {

....

// gRPC

compile "io.grpc:grpc-okhttp:$grpcVersion"

compile "io.grpc:grpc-protobuf-lite:$grpcVersion"

compile "io.grpc:grpc-stub:$grpcVersion"

compile 'javax.annotation:javax.annotation-api:1.2'

protobuf 'com.google.protobuf:protobuf-java:3.3.1'

compile group: 'com.google.api.grpc', name: 'grpc-google-cloud-speech-v1', version: '0.1.13'

// OAuth2 for Google API

compile('com.google.auth:google-auth-library-oauth2-http:0.7.0') {

exclude module: 'httpclient'

}

compile 'com.android.support:multidex:1.0.0'

}

Set Up to Authenticate With Your Project’s Credentials

In order to get Service Account Key, visit the Cloud Console, and navigate to: API Manager > Credentials > Create credentials > Service account key > New service account. Create a new service account, and download the JSON credentials file. Put the file in the app resources as app/src/main/res/raw/credential.json.

Generate Google Cloud Service Account

Streaming Speech API Recognition Requests

Streaming Speech API recognition call is designed for real-time capture and recognition of audio, within a bi-directional stream. Your application can send audio on the request stream and receive interim and final recognition results on the response stream in real time.

Interim results represent the current recognition result for a section of audio, while the final recognition result represents the last, best guess for that section of audio.

Streaming requests

You send both the configuration and audio within a single request, calling the streaming Speech API requires sending multiple requests. The first StreamingRecognizeRequest must contain a configuration of type StreamingRecognitionConfig without any accompanying audio. Subsequent StreamingRecognizeRequests sent over the same stream will then consist of consecutive frames of raw audio bytes.

A StreamingRecognitionConfig consists of the following fields:

config – (required) contains configuration information for the audio, of type RecognitionConfig.
single_utterance – (optional, defaults to false) indicates whether this request should automatically end after speech is no longer detected.
interim_results – (optional, defaults to false) indicates that this stream request should return temporary results that may be refined at a later time.

public void startRecognizing(int sampleRate) {
    if (mApi == null) {
          Log.w(TAG, "API not ready. Ignoring the request.");
          return;
    }

    // Configure the API
    mRequestObserver = mApi.streamingRecognize(mResponseObserver);

    StreamingRecognitionConfig streamingConfig = StreamingRecognitionConfig.newBuilder()
           .setConfig(RecognitionConfig.newBuilder()
           .setLanguageCode("en-US")
                        .setEncoding(RecognitionConfig.AudioEncoding.LINEAR16)
                        .setSampleRateHertz(sampleRate)
                        .build()
            )
            .setInterimResults(true)
            .setSingleUtterance(true)
            .build();

     StreamingRecognizeRequest streamingRecognizeRequest = StreamingRecognizeRequest.newBuilder().setStreamingConfig(streamingConfig).build();
     mRequestObserver.onNext(streamingRecognizeRequest);
}

   
public void recognize(byte[] data, int size) {
    if (mRequestObserver == null) {
         return;
    }
    // Call the streaming recognition API
    mRequestObserver.onNext(StreamingRecognizeRequest.newBuilder()
            .setAudioContent(ByteString.copyFrom(data, 0, size))
            .build());
}

public void startRecognizing(int sampleRate) {

if (mApi == null) {

Log.w(TAG, "API not ready. Ignoring the request.");

return;

}

// Configure the API

mRequestObserver = mApi.streamingRecognize(mResponseObserver);

StreamingRecognitionConfig streamingConfig = StreamingRecognitionConfig.newBuilder()

.setConfig(RecognitionConfig.newBuilder()

.setLanguageCode("en-US")

.setEncoding(RecognitionConfig.AudioEncoding.LINEAR16)

.setSampleRateHertz(sampleRate)

.build()

)

.setInterimResults(true)

.setSingleUtterance(true)

.build();

StreamingRecognizeRequest streamingRecognizeRequest = StreamingRecognizeRequest.newBuilder().setStreamingConfig(streamingConfig).build();

mRequestObserver.onNext(streamingRecognizeRequest);

}

public void recognize(byte[] data, int size) {

if (mRequestObserver == null) {

return;

}

// Call the streaming recognition API

mRequestObserver.onNext(StreamingRecognizeRequest.newBuilder()

.setAudioContent(ByteString.copyFrom(data, 0, size))

.build());

}

Streaming responses

Streaming speech recognition results are returned within a series of responses of type StreamingRecognitionResponse. Response consists of the following fields:

speechEventType contains events of type SpeechEventType.
results contains the list of results, which may be either interim or final result, of type
StreamingRecognitionResult. The results list contains following the sub-fields:
- alternatives contains a list of alternative transcriptions.
- isFinalinterim or are final.
- stability indicates the volatility of results obtained so far, with 0.0 indicating complete instability while 1.0 indicates complete stability.

private final StreamObserver<StreamingRecognizeResponse> mResponseObserver = new StreamObserver<StreamingRecognizeResponse>() {
    @Override
    public void onNext(StreamingRecognizeResponse response) {
        String text = null;
        boolean isFinal = false;
        if (response.getResultsCount() > 0) {
             final StreamingRecognitionResult result = response.getResults(0);
             isFinal = result.getIsFinal();
             if (result.getAlternativesCount() > 0) {
                  final SpeechRecognitionAlternative alternative = result.getAlternatives(0);
                  text = alternative.getTranscript();
             }
         }
         if (text != null) {
              for (Listener listener : mListeners) {
                   listener.onSpeechRecognized(text, isFinal);
               }
         }
     }
     @Override
     public void onError(Throwable t) {
         Log.e(TAG, "Error calling the API.", t);
     }

     @Override
     public void onCompleted() {
         Log.i(TAG, "API completed.");
     }

private final StreamObserver<StreamingRecognizeResponse> mResponseObserver = new StreamObserver<StreamingRecognizeResponse>() {

@Override

public void onNext(StreamingRecognizeResponse response) {

String text = null;

boolean isFinal = false;

if (response.getResultsCount() > 0) {

final StreamingRecognitionResult result = response.getResults(0);

isFinal = result.getIsFinal();

if (result.getAlternativesCount() > 0) {

final SpeechRecognitionAlternative alternative = result.getAlternatives(0);

text = alternative.getTranscript();

}

if (text != null) {

for (Listener listener : mListeners) {

listener.onSpeechRecognized(text, isFinal);

}

@Override

public void onError(Throwable t) {

Log.e(TAG, "Error calling the API.", t);

}

@Override

public void onCompleted() {

Log.i(TAG, "API completed.");

}

Download this project from GitHub

Google Cloud Natural Language API in Android APP

Google Cloud Vision API in Android APP

Speech Recognition Using TensorFlow

Google Cloud Natural Language API in Android APP

Brijesh Thumar / July 26, 2017May 20, 2018 / Machine Learning APIs

The Natural Language API lets you extract entities, sentiment, and syntax from your text.Real world example is customer feedback platform.

In this tutorial, I’ll introduce you to the Cloud Natural Language platform and show you how to use it to analyze text.

Prerequisites

Google Cloud Platform account(You can use 12 months free trial)

1.Acquiring an API Key

To Use Google Cloud Natural Language API services in the app, you need an API key. You can get one by creating a new project in the Google Cloud Platform console.

Once the project has been created, go to API Manager > Dashboard and press the Enable API button.

Enable Natural Language API

To get API key, go to the Credentials tab, press the Create Credentials button, and select API key.

Google Cloud Vision API Key

2.Creating a New Android Project

Google provides client libraries to simplify the process of building and sending requests and receiving and parsing responses.

Add the following compile dependencies to the app build.gradle:

android {
    .... 
    configurations.all {
        resolutionStrategy.force 'com.google.code.findbugs:jsr305:1.3.9'
    }
}
dependencies {
    ....
   compile group: 'com.google.apis', name: 'google-api-services-language', version: 'v1-rev388-1.22.0' exclude module: 'httpclient'
   compile group: 'com.google.api-client', name: 'google-api-client-android', version: '1.22.0' exclude module: 'httpclient'
  
}

android {

....

configurations.all {

resolutionStrategy.force 'com.google.code.findbugs:jsr305:1.3.9'

}

dependencies {

....

compile group: 'com.google.apis', name: 'google-api-services-language', version: 'v1-rev388-1.22.0' exclude module: 'httpclient'

compile group: 'com.google.api-client', name: 'google-api-client-android', version: '1.22.0' exclude module: 'httpclient'

}

Add INTERNET permission in the AndroidManifest.xml file.

<uses-permission android:name="android.permission.INTERNET"/>

1	<uses-permission android:name="android.permission.INTERNET"/>

To interact with the API using the Google API Client library, you must create a CloudNaturalLanguage object using the CloudNaturalLanguage.Builder class. Its constructor also expects an HTTP transport and a JSON factory.

Furthermore, by assigning a CloudNaturalLanguageRequestInitializer instance to it, you can force it to include your API key in all its
requests.

naturalLanguageService = new CloudNaturalLanguage.Builder(
                AndroidHttp.newCompatibleTransport(),
                new AndroidJsonFactory(),
                null
        ).setCloudNaturalLanguageRequestInitializer(
                new CloudNaturalLanguageRequestInitializer(API_KEY)
        ).build();

naturalLanguageService = new CloudNaturalLanguage.Builder(

AndroidHttp.newCompatibleTransport(),

new AndroidJsonFactory(),

null

).setCloudNaturalLanguageRequestInitializer(

new CloudNaturalLanguageRequestInitializer(API_KEY)

).build();

All the text you want to analyze using the API must be placed inside a Document object. The Document object must also contain configuration information, such as the language of the text and whether it is formatted as plain text or HTML. Add the following code:

Document document = new Document();
document.setType("PLAIN_TEXT");
document.setLanguage("en-US");
document.setContent(transcript);

Document document = new Document();

document.setType("PLAIN_TEXT");

document.setLanguage("en-US");

document.setContent(transcript);

Next, you must create a Features object specifying the features you are interested in analyzing. The following code shows you how to create a Features object that says you want to extract entities and run sentiment analysis only.

Features features = new Features();
features.setExtractEntities(true);
features.setExtractDocumentSentiment(true);

Features features = new Features();

features.setExtractEntities(true);

features.setExtractDocumentSentiment(true);

Use the Document and Features objects to compose an AnnotateTextRequest object, which can be passed to the annotateText() method to generate an AnnotateTextResponse object.

new AsyncTask<Object, Void, AnnotateTextResponse>() {
         @Override
         protected AnnotateTextResponse doInBackground(Object... params) {
                AnnotateTextResponse response = null;
                try {
                     response = naturalLanguageService.documents().annotateText(request).execute();

                } catch (IOException e) {
                     e.printStackTrace();
                }
                return response;
         }

         @Override
         protected void onPostExecute(AnnotateTextResponse response) {
              super.onPostExecute(response);
              if (response != null) {
                    Sentiment sent = response.getDocumentSentiment();
                    entityList.addAll(response.getEntities());
                    entityListAdapter.notifyDataSetChanged();
                    sentiment.setText("Score : " + sent.getScore() + " Magnitude : " + sent.getMagnitude());
              }
         }
}.execute();

new AsyncTask<Object, Void, AnnotateTextResponse>() {

@Override

protected AnnotateTextResponse doInBackground(Object... params) {

AnnotateTextResponse response = null;

try {

response = naturalLanguageService.documents().annotateText(request).execute();

} catch (IOException e) {

e.printStackTrace();

}

return response;

}

@Override

protected void onPostExecute(AnnotateTextResponse response) {

super.onPostExecute(response);

if (response != null) {

Sentiment sent = response.getDocumentSentiment();

entityList.addAll(response.getEntities());

entityListAdapter.notifyDataSetChanged();

sentiment.setText("Score : " + sent.getScore() + " Magnitude : " + sent.getMagnitude());

}

}.execute();

Entity Analysis

We get the name of the entity, for example, Google.The type of the entity is organisation. Then we get back some metadata.MID ID that maps to Google’s Knowledge Graph.

You can extract a list of entities from the AnnotateTextResponse object by calling its getEntities() method.

Analyze Entity

Sentiment Analysis

Analyze the sentiment of your text.If we have this restaurant review,

The food at that restaurant has stale,I will not be going back.

If I might want to flag the most positive and most negative once and then respond just to those. So we get two number back from the Natural Language API to help us do this.

The first thing we get back is score, which will tell us on a scale from -1 to 1 how positive or negative is this text? In this example, we get negative 0.8, which is almost fully negative.

Then we get magnitude, which tells us regardless of being positive or negative, how strong is the sentiment in this text?And this is a range from 0 to infinity, and it’s normalize based on the length of the text.So we get a pretty small number here,0.8 because this is just a small piece of text.

You can extract the overall sentiment of the transcript by calling the getDocumentSentiment() method. To get the actual score of the sentiment, however, you must also call the getScore() method, which returns a float.

analyze sentiment

Download this project from GitHub

Google Cloud Vision API in Android APP

Google Cloud Speech API in Android APP

Google Cloud Vision API in Android APP

Brijesh Thumar / July 24, 2017April 28, 2018 / Machine Learning APIs

Google recently launched a Cloud Machine Learning platform, which offers Neural Networks that have been pre-trained model to perform a variety of tasks.You can use them by simply making a few REST API calls or Client library.

Google Cloud Vision API use to understand the content of an image by machine learning models using REST API. It quickly classifies images into thousands of categories, detects objects and faces within images, and finds and reads printed words contained within images.

In this tutorial, I’ll introduce you to the Cloud Machine Learning platform and show you how to use it to create a smart Android app that can recognize the real-world object.

Prerequisites

A device running Android 4.4+
Google Cloud Platform account(You can use 12 months free trial)

1.Acquiring an API Key

To Use Google Vision API services in the app, you need an API key. You can get one by creating a new project in the Google Cloud Platform console.

Once the project has been created, go to API Manager > Dashboard and press the Enable API button.

Enable Google Vision API

To get API key, go to the Credentials tab, press the Create Credentials button, and select API key.

Google Cloud Vision API

2.Creating a New Android Project

Google provides client libraries to simplify the process of building and sending requests and receiving and parsing responses.

Add the following compile dependencies to app build.gradle:

android {
    .... 
    configurations.all {
        resolutionStrategy.force 'com.google.code.findbugs:jsr305:1.3.9'
    }
}
dependencies {
    ....
    compile 'com.google.api-client:google-api-client-android:1.20.0' exclude module: 'httpclient'
    compile 'com.google.http-client:google-http-client-gson:1.20.0' exclude module: 'httpclient'
    compile 'com.google.apis:google-api-services-vision:v1-rev2-1.21.0' exclude module: 'httpclient'

}

android {

....

configurations.all {

resolutionStrategy.force 'com.google.code.findbugs:jsr305:1.3.9'

}

dependencies {

....

compile 'com.google.api-client:google-api-client-android:1.20.0' exclude module: 'httpclient'

compile 'com.google.http-client:google-http-client-gson:1.20.0' exclude module: 'httpclient'

compile 'com.google.apis:google-api-services-vision:v1-rev2-1.21.0' exclude module: 'httpclient'

}

Add INTERNET permission in the AndroidManifest.xml file.

<uses-permission android:name="android.permission.INTERNET"/>

1	<uses-permission android:name="android.permission.INTERNET"/>

Step 1: Create an Intent

Creating a new intent with the ACTION_IMAGE_CAPTURE action and passing it to the startActivityForResult() method, you can ask the default camera app of the user’s device to take pictures and pass them on to your app. Add the following code to your Activity class:

public void takePictureFromCamera() {
     Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
     startActivityForResult(intent, CAMERA_REQUEST_CODE);
}

public void takePictureFromCamera() {

Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);

startActivityForResult(intent, CAMERA_REQUEST_CODE);

}

Receive the images captured by the default camera app in onActivityResult() method of activity class. you’ll have access to a Bundle object containing all the image data. You can render the image data by simply converting it into a Bitmap and passing it to the ImageView widget.

@Override
protected void onActivityResult(int requestCode, int resultCode,
                                    Intent data) {
    if (requestCode == CAMERA_REQUEST_CODE && resultCode == RESULT_OK) {
          bitmap = (Bitmap) data.getExtras().get("data");
          imageView.setImageBitmap(bitmap);
          callCloudVision(bitmap, feature);
      }
}

@Override

protected void onActivityResult(int requestCode, int resultCode,

Intent data) {

if (requestCode == CAMERA_REQUEST_CODE && resultCode == RESULT_OK) {

bitmap = (Bitmap) data.getExtras().get("data");

imageView.setImageBitmap(bitmap);

callCloudVision(bitmap, feature);

}

Step 3: Encode the Image

The Vision API cannot use Bitmap objects directly. It expects a Base64-encoded string of compressed image data.To compress the image data, you can use the compress() method of the Bitmap class. As its arguments, the method expects the compression format to use, the output quality desired, and a ByteArrayOutputStream object. The following code compresses the bitmap using the JPEG format.

@NonNull
private Image getImageEncodeImage(Bitmap bitmap) {
     Image base64EncodedImage = new Image();
     // Convert the bitmap to a JPEG
     // Just in case it's a format that Android understands but Cloud Vision
     ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
     bitmap.compress(Bitmap.CompressFormat.JPEG, 90, byteArrayOutputStream);
     byte[] imageBytes = byteArrayOutputStream.toByteArray();

     // Base64 encode the JPEG
     base64EncodedImage.encodeContent(imageBytes);
     return base64EncodedImage;
 }

@NonNull

private Image getImageEncodeImage(Bitmap bitmap) {

Image base64EncodedImage = new Image();

// Convert the bitmap to a JPEG

// Just in case it's a format that Android understands but Cloud Vision

ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();

bitmap.compress(Bitmap.CompressFormat.JPEG, 90, byteArrayOutputStream);

byte[] imageBytes = byteArrayOutputStream.toByteArray();

// Base64 encode the JPEG

base64EncodedImage.encodeContent(imageBytes);

return base64EncodedImage;

}

Step 4: Create Feature

The Feature indicates what type of image detection task to perform. Describe the type of Vision tasks to perform over images by using Features. Features encode the Vision vertical to operate on and the number of top-scoring results to return.

Feature feature = new Feature();
feature.setType("LANDMARK_DETECTION");
feature.setMaxResults(10);

Feature feature = new Feature();

feature.setType("LANDMARK_DETECTION");

feature.setMaxResults(10);

This is the Java data model class that specifies how to parse/serialize into the JSON that is transmitted over HTTP when working with the Cloud Vision API.

Step 5: Create Request

Create the request for performing Vision tasks over a user-provided image, with user-requested features.

AnnotateImageRequest annotateImageReq = new AnnotateImageRequest();
annotateImageReq.setFeatures(featureList);
annotateImageReq.setImage(getImageEncodeImage(bitmap));
annotateImageRequests.add(annotateImageReq);

AnnotateImageRequest annotateImageReq = new AnnotateImageRequest();

annotateImageReq.setFeatures(featureList);

annotateImageReq.setImage(getImageEncodeImage(bitmap));

annotateImageRequests.add(annotateImageReq);

Step 6: Process the Image

Now, you need to interact with the Vision API. Start by creating a HttpTransport and VisionRequestInitializer that contains your API key:

new AsyncTask<Object, Void, String>() {
    @Override
    protected String doInBackground(Object... params) {
          try {

              HttpTransport httpTransport = AndroidHttp.newCompatibleTransport();
              JsonFactory jsonFactory = GsonFactory.getDefaultInstance();
 
              VisionRequestInitializer requestInitializer = new VisionRequestInitializer(CLOUD_VISION_API_KEY);

              Vision.Builder builder = new Vision.Builder(httpTransport, jsonFactory, null);
              builder.setVisionRequestInitializer(requestInitializer);

              Vision vision = builder.build();

              BatchAnnotateImagesRequest batchAnnotateImagesRequest = new BatchAnnotateImagesRequest();
              batchAnnotateImagesRequest.setRequests(annotateImageRequests);

              Vision.Images.Annotate annotateRequest = vision.images().annotate(batchAnnotateImagesRequest);
              annotateRequest.setDisableGZipContent(true);
              BatchAnnotateImagesResponse response = annotateRequest.execute();
             
             return convertResponseToString(response);
         } catch (GoogleJsonResponseException e) {
              Log.d(TAG, "failed to make API request because " + e.getContent());
         } catch (IOException e) {
              Log.d(TAG, "failed to make API request because of other IOException " + e.getMessage());
       }
         return "Cloud Vision API request failed. Check logs for details.";
    }

    protected void onPostExecute(String result) {
         visionAPIData.setText(result);
         imageUploadProgress.setVisibility(View.INVISIBLE);
    }
}.execute();

new AsyncTask<Object, Void, String>() {

@Override

protected String doInBackground(Object... params) {

try {

HttpTransport httpTransport = AndroidHttp.newCompatibleTransport();

JsonFactory jsonFactory = GsonFactory.getDefaultInstance();

VisionRequestInitializer requestInitializer = new VisionRequestInitializer(CLOUD_VISION_API_KEY);

Vision.Builder builder = new Vision.Builder(httpTransport, jsonFactory, null);

builder.setVisionRequestInitializer(requestInitializer);

Vision vision = builder.build();

BatchAnnotateImagesRequest batchAnnotateImagesRequest = new BatchAnnotateImagesRequest();

batchAnnotateImagesRequest.setRequests(annotateImageRequests);

Vision.Images.Annotate annotateRequest = vision.images().annotate(batchAnnotateImagesRequest);

annotateRequest.setDisableGZipContent(true);

BatchAnnotateImagesResponse response = annotateRequest.execute();

return convertResponseToString(response);

} catch (GoogleJsonResponseException e) {

Log.d(TAG, "failed to make API request because " + e.getContent());

} catch (IOException e) {

Log.d(TAG, "failed to make API request because of other IOException " + e.getMessage());

}

return "Cloud Vision API request failed. Check logs for details.";

}

protected void onPostExecute(String result) {

visionAPIData.setText(result);

imageUploadProgress.setVisibility(View.INVISIBLE);

}

}.execute();

1.Label Detection

The Vision API can detect and extract information about entities within an image.Labels can identify objects, locations, activities, animal species, products, and more.

Vision API Label Detection

feature.setType("<b><em>LABEL_DETECTION</em></b>");

1	feature.setType("<b><em>LABEL_DETECTION</em></b>");

AnnotateImageResponse imageResponses = response.getResponses().get(0);
List<EntityAnnotation> entityAnnotations;

 entityAnnotations = imageResponses.getLabelAnnotations();
 if (entityAnnotation != null) {
        for (EntityAnnotation entity : entityAnnotation) {
               message = message + "    " + entity.getDescription() + " " + entity.getScore();
               message += "\n";
         }
 } else {
         message = "Nothing Found";
 }

AnnotateImageResponse imageResponses = response.getResponses().get(0);

List<EntityAnnotation> entityAnnotations;

entityAnnotations = imageResponses.getLabelAnnotations();

if (entityAnnotation != null) {

for (EntityAnnotation entity : entityAnnotation) {

message = message + " " + entity.getDescription() + " " + entity.getScore();

message += "\n";

}

} else {

message = "Nothing Found";

}

2.Landmark Detection

Landmark requests detect well-known natural and human-made landmarks and return identifying information such as an entity ID, the landmark’s name and location, and the bounding box that surrounds the landmark in the image.

Vision API Landmark Detection

feature.setType("LANDMARK_DETECTION");
....
entityAnnotations = imageResponses.getLogoAnnotations();

feature.setType("LANDMARK_DETECTION");

....

entityAnnotations = imageResponses.getLogoAnnotations();

3.Logo Detection

Logo detection requests detect popular product and corporate logos within an image.

feature.setType("LOGO_DETECTION");
...
entityAnnotations = imageResponses.getLogoAnnotations();

feature.setType("LOGO_DETECTION");

...

entityAnnotations = imageResponses.getLogoAnnotations();

4.Safe Search Detection

Safe Search requests examine an image for potentially unsafe or undesirable content. Likelihood of such imagery is returned in 4 categories:

adult indicates content generally suited for 18 years plus, such as nudity, sexual activity, and pornography (including cartoons or anime).
spoof indicates content that has been modified from the original to make it funny or offensive.
medical indicates content such as surgeries or MRIs.
violent indicates violent content, including but not limited to the presence of blood, war images, weapons, injuries, or car crashes.

feature.setType("SAFE_SEARCH_DETECTION");
...
SafeSearchAnnotation annotation = imageResponses.getSafeSearchAnnotation();
message = String.format("adult: %s\nmedical: %s\nspoofed: %s\nviolence: %s\n",
                annotation.getAdult(),
                annotation.getMedical(),
                annotation.getSpoof(),
                annotation.getViolence());

feature.setType("SAFE_SEARCH_DETECTION");

...

SafeSearchAnnotation annotation = imageResponses.getSafeSearchAnnotation();

message = String.format("adult: %s\nmedical: %s\nspoofed: %s\nviolence: %s\n",

annotation.getAdult(),

annotation.getMedical(),

annotation.getSpoof(),

annotation.getViolence());

5.Image Properties

An image properties request returns the dominant colors in the image as RGB values and percent of the total pixel count.

Vision API Image Property

feature.setType("IMAGE_PROPERTIES");
...
ImageProperties imageProperties = imageResponses.getImagePropertiesAnnotation();
message = getImageProperty(imageProperties);

DominantColorsAnnotation colors = imageProperties.getDominantColors();
for (ColorInfo color : colors.getColors()) {
        message = message + "" + color.getPixelFraction() + " - " + color.getColor().getRed() + " - " + color.getColor().getGreen() + " - " + color.getColor().getBlue();
        message = message + "\n";
}

feature.setType("IMAGE_PROPERTIES");

...

ImageProperties imageProperties = imageResponses.getImagePropertiesAnnotation();

message = getImageProperty(imageProperties);

DominantColorsAnnotation colors = imageProperties.getDominantColors();

for (ColorInfo color : colors.getColors()) {

message = message + "" + color.getPixelFraction() + " - " + color.getColor().getRed() + " - " + color.getColor().getGreen() + " - " + color.getColor().getBlue();

message = message + "\n";

}

Conclusion

In this tutorial, you learned how to use the Cloud Vision, which is part of the Google Cloud Machine Learning platform, in an Android app. There are many more such APIs offered by the platform. You can learn more about them by referring to the official documentation.

Download this project from GitHub

Android TensorFlow Machine Learning

Google Cloud Natural Language API in Android APP

Google Cloud Speech API in Android APP

Phone Number Authentication with Firebase Auth

Brijesh Thumar / July 15, 2017May 12, 2018 / Firebase

Over 90% of people who have issue login into an app is going to leave. Sign-in is such an important part of your growth funnel. Using phone number as an effective identity for authentication.

People prefer to use their phone as their identity instead of an email address or another form of identification. Phone numbers are higher quality.

Firebase Phone Auth

Many apps that actually don’t have phone number authentication today, building it in the first place can be a real struggle.

Firebase allow is it enables basic sign-in and sign-up flows. On Android Firebase actually, allow a few and built-in enhancements. Firebase has a really new cool piece of functionality called instant verification on Android.

Firebase enables basic sign-in and sign-up flows.

All a user would need to do is enter their phone number, and you can go ahead and pass it over to Firebase.

Firebase will validate the phone number, handle any normalization logic that needs to happen, if this is a new user, or a returning one, and go ahead and generate a one-time code associated with this request.

The new user will simply pass back the code into your application, and which you would just pass right back on to Firebase, and Firebase will verify it. Firebase will handle any of the complex logic, and deduplication, if you send multiple SMS by mistake or any of the other edge cases that might arise. The user will go ahead and enter their code. Firebase will validate it. Firebase will go ahead and mint cryptographically signed tokens, persist sessions the whole authentication. You won’t have to worry about how any of it.

Auto-retrieval

The best part of all of this is your user never actually had to leave your application, go to an SMS tab, wait for it and then copy and paste the code into your UI. They stayed in your application the entire time. It’s a really seamless experience for them overall. That’s all that you ever to do. All this powerful functionality is also really easy to use.

Configuration

As a pre-requisite, ensure your application is configured for use with Firebase.

Then, add the FirebaseUI auth library dependency. Please check the latest version

dependencies {
    // ...
    compile 'com.firebaseui:firebase-ui-auth:3.3.1'
}

dependencies {

// ...

compile 'com.firebaseui:firebase-ui-auth:3.3.1'

}

The second thing you need to do is just enable phone auth within your project in the Firebase console.

Enable Firebase Phone Auth

It’s really easy to use. In order to get all the awesome functionality the actual ability to generate and send codes, the auto-retrieval, even instant verification, comes down to just one single API call, and the implementation of the callbacks that you care about.

startActivityForResult(
             AuthUI.getInstance()
            .createSignInIntentBuilder()
            .setAvailableProviders(
                        Arrays.asList(new AuthUI.IdpConfig.Builder(AuthUI.PHONE_VERIFICATION_PROVIDER).build(),
                               new AuthUI.IdpConfig.Builder(AuthUI.GOOGLE_PROVIDER).build()))
                        .build(),
              RC_SIGN_IN);

startActivityForResult(

AuthUI.getInstance()

.createSignInIntentBuilder()

.setAvailableProviders(

Arrays.asList(new AuthUI.IdpConfig.Builder(AuthUI.PHONE_VERIFICATION_PROVIDER).build(),

new AuthUI.IdpConfig.Builder(AuthUI.GOOGLE_PROVIDER).build()))

.build(),

RC_SIGN_IN);

What’s about UI?

Building authentication UI can be pretty complex.FirebaseUI wrote out all the code for different phone authentication flows that.

The first thing you’ll notice is Firebase UI integrates with hint selector directly. You didn’t write any additional code for this. It’s able to pick up user’s phone number from the device, and I can just simply enter it directly into the app.

Now when you hit verify phone number, it’ll go ahead and send a code which will directly get right off the SMS. The code was sent to your device immediately written from the SIM. All I did was really tap my phone number and everything else was taken for me. The SMS was delivered. It was passed. It was taken. I never left the app.

Instant Verification

Instant verification checks if firebase has verified this phone number on the device recently. If it has, it can actually instantly verify the phone number the next time around without needing to send any additional SMS. It thinks that this means there is no wait time, on SMS, or anything else, just pure simplicity verify.

Automatic SMS Verification with the Phone Selector and SMS Retriever API

Brijesh Thumar / July 14, 2017May 20, 2018 / Firebase

Using SMS Retriever API, you can perform SMS-based verification in your app automatically, without requiring the user to manually type verification codes, and without requiring any extra app permissions.

The Google Play Services SDK(10.2 and newer) offering you to enable read the phone number and the verification SMS automatically without requiring these extra permissions.

1.Phone Selector API

Phone Selector API provides the Phone number to your app with a much better user experience and no extra permissions.Using this API, you can launch a dialogue, which shows the phone numbers on the device to the user.

First, you create a hint request object and set the phone number identifier supported field to true.

HintRequest hintRequest = new HintRequest.Builder()
                         .setHintPickerConfig(newCredentialPickerConfig.Builder().setShowCancelButton(true).build())
                         .setPhoneNumberIdentifierSupported(true)
                         .build();

HintRequest hintRequest = new HintRequest.Builder()

.setHintPickerConfig(newCredentialPickerConfig.Builder().setShowCancelButton(true).build())

.setPhoneNumberIdentifierSupported(true)

.build();

Then, you get a pending intent from that hint request for the phone number selector dialogue.

GoogleApiClient apiClient = new GoogleApiClient.Builder(getContext())
               .addApi(Auth.CREDENTIALS_API).enableAutoManage(getActivity(), GoogleApiHelper
               .getSafeAutoManageId(), new GoogleApiClient.OnConnectionFailedListener() {
                         @Override
                         public void onConnectionFailed(@NonNull ConnectionResult connectionResult) {
                             Log.e(TAG, "Client connection failed: " + connectionResult.getErrorMessage());
                         }
             }).build();


 PendingIntent intent = Auth.CredentialsApi.getHintPickerIntent(
            apiClient, hintRequest);

 startIntentSenderForResult(intent.getIntentSender(),
                            RESOLVE_HINT, null, 0, 0, 0);

GoogleApiClient apiClient = new GoogleApiClient.Builder(getContext())

.addApi(Auth.CREDENTIALS_API).enableAutoManage(getActivity(), GoogleApiHelper

.getSafeAutoManageId(), new GoogleApiClient.OnConnectionFailedListener() {

@Override

public void onConnectionFailed(@NonNull ConnectionResult connectionResult) {

Log.e(TAG, "Client connection failed: " + connectionResult.getErrorMessage());

}

}).build();

PendingIntent intent = Auth.CredentialsApi.getHintPickerIntent(

apiClient, hintRequest);

startIntentSenderForResult(intent.getIntentSender(),

RESOLVE_HINT, null, 0, 0, 0);

Once the user selects the phone number, that phone number is returned to your app in the onActivityResult().

@Override
public void onActivityResult(int requestCode, int resultCode, Intent data) {
    super.onActivityResult(requestCode, resultCode, data);
    if (requestCode == RC_PHONE_HINT) {
       if (data != null) {
          Credential cred = data.getParcelableExtra(Credential.EXTRA_KEY);
            if (cred != null) {
               final String unformattedPhone = cred.getId();
            }
       }    
    }
}

@Override

public void onActivityResult(int requestCode, int resultCode, Intent data) {

super.onActivityResult(requestCode, resultCode, data);

if (requestCode == RC_PHONE_HINT) {

if (data != null) {

Credential cred = data.getParcelableExtra(Credential.EXTRA_KEY);

if (cred != null) {

final String unformattedPhone = cred.getId();

}

2.Reading the verification code automatically using SMS retriever API

SMS Retriever API provides you the message content to your app without requiring any extra permissions.The key part is that it provides you only the message targeted your app.You have to verification code in your SMS message and include app-specific hash.This app-specific hash is a static hash that you can just include in the SMS template without requiring many code changes.

SMS Retriever API

Start the SMS retriever

This makes it wait for one matching SMS, which includes the app-specific hash.

SmsRetrieverClient client = SmsRetriever.getClient(mContext);

// Starts SmsRetriever, waits for ONE matching SMS message until timeout
// (5 minutes).
Task<Void> task = client.startSmsRetriever();

// Listen for success/failure of the start Task. 
task.addOnSuccessListener(new OnSuccessListener<Void>() {
  @Override
  public void onSuccess(Void aVoid) {
    // Log.d(TAG,"Successfully started retriever");
  }
});

task.addOnFailureListener(new OnFailureListener() {
  @Override
  public void onFailure(@NonNull Exception e) {
    Log.e(TAG,"Failed to start retriever");
  });
);

SmsRetrieverClient client = SmsRetriever.getClient(mContext);

// Starts SmsRetriever, waits for ONE matching SMS message until timeout

// (5 minutes).

Task<Void> task = client.startSmsRetriever();

// Listen for success/failure of the start Task.

task.addOnSuccessListener(new OnSuccessListener<Void>() {

@Override

public void onSuccess(Void aVoid) {

// Log.d(TAG,"Successfully started retriever");

}

});

task.addOnFailureListener(new OnFailureListener() {

@Override

public void onFailure(@NonNull Exception e) {

Log.e(TAG,"Failed to start retriever");

});

);

Once the SMS with the app-specific hash is received on the device, it is provided to your app via broadcast.

public class MySMSBroadcastReceiver extends BroadcastReceiver { 

  @Override
  public void onReceive(Context context, Intent intent) {
    if (SmsRetriever.SMS_RETRIEVED_ACTION.equals(intent.getAction())) {
      Bundle extras = intent.getExtras();
      Status status = (Status) extras.get(SmsRetriever.EXTRA_STATUS);

      switch(status.getStatusCode()) {
        case CommonStatusCodes.SUCCESS:
          // Get SMS message contents
          String message = (String) extras.get(SmsRetriever.EXTRA_SMS_MESSAGE);
          break;
        case CommonStatusCodes.TIMEOUT:
          Log.d(TAG,"timed out (5 minutes)"
          break;
      }
    }
  }
}

public class MySMSBroadcastReceiver extends BroadcastReceiver {

@Override

public void onReceive(Context context, Intent intent) {

if (SmsRetriever.SMS_RETRIEVED_ACTION.equals(intent.getAction())) {

Bundle extras = intent.getExtras();

Status status = (Status) extras.get(SmsRetriever.EXTRA_STATUS);

switch(status.getStatusCode()) {

case CommonStatusCodes.SUCCESS:

// Get SMS message contents

String message = (String) extras.get(SmsRetriever.EXTRA_SMS_MESSAGE);

break;

case CommonStatusCodes.TIMEOUT:

Log.d(TAG,"timed out (5 minutes)"

break;

}

In your broadcast receiver, you can get the message content from the extras.Once you have the message content, you can extract the verification code, and verify the code just like you would normally do.

<receiver android:name=".MySMSBroadcastReceiver" android:exported="true">
    <intent-filter>
        <action android:name="com.google.android.gms.auth.api.phone.SMS_RETRIEVED"/>
    </intent-filter>
</receiver>

<intent-filter>

</intent-filter>

</receiver>

After starting the SMS retriever, you can just send the SMS with the verification code and the app-specific hash to the phone using any backend infrastructure of yours.

3.Construct a verification message

Construct the verification message that you will send to the user’s device. This message must:

Be no longer than 140 bytes
Begin with one of the following strings:
- [#]
- Two consecutive zero-width space characters (U+200B)
End with an 11-character hash string that identifies your app

Otherwise, the contents of the verification message can be whatever you choose. It is helpful to create a message from which you can easily extract the one-time code later on. For example, a valid verification message might look like the following:

[#] Your ExampleApp code is: 123ABC78
FA+9qCX9VSu

1 2	[#] Your ExampleApp code is: 123ABC78 FA+9qCX9VSu

Computing your app’s hash string

You can get your app’s hash string with the AppSignatureHelper class from the SMS retriever sample app. However, if you use the helper class, be sure to remove it from your app after you get the hash string. Do not use hash strings dynamically computed on the client in your verification messages.

Android TensorFlow Machine Learning

Brijesh Thumar / July 8, 2017May 20, 2018 / TensorFlow

TensorFlow is an open source software library for machine learning, developed by Google and currently used in many of their projects.

In this article, we will create an Android app that can recognize five types of fruits.

Machine learning model inside your mobile app

Using the machine learnings, you can reduce the significant amount of traffic, and you can get much faster responses from your web server.Because you can extract the meaning from the raw data at client side.

For example, if you are using machine learning for image recognition, you can have the machine learning model running inside your mobile application so that your mobile application can recognize what kind of object is in each image.So that you can just send the label, such as a flower or human face, to the server.That can reduce the traffic.

Build an application that is powered by machine learning

TensorFlow is very valuable for people like me because I don’t have any sophisticated mathematical background.

Implement TensorFlow in Android

Android added a JSON integration, which makes step easier.Just add one line to the build.gradle, and the Gradle take care or the rest of steps.Under the library archive, holding TensorFlow shared object is downloaded from JCenter, linked against the application automatically.

dependencies {
 ....
 compile 'org.tensorflow:tensorflow-android:1.2.0-preview'
}

dependencies {

....

compile 'org.tensorflow:tensorflow-android:1.2.0-preview'

}

Android release inference library to integrate TensorFlow for Java Application.

Add your model to the project

We need the pre-trained model and label file.In the previous tutorial, we train model which does the image classification on a given image.You can download the model from here.Unzip this zip file, you will get retrained_labels.txt(label for objects) and rounded_graph.pb (pre-trained model).

Put retrained_labels.txt and rounded_graph.pb into android/assets directory.

public List<Recognition> recognizeImage(final Bitmap bitmap) {
     // Log this method so that it can be analyzed with systrace.
     Trace.beginSection("recognizeImage");

     Trace.beginSection("preprocessBitmap");
     // Preprocess the image data from 0-255 int to normalized float based
     // on the provided parameters.
     bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());
     for (int i = 0; i < intValues.length; ++i) {
         final int val = intValues[i];
         floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd;
         floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd;
         floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd;
     }
     Trace.endSection();

     // Copy the input data into TensorFlow.
     Trace.beginSection("feed");
     inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3);
     Trace.endSection();

     // Run the inference call.
     Trace.beginSection("run");
     inferenceInterface.run(outputNames, logStats);
     Trace.endSection();

     // Copy the output Tensor back into the output array.
     Trace.beginSection("fetch");
     inferenceInterface.fetch(outputName, outputs);
     Trace.endSection();

     // Find the best classifications.
      PriorityQueue<Recognition> pq =new PriorityQueue<Recognition>(
                        3,
                        new Comparator<Recognition>() {
                            @Override
                            public int compare(Recognition lhs, Recognition rhs) {
                                // Intentionally reversed to put high confidence at the head of the queue.
                                return Float.compare(rhs.getConfidence(), lhs.getConfidence());
                            }
                        });
        for (int i = 0; i < outputs.length; ++i) {
            if (outputs[i] > THRESHOLD) {
                pq.add(
                        new Recognition(
                                "" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i], null));
            }
        }
        final ArrayList<Recognition> recognitions = new ArrayList<Recognition>();
        int recognitionsSize = Math.min(pq.size(), MAX_RESULTS);
        for (int i = 0; i < recognitionsSize; ++i) {
            recognitions.add(pq.poll());
        }
        Trace.endSection(); // "recognizeImage"
        return recognitions;
    }

public List<Recognition> recognizeImage(final Bitmap bitmap) {

// Log this method so that it can be analyzed with systrace.

Trace.beginSection("recognizeImage");

Trace.beginSection("preprocessBitmap");

// Preprocess the image data from 0-255 int to normalized float based

// on the provided parameters.

bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());

for (int i = 0; i < intValues.length; ++i) {

final int val = intValues[i];

floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd;

floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd;

floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd;

}

Trace.endSection();

// Copy the input data into TensorFlow.

Trace.beginSection("feed");

inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3);

Trace.endSection();

// Run the inference call.

Trace.beginSection("run");

inferenceInterface.run(outputNames, logStats);

Trace.endSection();

// Copy the output Tensor back into the output array.

Trace.beginSection("fetch");

inferenceInterface.fetch(outputName, outputs);

Trace.endSection();

// Find the best classifications.

PriorityQueue<Recognition> pq =new PriorityQueue<Recognition>(

new Comparator<Recognition>() {

@Override

public int compare(Recognition lhs, Recognition rhs) {

// Intentionally reversed to put high confidence at the head of the queue.

return Float.compare(rhs.getConfidence(), lhs.getConfidence());

}

});

for (int i = 0; i < outputs.length; ++i) {

if (outputs[i] > THRESHOLD) {

pq.add(

new Recognition(

"" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i], null));

}

final ArrayList<Recognition> recognitions = new ArrayList<Recognition>();

int recognitionsSize = Math.min(pq.size(), MAX_RESULTS);

for (int i = 0; i < recognitionsSize; ++i) {

recognitions.add(pq.poll());

}

Trace.endSection(); // "recognizeImage"

return recognitions;

}

At first, create TensorFlow inference interface, opening the model file from the asset in the APK.Then, Set up the input feed using Feed API.

On mobile, the input feed tends to be retrieved from various sensors like a camera, accelerometer, then run the interface.

Finally, you can fetch the results using fetch method over there.You would notice that those calls are all blocking calls.So you’d want to run them in a worker thread, rather than the main thread because API would take a long time.This one is Java API.you can use regular C++ API as well.

Download this project from GitHub

Google Cloud Vision API in Android APP

TenserFlow Lite

Train Image classifier with TensorFlow

Train your Object Detection model locally with TensorFlow

Retrain TensorFlow Model for Image Classification

Brijesh Thumar / July 8, 2017May 18, 2018 / TensorFlow

In this post, we’re going to retrain an Image Classifier TensorFlow Model. You’ll need to install TensorFlow and you’ll need to understand how to use the command line.

1.Collect training data

We’re going to write a function to classify a piece of fruit Image. For starters, it will take an image of the fruit as input and predict whether it’s an apple or oranges as output.The more training data you have, the better a classifier you can create (at least 50 images of each, more is better).

The example folder fruits images should have a structure like this:

Image classifier folder

We will create a ~/tf_files/fruits folder and place each set of jpeg images in subdirectories (such as ~/tf_files/fruits/apple, ~/tf_files/fruits/orange etc)

The subfolder names are important.They define what label is applied to
each image, but the filenames themselves don’t matter.

A quick way to download multiple images at once is Chrome extension for batch download.

To retrain your classifier you need to run a couple of scripts. You only need to provide one thing–training data.

Step1: The retrain.py script is part of the TensorFlow repo.You need to download it manually, to the current directory(~/tf_files):

curl -O https://raw.githubusercontent.com/tensorflow/hub/master/examples/image_retraining/retrain.py

1	curl -O https://raw.githubusercontent.com/tensorflow/hub/master/examples/image_retraining/retrain.py

download retrain.py
Now, we have a trainer, we have data(Image), so let’s train! We will re-train the Inception v3 network.

Step2: Before starting the training active TensorFlow.

active tensorflow environment

Step3: Start your image retraining with one big command.

python retrain.py \
  --bottleneck_dir=bottlenecks \
  --how_many_training_steps=4000 \
  --model_dir=inception \
  --summaries_dir=training_summaries/basic \
  --output_graph=retrained_graph.pb \
  --output_labels=retrained_labels.txt \
  --image_dir=fruits

python retrain.py \

--bottleneck_dir=bottlenecks \

--how_many_training_steps=4000 \

--model_dir=inception \

--summaries_dir=training_summaries/basic \

--output_graph=retrained_graph.pb \

--output_labels=retrained_labels.txt \

--image_dir=fruits

Train Image classifier

These commands download the inception model and retrain it to classify images for ~/tf_files/fruits.

Train Image classifier accuracy

This operation can take several minutes depending on how many images you have and how many training steps you specified.

The script will generate two files: the model in a protobuf file (retrained_graph.pb) and a label list of all the objects it can recognize (retrained_labels.txt).

retrained grape files

Clone the Git repository for testing model

The following command will clone the Git repository containing the files required for the test model.

git clone https://github.com/googlecodelabs/tensorflow-for-poets-2

1	git clone https://github.com/googlecodelabs/tensorflow-for-poets-2

Copy tf file

The repo contains two directories: android/, and scripts/

1.android/: Directory contains nearly all the files necessary to build a simple Android app that classifies images.

2.scripts/: Directory contains the python scripts. These include scripts to prepare, test and evaluate the model.

Now copy the tf_files directory from the first part, into /tensorflow-for-poets-2 working directory.

Test the Model

The scripts/ directory contains a simple command line script, label_image.py, to test the network.

python -m scripts.label_image \
tf_files/fruits/grapes/grapes_2.jpg \
tf_files/retrained_graph.pb

python -m scripts.label_image \

tf_files/fruits/grapes/grapes_2.jpg \

tf_files/retrained_graph.pb

test trained model

Optimize model for Android

TensorFlow installation includes a tool, optimize_for_inference, that removes all nodes that aren’t needed for a given set of input and output nodes.

python -m tensorflow.python.tools.optimize_for_inference \
  --input=tf_files/retrained_graph.pb \
  --output=tf_files/optimized_graph.pb \
  --input_names="Cast" \
  --output_names="final_result"

python -m tensorflow.python.tools.optimize_for_inference \

--input=tf_files/retrained_graph.pb \

--output=tf_files/optimized_graph.pb \

--input_names="Cast" \

--output_names="final_result"

Optimize for inference

It creates a new file at tf_files/optimized_graph.pb.

Make the model compressible

The retrained model is still 84MB in size at this point. That large download size may be a limiting factor for any app that includes it.

Neural network operation requires a bunch of matrix characterizations, which means tons of multiply and add operations. Current mobile devices are capable of doing some of them with specialized hardware.

Quantization

Quantization is one of the techniques to reduce both memory footprint and computer load.

Now use the quantize_graph script to apply changes:

python -m scripts.quantize_graph \
  --input=tf_files/optimized_graph.pb \
  --output=tf_files/rounded_graph.pb \
  --output_node_names=final_result \
  --mode=weights_rounded

python -m scripts.quantize_graph \

--input=tf_files/optimized_graph.pb \

--output=tf_files/rounded_graph.pb \

--output_node_names=final_result \

--mode=weights_rounded

quantize graph

It does this without any changes to the structure of the network, it simply quantizes the constants in place.It creates a new file at tf_files/rounded_graph.pb.

Every mobile app distribution system compresses the package before distribution. So test how much the graph can be compressed:

Quantize compare

You should see a significant improvement. I get 73% optimize model.

Image Classify Using TensorFlow Lite

Train your Object Detection model locally with TensorFlow

Android TensorFlow Machine Learning

TenserFlow Lite

Prerequisites

1.Acquiring an API Key

2.Creating a New Android Project

Set Up to Authenticate With Your Project’s Credentials

Streaming Speech API Recognition Requests

Streaming requests

Streaming responses

Related Post

Prerequisites

1.Acquiring an API Key

2.Creating a New Android Project

Entity Analysis

Sentiment Analysis

Related Post

Prerequisites

1.Acquiring an API Key

2.Creating a New Android Project

Step 1: Create an Intent

Step 3: Encode the Image

Step 4: Create Feature

Step 5: Create Request

Step 6: Process the Image

1.Label Detection

2.Landmark Detection

3.Logo Detection

4.Safe Search Detection

5.Image Properties

Conclusion

Related Post

Firebase Phone Auth

Auto-retrieval

Configuration

What’s about UI?

Instant Verification

1.Phone Selector API

2.Reading the verification code automatically using SMS retriever API

Start the SMS retriever

3.Construct a verification message

Computing your app’s hash string

Machine learning model inside your mobile app

Build an application that is powered by machine learning

Implement TensorFlow in Android

Add your model to the project

Related Past

1.Collect training data

Clone the Git repository for testing model

Test the Model

Optimize model for Android

Make the model compressible

Quantization

Related Post