Rest API Pagination with Paging Library.

Almost every REST API your app call requires to handle pagination. When calling a REST API for resource delivering all of the results could be time-consuming, a REST API will typically make you paginate through the results. If you are trying to support pagination on a client you need to handle it gracefully.

Let’s say that there’s an API endpoint that returns a list of users. In the database of the API, there are 100,000+ users. It would be impractical to fetch all 100,000 users from the API all in one request. In fact, we would probably get an OutOfMemory exception. To avoid this, we want to paginate through our list of the user when making requests to our API.

Paging Library call REST API

 

Paging Library


The new paging library makes it easier for your app to gradually load information as needed from a REST API, without overloading the device or waiting too long for an all result.This library contains several classes to streamline the process of requesting data as you need it. These classes also work seamlessly with existing architecture components, like Room.

1.Adding Components to your Project


Architecture Components are available from Google’s Maven repository. To use them, follow these steps:

Open the build.gradle file for your project and add the line as shown below:

Add Architecture Components

In this tutorial, we are using LiveData, and ViewModel.

Open the build.gradle file for your app or module and add the artifacts that you need as dependencies:

2.Setting up Retrofit for Pagination


The code samples below for making the API calls are using Retrofit with GSON.We are working with the GitHub API and making a call to the GitHub User endpoint.

Now, let’s write our class that will generate our RetrofitService.

3.Create ItemKeyedDataSource


Use ItemKeyedDataSource class to define a data source.It uses for incremental data loading for paging keyed content, where loaded content uses previously loaded items as input to next loads.

Implement a DataSource using ItemKeyedDataSource if you need to use data from item N - 1 to load item N. This is common, for example, in sorted database queries where attributes of the item such just before the next query define how to execute it.

To implement, extend ItemKeyedDataSourcesubclass.

4.DataSourceFactory


A simple data source factory which also provides a way to observe the last created data source.This allows us to channel its network request status etc back to the UI.

5.Create ViewModel


In the ViewModel, we would extend from the Architecture Component ViewModel, and then we would keep a reference to that LiveData of our PagedList.

LivePagedListBuilder:This class generates a LiveData<PagedList> from the DataSource.Factory you provide.
PagedList: A PagedList is a List which loads its data in chunks (pages) from a DataSource.All data in a PagedList is loaded from its DataSource. Creating a PagedList loads data from the DataSource immediately, and should, for this reason, be done on a background thread. The constructed PagedList may then be passed to and used on the UI thread. This is done to prevent passing a list with no loaded content to the UI thread, which should generally not be presented to the user.

6.Create ProgressBar as Footer in a RecyclerView


Create RecyclerView with 2 type of items one is our usual items the second is a progress bar, then we need to listen NetworkState LiveData and decide are we going to show ProgressBar or not.

network_state_item.xml

Now, set the Network State and add or remove ProgressBar in Adapter.

hasExtraRow() check the Network State and return true or false.

To tell the PagedListAdapter how to compute the difference between the two elements, you’ll need to implement a new class, DiffCallback.Here, you will define two things.

You will define how to compute whether the contents are the same, and how to define whether the items are the same.
Let’s look at the adapter.So our adapter would extend PagedListAdapter, and then it will connect the user, which is the information that’s being displayed, with the user ViewHolder.

We define the callback, the DIFF_CALLBACK, for our user objects and then in onBindViewHolder, all we need to do is bind the item to the ViewHolder.

7.Create RecyclerView


In the onCreate, we get the reference to our ViewModel and get the reference to the RecyclerView, and we create our adapter.

 

Download this project from GitHub

 

Related Post

Architecture Components: Paging Library.

 

 

Architecture Components:Paging Library

Database queries take a long time to run and use a lot of memory. Android has a new paging library that can help you with all of this.

Main Classes of Paging Library


The main classes of the paging library are PagedListAdapter, that actually extends the RecyclerViewAdapter, PagedList, and DataSource.

DataSource: The DataSource is an interface for page source to provide the data gradually.

If you use the Room persistence library to manage your data, it can generate a DataSource.Factory to producePositionalDataSources for you automatically, for example:

PagedList: The PagedList is a component that loads the data automatically and can provide update signal, for example, to the RecyclerViewAdapter. The data is loaded automatically from a DataSource on the background thread.But then, it’s consumed on the main thread.It supports both an infinite scrolling list, but also countable lists.

You can configure several things.You can configure the initial load size, the page size, but also the prefetch distance.

PagedListAdapter:This class is an implementation of RecyclerView.Adapter that presents data from a PagedList. For example, when a new page is loaded, the PagedListAdapter signals the RecyclerView that the data has arrived; this lets the RecyclerView replace any placeholders with the actual items, performing the appropriate animation.

The PagedListAdapter also uses a background thread to compute changes from one PagedList to the next for example, when a database change produces a new PagedList with updated data, and calls the notifyItem…()methods as needed to update the list’s contents. RecyclerView then performs the necessary changes. For example, if an item changes position between PagedList versions, the RecyclerView animates that item moving to the new location in the list.

DataFlow


We have some data that we put on the DataSource in the background thread.The DataSource invalidates the PagedList and +updates its value.

Then on the main thread, the PagedList notifies its observers of the new value.Now the PagedListAdapter knows about the new value.

The background thread, the PageListAdapter needs to compute what’s changed, whats’ the difference and back on the UI thread, the View is updated in the onBinderViewHolder.

So all of this happens automatically.You just insert an item in that database, and then you see it animated in and no UI code is required.

Paging Library Example

Architecture Components Paging Demo

1.Adding Components to your Project


Architecture Components are available from Google’s Maven repository. To use them, follow these steps:

Open the build.gradle file for your project and add the line as shown below:

Add Architecture Components

In this tutorial, we are using Room, LiveData, and ViewModel.

Open the build.gradle file for your app or module and add the artifacts that you need as dependencies:

2.Create DataSource


Create Entity

Represents a class that holds a database row. For each entity, a database table is created to hold the items. You must reference the entity class in the Database class.

Data Access Objects (DAO)

To simplify the connection between the DataSource and the RecyclerView, we can use a LivePagedListProvider.So this will expose, actually, a LiveData of a PageList of our user.So all you will need to do is provide a DataSource.But if that DataSource is true, then it will be generated for you in the DAO.You don’t need to write any invalidating handling code.You can simply bind the LiveData of a PagedList to a PagedListAdapter and get updates, invalidates, and lifecycle cleanup with a single line of binding code.

So in our user DAO, we would return a LivePagedListProvider of our user to get the users by the last name.

Create Database

The annotation defines the list of entities, and the class’s content defines the list of data access objects (DAOs) in the database. It is also the main access point for the underlying connection.

The annotated class should be an abstract class that extends RoomDatabase.

3.Create ViewModel


In the ViewModel, we would extend from the Architecture Component ViewModel, and then we would keep a reference to that LiveData of our PagedList and we will get that reference from the DAO by calling getUsers(), and then call Create using the configuration that you want.So for example, setting the page size to 50, setting the prefetch distance to 50 and so on.

In the onCreate, we get the reference to our ViewModel.We get the reference to the RecyclerView, and we create our adapter.

4.Create Adapter


To tell the PagedListAdapter how to compute the difference between the two elements, you’ll need to implement a new class, DiffCallback.Here, you will define two things.

You will define how to compute whether the contents are the same, and how to define whether the items are the same.

Let’s look at the adapter.So our adapter would extend PagedListAdapter, and then it will connect the user, which is the information that’s being displayed, with the user ViewHolder.

We define the callback, the DIFF_CALLBACK, for our user objects and then in onBindViewHolder, all we need to do is bind the item to the ViewHolder.That’s all.

 

Download this project from GitHub

 

Related Post

Rest API Pagination with Paging Library.

 

Room: Database Relationships

Most modern applications today use databases for offline storage. This interaction is very easy using Room Persistence Library.In this tutorial, we’ll learn how to work with multiple tables that have relationships with each other. First, we will go over some core concepts, and then will begin working with JOIN queries in SQL.

There are several types of database relationships. Today we are going to cover the following:

  • One to One Relationships
  • One to Many and Many to One Relationships
  • Many to Many Relationships

When selecting data from multiple tables with relationships, we will be using the JOIN query.

Room: One-To-One Relationships


In this example you will learn how to map one-to-one relationship using Room. Consider the following relationship between Customer and Address entity.
Room: One-To-One mappingTo create this relationship you need to have a CUSTOMER and ADDRESS table. The relational model is shown below.

Room: One-To-One mapping

Use a primary key

Each entity must define at least 1 field as a primary key. Even when there is only 1 field, you still need to annotate the field with the @PrimaryKey annotation. Also, if you want Room to assign automatic IDs to entities, you can set the @PrimaryKey‘s autoGenerate property.

Define relationships between objects

You need to specify relationships between customer and address objects. The Room allows you to define Foreign Key constraints between entities.

For example, if there’s Customer entity, you can define its relationship to the Address entity using the @ForeignKeyannotation, as shown in the following code snippet.

Now we have a relationship between the Customers table and the Addresses table. If each address can belong to only one customer, this relationship is “One to One”. Keep in mind that this kind of relationship is not very common.

Our initial table that included the address along with the customer could have worked fine in most cases.Notice that now there is a field named “address_id” in the Customers table, that refers to the matching record in the Address table.

Room: One-To-Many Relationships


This is the most commonly used type of relationship. Consider an e-commerce app, with the following:

  • Customers can make many orders
  • Orders can contain many items
  • Items can have descriptions in many languages

In these cases, we would need to create “One to Many” relationships. Here is an example:
In the following example, you will learn how to map one-to-many relationship using Room. Consider the following relationship between Customer and Order entity.
Room: One-To-ManyAccording to the relationship, a Customer can have any number of Orders.
To create this relationship you need to have a Customer and Order table. The relational model is shown below.
Room: One-To-Many

Each customer may have zero, one or multiple orders. But an order can belong to only one customer.

To create Order table you need to create the following Java bean class.

Room: Many to Many Relationships


In some cases, you may need multiple instances on both sides of the relationship. For example, each order can contain multiple items. And each item can also be in multiple orders.

Room: Many to Many RelationshipsFor these relationships, we need to create an extra table:
Room: Many to Many Relationships
The item_order table has only one purpose, and that is to create a “Many to Many” relationship between the items and the orders.

To create the items and item_order tables you need to create the following Java Bean class.

Annotate indices and uniqueness

Sometimes, certain fields or groups of fields in a database must be unique. You can enforce this uniqueness property by setting the unique property of an @Index annotation to true. The following code sample prevents a table from having two rows that contain the same set of values for the order_id and item_id columns:

Join Queries


Some of your queries might require join tables to calculate the result. Room allows you to write any query. Furthermore, if the response is an observable data type, such as Flowable or LiveData, Room watches all tables referenced in the query for invalidation.

The following code snippet shows how to perform a table join to consolidate information between a table.

 

Related Post

Room Persistence Library

How to use DateTime datatype in SQLite Using Room

Room database Migrating

 

Image Classify Using TensorFlow Lite

Machine learning adds power to your application.TensorFlow Lite is a lightweight ML library for mobile and embedded devices.TensorFlow works well on large devices and TensorFlow Lite works really well on small devices. So that it’s easier and faster and smaller to work on mobile devices.

Getting Started TensorFlowLite


This post contains an example application using TensorFlow Lite for Android App. The app is a simple camera app that classifies images continuously using a quantized MobileNets model.

Step 1: Decide which Model to use


Depending on the use case, you may choose to use one of the popular open-sourced models such as InceptionV3 or MobileNets or re-train these models with their own custom data set or even build their own custom model.In this example, we use pre-train MobileNets model.

Step 2: Add TensorFlow Lite Android AAR


Android apps need to be written in Java, and core TensorFlow is in C++, a JNI library is provided to interface between the two.

Its interface is aimed only at inference, so it provides the ability to load a graph, set up inputs, and run the model to calculate particular outputs.

This app uses a pre-compiled TFLite Android Archive (AAR). This AAR is hosted on jcenter.

The following lines in the app’s build.gradle file includes the newest version of the AAR, from the TensorFlow maven repository, in the project.

We use the following block, to instruct the Android Asset Packaging Tool that .lite or .tflite assets should not be compressed. This is important as the .lite file will be memory-mapped, and that will not work when the file is compressed.

Step 3: Add your model files to the project


Download the quantized Mobilenet TensorFlow Lite model from here, unzip and copy mobilenet_quant_v1_224.tflite and label.txt to the assets directory: src/main/assets

Step 4: Load TensorFlow Lite Model


TensorFlow Lite’s Java API supports on-device inference and is provided as an Android Studio Library that allows loading models, feeding inputs, and retrieving inference outputs.

The Interpreter.java class drives model inference with TensorFlow Lite. In most of the cases, this is the only class an app developer will need.Initializing an Interpreter with a Model File.The Interpreter can be initialized with a MappedByteBuffer:

This byte buffer is sized to contain the image data once converted to float. The interpreter can accept float arrays directly as input, but the ByteBuffer is more efficient as it avoids extra copies in the interpreter.

The following lines load the label list and create the output buffer:

The output buffer is a float array with one element for each label where the model will write the output probabilities.

Running Model Inference

If a model takes only one input and returns only one output, the following will trigger an inference run:

For models with multiple inputs, or multiple outputs, use:

where each entry in inputs corresponds to an input tensor and map_of_indices_to_outputs maps indices of output tensors to the corresponding output data. In both cases the tensor indices should correspond to the values given to the TensorFlow Lite Optimized Converter when the model was created. Be aware that the order of tensors in input must match the order given to the TensorFlow Lite Optimized Converter.

Following method takes a Bitmap as input, runs the model and returns the text to print in the app.

This method does three things. First converts and copies the input Bitmap to the imgData ByteBuffer for input to the model. Then it calls the interpreter’s run method, passing the input buffer and the output array as arguments. The interpreter sets the values in the output array to the probability calculated for each class. The input and output nodes are defined by the arguments to the toco conversion step that created the .lite model file earlier.

 

 

Download this project from GitHub

Related Post

TensorFlow Lite

Train Image classifier with TensorFlow

 

TensorFlow Lite

If you want to bring the TensorFlow into your mobile applications there are some challenges you have to face. The neural network is big compared with the other classic machine learning models because deep learning you have to multiple layers. The total amount of the parameters and amount of the calculation is very large.

Freeze Graph

You can remove the all the variables from the TensorFlow graph and convert it into the constants. Once you have finish training you don’t have to those parameters in the variable. You can put everything into constant. Converting from variables to constants you can get much faster learning time.

Quantization in TensorFlow

Quantization is another optimization you can take for the mobile app. Quantizations means that you can compress the precision of each variable in parameters, weights, and biases into fewer operations. For example, TensorFlow uses the 32-bit floating point numbers for representing any weights and biases. But by using quantization, you can compress that into an 8-bit integer.

TensorFlow Lite


TensorFlow Lite is a lightweight ML library for mobile and embedded devices.TensorFlow works well on large devices and TensorFlow Lite works really well on small devices. It’s easier and faster and smaller to work on mobile devices.

How to build a model using TensorFlow Lite.

There are two aspects one is the workstation side and other one is the mobile side and let’s walk through the complete lifecycle.
TensorFlow Lite lifecycle

The first step is to decide what model you want to use. One option is to use a pre-trained model the other option would be to retrain just the last layers like you did in the post. You can write custom model and train and generate a graph this is nothing specific to TensorFlow Lite this is as good as standard TensorFlow where you build a model and generate graph depths and checkpoints.

The next step is specific to TensorFlow Lite is to convert the generated model into a format the TensorFlow Lite understands. Prerequisite to converting it is to freeze graph.

Freezing the graph is a step where you combine these two results and feed it to your converter. The converter is provided as part of the TensorFlow Lite software. You can use this to convert your model into the format that you need. Once conversion step is completed you will have what is called as a .lite binary file.

Move the model to the mobile side

You feed this TensorFlow Lite model into the interpreter.The interpreter executes the model using a set of operators.If the interpreter is running a CPU then this can be executed directly on the CPU otherwise if there is hardware acceleration then it can be executed on the hardware accelerated hardware as well.

Components of TensorFlow Lite

TensorFlow Lite ComponentThe main components of TensorFlow Lite are the model file format, the interpreter for processing the graph, a set of kernels to work to or where the interpreter can invoke a set of kernels, and lastly an interface to the hardware acceleration layer.

 

 

1.Model File

TensorFlow Lite has a special model file formate and this is lightweight and has very few dependencies and most graph calculations are done using 32-bit float

2.Interpreter

The interpreter is engineered to be work with low overhead and on small devices. TensorFlow Lite has very few dependencies and it is easy to build on simple devices.TensorFlow Lite kept the binary size of 70KB and 300KB with operators.

It uses FlatBuffers. So it can load really and the speed comes at the cost of flexibility.TensFolw Lite support only a subset of operators that TensorFlow has.

3.Ops/Kernels

The set of operators are smaller. Every model will be not supported them, in particular, TensorFlow Lite provides a set of core built-in ops and these have been optimized for arm CPU using neon and they work in both float and quantized.

4.Interface to Hardware Acceleration

It targets custom hardware. It is the neural network API TensorFlow lite comes pre-loaded with hooks for neural network API. If your device supports NN API then tensor flow lite will delegate these operators into NN API and if you have a device that does not support NN API it’s executed directly on the CPU.

Android Neural Network API


Android Neural Network API is supported for Android with 8.1+ release in Oreo. It will support various hardware acceleration. It uses TensorFlow as a core technology.

You can use TensorFlow to write your mobile app and your app will get the benefits of hardware acceleration through your NN API. It basically abstracts the hardware layer for ML inference, for example, if a device has ML DSP it can transparently map to it and it uses NN primitives that are very similar to TensorFlow Lite.

android neural network architecture

Architecture for neural network API’s looks like this essentially there’s an android app. On top typically there is no need for the Android app to access the neural network API directly it will be accessing it through the machine learning interface which is the TensorFlow Lite interpreter and the NN runtime. The neural network runtime can talk to the hardware abstraction layer and then which talks to their device and run various accelerators.

 

Related Post

Image Classify Using TensorFlow Lite

Train Image classifier with TensorFlow

Train your Object Detection model locally with TensorFlow

Android TensorFlow Machine Learning