Object Identification using Google Cloud Vision in Salesforce

Object Detection is used to locate the presence of objects in the image. We can find the location of the object as well in the image. It can detect a single object as well as multiple objects in the image.

As an example in the above image, we can get the number of objects in the picture, the classification of the object (cat, dog, duck), and the location of the object.

Lots of APIs are available for object detection. Here is a list of those top APIs

• Google CloudVision API
• Amazon Rekognition
• IBM Watson Visual Recognition
• Microsoft Image Processing API
• Clarifai
• Imagga

This post is using Google Cloud Vision API to detect objects in the image. Below steps are required for the Object Detection API call

• Get Access Token from Google to call API
• Create Apex class to call API
• Create an LWC Component to show an object list

1. Get an Access Token from Google to call the API

Refer blog http://salesforcecodex.com/2020/02/integrating-google-api-with-salesforce/ to get an access token for Google Cloud Vision API.

You need to enable Google Cloud Vision API for access token from https://console.cloud.google.com/apis/library/vision.googleapis.com. Use the below scope in GoogleAuthService class.

private static string scope='https://www.googleapis.com/auth/cloud-vision';

2. Create Apex class to call API

Let us create an apex class to call Google Cloud Vision API. We have to use auth token generated from the above step. We have to get auth token from custom metadata GoogleAuthSetting__mdt before calling API. We are updating this metadata in the previous step after getting the token.

API Detail

API Url : https://vision.googleapis.com/v1/images:annotate

Above API will be called as POST method. For this post, I am using blob image to get brand information. Sample JSON request data will be like this

{
  "requests": [
    {
      "image": {
        "content": "base64-encoded-image" //This will be content version data
      },
      "features": [
        {
          "maxResults": 10,  // Number of object which will be return
          "type": "OBJECT_LOCALIZATION"
        },
      ]
    }
  ]
}

Response data will be like this.

 {
          "mid": "/m/01bqk0",
          "name": "Bicycle wheel",
          "score": 0.93416494,
          "boundingPoly": {
            "normalizedVertices": [
              {
                "x": 0.5033941,
                "y": 0.7553
              },
              {
                "x": 0.6290014,
                "y": 0.7553
              },
              {
                "x": 0.6290014,
                "y": 0.9428198
              },
              {
                "x": 0.5033941,
                "y": 0.9428198
              }
            ]
          }
        },

Request, response, and API call will be like the below code. ObjectRecogRequest and ObjectRecogResponse are wrapper classes for the above request and response JSON data.

Apex Method Detail :

getAccessToken : This method will get token detail which is generated from step 1 and used in getObjectCases methods.

getObjectCases: This will get all cases which are for object recognition. This will be used in LWC to show dropdown selection.

getImage : This will get image information related to case. Image is shown in frame to visualize which image is being processed by API.

getObjectInformation: This will get objects information by calling Google Cloud Vision API. It will return list of objects in the image.

ObjectInformation wrapper class list is returned with object name and accuracy score.

By default, Google Cloud Vision API returns 10 rows. If you need to get more rows then provide maxResults value. This attribute shows how many rows Google API will return.

3. Create LWC Component to show Image and Object Identification List

Create LWC component to show and process images. Based on the shown image it will process requests.

We can create record type in case object to get cases for object recognition. We can process only those records.

When LWC Component is opened, it will show a list of all cases which is created as Object Recognition record type.

handleCaseChange : This will call apex method getImage to get image and show that in iframe.

getObjectDetail: This will call apex method getObjectInformation to get object information and show detail in data table.

Sample Output

Demo:

Reference:

https://cloud.google.com/vision/docs

https://cloud.google.com/vision/docs/detecting-logos

https://developer.salesforce.com/docs/component-library/documentation/en/lwc