Speech-To-Text API

In previous post, I have given understanding of Text-to-Speech feature of Web Speech API. In this post, I will give detail of Speech-To-Text feature of this API.

We will create a demo lightning component. This component will get voice command and salesforce object record will open.
Voice command can be integrated using many APIs. Below are some important APIs which can be used for Speech Recognition.

• Web Speech API
• Google Speech-To-Text
• Microsoft Cognitive Services
• Dialogflow
• IBM Watson
• Speechmatics

We will use Web Speech API for speech Recognition. This API use browser’s audio stream to convert speech into text.

SpeechRecognition : This is speech recognition interface and it is available in browser’s window object. This object is present as SpeechRecognition in Firefox and as webkitSpeechRecognition in Chrome.

Below code will set recognition interface to SpeechRecognition.

window.SpeechRecognition = window.webkitSpeechRecognition || window.SpeechRecognition;

After setting recognition, create SpeechRecognition object using window.SpeechRecognition()

const recognition = new window.SpeechRecognition();

This recognition object has many properties, methods and event handlers.
Methods:

Properties:

Events

Let us create voice app which open record page based on what we speak using above methods, properties and events.

voiceCommand.cmp

<aura:component controller="VoiceCommandController" implements="force:appHostable,flexipage:availableForAllPageTypes,flexipage:availableForRecordHome,force:hasRecordId,forceCommunity:availableForAllPageTypes,force:lightningQuickAction" access="global" >
    <aura:attribute name="value" type="string" default=""></aura:attribute> 
    <lightning:card title="Speech-to-Text">
        <lightning:textarea aura:id="speechText1" value="{!v.value}"></lightning:textarea>
        <lightning:button variant="success" label="Click to Speak" title="Speak" onclick="{! c.handleSpeechText }"/>
    </lightning:card>   
</aura:component>

voiceCommandController.js

({
    handleSpeechText: function(component, event, helper) {
      window.SpeechRecognition = window.webkitSpeechRecognition || window.SpeechRecognition;
        
        if ('SpeechRecognition' in window) {
            console.log('supported speech')
        } else {
            console.error('speech not supported')
        }
        const recognition = new window.SpeechRecognition();
        recognition.lang = 'en-IN';
        recognition.continuous = true;
        recognition.onresult = (event) => {
            component.set("v.value",event.results[event.results.length -1][0].transcript);
            var commandText=event.results[event.results.length -1][0].transcript;
            var commands=commandText.split(' ');
            if(commands.length>0)
            {
            	var obj=commands[1];
            	var condition=commands[3];
            	helper.getRecords(component,obj,condition);
        	}
        }
        recognition.start();
    }
});

event.results[event.results.length -1][0].transcript this will give voice transcript.  After that we will split those transcript into tags/terms which need to fire in salesforce to open records.

We will get command in format of “open {object} for/of {objectname}”. For example “open account of dhanik” or “open lead for dhanik” .  In this transcript, we have two important term: object name and data value like account and dhanik.  We will split text transcript to get object name and data values using commandText.split(‘ ‘);.

voiceCommandHelper.js

Apex method is called to get specific record id. if multiple records are matching then it will return first record. Based on record id, application is navigated to record.

getRecords : function(cmp,objName,name) {
     	// create a one-time use instance of the serverEcho action
        // in the server-side controller
        var action = cmp.get("c.getRecord");
       	action.setParams(
           {
               objectName:objName,
               names : name
           });

        // Create a callback that is executed after 
        // the server-side action returns
        action.setCallback(this, function(response) {
            var state = response.getState();
            debugger;
            if (state === "SUCCESS") {
                  var data=response.getReturnValue();
                  var navEvt = $A.get("e.force:navigateToSObject");
                    navEvt.setParams({
                    "recordId": data.Id
                });
                navEvt.fire();
            }
            else if (state === "INCOMPLETE") {
                // do something
            }
            else if (state === "ERROR") {
                var errors = response.getError();
                if (errors) {
                    if (errors[0] && errors[0].message) {
                        console.log("Error message: " + 
                                 errors[0].message);
                    }
                } else {
                    console.log("Unknown error");
                }
            }
        });
        $A.enqueueAction(action);
   }

Above helper class will call Apex method to get record id of first matching records. This can be changed based on requirement.

public class VoiceCommandController {
    @Auraenabled
	public static sObject getRecord(string objectName, string names)
    {
        String name= '%'+names+'%';
        string soql='select id from '+objectName+' where name like : name';
        List<sObject> sobjList = Database.query(soql);
	if(sobjList.size()>0)
        {
    		return sobjList[0];
        }
        return null;
    }
}

Demo Time

The Speech Recognition API is can be useful for data entry, record navigation and other useful commands. We can create application to capture instant transcripts.