Home Salesforce Audio to Text by Google Speech API in Salesforce Lightning

Audio to Text by Google Speech API in Salesforce Lightning

by Dhanik Lal Sahni

Recently we were having a requirement for getting text from audio or video file.  These audios were majorly from customer calls with agents.  Once we get text from audio files , we can review those conversions. We can check how agents are discussing with customers. This will increase customer happiness.

There are many solutions available for converting audio/video file to text. Some of major solutions are below

  1. Google Speech API 
  2. Amazon Transcribe
  3. Microsoft Speech API
  4. Watson Speech to Text

This post is using Google Speech API to transcribe audio file into text. This audio file is attached in case record.  As a requirement, we need to convert attached audio file and add that to case comment in same record.

To make this requirement achievable, we have to create two lightning component. One component will get access token from Google authorization service and second component will use this access token to generate text from audio file.

Prerequisite for this solution:

  1. Create user in Google cloud at https://cloud.google.com/speech-to-text/
  2. Create project in Google Cloud and enable Cloud Speech-to-Text API at  https://console.cloud.google.com/apis/library
  3. Authorize domain in OAuth Consent Screen. Use Lightning url of your org without https.
  4. Create credential for this API. We need to set redirect url which will be lighting component url for authenticating and getting access token.

As mentioned above, to get transcription from audio we have to create two lightning components.

  1. First Lightning Component to get Access Token from Google Authentication Service
  2. Second Lightning Component to get Transcription from Speech API

1. Get Access Token from Google Authentication Service

Let us Authenticate Salesforce App and get access token from Google authentication service. Below steps will be used to get access token

  1. Create Apex class to generate URL for getting access token
  2. Create lightning component to get access token.
  3. Create custom metadata to store access token for later user.
  4. Make lightning component as tab
a. Create Apex class to generate URL for getting access token

Create apex class GoogleAuthService to get authentication url. This url will generate token after you authenticate using google credential created in step #1.

createAuthURL – This apex method will create authentication redirect url.

getAccessToken: This apex method will generate access token. This token will be saved in custom metadata for further use.

b. Create Lightning component to get access token

GoogleAuthComponent component is created for getting access token. It will use above mentioned GoogleAuthService class to generate token.


c. Create custom metadata to store access token for later user

Create one custom metadata type named GoogleAuthSetting and add field AccessToken in it. Add class MetadataService to update access token value. This class will be called from class GoogleAuthService.

d. Make lightning component as tab

Create a tab for this lightning component so that it will be accessible easily. Url of this tab should be added in step 3 of prerequisites.

2. Get Transcription from Speech API

Let us create another lightning component. This component will take case id and generate transcribed text from any attached audio file on specific record. This transcribed text is added in case comment.


As we have access token from first lightning component into custom metadata.  Let us use that access token from metadata to create transcribe text. Apex class GoogleSpeechService will generate translated text using https://speech.googleapis.com/v1p1beta1/speech:recognize API.

Add this component to case page layout.

Demo Time :

References:

https://cloud.google.com/speech-to-text

You may also like

21 comments

Iftekhar February 18, 2020 - 6:14 am

Hi Dhanik,
Excellent work on this project for speech to text conversion. I tried to run this and it failed from Line 33 of GoogleSpeechService.apxc class. I am sure you have forgot to add some other instructions. Please share this it this would help for loads of Dev communities. Keep up with your good work. Best regards, Iftekhar.

AudioData data=new AudioData();
AudioData.Audio audio=new AudioData.Audio();
audio.content=EncodingUtil.base64Encode(file_body);
data.audio=audio;
AudioData.Config config=new AudioData.Config();
config.encoding=’MP3′;
config.sampleRateHertz=16000;
config.languageCode=’en-US’;
config.enableWordTimeOffsets=false;
data.config=config;
string jsondta=system.JSON.serialize(data);

Reply
Dhanik Lal Sahni February 21, 2020 - 2:05 pm

Hello Ifte,

AudioData class was missing. I have added that class file. Please check now.

Thank You,
Dhanik

Reply
Iftekhar March 22, 2020 - 9:40 am

Hi Dhanik,
It’s a nice tool to have and really appreciate your help towards Dev Community. Google Speech API settings is not well defined on this page. An excellent example defined everything in details as on this this link: http://santanuboral.blogspot.com/2020/01/LWC-FileUpload-AWS.html?m=1 . While it’s convenient for you please provide full details.

Best regards,
Iftekhar Uddin.

Reply
Linda Swanson September 9, 2021 - 9:44 pm

I am getting this error on the GoogleSpeechService: I am getting 17 errors on the GoogleSpeechService.apxc

Reply
Dhanik Lal Sahni September 10, 2021 - 9:47 am

Hello Linda, As per your latest comment many of these are resolved. For remaining 2 bugs I have commented in your another question.

Thank You,
Dhanik

Reply
Linda Swanson September 9, 2021 - 10:09 pm

I have it down to 2 errors on the GoogleSpeechService:
Invalid type: GoogleAuthSetting__mdt
and
Variable does not exist: mapping
Please advise, I am very much a novice at this.
Thanks in advance.

Reply
Dhanik Lal Sahni September 10, 2021 - 9:45 am

Hello Linda,
GoogleAuthSetting__mdt is custom metadata which you have to create. See section 1.c Create custom metadata to store access token for later user for creating it.

Thank You,
Dhanik

Reply
Linda Swanson September 15, 2021 - 12:20 am

Thank you for your help, I did set it up and have no errors in the code. In the googleAuthServicethe call to the MetadataService does not seem to be saving the token to the custom metadata so when I go to transcribe the log gives a 11:29:10:325 FATAL_ERROR System.QueryException: List has no rows for assignment to SObject I did a system debug on authuri and get:
11:42:17:010 USER_DEBUG [21]|DEBUG|authuri https://accounts.google.com/o/oauth2/auth?client_id=206717206662-jdhqo2p2cegn574vik31a1skr2pmfl5e.apps.googleusercontent.com&response_type=code&scope=https://www.googleapis.com/auth/cloud-platform&redirect_uri=https%3A%2F%2Fmultco–uat.lightning.force.com%2Flightning%2Fn%2FSpeech_to_Text&access_type=offline

System Deug does not return anything on messageBody or response.

Reply
Dhanik Lal Sahni September 15, 2021 - 12:12 pm

Hey Linda,

First check you are getting token or not. If you are getting token then you can try saving in custom setting or object. Let me know, if that is not working, we can connect to resolve your issue.

Thank You,
Dhanik

Reply
Linda Swanson September 15, 2021 - 11:06 pm

That would be very helpful. I believe I am getting the token, but cannot prove it. My Google person thinks things are set up correctly.

Reply
Linda Swanson September 17, 2021 - 6:52 pm

When would you be available?

Reply
Dhanik Lal Sahni September 18, 2021 - 9:40 pm

Hey Linda, Please ping me on linked-in or telegram for this.

Thank You
Dhanik

Linda Swanson September 20, 2021 - 10:33 pm

Unable to message in either place

This site can’t be reachedt.me unexpectedly closed the connection.
Try:

Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_CLOSED

Reply
Saurabh June 23, 2022 - 1:18 pm

Hello Sir, I am also getting the same Error
FATAL_ERROR System.QueryException: List has no rows for assignment to SObject

Reply
Dhanik Lal Sahni July 6, 2022 - 4:58 pm

Hello Saurabh,

As per error record is not being retrieved for some filter criteria. Please provide complete error detail like the error method and line to check where is the issue.

Thank You,
Dhanik

Reply
Saurabh June 23, 2022 - 1:30 pm

Hello LINDA,

It gives me same error. can you tell me next step to resolve it

FATAL_ERROR : System.QueryException: List has no rows for assignment to SObject

Reply
Saurabh June 23, 2022 - 8:14 pm

Hello Linda, Dhanik,

hope your issue has been resolved, can you please confirm , I am also facing same issue

System.QueryException: List has no rows for assignment to SObject

Saurabh

Reply
Dhanik Lal Sahni July 6, 2022 - 4:58 pm

Hello Saurabh,

As per error record is not being retrieved for some filter criteria. Please provide complete error detail like the error method and line to check where is the issue.

Thank You,
Dhanik

Reply
Pranshu Sanchania June 2, 2023 - 12:24 pm

Hello Dhanik Sir,

First of all, thanks for the solution on this very common use case. This is the most nearest solution I could find after hard searching for few days. The steps looks straight forward but when I was trying to authorise my sandbox’s domain in OAth Consent Screen, I am getting the error – “Must be a top private domain’. Searched extensively but couldn’t find an answer around this. If you can help me around this issue, that would be great. Thanks.

Reply
Dhanik Lal Sahni June 7, 2023 - 4:10 pm

Hello Pranshu,

Please check this post Must be a top private domain

Thank You,
Dhanik

Reply
Nagaraj July 21, 2023 - 6:22 pm

This Article is Awesome. It’s helped me a lot. Sir, Please keep up your good work.
https://revaalolabs.com/post/best-free-speech-to-text-apis-and-open-source-libraries

Reply

Leave a Comment