Salesforce Bulk API is required when we need to process high volume of data in Salesforce. This is quickest way to load data using programmatic ways.
When to use bulk api?
- Bulk API is based on REST principles and is optimized for loading or deleting large sets of data. Example: You have new client and they want to move their data from existing system to Salesforce.
- Salesforce processes batches in the background. So there is no problem of network speed. Once the records are upload, Salesforce will process it internally.
- We can easily control API calls using standard features (setup menu).
- We can also load data with Data Loader using CSV files
- Easiest way to use Bulk API is to enable it for processing records in Data Loader using CSV files.
Parallel Vs Serial Mode
We can process record using Bulk API V1.0 or V2.0. By default bulk API records are process using parallel mode, where multiple records are process in multiple batches. It might throw record lock or record lookup issue. Record lookup issue is where one records is being processed in one batch and referenced record is being processed in other batch, so it will create issue in processing. To avoid that situation we can go with serial mode or we can organize related records in same batch.
Serial mode processing is only possible in Bulk API V1.0. Bulk API V2.0 only process record in parallel mode.
How to process record in bulk API
Let us see how we can process record using Bulk API V1.0. For processing records we need below steps
- Login to Salesforce to get authorization token.
- Create bulk job
- Add batch to job
- Close job to start processing
- Get Job Status
We are using POSTMAN to execute Bulk API with above steps. In real implementation, POSTMAN request can be changed to specific language code. As we need to call Salesforce Bulk API from other system (in this case POSTMAN), we need connected app for this. So create one connected app in your Salesforce Orgs.
Creating Connected App:
A connected app is a framework that enables an external application like ASP.NET, Java etc. to integrate with Salesforce using APIs and standard protocols. Connected app can be created from Setup menu. Set below detail to create connected app.
| App Property | Description/Values |
|---|---|
| Connected App Name | External System App |
| Contact Email | put your email |
| Enable OAuth Setting | checked |
| Callback Url | https://www.getpostman.com/oauth2/callback |
| Selecte OAuth Usage | 1. Access and manage your data (api) 2. Provide access to your data via the Web 3. Perform requests on your behalf at any time (refresh-token, offline_access) |
Once connected app created, we will get Consumer Key and Consumer Secret. These detail along with Salesforce user will be used to run Bulk API in POSTMAN.
Let us start step by step process to process record using Bulk API.
1. Login to Salesforce to get Session Id
Bulk API V1.0 is a built as custom REST API, it is not a standard REST API technology. In V1.0, session id is used as token in next API calls. For getting session id, we have to use SOAPAction in login request. Ideally this is a SOAP request.
Login Request:
| Login Url | https://login.salesforce.com/services/Soap/c/50.0/ |
| Http Method | POST |
| Headers Sections | |
| Content-Type | text/xml |
| Accept-Encoding | text/xml;charset=UTF-8 |
| SOAPAction | “” <blank value> |
Based on above request, Session id (highlighted) will be generated. Generated session id will be used in next steps API calls.
2. Create bulk job
After session id is generated from first step. Let us create bulk job to process record. Job can be created for XML or JSON based request.
Session Id which is generated in first step is passed as http header X-SFDC-Session property.
| API Property | Values |
|---|---|
| API Url | {{instanceUrl}}/services/async/{{apiversion}}/job/’ |
| Http Method | POST |
| Http Header: X-SFDC-Session | <session id which is generated from first step> |
| Body (raw) | <?xml version=”1.0″ encoding=”UTF-8″?> <jobInfo xmlns=”http://www.force.com/2009/06/asyncapi/dataload”> <operation>insert</operation> <object>Account</object> <contentType>XML</contentType> </jobInfo> |
In this request body xml, we are passing below values
- operation: What kind of DML operation need to perform using Bulk API.
- object: On which object these DML operation will be performed.
- contentType : How the request will be processed, XML or CSV records.
Job Id is created after response is generated. This job id is required for batch job processing. This is mandatory for next API calls.
3. Add batch to Job
In this step we will upload data which need to be processed. This can be done in multiple ways. If you need to upload file directly use binary option in body or if you want to process record directly then use raw type. In case of binary, you will select file which need to process.
| API Property | Values |
|---|---|
| API Url | {{instanceUrl}}/services/async/50.0/job/{{apiversion}}/batch |
| Http Method | POST |
| Http Headers | |
| X-SFDC-Session | <Session Id generated in first step> |
| Content-Type | application/xml or text/csv |
| Body(raw-xml) | <?xml version=”1.0″ encoding=”UTF-8″?> <sObjects xmlns=”http://www.force.com/2009/06/asyncapi/dataload”> <sObject> <description>Created from Bulk API</description> <name>[Bulk API] Account 0 (batch 0)</name> </sObject> <sObject> <description>Created from Bulk API</description> <name>[Bulk API] Account 1 (batch 0)</name> </sObject> </sObjects> |
| Body(raw-csv) | Name,FirstName__c,LastName__c,Phone,Email__c Dhanik,Dhanik,Sahni,740-6604,dhanik.sahni@yahoo.com Poorvansh,Poorvansh,Sahni,886-0190,poorva@yahoo.com |
| Body(binary) | Select file which need to processed |
Put body based on request type. If you are using XML based processing then use Body(raw-xml), when CSV based records need to be processed then use Body(raw-csv).
We can upload multiple records by repeating this step.
To upload record data using files. Use binary body format to upload files.
4. Close Job
Once record is uploaded, we have to close job so that record processing will be started. This API request also informs Salesforce that no more batches will be submitted. To close job, below detail will be required.
| API Property | Values |
|---|---|
| API Url | {{instanceUrl}}/services/async/{{apiversion}}/job/{{_jobId}} |
| Http Method | POST |
| Http Headers | |
| X-SFDC-Session | {{sessionid generated in first step}} |
| Content-Type | application/json; charset=UTF-8 |
| Body (raw) | <?xml version=”1.0″ encoding=”UTF-8″?> <jobInfo xmlns=”http://www.force.com/2009/06/asyncapi/dataload”> <state>Closed</state> </jobInfo> |
I have observed some time this step is not required and processing got started immediately after uploading batch is completed.
5. Get Job Status
Once job is closed for batch processing, we can check status of job processing. Below detail will be passed as API request to get status.
| API Property | Value |
|---|---|
| API Url | {{instanceUrl}}/services/async/{{apiversion}}/job/{{_jobId}}/batch |
| Http Method | Get |
| Http Headers | |
| X-SFDC-Session | {{session id generated in first step}} |
