Skip to content

Commit 5e3cac3

Browse files
yug-rajanikush-elastic
yug-rajani
and
kush-elastic
authored
[salesforce] Add salesforce package foundation along with login_rest data stream (elastic#4261)
* Add 'salesforce' package foundation along with 'login_rest' data stream * Add changelog.yml entry * Update mechanism to avoid duplication, pipeline and links in README * Remove entries from 'links_table.yml' * Minor updates to the pipeline * Add missing processors and update README as per review comments * Minor change to period description and update event.original testing * Add data stream descriptions and update field description as per review comment * Add Login Dashboard * Update Login dashboard * Update dashboard and Readme * Update Readme * Update dashboard timeframe * Address Review Comments * Address Review Comments * Update description * Address Review Comments * Address Review Comments * Update Readme * Update README Co-authored-by: kush-elastic <[email protected]>
1 parent f240fef commit 5e3cac3

File tree

22 files changed

+2852
-0
lines changed

22 files changed

+2852
-0
lines changed

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,7 @@
156156
/packages/radware @elastic/security-external-integrations
157157
/packages/redis @elastic/obs-service-integrations
158158
/packages/redisenterprise @elastic/obs-service-integrations
159+
/packages/salesforce @elastic/obs-service-integrations
159160
/packages/santa @elastic/security-external-integrations
160161
/packages/security_detection_engine @elastic/protections
161162
/packages/sentinel_one @elastic/security-external-integrations

‎packages/salesforce/_dev/build/build.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
dependencies:
2+
ecs:
3+
reference: [email protected]

‎packages/salesforce/_dev/build/docs/README.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Salesforce Integration
2+
3+
## Overview
4+
5+
The Salesforce integration allows you to monitor a [Salesforce](https://www.salesforce.com/) instance. Salesforce is a customer relationship management (CRM) platform. It provides an ecosystem for businesses to manage marketing, sales, commerce, service, and IT teams from anywhere with one integrated CRM platform.
6+
7+
Use the Salesforce integration to:
8+
- Gain insights into login and other operational activities by the users of your organization.
9+
- Create visualizations to monitor, measure and analyze the usage trend and key data, and derive business insights.
10+
- Create alerts to reduce the MTTD and also the MTTR by referencing relevant logs when troubleshooting an issue.
11+
12+
As an example, you can use the data from this integration to understand the activity patterns of users based on region or the distribution of users by license type.
13+
14+
## Data streams
15+
16+
The Salesforce integration collects log events using the REST API of Salesforce.
17+
18+
**Logs** help you keep a record of events happening in Salesforce.
19+
Log data streams collected by the Salesforce integration include [Login](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_eventlogfile_login.htm).
20+
21+
Data streams:
22+
- `login_rest`: Tracks login activity of users who log in to Salesforce.
23+
24+
## Compatibility
25+
26+
This integration has been tested against Salesforce `Spring '22 (v54.0) release`.
27+
28+
In order to find out the Salesforce version of your Instance, see below:
29+
30+
1. On the Home tab in Salesforce Classic, in the top right corner of the screen is a link to releases like `Summer '22`. This indicates your release.
31+
32+
2. An alternative way to find out the version of Salesforce is by hitting the following URL:
33+
- Format: (Salesforce Instance URL)/services/data
34+
- Example: https://elastic1234-dev-ed.my.salesforce.com/services/data
35+
36+
Example response:
37+
```xml
38+
<Versions>
39+
<Version>
40+
<label>Winter '22</label>
41+
<url>/services/data/v53.0</url>
42+
<version>53.0</version>
43+
</Version>
44+
<Version>
45+
<label>Spring '22</label>
46+
<url>/services/data/v54.0</url>
47+
<version>54.0</version>
48+
</Version>
49+
<Version>
50+
<label>Summer '22</label>
51+
<url>/services/data/v55.0</url>
52+
<version>55.0</version>
53+
</Version>
54+
</Versions>
55+
```
56+
The last one on the list is the release of your instance. In the example above, the version is `Summer '22` i.e. `v55.0`.
57+
58+
## Prerequisites
59+
60+
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
61+
You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended or self-manage the Elastic Stack on your own hardware.
62+
63+
In your Salesforce instance, ensure that `API Enabled permission` is selected for the user profile. Follow the below steps to enable the same:
64+
65+
1. Go to `Setup` > `Quick Find` > `Users`, and Click on `Users`.
66+
2. Click on the profile link associated with the `User Account` used for data collection.
67+
3. Search for `API Enabled` permission on the same page. In case it’s not present, search it under `System Permissions` and check if `API Enabled` privilege is selected. If not, enable it for data collection.
68+
69+
## Set Up
70+
71+
For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.
72+
73+
## Configuration
74+
75+
You need the following information from your Salesforce instance to configure this integration in Elastic:
76+
77+
### Salesforce Instance URL
78+
79+
The instance your Salesforce Organization uses is indicated in the URL of your browser's address bar in Salesforce Classic. The value before 'salesforce.com' is your Salesforce Instance.
80+
81+
Example URL: https://na9.salesforce.com/home/home.jsp
82+
83+
In the above example, the value before 'salesforce.com' is your Salesforce Instance. In this example, the Salesforce Organization is located on NA9.
84+
85+
The Salesforce Instance URL is: https://na9.salesforce.com
86+
87+
In Salesforce Lightning, it is available under the user name in the “View Profile” tab.
88+
89+
### Client Key and Client Secret for Authentication
90+
91+
In order to use this integration, you need to create a new Salesforce Application using OAuth. Follow the steps below to create a connected application in Salesforce:
92+
93+
1. Login to [Salesforce](https://login.salesforce.com/) with the same user credentials that you want to collect data with.
94+
2. Click on Setup on the top right menu bar. On the Setup page search `App Manager` in the `Search Setup` search box at the top of the page, then select `App Manager`.
95+
3. Click *New Connected App*.
96+
4. Provide a name for the connected application. This will be displayed in the App Manager and on its App Launcher tile.
97+
5. Enter the API name. The default is a version of the name without spaces. Only letters, numbers, and underscores are allowed. If the original app name contains any other characters, edit the default name.
98+
6. Enter the contact email for Salesforce.
99+
7. Under the API (Enable OAuth Settings) section of the page, select *Enable OAuth Settings*.
100+
8. In the Callback URL enter the Instance URL (Please refer to `Salesforce Instance URL`)
101+
9. Select the following OAuth scopes to apply to the connected app:
102+
- Manage user data via APIs (api).
103+
- Perform requests at any time (refresh_token, offline_access).
104+
- (Optional) In case of data collection, if any permission issues arise, add the Full access (full) scope.
105+
10. Select *Require Secret for the Web Server Flow* to require the app's client secret in exchange for an access token.
106+
11. Select *Require Secret for Refresh Token Flow* to require the app's client secret in the authorization request of a refresh token and hybrid refresh token flow.
107+
12. Click Save. It may take approximately 10 minutes for the changes to take effect.
108+
13. Click Continue and then under API details click Manage Consumer Details, Verify the user account using Verification Code.
109+
14. Copy `Consumer Key` and `Consumer Secret` from the Consumer Details section, which should be populated as value to Client ID and Client Secret respectively in the configuration.
110+
111+
For more details on how to Create a Connected App refer to the salesforce documentation [here](https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm).
112+
113+
### Username
114+
115+
User Id of the registered user in Salesforce.
116+
117+
### Password
118+
119+
Password used for authenticating the above user.
120+
121+
## Additional Information
122+
123+
Follow the steps below, in case you need to find the API version:
124+
125+
1. Go to `Setup` > `Quick Find` > `Apex Classes`.
126+
2. Click the `New` button.
127+
3. Click the `Version Settings` tab.
128+
4. Refer to the `Version` dropdown for the API Version number.
129+
130+
## Validation
131+
132+
After the integration is successfully configured, clicking on the Assets tab of the Salesforce Integration should display a list of available dashboards. Click on the dashboard available for your configured datastream. It should be populated with the required data.
133+
134+
## Troubleshooting
135+
136+
- In case of data ingestion if the user finds the following type of error logs:
137+
```
138+
{
139+
"log.level": "error",
140+
"@timestamp": "2022-11-24T12:59:36.835+0530",
141+
"log.logger": "input.httpjson-cursor",
142+
"log.origin": {
143+
"[file.name](http://file.name/)": "compat/compat.go",
144+
"file.line": 124
145+
},
146+
"message": "Input 'httpjson-cursor' failed with: input.go:130: input 8A049E17A5CA661D failed (id=8A049E17A5CA661D)\n\toauth2 client: error loading credentials using user and password: oauth2: cannot fetch token: 400 Bad Request\n\tResponse: {\"error\":\"invalid_grant\",\"error_description\":\"authentication failure\"}",
147+
"[service.name](http://service.name/)": "filebeat",
148+
"id": "8A049E17A5CA661D",
149+
"ecs.version": "1.6.0"
150+
}
151+
```
152+
Please check if the `API Enabled permission` is provided to the `profile` associated with the `username` used as part of the integration.
153+
Please refer to the Prerequisites section above for more information.
154+
155+
If the error continues follow these steps:
156+
157+
1. Go to `Setup` > `Quick Find` > `Manage Connected Apps`.
158+
2. Click on the Connected App name created by you to generate the client id and client secret (Refer to Client Key and Client Secret for Authentication) under the Master Label.
159+
3. Click on Edit Policies, and select `Relax IP restrictions` from the dropdown for IP Relaxation.
160+
161+
## Logs reference
162+
163+
### Login Rest
164+
165+
This is the `login_rest` data stream. It represents events containing details about your organization's user login history.
166+
167+
{{event "login_rest"}}
168+
169+
{{fields "login_rest"}}

‎packages/salesforce/_dev/deploy/docker/docker-compose.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
version: '2.3'
2+
services:
3+
salesforce:
4+
image: docker.elastic.co/observability/stream:v0.8.0
5+
hostname: salesforce
6+
ports:
7+
- 8010
8+
volumes:
9+
- ./files:/files:ro
10+
environment:
11+
PORT: 8010
12+
command:
13+
- http-server
14+
- --addr=:8010
15+
- --config=/files/config.yml

‎packages/salesforce/_dev/deploy/docker/files/config.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
rules:
2+
- path: /services/oauth2/token
3+
methods: ["POST"]
4+
responses:
5+
- status_code: 200
6+
headers:
7+
content-type: ['application/json']
8+
body: '{"access_token":"access_token","instance_url":"https://temporary-intance-url","id":"https://login.salesforce.com/id/temp_id/temp_token","token_type":"Bearer","issued_at":"1633689089545","signature":"signature"}'
9+
- path: /services/data/v54.0/query
10+
methods: ["GET"]
11+
query_params:
12+
q: ["SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE Interval = 'Hourly' AND EventType = 'Login' ORDER BY LogDate ASC NULLS FIRST"]
13+
responses:
14+
- status_code: 200
15+
body: '{"totalSize":1,"done":true,"records":[{"attributes":{"type":"EventLogFile","url":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GVrfnGAD"},"Id":"0AT5j00002GVrfnGAD","CreatedDate":"2022-09-14T21:43:41.000+0000","LogDate":"2022-09-13T00:00:00.000+0000","LogFile":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GVrfnGAD/LogFile"}]}'
16+
headers:
17+
content-type: ["text/json"]
18+
- path: /services/data/v54.0/query
19+
methods: ["GET"]
20+
query_params:
21+
q: ["SELECT Action,CreatedByContext,CreatedById,CreatedByIssuer,CreatedDate,DelegateUser,Display,Id,ResponsibleNamespacePrefix,Section FROM SetupAuditTrail ORDER BY CreatedDate ASC NULLS FIRST"]
22+
responses:
23+
- status_code: 200
24+
body: '{"done":true,"records":[{"Action":"insertConnectedApplication","CreatedByContext":"Einstein","CreatedById":"0055j000000utlPAAQ","CreatedByIssuer":null,"CreatedDate":"2022-08-16T09:26:38.000+0000","DelegateUser":"user1","Display":"For user [email protected], the User Verified Email status changed to verified","Id":"0Ym5j000019nwonCAA","ResponsibleNamespacePrefix":"namespaceprefix","Section":"Connected Apps","attributes":{"type":"SetupAuditTrail","url":"/services/data/v54.0/sobjects/SetupAuditTrail/0Ym5j000019nwonCAA"}}],"totalSize":160}'
25+
headers:
26+
content-type: ["text/json"]
27+
- path: /services/data/v54.0/sobjects/EventLogFile/0AT5j00002GVrfnGAD/LogFile
28+
methods: ["GET"]
29+
responses:
30+
- status_code: 200
31+
headers:
32+
content-type: ["text/csv"]
33+
body: |-
34+
"EVENT_TYPE","TIMESTAMP","REQUEST_ID","ORGANIZATION_ID","USER_ID","RUN_TIME","CPU_TIME","URI","SESSION_KEY","LOGIN_KEY","USER_TYPE","REQUEST_STATUS","DB_TOTAL_TIME","BROWSER_TYPE","API_TYPE","API_VERSION","USER_NAME","TLS_PROTOCOL","CIPHER_SUITE","AUTHENTICATION_METHOD_REFERENCE","TIMESTAMP_DERIVED","USER_ID_DERIVED","CLIENT_IP","URI_ID_DERIVED","LOGIN_STATUS","SOURCE_IP"
35+
"Login","20221122044615.591","4ehU_U-nbQyAPFl1cJILm-","00D5j000000VI3n","0055j000000utlP","83","30","/index.jsp","","QfNecrLXSII6fsBq","Standard","Success","52435102","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36","f","9998.0","[email protected]","TLSv1.2","ECDHE-RSA-AES256-GCM-SHA384","","2022-11-22T04:46:15.591Z","0055j000000utlPAAQ","81.2.69.142","s4heK3WbH-lcJIL3-n","LOGIN_NO_ERROR","81.2.69.142"
36+
- path: /services/data/v54.0/query
37+
methods: ["GET"]
38+
query_params:
39+
q: ["SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE Interval = 'Hourly' AND EventType = 'Logout' ORDER BY LogDate ASC NULLS FIRST"]
40+
responses:
41+
- status_code: 200
42+
body: '{"totalSize":1,"done":true,"records":[{"attributes":{"type":"EventLogFile","url":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAX"},"Id":"0AT5j00002GWEsRGAX","CreatedDate":"2022-09-19T21:03:41.000+0000","LogDate":"2022-09-18T00:00:00.000+0000","LogFile":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAX/LogFile"}]}'
43+
headers:
44+
content-type: ["text/json"]
45+
- path: /services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAX/LogFile
46+
methods: ["GET"]
47+
responses:
48+
- status_code: 200
49+
headers:
50+
content-type: ["text/csv"]
51+
body: |-
52+
"EVENT_TYPE","TIMESTAMP","REQUEST_ID","ORGANIZATION_ID","USER_ID","USER_TYPE","SESSION_TYPE","SESSION_LEVEL","BROWSER_TYPE","PLATFORM_TYPE","RESOLUTION_TYPE","APP_TYPE","CLIENT_VERSION","API_TYPE","API_VERSION","USER_INITIATED_LOGOUT","SESSION_KEY","LOGIN_KEY","TIMESTAMP_DERIVED","USER_ID_DERIVED","CLIENT_IP"
53+
"Logout","20221122073725.779","4exLFFQZ1234xFl1cJNwOV","00D5j001234VI3n","0055j000000utlP","S","O","1","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36","1015","9999","1000","9998","f","54.0","0","WvtsJ1235oW24EbH","Obv9123BzbaxqCo1","2022-11-22T07:37:25.779Z","0055j000000utlPAAQ","81.2.69.142"
54+
- path: /services/data/v54.0/query
55+
methods: ["GET"]
56+
query_params:
57+
q: ["SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE Interval = 'Hourly' AND (EventType = 'ApexCallout' OR EventType = 'ApexExecution' OR EventType = 'ApexRestApi' OR EventType = 'ApexSoap' OR EventType = 'ApexTrigger' OR EventType = 'ExternalCustomApexCallout') ORDER BY LogDate ASC NULLS FIRST"]
58+
responses:
59+
- status_code: 200
60+
body: '{"done":true,"records":[{"CreatedDate":"2022-10-01T23:22:27.000+0000","Id":"0AT5j00002GWEsRGAY","LogDate":"2022-09-30T00:00:00.000+0000","LogFile":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAY/LogFile","attributes":{"type":"EventLogFile","url":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAY"}}],"totalSize":1}'
61+
headers:
62+
content-type: ["text/json"]
63+
- path: /services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAY/LogFile
64+
methods: ["GET"]
65+
responses:
66+
- status_code: 200
67+
headers:
68+
content-type: ["text/csv"]
69+
body: |-
70+
"EVENT_TYPE","TIMESTAMP","REQUEST_ID","ORGANIZATION_ID","USER_ID","RUN_TIME","CPU_TIME","URI","SESSION_KEY","LOGIN_KEY","TYPE","METHOD","SUCCESS","TIME","REQUEST_SIZE","RESPONSE_SIZE","URL","TIMESTAMP_DERIVED","USER_ID_DERIVED","CLIENT_IP","URI_ID_DERIVED"
71+
"ApexCallout","20221122044615.591","ABCDE","00D5j000000VABC","0055j000000ABCD","1305","10","CALLOUT-LOG","ABCDEF","ABCDEFGH","OData","GET","1","1293","10","256","https://temp.sh/odata/Accounts","2022-11-22T04:46:15.591Z","0055j012345utlPAAQ","127.0.0.1","0055j000000utlPABCD"

‎packages/salesforce/changelog.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# newer versions go on top
2+
3+
- version: 0.1.0
4+
changes:
5+
- description: Salesforce integration package with "login_rest" data stream.
6+
link: https://github.com/elastic/integrations/pull/4261
7+
type: enhancement

‎packages/salesforce/data_stream/login_rest/_dev/test/pipeline/test-common-config.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
dynamic_fields:
2+
event.ingested: ".*"
3+
fields:
4+
tags:
5+
- preserve_original_event

‎packages/salesforce/data_stream/login_rest/_dev/test/pipeline/test-login-rest.log‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"API_TYPE":"f","API_VERSION":"9998.0","AUTHENTICATION_METHOD_REFERENCE":"","BROWSER_TYPE":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/IP_ADDRESS_REMOVED Safari/537.36","CIPHER_SUITE":"ECDHE-RSA-AES256-GCM-SHA384","CLIENT_IP":"175.16.199.0","CPU_TIME":"63","DB_TOTAL_TIME":"93531912","EVENT_TYPE":"Login","LOGIN_KEY":"NK2oPJrJze8tH+vC","LOGIN_STATUS":"LOGIN_NO_ERROR","ORGANIZATION_ID":"00D5j000000VI3n","REQUEST_ID":"4lGJ1R0ZSWVXwFl1cJIRH-","REQUEST_STATUS":"Success","RUN_TIME":"156","SESSION_KEY":"","SOURCE_IP":"175.16.199.0","TIMESTAMP":"20220913052243.429","TIMESTAMP_DERIVED":"2022-09-13T05:22:43.429Z","TLS_PROTOCOL":"TLSv1.2","URI":"/index.jsp","URI_ID_DERIVED":"s4heK3WbH-lcJIL3-n","USER_ID":"0055j000000utlP","USER_ID_DERIVED":"0055j000000utlPAAQ","USER_NAME":"[email protected]","USER_TYPE":"Standard"}

0 commit comments

Comments
 (0)