AST

Author: chrisda

defender-office-365/attack-simulation-training-faq.md

@@ -19,7 +19,7 @@ ms.custom:
   - seo-marvel-apr2020
 description: Admins can learn about deployment considerations and frequently asked questions regarding Attack simulation and training in Microsoft 365 E5 or Microsoft Defender for Office 365 Plan 2 organizations.
 ms.service: defender-office-365
-ms.date: 6/22/2023
+ms.date: 06/14/2024
 appliesto:
   - ✅ <a href="https://learn.microsoft.com/defender-office-365/mdo-about#defender-for-office-365-plan-1-vs-plan-2-cheat-sheet" target="_blank">Microsoft Defender for Office 365 Plan 2</a>
 ---
@@ -144,8 +144,17 @@ There are no built-in options to add safety tips to payloads, but you can use th
 
 Yes. For more information, see [Training campaigns in Attack simulation training](attack-simulation-training-training-campaigns.md).
 
+### How do I find out about simulation messages that weren't delivered?
+
+The [Users tab](attack-simulation-training-simulations.md#users-tab) for the simulation is filterable by **Simulation message delivery: Failed to deliver**.
+
+If you won the sender domain, the undelivered simulation report is returned in a non-delivery report (also known as an NDR or bounce message). For more information about the codes in the NDR, see [Email non-delivery reports and SMTP errors in Exchange Online](/exchange/troubleshoot/email-delivery/ndr/non-delivery-reports-in-exchange-online).
+
 ## Issues with Attack simulation training reporting
 
+> [!TIP]
+> Simulation data recording start a few minutes after the simulation is launched and after users begin interacting with the simulation messages. There's no fixed start time. Events are still captured after the simulation ends.
+
 ### Differences in user activity data from Attack simulation training reports and other reports
 
 For reporting on user activity related to simulation messages, we recommend using the [built-in simulation reports](attack-simulation-training-insights.md). Reports from other sources (for example, [Advanced hunting](/defender-xdr/advanced-hunting-overview)) might not be accurate.
@@ -177,6 +186,9 @@ Attack simulation training supports on-premises mailboxes, but with reduced repo
 - Data on whether users read, forwarded, or deleted the simulation email isn't available for on-premises mailboxes.
 - The number of users who reported the simulation email isn't available for on-premises mailboxes.
 
+> [!TIP]
+> Other than the simulation messages being sent via the transport pipeline vs. direct injection in Microsoft 365, the training, automation, and content management experiences are the same for on-premises mailboxes.
+
 ### Simulation reports aren't updated immediately
 
 Detailed simulation reports aren't updated immediately after you launch a campaign. Don't worry; this behavior is expected.
@@ -251,6 +263,12 @@ Managing a large CSV file or adding many individual recipients can be cumbersome
 
 > [!TIP]
 > Currently, shared mailboxes aren't supported in Attack simulation training. Simulations should target user mailboxes or groups containing user mailboxes.
+>
+> Distribution groups are expanded and the list of users is generated at the time of saving the simulation or simulation automation.
+
+### Q: Are the limits for the number of simulations that can be deployed during a specific time interval?
+
+A. No, although you might experience slowness if you launch many parallel simulations. Message rates (including simulation message rates) are constrained by the [message rate limits of the service](/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits#receiving-and-sending-limits).
 
 ### Q: Does Microsoft provide payloads in other languages?
 
@@ -293,27 +311,40 @@ At 9:00 AM on the same day, the simulation message is sent to UserB. With region
 
 So, on the initial run of a campaign with region aware delivery enabled, it might appear that the simulation message was sent only to users in a specific time zone. But, as time passes and more users come into scope, the targeted users increase.
 
+If you don't use region aware delivery, the campaign starts based on the time zone of the user who's setting it up.
+
 ### Q: Does Microsoft collect or store any information that users enter at the Credential Harvest sign-in page, used in the Credential Harvest simulation technique?
 
 A: No. Any information entered at the credential harvest sign-in page is discarded silently. Only the 'click' is recorded to capture the compromise event. Microsoft doesn't collect, log or store any details that users enter at this step.
 
 ### Q: How long is simulation information retained? Can I delete simulation data?
 
-A: Simulation data is retained for 18 months from the date of generation. Data includes:
-
-- Simulation metadata
-- User activity
-- Aggregate reports
-
-The following data is retained until you [delete the simulation](attack-simulation-training-simulations.md#remove-simulations):
-
-- Tenant payloads
-- Notifications
-- Login pages
+A: See the following table:
+
+|Data type|Retention|
+|---|---|
+|Simulation metadata|18 months unless the [simulation is deleted sooner by an admin](attack-simulation-training-simulations.md#remove-simulations).|
+|Simulation automation|18 months unless the [simulation automation is deleted sooner by an admin](attack-simulation-training-simulation-automations.md#remove-simulation-automations).|
+|Payload automation|18 months unless the [payload automation is deleted sooner by an admin](attack-simulation-training-payload-automations.md#remove-payload-automations).|
+|User activity in simulation metadata|18 months unless deleted by an admin.|
+|Global payloads|Persisted unless deleted by Microsoft.|
+|Tenant payloads|18 months unless the [archived payload is deleted sooner by an admin](attack-simulation-training-payloads.md#remove-archived-payloads).|
+|User activity in training metadata|18 months unless the [simulation is deleted sooner by an admin](attack-simulation-training-simulations.md#remove-simulations).|
+|MDO recommended payloads|6 months.|
+|Global end user notifications|Persisted unless deleted by Microsoft.|
+|Tenant end user notifications|18 months unless the [notification is deleted sooner by an admin](attack-simulation-training-end-user-notifications.md#remove-end-user-notifications).|
+|Global login pages|Persisted unless deleted by the service.|
+|Tenant login pages|18 months unless the [login page is deleted sooner by an admin](attack-simulation-training-login-pages.md#remove-login-pages).|
+|Global landing pages|Persisted unless deleted by Microsoft|
+|Tenant landing pages|18 months unless the [landing page is deleted sooner by an admin](attack-simulation-training-landing-pages.md#remove-landing-pages).|
+
+If the entire tenant is deleted, attack simulation training data is deleted after 90 days.
+
+For more information, see [Data retention information for Microsoft Defender for Office 365](mdo-data-retention.md).
 
 ### Q: Can I create, view, and manage simulations using an API?
 
-A: Read and write scenarios are supported using the Microsoft Graph API:
+A: Yes. Read and write scenarios are supported using the Microsoft Graph API:
 
 - `AttackSimulation.Read.All`:
   - Read simulation metadata
@@ -323,3 +354,11 @@ A: Read and write scenarios are supported using the Microsoft Graph API:
 - `AttackSimulation.ReadWrite.All`: Run simulations using the specified payloads, notifications, and login pages.
 
 For more information, see [List simulations](/graph/api/attacksimulationroot-list-simulations) and [Reports API overview for attack simulation training as part of Microsoft Defender for Office 365](/graph/api/resources/report-m365defender-reports-overview).
+
+### Q: Can I delete custom payloads?
+
+A: Yes. First you archive the payload, then you delete the archived payload. For instructions, see [Archive payloads](attack-simulation-training-payloads.md#archive-payloads).
+
+### Q: Can I modify the built-in payloads?
+
+A: Not directly. You can copy the payload and then modify the copy. For instructions, see [Copy payloads](attack-simulation-training-payloads.md#copy-payloads).

defender-office-365/attack-simulation-training-insights.md

@@ -51,7 +51,7 @@ To go to the **Overview** tab, open the Microsoft Defender portal at <https://se
 The distribution of insights on the tabs is described in the following table:
 
 |Report|Overview tab|Reports tab|
-|---|:---:|:---:|
+|---|:---:|:---:.|
 |[Recent simulations card](#recent-simulations-card)|✔||
 |[Recommendations card](#recommendations-card)|✔||
 |[Simulation coverage card](#simulation-coverage-card)|✔|✔|
@@ -313,7 +313,7 @@ The **Delivery status** section on **Report** tab** for a simulation shows the n
 
 - **Successfully received message**
 - **Positive reinforcement message delivered**
-- **Just simulation message delivered
+- **Just simulation message delivered**
 
 Select **View users to whom message delivery failed** to go to the [Users tab](attack-simulation-training-simulations.md#users-tab) tab in the report where the results are filtered by **Simulation message delivery: Failed to deliver**.
 
@@ -380,6 +380,71 @@ You can hover over a section in the chart to see the actual numbers in each cate
 
 :::image type="content" source="media/attack-sim-report-training-campaign-report-tab-all-user-activity.png" alt-text="The All user activity section on the Report tab in the Training campaign report in Attack simulation training." lightbox="media/attack-sim-report-training-campaign-report-tab-all-user-activity.png":::
 
+## Appendix
+
+When you export information from the reports, the CSV file contains more information than what's shown in the report, even if you have all column shown. The fields are described in the following table.
+
+> [!TIP]
+> For maximum information, verify that all available columns in the report are visible before you export.
+
+|Field Name|Description|
+|---|---|
+|UserName|Username of the user who did the activity.|
+|UserMail|Email address of the user who did the activity.|
+|Compromised|Indicates if the user was compromised. Values are Yes or No.|
+|AttachmentOpened_TimeStamp|When the attachment was opened.|
+|AttachmentOpened_Browser|When the attachment was opened in a web browser. This information comes from UserAgent.|
+|AttachmentOpened_IP|The IP address where the attachment was opened. This information comes from UserAgent.|
+|AttachmentOpened_Device|The device where the attachment was opened. This information comes from UserAgent.|
+|AttachmentLinkClicked_TimeStamp|When the attachment link was clicked.|
+|AttachmentLinkClicked_Browser|The web browser that was used to click the attachment link. This information comes from UserAgent.|
+|AttachmentLinkClicked_IP|The IP address where the attachment link was clicked. This information comes from UserAgent.|
+|AttachmentLinkClicked_Device|The device where the attachment link was clicked. This information comes from UserAgent.|
+|CredSupplied_TimeStamp(Compromised)|When the user entered their credentials.|
+|CredSupplied_Browser|The web browser that was used when the user entered their credentials. This information comes from UserAgent.|
+|CredSupplied_IP|The IP address where the user entered their credentials. This information comes from UserAgent.|
+|CredSupplied_Device|The device where the user entered their credentials. This information comes from UserAgent.|
+|SuccessfullyDeliveredEmail_TimeStamp|When the simulation email message was delivered to the user.|
+|MessageRead_TimeStamp|When the simulation message was read.|
+|MessageDeleted_TimeStamp|When the simulation message was deleted.|
+|MessageReplied_TimeStamp|When the user replied to the simulation message.|
+|MessageForwarded_TimeStamp|When the user forwarded the simulation message.|
+|OutOfOfficeDays|Determines whether the user is out of office. This information comes from the Automatic replies setting in Outlook.|
+|PositiveReinforcementMessageDelivered_TimeStamp|When the positive reinforcement message was delivered to the user.|
+|PositiveReinforcementMessageFailed_TimeStamp|When the positive reinforcement message failed to be delivered to the user.|
+|JustSimulationMessageDelivered_TimeStamp|When the simulation message was delivered to the user as part of a simulation with no trainings assigned (**No training** was selected on the **Assign training** page of the new simulation wizard).|
+|JustSimulationMessageFailed_TimeStamp|When the simulation email message failed to be delivered to the user, and the simulation had no trainings assigned.|
+|TrainingAssignmentMessageDelivered_TimeStamp|When the training assignment message was delivered to the user. This value is empty if no trainings were assigned in the simulation.|
+|TrainingAssignmentMessageFailed_TimeStamp|When the training assignment message failed to be delivered to the user. This value is empty if no trainings were assigned in the simulation.|
+|FailedToDeliverEmail_TimeStamp|When the simulation email message failed to be delivered to the user.|
+|Last Simulation Activity|The last simulation activity of the user (whether they passed or were compromised).|
+|Assigned Trainings|The list of trainings assigned to the user as part of the simulation.|
+|Completed Trainings|The list of trainings completed by the user as part of the simulation..|
+|Training Status|The current status of trainings for the user as part of the simulation.|
+|Phishing Reported On|When the user reported the simulation message as phishing.|
+|Department|The user's Department property value in Microsoft Entra ID at the time of simulation.|
+|Company|The user's Company property value in Microsoft Entra ID at the time of simulation.|
+|Title|The user's Title property value in Microsoft Entra ID at the time of simulation.|
+|Office|The user's Office property value in Microsoft Entra ID at the time of simulation.|
+|City|The user's City property value in Microsoft Entra ID at the time of simulation.|
+|Country|The user's Country property value in Microsoft Entra ID at the time of simulation.|
+|Manager|The user's Manager property value in Microsoft Entra ID at the time of simulation.|
+
+How user activity signals are captured is described in the following table.
+
+|Field|Description|Calculation logic|
+|---|---|---|
+|DownloadAttachment|A user downloaded the attachment.|The signal comes from the client (for example, Outlook or Word).|
+|Opened Attachment|A user opened the attachment.|The signal comes from the client (for example, Outlook or Word).|
+|Read Message|The user read the simulation message.|Message read signals might experience issues in the following scenarios: <ul><li>The user reported the message as phishing in Outlook without leaving the reading pane, and **Mark items as read when viewed in the Reading Pane** wasn't configured (default).</li><li>The user reported the unread message as phishing in Outlook, the message was deleted, and **Mark messages as read when deleted** wasn't configured (default).</li></ul>|
+|Out of Office|Determines whether the user is out of office.|Currently calculated by the Automatic replies setting from Outlook.|
+|Compromised User|Indicates if a user been compromised. The compromise signals can vary based on the attack type.|<ul><li>**Credential Harvest**: The user enters their credentials in the login page (credentials aren't stored by Microsoft).</li><li>**Malware Attachment**: The user opens the file and enables editing in protected view.</li><li>**Link in attachment**: The user opens the attachment, and clicks on the link.</li><li>**Link to Malware**: The user clicks on the link and enters their credentials.</li><li>**Drive by URL**: The user clicks on the link (entering credentials isn't required).</li><li>**OAuth**: The user clicks on the link and accepts to share permissions.</li></ul>|
+|Clicked Message Link|Indicates if a user clicked on the message .|The URL in the simulation is unique for each user, which allows individual user activity tracking. Third-party filtering services or email forwarding can lead to false positives. For more information, see [I see clicks or compromise events from users who insist they didn't click the link in the simulation message](attack-simulation-training-faq.md#i-see-clicks-or-compromise-events-from-users-who-insist-they-didnt-click-the-link-in-the-simulation-message).|
+|Forwarded Message|Indicates if a user forwarded on the message .||
+|Replied to Message|Indicates if an end users has replied on the message.||
+|Deleted message|Indicates if an end users has deleted the message.|The signal comes from the Outlook activity of the user. If the user reports the message as phishing, the message might be moved to the Deleted Items folder, which is identified as a deletion.|
+|Permissions granted|Indicates if a user shared permissions in an Oauth-based attack.||
+
 ## Related Links
 
 [Get started using Attack simulation training](attack-simulation-training-get-started.md)

defender-office-365/attack-simulation-training-login-pages.md

@@ -121,6 +121,13 @@ When you select a login page from the list by clicking anywhere in the row other
 
      - **Code** tab: You can view and modify the HTML code directly.
 
+       > [!TIP]
+       > To avoid sending passwords in plain text from custom login pages, avoid using the variable **name** in HTML code. Instead, use **type**, **id**, or **class**. For example:
+       >
+       > ```html
+       > <input id="input-field-loginPage" type="password" placeholder="Password">
+       > ```
+
    You can preview the results by clicking the **Preview email** button at the top of the page.
 
    When you're finished on the **Review login page** page, select **Next**.

defender-office-365/attack-simulation-training-payloads.md

@@ -13,7 +13,7 @@ ms.collection:
 ms.custom:
 description: Admins can learn how to create and manage payloads for Attack simulation training in Microsoft Defender for Office 365 Plan 2.
 search.appverid: met150
-ms.date: 06/12/2024
+ms.date: 06/14/2024
 appliesto:
   - ✅ <a href="https://learn.microsoft.com/defender-office-365/mdo-about#defender-for-office-365-plan-1-vs-plan-2-cheat-sheet" target="_blank">Microsoft Defender for Office 365 Plan 2</a>
 ---
@@ -377,15 +377,15 @@ The create payload wizard opens with the settings and values of the selected pay
 
 ## Archive payloads
 
-You can't delete custom payloads from the **Tenant payloads** tab, but you can archive them.
-
 To archive an existing payload on the **Tenant payloads** tab, select the payload by clicking **⋮** (**Actions**) at the end of the row, and then select :::image type="icon" source="media/m365-cc-sc-archive-icon.png" border="false"::: **Archive**.
 
 The **Status** value of the payload changes to **Archive**, and the payload is no longer visible on the **Tenant payloads** table when **Show archived payloads** is toggled off :::image type="icon" source="media/scc-toggle-off.png" border="false":::.
 
 To see archived payloads on the **Tenant payloads** tab, toggle **Show archived payloads** to on :::image type="icon" source="media/scc-toggle-on.png" border="false":::.
 
-## Restore archived payloads
+After you archive a payload, you can restore it or remove it as described in the following subsections.
+
+### Restore archived payloads
 
 To restore an archive payload on the **Tenant payloads** tab, do the following steps:
 
@@ -394,6 +394,13 @@ To restore an archive payload on the **Tenant payloads** tab, do the following s
 
 After you restore the archived payload, the **Status** value changes to **Draft**. Toggle **Show archived payloads** to off :::image type="icon" source="media/scc-toggle-off.png" border="false"::: to see the restored payload. To return the payload to the **Status** value **Ready**, [edit the payload](#modify-payloads), review or change the settings, and then select **Submit**.
 
+### Remove archived payloads
+
+To remove an archived payload from the **Tenant payloads** tab, do the following steps:
+
+1. Set the **Show archived payloads** toggle to on :::image type="icon" source="media/scc-toggle-on.png" border="false":::.
+2. Select the payload by clicking **⋮** (**Actions**) at the end of the row, select :::image type="icon" source="media/m365-cc-sc-delete-icon.png" border="false"::: **Delete**, and then select **Confirm** in the confirmation dialog.
+
 ## Send a test
 
 On the **Tenant payloads** or **Global payloads** tabs, you can send a copy of the payload email to yourself (the currently logged in user) for inspection.