--- title: "AI CTRL Prompt Monitoring and Alerting" slug: "ai-ctrl-prompt-monitoring-and-alerting" updated: 2026-08-04T15:15:25Z published: 2026-08-04T15:15:25Z canonical: "kb.expedient.com/ai-ctrl-prompt-monitoring-and-alerting" --- > ## Documentation Index > Fetch the complete documentation index at: https://kb.expedient.com/llms.txt > Use this file to discover all available pages before exploring further. # AI CTRL Prompt Monitoring and Alerting # KB \- AI Prompt Monitoring NEW ## Overview AI CTRL gives you full visibility into how your organization uses AI, including the prompts your users submit. **Prompt monitoring alerts** build on that visibility by automatically notifying your team whenever a prompt contains content you care about, such as a sensitive keyword, a confidential phrase, or personally identifiable information (PII). The alert creation process described in this article takes place in the Compliance Platform (Elastic), which is the part of AI CTRL that provides security monitoring and detection capabilities. This article explains how to create a prompt monitoring alert in your AI CTRL Elastic instance, from defining what to watch for through choosing where notifications are delivered. The examples use a PII alert as a reference, but the same steps apply to any word or phrase you want to monitor. ### How alerts are generated Alerts are generated on a per-prompt basis, giving you an accurate, prompt-by-prompt record of matching activity: 1. When a user submits a prompt that matches your criteria, one alert is created. 2. If the same user submits another matching prompt later in the same conversation, a separate alert is created. 3. Each alert captures the user, the prompt text, and the time it occurred. ### What You Can Monitor You can alert on a single word, an exact phrase, a partial match, or several terms at once. The table below shows how to write each option. | | | | | --- | --- | --- | | **What you want to match** | **How to write it** | **Example** | | A single word | No quotes | confidential | | An exact phrase | Double quotes | "trade secret" | | A partial match | Asterisks around the term | \*contract\* | | Several words at once | OR inside parentheses | ("word1" OR "word2") | | Personally identifiable information (PII) | Matches the redaction marker | *\**REDACTED\* | #### Matching best practices - Matching is not case sensitive, so you do not need to worry about capitalization. - A plain word matches whole words only. For example, confidential will not match *confidentiality*. Use the partial match form (\*confidential\*) when you want to catch variations. ## Creating an Alert From the menu in the top left of Kibana, go to **Security → Rules → Detection rules (SIEM)**. Select **Create rule** in the top right, then choose **Manual rule creation**. The rule is set up across the five short steps below. ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-MD9CP8NA.png){height="" width=""} ### Step 1: Create a new rule On the create rule page, make sure **Custom query** is selected as the rule type. This is the default and is all that is needed for prompt monitoring. ### Step 2: Define what to monitor Enter the settings below. The only part you change for each alert is the keyword in the query. | | | | --- | --- | | **Setting** | **Value** | | Source | Index Patterns | | Index pattern | .ds-prompt\_data-\* | | Suppress alerts by | message.id | | If a suppression field is missing | Do not suppress alerts for events with missing fields | | Required fields | message.id , user\_name, request\_text | For the **Custom query**, use the following. Replace \ with your own word or phrase, using the formats from *What You Can Monitor* above: #### Ready-to-use queries Copy the query that matches your use case, then replace the keyword with your own. Keep the rest of the query exactly as shown. | | | | --- | --- | | **Use case** | **Full custom query** | | **Single word** | request\_text: confidential AND NOT request\_text: "### Task" AND NOT user\_name: "api\_key\_user" | | **Exact phrase** | request\_text: "trade secret" AND NOT request\_text: "### Task" AND NOT user\_name: "api\_key\_user" | | **Partial match** | request\_text: \*contract\* AND NOT request\_text: "### Task" AND NOT user\_name: "api\_key\_user" | | **Several words** | request\_text: ("confidential" OR "restricted") AND NOT request\_text: "### Task" AND NOT user\_name: "api\_key\_user" | | | | --- | | **Always keep the ending of the query**Keep AND NOT request\_text: "### Task" AND NOT user\_name: "api\_key\_user" at the end of every query, no matter which keyword you use. This filters out automated system traffic and prevents duplicate alerts. | ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-7ETOGG1V.png){height="" width=""} ### Step 3: Name and describe the alert Give the alert a clear name and description so anyone reviewing it later understands its purpose, then set a severity. - **Name**: use something descriptive, for example PII Monitoring or Confidential Keyword Alert. - **Description**: state plainly what the alert detects, for example “Generates an alert when a prompt contains PII.” - **Severity**: set this to reflect how important the alert is. For a PII alert, **High** is recommended. A suggested approach: | | | | --- | --- | | **Type of alert** | **Suggested severity** | | PII or other sensitive data | High | | Compliance or policy keyword | Medium | | General keyword monitoring | Low to Medium | | | | --- | | **Tip**Setting every alert to High makes it harder to tell which alerts truly need urgent attention. Matching severity to the type of content keeps your most important alerts easy to spot. | ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-XFTQHLRF.png){height="" width=""} ### Step 4: Set the schedule The schedule controls how often the alert checks for matching prompts. | | | | --- | --- | | **Setting** | **Value** | | Runs every | 1 minute | | Additional look-back time | 1 minute | ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-W08GATA0.png){height="" width=""} | | | --- | | **How the schedule works**The alert runs once every minute and checks for new prompts that match your query. A one minute look-back is added to each run so prompts are not missed if there is a brief delay in indexing. You will not receive duplicate alerts, because each prompt is only ever alerted on once. | ### Step 5: Choose where alerts are sent Decide how your team is notified when an alert fires. Microsoft Teams is the most common choice, but you can select any connector type your organization uses. 1. Under **Select a connector type**, choose your destination, for example Microsoft Teams. ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-RMFOU4F5.png){height="" width=""} 2. Set the notification frequency to **For each alert** so every matching prompt sends a notification. ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-0O3YKGPS.png){height="" width=""} 3. Add a short message so notifications are easy to read at a glance. A simple starting point: | | | --- | | AI CTRL alert: {{context.rule.name}} | | User: {{context.alerts.0.user\_name}} | | Prompt: {{context.alerts.0.request\_text}} | ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-Z4ZJ353U.png){height="" width=""} | | | --- | | **Tip**Use the *Add variable* button in the message editor to insert the correct fields (such as user and prompt text). | #### Preview and save Before saving, select **Rule preview** in the top right to check your alert against recent activity. This confirms it matches what you expect and helps catch any typos in the query. When you are satisfied, save the rule to make it active. # Testing Your Alert 1. In your AI CTRL chat interface, enter a prompt that should trigger the alert. For a PII alert, use a prompt such as “Tell me about John Doe 900-12-3456”. For a keyword alert, enter a prompt containing your keyword. ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-H0TC1TYJ.png){height="" width=""} 2. In Kibana, go to **Security → Alerts** and confirm the alert appears. ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-UR3JZBVE.png){height="" width=""} 3. Check your notification destination (for example, your Teams channel) to confirm the message was delivered and the details look correct. # Reviewing prompts in Elastic Elastic gives you the ability to review prompt and response logs and can give a detailed view into users usage. To view the logs follow the steps below 1. From Kibana, select the hamburger menu on the top left. 2. Look for the ‘Analytics’ section and choose ‘Dashboards’ ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-IX2E5VLS.png){height="" width=""} 3. This will open the Dashboards menu. From here select the Dashboard titled, ‘AI-Gateway Monitoring’ (as showed in the picture below) ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-J3K0UU7B.png){height="" width=""} 4. Once the Dashboard loads, you will see graphs that shows model usage and token usage. 5. One thing to note here is that the information can be filtered by any time frame. For example if you wanted to view the Cumulative Token Usage or prompt/response history for the last 30 days, select the calendar on the top right of the screen and click ‘Last 30 days’. The information will refresh and show only stats inside of the set timeframe. ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-MT1RIWB4.png){height="" width=""} 6. To view prompt and response details, scroll down until the ‘request and response detail’ board is shown. This is where you can confirm that the PII has been redacted. It will also give a timestamp, user name, model used and both the input and response texts. ![Image](https://cdn.document360.io/69e9f9c7-5da8-45e7-a671-4b8287f36122/Images/Documentation/image-A75VYNPK.png){height="" width=""} | | | --- | | **Note**The AI may decline to answer a prompt that contains sensitive information. The alert is still generated, because monitoring is based on the prompt the user submitted, not on the AI's response. | # Managing Your Alerts - **Review alerts** at any time under **Security → Alerts** in Kibana. - **Reduce noise**: if an alert is firing on prompts you do not need, refine the keyword to be more specific rather than turning the alert off. - **Pause monitoring**: to stop an alert temporarily, disable it rather than deleting it, so its settings are preserved for later. **Getting Help** If you have questions about creating or tuning prompt monitoring alerts, or you would like Expedient to set one up for you, [open a support ticket](https://support.expedient.com/) through the Support Management Console, or contact your Expedient Account Manager.