Skip to content

feat: Support stt mode auto send message #1977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
liuruibin merged 1 commit into from
Jan 6, 2025
Merged

feat: Support stt mode auto send message #1977

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions apps/application/migrations/0023_application_stt_autosend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Generated by Django 4.2.15 on 2025-01-06 10:37

from django.db import migrations, models


class Migration(migrations.Migration):

dependencies = [
('application', '0022_application_tts_autoplay'),
]

operations = [
migrations.AddField(
model_name='application',
name='stt_autosend',
field=models.BooleanField(default=False, verbose_name='自动发送'),
),
]
1 change: 1 addition & 0 deletions apps/application/models/application.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ class Application(AppModelMixin):
stt_model_enable = models.BooleanField(verbose_name="语音识别模型是否启用", default=False)
tts_type = models.CharField(verbose_name="语音播放类型", max_length=20, default="BROWSER")
tts_autoplay = models.BooleanField(verbose_name="自动播放", default=False)
stt_autosend = models.BooleanField(verbose_name="自动发送", default=False)
clean_time = models.IntegerField(verbose_name="清理时间", default=180)
file_upload_enable = models.BooleanField(verbose_name="文件上传是否启用", default=False)
file_upload_setting = models.JSONField(verbose_name="文件上传相关设置", default=dict)
Expand Down
4 changes: 3 additions & 1 deletion apps/application/serializers/application_serializers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1009,7 +1009,7 @@ def edit(self, instance: Dict, with_valid=True):
update_keys = ['name', 'desc', 'model_id', 'multiple_rounds_dialogue', 'prologue', 'status',
'dataset_setting', 'model_setting', 'problem_optimization', 'dialogue_number',
'stt_model_id', 'tts_model_id', 'tts_model_enable', 'stt_model_enable', 'tts_type',
'tts_autoplay', 'file_upload_enable', 'file_upload_setting',
'tts_autoplay', 'stt_autosend', 'file_upload_enable', 'file_upload_setting',
'api_key_is_active', 'icon', 'work_flow', 'model_params_setting', 'tts_model_params_setting',
'problem_optimization_prompt', 'clean_time']
for update_key in update_keys:
Expand Down Expand Up @@ -1075,6 +1075,8 @@ def get_work_flow_model(instance):
instance['tts_type'] = node_data['tts_type']
if 'tts_autoplay' in node_data:
instance['tts_autoplay'] = node_data['tts_autoplay']
if 'stt_autosend' in node_data:
instance['stt_autosend'] = node_data['stt_autosend']
if 'tts_model_params_setting' in node_data:
instance['tts_model_params_setting'] = node_data['tts_model_params_setting']
if 'file_upload_enable' in node_data:
Expand Down
1 change: 1 addition & 0 deletions ui/src/api/type/application.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ interface ApplicationFormType {
tts_model_enable?: boolean
tts_type?: string
tts_autoplay?: boolean
stt_autosend?: boolean
}
interface Chunk {
real_node_id: string
Expand Down
39 changes: 24 additions & 15 deletions ui/src/components/ai-chat/component/chat-input-operate/index.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@
</div>
</template>
<script setup lang="ts">
import { ref, computed, onMounted } from 'vue'
import { ref, computed, onMounted, nextTick } from 'vue'
import Recorder from 'recorder-core'
import applicationApi from '@/api/application'
import { MsgAlert } from '@/utils/message'
Expand Down Expand Up @@ -254,7 +254,7 @@ const getAcceptList = () => {
if (video) {
accepts = [...accepts, ...videoExtensions]
}
// console.log(accepts)

if (accepts.length === 0) {
return '.请在文件上传配置中选择文件类型'
}
Expand Down Expand Up @@ -462,7 +462,12 @@ const uploadRecording = async (audioBlob: Blob) => {
recorderLoading.value = false
mediaRecorder.value.close()
inputValue.value = typeof response.data === 'string' ? response.data : ''
// chatMessage(null, res.data)
// 自动发送
if (props.applicationDetails.stt_autosend) {
nextTick(() => {
autoSendMessage()
})
}
})
} catch (error) {
recorderLoading.value = false
Expand All @@ -486,24 +491,28 @@ const handleTimeChange = () => {
}, 1000)
}

function autoSendMessage() {
props.sendMessage(inputValue.value, {
image_list: uploadImageList.value,
document_list: uploadDocumentList.value,
audio_list: uploadAudioList.value,
video_list: uploadVideoList.value
})
inputValue.value = ''
uploadImageList.value = []
uploadDocumentList.value = []
uploadAudioList.value = []
uploadVideoList.value = []
quickInputRef.value.textareaStyle.height = '45px'
}

function sendChatHandle(event?: any) {
if (!event?.ctrlKey) {
// 如果没有按下组合键ctrl，则会阻止默认事件
event?.preventDefault()
if (!isDisabledChart.value && !props.loading && !event?.isComposing) {
if (inputValue.value.trim()) {
props.sendMessage(inputValue.value, {
image_list: uploadImageList.value,
document_list: uploadDocumentList.value,
audio_list: uploadAudioList.value,
video_list: uploadVideoList.value
})
inputValue.value = ''
uploadImageList.value = []
uploadDocumentList.value = []
uploadAudioList.value = []
uploadVideoList.value = []
quickInputRef.value.textareaStyle.height = '45px'
autoSendMessage()
}
}
} else {
Copy link
Contributor Author

@shaohuzhang1 shaohuzhang1 Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code seems to be a Vue.js component that handles file uploads, real-time recording of voice messages, and sending text-based messages with attachments. Here are some observations, potential issues, and optimization suggestions:

Observations:

  1. Import Statements: The nextTick utility is used multiple times, which can optimize rendering by ensuring that DOM updates occur after certain operations have been completed.
  2. Auto-Message Sending: The autoSendMessage function is called when the user hits the Enter key (after disabling the default behavior), but it could benefit from more robust logic, such as checking if the message is not empty before proceeding.
  3. Input Value Resetting: After uploading messages or files, there's no explicit reset strategy for input fields like inputValue, uploadImageList, etc., especially during automated sends. This might lead to unexpected state management.
  4. Template Structure: The template structure looks correct, but ensure all components (RecordPlayer.vue, quick-input-ref) are properly imported and referenced.

Potential Issues:

  1. Data Management: There's no clear separation between data states (e.g., inputValue, upload*List) and computed properties, which can make it harder to debug and maintain.
  2. Error Handling: Although error handling exists, the responses should include meaningful feedback to users, possibly using the existing MsgAlert utility.
  3. Performance Optimization: Ensure that critical operations do not block the main thread (such as API calls, DOM updates) unnecessarily.

Optimization Suggestions:

  1. State Reconciliation: Implement a consistent approach for resetting UI elements after actions like message sending. For example, use Vuex or another state management library to keep track of form inputs and output lists.
  2. Lazy Load Components: If RecordPlayer.vue is complex or heavy, consider lazy loading it only when needed.
  3. Improve Feedback Mechanisms: Enhance the messaging system to provide immediate visual feedback (like success animations) or toast notifications about successful actions.
  4. Code Refactoring: Consider breaking down functions into smaller, more manageable pieces, adhering to principles like Single Responsibility Principle (SRP).
  5. Type Safety: While TypeScript types seem present, ensure they cover all necessary scenarios where errors can occur (e.g., incorrect file formats).

Overall, the code appears functional and complete, but refining its structure, performance, and clarity will enhance its sustainability and reliability. Let me know if you need further assistance!

Expand Down
23 changes: 9 additions & 14 deletions ui/src/views/application/ApplicationSetting.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,21 +319,16 @@
<el-form-item>
<template #label>
<div class="flex-between">
<div class="flex align-center">
<span class="mr-4">语音输入</span>
<!-- <el-tooltip
effect="dark"
content="开启后，需要设定语音转文本模型，语音输入完成后会转化为文字直接发送提问"
placement="right"
>
<AppIcon iconName="app-warning" class="app-warning-icon"></AppIcon>
</el-tooltip> -->
<span class="mr-4">语音输入</span>
<div class="flex">
<el-checkbox v-model="applicationForm.stt_autosend">自动发送</el-checkbox>
<el-switch
class="ml-8"
size="small"
v-model="applicationForm.stt_model_enable"
@change="sttModelEnableChange"
/>
</div>
<el-switch
size="small"
v-model="applicationForm.stt_model_enable"
@change="sttModelEnableChange"
/>
</div>
</template>
<el-select
Copy link
Contributor Author

@shaohuzhang1 shaohuzhang1 Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code has several issues:

  1. Duplicate el-tooltip component: There seems to be an issue with the <el-tooltip> component where it appears multiple times within the same template slot.

  2. Unnecessary spacing: There is extra space around certain elements, which could affect layout readability.

  3. Inconsistent formatting: The HTML structure and CSS classes are inconsistent throughout the file. It's important to maintain a consistent style for better readability and maintainability.

  4. Unused import statement: There is an unused import import AppIcon from '@/components/comm/AppIcon.vue'; at the top of the file. This should be removed since it's not being used anywhere.

Here’s a suggested optimized version of the code:

<template>
  <el-form-item :label-width="formLabelWidth">
    <div class="flex-between">
      <span class="mr-4">语音输入</span>
      <div class="flex">
        <el-checkbox v-model="applicationForm.stt_autosend">自动发送</el-checkbox>
        <el-switch 
          class="ml-8"
          size="small"
          v-model="applicationForm.stt_model_enable"
          @change="sttModelEnableChange"
        ></el-switch>
      </div>
    </div>
    <!-- Other form-items... -->
  </el-form-item>
</template>

<script>
export default {
  // ...
};
</script>

<style scoped>
.flex-between {
  display: flex;
  justify-content: space-between;
}

.el-checkbox {
  margin-right: 8px; /* Adjusted margin */
}
</style>

Optimization Suggested Points:

  • Consistent Class Names: Use the .flex-between class consistently across similar items.
  • Remove Unused Imports: Review imports and ensure all components are correctly imported and used.
  • Use Template Slots Properly: If necessary, move tooltip logic outside the item label if it doesn't fit well there.
  • Adjust Styles: Ensure styles like padding, margins, and font sizes are appropriate for your design requirements.

By applying these changes, the code becomes more uniform, easier to read, and maintains consistency in its styling.

Expand Down
23 changes: 9 additions & 14 deletions ui/src/workflow/nodes/base-node/index.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,21 +84,16 @@
<el-form-item>
<template #label>
<div class="flex-between">
<div class="flex align-center">
<span class="mr-4">语音输入</span>
<!-- <el-tooltip
effect="dark"
content="开启后，需要设定语音转文本模型，语音输入完成后会转化为文字直接发送提问"
placement="right"
>
<AppIcon iconName="app-warning" class="app-warning-icon"></AppIcon>
</el-tooltip> -->
<span class="mr-4">语音输入</span>
<div class="flex">
<el-checkbox v-model="form_data.stt_autosend">自动发送</el-checkbox>
<el-switch
class="ml-8"
size="small"
v-model="form_data.stt_model_enable"
@change="sttModelEnableChange"
/>
</div>
<el-switch
size="small"
v-model="form_data.stt_model_enable"
@change="sttModelEnableChange"
/>
</div>
</template>

Copy link
Contributor Author

@shaohuzhang1 shaohuzhang1 Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code you provided has a few issues:

  1. Redundant <template #label> Content: The #label template slot now includes the auto-send checkbox without any separators or explanations.

  2. Conflicting Properties: There appears to be a mix-up with properties (form_data.stt_autosend and v-model="form_data.stt_model_enable"). These should either use one of them or separate them appropriately.

  3. Improper Structure: The checkboxes and switches may not behave as expected due to the lack of proper structure within their containers.

Here's an optimized version of the code:

<template>
  <el-form-item>
    <template #label>
      <div class="flex-between">
        <p>
          <span class="mr-4">语音输入</span>
          <!-- Add description here if needed -->
        </p>

        <div class="flex align-center">
          <el-checkbox v-model="form_data.stt_autosend">{{ $t('common.auto_send') }}</el-checkbox>
          
          <el-switch
            class="ml-8"
            style="--el-slider-button-size: 0;"
            :active-text="$t('settings.enable_stt')"
            size="small"
            v-model="form_data.stt_model_enable"
            @change="sttModelEnableChange"
          />
        </div>

      </div>
    </template>
    <!-- Rest of form item elements -->
  </el-form-item>
</template>

<script>
export default {
  data() {
    return {
      form_data: { ... } // Initialize your form_data object here
    };
  },
  methods: {
    sttModelEnableChange(enabled) {
      this.$emit('update:model-value', { ...this.form_data, stt_enabled: enabled });
      this.handleSttInput(enabled);
    },
    handleSttInput(autostart) {
      // Implement logic to enable/disable automatic start based on autostart value
    }
  }
};
</script>

Key Changes:

  • Simplified label template for better readability and maintainability.
  • Removed redundant switch element.
  • Added tooltips using i18n for translations.
  • Ensured consistent spacing and alignment in the label area.
  • Used appropriate HTML tags and attributes to improve SEO performance.

These changes should resolve most issues while maintaining clear presentation and functionality. Make sure to adjust the labels and placeholder text according to your application requirements.

Expand Down
Loading