gavin/Fay
1
0
Fork 0
mirror of https://github.com/xszyou/Fay.git synced 2026-03-12 17:51:28 +08:00
Code Issues Packages Projects Releases Wiki Activity

年番更新

Browse Source 
1、ui显示deepseek 思考窗口；
2、提供Fay ai编程工具二开指南：https://qqk9ntwbcit.feishu.cn/wiki/FKFywXWaeiBH28k4Q67c3eF7njC
3、修复使用gpt stream是声音重复合成播放的bug;
4、优化<think>标签内容处理逻辑，只在显示上和声音输出上做处理；
5、去除开启自动运行脚本功能，提高linux和mac的兼容性；
6、优化前端自动获取后端接口地址的方式；
7、agent 模式兼容deepseek。https://qqk9ntwbcit.feishu.cn/wiki/WLg5wde5di5ACqkUu6IcD4w7n0e
This commit is contained in:
xszyou
2025-02-20 00:22:51 +08:00
parent 44982f0ea1
commit d70547d70f
11 changed files with 333 additions and 167 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
aidev.txt Normal file
View File

@@ -0,0 +1,124 @@
此文件是Fay框架项目二次开发约定，供程序员和ai 编程工具阅读。
一、Fay框架的作用
1、实现简单的数字人交互
这种实现方式最为简单，在Fay框架基础上，你只需要考虑用什么皮肤和llm,以及在什么地方接收用户的信息，然后又把信息输出到那里。
a、接收的信息
接收的信息可以是文本信息（flask_server.py中实现），也可以是语音信息(fay_booter.py中实现)。可以从本机接收，也可以从网络接收。
b、输出的信息
输出的信息可以是文本信息（flask_server.py中实现），也可以是语音信息(fay_booter.py中实现)。可以向本机输出，也可以向网络输出。
c、皮肤
Fay框架通过数字人接口来驱动皮肤，若用户想驱动自己的皮肤，可以对接咱们定义的数字人接口（wsa_server.py,10002端口）。
d、llm
在llm目录下我们已经为大量的llm实现了对接，只需要在system.conf配置chat_module即可，若用户需要补充，可以参考nlp_gpt.py的实现，并在fay_core.py中引入。
2、为数字人提供“眼睛”
我们将智能硬件、摄像头、手机等外部终端设备视为眼睛。终端设备把识别的数据和文本交互信息一起发给Fay（flask_server.py中api_send_v1_chat_completions()方法的observation），Fay根据数据进行处理及回应。更复杂的情况可以参照flask_server.py中的打招呼或唤醒补充其他意图接口。
3、为数字人提供“手”
通过把chat_module配置为agent,并参照llm/agent/tools/下的工具的实现补充更丰富的工具，可以让数字人的手为我们工作。
4、调度三方系统
构建以数字人为中心的高度智能系统，只需要“让眼看见，让手触达”。
二、Fay框架的使用原则
1、首先明确用户使用Fay框架的目的。
2、然后评估Fay的功能是否已经满足用户的需求。
3、最后再考虑怎么样做最少的修改来达到用户的目的。
4、只实现用户最直接需求，不要给过多的建议。
5、请尽量遵照原工程的编码规范。
6、如果你不知道用户目的，请先问清楚。
7、遵循 PEP 8 编码规范，注重提高代码可读性和一致性​。
8、请说中文，不要说英文。
三、Fay框架的源码结构
├── main.py # 主程序入口
├── fay_booter.py # 启动器，管理录音设备和系统初始化
├── config.json # 配置文件
├── system.conf # 系统配置
├── requirements.txt # 依赖包列表
├── fay.db # 主数据库
├── timer.db # 定时任务数据库
├── user_profiles.db # 用户配置数据库
├── qa.csv # QA问答数据
├── verifier.json # 验证配置
│
├── core/ # 数字人核心目录
│ ├── fay_core.py # 核心控制器，处理交互逻辑和TTS
│ ├── recorder.py # 录音模块，音频采集和VAD检测
│ ├── wsa_server.py # WebSocket服务，处理实时通信
│ ├── qa_service.py # QA服务，处理问答逻辑
│ ├── interact.py # 交互基类，定义交互接口
│ ├── content_db.py # 内容数据库管理
│ ├── member_db.py # 用户数据库管理
│ ├── authorize_tb.py # 认证表管理
│ └── socket_bridge_service.py # Socket桥接服务
│
├── asr/ # 语音识别模块
│ ├── ali_nls.py # 阿里云语音识别
│ ├── funasr.py # FunASR语音识别
│ └── funasr/ # FunASR服务器相关文件
│
├── tts/ # 语音合成模块
│ ├── ms_tts_sdk.py # 微软TTS实现
│ ├── ali_tss.py # 阿里云TTS实现
│ ├── gptsovits.py # GPT-SoVITS实现
│ ├── gptsovits_v3.py # GPT-SoVITS V3实现
│ ├── volcano_tts.py # 火山TTS实现
│ └── tts_voice.py # 语音类型和风格定义
│
├── llm/ # 大语言模型模块
│ ├── agent/ # AI代理目录
│ ├── nlp_gpt.py # ChatGPT实现
│ ├── nlp_rasa.py # Rasa实现
│ ├── nlp_lingju.py # 灵聚AI实现
│ ├── nlp_xingchen.py # 星尘AI实现
│ ├── nlp_coze.py # Coze AI实现
│ ├── nlp_qingliu.py # 清流AI实现
│ ├── nlp_accompany.py # 陪伴AI实现
│ ├── nlp_ChatGLM3.py # ChatGLM3实现
│ ├── nlp_VisualGLM.py # VisualGLM实现
│ ├── nlp_ollama_api.py # Ollama API实现
│ ├── nlp_privategpt.py # Private GPT实现
│ ├── nlp_rwkv.py # RWKV实现
│ └── VllmGPT.py # vLLM GPT实现
│
├── ai_module/ # AI功能模块
│ ├── baidu_emotion.py # 百度情感分析
│ └── nlp_cemotion.py # 自定义情感分析
│
├── gui/ # 图形界面模块
│ ├── flask_server.py # Flask Web服务器
│ ├── window.py # 窗口程序实现
│ ├── robot/ # 机器人资源目录
│ ├── static/ # 静态资源目录
│ └── templates/ # 页面模板目录
│
├── utils/ # 工具模块
│ ├── config_util.py # 配置管理工具
│ ├── util.py # 通用工具函数
│ ├── stream_util.py # 流处理工具
│ └── openai_api/ # OpenAI API工具
│
├── scheduler/ # 调度管理
│ └── thread_manager.py # 线程管理器
│
├── cache_data/ # 缓存数据目录
│ └── input.wav # 临时音频文件
│
├── samples/ # 音频样本目录
│ └── sample-*.wav # TTS生成的音频文件
│
├── logs/ # 日志目录
│ ├── asr_result.txt # 语音识别结果日志
│ └── answer_result.txt # 回答结果日志
│
├── docker/ # Docker相关
│ └── Dockerfile # Docker配置文件
│
├── shell/ # Shell脚本
│ └── *.sh # 各种脚本文件
│
└── test/ # 测试目录
└── * # 测试文件和资源

7
core/fay_core.py
View File

@@ -204,8 +204,11 @@ class FeiFei:
content = {'Topic': 'Unreal', 'Data': {'Key': 'text', 'Value': text}, 'Username' : username, 'robot': f'http://{cfg.fay_url}:5000/robot/Speaking.jpg'}
wsa_server.get_instance().add_cmd(content)
#声音输出
MyThread(target=self.say, args=[interact, text]).start()
#声音输出(gpt_stream在stream_manager.py中调用了say函数)
if cfg.key_chat_module != 'gpt_stream':
if text in "</think>":
text = text.split("</think>")[1]
MyThread(target=self.say, args=[interact, text]).start()
return text

15
fay_booter.py
View File

@@ -288,14 +288,6 @@ def stop():
global socket_service_instance
global deviceSocketServer
#停止外部应用
if os.name == 'nt':
util.log(1, '停止外部应用...')
startup_script = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'shell', 'run_startup.py')
if os.path.exists(startup_script):
from shell.run_startup import stop_all_processes
stop_all_processes()
util.log(1, '正在关闭服务...')
__running = False
if recorderListener is not None:
@@ -330,13 +322,6 @@ def start():
global recorderListener
global __running
global socket_service_instance
#启动外部应用
util.log(1,'启动外部应用...')
startup_script = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'shell', 'run_startup.py')
if os.path.exists(startup_script):
subprocess.Popen([sys.executable, startup_script],
creationflags=subprocess.CREATE_NEW_CONSOLE)
util.log(1, '开启服务...')
__running = True

59
gui/static/css/index.css
View File

@@ -1,4 +1,3 @@
html {
font-size: 14px;
}
@@ -365,4 +364,60 @@ html {
border-radius: 4px;
font-size: 12px;
white-space: nowrap;
}
}
.think-panel-container {
position: fixed;
top: 100px; /* 保证不与顶部导航栏重叠 */
right: 120px;
width: 250px;
height: auto;
padding: 10px;
background: #fff;
border: 1px solid #ccc;
border-radius: 4px;
z-index: 1000;
}
.think-panel {
margin-top: 10px;
padding: 10px;
background: #f9f9f9;
border: 1px dashed #ccc;
border-radius: 4px;
max-height: 620px;
overflow-y: auto;
white-space: pre-wrap; /* 保持原始格式 */
}
.think-panel-header {
display: flex;
justify-content: space-between;
align-items: center;
padding-right: 10px;
}
.think-panel-minimize {
background: none;
border: none;
color: #617bab;
cursor: pointer;
padding: 5px;
line-height: 1;
display: flex;
align-items: center;
justify-content: center;
}
.think-panel-minimize:hover {
color: #0064fb;
}
.think-panel-icon {
font-size: 16px;
display: inline-block;
line-height: 1;
}
.think-panel.minimized {
height: 40px;
overflow: hidden;
}

72
gui/static/js/index.js
View File

@@ -38,12 +38,23 @@ class FayInterface {
async fetchData(url, options = {}) {
try {
// Ensure headers are properly set for POST requests
if (options.method === 'POST') {
options.headers = {
'Content-Type': 'application/json',
...options.headers
};
}
const response = await fetch(url, options);
if (!response.ok) throw new Error(`HTTP error! Status: ${response.status}`);
return await response.json();
if (!response.ok) {
throw new Error(`HTTP error! Status: ${response.status}`);
}
const data = await response.json();
return data;
} catch (error) {
console.error('Error fetching data:', error);
return null;
throw error; // Rethrow to handle in the calling function
}
}
@@ -246,25 +257,65 @@ new Vue({
panelMsg: '',
panelReply: '',
robot:'static/images/Normal.gif',
base_url: 'http://127.0.0.1:5000',
base_url: window.location.protocol + '//' + window.location.hostname + ':' + window.location.port,
play_sound_enabled: false,
source_record_enabled: false,
userListTimer: null, // 添加定时器变量
userListTimer: null,
thinkPanelExpanded: false,
thinkContent: '',
isThinkPanelMinimized: false,
};
},
watch: {
messages: {
handler(newMessages) {
for (let i = newMessages.length - 1; i >= 0; i--) {
let msg = newMessages[i];
if (msg.type === 'fay') {
const regex = /<think>([\s\S]*?)<\/think>/;
const match = msg.content.match(regex);
if (match && match[1]) {
this.thinkContent = match[1];
// 从原始消息中移除think标签及其内容，并去除多余空格
msg.content = msg.content.replace(regex, '').trim();
break;
}
}
}
},
deep: true
}
},
created() {
this.initFayService();
this.getData();
// 启动定时扫描用户列表
this.startUserListTimer();
},
methods: {
initFayService() {
this.fayService = new FayInterface('ws://127.0.0.1:10003', this.base_url, this);
const wsProtocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
const wsHost = window.location.hostname;
const wsUrl = `${wsProtocol}//${wsHost}:10003`;
this.fayService = new FayInterface(wsUrl, this.base_url, this);
this.fayService.connectWebSocket();
this.fayService.websocket.addEventListener('open', () => {
this.loadUserList();
});
});
},
async loadUserList() {
try {
const result = await this.fayService.getUserList();
if (result && result.list) {
this.userList = result.list;
if (this.userList.length > 0) {
this.selectedUser = this.userList[0];
await this.loadMessageHistory(this.selectedUser[1]);
}
}
} catch (error) {
console.error('Failed to load user list:', error);
this.$message.error('Failed to load user list. Please try again.');
}
},
sendMessage() {
let _this = this;
@@ -494,5 +545,10 @@ this.fayService.fetchData(`${this.base_url}/api/adopt_msg`, {
});
}
,
minimizeThinkPanel() {
this.isThinkPanelMinimized = !this.isThinkPanelMinimized;
const panel = document.querySelector('.think-panel');
panel.classList.toggle('minimized');
},
}
});

6
gui/static/js/setting.js
View File

@@ -172,7 +172,7 @@ new Vue({
}],
automatic_player_status: false,
automatic_player_url: "",
host_url: "http://127.0.0.1:5000"
host_url: window.location.protocol + '//' + window.location.hostname + ':' + window.location.port
};
},
created() {
@@ -181,7 +181,9 @@ new Vue({
},
methods: {
initFayService() {
this.fayService = new FayInterface('ws://127.0.0.1:10003', this.host_url, this);
const wsProtocol = window.location.protocol === 'https:' ? 'wss:' : 'ws';
const wsHost = window.location.hostname;
this.fayService = new FayInterface(`${wsProtocol}://${wsHost}:10003`, this.host_url, this);
this.fayService.connectWebSocket();
},
getData() {

137
gui/templates/index.html
View File

@@ -13,99 +13,118 @@
<script src="{{ url_for('static',filename='js/index.js') }}" defer></script>
<script src="{{ url_for('static',filename='js/script.js') }}" defer></script>
</head>
<body >
<div id="app" class="main_bg">
<div class="main_left">
<div class="main_left_logo" ><img src="{{ url_for('static',filename='images/Logo.png') }}" alt="">
<body>
<div id="app" class="main_bg">
<div class="main_left">
<div class="main_left_logo">
<img src="{{ url_for('static',filename='images/Logo.png') }}" alt="">
</div>
<div class="main_left_menu">
<ul>
<li class="changeImg"><a href="/"><span class="iconimg1">消息</span></a></li>
<li class="changeImg2"><a href="/setting"><span class="iconimg2">设置</span></a></li>
</ul>
</div>
<div class="main_left_emoji"><img style="padding-top: 60px; max-width: 140px;" :src="robot" alt="" >
<div class="main_left_menu">
<ul>
<li class="changeImg"><a href="/"><span class="iconimg1">消息</span></a></li>
<li class="changeImg2"><a href="/setting"><span class="iconimg2">设置</span></a></li>
</ul>
</div>
</div>
<div class="main_left_emoji">
<img style="padding-top: 60px; max-width: 140px;" :src="robot" alt="">
</div>
</div>
<div class="main_right">
<div class="top_info"><span class="top_info_text">消息：</span>[[panelMsg]]</div>
<!-- 以上是即时信息显示 -->
<div class="chatmessage">
<div class="chat-container" id="user0" >
<div v-for="(item, index) in messages" :key="index" >
<div class="top_info">
<span class="top_info_text">消息：</span>[[panelMsg]]
</div>
<div class="chatmessage">
<div class="chat-container" id="user0">
<div v-for="(item, index) in messages" :key="index">
<div class="message receiver-message" v-if="item.type == 'fay'">
<img class="avatar" src="{{ url_for('static',filename='images/Fay_send.png') }}" alt="接收者头像">
<div class="message-content">
<div class="message-bubble">[[item.content]]</div>
<div class="message-time"><span class="what-time">[[item.timetext]]</span>
<div class="message-time">
<span class="what-time">[[item.timetext]]</span>
<div v-if="item.is_adopted == 0" class="adopt-button" @click="adoptText(item.id)">
<img src="{{ url_for('static',filename='images/adopt.png') }}" alt="采纳图标" class="adopt-img" />
</div>
<div v-else class="adopt-button">
<img src="{{ url_for('static',filename='images/adopted.png') }}" alt="采纳图标" class="adopt-img" />
</div>
<img src="{{ url_for('static',filename='images/adopt.png') }}" alt="采纳图标" class="adopt-img" />
</div>
<div v-else class="adopt-button">
<img src="{{ url_for('static',filename='images/adopted.png') }}" alt="采纳图标" class="adopt-img" />
</div>
</div>
</div>
</div>
<div class="message sender-message" v-else>
<div class="message-content">
<div class="sender-message message-bubble">[[item.content]]</div>
<div class="sender-message-time">[[item.timetext]]</div>
<div class="message sender-message" v-else>
<div class="message-content">
<div class="sender-message message-bubble">[[item.content]]</div>
<div class="sender-message-time">[[item.timetext]]</div>
</div>
<img class="avatar" src="{{ url_for('static',filename='images/User_send.png') }}" alt="发送者头像">
</div>
<img class="avatar" src="{{ url_for('static',filename='images/User_send.png') }}" alt="发送者头像">
</div>
</div>
<div >
</div>
</div>
</div>
<!-- 以上是聊天对话 -->
<div class="inputmessage">
<div class="inputmessage_voice" >
<img v-if="!source_record_enabled" src="{{ url_for('static',filename='images/recording.png') }}" alt="" @click=changeRecord() >
<img v-else src="{{ url_for('static',filename='images/record.png') }}" alt="" @click=changeRecord() >
<div class="inputmessage_voice">
<img v-if="!source_record_enabled" src="{{ url_for('static',filename='images/recording.png') }}" alt="" @click="changeRecord()">
<img v-else src="{{ url_for('static',filename='images/record.png') }}" alt="" @click="changeRecord()">
</div>
<div class="inputmessage_input">
<textarea class="text_in" placeholder="请输入内容" v-model="newMessage" @keyup.enter="sendMessage" style="padding-top: 13px;"></textarea>
</div>
<div class="inputmessage_send">
<img src="{{ url_for('static',filename='images/send.png') }}" alt="发送信息" @click="sendMessage">
</div>
<div class="inputmessage_input"> <textarea class="text_in" placeholder="请输入内容" v-model="newMessage" @keyup.enter="sendMessage" style="padding-top: 13px;"></textarea></div>
<div class="inputmessage_send"><img src="{{ url_for('static',filename='images/send.png') }}" alt="发送信息" @click="sendMessage"></div>
<div v-if="liveState == 1" class="inputmessage_open">
<img v-if="!play_sound_enabled" src="{{ url_for('static',filename='images/sound_off.png') }}" @click=changeSound() >
<img v-else src="{{ url_for('static',filename='images/sound_on.png') }}" @click=changeSound() >
<img v-if="!play_sound_enabled" src="{{ url_for('static',filename='images/sound_off.png') }}" @click="changeSound()">
<img v-else src="{{ url_for('static',filename='images/sound_on.png') }}" @click="changeSound()">
</div>
<div v-else class="inputmessage_open">
<img src="{{ url_for('static',filename='images/open.png') }}" @click=startLive() >
<img src="{{ url_for('static',filename='images/open.png') }}" @click="startLive()">
</div>
</div>
<div class="Userchange">
<button id="prevButton" ><img src="{{ url_for('static',filename='images/scrollleft.png') }}" alt="向左滑动" ></button>
<button id="prevButton">
<img src="{{ url_for('static',filename='images/scrollleft.png') }}" alt="向左滑动">
</button>
<div class="menu" ref="menuContainer">
<div class="tag" v-for="user in userList" :key="user[0]" :class="{'selected': selectedUser && selectedUser[0] === user[0]}" @click="selectUser(user)">
[[ user[1] ]]
</div>
</div>
<button id="nextButton" ><img src="{{ url_for('static',filename='images/scrollright.png') }}" alt="向右滑动" ></button>
</div>
<button id="nextButton">
<img src="{{ url_for('static',filename='images/scrollright.png') }}" alt="向右滑动">
</button>
</div>
</div>
<!-- 以上是多用户切换 -->
</div>
<!-- 固定在右侧的 think 信息面板 -->
<div class="think-panel-container" v-show="thinkContent">
<div class="think-panel">
<div class="think-panel-header">
<h4>思考内容</h4>
<button class="think-panel-minimize" @click="minimizeThinkPanel">
<span class="think-panel-icon" v-if="isThinkPanelMinimized">+</span>
<span class="think-panel-icon" v-else>-</span>
</button>
</div>
<p>[[thinkContent]]</p>
</div>
</div>
</div>
</body>
<script>
document.addEventListener('DOMContentLoaded', function() {
// 初始化 thinkContent
if (typeof app !== 'undefined') {
app.thinkContent = "";
}
});
</script>
</body>
</html>

2
llm/nlp_ollama_api.py
View File

@@ -64,8 +64,6 @@ def question(cont, uid=0, observation=""):
result = json.loads(response.text)
response_text = result["message"]["content"]
if "</think>" in response_text:
response_text = response_text.split("</think>", 1)[1]
except requests.exceptions.RequestException as e:
print(f"请求失败: {e}")

16
main.py
View File

@@ -38,16 +38,7 @@ def __clear_logs():
os.mkdir("./logs")
for file_name in os.listdir('./logs'):
if file_name.endswith('.log'):
os.remove('./logs/' + file_name)
#ip替换
def replace_ip_in_file(file_path, new_ip):
with open(file_path, "r", encoding="utf-8") as file:
content = file.read()
content = re.sub(r"127\.0\.0\.1", new_ip, content)
content = re.sub(r"localhost", new_ip, content)
with open(file_path, "w", encoding="utf-8") as file:
file.write(content)
os.remove('./logs/' + file_name)
def kill_process_by_port(port):
for conn in psutil.net_connections(kind='inet'):
@@ -123,11 +114,6 @@ if __name__ == '__main__':
contentdb = content_db.new_instance()
contentdb.init_db()
#ip替换
if config_util.fay_url != "127.0.0.1":
replace_ip_in_file("gui/static/js/index.js", config_util.fay_url)
replace_ip_in_file("gui/static/js/setting.js", config_util.fay_url)
#启动数字人接口服务
ws_server = wsa_server.new_instance(port=10002)
ws_server.start_server()

62
shell/run_startup.py
View File

@@ -1,62 +0,0 @@
import subprocess
import os
import signal
# 存储所有启动的进程
running_processes = []
def run_startup_apps():
# Get the directory of the current script
script_dir = os.path.dirname(os.path.abspath(__file__))
startup_file = os.path.join(script_dir, 'startup.txt')
if not os.path.exists(startup_file):
return
# Read and process each line in the startup file
with open(startup_file, 'r', encoding='utf-8') as f:
for line in f:
# Skip empty lines
line = line.strip()
if not line:
continue
try:
# Split the command into program path and arguments
parts = line.split()
program = parts[0]
args = parts[1:] if len(parts) > 1 else []
# Create the process with no window
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE
# Start the process
process = subprocess.Popen(
[program] + args,
startupinfo=startupinfo,
creationflags=subprocess.CREATE_NEW_CONSOLE
)
running_processes.append(process)
print(f"Started: {line}")
except Exception as e:
print(f"Error starting {line}: {str(e)}")
def stop_all_processes():
"""停止所有启动的进程"""
for process in running_processes:
try:
if process.poll() is None: # 检查进程是否还在运行
process.terminate() # 尝试正常终止
try:
process.wait(timeout=3) # 等待进程终止，最多等待3秒
except subprocess.TimeoutExpired:
process.kill() # 如果进程没有及时终止，强制结束
print(f"Stopped process with PID: {process.pid}")
except Exception as e:
print(f"Error stopping process: {str(e)}")
running_processes.clear()
if __name__ == "__main__":
run_startup_apps()

0
shell/startup.txt
View File