-
- Fay数字人 AI Agent版
- “agent”即代理，它能够代替你完成决策规划并执行，这一切都依赖目前最强的大语言模型的ReAct能力。不同于助理版的一问一答，agent版的Fay可以实现自动代理执行的同时，在它认为必要时候会触发数字人或者直接的声音输出。 -
+ Fay开源数字人框架
-**请先想明白** 如果你需要是一个线上线下的销售员，请移步[`带货完整版`](https://github.com/TheRamU/Fay/tree/fay-sales-edition) 如果你需要的是一个人机交互的数字人助理（当然，你也可以命令它开关设备），请移步 [`助理完整版`](https://github.com/TheRamU/Fay/tree/fay-assistant-edition) +如果你需要是一个可以自主决策、主动联系主人的agent，请移步[`agent版`](https://github.com/TheRamU/Fay/tree/fay-agent-edition) - -***然后，“优秀的产品都值得用数字人从新做一遍”*** -1、基于日程维护的助理模式：执行及维护你的日程，绝不是一个简单的闹钟 -
-
-2、强大的规划执行（ReAct）能力：规划->执行<->反思->总结
-
-
-3、LLM Chain与React Agent自动切换：保留规划执行能力的同时兼顾聊天能力（还需优化）
-
-
-4、双记忆机制：斯坦福AI小镇的记忆流（时间、重要性、相关度）实现长时记忆，邻近对话记忆实现连贯对话
-
-
-5、易于扩展的agent 工具
-
-
-6、配套24小时后台运行的android 连接器
-
-
-
-## **安装说明**
-
-
-### **环境要求**
-
-- Python 3.9、3.10
-- Windows、macos、linux
-
-### **安装依赖**
-
-```shell
-pip install -r requirements.txt
-```
-
-### **配置应用密钥**
-
-+ 将GPT-4 key填入 `./system.conf` 中
-
-### **启动控制器**
-
-启动Fay控制器
-
-```shell
-python main.py
-```
-
-### **启动数字人（非必须）**
-
-+ 仓库地址：https://github.com/xszyou/fay-ue5
-
-### **启动android 连接器（非必须）**
-
-+ 仓库地址：https://github.com/xszyou/fay-android
-
-### **更新日志**
-2024.01.01:
-openai token计算✓
-优化ReAct Agent 与 LLM Chain自动切换逻辑✓
-*添加双记忆机制：长时记忆流及短时聊天记忆✓
-修复record.py asr bug✓
-提高远程音频（android 连接器）的稳定性✓
-修复执行时间计算bug✓
-优化语音输出逻辑✓
-
-2023.12.25:
-
-*实现agent ReAct与LLM chain自动切换逻辑✓
-聊天窗区分任务消息✓
-修复删除日程bug✓
-优化远程音频逻辑✓
-等待处理引入加载中效果✓
-优化prompt以解决日程任务递归调用问题✓
-修复一次性日程清除的bug✓
-
-
-### **技术交流**
+框架文档：https://qqk9ntwbcit.feishu.cn/wiki/JzMJw7AghiO8eHktMwlcxznenIg
-
\ No newline at end of file
+联系我们，请关注微信公众号 fay数字人
+
diff --git a/README_EN.md b/README_EN.md
deleted file mode 100644
index 9e28e5e..0000000
--- a/README_EN.md
+++ /dev/null
@@ -1,100 +0,0 @@
-[`中文`](https://github.com/TheRamU/Fay/blob/main/README.md)
-
-
-
-
-
-
-
-
-
-**Please Understand First**
-
-If you need an online and offline salesperson, please go to [`Complete Retail Version`](https://github.com/TheRamU/Fay/tree/fay-sales-edition)
-
-If you need a digital human assistant for human-computer interaction (and yes, you can command it to switch devices on and off), please go to [`Complete Assistant Version`](https://github.com/TheRamU/Fay/tree/fay-assistant-edition)
-
-**"Excellent products deserve to be redone with digital humans."**
-1.Assistant mode based on schedule maintenance: Managing and maintaining your schedule, not just a simple alarm clock.
--
- Fay Digital Human AI Agent Version
- An "agent" is a representative that can make decisions and execute plans for you, relying on the powerful ReAct capability of the most advanced large language models. -
-
-2.Powerful planning and execution (ReAct) capability: Plan -> Execute <-> Reflect -> Summarize.
-
-
-3.Automatic switching between LLM Chain and React Agent: Retains planning and execution capabilities while considering chatting abilities (still needs optimization).
-
-
-4.Dual memory mechanism: Stanford AI Town's memory stream (time, importance, relevance) for long-term memory, and adjacent conversation memory for coherent conversations.
-
-
-5.Easily expandable agent tools.
-
-
-6.Accompanying 24-hour background running Android connector.
-
-
-## **Installation Instructions**
-
-### **System Requirements**
-
-- Python 3.9, 3.10
-- Windows, macOS, Linux
-
-### **Installing Dependencies**
-
-```shell
-pip install -r requirements.txt
-```
-
-### **Configuring Application Keys**
-
-+ Enter your GPT-4 key in `./system.conf`
-
-### **Launching the Controller**
-
-Start the Fay controller
-
-```shell
-python main.py
-```
-
-### **Launching the Digital Human (Optional)**
-
-Repository URL:https://github.com/xszyou/fay-ue5
-
-
-### **Launch of Android Connector (Optional)**
-Repository URL: https://github.com/xszyou/fay-android
-
-
-### **Changelog**
-2024.01.01:
-
-OpenAI token calculation ✓
-Optimized ReAct Agent and LLM Chain auto-switching logic ✓
-*Added dual memory mechanism: long-term memory stream and short-term chat memory ✓
-Fixed record.py ASR bug ✓
-Improved stability of remote audio (Android connector) ✓
-Fixed execution time calculation bug ✓
-Optimized voice output logic ✓
-
-2023.12.25:
-
-Implemented the automatic switching logic between agent ReAct and LLM chain ✓
-Distinguished task messages in the chat window ✓
-Fixed the bug in deleting schedules ✓
-Optimized remote audio logic ✓
-Introduced loading effects for pending processes ✓
-Optimized prompts to resolve recursive calling issues in schedule tasks ✓
-Fixed the bug in clearing one-time schedules ✓
-
-
-### **Contact**
-
-**Business QQ: 467665317**
-
-Join the discussion group by following the public account Fay Digital Human (please star this repository first)
-
-
diff --git a/WebSocket.md b/WebSocket.md
deleted file mode 100644
index b006adc..0000000
--- a/WebSocket.md
+++ /dev/null
@@ -1,114 +0,0 @@
-#这是与数字人通讯的接口说明
-## 消息格式
-
-通讯地址: [`ws://127.0.0.1:10002`](ws://127.0.0.1:10002)
-
-注：ue作为客户端
-
-
-
-### 发送情绪值
-
-```json
-{
- "Topic": "Unreal",
- "Data": {
- "Key": "mood",
- "Value": 1.0
- }
-}
-```
-
-
-
-| 参数 | 描述 | 类型 | 范围 |
-| ---------- | ------ | ----- | ------- |
-| Data.Value | 情绪值 | float | [-1, 1] |
-
-
-
-
-
-### 发送音频
-
-```json
-{
- "Topic": "Unreal",
- "Data": {
- "Key": "audio",
- "Value": "C:\\samples\\sample-1.wav",
- "Text" : "很高兴见到你"
- "Lips":[{"Lip": "sil", "Time": 180}, {"Lip": "FF", "Time": 144}],
- "Time": 10,
- "Type": "interact"
- }
-}
-```
-
-
-
-| 参数 | 描述 | 类型 | 范围 |
-| ---------- | ---------------- | ----- | --------------- |
-| Data.Value | 音频文件绝对路径 | str | |
-| Data.Time | 音频时长 (秒) | float | |
-| Data.Type | 发言类型 | str | interact/script |
-| Data.Lips | 视音素 | array | |
-| Data.text | 文本 | str | |
-
-
-
-
-
-### 发送回复文字
-
-```json
-{
- "Topic": "Unreal",
- "Data": {
- "Key": "text",
- "Value": "很高兴见到你"
- }
-}
-```
-
-
-
-| 参数 | 描述 | 类型 | 范围 |
-| ---------- | ---------------- | ----- | --------------- |
-| Data.text | 文本 | str | |
-
-### 发送询问文字
-
-```json
-{
- "Topic": "Unreal",
- "Data": {
- "Key": "question",
- "Value": "很高兴见到你"
- }
-}
-```
-
-
-
-| 参数 | 描述 | 类型 | 范围 |
-| ---------- | ---------------- | ----- | --------------- |
-| Data.text | 文本 | str | |
-
-### 发送日志文字
-
-```json
-{
- "Topic": "Unreal",
- "Data": {
- "Key": "log",
- "Value": "很高... "
- }
-}
-```
-
-
-
-| 参数 | 描述 | 类型 | 范围 |
-| ---------- | ---------------- | ----- | --------------- |
-| Data.text | 文本 | str | |
diff --git a/agent/agent_service.py b/agent/agent_service.py
deleted file mode 100644
index f7b9adf..0000000
--- a/agent/agent_service.py
+++ /dev/null
@@ -1,125 +0,0 @@
-import sqlite3
-import threading
-import datetime
-import time
-from agent.fay_agent import FayAgentCore
-from core import fay_core
-
-scheduled_tasks = {}
-agent_running = False
-agent = FayAgentCore()
-
-
-# 数据库初始化
-def init_db():
- conn = sqlite3.connect('timer.db')
- cursor = conn.cursor()
- cursor.execute('''
- CREATE TABLE IF NOT EXISTS timer (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- time TEXT NOT NULL,
- repeat_rule TEXT NOT NULL,
- content TEXT NOT NULL
- )
- ''')
- conn.commit()
- conn.close()
-
-
-
-# 插入测试数据
-def insert_test_data():
- conn = sqlite3.connect('timer.db')
- cursor = conn.cursor()
- cursor.execute("INSERT INTO timer (time, repeat_rule, content) VALUES (?, ?, ?)", ('16:20', '1010001', 'Meeting Reminder'))
- conn.commit()
- conn.close()
-
-# 解析重复规则返回待执行时间，None代表不在今天的待执行计划
-def parse_repeat_rule(rule, task_time):
- today = datetime.datetime.now()
- if rule == '0000000': # 不重复
- task_datetime = datetime.datetime.combine(today.date(), task_time)
- if task_datetime > today:
- return task_datetime
- else:
- return None
- for i, day in enumerate(rule):
- if day == '1' and today.weekday() == i:
- task_datetime = datetime.datetime.combine(today.date(), task_time)
- if task_datetime > today:
- return task_datetime
- return None
-
-# 执行任务
-def execute_task(task_time, id, content):
- agent.is_chat = False
- fay_core.send_for_answer("执行任务->立刻" + content)
- if id in scheduled_tasks:
- del scheduled_tasks[id]
- # 如果不重复，执行后删除记录
- conn = sqlite3.connect('timer.db')
- cursor = conn.cursor()
- cursor.execute("DELETE FROM timer WHERE repeat_rule = '0000000' AND time = ? AND content = ?", (task_time.strftime('%H:%M'), content))
- conn.commit()
- conn.close()
-
-
-# 30秒扫描一次数据库，当扫描到今天的不存在于定时任务列表的记录，则添加到定时任务列表。执行完的记录从定时任务列表中清除。
-def check_and_execute():
- while agent_running:
- conn = sqlite3.connect('timer.db')
- cursor = conn.cursor()
- cursor.execute("SELECT * FROM timer")
- rows = cursor.fetchall()
-
- for row in rows:
- id, task_time_str, repeat_rule, content = row
- task_time = datetime.datetime.strptime(task_time_str, '%H:%M').time()
- next_execution = parse_repeat_rule(repeat_rule, task_time)
-
- if next_execution and id not in scheduled_tasks:
- timer_thread = threading.Timer((next_execution - datetime.datetime.now()).total_seconds(), execute_task, [next_execution, id, content])
- timer_thread.start()
- scheduled_tasks[id] = timer_thread
-
- conn.close()
- time.sleep(30) # 30秒扫描一次
-
-# agent启动
-def agent_start():
- global agent_running
- global agent
-
- agent_running = True
- init_db()
- check_and_execute_thread = threading.Thread(target=check_and_execute)
- check_and_execute_thread.start()
-
- #初始计划
- fay_core.send_for_answer("""执行任务-->
- 你是一个智慧农业实验箱里的ai，你的责任是陪伴主人生活、工作，以及协助主人打理好农业种植箱里的农作物：
- 1、在每天早上8点、中午12点、晚上10点检查农业种植箱的状态是否附合设定的预期执行，如果不符合请告知我调整。
- 2、每天12点“语音提醒主人吃饭”;
- 3、在星期一到星期五13:30“语音提醒主人开始工作”;
- 4、在星期一到星期五15:15“语音提醒主人冲咖啡”;
- 5、在星期一、星期三的11:15“语音提醒主人开会”;
- 6、在星期五17:30“语音提醒主人开会”;
- 7、在星期一到星期五18:00“语音提醒主人下班”;
- 8、在每天21点陪主人聊聊天;
- 9、在每天晚上10:30会跟据第二天的天气预报信息和当天的运行情况，检查iotm系统当天的控制规则；
-
- """)
-
-def agent_stop():
- global agent_running
- global scheduled_tasks
- # 取消所有定时任务
- for task in scheduled_tasks.values():
- task.cancel()
- agent_running = False
- scheduled_tasks = {}
-
-
-if __name__ == "__main__":
- agent_start()
diff --git a/agent/fay_agent.py b/agent/fay_agent.py
deleted file mode 100644
index d52680d..0000000
--- a/agent/fay_agent.py
+++ /dev/null
@@ -1,285 +0,0 @@
-import os
-import time
-import math
-
-from langchain.embeddings.openai import OpenAIEmbeddings
-from langchain.chat_models import ChatOpenAI
-from langchain.memory import VectorStoreRetrieverMemory
-import faiss
-from langchain.docstore import InMemoryDocstore
-from langchain.vectorstores import FAISS
-from langchain.agents import AgentExecutor, Tool, ZeroShotAgent, initialize_agent, agent_types
-from langchain.chains import LLMChain
-from langchain.prompts import PromptTemplate
-
-from agent.tools.MyTimer import MyTimer
-from agent.tools.QueryTime import QueryTime
-from agent.tools.Weather import Weather
-from agent.tools.Calculator import Calculator
-from agent.tools.CheckSensor import CheckSensor
-from agent.tools.Switch import Switch
-from agent.tools.Knowledge import Knowledge
-from agent.tools.Say import Say
-from agent.tools.QueryTimerDB import QueryTimerDB
-from agent.tools.DeleteTimer import DeleteTimer
-from agent.tools.GetSwitchLog import GetSwitchLog
-from agent.tools.getOnRunLinkage import getOnRunLinkage
-from agent.tools.SetChatStatus import SetChatStatus
-from langchain.callbacks import get_openai_callback
-from langchain.retrievers import TimeWeightedVectorStoreRetriever
-from langchain.memory import ConversationBufferWindowMemory
-
-import utils.config_util as utils
-from core import wsa_server
-import fay_booter
-from utils import util
-
-
-class FayAgentCore():
- def __init__(self):
- utils.load_config()
- os.environ['OPENAI_API_KEY'] = utils.key_gpt_api_key
- #使用open ai embedding
- embedding_size = 1536 # OpenAIEmbeddings 的维度
- index = faiss.IndexFlatL2(embedding_size)
- embedding_fn = OpenAIEmbeddings()
-
- #创建llm
- self.llm = ChatOpenAI(model="gpt-4-1106-preview", verbose=True)
-
- #创建向量数据库
- def relevance_score_fn(self, score: float) -> float:
- return 1.0 - score / math.sqrt(2)
- vectorstore = FAISS(embedding_fn, index, InMemoryDocstore({}), {}, relevance_score_fn=relevance_score_fn)
-
- # 创建记忆(斯坦福小镇同款记忆检索机制:时间、相关性、重要性三个维度)
- retriever = TimeWeightedVectorStoreRetriever(vectorstore=vectorstore, other_score_keys=["importance"], k=3)
- self.agent_memory = VectorStoreRetrieverMemory(memory_key="history", retriever=retriever)
-
- # 保存基本信息到记忆
- utils.load_config()
- attr_info = ", ".join(f"{key}: {value}" for key, value in utils.config["attribute"].items())
- self.agent_memory.save_context({"input": "我的基本信息是?"}, {"output": attr_info})
-
- #内存保存聊天历史
- self.chat_history = []
-
- #创建agent chain
- my_timer = MyTimer()
- query_time_tool = QueryTime()
- weather_tool = Weather()
- calculator_tool = Calculator()
- check_sensor_tool = CheckSensor()
- switch_tool = Switch()
- knowledge_tool = Knowledge()
- say_tool = Say()
- query_timer_db_tool = QueryTimerDB()
- delete_timer_tool = DeleteTimer()
- get_switch_log = GetSwitchLog()
- get_on_run_linkage = getOnRunLinkage()
- set_chat_status_tool = SetChatStatus()
-
- self.tools = [
- Tool(
- name=my_timer.name,
- func=my_timer.run,
- description=my_timer.description
- ),
- Tool(
- name=query_time_tool.name,
- func=query_time_tool.run,
- description=query_time_tool.description
- ),
- Tool(
- name=weather_tool.name,
- func=weather_tool.run,
- description=weather_tool.description
- ),
- Tool(
- name=calculator_tool.name,
- func=calculator_tool.run,
- description=calculator_tool.description
- ),
- Tool(
- name=check_sensor_tool.name,
- func=check_sensor_tool.run,
- description=check_sensor_tool.description
- ),
- Tool(
- name=switch_tool.name,
- func=switch_tool.run,
- description=switch_tool.description
- ),
- Tool(
- name=knowledge_tool.name,
- func=knowledge_tool.run,
- description=knowledge_tool.description
- ),
- Tool(
- name=say_tool.name,
- func=say_tool.run,
- description=say_tool.description
- ),
- Tool(
- name=query_timer_db_tool.name,
- func=query_timer_db_tool.run,
- description=query_timer_db_tool.description
- ),
- Tool(
- name=delete_timer_tool.name,
- func=delete_timer_tool.run,
- description=delete_timer_tool.description
- ),
- Tool(
- name=get_switch_log.name,
- func=get_switch_log.run,
- description=get_switch_log.description
- ),
- Tool(
- name=get_on_run_linkage.name,
- func=get_on_run_linkage.run,
- description=get_on_run_linkage.description
- ),
- Tool(
- name=set_chat_status_tool.name,
- func=set_chat_status_tool.run,
- description=set_chat_status_tool.description
- ),
- ]
-
- #agent用于执行任务
- self.agent = initialize_agent(agent_types=agent_types.AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION,
- tools=self.tools, llm=self.llm, verbose=True,
- max_history=5, handle_parsing_errors=True)
-
- #llm chain 用于聊天
- self.is_chat = False#聊天状态
-
- #记录一轮执行有无调用过say tool
- self.is_use_say_tool = False
- self.say_tool_text = ""
-
- self.total_tokens = 0
- self.total_cost = 0
-
-
- def format_history_str(self, str):
- result = ""
- history_string = str['history']
-
- # Split the string into lines
- lines = history_string.split('input:')
-
- # Initialize an empty list to store the formatted history
- formatted_history = []
-
- #处理记忆流格式
- for line in lines:
- if "output" in line:
- input_line = line.split("output:")[0].strip()
- output_line = line.split("output:")[1].strip()
- formatted_history.append({"input": input_line, "output": output_line})
-
-
- # 记忆流转换成字符串
- result += "-以下是与用户说话关连度最高的记忆：\n"
- for i in range(len(formatted_history)):
- if i >= 3:
- break
- line = formatted_history[i]
- result += f"--input：{line['input']}\n--output：{line['output']}\n"
- if len(formatted_history) == 0:
- result += "--没有记录\n"
-
-
- #添加内存记忆
- formatted_history = []
- for line in self.chat_history:
- formatted_history.append({"input": line[0], "output": line[1]})
-
- #格式化内存记忆字符串
- result += "\n-以下刚刚的对话：\n"
- for i in range(len(formatted_history)):
- line = formatted_history[i]
- result += f"--input：{line['input']}\n--output：{line['output']}\n"
- if len(formatted_history) == 0:
- result += "--没有记录\n"
-
- return result
-
-
- def get_llm_chain(self, history):
- tools_prompt = "["
- tool_names = [tool.name for tool in self.tools if tool.name != SetChatStatus().name and tool.name != Say().name]
- tools_prompt += "、".join(tool_names) + "]"
- template = """
-你是一个智能家居系统中的AI，负责协助主人处理日常事务和智能设备的操作。当主人提出要求时，如果需要使用特定的工具或执行特定的操作，请严格回复“agent: {human_input}”字符串。如果主人只是进行普通对话或询问信息，直接以文本内容回答即可。你可以使用的工具或执行的任务包括：。
-""" + tools_prompt + "等。" +"""
-现在时间是：now_time
-请依据以下信息回复主人：
-chat_history
-
-input：
-{human_input}
-output：""".replace("chat_history", history).replace("now_time", QueryTime().run(""))
- prompt = PromptTemplate(
- input_variables=["human_input"], template=template
- )
-
- llm_chain = LLMChain(
- llm=self.llm,
- prompt=prompt,
- verbose=True
- )
- return llm_chain
-
- def run(self, input_text):
- self.is_use_say_tool = False
- self.say_tool_text = ""
-
- result = ""
- history = self.agent_memory.load_memory_variables({"input":input_text.replace('主人语音说了：', '').replace('主人文字说了：', '')})
- history = self.format_history_str(history)
- try:
- #判断执行聊天模式还是agent模式，双模式在运行过程中会主动切换
- if self.is_chat:
- llm_chain = self.get_llm_chain(history)
- with get_openai_callback() as cb:
- result = llm_chain.predict(human_input=input_text.replace('主人语音说了：', '').replace('主人文字说了：', ''))
- self.total_tokens = self.total_tokens + cb.total_tokens
- self.total_cost = self.total_cost + cb.total_cost
- util.log(1, "本次消耗token:{}， Cost (USD):{}，共消耗token:{}， Cost (USD):{}".format(cb.total_tokens, cb.total_cost, self.total_tokens, self.total_cost))
-
- if "agent:" in result.lower() or not self.is_chat:
- self.is_chat = False
- input_text = result.lower().replace("agent:", "") if "agent:" in result.lower() else input_text.replace('主人语音说了：', '').replace('主人文字说了：', '')
- agent_prompt = """
-现在时间是：{now_time}。请依据以下信息为主人服务 ：
-{history}
-input：{input_text}
-output：
-""".format(history=history, input_text=input_text, now_time=QueryTime().run(""))
- print(agent_prompt)
- with get_openai_callback() as cb:
- result = self.agent.run(agent_prompt)
- self.total_tokens = self.total_tokens + cb.total_tokens
- self.total_cost = self.total_cost + cb.total_cost
- util.log(1, "本次消耗token:{}， Cost (USD):{}，共消耗token:{}， Cost (USD):{}".format(cb.total_tokens, cb.total_cost, self.total_tokens, self.total_cost))
-
- except Exception as e:
- print(e)
-
- result = "执行完毕" if result is None or result == "N/A" else result
- chat_text = self.say_tool_text if self.is_use_say_tool else result
-
- #保存到记忆流和聊天对话
- self.agent_memory.save_context({"input": input_text.replace('主人语音说了：', '').replace('主人文字说了：', '')},{"output": result})
- self.chat_history.append((input_text.replace('主人语音说了：', '').replace('主人文字说了：', ''), chat_text))
- if len(self.chat_history) > 5:
- self.chat_history.pop(0)
-
- return self.is_use_say_tool, chat_text
-
-if __name__ == "__main__":
- agent = FayAgentCore()
- print(agent.run("你好"))
diff --git a/agent/tools/Calculator.py b/agent/tools/Calculator.py
deleted file mode 100644
index 6599694..0000000
--- a/agent/tools/Calculator.py
+++ /dev/null
@@ -1,32 +0,0 @@
-import abc
-import math
-from typing import Any
-
-from langchain.tools import BaseTool
-
-
-class Calculator(BaseTool, abc.ABC):
- name = "Calculator"
- description = "Useful for when you need to answer questions about math（不能用于非数字计算）"
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str) -> str:
- para = para.replace("^", "**")
- if "sqrt" in para:
- para = para.replace("sqrt", "math.sqrt")
- elif "log" in para:
- para = para.replace("log", "math.log")
- return eval(para)
-
-
-if __name__ == "__main__":
- calculator_tool = Calculator()
- result = calculator_tool.run("sqrt(2) + 3")
- print(result)
diff --git a/agent/tools/CheckSensor.py b/agent/tools/CheckSensor.py
deleted file mode 100644
index 395f354..0000000
--- a/agent/tools/CheckSensor.py
+++ /dev/null
@@ -1,79 +0,0 @@
-import os
-from typing import Any
-
-import requests
-from langchain.tools import BaseTool
-import time
-import agent.tools.IotmService as IotmService
-from datetime import datetime
-
-class CheckSensor(BaseTool):
- name = "CheckSensor"
- description = "此工具用于查询农业箱在线状态、传感器数据、设备开关状态"
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str) -> str:
- #箱子信息
- building_infos = IotmService.get_building_unit()
- is_online = building_infos.get('isonline', 0)
- #传感器数据
- sensor_all_infos = IotmService.get_latest_list()
- sensor_infos = sensor_all_infos['data']
- desc_list = {
- 'temperature': '温度',
- 'humidity': '湿度',
- 'co2': '二氧化碳',
- 'light': '箱内的光照强度的值，当箱内光照强度太低时，生长灯会被打开，传感器位置是可以检测到生长灯的亮度的',
- 'air': '污染气体',
- 'nh3': '氨气'
- }
-
- infos = []
- for sensor_type, sensor_data in sensor_infos.items():
- for data_point in sensor_data:
- if sensor_type == 'temperature':
- if data_point['port'] == 'MP14':
- description = '箱外温度'
- else:
- description = '箱内温度'
-
- elif sensor_type == 'humidity':
- if data_point['port'] == 'MP14':
- description = '箱外湿度'
- elif data_point['port'] == 'S34':
- description = '箱内土壤的湿度，检测的数所有延迟，水在土壤里有个渗透的过程'
- else:
- description = '箱内湿度'
- else:
- description = desc_list.get(sensor_type, 'Unknown') # Get description from desc_list, default to 'Unknown'
- timestamp = data_point['ts']
- value = data_point['val']
- infos.append({'ts': timestamp, 'val': value, 'desc':description })
- #开关数据
- switch_all_infos = IotmService.get_switch_info()
- switch_infos = {}
- switch_dict = switch_all_infos[0]
- #设备配置
- switch_infos['小风扇'] = 'on' if switch_dict.get('onoff1', '') == '1' else 'off'
- switch_infos['电热风扇'] = 'on' if switch_dict.get('onoff2', '') == '1' else 'off'
- switch_infos['制冷风扇'] = 'on' if switch_dict.get('onoff3', '') == '1' else 'off'
- switch_infos['水开关'] = 'on' if switch_dict.get('onoff4', '') == '1' else 'off'
- switch_infos['肥料开关'] = 'on' if switch_dict.get('onoff5', '') == '1' else 'off'
- switch_infos['植物生长灯'] = 'on' if switch_dict.get('onoff6', '') == '1' else 'off'
- switch_infos['二氧化碳'] = 'on' if switch_dict.get('onoff7', '') == '1' else 'off'
- current_time = datetime.now()
- current_time_str = current_time.strftime("%Y-%m-%d %H:%M:%S")
- result = {'sensor_infos': infos, 'switch_infos': switch_infos, 'is_online': is_online, 'ts' : current_time_str}
- return result
-
-if __name__ == "__main__":
- tool = CheckSensor()
- info = tool.run("")
- print(info)
diff --git a/agent/tools/DeleteTimer.py b/agent/tools/DeleteTimer.py
deleted file mode 100644
index 531558c..0000000
--- a/agent/tools/DeleteTimer.py
+++ /dev/null
@@ -1,39 +0,0 @@
-import sqlite3
-from typing import Any
-
-from langchain.tools import BaseTool
-
-from agent import agent_service
-
-
-class DeleteTimer(BaseTool):
- name = "DeleteTimer"
- description = "用于删除某一个日程，接受任务id作为参数，如：2"
-
- def __init__(self):
- super().__init__()
-
- def _run(self, para) -> str:
- try:
- id = int(para)
- except ValueError:
- return "输入的 ID 无效，必须是数字。"
-
- if id in agent_service.scheduled_tasks:
- del agent_service.scheduled_tasks[id]
-
- try:
- with sqlite3.connect('timer.db') as conn:
- cursor = conn.cursor()
- cursor.execute("DELETE FROM timer WHERE id = ?", (id,))
- conn.commit()
- except sqlite3.Error as e:
- return f"数据库错误: {e}"
-
- return f"任务 {id} 取消成功。"
-
-
-if __name__ == "__main__":
- tool = DeleteTimer()
- result = tool.run("1")
- print(result)
diff --git a/agent/tools/GetSwitchLog.py b/agent/tools/GetSwitchLog.py
deleted file mode 100644
index 8a3b318..0000000
--- a/agent/tools/GetSwitchLog.py
+++ /dev/null
@@ -1,48 +0,0 @@
-import os
-from typing import Any
-
-from langchain.tools import BaseTool
-import tools.IotmService as IotmService
-
-class GetSwitchLog(BaseTool):
- name = "GetSwitchLog"
- description = "此工具用于查询农业箱的设备开关当天的操作历史记录"
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str):
- logs = IotmService.get_switch_log()
- device_logs = {}
-
- switch_mapping = {
- 1: '小风扇',
- 2: '电热风扇',
- 3: '制冷风扇',
- 4: '水开关',
- 5: '肥料开关',
- 6: '植物生长灯',
- 7: '二氧化碳'
- }
-
- for val in logs:
- switch_name = switch_mapping[val['number']]
- status = 'on' if val['status'] == 1 else 'off'
- info = val['timetText']
-
- if switch_name not in device_logs:
- device_logs[switch_name] = {'on': [], 'off': []}
-
- device_logs[switch_name][status].append(info)
-
- return device_logs
-
-if __name__ == "__main__":
- tool = GetSwitchLog()
- info = tool.run("")
- print(info)
diff --git a/agent/tools/IotmService.py b/agent/tools/IotmService.py
deleted file mode 100644
index ff2d2d3..0000000
--- a/agent/tools/IotmService.py
+++ /dev/null
@@ -1,47 +0,0 @@
-
-#获取传感器状态
-def get_latest_list():
- info ="""
- {'result': True, 'code': 1, 'msg': '查询成功', 'data': {'co2': [{'ts': '2023-12-18 16:07:28.124', 'val': 8, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'S36', 'sensorid': 297}], 'air': [{'ts': '2023-12-18 16:07:28.124', 'val': 15, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'S37', 'sensorid': 298}], 'humidity': [{'ts': '2023-12-18 16:06:20.152', 'val': 49.7, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'MP14', 'sensorid': 302}, {'ts': '2023-12-18 16:06:57.861', 'val': 40.8, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'MP21', 'sensorid': 300}, {'ts': '2023-12-18 16:07:28.124', 'val': 99.41003, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'S34', 'sensorid': 299}], 'light': [{'ts': '2023-12-18 16:07:28.124', 'val': 185, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'bh1', 'sensorid': 301}], 'nh3': [{'ts': '2023-12-18 16:07:28.124', 'val': 14, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'S37', 'sensorid': 303}], 'temperature': [{'ts': '2023-12-18 16:03:58.326', 'val': 18.6, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'MP14', 'sensorid': 304}, {'ts': '2023-12-18 16:07:28.124', 'val': 22.9, 'istext': False, 'content_des': '', 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'port': 'MP21', 'sensorid': 305}]}}
- """
- return info
-
-#获取开关状态
-def get_switch_info():
- info = """
- [{'id': 16, 'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'onoff1': '1', 'onoff2': '0', 'onoff3': '0', 'onoff4': '0', 'onoff5': '0', 'onoff6': '1', 'onoff7': '1', 'onoff8': '0', 'onoff9': '0', 'onoff10': '0', 'onoff11': '0', 'onoff12': '0', 'onoff13': '0', 'onoff14': '0', 'onoff15': '0', 'onoff16': '0', 'updatetime': 1702886988874}]
- """
- return info
-
-#设备开关操作
-def do_switch_operation(num,onoff):
- return True
-
-#获取传感器基本信息
-def get_building_unit():
- info = """
- {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'lat': '0.0000000000000', 'lng': '0.0000000000000', 'sensor': [{'id': 297, 'title': '二氧化碳传感器', 'label': 'co2'}, {'id': 298, 'title': '空气质量传感器', 'label': 'air'}, {'id': 299, 'title': '土壤湿度传感器', 'label': 'humidity'}, {'id': 300, 'title': '温湿度传感器', 'label': 'humidity'}, {'id': 301, 'title': '光照传感器', 'label': 'light'}, {'id': 302, 'title': '温湿度传感器', 'label': 'humidity'}, {'id': 303, 'title': '氨气传感器', 'label': 'nh3'}, {'id': 304, 'title': '温湿度传感器', 'label': 'temperature'}, {'id': 305, 'title': '温湿度传感器', 'label': 'temperature'}], 'isonline': 1
- """
- return info
-
-#获取开关记录日志
-def get_switch_log():
- info = """
- [{'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 3, 'status': 0, 'createTime': 1702732876735, 'timetText': '2023-12-16 21:21:16'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 1, 'status': 1, 'createTime': 1702667478198, 'timetText': '2023-12-16 03:11:18'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 7, 'status': 1, 'createTime': 1702664989048, 'timetText': '2023-12-16 02:29:49'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 7, 'status': 0, 'createTime': 1702657012799, 'timetText': '2023-12-16 00:16:52'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 7, 'status': 1, 'createTime': 1702648220859, 'timetText': '2023-12-15 21:50:20'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 3, 'status': 1, 'createTime': 1702646816090, 'timetText': '2023-12-15 21:26:56'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 3, 'status': 0, 'createTime': 1702646531391, 'timetText': '2023-12-15 21:22:11'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 1, 'status': 0, 'createTime': 1702646530372, 'timetText': '2023-12-15 21:22:10'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 3, 'status': 1, 'createTime': 1702645992974, 'timetText': '2023-12-15 21:13:12'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 1, 'status': 1, 'createTime': 1702644950252, 'timetText': '2023-12-15 20:55:50'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 1, 'status': 0, 'createTime': 1702644949600, 'timetText': '2023-12-15 20:55:49'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 1, 'createTime': 1702634257442, 'timetText': '2023-12-15 17:57:37'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 0, 'createTime': 1702633183083, 'timetText': '2023-12-15 17:39:43'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 1, 'createTime': 1702631382970, 'timetText': '2023-12-15 17:09:42'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 0, 'createTime': 1702629480618, 'timetText': '2023-12-15 16:38:00'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 1, 'createTime': 1702628371951, 'timetText': '2023-12-15 16:19:31'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 0, 'createTime': 1702626695422, 'timetText': '2023-12-15 15:51:35'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 1, 'createTime': 1702625360795, 'timetText': '2023-12-15 15:29:20'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 0, 'createTime': 1702624152081, 'timetText': '2023-12-15 15:09:12'}, {'did': 'bbb14d38-2814-11ed-b20a-e45f019833ac', 'number': 6, 'status': 1, 'createTime': 1702622351970, 'timetText': '2023-12-15 14:39:11'}]
- """
- return info
-
-#获取联动规则
-def get_on_run_linkage():
- info = """
- {'result': True, 'code': 1, 'msg': '获取成功', 'data': [{'port': 'S34', 'sensorTitle': '土壤湿度传感器', 'label': 'humidity', 'minVal': 0, 'maxVal': 70, 'taskId': 'linkage_135', 'onoff': 1, 'switchNum': 4, 'keeptime': '0.25', 'delaytime': 30}, {'port': 'S36', 'sensorTitle': '二氧化碳传感器', 'label': 'co2', 'minVal': 0, 'maxVal': 8, 'taskId': 'linkage_138', 'onoff': 1, 'switchNum': 7, 'keeptime': '30.00', 'delaytime': 0}, {'port': 'MP14', 'sensorTitle': '温湿度传感器', 'label': 'temperature', 'minVal': 0, 'maxVal': 28, 'taskId': 'linkage_143', 'onoff': 1, 'switchNum': 1, 'keeptime': '0.00', 'delaytime': 0}, {'port': 'MP14', 'sensorTitle': '温湿度传感器', 'label':
-'temperature', 'minVal': 30, 'maxVal': 999999, 'taskId': 'linkage_144', 'onoff': 0, 'switchNum': 1, 'keeptime': '0.00', 'delaytime': 0}, {'port': 'MP14', 'sensorTitle': '温湿度传感器', 'label': 'temperature', 'minVal': 30, 'maxVal': 999999, 'taskId': 'linkage_145', 'onoff': 0, 'switchNum': 1, 'keeptime': '0.00', 'delaytime': 0}, {'port': 'bh1', 'sensorTitle': '光照传感器', 'label': 'light', 'minVal': 0, 'maxVal': 100, 'taskId': 'linkage_147', 'onoff': 1, 'switchNum': 6, 'keeptime': '30.00', 'delaytime': 50}]}
-[{'port': 'S34', 'sensorTitle': '土壤湿度传感器', 'label': 'humidity', 'minVal': 0, 'maxVal': 70, 'taskId': 'linkage_135', 'onoff': 1, 'switchNum': 4, 'keeptime': '0.25', 'delaytime': 30}, {'port': 'S36', 'sensorTitle': '二氧化碳传感器', 'label': 'co2', 'minVal': 0, 'maxVal': 8, 'taskId': 'linkage_138', 'onoff': 1, 'switchNum': 7, 'keeptime': '30.00', 'delaytime': 0}, {'port': 'MP14', 'sensorTitle': '温湿度传感器', 'label': 'temperature', 'minVal': 0, 'maxVal': 28, 'taskId': 'linkage_143', 'onoff':
-1, 'switchNum': 1, 'keeptime': '0.00', 'delaytime': 0}, {'port': 'MP14', 'sensorTitle': '温湿度传感器', 'label': 'temperature', 'minVal': 30, 'maxVal': 999999, 'taskId': 'linkage_144', 'onoff': 0, 'switchNum': 1, 'keeptime': '0.00', 'delaytime': 0}, {'port': 'MP14', 'sensorTitle': '温湿度传感器', 'label': 'temperature', 'minVal': 30,
-'maxVal': 999999, 'taskId': 'linkage_145', 'onoff': 0, 'switchNum': 1, 'keeptime': '0.00', 'delaytime': 0}, {'port': 'bh1', 'sensorTitle': '光照传感器', 'label': 'light', 'minVal': 0, 'maxVal': 100, 'taskId': 'linkage_147', 'onoff': 1, 'switchNum': 6, 'keeptime': '30.00', 'delaytime': 50}]
- """
- return info
-
-if __name__ == "__main__":
- str = get_on_run_linkage()
- print(str)
\ No newline at end of file
diff --git a/agent/tools/Knowledge.py b/agent/tools/Knowledge.py
deleted file mode 100644
index a59066c..0000000
--- a/agent/tools/Knowledge.py
+++ /dev/null
@@ -1,28 +0,0 @@
-import os
-from typing import Any
-
-import requests
-from langchain.tools import BaseTool
-
-
-class Knowledge(BaseTool):
- name = "Knowledge"
- description = """此工具用于查询箱内植物的专业知识，使用时请传入相关问题作为参数，例如：“草梅最适合的生长温度”"""
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str) -> str:
- return "查询知识库：" + para
-
-
-
-if __name__ == "__main__":
- tool = Knowledge()
- info = tool.run("草梅最适合的生长温度")
- print(info)
diff --git a/agent/tools/MyTimer.py b/agent/tools/MyTimer.py
deleted file mode 100644
index b9d61ff..0000000
--- a/agent/tools/MyTimer.py
+++ /dev/null
@@ -1,35 +0,0 @@
-import abc
-import sqlite3
-from typing import Any
-import ast
-
-
-from langchain.tools import BaseTool
-
-
-class MyTimer(BaseTool, abc.ABC):
- name = "MyTimer"
- description = "用于设置日程,使用的时候需要接受3个参数，第1个参数是时间,第2个参数是循环规则（如:'1000100'代表星期一和星期五循环，'0000000'代表不循环），第3个参数代表要执行的事项,如：('15:15', '0000001', '提醒主人叫咖啡')"
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para) -> str:
- para = ast.literal_eval(para)
- conn = sqlite3.connect('timer.db')
- cursor = conn.cursor()
- cursor.execute("INSERT INTO timer (time, repeat_rule, content) VALUES (?, ?, ?)", (para[0], para[1], para[2]))
- conn.commit()
- conn.close()
- return "日程设置成功"
-
-
-if __name__ == "__main__":
- calculator_tool = MyTimer()
- result = calculator_tool.run("sqrt(2) + 3")
- print(result)
diff --git a/agent/tools/QueryTime.py b/agent/tools/QueryTime.py
deleted file mode 100644
index 5e31428..0000000
--- a/agent/tools/QueryTime.py
+++ /dev/null
@@ -1,46 +0,0 @@
-import abc
-import math
-from typing import Any
-from datetime import datetime
-from langchain.tools import BaseTool
-
-class QueryTime(BaseTool, abc.ABC):
- name = "QueryTime"
- description = "用于查询当前日期、星期几及时间"
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
- def _run(self, para) -> str:
- # 获取当前时间
- now = datetime.now()
- # 获取当前日期
- today = now.date()
- # 获取星期几的信息
- week_day = today.strftime("%A")
- # 将星期几的英文名称转换为中文
- week_day_zh = {
- "Monday": "星期一",
- "Tuesday": "星期二",
- "Wednesday": "星期三",
- "Thursday": "星期四",
- "Friday": "星期五",
- "Saturday": "星期六",
- "Sunday": "星期日",
- }.get(week_day, "未知")
- # 将日期格式化为字符串
- date_str = today.strftime("%Y年%m月%d日")
-
- # 将时间格式化为字符串
- time_str = now.strftime("%H:%M")
-
- return "现在时间是：{0} {1} {2}".format(time_str, week_day_zh, date_str)
-
-if __name__ == "__main__":
- tool = QueryTime()
- result = tool.run("")
- print(result)
diff --git a/agent/tools/QueryTimerDB.py b/agent/tools/QueryTimerDB.py
deleted file mode 100644
index 94a9336..0000000
--- a/agent/tools/QueryTimerDB.py
+++ /dev/null
@@ -1,41 +0,0 @@
-import abc
-import sqlite3
-from typing import Any
-import ast
-
-
-from langchain.tools import BaseTool
-
-
-class QueryTimerDB(BaseTool, abc.ABC):
- name = "QueryTimerDB"
- description = "用于查询所有日程，返回的数据里包含3个参数:时间、循环规则（如:'1000100'代表星期一和星期五循环，'0000000'代表不循环）、执行的事项"
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para) -> str:
- conn = sqlite3.connect('timer.db')
- cursor = conn.cursor()
- # 执行查询
- cursor.execute("SELECT * FROM timer")
- # 获取所有记录
- rows = cursor.fetchall()
- # 拼接结果
- result = ""
- for row in rows:
- result = result + str(row) + "\n"
- conn.commit()
- conn.close()
- return result
-
-
-if __name__ == "__main__":
- tool = QueryTimerDB()
- result = tool.run("")
- print(result)
diff --git a/agent/tools/Say.py b/agent/tools/Say.py
deleted file mode 100644
index 66868aa..0000000
--- a/agent/tools/Say.py
+++ /dev/null
@@ -1,36 +0,0 @@
-import os
-from typing import Any
-
-import requests
-from langchain.tools import BaseTool
-import fay_booter
-from core.interact import Interact
-from agent import agent_service
-
-
-class Say(BaseTool):
- name = "语音输出工具"
- description = """此工具用于语音输出，使用时请传入说话内容作为参数，例如：“该下班了，请注意休息”"""
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str) -> str:
- agent_service.agent.is_chat = True
- agent_service.agent.is_use_say_tool = True
- agent_service.agent.say_tool_text = para
- interact = Interact("audio", 1, {'user': '', 'msg': para})
- fay_booter.feiFei.on_interact(interact)
- return "语音输出了：" + para
-
-
-
-if __name__ == "__main__":
- tool = Say()
- info = tool.run("该下班了，请注意休息")
- print(info)
diff --git a/agent/tools/SetChatStatus.py b/agent/tools/SetChatStatus.py
deleted file mode 100644
index 7f23b59..0000000
--- a/agent/tools/SetChatStatus.py
+++ /dev/null
@@ -1,29 +0,0 @@
-import os
-from typing import Any
-
-from langchain.tools import BaseTool
-from agent import agent_service
-
-
-class SetChatStatus(BaseTool):
- name = "SetChatStatus"
- description = """此工具用于设置聊天状态，当识别到主人想进行交流聊天时使用此工具"""
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str) -> str:
- agent_service.agent.is_chat = True
- return "设置聊天状态成功"
-
-
-
-if __name__ == "__main__":
- tool = SetChatStatus()
- info = tool.run("该下班了，请注意休息")
- print(info)
diff --git a/agent/tools/Switch.py b/agent/tools/Switch.py
deleted file mode 100644
index 2e05741..0000000
--- a/agent/tools/Switch.py
+++ /dev/null
@@ -1,55 +0,0 @@
-import os
-import ast
-from typing import Any
-
-import requests
-from langchain.tools import BaseTool
-import json
-import sys
-sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
-import agent.tools.IotmService as IotmService
-
-class Switch(BaseTool):
- name = "Switch"
- description = '此工具用于控制小风扇、电热风扇、制冷风扇、水开关、肥料开关、植物生长灯、二氧化碳的开关，参数格式:("小风扇","on"),返回True为成功'
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str) -> str:
- try:
- if not para:
- return "参数不能为空"
- para = ast.literal_eval(para)
- if not para:
- return "参数格式不正确"
- switch = para[0]
- switch_mapping = {
- '小风扇': 1,
- '电热风扇': 2,
- '制冷风扇': 3,
- '水开关': 4,
- '肥料开关': 5,
- '植物生长灯': 6,
- '二氧化碳': 7
- }
- if switch not in switch_mapping:
- return "未知的设备类型，请检查 'switch' 字段值"
- num = switch_mapping[switch]
- onoff = para[1]
- re = IotmService.do_switch_operation(num, onoff)
- return re
- except json.JSONDecodeError:
- return '参数格式不正确，请使用正确的 JSON 格式表示方式，例如 {"switch": "小风扇", "onoff": "on"}'
-
-
-
-if __name__ == "__main__":
- tool = Switch()
- info = tool.run('("小风扇","off")')
- print(info)
diff --git a/agent/tools/Weather.py b/agent/tools/Weather.py
deleted file mode 100644
index 9e6d4d9..0000000
--- a/agent/tools/Weather.py
+++ /dev/null
@@ -1,53 +0,0 @@
-import os
-from typing import Any
-
-import requests
-from langchain.tools import BaseTool
-from urllib.parse import quote
-
-class Weather(BaseTool):
- name = "weather"
- description = "此工具用于获取天气预报信息，需传入英文的城市名，参数格式：Guangzhou"
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str) -> str:
- try:
- if not para:
- return "参数不能为空"
- encoded_city = quote(para)
-
- api_url = f"http://api.openweathermap.org/data/2.5/weather?q={encoded_city}&appid=272fcb70d2c4e6f5134c2dce7d091df6"
- response = requests.get(api_url)
- if response.status_code == 200:
- weather_data = response.json()
- # 提取天气信息
- temperature_kelvin = weather_data['main']['temp']
- temperature_celsius = temperature_kelvin - 273.15
- min_temperature_kelvin = weather_data['main']['temp_min']
- max_temperature_kelvin = weather_data['main']['temp_max']
- min_temperature_celsius = min_temperature_kelvin - 273.15
- max_temperature_celsius = max_temperature_kelvin - 273.15
- description = weather_data['weather'][0]['description']
- wind_speed = weather_data['wind']['speed']
-
- # 构建天气描述
- weather_description = f"今天天气：{description}，气温：{temperature_celsius:.2f}摄氏度，风速：{wind_speed} m/s。"
-
- return f"天气预报信息：{weather_description}"
- else:
- return f"无法获取天气预报信息，状态码：{response.status_code}"
- except Exception as e:
- return f"发生错误：{str(e)}"
-
-
-if __name__ == "__main__":
- weather_tool = Weather()
- weather_info = weather_tool.run("Guangzhou")
- print(weather_info)
diff --git a/agent/tools/getOnRunLinkage.py b/agent/tools/getOnRunLinkage.py
deleted file mode 100644
index 1a8a124..0000000
--- a/agent/tools/getOnRunLinkage.py
+++ /dev/null
@@ -1,63 +0,0 @@
-import os
-from typing import Any
-from itertools import groupby
-
-from langchain.tools import BaseTool
-import tools.IotmService as IotmService
-
-class getOnRunLinkage(BaseTool):
- name = "getOnRunLinkage"
- description = "此工具用于查询农业箱当前在运行的联动"
-
- def __init__(self):
- super().__init__()
-
- async def _arun(self, *args: Any, **kwargs: Any) -> Any:
- # 用例中没有用到 arun 不予具体实现
- pass
-
-
- def _run(self, para: str) -> str:
- logs = IotmService.get_on_run_linkage()
- desc_list = {
- 'co2_S36': '二氧化碳',
- 'light_bh1': '箱内的光照强度',
- 'air_S37': '污染气体',
- 'nh3_S37': '氨气',
- 'temperature_MP14': '箱外温度',
- 'temperature_MP21': '箱内温度',
- 'humidity_MP14': '箱外湿度',
- 'humidity_S34': '箱内土壤的湿度',
- }
- infos = {}
-
-
- logs.sort(key=lambda x: (x['label'], x['port']))
-
- for (sensor_type, port), group in groupby(logs, key=lambda x: (x['label'], x['port'])):
-
- group_infos = []
- for val in group:
-
- onoff = '开启设备开关' if val['onoff'] == 1 else '关闭设备开关'
-
- info = {
- 'max': val['maxVal'],
- 'min': val['minVal'],
- 'onoff': onoff,
- }
- if float(val['keeptime']) > 0:
- info["持续时间（若需执行开启设备，持续时间过后执行关闭）,单位为分钟"] = val['keeptime']
- if float(val['delaytime']) > 0:
- info["执行后下次检查相距时间,单位为分钟"] = val['delaytime']
-
- group_infos.append(info)
-
- key_str = f"{sensor_type}_{port}"
- infos[desc_list.get(key_str, 'Unknown')] = group_infos
-
- return infos
-if __name__ == "__main__":
- tool = getOnRunLinkage()
- info = tool.run("")
- print(info)
diff --git a/ai_module/ali_nls.py b/ai_module/ali_nls.py
deleted file mode 100644
index f554030..0000000
--- a/ai_module/ali_nls.py
+++ /dev/null
@@ -1,184 +0,0 @@
-from threading import Thread
-
-import websocket
-import json
-import time
-import ssl
-import _thread as thread
-from aliyunsdkcore.client import AcsClient
-from aliyunsdkcore.request import CommonRequest
-
-from core import wsa_server
-from scheduler.thread_manager import MyThread
-from utils import util
-from utils import config_util as cfg
-
-__running = True
-__my_thread = None
-_token = ''
-
-
-def __post_token():
- global _token
- __client = AcsClient(
- cfg.key_ali_nls_key_id,
- cfg.key_ali_nls_key_secret,
- "cn-shanghai"
- )
-
- __request = CommonRequest()
- __request.set_method('POST')
- __request.set_domain('nls-meta.cn-shanghai.aliyuncs.com')
- __request.set_version('2019-02-28')
- __request.set_action_name('CreateToken')
- _token = json.loads(__client.do_action_with_exception(__request))['Token']['Id']
-
-
-def __runnable():
- while __running:
- __post_token()
- time.sleep(60 * 60 * 12)
-
-
-def start():
- MyThread(target=__runnable).start()
-
-
-class ALiNls:
- # 初始化
- def __init__(self):
- self.__URL = 'wss://nls-gateway-cn-shenzhen.aliyuncs.com/ws/v1'
- self.__ws = None
- self.__connected = False
- self.__frames = []
- self.__state = 0
- self.__closing = False
- self.__task_id = ''
- self.done = False
- self.finalResults = ""
-
- def __create_header(self, name):
- if name == 'StartTranscription':
- self.__task_id = util.random_hex(32)
- header = {
- "appkey": cfg.key_ali_nls_app_key,
- "message_id": util.random_hex(32),
- "task_id": self.__task_id,
- "namespace": "SpeechTranscriber",
- "name": name
- }
- return header
-
- def __on_msg(self):
- pass
-
- # 收到websocket消息的处理
- def on_message(self, ws, message):
- try:
- data = json.loads(message)
- header = data['header']
- name = header['name']
- if name == 'SentenceEnd':
- self.done = True
- self.finalResults = data['payload']['result']
- wsa_server.get_web_instance().add_cmd({"panelMsg": self.finalResults})
- if not cfg.config["interact"]["playSound"]: # 非展板播放
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': self.finalResults}}
- wsa_server.get_instance().add_cmd(content)
- self.__on_msg()
- elif name == 'TranscriptionResultChanged':
- self.finalResults = data['payload']['result']
- wsa_server.get_web_instance().add_cmd({"panelMsg": self.finalResults})
- if not cfg.config["interact"]["playSound"]: # 非展板播放
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': self.finalResults}}
- wsa_server.get_instance().add_cmd(content)
- self.__on_msg()
-
- except Exception as e:
- print(e)
- # print("### message:", message)
- if self.__closing:
- try:
- self.__ws.close()
- except Exception as e:
- print(e)
-
- # 收到websocket错误的处理
- def on_close(self, ws, code, msg):
- self.__connected = False
- print("### CLOSE:", msg)
-
- # 收到websocket错误的处理
- def on_error(self, ws, error):
- print("### error:", error)
-
- # 收到websocket连接建立的处理
- def on_open(self, ws):
- self.__connected = True
-
- # print("连接上了！！！")
-
- def run(*args):
- while self.__connected:
- try:
- if len(self.__frames) > 0:
- frame = self.__frames[0]
-
- self.__frames.pop(0)
- if type(frame) == dict:
- ws.send(json.dumps(frame))
- elif type(frame) == bytes:
- ws.send(frame, websocket.ABNF.OPCODE_BINARY)
- #print('发送 ------> ' + str(type(frame)))
- except Exception as e:
- print(e)
- time.sleep(0.04)
-
- thread.start_new_thread(run, ())
-
- def __connect(self):
- self.finalResults = ""
- self.done = False
- self.__frames.clear()
- self.__ws = websocket.WebSocketApp(self.__URL + '?token=' + _token, on_message=self.on_message)
- self.__ws.on_open = self.on_open
- self.__ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
-
- def add_frame(self, frame):
- self.__frames.append(frame)
-
- def send(self, buf):
- self.__frames.append(buf)
-
- def start(self):
- Thread(target=self.__connect, args=[]).start()
- data = {
- 'header': self.__create_header('StartTranscription'),
- "payload": {
- "format": "pcm",
- "sample_rate": 16000,
- "enable_intermediate_result": True,
- "enable_punctuation_prediction": False,
- "enable_inverse_text_normalization": True,
- "speech_noise_threshold": -1
- }
- }
- self.add_frame(data)
-
- def end(self):
- if self.__connected:
- try:
- for frame in self.__frames:
- self.__frames.pop(0)
- if type(frame) == dict:
- self.__ws.send(json.dumps(frame))
- elif type(frame) == bytes:
- self.__ws.send(frame, websocket.ABNF.OPCODE_BINARY)
- time.sleep(0.4)
- self.__frames.clear()
- frame = {"header": self.__create_header('StopTranscription')}
- self.__ws.send(json.dumps(frame))
- except Exception as e:
- print(e)
- self.__closing = True
- self.__connected = False
diff --git a/ai_module/funasr.py b/ai_module/funasr.py
deleted file mode 100644
index 7a6f8b0..0000000
--- a/ai_module/funasr.py
+++ /dev/null
@@ -1,122 +0,0 @@
-"""
-感谢北京中科大脑神经算法工程师张聪聪提供funasr集成代码
-"""
-from threading import Thread
-import websocket
-import json
-import time
-import ssl
-import _thread as thread
-
-from core import wsa_server
-from utils import config_util as cfg
-
-class FunASR:
- # 初始化
- def __init__(self):
- self.__URL = "ws://{}:{}".format(cfg.local_asr_ip, cfg.local_asr_port)
- self.__ws = None
- self.__connected = False
- self.__frames = []
- self.__state = 0
- self.__closing = False
- self.__task_id = ''
- self.done = False
- self.finalResults = ""
-
-
- def __on_msg(self):
- pass
-
- # 收到websocket消息的处理
- def on_message(self, ws, message):
- try:
- self.done = True
- self.finalResults = message
- wsa_server.get_web_instance().add_cmd({"panelMsg": self.finalResults})
- if not cfg.config["interact"]["playSound"]: # 非展板播放
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': self.finalResults}}
- wsa_server.get_instance().add_cmd(content)
- self.__on_msg()
-
- except Exception as e:
- print(e)
-
- if self.__closing:
- try:
- self.__ws.close()
- except Exception as e:
- print(e)
-
- # 收到websocket错误的处理
- def on_close(self, ws, code, msg):
- self.__connected = False
- print("### CLOSE:", msg)
-
- # 收到websocket错误的处理
- def on_error(self, ws, error):
- print("### error:", error)
-
- # 收到websocket连接建立的处理
- def on_open(self, ws):
- self.__connected = True
-
- def run(*args):
- while self.__connected:
- try:
- if len(self.__frames) > 0:
- frame = self.__frames[0]
-
- self.__frames.pop(0)
- if type(frame) == dict:
- ws.send(json.dumps(frame))
- elif type(frame) == bytes:
- ws.send(frame, websocket.ABNF.OPCODE_BINARY)
- # print('发送 ------> ' + str(type(frame)))
- except Exception as e:
- print(e)
- time.sleep(0.04)
-
- thread.start_new_thread(run, ())
-
- def __connect(self):
- self.finalResults = ""
- self.done = False
- self.__frames.clear()
- websocket.enableTrace(False)
- self.__ws = websocket.WebSocketApp(self.__URL, on_message=self.on_message,on_close=self.on_close,on_error=self.on_error,subprotocols=["binary"])
- self.__ws.on_open = self.on_open
-
- self.__ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
-
- def add_frame(self, frame):
- self.__frames.append(frame)
-
- def send(self, buf):
- self.__frames.append(buf)
-
- def start(self):
- Thread(target=self.__connect, args=[]).start()
- data = {
- 'vad_need':False,
- 'state':'StartTranscription'
- }
- self.add_frame(data)
-
- def end(self):
- if self.__connected:
- try:
- for frame in self.__frames:
- self.__frames.pop(0)
- if type(frame) == dict:
- self.__ws.send(json.dumps(frame))
- elif type(frame) == bytes:
- self.__ws.send(frame, websocket.ABNF.OPCODE_BINARY)
- time.sleep(0.4)
- self.__frames.clear()
- frame = {'vad_need':False,'state':'StopTranscription'}
- self.__ws.send(json.dumps(frame))
- except Exception as e:
- print(e)
- self.__closing = True
- self.__connected = False
diff --git a/ai_module/nlp_cemotion.py b/ai_module/nlp_cemotion.py
deleted file mode 100644
index 2b93fae..0000000
--- a/ai_module/nlp_cemotion.py
+++ /dev/null
@@ -1,10 +0,0 @@
-
-def get_sentiment(c,text):
- try:
- return c.predict(text)
- except BaseException as e:
- print("请稍后")
- print(e)
-
-
-
diff --git a/ai_module/openai_tts.py b/ai_module/openai_tts.py
deleted file mode 100644
index ffa8aed..0000000
--- a/ai_module/openai_tts.py
+++ /dev/null
@@ -1,59 +0,0 @@
-import requests
-import json
-import time
-from utils import util, config_util
-
-class Speech:
- def __init__(self):
- self.api_key = config_util.key_gpt_api_key
- self.history_data = []
-
- def __get_history(self, text):
- for data in self.history_data:
- if data[0] == text:
- return data[1]
- return None
-
- def connect(self):
- pass
-
- def close(self):
- pass
-
- def to_sample(self, text, voice="nova", response_format="mp3", speed=1):
- history = self.__get_history(text)
- if history is not None:
- return history
-
- headers = {
- 'Authorization': f'Bearer {self.api_key}',
- 'Content-Type': 'application/json'
- }
- url = "https://api.openai.com/v1/audio/speech"
- query = {
- "model": "tts-1-hd",#tts-1、tts-1-hd
- "input": text,
- "voice": voice,
- "response_format": response_format,
- "speed": speed
- }
- try:
- response = requests.post(url=url, data=json.dumps(query), headers=headers)
-
- file_url = './samples/sample-' + str(int(time.time() * 1000)) + '.mp3'
- with open(file_url, "wb") as audio_file:
- audio_file.write(response.content)
-
- self.history_data.append((text, file_url))
- except Exception as e :
- util.log(1, "[x] 语音转换失败！")
- util.log(1, "[x] 原因: " + str(str(e)))
- file_url = None
- return file_url
-
-if __name__ == '__main__':
- openai_tts = Speech(api_key='') # 替换为您的 OpenAI API Key
- text = "你好！我是FAY！今天天气真好！"
- audio_file_url = openai_tts.to_sample(text)
-
- print("音频文件已保存:", audio_file_url)
\ No newline at end of file
diff --git a/ai_module/xf_ltp.py b/ai_module/xf_ltp.py
deleted file mode 100644
index 51771f6..0000000
--- a/ai_module/xf_ltp.py
+++ /dev/null
@@ -1,59 +0,0 @@
-import time
-import urllib.request
-import urllib.parse
-import json
-import hashlib
-import base64
-from utils import config_util as cfg
-
-__URL = "https://ltpapi.xfyun.cn/v2/sa"
-
-
-def __quest(text):
- body = urllib.parse.urlencode({'text': text}).encode('utf-8')
- param = {"type": "dependent"}
- x_param = base64.b64encode(json.dumps(param).replace(' ', '').encode('utf-8'))
- x_time = str(int(time.time()))
- x_checksum = hashlib.md5(cfg.key_xf_ltp_api_key.encode('utf-8') + str(x_time).encode('utf-8') + x_param).hexdigest()
- x_header = {
- 'X-Appid': cfg.key_xf_ltp_app_id,
- 'X-CurTime': x_time,
- 'X-Param': x_param,
- 'X-CheckSum': x_checksum
- }
- req = urllib.request.Request(__URL, body, x_header)
- result = urllib.request.urlopen(req)
- result = result.read()
- return json.loads(result.decode('utf-8'))
-
-
-"""
-情感分析
-
-:param text: 文本
-
-:returns: 情感分数 (0.7以上为褒义, 0.3-0.7为中性 0.3以下为贬义,, -1为分析失败)
-"""
-
-
-def get_score(text):
- result = __quest(text)
- if result['desc'] == 'success':
- return float(result['data']['score'])
- return -1
-
-
-"""
-情感分析
-
-:param text: 文本
-
-:returns: 情感极性分类 (2为褒义, 1为中性 0为贬义,, -1为分析失败)
-"""
-
-
-def get_sentiment(text):
- result = __quest(text)
- if result['desc'] == 'success':
- return int(result['data']['sentiment']) + 1
- return -1
diff --git a/ai_module/yolov8.py b/ai_module/yolov8.py
deleted file mode 100644
index d20b799..0000000
--- a/ai_module/yolov8.py
+++ /dev/null
@@ -1,148 +0,0 @@
-from ultralytics import YOLO
-from scipy.spatial import procrustes
-import numpy as np
-import cv2
-import time
-from scheduler.thread_manager import MyThread
-
-__fei_eyes = None
-class FeiEyes:
-
- def __init__(self):
-
- """
- 鼻子（0）
- 左眼（1），右眼（2）
- 左耳（3），右耳（4）
- 左肩（5），右肩（6）
- 左肘（7），右肘（8）
- 左腕（9），右腕（10）
- 左髋（11），右髋（12）
- 左膝（13），右膝（14）
- 左脚踝（15），右脚踝（16）
- """
- self.POSE_PAIRS = [
- (3, 5), (5, 6), # upper body
- (5, 7), (6, 8), (7, 9), (8, 10), # lower body
- (11, 12), (11, 13), (12, 14), (13, 15) # arms
- ]
- self.my_face = np.array([[154.4565, 193.7006],
- [181.8575, 164.8366],
- [117.1820, 164.3602],
- [213.5605, 193.0460],
- [ 62.7056, 193.5217]])
- self.is_running = False
- self.img = None
-
- def is_sitting(self, keypoints):
- if len(keypoints) < 17: # 确保有足够的关键点
- return False
- # 检查每个关键点的置信度
- if keypoints[11][2] < 0.5 or keypoints[12][2] < 0.5 or keypoints[13][2] < 0.5 or keypoints[14][2] < 0.5 or keypoints[15][2] < 0.5 or keypoints[16][2] < 0.5:
- return False
-
- left_hip, right_hip = keypoints[11][:2], keypoints[12][:2]
- left_knee, right_knee = keypoints[13][:2], keypoints[14][:2]
- left_ankle, right_ankle = keypoints[15][:2], keypoints[16][:2]
-
- hip_knee_y = (left_hip[1] + right_hip[1] + left_knee[1] + right_knee[1]) / 4
- knee_ankle_y = (left_knee[1] + right_knee[1] + left_ankle[1] + right_ankle[1]) / 4
-
- return hip_knee_y < knee_ankle_y
-
- def is_standing(self, keypoints):
- if len(keypoints) < 17 or keypoints[0][2] < 0.5 or keypoints[15][2] < 0.5 or keypoints[16][2] < 0.5:
- return False
-
- head = keypoints[0][:2]
- left_ankle, right_ankle = keypoints[15][:2], keypoints[16][:2]
-
- return head[1] > left_ankle[1] and head[1] > right_ankle[1]
-
- def get_counts(self):
- if not self.is_running:
- return 0,0,0
- return self.person_count, self.stand_count, self.sit_count
-
- def get_status(self):
- return self.is_running
-
- def get_img(self):
- if self.is_running:
- return self.img
- else:
- return None
-
- def start(self):
- cap = cv2.VideoCapture(0)
- if cap.isOpened():
- self.is_running = True
- MyThread(target=self.run, args=[cap]).start()
-
- def stop(self):
- self.is_running = False
-
- def run(self, cap):
- model = YOLO("yolov8n-pose.pt")
- while self.is_running:
- time.sleep(0.033)
- ret, frame = cap.read()
- self.img = frame
- operated_frame = frame.copy()
- if not ret:
- break
- results = model.predict(operated_frame, verbose=False)
- person_count = 0
- sit_count = 0
- stand_count = 0
- for res in results: # loop over results
- for box, cls in zip(res.boxes.xyxy, res.boxes.cls): # loop over detections
- x1, y1, x2, y2 = box
- cv2.rectangle(operated_frame, (int(x1.item()), int(y1.item())), (int(x2.item()), int(y2.item())), (0, 255, 0), 2)
- cv2.putText(operated_frame, f"{res.names[int(cls.item())]}", (int(x1.item()), int(y1.item()) - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
- if res.keypoints is not None and res.keypoints.xy.numel() > 0: # check if keypoints exist
- keypoints = res.keypoints[0]
- #总人数
- person_count += 1
- #坐着的人数
- if self.is_sitting(keypoints):
- sit_count += 1
- #站着的人数
- elif self.is_standing(keypoints):
- stand_count += 1
-
- for keypoint in keypoints: # loop over keypoints
-
- if len(keypoint) == 3:
- x, y, conf = keypoint
- if conf > 0.5: # draw keypoints with confidence greater than 0.5
- cv2.circle(operated_frame, (int(x.item()), int(y.item())), 3, (0, 0, 255), -1)
-
- # Draw lines connecting keypoints
- for pair in self.POSE_PAIRS:
- if pair[0] < len(keypoints) and pair[1] < len(keypoints):
- pt1, pt2 = keypoints[pair[0]][:2], keypoints[pair[1]][:2]
- conf1, conf2 = keypoints[pair[0]][2], keypoints[pair[1]][2]
- if conf1 > 0.5 and conf2 > 0.5:
- # cv2.line(operated_frame, (int(pt1[0].item()), int(pt1[1].item())), (int(pt2[0].item()), int(pt2[1].item())), (255, 255, 0), 2)
- pass
- self.person_count = person_count
- self.sit_count = sit_count
- self.stand_count = stand_count
- cv2.imshow("YOLO v8 Fay Eyes", operated_frame)
- cv2.waitKey(1)
-
- cap.release()
- cv2.destroyAllWindows()
-
-
-def new_instance():
- global __fei_eyes
- if __fei_eyes is None:
- __fei_eyes = FeiEyes()
- return __fei_eyes
-
-
-
-
-
diff --git a/config.json b/config.json
deleted file mode 100644
index efdde3e..0000000
--- a/config.json
+++ /dev/null
@@ -1,35 +0,0 @@
-{
- "attribute": {
- "age": "",
- "birth": "",
- "constellation": "",
- "contact": "",
- "gender": "",
- "hobby": "",
- "job": "",
- "name": "",
- "voice": "alloy",
- "zodiac": ""
- },
- "interact": {
- "QnA": "",
- "maxInteractTime": 15,
- "perception": {
- "chat": 0,
- "follow": 0,
- "gift": 0,
- "indifferent": 0,
- "join": 0
- },
- "playSound": true,
- "visualization": false
- },
- "items": [],
- "source": {
- "record": {
- "device": "\u9ea6\u514b\u98ce (HD Webcam C525)",
- "enabled": false
- },
- "wake_word_enabled": false
- }
-}
\ No newline at end of file
diff --git a/core/content_db.py b/core/content_db.py
deleted file mode 100644
index a534736..0000000
--- a/core/content_db.py
+++ /dev/null
@@ -1,84 +0,0 @@
-import sqlite3
-import time
-import threading
-import functools
-from utils import util
-def synchronized(func):
- @functools.wraps(func)
- def wrapper(self, *args, **kwargs):
- with self.lock:
- return func(self, *args, **kwargs)
- return wrapper
-class Content_Db:
-
- def __init__(self) -> None:
- self.lock = threading.Lock()
-
-
-
- #初始化
- def init_db(self):
- conn = sqlite3.connect('fay.db')
- c = conn.cursor()
- c.execute('''CREATE TABLE IF NOT EXISTS T_Msg
- (id INTEGER PRIMARY KEY autoincrement,
- type char(10),
- way char(10),
- content TEXT NOT NULL,
- createtime Int);''')
- conn.commit()
- conn.close()
-
-
-
-
- #添加对话
- @synchronized
- def add_content(self,type,way,content):
- conn = sqlite3.connect("fay.db")
- cur = conn.cursor()
- try:
- cur.execute("insert into T_Msg (type,way,content,createtime) values (?,?,?,?)",(type,way,content,int(time.time())))
- conn.commit()
- except:
- util.log(1, "请检查参数是否有误")
- conn.close()
- return 0
- conn.close()
- return cur.lastrowid
-
-
-
- #获取对话内容
- @synchronized
- def get_list(self,way,order,limit):
- conn = sqlite3.connect("fay.db")
- cur = conn.cursor()
- if(way == 'all'):
- cur.execute("select type,way,content,createtime,datetime(createtime, 'unixepoch', 'localtime') as timetext from T_Msg order by id "+order+" limit ?",(limit,))
- elif(way == 'notappended'):
- cur.execute("select type,way,content,createtime,datetime(createtime, 'unixepoch', 'localtime') as timetext from T_Msg where way != 'appended' order by id "+order+" limit ?",(limit,))
- else:
- cur.execute("select type,way,content,createtime,datetime(createtime, 'unixepoch', 'localtime') as timetext from T_Msg where way = ? order by id "+order+" limit ?",(way,limit,))
-
- list = cur.fetchall()
- conn.close()
- return list
-
-
-
-
-
-# a = Content_Db()
-# s = a.get_list('all','desc',10)
-# print(s)
-
-
-
-
-
-
-
-
-
-
diff --git a/core/fay_core.py b/core/fay_core.py
deleted file mode 100644
index ba90676..0000000
--- a/core/fay_core.py
+++ /dev/null
@@ -1,348 +0,0 @@
-
-import math
-import os
-import time
-import socket
-
-import eyed3
-import logging
-
-
-# 适应模型使用
-import numpy as np
-import fay_booter
-from ai_module import xf_ltp
-from ai_module.openai_tts import Speech
-from core import wsa_server
-from core.interact import Interact
-from scheduler.thread_manager import MyThread
-from utils import util, config_util
-
-import pygame
-from utils import config_util as cfg
-from ai_module import nlp_cemotion
-import platform
-from ai_module import yolov8
-from agent import agent_service
-import fay_booter
-from core.content_db import Content_Db
-if platform.system() == "Windows":
- import sys
- sys.path.append("test/ovr_lipsync")
- from test_olipsync import LipSyncGenerator
-
-
-#文本消息处理（20231211增加：agent操作）
-def send_for_answer(msg):
- #记录运行时间
- fay_booter.feiFei.last_quest_time = time.time()
-
- #消息保存
- contentdb = Content_Db()
- contentdb.add_content('member', 'agent', msg.replace('主人语音说了：', '').replace('主人文字说了：', ''))
- wsa_server.get_web_instance().add_cmd({"panelReply": {"type":"member","content":msg.replace('主人语音说了：', '').replace('主人文字说了：', '')}})
-
- # 发送给数字人端
- if not config_util.config["interact"]["playSound"]:
- content = {'Topic': 'Unreal', 'Data': {'Key': 'question', 'Value': msg}}
- wsa_server.get_instance().add_cmd(content)
-
- #思考中...
- wsa_server.get_web_instance().add_cmd({"panelMsg": "思考中..."})
- if not cfg.config["interact"]["playSound"]: # 非展板播放
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': "思考中..."}}
- wsa_server.get_instance().add_cmd(content)
-
- #agent 或llm chain处理
- is_use_say_tool, text = agent_service.agent.run(msg)
-
- #语音输入强制语音输出
- if text and "语音说了" in msg and not is_use_say_tool:
- interact = Interact("audio", 1, {'user': '', 'msg': text})
- fay_booter.feiFei.on_interact(interact)
-
- #消息保存
- contentdb.add_content('fay','agent', text)
- wsa_server.get_web_instance().add_cmd({"panelReply": {"type":"fay","content":text}})
- util.log(1, 'ReAct Agent或LLM Chain处理总时长：{} ms'.format(math.floor((time.time() - fay_booter.feiFei.last_quest_time) * 1000)))
-
- #推送数字人
- if not cfg.config["interact"]["playSound"]:
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': text}}
- wsa_server.get_instance().add_cmd(content)
- if not config_util.config["interact"]["playSound"]:
- content = {'Topic': 'Unreal', 'Data': {'Key': 'text', 'Value': text}}
- wsa_server.get_instance().add_cmd(content)
-
- return text
-
-
-class FeiFei:
- def __init__(self):
- pygame.mixer.init()
- self.q_msg = '你叫什么名字？'
- self.a_msg = 'hi,我叫菲菲，英文名是fay'
- self.mood = 0.0 # 情绪值
- self.old_mood = 0.0
- self.connect = False
- self.item_index = 0
- self.deviceSocket = None
- self.deviceConnect = None
-
- #启动音频输入输出设备的连接服务
- self.deviceSocketThread = MyThread(target=self.__accept_audio_device_output_connect)
- self.deviceSocketThread.start()
-
- self.X = np.array([1, 0, 0, 0, 0, 0, 0, 0]).reshape(1, -1) # 适应模型变量矩阵
- # self.W = np.array([0.01577594,1.16119452,0.75828,0.207746,1.25017864,0.1044121,0.4294899,0.2770932]).reshape(-1,1) #适应模型变量矩阵
- self.W = np.array([0.0, 0.6, 0.1, 0.7, 0.3, 0.0, 0.0, 0.0]).reshape(-1, 1) # 适应模型变量矩阵
-
- self.wsParam = None
- self.wss = None
- self.sp = Speech()
- self.speaking = False
- self.interactive = []
- self.sleep = False
- self.__running = True
- self.sp.connect() # 预连接
- self.last_quest_time = time.time()
- self.playing = False
- self.muting = False
- self.cemotion = None
-
-
- def __auto_speak(self):
- while self.__running:
- time.sleep(0.1)
- if self.speaking or self.sleep:
- continue
-
- try:
- if len(self.interactive) > 0:
- interact: Interact = self.interactive.pop()
- #开启fay eyes，无人时不回复
- fay_eyes = yolov8.new_instance()
- if fay_eyes.get_status():#YOLO正在运行
- person_count, stand_count, sit_count = fay_eyes.get_counts()
- if person_count < 1: #看不到人，不互动
- wsa_server.get_web_instance().add_cmd({"panelMsg": "看不到人，不互动"})
- if not cfg.config["interact"]["playSound"]: # 非展板播放
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': "看不到人，不互动"}}
- wsa_server.get_instance().add_cmd(content)
- continue
-
- self.speaking = True
- self.a_msg = interact.data["msg"]
- MyThread(target=self.__say, args=['interact']).start()
-
- except BaseException as e:
- print(e)
-
- def on_interact(self, interact: Interact):
- self.interactive.append(interact)
- MyThread(target=self.__update_mood, args=[interact.interact_type]).start()
-
-
- # 适应模型计算(用于学习真人的性格特质，开源版本暂不使用)
- def __fay(self, index):
- if 0 < index < 8:
- self.X[0][index] += 1
- # PRED = 1 /(1 + tf.exp(-tf.matmul(tf.constant(self.X,tf.float32), tf.constant(self.W,tf.float32))))
- PRED = np.sum(self.X.reshape(-1) * self.W.reshape(-1))
- if 0 < index < 8:
- print('***PRED:{0}***'.format(PRED))
- print(self.X.reshape(-1) * self.W.reshape(-1))
- return PRED
-
- # 发送情绪
- def __send_mood(self):
- while self.__running:
- time.sleep(3)
- if not self.sleep and not config_util.config["interact"]["playSound"] and wsa_server.get_instance().isConnect:
- content = {'Topic': 'Unreal', 'Data': {'Key': 'mood', 'Value': self.mood}}
- if not self.connect:
- wsa_server.get_instance().add_cmd(content)
- self.connect = True
- else:
- if self.old_mood != self.mood:
- wsa_server.get_instance().add_cmd(content)
- self.old_mood = self.mood
-
- else:
- self.connect = False
-
- # 更新情绪
- def __update_mood(self, typeIndex):
- perception = config_util.config["interact"]["perception"]
- if typeIndex == 1:
- try:
- if cfg.ltp_mode == "cemotion":
- result = nlp_cemotion.get_sentiment(self.cemotion,self.q_msg)
- chat_perception = perception["chat"]
- if result >= 0.5 and result <= 1:
- self.mood = self.mood + (chat_perception / 200.0)
- elif result <= 0.2:
- self.mood = self.mood - (chat_perception / 100.0)
- else:
- result = xf_ltp.get_sentiment(self.q_msg)
- chat_perception = perception["chat"]
- if result == 1:
- self.mood = self.mood + (chat_perception / 200.0)
- elif result == -1:
- self.mood = self.mood - (chat_perception / 100.0)
- except BaseException as e:
- print("[System] 情绪更新错误！")
- print(e)
-
- elif typeIndex == 2:
- self.mood = self.mood + (perception["join"] / 100.0)
-
- elif typeIndex == 3:
- self.mood = self.mood + (perception["gift"] / 100.0)
-
- elif typeIndex == 4:
- self.mood = self.mood + (perception["follow"] / 100.0)
-
- if self.mood >= 1:
- self.mood = 1
- if self.mood <= -1:
- self.mood = -1
-
- def __get_mood_voice(self):
- voice = config_util.config["attribute"]["voice"]
- return voice
-
- # 合成声音
- def __say(self, styleType):
- try:
- if len(self.a_msg) < 1:
- self.speaking = False
- else:
- util.printInfo(1, '菲菲', '({}) {}'.format(self.__get_mood_voice(), self.a_msg))
- util.log(1, '合成音频...')
- tm = time.time()
- #文字也推送出去，为了ue5
- result = self.sp.to_sample(self.a_msg, self.__get_mood_voice())
- util.log(1, '合成音频完成. 耗时: {} ms 文件:{}'.format(math.floor((time.time() - tm) * 1000), result))
- if result is not None:
- MyThread(target=self.__send_or_play_audio, args=[result, styleType]).start()
- return result
- except BaseException as e:
- print(e)
- self.speaking = False
- return None
-
- def __play_sound(self, file_url):
- util.log(1, '播放音频...')
- util.log(1, 'agent处理总时长：{} ms'.format(math.floor((time.time() - self.last_quest_time) * 1000)))
- pygame.mixer.music.load(file_url)
- pygame.mixer.music.play()
-
-
- def __send_or_play_audio(self, file_url, say_type):
- try:
- try:
- logging.getLogger('eyed3').setLevel(logging.ERROR)
- audio_length = eyed3.load(file_url).info.time_secs #mp3音频长度
- except Exception as e:
- audio_length = 3
-
- # with wave.open(file_url, 'rb') as wav_file: #wav音频长度
- # audio_length = wav_file.getnframes() / float(wav_file.getframerate())
- # print(audio_length)
- # if audio_length <= config_util.config["interact"]["maxInteractTime"] or say_type == "script":
- if config_util.config["interact"]["playSound"]: # 展板播放
- self.__play_sound(file_url)
- else:#发送音频给ue
- #推送ue
- content = {'Topic': 'Unreal', 'Data': {'Key': 'audio', 'Value': os.path.abspath(file_url), 'Text': self.a_msg, 'Time': audio_length, 'Type': say_type}}
- #计算lips
- if platform.system() == "Windows":
- try:
- lip_sync_generator = LipSyncGenerator()
- viseme_list = lip_sync_generator.generate_visemes(os.path.abspath(file_url))
- consolidated_visemes = lip_sync_generator.consolidate_visemes(viseme_list)
- content["Data"]["Lips"] = consolidated_visemes
- except Exception as e:
- util.log(1, "唇型数字生成失败，无法使用新版ue5工程")
- wsa_server.get_instance().add_cmd(content)
-
- #推送远程音频
- if self.deviceConnect is not None:
- try:
- self.deviceConnect.send(b'\x00\x01\x02\x03\x04\x05\x06\x07\x08') # 发送音频开始标志，同时也检查设备是否在线
- wavfile = open(os.path.abspath(file_url),'rb')
- data = wavfile.read(1024)
- total = 0
- while data:
- total += len(data)
- self.deviceConnect.send(data)
- data = wavfile.read(1024)
- time.sleep(0.001)
- self.deviceConnect.send(b'\x08\x07\x06\x05\x04\x03\x02\x01\x00')# 发送音频结束标志
- util.log(1, "远程音频发送完成：{}".format(total))
- except socket.error as serr:
- util.log(1,"远程音频输入输出设备已经断开：{}".format(serr))
-
- time.sleep(audio_length + 0.5)
- wsa_server.get_web_instance().add_cmd({"panelMsg": ""})
- if not cfg.config["interact"]["playSound"]: # 非展板播放
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': ""}}
- wsa_server.get_instance().add_cmd(content)
- if config_util.config["interact"]["playSound"]:
- util.log(1, '结束播放！')
- self.speaking = False
- except Exception as e:
- print(e)
-
- def __device_socket_keep_alive(self):
- while True:
- if self.deviceConnect is not None:
- try:
- self.deviceConnect.send(b'\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8')#发送心跳包
- except Exception as serr:
- util.log(1,"远程音频输入输出设备已经断开：{}".format(serr))
- self.deviceConnect = None
- time.sleep(5)
-
- def __accept_audio_device_output_connect(self):
- self.deviceSocket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
- self.deviceSocket.bind(("0.0.0.0",10001))
- self.deviceSocket.listen(1)
- addr = None
- try:
- while True:
- self.deviceConnect,addr=self.deviceSocket.accept() #接受TCP连接，并返回新的套接字与IP地址
- MyThread(target=self.__device_socket_keep_alive).start() # 开启心跳包检测
- util.log(1,"远程音频输入输出设备连接上：{}".format(addr))
- while self.deviceConnect: #只允许一个设备连接
- time.sleep(1)
- except Exception as err:
- pass
-
- def set_sleep(self, sleep):
- self.sleep = sleep
-
- def start(self):
- if cfg.ltp_mode == "cemotion":
- from cemotion import Cemotion
- self.cemotion = Cemotion()
- MyThread(target=self.__send_mood).start()
- MyThread(target=self.__auto_speak).start()
-
-
- def stop(self):
- self.__running = False
- self.speaking = False
- self.playing = False
- self.sp.close()
- wsa_server.get_web_instance().add_cmd({"panelMsg": ""})
- if not cfg.config["interact"]["playSound"]: # 非展板播放
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': ""}}
- wsa_server.get_instance().add_cmd(content)
- if self.deviceConnect is not None:
- self.deviceConnect.close()
- self.deviceConnect = None
- if self.deviceSocket is not None:
- self.deviceSocket.close()
-
diff --git a/core/interact.py b/core/interact.py
deleted file mode 100644
index ee1e836..0000000
--- a/core/interact.py
+++ /dev/null
@@ -1,6 +0,0 @@
-class Interact:
-
- def __init__(self, interleaver: str, interact_type: int, data: dict):
- self.interleaver = interleaver
- self.interact_type = interact_type
- self.data = data
diff --git a/core/recorder.py b/core/recorder.py
deleted file mode 100644
index b65b968..0000000
--- a/core/recorder.py
+++ /dev/null
@@ -1,232 +0,0 @@
-import audioop
-import math
-import time
-import threading
-from abc import abstractmethod
-
-from ai_module.ali_nls import ALiNls
-from ai_module.funasr import FunASR
-from core import wsa_server
-from scheduler.thread_manager import MyThread
-from utils import util
-from utils import config_util as cfg
-import numpy as np
-# 启动时间 (秒)
-_ATTACK = 0.2
-
-# 释放时间 (秒)
-_RELEASE = 0.75
-
-
-class Recorder:
-
- def __init__(self, fay):
- self.__fay = fay
-
-
-
- self.__running = True
- self.__processing = False
- self.__history_level = []
- self.__history_data = []
- self.__dynamic_threshold = 0.5 # 声音识别的音量阈值
-
- self.__MAX_LEVEL = 25000
- self.__MAX_BLOCK = 100
-
- #Edit by xszyou in 20230516:增加本地asr
- self.ASRMode = cfg.ASR_mode
- self.__aLiNls = self.asrclient()
- self.is_awake = False
- self.wakeup_matched = False
- if cfg.config['source']['wake_word_enabled']:
- self.timer = threading.Timer(60, self.reset_wakeup_status) # 60秒后执行reset_wakeup_status方法
-
- def asrclient(self):
- asrcli = None
- if self.ASRMode == "ali":
- asrcli = ALiNls()
- elif self.ASRMode == "funasr":
- asrcli = FunASR()
- return asrcli
-
- def __get_history_average(self, number):
- total = 0
- num = 0
- for i in range(len(self.__history_level) - 1, -1, -1):
- level = self.__history_level[i]
- total += level
- num += 1
- if num >= number:
- break
- return total / num
-
- def __get_history_percentage(self, number):
- return (self.__get_history_average(number) / self.__MAX_LEVEL) * 1.05 + 0.02
-
- def __print_level(self, level):
- text = ""
- per = level / self.__MAX_LEVEL
- if per > 1:
- per = 1
- bs = int(per * self.__MAX_BLOCK)
- for i in range(bs):
- text += "#"
- for i in range(self.__MAX_BLOCK - bs):
- text += "-"
- print(text + " [" + str(int(per * 100)) + "%]")
-
- def reset_wakeup_status(self):
- self.wakeup_matched = False
-
- def __waitingResult(self, iat: asrclient):
- if self.__fay.playing:
- return
- self.processing = True
- t = time.time()
- tm = time.time()
- # 等待结果返回
- while not iat.done and time.time() - t < 1:
- time.sleep(0.01)
- text = iat.finalResults
- util.log(1, "语音处理完成！ 耗时: {} ms".format(math.floor((time.time() - tm) * 1000)))
- if len(text) > 0:
- if cfg.config['source']['wake_word_enabled']:
- if not self.wakeup_matched:
- #唤醒词判断
- wake_word = cfg.config['source']['wake_word']
- wake_word_list = wake_word.split(',')
- wake_up = False
- for word in wake_word_list:
- if word in text:
- wake_up = True
-
- if wake_up:
- self.wakeup_matched = True # 唤醒成功
- util.log(1, "唤醒成功！")
- self.on_speaking('唤醒')
- self.processing = False
- self.timer.cancel() # 取消之前的计时器任务
- else:
- util.log(1, "[!] 待唤醒！")
- wsa_server.get_web_instance().add_cmd({"panelMsg": ""})
-
- else:
- self.on_speaking(text)
- self.processing = False
- self.timer.cancel() # 取消之前的计时器任务
- self.timer = threading.Timer(60, self.reset_wakeup_status) # 重设计时器为60秒
- self.timer.start()
- else:
- self.on_speaking(text)
- self.processing = False
- else:
- if self.wakeup_matched:
- self.wakeup_matched = False
- util.log(1, "[!] 语音未检测到内容！")
- self.processing = False
- self.dynamic_threshold = self.__get_history_percentage(30)
- wsa_server.get_web_instance().add_cmd({"panelMsg": ""})
- if not cfg.config["interact"]["playSound"]: # 非展板播放
- content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': ""}}
- wsa_server.get_instance().add_cmd(content)
-
- def __record(self):
- try:
- stream = self.get_stream() #把get stream的方式封装出来方便实现麦克风录制及网络流等不同的流录制子类
- except Exception as e:
- print(e)
- util.log(1, "请检查设备是否有误，再重新启动!")
- return
- isSpeaking = False
- last_mute_time = time.time()
- last_speaking_time = time.time()
- data = None
- while self.__running:
- try:
- data = stream.read(1024, exception_on_overflow=False)
- except Exception as e:
- data = None
- print(e)
- util.log(1, "请检查设备是否有误，再重新启动!")
- return
- if not data:
- continue
-
- if cfg.config['source']['record']['enabled'] and not self.is_remote():
- if len(cfg.config['source']['record'])<3:
- channels = 1
- else:
- channels = int(cfg.config['source']['record']['channels'])
- #只获取第一声道
- data = np.frombuffer(data, dtype=np.int16)
- data = np.reshape(data, (-1, channels)) # reshaping the array to split the channels
- mono = data[:, 0] # taking the first channel
- data = mono.tobytes()
-
- level = audioop.rms(data, 2)
- if len(self.__history_data) >= 5:
- self.__history_data.pop(0)
- if len(self.__history_level) >= 500:
- self.__history_level.pop(0)
- self.__history_data.append(data)
- self.__history_level.append(level)
-
- percentage = level / self.__MAX_LEVEL
- history_percentage = self.__get_history_percentage(30)
-
- if history_percentage > self.__dynamic_threshold:
- self.__dynamic_threshold += (history_percentage - self.__dynamic_threshold) * 0.0025
- elif history_percentage < self.__dynamic_threshold:
- self.__dynamic_threshold += (history_percentage - self.__dynamic_threshold) * 1
-
- soon = False
- if percentage > self.__dynamic_threshold and not self.__fay.speaking:
- last_speaking_time = time.time()
- if not self.__processing and not isSpeaking and time.time() - last_mute_time > _ATTACK:
- soon = True #
- isSpeaking = True #用户正在说话
- util.log(3, "聆听中...")
- self.__aLiNls = self.asrclient()
- try:
- self.__aLiNls.start()
- except Exception as e:
- print(e)
- for buf in self.__history_data:
- self.__aLiNls.send(buf)
- else:
- last_mute_time = time.time()
- if isSpeaking:
- if time.time() - last_speaking_time > _RELEASE:
- isSpeaking = False
- self.__aLiNls.end()
- util.log(1, "语音处理中...")
- self.__fay.last_quest_time = time.time()
- self.__waitingResult(self.__aLiNls)
- if not soon and isSpeaking:
- self.__aLiNls.send(data)
-
-
- def set_processing(self, processing):
- self.__processing = processing
-
- def start(self):
- MyThread(target=self.__record).start()
-
- def stop(self):
- self.__running = False
- self.__aLiNls.end()
-
- @abstractmethod
- def on_speaking(self, text):
- pass
-
- #TODO Edit by xszyou on 20230113:把流的获取方式封装出来方便实现麦克风录制及网络流等不同的流录制子类
- @abstractmethod
- def get_stream(self):
- pass
-
- #TODO Edit by xszyou on 20231225:子类实现返回是否远程音频
- @abstractmethod
- def is_remote(self):
- pass
diff --git a/core/wsa_server.py b/core/wsa_server.py
deleted file mode 100644
index b9bf444..0000000
--- a/core/wsa_server.py
+++ /dev/null
@@ -1,232 +0,0 @@
-from asyncio import AbstractEventLoop
-
-import websockets
-import asyncio
-import json
-from abc import abstractmethod
-from websockets.legacy.server import Serve
-
-from scheduler.thread_manager import MyThread
-from utils import util
-
-
-class MyServer:
- def __init__(self, host='0.0.0.0', port=10000):
- self.__host = host # ip
- self.__port = port # 端口号
- self.__listCmd = [] # 要发送的信息的列表
- self.__server: Serve = None
- self.__event_loop: AbstractEventLoop = None
- self.__running = True
- self.__pending = None
- self.isConnect = False
-
- def __del__(self):
- self.stop_server()
-
- # 接收处理
- async def __consumer_handler(self, websocket, path):
- try:
- async for message in websocket:
- await asyncio.sleep(0.01)
- await self.__consumer(message)
- except websockets.exceptions.ConnectionClosedError as e:
- util.log(1, f"WebSocket 连接关闭: {e}")
- self.isConnect = False
- self.on_close_handler()
-
- async def __producer_handler(self, websocket, path):
- try:
- while self.__running:
- await asyncio.sleep(0.01)
- message = await self.__producer()
- if message:
- await websocket.send(message)
- except websockets.exceptions.ConnectionClosedError as e:
- util.log(1, f"WebSocket 连接关闭: {e}")
- self.isConnect = False
- self.on_close_handler()
-
- async def __handler(self, websocket, path):
- self.isConnect = True
- util.log(1,"websocket连接上:{}".format(self.__port))
- self.on_connect_handler()
- consumer_task = asyncio.ensure_future(self.__consumer_handler(websocket, path))#接收
- producer_task = asyncio.ensure_future(self.__producer_handler(websocket, path))#发送
- done, self.__pending = await asyncio.wait([consumer_task, producer_task], return_when=asyncio.FIRST_COMPLETED, )
- for task in self.__pending:
- task.cancel()
- self.isConnect = False
- util.log(1,"websocket连接断开:{}".format(self.__port))
- self.on_close_handler()
-
- async def __consumer(self, message):
- self.on_revice_handler(message)
-
- async def __producer(self):
- if len(self.__listCmd) > 0:
- message = self.on_send_handler(self.__listCmd.pop(0))
- return message
- else:
- return None
-
-
- #Edit by xszyou on 20230113:通过继承此类来实现服务端的接收后处理逻辑
- @abstractmethod
- def on_revice_handler(self, message):
- pass
-
- #Edit by xszyou on 20230114:通过继承此类来实现服务端的连接处理逻辑
- @abstractmethod
- def on_connect_handler(self):
- pass
-
- #Edit by xszyou on 20230804:通过继承此类来实现服务端的发送前的处理逻辑
- @abstractmethod
- def on_send_handler(self, message):
- return message
-
- #Edit by xszyou on 20230816:通过继承此类来实现服务端的断开后的处理逻辑
- @abstractmethod
- def on_close_handler(self):
- pass
-
- # 创建server
- def __connect(self):
- self.__event_loop = asyncio.new_event_loop()
- asyncio.set_event_loop(self.__event_loop)
- self.__isExecute = True
- if self.__server:
- util.log(1, 'server already exist')
- return
- self.__server = websockets.serve(self.__handler, self.__host, self.__port)
- asyncio.get_event_loop().run_until_complete(self.__server)
- asyncio.get_event_loop().run_forever()
-
- # 往要发送的命令列表中，添加命令
- def add_cmd(self, content):
- if not self.__running:
- return
- jsonObj = json.dumps(content)
- self.__listCmd.append(jsonObj)
- # util.log('命令 {}'.format(content))
-
- # 开启服务
- def start_server(self):
- MyThread(target=self.__connect).start()
-
- # 关闭服务
- def stop_server(self):
- self.__running = False
- self.isConnect = False
- if self.__server is None:
- return
- self.__server.ws_server.close()
- self.__server = None
- try:
- all_tasks = asyncio.all_tasks(self.__event_loop)
- for task in all_tasks:
- while not task.cancel():
- util.log(1, "无法关闭！")
- self.__event_loop.stop()
- self.__event_loop.close()
- except BaseException as e:
- util.log(1, "Error: {}".format(e))
-
-
-
-
-
-
-
-#ui端server
-class WebServer(MyServer):
- def __init__(self, host='0.0.0.0', port=10000):
- super().__init__(host, port)
-
- def on_revice_handler(self, message):
- pass
-
- def on_connect_handler(self):
- self.add_cmd({"panelMsg": "使用提示：Fay可以独立使用，启动数字人将自动对接。"})
-
- def on_send_handler(self, message):
- return message
-
- def on_close_handler(self):
- pass
-
-#数字人端server
-class HumanServer(MyServer):
- def __init__(self, host='0.0.0.0', port=10000):
- super().__init__(host, port)
-
- def on_revice_handler(self, message):
- pass
-
- def on_connect_handler(self):
- web_server_instance = get_web_instance()
- web_server_instance.add_cmd({"is_connect": True})
-
-
- def on_send_handler(self, message):
- # util.log(1, '向human发送 {}'.format(message))
- if not self.isConnect:
- return None
- return message
-
- def on_close_handler(self):
- web_server_instance = get_web_instance()
- web_server_instance.add_cmd({"is_connect": False})
-
-
-
-#测试
-class TestServer(MyServer):
- def __init__(self, host='0.0.0.0', port=10000):
- super().__init__(host, port)
-
- def on_revice_handler(self, message):
- print(message)
-
- def on_connect_handler(self):
- print("连接上了")
-
- def on_send_handler(self, message):
- return message
-
- def on_close_handler(self):
- pass
-
-
-
-#单例
-
-__instance: MyServer = None
-__web_instance: MyServer = None
-
-
-def new_instance(host='0.0.0.0', port=10000) -> MyServer:
- global __instance
- if __instance is None:
- __instance = HumanServer(host, port)
- return __instance
-
-
-def new_web_instance(host='0.0.0.0', port=10000) -> MyServer:
- global __web_instance
- if __web_instance is None:
- __web_instance = WebServer(host, port)
- return __web_instance
-
-
-def get_instance() -> MyServer:
- return __instance
-
-
-def get_web_instance() -> MyServer:
- return __web_instance
-
-if __name__ == '__main__':
- testServer = TestServer(host='0.0.0.0', port=10000)
- testServer.start_server()
\ No newline at end of file
diff --git a/favicon.ico b/favicon.ico
deleted file mode 100644
index cd70e43..0000000
Binary files a/favicon.ico and /dev/null differ
diff --git a/fay_booter.py b/fay_booter.py
deleted file mode 100644
index 0fa351e..0000000
--- a/fay_booter.py
+++ /dev/null
@@ -1,243 +0,0 @@
-import time
-import pyaudio
-from core.interact import Interact
-from core.recorder import Recorder
-from core.fay_core import FeiFei
-from scheduler.thread_manager import MyThread
-from utils import util, config_util, stream_util, ngrok_util
-from core.wsa_server import MyServer
-from scheduler.thread_manager import MyThread
-from agent.agent_service import agent_start, agent_stop
-from core import fay_core
-
-feiFei: FeiFei = None
-recorderListener: Recorder = None
-
-__running = False
-
-#录制麦克风音频输入并传给aliyun
-class RecorderListener(Recorder):
-
- def __init__(self, device, fei):
- self.__device = device
- self.__RATE = 16000
- self.__FORMAT = pyaudio.paInt16
- self.__running = False
-
- super().__init__(fei)
-
- def on_speaking(self, text):
- if len(text) > 1:
- util.printInfo(3, "语音", '{}'.format(text), time.time())
- fay_core.send_for_answer("主人语音说了：" + text)
- time.sleep(2)
-
- def get_stream(self):
- self.paudio = pyaudio.PyAudio()
- device_id,devInfo = self.__findInternalRecordingDevice(self.paudio)
- if device_id < 0:
- return
- channels = int(devInfo['maxInputChannels'])
- if channels == 0:
- util.log(1, '请检查设备是否有误，再重新启动!')
- return
- self.stream = self.paudio.open(input_device_index=device_id, rate=self.__RATE, format=self.__FORMAT, channels=channels, input=True)
- self.__running = True
- MyThread(target=self.__pyaudio_clear).start()
- return self.stream
-
- def __pyaudio_clear(self):
- while self.__running:
- time.sleep(30)
-
-
- def __findInternalRecordingDevice(self, p):
- for i in range(p.get_device_count()):
- devInfo = p.get_device_info_by_index(i)
- if devInfo['name'].find(self.__device) >= 0 and devInfo['hostApi'] == 0:
- config_util.config['source']['record']['channels'] = devInfo['maxInputChannels']
- config_util.save_config(config_util.config)
- return i, devInfo
- util.log(1, '[!] 无法找到内录设备!')
- return -1, None
-
- def stop(self):
- super().stop()
- self.__running = False
- try:
- self.stream.stop_stream()
- self.stream.close()
- self.paudio.terminate()
- except Exception as e:
- print(e)
- util.log(1, "请检查设备是否有误，再重新启动!")
-
- def is_remote(self):
- return False
-
-
-
-#Edit by xszyou on 20230113:录制远程设备音频输入并传给aliyun
-class DeviceInputListener(Recorder):
- def __init__(self, fei):
- super().__init__(fei)
- self.__running = True
- self.ngrok = None
- self.streamCache = None
- self.thread = MyThread(target=self.run)
- self.thread.start() #启动远程音频输入设备监听线程
-
- def run(self):
- #启动ngork
- self.streamCache = stream_util.StreamCache(1024*1024*20)
- if config_util.key_ngrok_cc_id and config_util.key_ngrok_cc_id is not None and config_util.key_ngrok_cc_id.strip() != "":
- MyThread(target=self.start_ngrok, args=[config_util.key_ngrok_cc_id]).start()
- addr = None
- while self.__running:
- try:
-
- data = b""
- while feiFei.deviceConnect:
- data = feiFei.deviceConnect.recv(1024)
- self.streamCache.write(data)
- time.sleep(0.005)
- self.streamCache.clear()
-
- except Exception as err:
- pass
- time.sleep(1)
-
- def on_speaking(self, text):
- global feiFei
-
- if len(text) > 1:
- util.printInfo(3, "语音", '{}'.format(text), time.time())
- fay_core.send_for_answer("主人语音说了：" + text)
- time.sleep(1)
-
- #recorder会等待stream不为空才开始录音
- def get_stream(self):
- while not feiFei.deviceConnect:
- time.sleep(1)
- pass
- return self.streamCache
-
- def stop(self):
- super().stop()
- self.__running = False
- if config_util.key_ngrok_cc_id and config_util.key_ngrok_cc_id is not None and config_util.key_ngrok_cc_id.strip() != "":
- self.ngrok.stop()
-
- def start_ngrok(self, clientId):
- self.ngrok = ngrok_util.NgrokCilent(clientId)
- self.ngrok.start()
-
- def is_remote(self):
- return True
-
-
-
-
-def console_listener():
- global feiFei
- while __running:
- text = input()
- args = text.split(' ')
-
- if len(args) == 0 or len(args[0]) == 0:
- continue
-
- if args[0] == 'help':
- util.log(1, 'in