From b9729d19e356ef3c9330456d7142a1244df9b1d6 Mon Sep 17 00:00:00 2001 From: "jinli.yl" Date: Wed, 11 Sep 2024 17:38:47 +0800 Subject: [PATCH 01/11] update logo size & update version --- README.md | 18 +++++++++++------- README_ZH.md | 17 +++++++++++------ memoryscope/__init__.py | 2 +- setup.py | 2 +- 4 files changed, 24 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 0df8946d..f08340b2 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,13 @@ English | [**中文**](./README_ZH.md) # MemoryScope - -![MemoryScope Logo](./docs/images/logo.png) - +

+ MemoryScopeLogo +

Equip your LLM chatbot with a powerful and flexible long term memory system. [![](https://img.shields.io/badge/python-3.10+-blue)](https://pypi.org/project/memoryscope/) -[![](https://img.shields.io/badge/pypi-v0.1.1-blue?logo=pypi)](https://pypi.org/project/memoryscope/) +[![](https://img.shields.io/badge/pypi-v0.1.1.0-blue?logo=pypi)](https://pypi.org/project/memoryscope/) [![](https://img.shields.io/badge/license-Apache--2.0-black)](./LICENSE) [![](https://img.shields.io/badge/Docs-English%7C%E4%B8%AD%E6%96%87-blue?logo=markdown)](https://modelscope.github.io/memoryscope/#welcome-to-memoryscope-tutorial-hub) [![](https://img.shields.io/badge/Docs-API_Reference-blue?logo=markdown)](https://modelscope.github.io/memoryscope/) @@ -16,7 +16,7 @@ Equip your LLM chatbot with a powerful and flexible long term memory system. ---- ## 📰 News -- **[2024-09-10]** We release MemoryScope v0.1.1 now, which is also available in [PyPI](https://pypi.org/simple/memoryscope/)! +- **[2024-09-10]** We release MemoryScope v0.1.1.0 now, which is also available in [PyPI](https://pypi.org/simple/memoryscope/)! ---- ## 🌟 What is MemoryScope？ MemoryScope provides LLM chatbots with powerful and flexible long-term memory capabilities, offering a framework for building such abilities. @@ -24,10 +24,14 @@ It can be applied to scenarios like personal assistants and emotional companions This allows users to gradually experience a sense of "understanding" when using the LLM. ### Demo -![english-gif](https://github.com/user-attachments/assets/1754c814-1342-4288-a8a3-74d0b40f59a6) +

+ en_demo +

### Framework -![Framework](./docs/images/framework.png) +

+ Framework +

💾 Memory Database: MemoryScope is equipped with a vector database (default is *ElasticSearch*) to store all memory fragments recorded in the system. diff --git a/README_ZH.md b/README_ZH.md index 1e108c90..cc975cd5 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -1,9 +1,9 @@ [**English**](./README.md) | 中文 # MemoryScope - -![MemoryScope Logo](./docs/images/logo.png) - +

+ MemoryScopeLogo +

为您的大语言模型聊天机器人配备强大且灵活的长期记忆系统。 [![](https://img.shields.io/badge/python-3.10+-blue)](https://pypi.org/project/memoryscope/) @@ -21,7 +21,7 @@ ---- ## 📰 新闻 -- **[2024-09-10]** 我们现在发布了 MemoryScope v0.1.1，该版本也可以在 [PyPI](https://pypi.org/simple/memoryscope/) 上获取！ +- **[2024-09-10]** 我们现在发布了 MemoryScope v0.1.1.0，该版本也可以在 [PyPI](https://pypi.org/simple/memoryscope/) 上获取！ ---- ## 🌟 什么是MemoryScope？ @@ -29,10 +29,15 @@ MemoryScope可以为LLM聊天机器人提供强大且灵活的长期记忆能力 MemoryScope可以用于个人助理、情感陪伴等记忆场景，通过长期记忆能力来不断学习，记得用户的基础信息以及各种习惯和喜好，使得用户在使用LLM时逐渐感受到一种“默契”。 ### Demo -![chinese-gif](https://github.com/user-attachments/assets/57519274-8c01-4d88-bcd2-0ebce3551e5d) +

+ zh_demo +

+ ### 核心框架： -![Framework](./docs/images/framework.png) +

+ Framework +

💾 记忆数据库: MemoryScope配备了向量数据库(默认是*ElasticSearch*)，用于存储系统中记录的所有记忆片段。 diff --git a/memoryscope/__init__.py b/memoryscope/__init__.py index c5be9bb2..a5d21147 100644 --- a/memoryscope/__init__.py +++ b/memoryscope/__init__.py @@ -1,5 +1,5 @@ """ Version of MemoryScope.""" -__version__ = "0.1.0.10" +__version__ = "0.1.1.0" import fire from memoryscope.core.config.arguments import Arguments # noqa: F401 diff --git a/setup.py b/setup.py index 4803d1fc..a5a39d8a 100644 --- a/setup.py +++ b/setup.py @@ -54,7 +54,7 @@ def package_files(directory): setuptools.setup( name="memoryscope", - version="0.1.0.10", + version="0.1.1.0", author=', '.join([author['name'] for author in authors]), author_email=', '.join([author['email'] for author in authors]), description="MemoryScope is a powerful and flexible long term memory system for LLM chatbots. It consists of a " From 05500f1279cbef5f9bf5d106e28855a18a30edb7 Mon Sep 17 00:00:00 2001 From: "jinli.yl" Date: Mon, 14 Oct 2024 14:37:58 +0800 Subject: [PATCH 02/11] add memory view --- examples/cli/CLI_README.md | 9 +++++++++ examples/cli/CLI_README_ZH.md | 9 +++++++++ 2 files changed, 18 insertions(+) diff --git a/examples/cli/CLI_README.md b/examples/cli/CLI_README.md index 5c747311..f247ff4e 100644 --- a/examples/cli/CLI_README.md +++ b/examples/cli/CLI_README.md @@ -53,3 +53,12 @@ Here are the available options that can be set through either method: - `--enable_ranker`: A boolean indicating whether to use a dummy ranker (default is `False`). - `--rank_backend`: The backend used for ranking responses. - `--rank_model`: The model used for ranking responses. + +### 3. View Memory +You can open two command line windows following the method in the second step. +In one command line window, you can have a conversation with the AI, while in the other, you can check the AI's long-term memory about the user. +Use /help to open the command line help, and find the command /list_memory along with the corresponding auto-refresh instruction. +``` +/list_memory refresh_time=5 +``` +Then you can enjoy a pleasant conversation with the AI! \ No newline at end of file diff --git a/examples/cli/CLI_README_ZH.md b/examples/cli/CLI_README_ZH.md index 4ba1c640..b9debdda 100644 --- a/examples/cli/CLI_README_ZH.md +++ b/examples/cli/CLI_README_ZH.md @@ -53,3 +53,12 @@ memoryscope --language="en" \ - `--enable_ranker`: 一个布尔值，指示是否使用排名器（默认为 False）。 - `--rank_backend`: 用于排名回复的后端。 - `--rank_model`: 用于排名回复的模型。 + +### 3. 查看记忆 +按照第二步的方式可以打开两个命令行的窗口。 +其中一个命令行窗口可以和AI进行对话，另一个命令行窗口可以查看AI关于用户的长期记忆 +使用/help打开命令行帮助，找到/list_memory的命令和对应自动刷新的指令。 +``` +/list_memory refresh_time=5 +``` +接下来就可以和AI进行愉快地交流啦。 \ No newline at end of file From 4f72424fe6ec78db88ba83b65a9f1f090652a790 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=9D=92=E8=BD=A9?= Date: Tue, 10 Sep 2024 11:47:49 +0800 Subject: [PATCH 03/11] amend github action --- .github/workflows/docker-image.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml index 78a34b6c..e157f525 100644 --- a/.github/workflows/docker-image.yml +++ b/.github/workflows/docker-image.yml @@ -8,7 +8,7 @@ on: env: REGISTRY: ghcr.io - IMAGE_NAME: ${{ github.repository }} + IMAGE_NAME: ${{ github.repository }}_standard jobs: build-and-push-image: From 32ce03f2c593d5da7659f82d83a98c0aa653ad56 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=9D=92=E8=BD=A9?= Date: Tue, 10 Sep 2024 11:52:29 +0800 Subject: [PATCH 04/11] amend github action --- .github/workflows/docker-image.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml index e157f525..78a34b6c 100644 --- a/.github/workflows/docker-image.yml +++ b/.github/workflows/docker-image.yml @@ -8,7 +8,7 @@ on: env: REGISTRY: ghcr.io - IMAGE_NAME: ${{ github.repository }}_standard + IMAGE_NAME: ${{ github.repository }} jobs: build-and-push-image: From 222f1c005995fbd763e71666e169a55113b6e1c9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=9D=92=E8=BD=A9?= Date: Wed, 11 Sep 2024 13:30:58 +0800 Subject: [PATCH 05/11] fix: limit elastic search memory usage --- Dockerfile | 4 ++-- docker-compose.yml | 15 ++++----------- docs/installation.md | 2 +- docs/installation_zh.md | 2 +- 4 files changed, 8 insertions(+), 15 deletions(-) diff --git a/Dockerfile b/Dockerfile index 2627e66c..68f4ab1d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,9 +11,9 @@ # sudo docker build --network=host -t memoryscope . # To run docker image: -# sudo docker run -it --rm --net=host memoryscope +# sudo docker run -it --rm --memory=4G --net=host memoryscope # To run docker image with arguments (refer to memoryscope/core/config/arguments.py): -# sudo docker run -it --rm --net=host -e "OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -e "language=en" -e "human_name=superman" -e "generation_backend=openai_generation" -e "generation_model=gpt-4o" -e "embedding_backend=openai_embedding" -e "embedding_model=text-embedding-3-small" -e "enable_ranker=False" memoryscope +# sudo docker run -it --rm --memory=4G --net=host -e "OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -e "language=en" -e "human_name=superman" -e "generation_backend=openai_generation" -e "generation_model=gpt-4o" -e "embedding_backend=openai_embedding" -e "embedding_model=text-embedding-3-small" -e "enable_ranker=False" memoryscope FROM python:3.11 diff --git a/docker-compose.yml b/docker-compose.yml index deaa259e..0820bf3e 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -9,14 +9,7 @@ services: # OPENAI_API_KEY: "sk-0000000000" volumes: - ./memoryscope/core/config:/memory_scope_project/memoryscope/memoryscope/core/config - depends_on: - - elasticsearch - - elasticsearch: - image: docker.elastic.co/elasticsearch/elasticsearch:8.13.2 - ports: - - "9200:9200" - environment: - discovery.type: "single-node" - xpack.security.enabled: "false" - xpack.license.self_generated.type: "trial" + deploy: + resources: + limits: + memory: 4G diff --git a/docs/installation.md b/docs/installation.md index 413214a3..b4453172 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -38,7 +38,7 @@ OPENAI_API_KEY: "sk-0000000000" ``` -3. Run `docker-compose up` to build and launch the memory-scope cli interface. +3. Run `docker-compose run memory_scope_main` to build and launch the memory-scope cli interface. ## III. Install from PyPI diff --git a/docs/installation_zh.md b/docs/installation_zh.md index 41d29f46..0803299b 100644 --- a/docs/installation_zh.md +++ b/docs/installation_zh.md @@ -38,7 +38,7 @@ DASHSCOPE_API_KEY: "sk-0000000000" ``` -3. 运行 `docker-compose up` 命令来构建并启动 MemoryScope CLI 界面。 +3. 运行 `docker-compose run memory_scope_main` 命令来构建并启动 MemoryScope CLI 界面。 ## 三、通过 PYPI 安装 From 2bd628a21823bf550830552c9e8abc2745780e10 Mon Sep 17 00:00:00 2001 From: Sen Huang <48879559+ployts@users.noreply.github.com> Date: Wed, 11 Sep 2024 17:31:13 +0800 Subject: [PATCH 06/11] generate docs automatically --- .github/workflows/deploy_sphinx_docs.yml | 40 ++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 .github/workflows/deploy_sphinx_docs.yml diff --git a/.github/workflows/deploy_sphinx_docs.yml b/.github/workflows/deploy_sphinx_docs.yml new file mode 100644 index 00000000..d1918737 --- /dev/null +++ b/.github/workflows/deploy_sphinx_docs.yml @@ -0,0 +1,40 @@ +name: Deploy Sphinx documentation to Pages + +on: + pull_request: + types: [opened, synchronize] + paths: + - 'docs/sphinx_doc/**/*' + push: + branches: + - main + +jobs: + pages: + runs-on: ubuntu-20.04 + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Setup Python ${{ matrix.python-version }} + uses: actions/setup-python@master + with: + python_version: ${{ matrix.python-version }} + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r docs/sphinx_doc/requirements.txt + - id: build + name: Build Documentation + run: | + cd docs/sphinx_doc + bash build_sphinx_doc.sh + - name: Upload Documentation + uses: actions/upload-artifact@v3 + with: + name: SphinxDoc + path: 'docs/sphinx_doc/build/html' + - uses: peaceiris/actions-gh-pages@v3 + if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: 'docs/sphinx_doc/build/html' From 46b5fee6104e8473978704ed8d8f8da0ea4882a2 Mon Sep 17 00:00:00 2001 From: Sen Huang <48879559+ployts@users.noreply.github.com> Date: Thu, 12 Sep 2024 11:36:48 +0800 Subject: [PATCH 07/11] sphinxdoc deploy (#2) * fix sphinxdoc yaml * fix link to doc --------- Co-authored-by: hs --- .github/workflows/deploy_sphinx_docs.yml | 62 +++++++++++++++++++++--- README.md | 6 +-- README_ZH.md | 6 +-- docs/sphinx_doc/requirements.txt | 1 - 4 files changed, 61 insertions(+), 14 deletions(-) diff --git a/.github/workflows/deploy_sphinx_docs.yml b/.github/workflows/deploy_sphinx_docs.yml index d1918737..8309017b 100644 --- a/.github/workflows/deploy_sphinx_docs.yml +++ b/.github/workflows/deploy_sphinx_docs.yml @@ -1,4 +1,4 @@ -name: Deploy Sphinx documentation to Pages +name: deploy-sphinx-documentation-to-pages on: pull_request: @@ -15,16 +15,63 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 - - name: Setup Python ${{ matrix.python-version }} + - name: Setup Python uses: actions/setup-python@master with: - python_version: ${{ matrix.python-version }} - - name: Install dependencies + python-version: '3.10' + - name: Choose Pandoc + shell: bash + run: | + case $RUNNER_OS in + "Linux") + printf 'INSTALLER_SUFFIX=1-amd64.deb' >> $GITHUB_ENV + ;; + "macOS") + printf 'INSTALLER_SUFFIX=macOS.pkg' >> $GITHUB_ENV + ;; + *) + printf 'Do not know how to install pandoc on %s\n' "$RUNNER_OS" + exit 1 + ;; + esac + - name: Download Pandoc + shell: bash + env: + GITHUB_TOKEN: ${{ github.token }} + REPO: jgm/pandoc + DOWNLOAD_URL: 'https://github.com/jgm/pandoc/releases/download/' + run: | + gh release download ${{ inputs.version }} \ + --repo "$REPO" \ + --pattern '*'${{ env.INSTALLER_SUFFIX }} + printf 'INSTALLER_VERSION=%s' \ + "$(ls pandoc-*-${{ env.INSTALLER_SUFFIX }} | \ + sed 's/pandoc-\([0-9.]*\)-.*/\1/')" \ + >> $GITHUB_ENV + - name: Install Pandoc + shell: bash + env: + INSTALLER: pandoc-${{ env.INSTALLER_VERSION }}-${{ env.INSTALLER_SUFFIX }} + run: | + case $RUNNER_OS in + "Linux") + sudo apt install ./$INSTALLER + ;; + "macOS") + sudo installer -pkg ./$INSTALLER -target '/' + ;; + *) + echo "$RUNNER_OS not supported" + exit 1 + ;; + esac + rm $INSTALLER + - name: Install Sphinx Dependencies run: | python -m pip install --upgrade pip + pip install -r requirements.txt pip install -r docs/sphinx_doc/requirements.txt - - id: build - name: Build Documentation + - name: Build Documentation run: | cd docs/sphinx_doc bash build_sphinx_doc.sh @@ -33,7 +80,8 @@ jobs: with: name: SphinxDoc path: 'docs/sphinx_doc/build/html' - - uses: peaceiris/actions-gh-pages@v3 + - name: Push Pages + uses: peaceiris/actions-gh-pages@v3 if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} with: github_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/README.md b/README.md index f08340b2..30959776 100644 --- a/README.md +++ b/README.md @@ -9,9 +9,9 @@ Equip your LLM chatbot with a powerful and flexible long term memory system. [![](https://img.shields.io/badge/python-3.10+-blue)](https://pypi.org/project/memoryscope/) [![](https://img.shields.io/badge/pypi-v0.1.1.0-blue?logo=pypi)](https://pypi.org/project/memoryscope/) [![](https://img.shields.io/badge/license-Apache--2.0-black)](./LICENSE) -[![](https://img.shields.io/badge/Docs-English%7C%E4%B8%AD%E6%96%87-blue?logo=markdown)](https://modelscope.github.io/memoryscope/#welcome-to-memoryscope-tutorial-hub) -[![](https://img.shields.io/badge/Docs-API_Reference-blue?logo=markdown)](https://modelscope.github.io/memoryscope/) -[![](https://img.shields.io/badge/Contribute-Welcome-green)](https://modelscope.github.io/memoryscope/tutorial/contribute.html) +[![](https://img.shields.io/badge/Docs-English%7C%E4%B8%AD%E6%96%87-blue?logo=markdown)](https://modelscope.github.io/MemoryScope/en/index.html#welcome-to-memoryscope-tutorial) +[![](https://img.shields.io/badge/Docs-API_Reference-blue?logo=markdown)](https://modelscope.github.io/MemoryScope/en/docs/api.html) +[![](https://img.shields.io/badge/Contribute-Welcome-green)](https://modelscope.github.io/MemoryScope/en/docs/contribution.html) ---- ## 📰 News diff --git a/README_ZH.md b/README_ZH.md index cc975cd5..6785027d 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -9,9 +9,9 @@ [![](https://img.shields.io/badge/python-3.10+-blue)](https://pypi.org/project/memoryscope/) [![](https://img.shields.io/badge/pypi-v0.1.1-blue?logo=pypi)](https://pypi.org/project/memoryscope/) [![](https://img.shields.io/badge/license-Apache--2.0-black)](./LICENSE) -[![](https://img.shields.io/badge/Docs-English%7C%E4%B8%AD%E6%96%87-blue?logo=markdown)](https://modelscope.github.io/memoryscope/#welcome-to-memoryscope-tutorial-hub) -[![](https://img.shields.io/badge/Docs-API_Reference-blue?logo=markdown)](https://modelscope.github.io/memoryscope/) -[![](https://img.shields.io/badge/Contribute-Welcome-green)](https://modelscope.github.io/memoryscope/tutorial/contribute.html) +[![](https://img.shields.io/badge/Docs-English%7C%E4%B8%AD%E6%96%87-blue?logo=markdown)](https://modelscope.github.io/MemoryScope/zh/index.html#id1) +[![](https://img.shields.io/badge/Docs-API_Reference-blue?logo=markdown)](https://modelscope.github.io/MemoryScope/zh/docs/api.html) +[![](https://img.shields.io/badge/Contribute-Welcome-green)](https://modelscope.github.io/MemoryScope/zh/docs/contribution.html)