4、开源搜索引擎集成入langchain.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "837bb7ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "acfd1307",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"关于ChatGPT编写和调试计算机程序的能力的训练，由于 深度学习 模型不懂编程，与所有其他基于 深度学习 的 语言模型 一样，只是在获取代码片段之间的统计相关性。 [16] 斯坦福大学的研究发现，GPT3已经可以解决70%的 心智理论 任务，相当于7岁儿童；至于GPT3.5（ChatGPT的同源模型），更是解决了93%的任务，心智相当于9岁儿童。 但这并不意味着，ChatGPT就真正有 心智理论 。 可能即使不將它设计到AI系统中，也可以作为“副产品”通过训练得到。 因此，相比探究GPT3.5是不是真的有了心智还是像有心智，更需要反思的是这些测试本身。 [17] [18] 特点和局限 特点 虽然聊天机器人的核心功能是模仿人类对话者，但ChatGPT用途广泛。\\n\\nChatGPT is a sibling model to InstructGPT, which is trained to follow an instruction in a prompt and provide a detailed response. We are excited to introduce ChatGPT to get users’ feedback and learn about its strengths and weaknesses. During the research preview, usage of ChatGPT is free. Try it now at chat.openai.com. Samples\\n\\nChatGPT是个啥？ 近期，OpenAI 发布了 ChatGPT，是一个可以对话的方式进行交互的模型，因为它的智能化，得到了很多用户的欢迎。 ChatGPT 也是OpenAI之前发布的 InstructGPT 的亲戚， ChatGPT 模型的训练是使用RLHF（Reinforcement learning with human feedback）也许 ChatGPT的到来，也是OpenAI 的GPT-4正式推出之前的序章。 什么是GPT？ 从GPT-1到GPT-3 Generative Pre-trained Transformer (GPT)，是一种基于互联网可用数据训练的文本生成深度学习模型。 它用于问答、文本摘要生成、机器翻译、分类、代码生成和对话 AI。\\n\\n推荐： 国内注册ChatGPT的方法 (100%可用) GPT-4 有什么新功能？ GPT 是 Generative Pre-trained Transformer 的缩写，是一种深度学习技术，使用人工神经网络来像人类一样书写。 GPT-4 是 OpenAI 创建的一种新的语言模型，可以生成类似于人类语音的文本，改进了目前基于 GPT-3.5 的 ChatGPT 使用的技术。 GPT-4 这种语言模型在三个关键领域更加先进： 创造力、视觉输入和更长的上下文 。 在创造力方面，GPT-4 在创造和与用户合作进行创意项目方面要好得多。 如何使用GPT-4 (100%可用免费) GPT-4 现在可以处理来自用户的多达 25,000 个单词的文本。\\n\\n🤖 ChatGPT 中文指南 🤖 什么是 ChatGPT ? 使用途径 💻 OpenAI 官网 Plus 开通教程 💻 poe 💻 微软必应 国内可使用ChatGPT镜像站点 💻 第三方开发者开发的 ChatGPT 客户端 💻 国外竞品 💻 国产 ChatGPT 类似产品 如何与 ChatGPT 高效对话？\\n\\nWhat is ChatGPT? ChatGPT is a large language model (LLM) developed by OpenAI that can be used for natural language processing tasks such as text generation and language translation. It is based on the GPT-3.5 (Generative Pretrained Transformer 3.5) and GPT-4 model, which is one of the largest and most advanced language models currently available.\\n\\nChatGPT is a large language model chatbot developed by OpenAI based on GPT-3.5. It has a remarkable ability to interact in conversational dialogue form and provide responses that can appear...\\n\\nChatGPT于2022年11月30日由总部位于旧金山的OpenAI推出。 该服务最初是免费向公众推出，并计划以后用该服务获利 。 到12月4日，OpenAI估计ChatGPT已有超过一百万用户 。 2023年1月，ChatGPT的用户数超过1亿，成为该时间段内增长最快的消费者应用程序 。. 2022年12月15日，全国广播公司商业频道写道，该服务 ...\\n\\nChatGPT is a powerful new AI technology that is likely to change the world profoundly. ChatGPT is not inherently good or bad, but it's powerful and can offer tremendous benefits and create major ...\\n\\n什么是ChatGPT - Data Application Lab · Enterprise Service. Machine Learning & AI; FinTech · Natural Language Processing · Steam Game Recommendation System ...\""
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.utilities import SearxSearchWrapper\n",
    "s = SearxSearchWrapper(searx_host=\"http://localhost:8080\")\n",
    "s.run(\"什么是chatgpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "15705a35",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'深度学习：是机器学习的分支，是一种以人工神经网络为架构，对数据进行表征学习的算法。 在传统机器学习中，手工设计特征对学习效果很重要，但是特征工程非常繁琐。 而深度学习能够从大数据中自动学习特征，这也是深度学习在大数据时代受欢迎的一大原因。2019年11月19日'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search = SearxSearchWrapper(searx_host=\"http://localhost:8080\", k=5)\n",
    "search.run(\"深度学习\", language='zh', engines=['wiki'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "ed54db24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'snippet': 'In the following sample, ChatGPT asks the clarifying questions '\n",
      "             'to debug code. In the following sample, ChatGPT initially '\n",
      "             'refuses to answer a question that could be about illegal '\n",
      "             'activities but responds after the user clarifies their intent. '\n",
      "             'In the following sample, ChatGPT is able to understand the '\n",
      "             'reference (\"it\") to the subject of the previous question '\n",
      "             '(\"fermat\\'s little theorem\").',\n",
      "  'title': 'Introducing ChatGPT',\n",
      "  'link': 'https://openai.com/blog/chatgpt',\n",
      "  'engines': ['google', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'ChatGPT version: OpenAI included a bit of fine print below the '\n",
      "             'text input area, where you can read the disclaimer that this is '\n",
      "             'a \"Free Research Preview. ChatGPT may produce inaccurate '\n",
      "             'information ...',\n",
      "  'title': 'How to use ChatGPT: What you need to know now',\n",
      "  'link': 'https://www.zdnet.com/article/how-to-use-chatgpt/',\n",
      "  'engines': ['google', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'ChatGPT is a natural language processing tool driven by AI '\n",
      "             'technology that allows you to have human-like conversations and '\n",
      "             'much more with the chatbot. The language model can answer '\n",
      "             'questions and ...',\n",
      "  'title': \"What is ChatGPT and why does it matter? Here's what you ...\",\n",
      "  'link': 'https://www.zdnet.com/article/what-is-chatgpt-and-why-does-it-matter-heres-everything-you-need-to-know/',\n",
      "  'engines': ['google', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'ChatGPT has continued to dazzle the internet with AI-generated '\n",
      "             'content, morphing from a novel chatbot into a piece of '\n",
      "             'technology that is driving the next era of technological '\n",
      "             'innovation. Not ...',\n",
      "  'title': \"ChatGPT: How to use the AI tool that's changing everything\",\n",
      "  'link': 'https://www.digitaltrends.com/computing/how-to-use-openai-chatgpt-text-generation-chatbot/',\n",
      "  'engines': ['google', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Mar 13, 2023 — GPT-4 surpasses ChatGPT in its advanced reasoning '\n",
      "             'capabilities. ... GPT-4 outperforms ChatGPT by scoring in higher '\n",
      "             'approximate percentiles among test-takers ...',\n",
      "  'title': 'GPT-4',\n",
      "  'link': 'https://openai.com/product/gpt-4',\n",
      "  'engines': ['google'],\n",
      "  'category': 'general'}]\n"
     ]
    }
   ],
   "source": [
    "results = search.results(\"chatgpt\", num_results=5, categories='science', time_range='year')\n",
    "pprint.pp(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "26a9aa84",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'snippet': 'Thanks to the advanced improvement of large pre-trained language '\n",
      "             'models, prompt-based fine-tuning is shown to be effective on a '\n",
      "             'variety of downstream tasks. Though many prompting methods have '\n",
      "             'been investigated, it remains unknown which type of prompts are '\n",
      "             'the most effective among three types of prompts (i.e., '\n",
      "             'human-designed prompts, schema prompts and null prompts). In '\n",
      "             'this work, we empirically compare the three types of prompts '\n",
      "             'under both few-shot and fully-supervised settings. Our '\n",
      "             'experimental results show that schema prompts are the most '\n",
      "             'effective in general. Besides, the performance gaps tend to '\n",
      "             'diminish when the scale of training data grows large.',\n",
      "  'title': 'Do Prompts Solve NLP Tasks Using Natural Language?',\n",
      "  'link': 'http://arxiv.org/abs/2203.00902v1',\n",
      "  'engines': ['arxiv'],\n",
      "  'category': 'science'},\n",
      " {'snippet': 'Cross-prompt automated essay scoring (AES) requires the system '\n",
      "             'to use non target-prompt essays to award scores to a '\n",
      "             'target-prompt essay. Since obtaining a large quantity of '\n",
      "             'pre-graded essays to a particular prompt is often difficult and '\n",
      "             'unrealistic, the task of cross-prompt AES is vital for the '\n",
      "             'development of real-world AES systems, yet it remains an '\n",
      "             'under-explored area of research. Models designed for '\n",
      "             'prompt-specific AES rely heavily on prompt-specific knowledge '\n",
      "             'and perform poorly in the cross-prompt setting, whereas current '\n",
      "             'approaches to cross-prompt AES either require a certain quantity '\n",
      "             'of labelled target-prompt essays or require a large quantity of '\n",
      "             'unlabelled target-prompt essays to perform transfer learning in '\n",
      "             'a multi-step manner. To address these issues, we introduce '\n",
      "             'Prompt Agnostic Essay Scorer (PAES) for cross-prompt AES. Our '\n",
      "             'method requires no access to labelled or unlabelled '\n",
      "             'target-prompt data during training and is a single-stage '\n",
      "             'approach. PAES is easy to apply in practice and achieves '\n",
      "             'state-of-the-art performance on the Automated Student Assessment '\n",
      "             'Prize (ASAP) dataset.',\n",
      "  'title': 'Prompt Agnostic Essay Scorer: A Domain Generalization Approach to '\n",
      "           'Cross-prompt Automated Essay Scoring',\n",
      "  'link': 'http://arxiv.org/abs/2008.01441v1',\n",
      "  'engines': ['arxiv'],\n",
      "  'category': 'science'},\n",
      " {'snippet': 'Research on prompting has shown excellent performance with '\n",
      "             'little or even no supervised training across many tasks. '\n",
      "             'However, prompting for machine translation is still '\n",
      "             'under-explored in the literature. We fill this gap by offering a '\n",
      "             'systematic study on prompting strategies for translation, '\n",
      "             'examining various factors for prompt template and demonstration '\n",
      "             'example selection. We further explore the use of monolingual '\n",
      "             'data and the feasibility of cross-lingual, cross-domain, and '\n",
      "             'sentence-to-document transfer learning in prompting. Extensive '\n",
      "             'experiments with GLM-130B (Zeng et al., 2022) as the testbed '\n",
      "             'show that 1) the number and the quality of prompt examples '\n",
      "             'matter, where using suboptimal examples degenerates translation; '\n",
      "             '2) several features of prompt examples, such as semantic '\n",
      "             'similarity, show significant Spearman correlation with their '\n",
      "             'prompting performance; yet, none of the correlations are strong '\n",
      "             'enough; 3) using pseudo parallel prompt examples constructed '\n",
      "             'from monolingual data via zero-shot prompting could improve '\n",
      "             'translation; and 4) improved performance is achievable by '\n",
      "             'transferring knowledge from prompt examples selected in other '\n",
      "             'settings. We finally provide an analysis on the model outputs '\n",
      "             'and discuss several problems that prompting still suffers from.',\n",
      "  'title': 'Prompting Large Language Model for Machine Translation: A Case '\n",
      "           'Study',\n",
      "  'link': 'http://arxiv.org/abs/2301.07069v2',\n",
      "  'engines': ['arxiv'],\n",
      "  'category': 'science'},\n",
      " {'snippet': 'Prevailing methods for mapping large generative language models '\n",
      "             \"to supervised tasks may fail to sufficiently probe models' novel \"\n",
      "             'capabilities. Using GPT-3 as a case study, we show that 0-shot '\n",
      "             'prompts can significantly outperform few-shot prompts. We '\n",
      "             'suggest that the function of few-shot examples in these cases is '\n",
      "             'better described as locating an already learned task rather than '\n",
      "             'meta-learning. This analysis motivates rethinking the role of '\n",
      "             'prompts in controlling and evaluating powerful language models. '\n",
      "             'In this work, we discuss methods of prompt programming, '\n",
      "             'emphasizing the usefulness of considering prompts through the '\n",
      "             'lens of natural language. We explore techniques for exploiting '\n",
      "             'the capacity of narratives and cultural anchors to encode '\n",
      "             'nuanced intentions and techniques for encouraging deconstruction '\n",
      "             'of a problem into components before producing a verdict. '\n",
      "             'Informed by this more encompassing theory of prompt programming, '\n",
      "             'we also introduce the idea of a metaprompt that seeds the model '\n",
      "             'to generate its own natural language prompts for a range of '\n",
      "             'tasks. Finally, we discuss how these more general methods of '\n",
      "             'interacting with language models can be incorporated into '\n",
      "             'existing and future benchmarks and practical applications.',\n",
      "  'title': 'Prompt Programming for Large Language Models: Beyond the Few-Shot '\n",
      "           'Paradigm',\n",
      "  'link': 'http://arxiv.org/abs/2102.07350v1',\n",
      "  'engines': ['arxiv'],\n",
      "  'category': 'science'},\n",
      " {'snippet': 'Large language models can perform new tasks in a zero-shot '\n",
      "             'fashion, given natural language prompts that specify the desired '\n",
      "             'behavior. Such prompts are typically hand engineered, but can '\n",
      "             'also be learned with gradient-based methods from labeled data. '\n",
      "             'However, it is underexplored what factors make the prompts '\n",
      "             'effective, especially when the prompts are natural language. In '\n",
      "             'this paper, we investigate common attributes shared by effective '\n",
      "             'prompts. We first propose a human readable prompt tuning method '\n",
      "             '(F LUENT P ROMPT) based on Langevin dynamics that incorporates a '\n",
      "             'fluency constraint to find a diverse distribution of effective '\n",
      "             'and fluent prompts. Our analysis reveals that effective prompts '\n",
      "             'are topically related to the task domain and calibrate the prior '\n",
      "             'probability of label words. Based on these findings, we also '\n",
      "             'propose a method for generating prompts using only unlabeled '\n",
      "             'data, outperforming strong baselines by an average of 7.0% '\n",
      "             'accuracy across three tasks.',\n",
      "  'title': \"Toward Human Readable Prompt Tuning: Kubrick's The Shining is a \"\n",
      "           'good movie, and a good prompt too?',\n",
      "  'link': 'http://arxiv.org/abs/2212.10539v1',\n",
      "  'engines': ['arxiv'],\n",
      "  'category': 'science'}]\n"
     ]
    }
   ],
   "source": [
    "results = search.results(\"Large Language Model prompt\", num_results=5, engines=['arxiv'])\n",
    "pprint.pp(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "de211459",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'snippet': 'This repo includes ChatGPT prompt curation to use ChatGPT '\n",
      "             'better.',\n",
      "  'title': 'awesome-chatgpt-prompts',\n",
      "  'link': 'https://github.com/f/awesome-chatgpt-prompts',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'ChatGPT 中文调教指南。各种场景使用指南。学习怎么让它听你的话。',\n",
      "  'title': 'awesome-chatgpt-prompts-zh',\n",
      "  'link': 'https://github.com/PlexPt/awesome-chatgpt-prompts-zh',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': '🔮 ChatGPT Desktop Application (Mac, Windows and Linux)',\n",
      "  'title': 'ChatGPT',\n",
      "  'link': 'https://github.com/lencx/ChatGPT',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'Examples and guides for using the OpenAI API',\n",
      "  'title': 'openai-cookbook',\n",
      "  'link': 'https://github.com/openai/openai-cookbook',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'decentralising the Ai Industry, just some language model '\n",
      "             \"api's...\",\n",
      "  'title': 'gpt4free',\n",
      "  'link': 'https://github.com/xtekky/gpt4free',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': '为GPT/GLM提供图形交互界面，特别优化论文阅读润色体验，模块化设计支持自定义快捷按钮&函数插件，支持代码块表格显示，Tex公式双显示，新增Python和C++项目剖析&自译解功能，PDF/LaTex论文翻译&总结功能，支持并行问询多种LLM模型，支持清华chatglm等本地模型。兼容复旦MOSS, '\n",
      "             'llama, rwkv, 盘古等。',\n",
      "  'title': 'gpt_academic',\n",
      "  'link': 'https://github.com/binary-husky/gpt_academic',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'OpenAssistant is a chat-based assistant that understands tasks, '\n",
      "             'can interact with third-party systems, and retrieve information '\n",
      "             'dynamically to do so.',\n",
      "  'title': 'Open-Assistant',\n",
      "  'link': 'https://github.com/LAION-AI/Open-Assistant',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': '🐙 Guides, papers, lecture, notebooks and resources for prompt '\n",
      "             'engineering',\n",
      "  'title': 'Prompt-Engineering-Guide',\n",
      "  'link': 'https://github.com/dair-ai/Prompt-Engineering-Guide',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'Reverse engineered ChatGPT API',\n",
      "  'title': 'ChatGPT',\n",
      "  'link': 'https://github.com/acheong08/ChatGPT',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'One-Click to deploy well-designed ChatGPT web UI on Vercel. '\n",
      "             '一键拥有你自己的 ChatGPT 网页服务。',\n",
      "  'title': 'ChatGPT-Next-Web',\n",
      "  'link': 'https://github.com/Yidadaa/ChatGPT-Next-Web',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': '用 Express 和 Vue3 搭建的 ChatGPT 演示网页',\n",
      "  'title': 'chatgpt-web',\n",
      "  'link': 'https://github.com/Chanzhaoyu/chatgpt-web',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': '基于vite+vue3+gin搭建的开发基础平台（支持TS,JS混用），集成jwt鉴权，权限管理，动态路由，显隐可控组件，分页封装，多点登录拦截，资源权限，上传下载，代码生成器，表单生成器,chatGPT自动查表等开发必备功能。',\n",
      "  'title': 'gin-vue-admin',\n",
      "  'link': 'https://github.com/flipped-aurora/gin-vue-admin',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'The ChatGPT Retrieval Plugin lets you easily search and find '\n",
      "             'personal or work documents by asking questions in everyday '\n",
      "             'language.',\n",
      "  'title': 'chatgpt-retrieval-plugin',\n",
      "  'link': 'https://github.com/openai/chatgpt-retrieval-plugin',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': '基于 ChatGPT API 的划词翻译浏览器插件和跨平台桌面端应用 - Browser extension and '\n",
      "             'cross-platform desktop application for translation based on '\n",
      "             'ChatGPT API.',\n",
      "  'title': 'openai-translator',\n",
      "  'link': 'https://github.com/yetone/openai-translator',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': '🤱🏻 Turn any webpage into a desktop app with Rust. 🤱🏻 很简单的用 Rust '\n",
      "             '打包网页生成很小的桌面 App',\n",
      "  'title': 'Pake',\n",
      "  'link': 'https://github.com/tw93/Pake',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': '🏆 A ranked list of awesome machine learning Python libraries. '\n",
      "             'Updated weekly.',\n",
      "  'title': 'best-of-ml-python',\n",
      "  'link': 'https://github.com/ml-tooling/best-of-ml-python',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'Node.js client for the official ChatGPT API. 🔥',\n",
      "  'title': 'chatgpt-api',\n",
      "  'link': 'https://github.com/transitive-bullshit/chatgpt-api',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'An open source ChatGPT UI.',\n",
      "  'title': 'chatbot-ui',\n",
      "  'link': 'https://github.com/mckaywrigley/chatbot-ui',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'A browser extension that enhance search engines with ChatGPT',\n",
      "  'title': 'chatgpt-google-extension',\n",
      "  'link': 'https://github.com/wong2/chatgpt-google-extension',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'},\n",
      " {'snippet': 'Curso para aprender el lenguaje de programación Python desde '\n",
      "             'cero y para principiantes. Más de 30 clases, 25 horas en vídeo, '\n",
      "             'código y grupo de chat. Desde sus fundamentos hasta la creación '\n",
      "             'de un API Backend con base de datos y más...',\n",
      "  'title': 'Hello-Python',\n",
      "  'link': 'https://github.com/mouredev/Hello-Python',\n",
      "  'engines': ['github'],\n",
      "  'category': 'it'}]\n"
     ]
    }
   ],
   "source": [
    "results = search.results(\"chatgpt\", num_results = 20, categories='it',engines=['github'])\n",
    "pprint.pp(results)\n",
    "# pprint.pp(list(filter(lambda r: r['engines'][0] == 'github', results)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "0ca9545f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'snippet': 'A large language model ( LLM) is a language model consisting of '\n",
      "             'a neural network with many parameters (typically billions of '\n",
      "             'weights or more), trained on large quantities of unlabeled text '\n",
      "             'using self-supervised learning or semi-supervised learning. [1] '\n",
      "             'LLMs emerged around 2018 and perform well at a wide variety of '\n",
      "             'tasks.',\n",
      "  'title': 'Large language model',\n",
      "  'link': 'https://en.wikipedia.org/wiki/Large_language_model',\n",
      "  'engines': ['wikipedia', 'duckduckgo', 'wikidata', 'qwant'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'LLMs are natural language processingcomputer programs that use '\n",
      "             'artificial neural networksto generate text and source code. Some '\n",
      "             'notable ones are GPT-3, GPT-4, LaMDA, BLOOM, and LLaMA. LLMs '\n",
      "             'power many applications, such as AI chatbots and AI search '\n",
      "             'engines.',\n",
      "  'title': 'Wikipedia:Large language models - Wikipedia',\n",
      "  'link': 'https://en.wikipedia.org/wiki/Wikipedia:Large_language_models',\n",
      "  'engines': ['qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Large language models are deep learning neural networks, a '\n",
      "             'subset of artificial intelligence and machine learning. Large '\n",
      "             'language models are first pre-trained so that they learn basic '\n",
      "             'language tasks and functions. Pretraining is the step that '\n",
      "             'requires massive computational power and cutting-edge hardware. '\n",
      "             'Figure 2: Pre-training vs. fine-tuning',\n",
      "  'title': 'Large Language Models: Complete Guide in 2023',\n",
      "  'link': 'https://research.aimultiple.com/large-language-models/',\n",
      "  'engines': ['qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Download PDF Abstract: Recent work claims that large language '\n",
      "             'models display emergent abilities, abilities not present in '\n",
      "             'smaller-scale models that are present in larger-scale models. '\n",
      "             'What makes emergent abilities intriguing is two-fold: their '\n",
      "             'sharpness, transitioning seemingly instantaneously from not '\n",
      "             'present to present, and their unpredictability, appearing at '\n",
      "             'seemingly unforeseeable model ...',\n",
      "  'title': 'Are Emergent Abilities of Large Language Models a Mirage?',\n",
      "  'link': 'https://arxiv.org/abs/2304.15004',\n",
      "  'engines': ['qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'What Is a Large Language Model? In its simplest terms, an LLM is '\n",
      "             'a massive database of text data that can be referenced to '\n",
      "             'generate human-like responses to your prompts. The text comes '\n",
      "             'from a range of sources and can amount to billions of words. '\n",
      "             'Among common sources of text data used are:',\n",
      "  'title': 'What Are Large Language Models (LLMs) and How Do They Work? - MUO',\n",
      "  'link': 'https://www.makeuseof.com/what-are-large-langauge-models-how-do-they-work/',\n",
      "  'engines': ['qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'ChatGPT, Google Bard, and other bots like them, are examples of '\n",
      "             \"large language models, or LLMs, and it's worth digging into how \"\n",
      "             \"they work. It means you'll be able to better make use of \"\n",
      "             'them,...',\n",
      "  'title': 'How ChatGPT and Other LLMs Work—and Where They Could Go Next',\n",
      "  'link': 'https://www.wired.com/story/how-chatgpt-works-large-language-model/',\n",
      "  'engines': ['duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Fig.2- Large Language Models. One of the most well-known large '\n",
      "             'language models is GPT-3, which has 175 billion parameters. In '\n",
      "             'GPT-4, Which is even more powerful than GPT-3 has 1 Trillion '\n",
      "             \"Parameters. It's awesome and scary at the same time. These \"\n",
      "             'parameters essentially represent the \"knowledge\" that the model '\n",
      "             'has acquired during its training.',\n",
      "  'title': 'Large Language Models and GPT-4 Explained | Towards AI',\n",
      "  'link': 'https://pub.towardsai.net/large-language-models-and-gpt-4-architecture-and-openai-api-d8f1c070e0fc',\n",
      "  'engines': ['duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Fig.2- Large Language Models. One of the most well-known large '\n",
      "             'language models is GPT-3, which has 175 billion parameters. In '\n",
      "             'GPT-4, Which is even more powerful than GPT-3 has 1 Trillion '\n",
      "             \"Parameters. It's awesome and scary at the same time. These \"\n",
      "             'parameters essentially represent the \"knowledge\" that the model '\n",
      "             'has acquired during its training.',\n",
      "  'title': 'Large Language Models and GPT-4: Architecture and OpenAI API',\n",
      "  'link': 'https://towardsai.net/p/machine-learning/large-language-models-and-gpt-4-architecture-and-openai-api',\n",
      "  'engines': ['duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': \"Large language models like OpenAI's GPT-3 are massive neural \"\n",
      "             'networks that can generate human-like text, from poetry to '\n",
      "             'programming code. Trained using troves of internet data, these '\n",
      "             'machine-learning models take a small bit of input text and then '\n",
      "             \"predict the text that is likely to come next. But that's not all \"\n",
      "             'these models can do.',\n",
      "  'title': 'Solving a machine-learning mystery | MIT News | Massachusetts '\n",
      "           'Institute ...',\n",
      "  'link': 'https://news.mit.edu/2023/large-language-models-in-context-learning-0207',\n",
      "  'engines': ['duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'A large language model (LLM) is a type of artificial '\n",
      "             'intelligence ( AI) algorithm that uses deep learning techniques '\n",
      "             'and massively large data sets to understand, summarize, generate '\n",
      "             'and predict new content.',\n",
      "  'title': 'What is a large language model (LLM)? – TechTarget Definition',\n",
      "  'link': 'https://www.techtarget.com/whatis/definition/large-language-model-LLM',\n",
      "  'engines': ['google', 'qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'What is a large language model? LLMs are machine learning models '\n",
      "             'that utilize deep learning algorithms to process and understand '\n",
      "             'language. They’re trained with immense amounts of data to...',\n",
      "  'title': 'What is a large language model and how does it work?',\n",
      "  'link': 'https://www.fastcompany.com/90884581/what-is-a-large-language-model',\n",
      "  'engines': ['google', 'qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'A large language model, or LLM, is a deep learning algorithm '\n",
      "             'that can recognize, summarize, translate, predict and generate '\n",
      "             'text and other content based on knowledge gained from massive '\n",
      "             'datasets. Large language models are among the most successful '\n",
      "             'applications of transformer models.',\n",
      "  'title': 'What Are Large Language Models Used For? - blogs.nvidia.com',\n",
      "  'link': 'https://blogs.nvidia.com/blog/2023/01/26/what-are-large-language-models-used-for/',\n",
      "  'engines': ['google', 'qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'GPT-3 can translate language, write essays, generate computer '\n",
      "             'code, and more — all with limited to no supervision. In July '\n",
      "             '2020, OpenAI unveiled GPT-3, a language model that was easily '\n",
      "             'the largest known at the time. Put simply, GPT-3 is trained to '\n",
      "             'predict the next word in a sentence, much like how a text '\n",
      "             'message autocomplete feature works.',\n",
      "  'title': 'How Large Language Models Will Transform Science, Society, and AI',\n",
      "  'link': 'https://hai.stanford.edu/news/how-large-language-models-will-transform-science-society-and-ai',\n",
      "  'engines': ['google', 'qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'A large language model, or LLM, is a deep learning model that '\n",
      "             'can understand, learn, summarize, translate, predict, and '\n",
      "             'generate text and other content based on knowledge gained from '\n",
      "             'massive datasets. Large language models - successful '\n",
      "             'applications of transformer models.',\n",
      "  'title': 'An Introduction to Large Language Models (LLMs)',\n",
      "  'link': 'https://www.analyticsvidhya.com/blog/2023/03/an-introduction-to-large-language-models-llms/',\n",
      "  'engines': ['google', 'qwant', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': \"3) Massive sparse expert models. Today's most prominent large \"\n",
      "             'language models all have effectively the same architecture. Meta '\n",
      "             'AI chief Yann LeCun said recently: \"In terms of underlying ...',\n",
      "  'title': 'The Next Generation Of Large Language Models',\n",
      "  'link': 'https://www.forbes.com/sites/robtoews/2023/02/07/the-next-generation-of-large-language-models/',\n",
      "  'engines': ['google', 'duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Apr 28, 2023 — A large language model (LLM) is a type of machine '\n",
      "             'learning model that can perform a variety of natural language '\n",
      "             'processing (NLP) tasks, ...',\n",
      "  'title': 'What is Large Language Model (LLM)?',\n",
      "  'link': 'https://www.techopedia.com/definition/34948/large-language-model-llm',\n",
      "  'engines': ['google'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Large language models are computer programs that open new '\n",
      "             'possibilities of text understanding and generation in software '\n",
      "             'systems. Consider this: ...',\n",
      "  'title': 'Introduction to Large Language Models',\n",
      "  'link': 'https://docs.cohere.com/docs/introduction-to-large-language-models',\n",
      "  'engines': ['google'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Large language models (LLMs) power ChatGPT, and these models are '\n",
      "             'the topic of this post. Before considering LLMs more carefully, '\n",
      "             'we would first like to establish what a language model does. A '\n",
      "             'language model gives a probability distribution of a word being '\n",
      "             'valid in a sequence of words. Essentially, the job of a language '\n",
      "             'model is to predict which ...',\n",
      "  'title': 'An Introduction to Large Language Models: Prompt Engineering and P '\n",
      "           '...',\n",
      "  'link': 'https://developer.nvidia.com/blog/an-introduction-to-large-language-models-prompt-engineering-and-p-tuning/',\n",
      "  'engines': ['duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Large Language Model (LLM) AI is a term that refers to AI models '\n",
      "             'that can generate natural language texts from large amounts of '\n",
      "             'data. Large language models use deep neural networks, such as '\n",
      "             'transformers, to learn from billions or trillions of words, and '\n",
      "             'to produce texts on any topic or domain.',\n",
      "  'title': 'Concepts Overview for LLM AI | Microsoft Learn',\n",
      "  'link': 'https://learn.microsoft.com/en-us/semantic-kernel/concepts-ai/',\n",
      "  'engines': ['duckduckgo'],\n",
      "  'category': 'general'},\n",
      " {'snippet': 'Large language models (LLMs) represent a major advancement in '\n",
      "             'AI, with the promise of transforming domains through learned '\n",
      "             'knowledge. LLM sizes have been increasing 10X every year for the '\n",
      "             'last few years, and as these models grow in complexity and size, '\n",
      "             'so do their capabilities.',\n",
      "  'title': 'Large language models (LLMs) | NVIDIA',\n",
      "  'link': 'https://www.nvidia.com/en-in/deep-learning-ai/solutions/large-language-models/',\n",
      "  'engines': ['duckduckgo'],\n",
      "  'category': 'general'}]\n"
     ]
    }
   ],
   "source": [
    "results = search.results(\"large language model\", num_results = 20, engines=['github', 'gitlab'])\n",
    "pprint.pp(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "b1a31457",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 您还可以将此包装器作为工具加载\n",
    "from langchain.agents import load_tools\n",
    "tools = load_tools([\"searx-search\"],\n",
    "                    searx_host=\"http://localhost:8888\",\n",
    "                    engines=[\"github\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "047d107f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 请注意，我们可以选择传递要使用的自定义引擎。\n",
    "# 如果你想获得元数据作为json 的结果，你可以使用\n",
    "tools = load_tools([\"searx-search-results-json\"],\n",
    "                    searx_host=\"http://localhost:8888\",\n",
    "                    num_results=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96a1f52b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}