Skip to content

Latest commit

 

History

History
286 lines (267 loc) · 8.46 KB

README_en.md

File metadata and controls

286 lines (267 loc) · 8.46 KB
Raw

search legend

[中文版|English|Release Notes|Contributors

what it is?

In 2023 or the second half of 2022, after several years of dormancy, the big model finally showed his powerful ability to the world in the form of chatgpt. This project is the entry project of the Gottingen organization, which aims to introduce how to implement a search system and provide a suitable way to access the components of the large model. The project takes engineering realization and actual landing projects as the main line, and gradually explains the principle, design and function of the system.

Technical discussions

reference project

flowchart TB
    A[turbo]-->B[libtext]
    A[turbo]-->BB[titan-redis]
    A[turbo]-->BC[titan-ann]
    BB-->C[lambda]
    BC-->C[lambda]
    A[turbo]-->E[rigel]
    E[rigel]-->D[hercules]
    D[hercules]-->F[hvm]
    C[lambda]-->F[hvm]
    B[libtext]-->F[hvm]
    E[rigel]-->G[melon-rpc]
    G[melon-rpc]-->F[hvm]
Loading

  • turbo c++ foundation library.

  • libtext chinese segment and normalize library.

  • titan-redis disk storage using redis interface.

  • titan-ann ann search engine base on graph.

  • lambda host local search engine for mix term and vector search.

  • rigel ai and rpc runtime library, to manage memory and fiber or thread schedule.

  • hercules python Aot framework for integrated components interface rpc and ai backend like tesorflow or pytorch

  • hvm (not start now) framework gateway, let us write python and generate c++ code and run in llvm.

the goals

flowchart LR
    A[python code]-->B[hvm.compile]
    B-->C[so library and graph]
    G[ci/cd]-->C[so library and graph]
    D[c++ runtime]-.->G
    D[c++ runtime]-->E[c++ app]
    F[service mess]-->E[c++ app]
Loading

it will reduce c++ communication and development phrase during business development。 may reduce business development days a lot and provide bug free code.

how ever, in some Modularity is not good system, you need to do a lot work int test and online, it is a very frustrating thing that， if many people develop together, so fuzz..

flowchart LR
  A[python algorithm]-->B[c++ review]-->C[c++ develop]-->D[test and online]
  AA[python algorithm]-->E[generate c++]-->D[test and online]
Loading

inspire by the ai aot. design it write a few python code like below

import hvm

q:str='刘德华'
def do_search(q:tr)->List[Any]
  qp = hvm.QU.parse(q)
  qt = hvm.query.boolen_query
  sq = hvm.query.term_query(qp)
  sq.must('title')
  qt.add_sub_query(sq)
  searcher  = hvm.search
  searcher.set_relavece('bm25')
  searcher.set_ranker('dw_fm')
  searcher.set_l1_sort('default')
  searcher.set_l2_sort('ctr_pri_sorter')
  result = searcher.search(engine)
  return result
hvm.compile(do_search, "./search_demo")

at c++ side,

hvm::Session session;
bool rs = session.load("./search_demo")
if(!rs){
    return false;
}      
string query;
session.run(query);

so it can do things in c++ code,Enjoy the joy brought by c++. and hidden the difficult code via c++ and hidden complex service governance logic behind.

nlp roadmap

Table of contents