Important Takeaways

I learned a lot about the inner workings of large language models and technical tools that can help fine tune them for specific purposes. Here are some important takeaways I want to share:

Metadata are crucial in this project. With metadata, files will not only be more accurately retrieved, but responses are also more accurately generated by the language model. Metadata also includes important and otherwise unavailable information such as the relevance score for retrieved files: the response coming back from Azure Cognitive Search contains a floating point number on a scale of 0 to 1.0, that represents how relevant each file is given the user query. In addition, the LLM also gives a numeric value for the usage of each source file when generating responses.

It was surprising to see these numeric values exist to help reflect the accuracy of language models. They exist thanks to the use of vectors. Language models uses vectors to store and comprehend information semantically: the closer the distance between two vectors, the closer their semantic meanings are. There are multiple ways to measure vector distances, including the more well-known Euclidean distance. In this project, the default distance calculation method cosine distance is used. This emphasizes the angle between two vectors regardless of their respective size.

Using LlamaIndex and LangChain, the app is able to process uploaded file into “embeddings”, which are then stored as “chunks” of Document objects. Such document objects contain vectors that represent semantic meanings. In this way, the app can communicate with the language model in terms of vectors.