From 80151850ea540fccb869eae5d5286b0b9560fd4a Mon Sep 17 00:00:00 2001 From: AlexCHEN Date: Thu, 22 Feb 2024 23:21:35 -0800 Subject: [PATCH] Built site for gh-pages --- .nojekyll | 2 +- about.html | 6 +----- hello.html | 6 +----- index.html | 6 +----- notes.html | 8 ++------ notes.xml | 13 +++++++++++++ notes/Diffusion Model/sd.html | 6 +----- notes/Large Language Model/llm_eval.html | 19 ++++++++++++++----- notes/Large Language Model/moe.html | 6 +----- notes/Large Language Model/rl_llm.html | 6 +----- search.json | 14 -------------- sitemap.xml | 6 +----- 12 files changed, 37 insertions(+), 61 deletions(-) diff --git a/.nojekyll b/.nojekyll index 68a0184..598faf8 100644 --- a/.nojekyll +++ b/.nojekyll @@ -1 +1 @@ -d52de7b8 \ No newline at end of file +5bb44101 \ No newline at end of file diff --git a/about.html b/about.html index ab9e271..9a89ca4 100644 --- a/about.html +++ b/about.html @@ -95,11 +95,7 @@
  • -Notes -
    • -
  • - -LLM code example +Articles
  • diff --git a/hello.html b/hello.html index d8d1caf..bb06e4d 100644 --- a/hello.html +++ b/hello.html @@ -158,11 +158,7 @@
  • -Notes -
    • -
  • - -LLM code example +Articles
  • diff --git a/index.html b/index.html index 5725aae..cada6d4 100644 --- a/index.html +++ b/index.html @@ -94,11 +94,7 @@
  • -Notes -
    • -
  • - -LLM code example +Articles
  • diff --git a/notes.html b/notes.html index 527d871..83ef583 100644 --- a/notes.html +++ b/notes.html @@ -156,11 +156,7 @@
  • -Notes -
    • -
  • - -LLM code example +Articles
  • @@ -224,7 +220,7 @@

    Research notes

  • -Notes -
    • -
  • - -LLM code example +Articles
  • diff --git a/notes/Large Language Model/llm_eval.html b/notes/Large Language Model/llm_eval.html index d8b2b30..2a7f8d9 100644 --- a/notes/Large Language Model/llm_eval.html +++ b/notes/Large Language Model/llm_eval.html @@ -97,11 +97,7 @@
  • -Notes -
    • -
  • - -LLM code example +Articles
  • @@ -259,6 +255,19 @@

    Large language model evaluation

    Benchmarks for evaluating large language models come in various forms, each serving a unique purpose. They can be broadly categorized into general benchmarks, which assess overall performance, and specialized benchmarks, designed to evaluate the model’s proficiency in specific areas such as understanding the Chinese language or its ability to perform function calls.

    +
    +
    +
    + +
    +
    +Tip +
    +
    +
    +

    Consider to learn Stanford CS224U if you want to learn more fundamental knowledge about the LLM evaluation.

    +
    +

    LLM leaderboard

    Numerous leaderboards exist for Large Language Models (LLMs), each compiled based on the benchmarks of these models. By examining these leaderboards, we can identify which benchmarks are particularly effective and informative for evaluating the capabilities of LLMs.

    diff --git a/notes/Large Language Model/moe.html b/notes/Large Language Model/moe.html index c5243f3..a48f8f5 100644 --- a/notes/Large Language Model/moe.html +++ b/notes/Large Language Model/moe.html @@ -160,11 +160,7 @@
  • -Notes -
    • -
  • - -LLM code example +Articles
  • diff --git a/notes/Large Language Model/rl_llm.html b/notes/Large Language Model/rl_llm.html index b3ca73b..1a75fdb 100644 --- a/notes/Large Language Model/rl_llm.html +++ b/notes/Large Language Model/rl_llm.html @@ -160,11 +160,7 @@
  • -Notes -
    • -
  • - -LLM code example +Articles
  • diff --git a/search.json b/search.json index 9d06e0d..58be494 100644 --- a/search.json +++ b/search.json @@ -128,20 +128,6 @@ "section": "", "text": "Large language model evaluation\n\n\n\n\n\n\nLarge Language Models\n\n\n\n\n\n\n\n\n\n11 min\n\n\n\n\n\n\n\n\n\n\n\n\nMixture of expert\n\n\n\n\n\n\nLarge Language Models\n\n\n\n\n\n\n\n\n\n5 min\n\n\n\n\n\n\n\n\n\n\n\n\nScalable diffusion models with transformers\n\n\n\n\n\n\nDiffusion Model\n\n\n\n\n\n\n\n\n\n1 min\n\n\n\n\n\n\n\n\n\n\n\n\nReinforcement learning for large language model\n\n\n\n\n\n\nLarge Language Models\n\n\n\n\n\n\n\n\n\n19 min\n\n\n\n\n\n\nNo matching items" }, - { - "objectID": "chat_model.html", - "href": "chat_model.html", - "title": "Chat model demo", - "section": "", - "text": "The chat models are very different from the other models. We should spend more time for the data, especially during training.\n\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.1\")\n\nchat = [\n {\"role\": \"user\", \"content\": \"Hello, how are you?\"},\n {\"role\": \"assistant\", \"content\": \"I'm doing great. How can I help you today?\"},\n {\"role\": \"user\", \"content\": \"I'd like to show off how chat templating works!\"},\n]\n\ntokenizer.apply_chat_template(chat, tokenize=False)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\"<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]\"\n\n\n\n# each model has different tokenizer! We can access the chat templates for each model\ntokenizer.chat_template\n\n\"{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token + ' ' }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}\"\n\n\n\n# During training, we should also use the chat templates to rearrange the dataset\nfrom transformers import AutoTokenizer\nfrom datasets import Dataset\n\ntokenizer = AutoTokenizer.from_pretrained(\"HuggingFaceH4/zephyr-7b-beta\")\n\nchat1 = [\n {\"role\": \"user\", \"content\": \"Which is bigger, the moon or the sun?\"},\n {\"role\": \"assistant\", \"content\": \"The sun.\"},\n]\nchat2 = [\n {\"role\": \"user\", \"content\": \"Which is bigger, a virus or a bacterium?\"},\n {\"role\": \"assistant\", \"content\": \"A bacterium.\"},\n]\n\ndataset = Dataset.from_dict({\"chat\": [chat1, chat2]})\ndataset = dataset.map(\n lambda x: {\n \"formatted_chat\": tokenizer.apply_chat_template(\n x[\"chat\"], tokenize=False, add_generation_prompt=False\n )\n }\n)\nprint(dataset[\"formatted_chat\"][0])\n\n\n\nWe can also fill the missing tokens in the middle of the sentence. We just need to tell the language model the content before and after the missing part. And the model will handle the missing part automatically.\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n \"stabilityai/stable-code-3b\", trust_remote_code=True\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n \"stabilityai/stable-code-3b\",\n trust_remote_code=True,\n torch_dtype=\"auto\",\n # attn_implementation=\"flash_attention_2\",\n)\nmodel.cuda()\n\n# <fim_prefix> is the prefix code before the missiong part\n# <fim_suffix> is the suffix code after the missiong part\n# <fim_middle> is the missing part, and we add the token in the final part so that the model can predict it\ninputs = tokenizer(\n \"<fim_prefix>def fib(n):<fim_suffix> else:\\n return fib(n - 2) + fib(n - 1)<fim_middle>\",\n return_tensors=\"pt\",\n).to(model.device)\ntokens = model.generate(\n **inputs,\n max_new_tokens=48,\n temperature=0.2,\n do_sample=True,\n)\nprint(tokenizer.decode(tokens[0], skip_special_tokens=True))" - }, - { - "objectID": "chat_model.html#fill-in-middle-model", - "href": "chat_model.html#fill-in-middle-model", - "title": "Chat model demo", - "section": "", - "text": "We can also fill the missing tokens in the middle of the sentence. We just need to tell the language model the content before and after the missing part. And the model will handle the missing part automatically.\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n \"stabilityai/stable-code-3b\", trust_remote_code=True\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n \"stabilityai/stable-code-3b\",\n trust_remote_code=True,\n torch_dtype=\"auto\",\n # attn_implementation=\"flash_attention_2\",\n)\nmodel.cuda()\n\n# <fim_prefix> is the prefix code before the missiong part\n# <fim_suffix> is the suffix code after the missiong part\n# <fim_middle> is the missing part, and we add the token in the final part so that the model can predict it\ninputs = tokenizer(\n \"<fim_prefix>def fib(n):<fim_suffix> else:\\n return fib(n - 2) + fib(n - 1)<fim_middle>\",\n return_tensors=\"pt\",\n).to(model.device)\ntokens = model.generate(\n **inputs,\n max_new_tokens=48,\n temperature=0.2,\n do_sample=True,\n)\nprint(tokenizer.decode(tokens[0], skip_special_tokens=True))" - }, { "objectID": "about.html", "href": "about.html", diff --git a/sitemap.xml b/sitemap.xml index 139d3ed..5cdde38 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -10,7 +10,7 @@ https://alexchen4ai.github.io/blog/notes/Large Language Model/llm_eval.html - 2024-02-23T07:11:00.761Z + 2024-02-23T07:17:16.360Z https://alexchen4ai.github.io/blog/index.html @@ -20,10 +20,6 @@ https://alexchen4ai.github.io/blog/notes.html 2024-02-03T23:13:18.582Z - - https://alexchen4ai.github.io/blog/chat_model.html - 2024-02-04T06:59:51.701Z - https://alexchen4ai.github.io/blog/about.html 2024-02-09T08:17:04.643Z