Feedforward Neural Network
Loss function
-
-
During forward propagation, the neural network makes predictions based on input data.
-The loss function compares these predictions to the true values and calculates a loss score.
-The loss score is a measure of how well the network is performing.
-The goal of training is to minimize the loss function.
-For regression problems, use MSE or MAE.
-For classification problems, use cross-entropy loss.
-For multi-class classification problems, use categorical cross-entropy loss.
+- During forward pass, the neural network makes predictions based on input data. +
- The loss function compares these predictions to the true values and calculates a loss score. +
- The loss score is a measure of how well the network is performing. +
- The goal of training is to minimize the loss function. +
Additional resources
+-
+
- What is a neural network? Video +
- Gradient descent, how neural networks learn Video +
- Backpropagation, how neural networks learn Video
diff --git a/search.json b/search.json
index af9b9f3..efef59d 100644
--- a/search.json
+++ b/search.json
@@ -32,7 +32,14 @@
"href": "2024/weeks/week02/slides.html#perceptron",
"title": "Week 02 Basics of Neural Networks",
"section": "Perceptron",
- "text": "Perceptron"
+ "text": "Perceptron\n\n\n\n\n\ngraph LR\n subgraph Inputs\n x1((x1))\n x2((x2))\n x3((x3))\n end\n\n sum((Σ))\n act[Activation]\n out((Output))\n b[Bias]\n\n x1 -->|w1| sum\n x2 -->|w2| sum\n x3 -->|w3| sum\n b --> sum\n sum --> act\n act --> out\n\n style Inputs fill:#87CEFA,stroke:#333,stroke-width:2px, fill-opacity: 0.5\n style x1 fill:#87CEFA,stroke:#333,stroke-width:2px\n style x2 fill:#87CEFA,stroke:#333,stroke-width:2px\n style x3 fill:#87CEFA,stroke:#333,stroke-width:2px\n style sum fill:#FFA07A,stroke:#333,stroke-width:2px\n style act fill:#98FB98,stroke:#333,stroke-width:2px\n style b fill:#FFFF00,stroke:#333,stroke-width:2px\n\n\n\n\n\n\n\nInput Nodes (x1, x2, x3): Each input is a number.\nWeights (w1, w2, w3): Each weight is a number that determines the importance of the corresponding input.\nBias (b): A constant value that shifts the output of the perceptron."
+ },
+ {
+ "objectID": "2024/weeks/week02/slides.html#section-3",
+ "href": "2024/weeks/week02/slides.html#section-3",
+ "title": "Week 02 Basics of Neural Networks",
+ "section": "",
+ "text": "Sum Node (Σ): Calculates the weighted sum of the inputs and the bias.\nActivation Function (\\(f\\)): Introduces non-linearity to the output of the perceptron.\nOutput Node: The final output of the perceptron.\n\n\\[\n\\text{Output} = f(w_1 \\times x_1 + w_2 \\times x_2 + w_3 \\times x_3 + b)\n\\]\n\nThe output of the perceptron is a weighted sum of the inputs and the bias passed through an activation function.\n\nWhy do we need non-linearity?\n\nNon-linearity allows the perceptron to learn complex patterns in the data.\nWithout non-linearity, the perceptron would be limited to learning linear patterns.\nActivation functions introduce non-linearity to the output of the perceptron."
},
{
"objectID": "2024/weeks/week02/slides.html#activation-functions",
@@ -42,15 +49,15 @@
"text": "Activation functions\n\nActivation functions are used to introduce non-linearity to the output of a neuron.\n\nSigmoid function\n\\[\nf(x) = \\frac{1}{1 + e^{-x}}\n\\]\nExample: \\(f(0) = 0.5\\)\n- f(x): This represents the output of the sigmoid function for a given input x.\n- e: This is the euler's number (approximately 2.71828).\n- x: This is the input to the sigmoid function.\n- 1: This is added to the denominator to avoid division by zero.\n\nThe sigmoid function takes any real number as input and outputs a value between 0 and 1.\nIt is used in the output layer of a binary classification problem."
},
{
- "objectID": "2024/weeks/week02/slides.html#section-3",
- "href": "2024/weeks/week02/slides.html#section-3",
+ "objectID": "2024/weeks/week02/slides.html#section-4",
+ "href": "2024/weeks/week02/slides.html#section-4",
"title": "Week 02 Basics of Neural Networks",
"section": "",
"text": "ReLU function\n\\[\nf(x) = \\max(0, x)\n\\]\nExample: \\(f(2) = 2\\)\nwhere:\n- f(x): This represents the output of the ReLU function for a given input x.\n- x: This is the input to the ReLU function.\n- max: This function returns the maximum of the two values.\n- 0: This is the threshold value.\n\nThe Rectified Linear Unit (ReLU) function is that outputs the input directly if it is positive, otherwise, it outputs zero.\nThe output of the ReLU function is between 0 and infinity.\nIt is a popular activation function used in deep learning models."
},
{
- "objectID": "2024/weeks/week02/slides.html#section-4",
- "href": "2024/weeks/week02/slides.html#section-4",
+ "objectID": "2024/weeks/week02/slides.html#section-5",
+ "href": "2024/weeks/week02/slides.html#section-5",
"title": "Week 02 Basics of Neural Networks",
"section": "",
"text": "Feedforward Neural Network\n\n\n\n\n\nflowchart LR\n %% Input Layer\n I1((I1)):::inputStyle\n I2((I2)):::inputStyle\n I3((I3)):::inputStyle\n B1((Bias)):::biasStyle\n %% Hidden Layer\n H1((H1)):::hiddenStyle\n H2((H2)):::hiddenStyle\n H3((H3)):::hiddenStyle\n B2((Bias)):::biasStyle\n %% Output Layer\n O1((O1)):::outputStyle\n O2((O2)):::outputStyle\n\n %% Connections\n I1 -->|w11| H1\n I1 -->|w12| H2\n I1 -->|w13| H3\n I2 -->|w21| H1\n I2 -->|w22| H2\n I2 -->|w23| H3\n I3 -->|w31| H1\n I3 -->|w32| H2\n I3 -->|w33| H3\n B1 -->|b1| H1\n B1 -->|b2| H2\n B1 -->|b3| H3\n H1 -->|v11| O1\n H1 -->|v12| O2\n H2 -->|v21| O1\n H2 -->|v22| O2\n H3 -->|v31| O1\n H3 -->|v32| O2\n B2 -->|b4| O1\n B2 -->|b5| O2\n\n %% Styles\n classDef inputStyle fill:#3498db,stroke:#333,stroke-width:2px;\n classDef hiddenStyle fill:#e74c3c,stroke:#333,stroke-width:2px;\n classDef outputStyle fill:#2ecc71,stroke:#333,stroke-width:2px;\n classDef biasStyle fill:#f39c12,stroke:#333,stroke-width:2px;\n\n %% Layer Labels\n I2 -.- InputLabel[Input Layer]\n H2 -.- HiddenLabel[Hidden Layer]\n O1 -.- OutputLabel[Output Layer]\n\n style InputLabel fill:none,stroke:none\n style HiddenLabel fill:none,stroke:none\n style OutputLabel fill:none,stroke:none"
@@ -63,11 +70,18 @@
"text": "Feedforward Neural Network\n\nFeedforward neural network with three layers: input, hidden, and output.\nThe input layer has three nodes (I1, I2, I3).\nThe hidden layer has three nodes (H1, H2, H3).\nThe output layer has two nodes (O1, O2).\nEach connection between the nodes has a weight (w) and a bias (b).\nThe weights and biases are learned during the training process."
},
{
- "objectID": "2024/weeks/week02/slides.html#section-5",
- "href": "2024/weeks/week02/slides.html#section-5",
+ "objectID": "2024/weeks/week02/slides.html#section-6",
+ "href": "2024/weeks/week02/slides.html#section-6",
"title": "Week 02 Basics of Neural Networks",
"section": "",
- "text": "Loss function\n\nDuring forward propagation, the neural network makes predictions based on input data.\nThe loss function compares these predictions to the true values and calculates a loss score.\nThe loss score is a measure of how well the network is performing.\nThe goal of training is to minimize the loss function.\nFor regression problems, use MSE or MAE.\nFor classification problems, use cross-entropy loss.\nFor multi-class classification problems, use categorical cross-entropy loss.\n\n\n\n\n\nLLMs in Lingustic Research WiSe 2024/25"
+ "text": "Loss function\n\nDuring forward pass, the neural network makes predictions based on input data.\nThe loss function compares these predictions to the true values and calculates a loss score.\nThe loss score is a measure of how well the network is performing.\nThe goal of training is to minimize the loss function."
+ },
+ {
+ "objectID": "2024/weeks/week02/slides.html#additional-resources",
+ "href": "2024/weeks/week02/slides.html#additional-resources",
+ "title": "Week 02 Basics of Neural Networks",
+ "section": "Additional resources",
+ "text": "Additional resources\n\nWhat is a neural network? Video\nGradient descent, how neural networks learn Video\nBackpropagation, how neural networks learn Video\n\n\n\n\n\nLLMs in Lingustic Research WiSe 2024/25"
},
{
"objectID": "2024/homework/homework01.html",
diff --git a/sitemap.xml b/sitemap.xml
index af542dd..0d246b8 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -2,42 +2,42 @@
https://llm4linguists.xyz/2024/weeks/week02/slides.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.912Z
https://llm4linguists.xyz/2024/homework/homework01.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.908Z
https://llm4linguists.xyz/2024/weeks/week01/slides.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.908Z
https://llm4linguists.xyz/2024/syllabus.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.908Z
https://llm4linguists.xyz/2024/assignments.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.908Z
https://llm4linguists.xyz/index.html
- 2024-10-16T09:15:14.758Z
+ 2024-10-16T12:09:05.944Z
https://llm4linguists.xyz/2024/homework.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.908Z
https://llm4linguists.xyz/2024/weeks/week01/page.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.908Z
https://llm4linguists.xyz/2024/weeks/week02/page.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.912Z
https://llm4linguists.xyz/2024/homework/homework02.html
- 2024-10-16T09:15:14.722Z
+ 2024-10-16T12:09:05.908Z