Introduction to AStyle
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
@@ -205,7 +206,7 @@ Introduction to AStyle
choco install astyle
+
+ Hadoop Commands
++
Introduction to Apache Hadoop
+Apache Hadoop is an open-source framework that allows for the distributed processing of large data sets + across + clusters of computers using simple programming models. It is designed to scale up from a single server + to + thousands of machines, each offering local computation and storage. Hadoop is known for its reliability + and + ability to handle vast amounts of data efficiently.
+ +The Hadoop Distributed File System (HDFS) is a fundamental component of Hadoop, providing scalable and + reliable data + storage. HDFS is designed to store very large files across a cluster of machines, ensuring fault + tolerance and high + availability. +
+ +Another critical component of Hadoop is the MapReduce programming model, which enables efficient data + processing across + the distributed storage provided by HDFS. MapReduce breaks down data processing tasks into two main + phases: the Map + phase, which filters and sorts the data, and the Reduce phase, which performs aggregation and + summarization.Together with YARN (Yet Another Resource Negotiator), which manages and schedules + resources in the Hadoop cluster, these + components make Hadoop a robust platform for big data analytics, allowing organizations to gain insights + from their data + at unprecedented scales.
+Prerequisites
+ +Before starting Hadoop, ensure you have:
+-
+
- Properly installed Hadoop on the system. +
- Configured the necessary environment variables (
HADOOP_HOME,JAVA_HOME, + etc.). +
+ - Configured the core Hadoop configuration files (
core-site.xml, +hdfs-site.xml, +yarn-site.xml, andmapred-site.xml). +
+
Example Workflow
+-
+
- Start Command Prompt in Admin Mode:
+
-
+
- Press
Windows Key + X, select Command Prompt (Admin), and + confirm any + prompts.
+
+ - Press
- Navigate to the
sbinDirectory: +-
+
cd C:\Hadoop\sbin+
+
+ - Start HDFS:
+
-
+
start-dfs.cmd+
+
+ - Start YARN:
+
-
+
start-yarn.cmd
+
+ - Verify HDFS:
+
-
+
- Open http://localhost:9870/ in the web + browser. +
+ - Verify YARN:
+
-
+
- Open http://localhost:8088/ in the web + browser. +
+
Following these steps will help ensure that Hadoop services are correctly started and running on the + local machine. + For more detailed information, refer to the official Apache Hadoop documentation. Detailed steps for starting this services are + provided later in + this page.
+Follow these steps to start the Hadoop Distributed File System (HDFS) and YARN (Yet + Another Resource + Negotiator): +
+Step 1: Open Command Prompt in Administrator Mode
+To avoid permission issues, it's important to run the command prompt with administrative privileges.
+-
+
- Press
Windows Key + Xand select Command Prompt (Admin) or + Windows + PowerShell (Admin). +
+ - Confirm any prompts to allow the command prompt to run as an administrator. +
Step 2: Navigate to the Hadoop sbin Directory
Change the directory to the Hadoop sbin directory where the startup scripts are located.
+
cd C:\Hadoop\sbin+
-
+
- The command
cd C:\Hadoop\sbinchanges the current directory to the +sbinfolder in + the Hadoop installation directory. +
+
-
+
- The prompt will change to
C:\Hadoop\sbin, indicating that you are now in the + correct directory. +
+
Step 3: Start the Hadoop Distributed File System (HDFS) +
Run the following command to start the HDFS daemons (NameNode, Secondary NameNode, and DataNode):
+start-dfs.cmd+
-
+
- The command
start-dfs.cmdstarts the Hadoop Distributed File System (HDFS) daemons, + including + the NameNode, Secondary NameNode, and DataNode processes.
+ - These processes are responsible for managing the distributed storage in Hadoop, ensuring data is + stored + across multiple nodes for fault tolerance and high availability. +
-
+
- Log messages indicating that the NameNode, Secondary NameNode, and DataNode services have + started + successfully. +
- The prompt will return, indicating the command has finished executing. +
Step 4: Start YARN +
Run the following command to start YARN daemons (ResourceManager and NodeManager):
+start-yarn.cmd+
-
+
- The command
start-yarn.cmdstarts the YARN (Yet Another Resource Negotiator) + daemons, including + the ResourceManager and NodeManager processes.
+ - The ResourceManager is responsible for managing resources across the cluster and scheduling + applications, + while the NodeManager manages resources on a single node. +
-
+
- Log messages indicating that the ResourceManager and NodeManager services have started + successfully. +
- The prompt will return, indicating the command has finished executing. +
Step 5: Verify Hadoop Services
+ +HDFS Verification
+Open a web browser and navigate to the HDFS web UI to verify that the DataNode and NameNode are running + correctly:
+ +This URL provides information about the HDFS NameNode status and allows you to browse the file system. +
+ +YARN Verification
+To verify that YARN is running correctly, navigate to the YARN ResourceManager web UI:
+ +This URL provides information about the cluster status, including running applications and available + resources. +
+Thank You
++ Thank you for visiting this website. If you have any questions or would + like to get in touch, please feel free to contact me. +
+ + + + + Visit on GitHub +