Code and Module Overview Starting Point

[matthewmcneill]

grottdata.py

The grottdata.py module is responsible for processing data received from Growatt inverters. It performs the following tasks:

1. Data Decryption: Decrypts the received data using a predefined mask ("Growatt").

2. Protocol Detection: Automatically detects the protocol and record layout used by the inverter based on the data header and length.

3. Data Parsing: Parses the decrypted data according to the detected record layout, extracting key-value pairs defined in a configuration file (conf.recorddict).

4. Date/Time Handling: Extracts the date and time from the data record if available, otherwise uses the server's time. Handles buffered records and timezones.

5. Data Formatting: Formats the extracted data into a JSON object.

6. Data Publishing: Publishes the JSON data to an MQTT broker and/or an InfluxDB database, if enabled in the configuration.

7. PVOutput Integration: Sends data to PVOutput, a solar monitoring platform, if enabled in the configuration.

8. Extension Support: Allows for custom data processing through user-defined extension modules.

Key Features:

• Automatic protocol and record layout detection.

• Support for both encrypted and unencrypted data.

• Flexible data parsing based on configurable record layouts.

• Date/Time handling with timezone support.

• Integration with MQTT, InfluxDB, and PVOutput.

• Extensible through user-defined modules.

Usage:

The procdata() function is the main entry point for processing data. It takes the configuration object (conf) and the raw data (data) as input and performs all the necessary processing steps.

Note:

The module relies on a configuration file (conf) that defines various parameters such as inverter serial number, MQTT broker details, InfluxDB connection settings, PVOutput API key, etc.

The data parsing in grottdata.py is driven by a configuration file represented by the conf.recorddict dictionary. This dictionary holds parsing instructions for different data record layouts received from Growatt inverters.

Here's a breakdown of the parsing process:

1. Layout Detection:

   • The code first determines the layout of the incoming data record based on its header and length. This layout is a string key used to access parsing instructions within conf.recorddict.

   • For example, a layout string could be "T050104NN" representing a specific protocol version and data type.

2. Keyword Iteration:

   • The code iterates through each keyword defined in the conf.recorddict for the detected layout. These keywords represent individual data points within the record, like "pvpowerin" for input power.

3. Inclusion Check:

   • For each keyword, the code checks if it should be included in the output. This is determined by an optional "incl" field in the configuration. If "incl" is set to "no", the keyword is skipped.

4. Type Handling:

   • The code then determines the data type of the keyword using the "type" field in the configuration. Supported types include:

     • "text": Extracts a string from the raw data and decodes it from hexadecimal to UTF-8.

     • "num": Extracts a value from the raw data and converts it from hexadecimal to an integer.

     • "numx": Extracts a value and converts it to a signed integer.

     • "log": Extracts a value from a log section of the data, which is pre-parsed into a list.

     • "logpos"/"logneg": Similar to "log", but only includes the value if it's positive or negative, respectively.

5. Value Extraction and Conversion:

   • Based on the data type, the code extracts the corresponding value from the raw data string using the "value" and "length" fields from the configuration.

   • The "value" field specifies the starting position of the data within the string, while "length" determines how many characters to extract.

   • Extracted values are then converted to the appropriate data type and stored in the definedkey dictionary.

6. Output Generation:

   • Once all keywords are processed, the definedkey dictionary, containing the parsed data points, is used to generate output in JSON format. This JSON data can then be sent to various destinations like MQTT brokers, InfluxDB, or PVOutput.

Example:

Let's say conf.recorddict contains the following entry for a keyword "pvpowerin":

"pvpowerin": {
    "value": 62,
    "length": 4,
    "type": "num",
    "divide": 10
}

This configuration indicates that the "pvpowerin" value starts at position 62 in the raw data string, has a length of 4 characters, is a numerical value, and should be divided by 10. The parsing code will extract the characters from position 62 to 65, convert them from hexadecimal to an integer, divide the result by 10, and store it in the definedkey dictionary under the "pvpowerin" key.

The logpos, logneg, and log types in grottdata.py are used to process data extracted from a specific section of the Growatt inverter data record referred to as the "log section". This section typically contains a list of comma-separated values.

Here's a breakdown of each type:

1. log:

• Purpose: Extracts a value from the log section at a specified position.

• Configuration:

  • "pos": (Required) An integer representing the position of the desired value within the log section (starting from 1).

• Example:

"total_energy": {
    "value": 0,
    "length": 0,
    "type": "log",
    "pos": 10,
    "divide": 10
}

This configuration extracts the 10th value from the log section, divides it by 10, and stores it in the definedkey dictionary under the key "total_energy".

2. logpos:

• Purpose: Extracts a value from the log section only if it's positive.

• Configuration:

  • "pos": (Required) Same as log.

• Example:

"positive_power": {
    "value": 0,
    "length": 0,
    "type": "logpos",
    "pos": 5
}

This configuration extracts the 5th value from the log section only if it's positive. If it's negative or zero, the value stored in definedkey for "positive_power" will be 0.

3. logneg:

• Purpose: Extracts a value from the log section only if it's negative.

• Configuration:

  • "pos": (Required) Same as log.

• Example:

"negative_power": {
    "value": 0,
    "length": 0,
    "type": "logneg",
    "pos": 5
}

This configuration extracts the 5th value from the log section only if it's negative. If it's positive or zero, the value stored in definedkey for "negative_power" will be 0.

Key Points:

• The log section is pre-parsed into a list of strings before these types are applied.

• The value and length fields in the configuration are ignored for these types, as the position within the log section determines the value to be extracted.

• These types are particularly useful for extracting specific data points from a larger set of logged values within the inverter data record.

grottconf.py

The module defines a Python class named Conf which is responsible for handling configuration settings for the "Grott" application, which is a tool for monitoring and logging data from Growatt solar inverters. The module focuses on:

1. Configuration Management:

   • Loading configuration settings from various sources:

     • Command-line arguments (using argparse).

     • Environment variables.

     • A configuration file (using configparser), with "grott.ini" as the default.

   • Providing a structured way to access and manage these settings through the Conf class.

2. Parameter Processing:

   • Parsing command-line arguments and validating their values.

   • Reading and interpreting configuration file entries.

   • Retrieving and handling environment variables.

   • Prioritizing settings from different sources (command-line > environment > config file > defaults).

3. Application Setup:

   • Initializing variables and data structures based on the loaded configuration.

   • Setting up connections to external services like MQTT brokers, InfluxDB databases, and PVOutput, if enabled.

   • Defining data layouts and whitelists for processing inverter data.

4. Output and Logging:

   • Controlling the verbosity of the application's output using the verbose and trace settings.

   • Redirecting output to a file or standard output.

In essence, the grottconf.py module acts as the configuration backbone for the Grott application, ensuring that it runs with the desired settings and connects to the appropriate services based on user preferences and system environment.

The set_reclayouts function in the provided Python code defines a dictionary named self.recorddict. This dictionary stores various data layouts used to parse information received from Growatt solar inverters.

The lines self.recorddict.update(self.recorddict1) is merging the contents of the dictionary self.recorddict1 into the dictionary self.recorddict.

Let's break it down:

• self.recorddict: This is a dictionary that, based on the code's purpose, likely stores various data layouts used to parse information from Growatt solar inverters.

• self.recorddict1: This is another dictionary containing a specific set of data layout definitions.

• .update(): This is a Python dictionary method. It takes another dictionary as an argument and adds all its key-value pairs into the dictionary it's called upon. If keys already exist, their values are updated with the values from the argument dictionary.

In essence, this line is adding (or updating) data layout definitions in self.recorddict with the definitions found in self.recorddict1. This is a common way in Python to combine dictionary contents, especially when building up a larger dictionary from smaller ones.

Here's a breakdown:

1. Purpose:

   • The function aims to create a mapping between specific data record types and their corresponding parsing instructions. This mapping is crucial for correctly extracting meaningful values from the raw data stream sent by the inverter.

2. Data Structure:

   • self.recorddict is a nested dictionary.

     • Outer Level Keys: These keys represent different data record layouts, identified by a combination of characters (e.g., "T02NNNN", "T05NNNN"). These identifiers likely correspond to specific protocol versions or data structures used by different Growatt inverter models.

     • Inner Level Dictionaries: Each outer key maps to another dictionary containing parsing instructions for that specific layout.

3. Parsing Instructions:

   • Within each inner dictionary, there are key-value pairs defining how to extract individual data points from the raw record.

     • Keys: These represent the names of the data points being extracted (e.g., "datalogserial", "pvpowerin", "pvtemperature").

     • Values: These are dictionaries themselves, containing the following fields:

       • "value": Specifies the starting position (offset) of the data point within the raw record.

       • "length": Indicates the number of characters/bytes occupied by the data point.

       • "type": Defines the data type of the value (e.g., "text", "num", "numx").

       • "divide": (Optional) If present, the extracted value is divided by this number for scaling or unit conversion.

       • "incl": (Optional) Controls whether this data point should be included in the output. A value of "no" excludes it.

4. **Example:**Consider this entry from the code:

"pvpowerin": {"value" :82, "length" : 4, "type" : "num", "divide" : 10}, 

This instruction indicates that the "pvpowerin" data point:

• Starts at position 82 in the raw record.

• Has a length of 4 characters/bytes.

• Represents a numerical value ("num" type).

• Needs to be divided by 10 after extraction.

5. **Usage:**Later in the code, when a data record is received, the code will:

   • Identify the record layout based on its header information.

   • Use the corresponding layout key from self.recorddict to access the correct parsing instructions.

   • Extract and process individual data points according to these instructions.

In summary, the set_reclayouts function plays a vital role in defining the data extraction rules for different Growatt inverter data formats, enabling the application to interpret and utilize the received information effectively.

grottproxy.py

grottproxy.py acts as a man-in-the-middle (proxy) between Growatt solar inverters and the Growatt server, allowing for data interception and manipulation. Here's a breakdown:

Core Functionality:

1. Data Forwarding: It establishes a connection to both the Growatt inverter and the official Growatt server, relaying data between them. This ensures the inverter can still communicate with the server for essential functions.

2. Data Interception: The proxy intercepts all data passing through it, giving you access to the raw data stream from the inverter.

3. Data Processing: It utilizes the procdata function from grottdata.py to parse and interpret the intercepted data, extracting meaningful information like energy generation, voltage, and temperature.

4. Data Manipulation (Optional): The proxy can be configured to block specific commands sent from the Growatt server to the inverter, preventing unwanted actions like firmware updates or configuration changes.

5. Data Logging/Publishing: The parsed data can be further processed, logged locally, or published to external services like MQTT brokers for real-time monitoring and analysis.

Key Features:

• Configuration: The proxy's behavior is controlled by a configuration object (conf) that defines parameters like:

  • IP addresses and ports for the inverter, Growatt server, and the proxy itself.

  • Minimum record length for processing data.

  • Command blocking rules.

  • Verbosity of log messages.

• Data Validation: It validates incoming data records for integrity using CRC checks (if libscrc is installed) and length verification, ensuring only valid data is processed.

• Selective Command Blocking: You can configure the proxy to block specific commands based on their hexadecimal codes, providing granular control over server-inverter communication.

• Modularity: It leverages functions from grottdata.py for data processing, promoting code reusability and separation of concerns.

Use Cases:

• Local Data Monitoring: Intercept and analyze inverter data without relying on the Growatt server or cloud platform.

• Data Security: Prevent unauthorized access or control of the inverter by blocking specific commands.

• Custom Integrations: Integrate inverter data with third-party monitoring systems or home automation platforms.

• Troubleshooting: Analyze the raw communication between the inverter and server to diagnose issues.

Overall, grottproxy.py provides a powerful tool for gaining deeper insights into your Growatt solar inverter's operation, enhancing data security, and enabling custom integrations.

grottserver.py

grottserver.py emulates the behavior of the official Growatt server (server.growatt.com), acting as a local server for Growatt solar inverters to communicate with. This is particularly useful for debugging, testing, and potentially even bypassing certain limitations of the official server.

Here's a breakdown of its key functionalities:

1. Network Communication:

• Socket Server: It creates a socket server that listens on a specified IP address and port (configurable via serverhost and serverport). This server handles incoming connections from Growatt inverters and dataloggers.

• HTTP Server: It also runs a simple HTTP server (using http.server) on a different port (configurable via httphost and httpport). This server provides a basic web interface for:

  • Displaying server status and connected devices.

  • Manually sending commands to the inverter or datalogger.

  • Viewing responses from these devices.

2. Data Handling:

• Data Reception: The socket server receives data packets from connected inverters.

• Data Decryption/Encryption: It handles both encrypted (for newer protocols like 05 and 06) and unencrypted (protocol 02) data packets, decrypting incoming data and encrypting outgoing responses as needed.

• Data Parsing: It parses the received data packets, extracting information like:

  • Protocol version

  • Sequence number

  • Device ID (inverter or datalogger)

  • Command type

  • Data values (e.g., energy generation, voltage)

• Data Validation: It performs basic data validation, checking for correct packet length and CRC (Cyclic Redundancy Check) for certain protocols.

3. Command Handling:

• Command Recognition: It recognizes different command types sent by the inverters, such as:

  • Ping (heartbeat signal)

  • Data upload

  • Command responses

• Command Processing: Based on the command type, it performs appropriate actions:

  • Responds to pings to maintain the connection.

  • Acknowledges data uploads.

  • Processes command responses, storing the received data for later retrieval via the HTTP interface.

• Command Generation: It can also generate commands to be sent to the inverter, such as:

  • Time synchronization commands

  • Register read/write commands (for retrieving or setting specific parameters)

4. Device Management:

• Device Registration: It keeps track of connected devices (inverters and dataloggers), storing their IP addresses, ports, and communication protocols.

• Connection Handling: It manages the lifecycle of connections, handling new connections, disconnections, and potential errors.

5. User Interface:

• Web Interface: The HTTP server provides a basic web interface for interacting with the server, allowing users to:

  • View server status and connected devices.

  • Send commands to specific devices.

  • Retrieve and view data from devices.

While grottserver.py doesn't directly implement the Modbus protocol, it interacts with devices that basically use Modbus. It emulates the behavior of a Modbus master, handling data requests, commands, and responses, using elements like CRC checks and register addressing that are characteristic of Modbus communication.

Here's how the protocols are used:

1. Modbus as the Underlying Protocol: Growatt inverters often use Modbus, a common industrial communication protocol, to exchange data with monitoring devices or servers. This data includes things like energy generation, voltage, and system status.

2. grottserver.py as a Modbus Master: The script acts as a server that these inverters connect to. Given its role as a central data collector and command issuer, it functions as a Modbus "master" in this context. A Modbus master initiates communication and queries the inverters (acting as "slaves") for data or sends them commands.

3. CRC for Data Integrity: The script uses CRC (Cyclic Redundancy Check) calculations (libscrc.modbus). CRC is a common error-detection mechanism used in Modbus to ensure data integrity during transmission. 

4. Register-Based Data Access: The code includes references to "registers" (e.g., register = int(result_string[36+offset:40+offset],16)). Modbus organizes data within devices using "registers," which are essentially memory locations holding specific data points. The script reads from and writes to these registers to get information from or send commands to the inverters.

The encryption/decryption aspect of the grottserver.py script is designed to handle the communication protocol used by certain Growatt solar inverters. Here's a breakdown:

1. Purpose:

• Data Security: The encryption adds a layer of security to the communication between the inverter and the server, making it harder for unauthorized parties to eavesdrop on or tamper with the data.

• Protocol Compatibility: Growatt seems to have introduced encryption in later versions of their communication protocol (likely protocols 05 and 06, as indicated in the code). The script needs to handle both encrypted and unencrypted communication to support different inverter models.

2. Encryption/Decryption Mechanism:

• XOR Cipher: The code uses a simple XOR (exclusive OR) cipher for encryption and decryption. This is a symmetric encryption method where each byte of the data is XORed with a corresponding byte from a secret key (the mask).

• Mask/Key: The key used for encryption and decryption is the string "Growatt". This key is hardcoded in the script, which is generally not considered a secure practice. Ideally, the key should be stored more securely and potentially be configurable.

• Limited Security: It's important to note that XOR encryption with a static and easily guessable key like this offers very limited security. It can be easily broken by anyone who knows the algorithm and the key.

3. Code Implementation:

• decrypt(decdata) Function: This function handles the decryption process. It takes the encrypted data (decdata) as input and returns the decrypted data as a hexadecimal string.

  • Mask Conversion: The "Growatt" mask is converted into a list of hexadecimal values (hex_mask).

  • XOR Operation: The function iterates through the encrypted data (excluding the first 8 bytes, which are assumed to be an unencrypted header) and performs an XOR operation between each data byte and the corresponding byte from the hex_mask.

  • Result Formatting: The decrypted bytes are then converted into a hexadecimal string and returned.

4. Usage in the Script:

• Protocol Detection: The script first determines the communication protocol version used by the connected inverter.

• Conditional Decryption: If the protocol version indicates encrypted communication (likely 05 or 06), the decrypt() function is used to decrypt the received data.

• Data Processing: The decrypted data is then parsed to extract relevant information like energy generation, voltage, etc.

• Encryption for Responses: Similarly, before sending any commands or responses to the inverter, the script encrypts the data if the protocol requires it.

Security Considerations:

• Weak Encryption: As mentioned earlier, the XOR cipher with a hardcoded key is very weak and doesn't provide strong security.

• Key Management: The hardcoded key is a significant security vulnerability. A more secure approach would involve storing the key securely and potentially allowing users to configure it.

Overall, while the encryption/decryption mechanism in the script adds a basic layer of obfuscation, it's essential to be aware of its limitations and not rely on it for strong security.