Workers
Start a Discussion or join our Discord for support and to help out.
While the default installation of Machinaris is standalone, management of multiple Machinaris workers on your local-area network (LAN) is also supported. Machinaris offers a single WebUI to control all the workers (plotters, harvesters, farmers) on your LAN. Here's an example network layout:
The Machinaris Controller also provides the Chia blockchain full-node. To farm your Chia plots to blockchain forks, you can launch those separate containers as full-nodes/harvesters. They will register with the Machinaris controller for remote management and monitoring. Generally, all blockchain fullnodes (one per container) will run on the same computer, however you can run some on a different machine.
NOTE: This is NOT intended for management of multiple instances at different geographic sites over a wide area network (WAN). Please investigate usage of VPN tunneling and/or secured web-proxy in that case. DO NOT simply port-forward ports 8447, 8926 or 8927 at your router. Ports 8447 (Chia farmer), 8926 (Machinaris WebUI) and 8927 (Machinaris API) should NOT be available to the public Internet!
First, you'll need a Machinaris controller/full-node running. Then you can launch workers such as 'harvester', 'plotter', 'harvester+plotter'. When you later add blockchain Forks to your farm, you will first launch that fork's full-node.
As with standalone, one full-node instance of Machinaris should be running as controller for your network. This controller will be the Chia full-node as well. Use the Launch Wizard to create a Docker Compose command for your system.
By default, a Machinaris full-node includes the ability to plot as an out-of-the-box all-in-one solution. However, if you only plot on other systems, you can specifically disable the Madmax and Bladebit plotters by passing these environment variables:
- bladebit_skip_build=true
- madmax_skip_build=true
There may be multiple plotters running on the LAN. Visit the Workers page of your Machinaris controller (above), and click the New Worker button to generate a Docker Compose for your system.
There may be multiple harvesters running on the LAN. Visit the Workers page of your Machinaris controller (above), and click the New Worker button to generate a Docker Compose for your system.
It's also possible to combine both harvesting and plotting duties in a single Machinaris container launch. Visit the Workers page of your Machinaris controller (above), and click the New Worker button to generate a Docker Compose for your system. You can check both the 'Harvester' and 'Plotter' mode buttons at the same time.
When running, browsing to Machinaris on your Controller instance will show status for all connected workers. The memory usage chart at the top shows ALL memory usage by all processes including the operating system on the host machine. This chart DOES NOT show only memory usage by Machinaris. To see, per-container memory usage by Machinaris, check the 'Memory' column in the table of workers below, and click to see a chart.
The communication between controller and worker is bi-directional via REST API on port 8927. Please ensure you have opened the required ports (8926 & 8927) in the firewalls on each computer running Machinaris. In example above, the worker (gimli) is sending its status to the controller (aragorn), but due to firewall on the worker, the controller shows it as Connection Refused.
With multiple Machinaris plotters on your LAN, you can configure and control them from the Plotting page on your Controller instance:
While the Workers design was primarily for a home LAN with a shared local DNS resolving all local machine hostnames, I discovered that most don't have a functional DNS on their LAN. As such, please ensure you use IP addresses for environment variables (farmer_address, controller_host and worker_address) when you create your worker's docker-compose.yml on the Workers page of the WebUI.
All controllers and workers are running a REST-API server on port tcp/8927. To allow multiple fork containers on a single worker, this default port in incremented for each container. Flax is 8928, N-Chain is 8929, etc. All of this is generated for you on the Workers page when you click the New Worker button.
In general, workers submit their own status (farm/plotting/plots/etc) to their controller's API endpoint. As well, the controller can send commands to a worker at their API endpoint, to start/stop plotting, view logs, etc. Currently, there is no authentication of these endpoints as it is intended for ease-of-use on a home LAN. As such, you mustn't expose these ports outside your LAN or VLAN, particularly not to the public Internet.
Yes, normally all fullnode containers run on a single machine, however you may run out of memory on that primary system with too many blockchains. In that case, leave the machinaris container (Chia) running on your primary system, along with original blockchains, and deploy new blockchain fullnodes (Flax, BPX, HDDCoin, etc) to a secondary computer. Basically, you are taking a docker-compose.yml generated at https://www.machinaris.app/ and splitting it up.
On the secondary fullnode system, be sure to manually create the ~/.machinaris folder and place a copy of your mnemonic.txt from your primary fullnode system. Then ensure the controller_host environment variable in the docker-compose.yml of your secondary system must be the IP address of your primary fullnode system (where Machinaris controller is running). This will allow the blockchain fullnodes on the secondary system to contact the Machinaris controller (primary system).
To "move" a blockchain fullnode for FORK:
- On primary system:
docker-compose stop machinaris-FORK && docker-compose rm -f machinaris-FORK - On primary system: cut that blockchain's section from
docker-compose.yml - Copy
~/.machinaris-FORKfolder from primary system to secondary system. - On secondary system: add that blockcchain's sectinon to
docker-compose.yml. Change all IPs to secondary machine exceptcontroller_hostwhich must still be IP of primary system. - On secondary system: ensure
~/.machinaris/mnemonic.txtexists. - On secondary system: ensure plots from primary system are remote mounted to secondary system via NFS/SMB. Edit Volume lines in
docker-compose.ymlto map them correctly. - On secondary system:
docker-compose up -d machinaris-FORK && docker logs -f machinaris-FORK
Here are some common tips for troubleshooting connection issues between the controller and workers.
If you have launched a Worker (harvester and/or plotter), and after 10-15 minutes still do not see it listed on the Workers page of the Machinaris WebUI, please check the following:
As outlined above, please ensure that each worker has the correct environment variables provided to it on container, which allow it to contact the Machinaris controller:
-
controller_host: The IP address (not hostname) of the Machinaris controller on your LAN. -
farmer_address: If the worker is a harvester, it must have the address of the full-node which will handle farming. Normally, this is the same IP address ascontroller_host.
Check docker logs -f machinaris (or docker logs -f machinaris-[FORK]) on the worker system to ensure a clean startup, without launch errors.
Similar to the general log troubleshooting guidance, please check that the ~/.machinaris[-FORK]/machinaris/logs/apisrv.log file does not have errors about failure to reach the controller. For example, you can test if a firewall is blocking the worker from registering with the controller by:
docker exec -it machinaris bash
curl http://${controller_host}:${controller_api_port}/ping/
Example successful response:
root@aragorn:/chives-blockchain# curl http://${controller_host}:${controller_api_port}/ping/
Pong!
root@aragorn:/chives-blockchain#
If you don't get a Pong! back, then the worker is unable to reach the controller, so it will not be listed on the Workers page.
If a Harvester worker is listed on the Workers page, but does not appear in the "Blockchain Challenges" chart on the Summary page, showing timely responses, then please ensure that certificates were automatically pulled on startup (first launch only). Please delete any old ~/.machinaris* folders on the harvester, and re-launch Machinaris worker on the harvester. Follow the log output with docker logs -f machinaris to see the certificates come over in a zip bundle, be inflated and imported to Chia.
As well, ensure you have the correct Timezone set for each worker. If you worker thinks it is hours behind the rest of your farm, then its blockchain challenges will not appear on the Summary page chart. To check this, browse to Workers page of Machinaris WebUI. Click into the particular worker by name and check Time on Worker field. Is it in the same timezone as rest of your farm? If not, you have overridden the default TZ environment variable automatically provided when using the New Worker wizard.
It is also possible to reset a Harvester, to ensure that certificates are configured correctly on first launch:
docker-compose stop
docker-compose rm -f
rm -rf ~/.machinaris-FORK
docker-compose pull
docker-compose up -d
docker logs -f machinaris-FORK
In the Docker logs of the FORK container, you should see the pull of certificates from the fullnode as zip, then unzip before the harvester is started. Once running, you can grep the harvester log for a successful connection to the fullnode.
docker exec -it machinaris-FORK
cd /root/.FORK/
grep -i connect mainnet/log/debug.log
Then verify that your Chia fullnode system sees the harvesters by running chia farm summary on the fullnode. You should see your harvesters listed by IP address, along with the count of plots they are harvesting.