Merge pull request #27 from rjsears/V0.6-Update

V0.6 update

README.md

@@ -1,7 +1,7 @@
  <h2 align="center">
   <a name="chia_drive_logo" href="https://github.com/rjsears/chia_plot_manager"><img src="https://github.com/rjsears/chia_plot_manager/blob/main/images/chia_plot_manager.png" alt="Chia Plot Manager"></a><br>
 
-  Chia Plot, Drive Manager & Coin Monitor (V0.5 - April 22th, 2021)
+  Chia Plot, Drive Manager & Coin Monitor (V0.6 - April 22th, 2021)
   </h2>
   <p align="center">
 Multi Server Chia Plot and Drive Management Solution
@@ -405,27 +405,29 @@ And if you choose, once a day you can get a daily update email:
 
 ```
 NAS Server: chianas01
-Daily Update Email - Generated at 10:20:53
-Current Plotting Drive (by mountpoint).................../mnt/enclosure0/front/column2/drive15
-Current Plotting Drive (by device)......................./dev/sdq1
-Drive Size...............................................10.9T
+Daily Update Email - Generated at 22:15:46
+Current Plotting Drive (by mountpoint)..................../mnt/enclosure0/front/column3/drive22
+Current Plotting Drive (by device)......................../dev/sdw1
+Drive Size................................................10.9T
 
 Environmental & Health
-Drive Serial Number......................................00000000
-Current Drive Temperature................................31°C
-Last Smart Test Health Assessment........................PASS
+Drive Serial Number.......................................00000000
+Current Drive Temperature.................................25°C
+Last Smart Test Health Assessment.........................PASS
 
 Other Information
-Total Plots on chianas01.................................1692
-Current Total Number of Plot Drives......................24
-Number of k32 Plots until full...........................925
-Max # of Plots with current # of Drives..................2640
+Total Plots on chianas01..................................2944
+Current Total Number of Plot Drives.......................44
+Number of k32 Plots until full............................1852
+Max # of Plots with current # of Drives...................4840
+Plots Being Farmed as reported by Chia....................2939
+Total Plot Space in use as reported by Chia...............291 TiB
 
 Plotting Speed
-Total Plots Last 24 Hours................................57
-Average Plots Per Hour...................................2.4
-Average Plotting Speed Last 24 Hours.....................5.757 TiB/Day
-Approx. # of Days to fill all Plot Drives................16
+Total Plots Last 24 Hours.................................79
+Average Plots Per Hour....................................3.3
+Average Plotting Speed Last 24 Hours......................7.979 TiB/Day
+Approx. # of Days to fill all Plot Drives.................23
 ```
 
 <h3>If you are using your NAS as a local plotter as well, read this.....</h3>
@@ -454,7 +456,7 @@ Staring with V0.3 (April 4th, 2021) (and updated again in V0.4) I have started t
 These options print out the help message or version information and exits.
 
 ```
-******** ChiaNAS Drive Manager - 0.5 (2021-04-22) ********
+******** ChiaNAS Drive Manager - 0.6 (2021-04-22) ********
 Running drive_manager.py with no arguments causes drive_manager to run in 'normal' mode.
 In this mode drive_manager will check the drive utilization and update which drive your
 Chia plots will be sent to when they arrive from your plotter. This is generally called
@@ -505,7 +507,7 @@ optional arguments:
   ```
 <br><br>
 <b> -dr    --drive_report</b><br>
-This option outputs the HTML versionof the Daily Drive Report email to the screen and also emails.
+This option outputs the HTML version of the Daily Drive Report email to the screen and also emails.
 This only works if configured. If this notification is set to off, this will do nothing.<br>
 
 <br>
@@ -564,16 +566,18 @@ and exits.
 ############################################################
 ################### chianas01 Plot Report ##################
 ############################################################
-Total Number of Plots on chianas01:                     1690
-Total Number of Systemwide Plots Drives:                  24
-Total Number of k32 Plots until full:                    926
-Maximum # of plots when full:                           2640
-Plots completed in the last 24 Hours:                     57
-Average Plots per Hours:                                 2.4
-Average Plotting Speed Last 24 Hours (TiB/Day):         5.76 
-Appx Number of Days to fill all current plot drives:      16  
-Current Plot Storage Drive:                        /dev/sdq1
-Temperature of Current Plot Drive:                      30°C
+Total Number of Plots on chianas01:                     2943
+Total Number of Plots Chia is Farming:                  2943
+Total Amount of Drive Space (TiB) Chia is Farming:       291
+Total Number of Systemwide Plots Drives:                  44
+Total Number of k32 Plots until full:                   1853
+Maximum # of plots when full:                           4840
+Plots completed in the last 24 Hours:                     79
+Average Plots per Hours:                                 3.3
+Average Plotting Speed Last 24 Hours (TiB/Day):         7.98 
+Appx Number of Days to fill all current plot drives:      23  
+Current Plot Storage Drive:                        /dev/sdw1
+Temperature of Current Plot Drive:                      26°C
 Latest Smart Drive Assessment of Plot Drive:            PASS
 ############################################################
 ```
@@ -691,6 +695,19 @@ strategy above, it is super easy to add more drives.
 
 ### <a name="changelog"></a>Changelog
 
+<b>V0.6 2021-04-22>/b>
+   - Minor updates to `move_local_plots.py` to utilize Disk I/O to verify that
+     we have not stalled a plot move. Attempts to auto correct and restart the
+     copy/move process if we detect a failure.
+   - Added plot checking via Chia logs to match against how many plots are 
+     currently reported as being on the system. Also included total drive space
+     in use by Chia.
+   - Updated the plot report as well as the Daily Report with above plot
+     information. It is not uncommon for there to be a little difference in the
+     numbers as a result of moves/copies in progress, etc. But if the number is
+     more than say 5 or 6 plots, you might want to check your logs for other
+     issues.
+
 <b>V0.5 2021-04-22</b>
    - Added ability to manage locally plotted plots in addition to remote plots.
      This allow you to plot on your NAS/Harvester and have those plots managed

chianas/drive_manager.py

@@ -3,7 +3,7 @@
 # -*- coding: utf-8 -*-
 
 __author__ = 'Richard J. Sears'
-VERSION = "0.5 (2021-04-22)"
+VERSION = "0.6 (2021-04-22)"
 
 ### Simple python script that helps to move my chia plots from my plotter to
 ### my nas. I wanted to use netcat as it was much faster on my 10GBe link than
@@ -14,6 +14,14 @@
 
 # Updates
 #
+#   V0.6 2021-04-22
+#   - Check Chia logs and report actual plots being farmed (per Chia) and
+#     total amount of drive space in use (also per Chia). It is not
+#     uncommon for the total number of plots on your system to be slightly
+#     different that what `drive_manager.py` reports due to plot moves, etc
+#     but if there is a large difference, you should check your logs for
+#     signs of other issues.
+#
 #   V0.5 2021-04-22
 #   - Updated to support local plot management via `move_local_plots.py`
 #
@@ -49,7 +57,6 @@
 #   - Updated necessary functions for read_config_data() change
 
 
-
 import os
 import sys
 
@@ -75,6 +82,7 @@
 import argparse
 import textwrap
 from natsort import natsorted
+import mmap
 
 # Define some colors for our help message
 red='\033[0;31m'
@@ -84,12 +92,10 @@
 blue='\033[0;34m'
 nc='\033[0m'
 
-# Remove if not using Sentry.io
-# also remove capture(e) in functions below.
 import sentry_sdk
 
 sentry_sdk.init(
-    "https://kjhsflkjshdflkjhdslkfjhaslkdjfhlakjsd.ingest.sentry.io/98793879",
+    "https://xxxxxxxxxxxxxxxxxxxxxxx.ingest.sentry.io/xxxxxxxxx",
 
     # Set traces_sample_rate to 1.0 to capture 100%
     # of transactions for performance monitoring.
@@ -103,11 +109,14 @@
 plot_size_k = 108995911228
 plot_size_g = 101.3623551
 receive_script = '/root/plot_manager/receive_plot.sh'
+chia_log_file = '/root/.chia/mainnet/log/debug.log'
 
+# Date and Time Stuff
 today = datetime.today().strftime('%A').lower()
 current_military_time = datetime.now().strftime('%H:%M:%S')
 current_timestamp = int(time.time())
 
+
 # Setup Module logging. Main logging is configured in system_logging.py
 setup_logging()
 level = read_logging_config('plot_manager_config', 'system_logging', 'log_level')
@@ -534,6 +543,8 @@ def send_new_plot_disk_email():
                                 current_drive_temperature=Device(get_device_by_mountpoint(get_plot_drive_to_use())[0][1]).temperature,
                                 smart_health_assessment=Device(get_device_by_mountpoint(get_plot_drive_to_use())[0][1]).assessment,
                                 total_serverwide_plots=get_all_available_system_space('used')[1],
+                                total_serverwide_plots_chia=check_plots()[0],
+                                total_serverwide_space_per_chia=check_plots()[1],
                                 total_number_of_drives=get_all_available_system_space('total')[0],
                                 total_k32_plots_until_full=get_all_available_system_space('free')[1],
                                 max_number_of_plots=get_all_available_system_space('total')[1],
@@ -560,6 +571,8 @@ def send_daily_update_email():
                                 total_number_of_drives=get_all_available_system_space('total')[0],
                                 total_k32_plots_until_full=get_all_available_system_space('free')[1],
                                 max_number_of_plots=get_all_available_system_space('total')[1],
+                                total_serverwide_plots_chia=check_plots()[0],
+                                total_serverwide_space_per_chia=check_plots()[1],
                                 total_plots_last_day=read_config_data('plot_manager_config', 'plotting_information', 'current_total_plots_daily', False),
                                 days_to_fill_drives=(int(get_all_available_system_space('free')[1] / int(read_config_data('plot_manager_config', 'plotting_information', 'current_total_plots_daily', False)))),
                                 average_plots_per_hour=round((int(read_config_data('plot_manager_config', 'plotting_information', 'current_total_plots_daily', False)))/24,1),
@@ -573,14 +586,16 @@ def space_report():
     print(f'{blue}################### {green}{nas_server} Plot Report{blue} ##################{nc}' )
     print(f'{blue}############################################################{nc}')
     print (f'Total Number of Plots on {green}{nas_server}{nc}:                     {yellow}{get_all_available_system_space("used")[1]}{nc}')
+    print (f'Total Number of Plots {green}Chia{nc} is Farming:                  {yellow}{check_plots()[0]}{nc}')
+    print (f'Total Amount of Drive Space (TiB) {green}Chia{nc} is Farming:       {yellow}{check_plots()[1]}{nc}')
     print (f'Total Number of Systemwide Plots Drives:                  {yellow}{get_all_available_system_space("total")[0]}{nc}')
     print (f'Total Number of k32 Plots until full:                   {yellow}{get_all_available_system_space("free")[1]}{nc}')
     print (f'Maximum # of plots when full:                           {yellow}{get_all_available_system_space("total")[1]}{nc}')
     print (f"Plots completed in the last 24 Hours:                     {yellow}{int(read_config_data('plot_manager_config', 'plotting_information', 'current_total_plots_daily', False))}{nc}")
     print (f"Average Plots per Hours:                                 {yellow}{round((int(read_config_data('plot_manager_config', 'plotting_information', 'current_total_plots_daily', False)))/24,1)}{nc}")
     print (f"Average Plotting Speed Last 24 Hours (TiB/Day):         {yellow}{round((int(read_config_data('plot_manager_config', 'plotting_information', 'current_total_plots_daily', False)) * int(plot_size_g)/1000),2)}{nc} ")
     print(f"Appx Number of Days to fill all current plot drives:     {yellow} {int(get_all_available_system_space('free')[1] / int(read_config_data('plot_manager_config', 'plotting_information', 'current_total_plots_daily', False)))} {nc} ")
-    print (f"Current Plot Storage Drive:                       {yellow}{(get_device_by_mountpoint(read_config_data('plot_manager_config', 'plotting_drives', 'current_plotting_drive', False))[0][1])}{nc}")
+    print (f"Current Plot Storage Drive:                        {yellow}{(get_device_by_mountpoint(read_config_data('plot_manager_config', 'plotting_drives', 'current_plotting_drive', False))[0][1])}{nc}")
     print (f"Temperature of Current Plot Drive:                      {yellow}{Device((get_device_by_mountpoint(read_config_data('plot_manager_config', 'plotting_drives', 'current_plotting_drive', False))[0][1])).temperature}°C{nc}")
     print (f"Latest Smart Drive Assessment of Plot Drive:            {yellow}{Device((get_device_by_mountpoint(read_config_data('plot_manager_config', 'plotting_drives', 'current_plotting_drive', False))[0][1])).assessment}{nc}")
     print(f'{blue}############################################################{nc}')
@@ -703,6 +718,19 @@ def send_new_plot_notification():
             notify('New Plot Received', 'New Plot Received')
         os.remove('new_plot_received')
 
+
+def check_plots():
+    with open(chia_log_file, 'rb', 0) as f:
+        m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
+        i = m.rfind(b'Loaded')
+        m.seek(i)
+        line = m.readline()
+        newline = line.decode("utf-8")
+        x = newline.split()
+        plots = x[4]
+        TiB = float(x[8])
+        return plots, f'{TiB:.0f}'
+
 def main():
     parser = init_argparser()
     args = parser.parse_args()

chianas/templates/daily_update.html

@@ -10,22 +10,24 @@ <h3><b>NAS Server: {{nas_server}}</b></h3>
 
 <br>
 <b>Environmental & Health</b><br>
-Drive Serial Number..........................................{{drive_serial_number}}<br>
-Current Drive Temperature................................{{current_drive_temperature}}°C<br>
-Last Smart Test Health Assessment..................{{smart_health_assessment}}<br>
+Drive Serial Number................................................{{drive_serial_number}}<br>
+Current Drive Temperature......................................{{current_drive_temperature}}°C<br>
+Last Smart Test Health Assessment........................{{smart_health_assessment}}<br>
 
 <br>
 <b>Other Information</b><br>
-Total Plots on <b>{{nas_server}}</b>..................................{{total_serverwide_plots}}<br>
-Current Total Number of Plot Drives..................{{total_number_of_drives}}<br>
-Number of k32 Plots until full.............................{{total_k32_plots_until_full}}<br>
-Max # of Plots with current # of Drives..............{{max_number_of_plots}}<br>
+Total Plots on <b>{{nas_server}}</b>.........................................{{total_serverwide_plots}}<br>
+Current Total Number of Plot Drives........................{{total_number_of_drives}}<br>
+Number of k32 Plots until full...................................{{total_k32_plots_until_full}}<br>
+Max # of Plots with current # of Drives....................{{max_number_of_plots}}<br>
+Plots Being Farmed as reported by Chia.................{{total_serverwide_plots_chia}}<br>
+Total Plot Space in use as reported by Chia............{{total_serverwide_space_per_chia}} <b>TiB</b><br>
 
 <br>
 <b>Plotting Speed</b><br>
-Total Plots Last 24 Hours....................................{{total_plots_last_day}}<br>
-Average Plots Per Hour.......................................{{average_plots_per_hour}}<br>
-Average Plotting Speed Last 24 Hours...............{{average_plotting_speed}} <b>TiB/Day</b><br>
-Approx. # of Days to fill all Plot Drives................{{days_to_fill_drives}}<br>
+Total Plots Last 24 Hours.........................................{{total_plots_last_day}}<br>
+Average Plots Per Hour...........................................{{average_plots_per_hour}}<br>
+Average Plotting Speed Last 24 Hours....................{{average_plotting_speed}} <b>TiB/Day</b><br>
+Approx. # of Days to fill all Plot Drives.....................{{days_to_fill_drives}}<br>
 </body>
 </html>

chianas/templates/new_plotting_drive.html

@@ -25,5 +25,7 @@ <h3><b>Server: {{nas_server}}</b></h3>
 Number of k32 Plots until full.............................{{total_k32_plots_until_full}}<br>
 Max # of Plots with current # of Drives..............{{max_number_of_plots}}<br>
 Approx. # of Days to fill all Plot Drives..............{{days_to_fill_drives}}<br>
+Plots Being Farmed as reported by Chia...............{{total_serverwide_plots_chia}}<br>
+Total Plot Space in use as reported by Chia...........{{total_serverwide_space_per_chia}}<b>TiB</b><br>
 </body>
 </html>