Merge pull request #26 from rjsears/move_local_plot_fixes

Move local plot fixes

README.md

@@ -134,6 +134,8 @@ I am running on Python 3.8.5 and pretty much everything else came installed with
   <li><a href="https://pypi.org/project/configparser/">ConfigParser (5.0.2)</a> - Used for reading and updating config files</li>
   <li><a href="https://github.com/truenas/py-SMART">py-SMART (0.3)</a> - Used for reading drive information</li>
   <li><a href="https://pypi.org/project/natsort/">Natsort (7.1.1)</a> - Used for natural sorting of drive numbers</li>
+ <li><a href="http://dag.wieers.com/home-made/dstat/">DStat (0.8.0)</a> - Used to monitor Disk I/O Stats</li>
+
  </ul>
  <hr>
 
@@ -426,6 +428,14 @@ Average Plotting Speed Last 24 Hours.....................5.757 TiB/Day
 Approx. # of Days to fill all Plot Drives................16
 ```
 
+<h3>If you are using your NAS as a local plotter as well, read this.....</h3>
+I also use my NAS/Harvester as a local plotter as well, and I need to be able to manage the plots that are created just as I do the remote plots. I use `Plotman` to manage the creation of my plots but it is not really designed with a setup like mine. You can point it to a single rsync location but you have to manage that plot on your own once it gets there. The `-d` drive you point `Plotman` at does not get monitored by `Plotman` (or chia for that matter) and so I learned the hard way that those drives can fill up and cause all of your plotting to fail. So I created `move_local_plots.py`.<br><br>
+This script is very simple, it looks in the location you tell it you are storing completed local plots and moves them to the currently selected plotting drive that is configured by `drive_manager.py`. It then verifies the plot sizes and deletes the plot from the monitored drive. It an attempt to verify that the copy is taking place and not failed, I utilize `Dstat` and a simple shell script that looks at the I/O on the drive you are moving the plot off of. If there is no I/O on the drive, it is assumed that there is no copy process. If the script detects that it thinks there is a copy going on yet there is no drive I/O, it attempts to reset itself and restart the copy, hopefully completing correctly this time around.<br><br>
+Make sure you install `Dstat` before trying to use this script otherwise it will fail. See the beginning of the script for settings.<br><br>
+One word of advice - if you are using one of your drives that `drive_manager.py` would normally use to place plots on as a temp local `-d` drive, you should offline the drive using the `./drive_manager.py --off drivexx` command, otherwise `drive_manager.py` could start placing plots on that drive causing issues.<br><br>
+
+
+
 ### <a name="cli"></a>Command Line Options
 
 Staring with V0.3 (April 4th, 2021) (and updated again in V0.4) I have started to add in command line options to make getting plot and other information easier and to generate reports on the fly. Currently the command line options that are availare are:
@@ -570,11 +580,11 @@ Latest Smart Drive Assessment of Plot Drive:            PASS
 
 <br><br>
 <b>-ud --update_daily</b><br>
-This option is really designed to be called from your crontab riught before you run your daily email:
+This option is really designed to be called from your crontab right before you run your daily email:
 
 ```
-01 00 * * * /usr/bin/python3 /root/plot_manager/drive_manager.py -ud >/dev/null 2>&1
-02 00 * * * /usr/bin/python3 /root/plot_manager/drive_manager.py -dr >/dev/null 2>&1
+01 00 * * * cd /root/plot_manager && /usr/bin/python3 /root/plot_manager/drive_manager.py -ud >/dev/null 2>&1
+02 00 * * * cd /root/plot_manager && /usr/bin/python3 /root/plot_manager/drive_manager.py -dr >/dev/null 2>&1
 ```
 
 It is designed to calculate the information necessary for the daily plot report. You can run it anytime

chianas/check_drive_activity.sh

@@ -0,0 +1,2 @@
+#! /bin/bash
+dstat -r -D $1 1 3 > drive_monitor.iostat

chianas/move_local_plots.py

@@ -20,12 +20,20 @@
 import sys
 sys.path.append('/root/plot_manager')
 import logging
+import psutil
 import configparser
 from system_logging import setup_logging
 from system_logging import read_logging_config
 import pathlib
 import shutil
 from timeit import default_timer as timer
+from drive_manager import get_device_by_mountpoint
+import subprocess
+
+
+# Set the below file paths as necessary. The `drive_activity_log` needs to match
+# the file you have set in the `check_drive_activity.sh` shell script in order to
+# work. Make sure you have Dstat installed otherwise this script will not work.
 
 
 # Are we testing?
@@ -34,10 +42,16 @@
     plot_dir = '/root/plot_manager/test_plots/'
     plot_size = 10000000
     status_file = '/root/plot_manager/local_transfer_job_running_testing'
+    drive_activity_test = '/root/plot_manager/check_drive_activity.sh'
+    drive_activity_log = '/root/plot_manager/drive_monitor.iostat'
 else:
-    plot_dir = '/mnt/enclosure1/front/column1/drive43/'
+    plot_dir = '/mnt/enclosure1/front/column1/drive43'
     plot_size = 108644374730  # Based on K32 plot size
     status_file = '/root/plot_manager/local_transfer_job_running'
+    drive_activity_test = '/root/plot_manager/check_drive_activity.sh'
+    drive_activity_log = '/root/plot_manager/drive_monitor.iostat'
+
+
 
 
 # Setup Module logging. Main logging is configured in system_logging.py
@@ -83,20 +97,24 @@ def process_plot():
         plot_to_process = get_list_of_plots()
         if plot_to_process and not testing:
             process_control('set_status', 'start')
-            plot_path = plot_dir + plot_to_process
+            plot_path = plot_dir + '/' + plot_to_process
             log.info(f'Processing Plot: {plot_path}')
             current_plotting_drive = read_config_data('plot_manager_config', 'plotting_drives', 'current_plotting_drive', False)
             log.debug(f'Current Plotting Drive is: {current_plotting_drive}')
             log.debug(f'Starting Copy of {plot_path}')
             start_time = timer()
-            shutil.copy2(plot_path, current_plotting_drive)
+            try:
+                shutil.copy2(plot_path, current_plotting_drive)
+            except:
+                log.debug(f'ERROR: There was a problem copying: {plot_dir}!')
+                exit()
             end_time = timer()
             if verify_plot_move(current_plotting_drive, plot_path, plot_to_process):
                 log.info('Plot Sizes Match, we have a good plot move!')
                 log.info(f'Total Elapsed Time: {end_time - start_time:.2f} seconds or {(end_time - start_time)/60:.2f} Minutes')
             else:
-                log.debug('FAILURE - Plot sizes DO NOT Match - Exiting') # ToDo Do some notification here and then...?
-                process_control('set_status', 'stop') #Set to stop so it will attempt to run again in the event we want to retry....
+                log.debug('FAILURE - Plot sizes DO NOT Match')
+                process_control('set_status', 'stop')  #Set to stop so it will attempt to run again in the event we want to retry....
                 main() # Try Again
             process_control('set_status', 'stop')
             os.remove(plot_path)
@@ -139,15 +157,34 @@ def process_control(command, action):
                 log.debug(f'Status File: [{status_file}] does not exist!')
                 return
     elif command == 'check_status':
-        if os.path.isfile(status_file):
-            log.debug(f'Checkfile Exists, We are currently Copying a Plot, Exiting')
+        if os.path.isfile(status_file) and check_drive_activity():
+            log.debug(f'Checkfile Exists and Disk I/O is present, We are currently Copying a Plot, Exiting')
             return True
+        elif os.path.isfile(status_file) and not check_drive_activity():
+            log.debug('WARNING! - Checkfile exists but there is no Disk I/O! Forcing Reset')
+            os.remove(status_file)
+            return False
         else:
             log.debug(f'Checkfile Does Not Exist!')
             return False
     else:
         return
 
+def check_drive_activity():
+    try:
+        subprocess.call([drive_activity_test, get_device_by_mountpoint(plot_dir)[0][1].split('/')[2]])
+    except subprocess.CalledProcessError as e:
+        log.warning(e.output)
+    with open(drive_activity_log, 'rb') as f:
+        f.seek(-2, os.SEEK_END)
+        while f.read(1) != b'\n':
+            f.seek(-2, os.SEEK_CUR)
+        last_line = f.readline().decode()
+    if float((str.split(last_line)[0])) > 10:
+        return True
+    else:
+        return False
+
 
 def main():
     process_plot()