-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathstatistics_graphs_and_visualizations.txt
74 lines (60 loc) · 9.57 KB
/
statistics_graphs_and_visualizations.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
Current Status
==============
TRAINING MODE
1 Accuracy Check Table
- when training is complete, FINAL_TEST_BATCH_SIZE test examples are classified by the last saved model (which is the best model produced during the run)
- these are displayed on the console, in 'truth', 'preds', 'delta' table format.
- since only test examples are used, and the model is the best one for the run, these outputs are experimentally meaningful
2 Confusion Matrices
- epoch-level test results are accumulated and and presented at the end of a run as a confusion matrix (using the same test batches as 1. above)
- if a job comprises a single run, the confusion matrix is not saved
- if a job comprises more than one run:
- job-level (accumulated) version of this table is also generated and presented, and two versions of it saved in logs:
- 20120909__stad_stad_mapping_file_MASTER_r2_e2_n469_b32_t20_lr0.0002_h1100_d20__job_level_classifications_matrix.csv, and
- 20120909__stad_stad_mapping_file_MASTER_r2_e2_n469_b32_t20_lr0.0002_h1100_d20__job_level_classifications_matrix_with_totals.csv
- examples are classified using the model as it exists at the end of each epoch (and not the best model produced during the run) therefore, these tables are not relevant from an experimental point of view. They are useful for hyperparameter tuning however
3 Box plots
- box plots are generated, displayed (via tensorboard) and saved. These show the average accuracy for each subtype across all runs in the job
- same data used as for Connfusion Matrices above
- for both image and rna inputs, at the end of each job, provided the job comprises more than MINIMUM_JOB_SIZE runs
TEST MODE
4 Visualizations: 'Annotated Patches':
- for images, a square patch of tiles of user specified size pushed through the last saved (best) model
- at the end of each run, display (via tensorboard) of two annotated versions ('ANNOTATED_TILES' and 'SCATTERGRAM') of the designated patch (image mode). 'SCATTERGRAM' has additional display options, notably 'PROBS_MATRIX'
5 Bar charts
- optional display (via tensorboard) and logging of two versions of bar charts which show the score of the predicted subtype.
- two scoring variants are calculated and displayed: (i) scoring by 'winner take all' tile count and (ii) scoring by 'aggregate tile-level probabilities'
- at the moment it shows the score of the classification (subtype) with the highest score, but this should be changed so that it shows the score of the correct classification (subtype) <<<<< TODO
- at the moment, bar charts are only generated for images. Need to add support for rna-seq) <<<<< TODO
BOTH TRAINING AND TEST MODE
6 Accuracy Check Table
- after each training epoch, one or more test batches (in accordance with to PCT_TEST) are pushed through and classified by the current version of the model, and
- Tensorboard: average test loss per eopoch as well as various other learning curves are presented
- Console: the last test batch of each epoch is displayed, in 'truth', 'preds', 'delta' table format
7 Confusion Matrices <<<<< TODO
- epoch-level test results are accumulated and and presented at the end of a run as a confusion matrix and two versions of it are saved in logs:
- 20120909__stad_stad_mapping_file_MASTER_r2_e2_n469_b32_t20_lr0.0002_h1100_d20__job_level_classifications_matrix.csv, and
- 20120909__stad_stad_mapping_file_MASTER_r2_e2_n469_b32_t20_lr0.0002_h1100_d20__job_level_classifications_matrix_with_totals.csv
- examples are classified using the model as it exists at the end of each epoch (and not the best model produced during the run) therefore, these tables are not relevant from an experimental point of view. They are useful for hyperparameter tuning however
8 Box plots <<<<< TODO
- box plots are generated, displayed (via tensorboard) and saved. These show the average accuracy for each subtype across all runs in the job
- same data used as for Connfusion Matrices above
- for both image and rna inputs, at the end of each job, provided the job comprises more than MINIMUM_JOB_SIZE runs
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
user variable level mode console tensorboard 'do_all.sh' 'just_test.sh' code location logged? example log file name
output? output? image rna image_rna image rna image_rna
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
output:
epoch level test batch run training YES YES YES YES test() NO
final test batch FINAL_TEST_BATCH_SIZE run training YES YES YES YES C NO
run level confusion matrices run training YES YES F NO
job level confusion matrices (I) job training YES YES G YES 20120909__stad_stad_mapping_file_MASTER_r2_e2_n469_b32_t20_lr0.0002_h1100_d20__job_level_classifications_matrix.csv
job level confusion matrices (II) G YES 20120909__stad_stad_mapping_file_MASTER_r2_e2_n469_b32_t20_lr0.0002_h1100_d20__job_level_classifications_matrix_with_totals.csv
box plots (I) BOX_PLOT, MINIMUM_JOB_SIZE job training YES YES YES G & box_plot_by_subtype() YES 20121021__stad_stad_mapping_file_MASTER_r6_e2_n4_b16_t15_lr0.001_h2000_d0__box_plot_landscape.png
box plots (II) YES 20121021__stad_stad_mapping_file_MASTER_r6_e2_n4_b16_t15_lr0.001_h2000_d0__box_plot_portrait.png
predictions (intermediate output) run test YES YES YES YES end of test() & analyse_probs() NO
patch annotation (I) ANNOTATED_TILES run test YES YES test() & analyse_probs() & plot_classes_preds() YES '2012081523 stad image VGG11 ADAM n=32 test=15.0% batch=16 lr=0.001 t'/
patch annotation (II) SCATTERGRAM run test YES YES test() & analyse_probs() & plot_classes_preds() & plot_scatter() YES '2012081523 stad image VGG11 ADAM n=32 test=15.0% batch=16 lr=0.001 t'/
patch annotation (III) PROBS_MATRIX run test YES YES test() & analyse_probs() & plot_classes_preds() '2012081523 stad image VGG11 ADAM n=32 test=15.0% batch=16 lr=0.001 t'/
bar charts (I) run test YES YES ToDo E YES 20120910__stad_r1_e5_n228_b16_t100_lr0.001_h2000_d0_bar_chart_winner_take_all.png
bar charts (II) E YES 20120910__stad_r1_e5_n228_b16_t100_lr0.001_h2000_d0_bar_chart_tile_aggregate_probabilities.png