Plotting FLAG column fails

[rpkeenan]

shadems calls with the FLAG column as -x -y -a or -c fail with the error "FLAG: column chnaged from discrete to continuous-valued. This is a bug, or a very weird MS." I've come across the same problem in the shadems version built into caracal and when building shadems from the current github master branch in a clean python 3.12 virtual envrionment.

This appears to be because line 362 of data_mappers.py: if coldata.dtype is bool or np.issubdtype(coldata.dtype, np.integer) evaluates to false, even though coldata.dtype is boolean for the FLAG column.

I am able to fix this by changing the condition in line 362 to if coldata.dtype == bool or np.issubdtype(coldata.dtype, np.integer), which causes a series of downstream errors related to FLAG having a boolean data. These errors all seem to be fixed by casting coldata to an integer type, but perhaps this may cause problems elsewhere.

As an example the command shadems [MS name] -x FREQ -y amp -a FLAG --field 0 --title 'test1' --col CORRECTED_DATA --png test.png --corr XX --ymin 0 --ymax 30 (desired output is a plot of frequency vs amplitude, shaded by flag fraction) produces the following:

2025-02-18 15:47:34 - shadems - INFO - : (1, 717) spectral windows and channels
2025-02-18 15:47:34 - shadems - INFO - : 3 fields: J0408-6545 J0420-6223 J0521+1638
2025-02-18 15:47:34 - shadems - INFO - : 10 scans: 1 2 4 5 7 8 10 11 13 14
2025-02-18 15:47:34 - shadems - INFO - : 58/58 antennas: 0:m000 1:m001 2:m002 3:m003 4:m004 5:m005 6:m006 7:m007 8:m008 9:m009 10:m010 11:m012 12:m013 13:m014 14:m015 15:m017 16:m018 17:m019 18:m020 19:m021 20:m022 21:m023 22:m024 23:m025 24:m026 25:m027 26:m028 27:m029 28:m030 29:m031 30:m032 31:m033 32:m034 33:m035 34:m036 35:m037 36:m038 37:m040 38:m041 39:m044 40:m045 41:m046 42:m047 43:m048 44:m050 45:m051 46:m052 47:m053 48:m054 49:m055 50:m056 51:m057 52:m058 53:m059 54:m060 55:m061 56:m062 57:m063
2025-02-18 15:47:34 - shadems - INFO - : 1711/1711 baselines present
2025-02-18 15:47:34 - shadems - INFO - : corrs/Stokes XX XY YX YY I Q U V
2025-02-18 15:47:34 - shadems - INFO - ------------------------------------------------------
2025-02-18 15:47:34 - shadems - INFO - : Data selected for plotting:
2025-02-18 15:47:34 - shadems - INFO - Antenna(s) : all
2025-02-18 15:47:34 - shadems - INFO - Baseline(s) : all except autocorrelations
2025-02-18 15:47:34 - shadems - INFO - Field(s) : J0408-6545
2025-02-18 15:47:34 - shadems - INFO - SPW(s) : all
2025-02-18 15:47:34 - shadems - INFO - Scan(s) : all
2025-02-18 15:47:34 - shadems - INFO - Channels : all
2025-02-18 15:47:34 - shadems - INFO - Corr/Stokes : XX
2025-02-18 15:47:34 - shadems - INFO - ------------------------------------------------------
2025-02-18 15:47:34 - shadems - INFO - loading minmax cache from 1603319074-cal_avg-minmax-cache.json
2025-02-18 15:47:34 - shadems - INFO - : plotting a flag column implies that flagged data will not be masked
2025-02-18 15:47:34 - shadems - INFO - axis: FREQ, range (None, None), discretization None
2025-02-18 15:47:34 - shadems - INFO - axis: amp(CORRECTED_DATA), corr None, range (0.0, 30.0), discretization None
2025-02-18 15:47:34 - shadems - INFO - axis: _(FLAG), corr False, range (None, None), discretization None
2025-02-18 15:47:34 - shadems - INFO - : you have asked for 1 plots employing 3 unique datums
2025-02-18 15:47:35 - shadems - INFO - : Indexing MS and building dataframes (123975 rows, chunk size is 5000)
Traceback (most recent call last):
File "/data/beegfs/astro-storage/groups/walter/keenan/venv/shadems/bin/shadems", line 8, in
sys.exit(driver())
^^^^^^^^
File "/data/beegfs/astro-storage/groups/walter/keenan/shadeMS/shade_ms/main.py", line 578, in driver
main([a for a in sys.argv[1:]])
File "/data/beegfs/astro-storage/groups/walter/keenan/shadeMS/shade_ms/main.py", line 419, in main
data_plots.get_plot_data(ms, group_cols, mytaql, ms.chan_freqs,
File "/data/beegfs/astro-storage/groups/walter/keenan/shadeMS/shade_ms/data_plots.py", line 197, in get_plot_data
value = axis.get_value(group, corr, extras, flag=flag, flag_row=flag_row, chanslice=chanslice)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/beegfs/astro-storage/groups/walter/keenan/shadeMS/shade_ms/data_mappers.py", line 383, in get_value
raise TypeError(f"{self.label}: column chnaged from discrete to continuous-valued. This is a bug, or a very weird MS.")
TypeError: FLAG___False: column chnaged from discrete to continuous-valued. This is a bug, or a very weird MS.

However, if I edit line 362 data_mappers.py to be:
if coldata.dtype == bool or np.issubdtype(coldata.dtype, np.integer):
and add the code

# Adding to treat flag as 0/1
if coldata.dtype == bool:
    coldata = coldata.astype(int)
    print(coldata.dtype)

after line 375, I get the following plot, which looks to be what I was aiming for:

[o-smirnov]

Thanks! Looks like some change in numpy tripping us up. What numpy version have you got it running with?

[rpkeenan]

numpy 2.1.3. Based on the release notes it looks like this may be either a numpy 2.0 change or a numpy 1.24 change https://numpy.org/doc/2.2/release/2.0.0-notes.html