Crashing with Mask and data not compatible

[landmanbester]

Not sure this is an issue with shadems per se but I have gotten myself into a situation where I get the following error

2021-01-20 17:08:03 - shadems - INFO - using colourmap colorcet.bkr
2021-01-20 17:08:03 - shadems - INFO - using colourmap cmasher.pride
2021-01-20 17:08:03 - shadems - INFO - using colourmap colorcet.glasbey_dark
2021-01-20 17:08:03 - shadems - INFO - /home/bester/venvs/shadems/bin/shadems --xaxis FREQ --yaxis UNCORRECTED_DATA:amp:XX --dir udatacol 1557347448_sdp_l0-cal.ms/
2021-01-20 17:08:03 - shadems - INFO - ------------------------------------------------------
2021-01-20 17:08:03 - shadems - INFO - : MS 1557347448_sdp_l0-cal.ms contains 708354 rows
2021-01-20 17:08:03 - shadems - INFO - :   (1, 4096) spectral windows and channels
2021-01-20 17:08:03 - shadems - INFO - :   1 fields: J1939-6342
2021-01-20 17:08:03 - shadems - INFO - :   8 scans: 1 3 5 7 9 11 13 15
2021-01-20 17:08:04 - shadems - INFO - :   58/58 antennas: 0:m000 1:m001 2:m002 3:m003 4:m004 5:m005 6:m006 7:m007 8:m008 9:m009 10:m010 11:m011 12:m012 13:m013 14:m015 15:m016 16:m018 17:m019 18:m020 19:m021 20:m022 21:m023 22:m024 23:m025 24:m026 25:m027 26:m028 27:m029 28:m030 29:m031 30:m033 31:m034 32:m035 33:m036 34:m037 35:m038 36:m039 37:m040 38:m041 39:m042 40:m043 41:m044 42:m047 43:m048 44:m049 45:m050 46:m051 47:m052 48:m053 49:m054 50:m055 51:m056 52:m057 53:m058 54:m060 55:m061 56:m062 57:m063
2021-01-20 17:08:04 - shadems - INFO - :   1711/1711 baselines present
2021-01-20 17:08:04 - shadems - INFO - :   corrs/Stokes XX XY YX YY I Q U V
2021-01-20 17:08:04 - shadems - INFO - ------------------------------------------------------
2021-01-20 17:08:04 - shadems - INFO - : Data selected for plotting:
2021-01-20 17:08:04 - shadems - INFO - Antenna(s)       : all
2021-01-20 17:08:04 - shadems - INFO - Baseline(s)      : all except autocorrelations
2021-01-20 17:08:04 - shadems - INFO - Field(s)         : all
2021-01-20 17:08:04 - shadems - INFO - SPW(s)           : all
2021-01-20 17:08:04 - shadems - INFO - Scan(s)          : all
2021-01-20 17:08:04 - shadems - INFO - Channels         : all
2021-01-20 17:08:04 - shadems - INFO - Corr/Stokes      : XX XY YX YY
2021-01-20 17:08:04 - shadems - INFO - ------------------------------------------------------
2021-01-20 17:08:04 - shadems - INFO - loading minmax cache from udatacol/1557347448_sdp_l0-cal-minmax-cache.json
2021-01-20 17:08:04 - shadems - INFO - axis: FREQ, range (None, None), discretization None
2021-01-20 17:08:04 - shadems - INFO - axis: amp(UNCORRECTED_DATA), corr 0, range (None, None), discretization None
2021-01-20 17:08:04 - shadems - INFO -                  : you have asked for 1 plots employing 2 unique datums
2021-01-20 17:08:04 - shadems - INFO - : Indexing MS and building dataframes (684342 rows, chunk size is 5000)
Traceback (most recent call last):
  File "/home/bester/venvs/shadems/bin/shadems", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/bester/shadeMS/bin/shadems", line 8, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/home/bester/shadeMS/shade_ms/main.py", line 418, in main
    row_chunk_size=options.row_chunk_size)
  File "/home/bester/shadeMS/shade_ms/data_plots.py", line 191, in get_plot_data
    value = axis.get_value(group, corr, extras, flag=flag, flag_row=flag_row, chanslice=chanslice)
  File "/home/bester/shadeMS/shade_ms/data_mappers.py", line 383, in get_value
    return dama.masked_array(coldata, da.logical_or(flag, bad_data))
  File "/home/bester/venvs/shadems/lib/python3.6/site-packages/dask/array/ma.py", line 229, in masked_array
    "%s." % (repr(data.shape), repr(mask.shape))
numpy.ma.core.MaskError: Mask and data not compatible: data shape is (684342, 4096), and mask shape is (684342, 4096, 4096).

UNCORRECTED_DATA is a column I have written to the measurement set myself using

https://github.com/ratt-ru/pfb-clean/blob/add_gains/scripts/kgb2col.py

Strangely, it seems to have the same shape as DATA and CORRECTED_DATA both of which plot just fine. Any idea what coul dbe going wrong here?

[IanHeywood]

I think this is the same as #64

[landmanbester]

Ah, thanks @IanHeywood. This is a duplicate