Subtraction of summed model columns prior to flagging (feature request)

[IanHeywood]

I think a useful feature would be to be able to pass column arithmetic to the -smc switch, similar to how CubiCal's model list and shadems operate.

For example, direction-dependent calibration scenarios where the model is partitioned into multiple columns require a lot of messing around in order to perform a flagging operation on the residual visibilities. Something like -smc MODEL_DATA+DIR1_DATA would get around this nicely.

Thanks.

[sjperkins]

@IanHeywood It sounds like a good addition.

Could I interest you in putting together a PR based on this?

[IanHeywood]

I don't know when I would get the uninterrupted several-hour window that would be necessary for me to even begin to figure this out. The spirit is willing but the calendar is bleak.

[sjperkins]

The spirit is willing but the calendar is bleak.

I think we're in similar straights. Let's see if my schedule clears up.

[IanHeywood]

It's very much an edge case use case so please don't go to any trouble with it.

[o-smirnov]

+1 from me.

[sjperkins]

So at the moment we have --data-column and --subtract-model-column arguments.

I would propose that we remove the --substract-model-column argument and support expressions within the data column argument.

There would be two components to this:

1. Extending the parsing of the --data-column argument (as done in shadems here). Therefore, the data column argument will have a fair amount flexibility. Lets say that it can just be one column or a binary expression containing two columns.
   i.e. -dc DATA or -dc DATA-MODEL_DATA or -dc DATA+DIR1
2. Applying the expression on the columns here

/cc @bennahugo for comments on the proposed change

[bennahugo]

One could, but I would not deprecate the existing smc option to keep backwards compatibility in pipelines. Multi-model column processing is not quite a standard way of most data reductions unless you do dE calibration with a dft

[sjperkins]

Ok, how about marking SMC for deprecation with a warning?

[bennahugo]

sure

…

On Mon, Oct 26, 2020 at 3:19 PM Simon Perkins ***@***.***> wrote: Ok, how about marking SMC for deprecation with a warning? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6RQ3ZBEJOYJXCCTUITSMVZPLANCNFSM4QXXOLWQ> .

-- -- Benjamin Hugo PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[bennahugo]

Actually what is even better is DATA / sum(MODELS). RFI typically stands out like a sore thumb

[sjperkins]

Actually what is even better is DATA / sum(MODELS). RFI typically stands out like a sore thumb

Nice to have, but I think this would need AST parsing, rather than a simple regex for a binary operation.

To help narrow this down, what operators should be supported? A nice simple calculator subset is: +, -, /, *, (, ), for example. Could we do without explicit functions like sum? i.e. could DATA / (DIR1_DATA + DIR2_DATA + MODEL_DATA) work instead?