Enable TAP for ESO module

[juanmcloaiza]

Addresses #3138

Changes:

• query_main
• list_instruments
• query_instruments
• list_surveys
• query_surveys
• query_apex_quicklooks
• deprecate open_form and cache arguments
• beta test - IN PROGRESS

[pep8speaks]

Hello @juanmcloaiza! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2025-01-17 14:13:03 UTC

[codecov]

Codecov Report

Attention: Patch coverage is 76.57658% with 52 lines in your changes missing coverage. Please review.

Project coverage is 69.88%. Comparing base (af2d6f4) to head (c63bc21).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
astroquery/eso/core.py	71.58%	52 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3141      +/-   ##
==========================================
+ Coverage   69.37%   69.88%   +0.50%     
==========================================
  Files         232      233       +1     
  Lines       19686    19580     -106     
==========================================
+ Hits        13657    13683      +26     
+ Misses       6029     5897     -132     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bsipocz]

One comment for now regarding the deprecation, otherwise is looking good so far (I didn't dive into the details, but had a quick look

[juanmcloaiza]

@bsipocz, hi, I'd like to ask for your feedback regarding deprecated functionality.

This PR's scope is to switch the back-end of the ESO module from WDB to TAP. In doing so, many of the functions will be renamed and their signatures changed (i.e. completely new functions are being written, while others are being removed).

If I understand correctly, astroquery requires that deprecated features are handled via @deprecated decorators. However, in this particular case, the changes are rather large --to some extent a @deprecated decorator would need to be added to the whole ESO module (take this last phrase with a grain of salt, I'm exaggerating).

The question I'd like to ask you as an astroquery maintainer is the following: Would it be admissible for this time to avoid adding deprecated decorators to each of the functions and arguments that we are discontinuing? My impression is that the @deprecated benefits will be small in comparison to the illegibility of the code they'll produce. I would prefer instead to make it very explicit in the documentation that whatever functionality available in past versions may not be available from this version on.

Please let us know what you think (cc @szampier). Thanks in advance for your feedback ;)

[bsipocz]

Given this is a full refactoring and switching out what backend to use on the server side, it should be OK to do it without decorating with deprecations and remove methods that don't make sense any more.
However, the ones that used to be around should issue deprecation for keywords that got removed, etc. Similarly, methods that got renamed should maybe stay around for a release cycle as an alias of the new methods.

[bsipocz]

Also let me know when I should come in and do a first round of review. I already noticed a few things that are odd and would preferably done differently, e.g. the custom caching functions (e.g. it would be nice to either reuse what we already have, or generalize these and put it into a more central location). Also, I see some workaround for the tests that preferably should be resolved, e.g. don't filter warnings but if they are legit, then check if they are raised, also, empty tests should not be around for the sake of fooling codecov (you can just ignore the report for now, it doesn't report on the remote tests)

[juanmcloaiza]

Hi @bsipocz , thanks for the feedback. I reply each of your points here:

• deprecation warnings - all clear, I'll do it as you suggest.
• fooling codecov - You are completely right. I wanted to understand how codecov worked and forgot to remove those tests. The nice thing I learned is that codecov does actually tell you, line by line, whether a line was executed by your test suite, so no way to fool it by writing empty tests - https://docs.codecov.com/docs/about-code-coverage. Really nice tool, it did help me to write new real and legit offline tests and now the report looks much better.
• filtering warnings - I'm afraid that what you suggest was my original intention, but for some function calls inside loops, the warning would be raised, while for some others it wouldn't. To do things properly, I have created specific tests to check that the warning is raised when it should, and not raised when it shouldn't.
• custom caching functions - I didn't find a way to reuse the caching functions available, but the ones I wrote are largely based on them. Since we want to keep this PR minimal, I will remove cache functionality for the time being. However, these functions to cache TAP queries could be useful. Should I open a new PR aiming to add them to a more central location?

I will ping you again when we're ready with our internal feedback rounds, so you can go through it more thoroughly. Thanks again for your preliminary feedback! 👍

[juanmcloaiza]

@bsipocz , hi, one quick question. Preparing for the future, we are thinking on developing ESO submodules, so that we can query ESO data in the following way:

import astroquery.eso.observations as observations # Current Pull Request
import astroquery.eso.catalogues as catalogues # To be developed in the future

raw_datasets = observations.list_raw_instruments()
# >>> ['MUSE', 'VVV', 'SPHERE', ... ]

# Query raw data:
raw_data = observations.Raw() # Raw data from table dbo.raw
my_astropy_table = raw_data.query(instrument="MUSE")

# Query raw data from an IST:
muse_data = observations.Raw("MUSE") # Raw data from table ist.muse
my_astropy_table = muse_data.query(...)

# Query Phase 3 data:
processed_data = observations.Processed() # Processed data from table ivoa.ObsCore
my_astropy_table = processed_data.query(collections=["MUSE"])

Would this be OK for astroquery standards?

[keflavich]

I'd recommend that some of what you're proposing should be keywords. e.g. observations.list_instruments(processed_status='raw') instead of list_raw_instruments.

If you're going to use observations.Raw("MUSE"), then keep the symmetry and use observations.Processed("MUSE") too.

Other archives generally split their queries by Observation first, Processed State next (e.g., MAST). Is there a structural reason you need to split by processed state first, or could you also support queries that return everything associated with a given observation, then filter from there? If you can support both ways, I think that's better - the average user will not want raw data, so your approach looks possibly better.

[bsipocz]

I would really like to keep the facility name in the name of classes, etc, so while submodules are fine I think it would be better to keep calling them something ESO.

import astroquery.eso.observations as EsoObservations # Current Pull Request
import astroquery.eso.catalogues as EsoCatalogues # To be developed in the future

(I know that mast is an outlier in this sense, and then gemini copied their approach, but I haven't given up on getting the renamed done as outlined in #1748)

[keflavich]

Strongly agree @bsipocz . What you've suggested is a documentation (suggested usage) change, right? i.e., import astroquery.eso.Observations as EsoObservations is how we'd recommend users import the module, but it wouldn't require any code changes?

[bsipocz]

i.e., import astroquery.eso.Observations as EsoObservations is how we'd recommend users import the module, but it wouldn't require any code changes?

No, I would strongly recommend to rename the class itself.

[juanmcloaiza]

Hi @bsipocz and @keflavich , thanks for the attention given to my question. My takeaway is as follows:

• Keep the facility name in the name of classes, etc; submodules are fine --> Cool, thanks! 👍

To quickly answer your questions:

• Is there a structural reason you need to split by processed state first? --> Yes
• Could you also support queries that return everything associated with a given observation? --> No

Not to deviate the discussion, I'm not going into more detailed answers, but am happy to discuss further elsewhere if you're interested. For the rest of your comments, I don't have a clear answer now but they gave important insights as to what external users expect. We will discuss internally and come back with a solution. Thanks a lot! 👍

[juanmcloaiza]

@bsipocz, @keflavich, hi, thanks again for the discussion last time.

We will implement a new interface, but to streamline the review process, we won't do it in this pull request. I will open a new one shortly.

@bsipocz, could you please start giving a more thorough look at this pull request? @szampier is also looking at it in parallel, but we don't expect large changes.

Thanks in advance, looking forward to your feedback!