Merge pull request #2 from oliverphilcox/restructuring

Restructuring

Author: oliverphilcox

.gitignore

@@ -1,11 +1,21 @@
-slurm/*
+slurm/compute*
+slurm/bk_fish.slurm
+slurm/test*
 *.pyc
 *.py~
 *.md~
 notebooks/*
+bk/*.param
+pk/*.param
+_*.py
 figs/*
+pk/_*
 bk/_*
+pk/*.pyc
+bk/*.pyc
+bk/old*
 src/*.pyc
+src/_*.py
 src/test*.py
 *test*.py
 *corr*.py

README.md

@@ -1,15 +1,17 @@
 # BOSS Without Windows
 
-This repository contains details of the window-free analysis of BOSS DR12 data. This uses the quadratic and cubic estimators described in [Philcox 2020](https://arxiv.org/abs/2012.09389), and [Philcox 2021](https://arxiv.org/abs/2107.06287), and contains the data used in the BOSS power spectrum and bispectrum analysis of [Philcox & Ivanov 2022](http://arxiv.org/abs/2112.04515).
+This repository contains details of the window-free analysis of non-uniform spectroscopic data. This uses the quadratic and cubic estimators described in [Philcox 2020](https://arxiv.org/abs/2012.09389), and [Philcox 2021](https://arxiv.org/abs/2107.06287), and additionally contains the data generated for the BOSS power spectrum and bispectrum analysis of [Philcox & Ivanov](https://arxiv.org/abs/2112.04515).
 
 ## Outline
 - [pk/](pk): Analysis code to estimate unwindowed power spectra. We additionally supply the raw power spectrum measurements of BOSS, 2048 Patchy simulations and 84 Nseries simulations.
 - [bk/](bk): Analysis code to estimate unwindowed bispectra. We additionally supply the raw bispectrum measurements of BOSS and 2048 Patchy simulations and 84 Nseries simulations.
+- [paramfiles/](paramfiles): Example parameter files.
 - [src/](src): Various Python utilities used in the unwindowed estimators.
+- [slurm/](slurm): Example SLURM submission script.
 - [generate_mask.py](generate_mask.py): Utility function to generate the background number density, n(r) from the survey mask and n(z) distribution. This is described in the code header.
 
 ### Requirements
-To run the analysis code one requires:
+The scripts in this repository have the following dependencies:
 - python (2 or 3)
 - numpy
 - scipy
@@ -19,7 +21,7 @@ To run the analysis code one requires:
 - [fasteners](https://pypi.org/project/fasteners/) (for providing file overwrites in bispectrum computation)
 - [mangle](https://github.com/mollyswanson/manglepy) (for reading survey mask files)
 
-We additionally use a number of the BOSS data products available on the BOSS [SAS](https://data.sdss.org/sas/dr12/boss/lss/). Once downloaded, the locations of these files should be specified in [this file](src/opt_utilities.py).
+When applying the code to the BOSS survey, we use a number of products available on the BOSS [SAS](https://data.sdss.org/sas/dr12/boss/lss/), which are references in the relevant [paramfiles](paramfiles).
 
 ## Acknowledgements
 
@@ -31,4 +33,6 @@ Codes and data from this repository are used in the following publications:
 
 - Philcox (2020, [Phys. Rev. D](https://journals.aps.org/prd/abstract/10.1103/PhysRevD.103.103504), [arXiv](https://arxiv.org/abs/2012.09389)): Description of the unwindowed power spectrum estimators.
 - Philcox (2021, [Phys. Rev. D](https://doi.org/10.1103/PhysRevD.104.123529), [arXiv](https://arxiv.org/abs/2107.06287)): Description of the unwindowed bispectrum estimators.
-- Philcox & Ivanov  (2022, [Phys. Rev. D](https://doi.org/10.1103/PhysRevD.105.043517), [arXiv](http://arxiv.org/abs/2112.04515)): Combined constraints on LambdaCDM from the BOSS power spectrum and bispectrum.
+- Philcox & Ivanov (2021, [Phys. Rev. D](https://doi.org/10.1103/PhysRevD.105.043517), [arXiv](https://arxiv.org/abs/2112.04515)): Combined constraints on LambdaCDM from the BOSS power spectrum and bispectrum.
+- Cabass et al. (2022, [arXiv](https://arxiv.org/abs/2201.07238)): Constraints on single-field inflation from the BOSS power spectrum and bispectrum.
+- Nunes et al. (2022, [arXiv](https://arxiv.org/abs/2203.08093)): Constraints on dark-sector interactions from the BOSS galaxy power spectrum.

bk/README.md

@@ -2,32 +2,19 @@
 
 ### Overview
 
-To compute bispectra, there are two steps:
-1. Run the ```compute_bk_randoms.py``` script to generate and analyze uniformly distributed particles. These are used to compute the Fisher and average q-alpha term matrix. This script should be run with 50-100 choices of input parameter ```rand_it```. We also provide the ```compute_bk_boss_randoms.py``` script to compute the analogous Fisher matrix for the BOSS data (which has a different random catalog).
-2. Run the ```compute_bk_data.py``` script to analyze a specific Patchy simulation or BOSS data. Step (1) must be computed before this is run.
+To compute bispectra, there are three steps:
+1. Run the ```compute_bk_randoms.py``` script to generate and analyze randomly distributed particles. These are used to compute the Fisher matrix and average q-alpha term, encoding the survey geometry. This script should be run with around 100 choices of input parameter ```rand_it```.
+2. Run the ```compute_bk_fisher.py``` script to combine together the various contributions computed in step (1).
+3. Run the ```compute_bk_data.py``` script to analyze a specific simulation or dataset. Steps (1) and (2) must be computed before this is run.
 
-The output power spectrum will be saved as ```bk_patchy{SIM_NO}....txt``` or ```bk_boss....txt``` in the specified output directory.
+The output bispectra will be saved as ```bk_{TYPE}{SIM_NO}....txt``` in the specified output directory. Note that the background density map ```n(r)``` must be generated before these scripts are run; this can be done using the [generate_mask.py](../generate_mask.py) script.
 
 ### Input Parameters
 On the command line, the following parameters can be specified:
-- ```rand_it```: Index of the uniform simulation to be created and analyzed (for ```compute_bk_randoms.py```).
-- ```sim_no```: Patchy simulation number (for ```compute_bk_data.py```). If set to -1, the true BOSS data is analyzed.
-- ```patch```: Which region of BOSS to use, either ```ngc``` or ```sgc```.
-- ```z_type```: Which redshift region, either ```z1``` or ```z3```.
-- ```wtype```: Flag to indicate the type of weights. If set to 0, we use FKP-like weights, else maximum likelihood (ML) weights if set to 1. Note that both weights give unwindowed, pixelation-corrected power spectra.
--  ```grid_factor```: Factor by which to inflate the pixel size, relative to the original BOSS release.
-
-Within the code we can specify the following additional parameters:
-- ```N_mc```: Number of Monte Carlo simulations to compute bias and Fisher matrices.
-- ```k_min```, ```k_max```, ```dk```: Desired (linear) k-binning strategy.
-- ```h_fid```, ```OmegaM_fid```: Fiducial parameters to use when converting redshifts and angles into Cartesian co-ordinates.
-- ```tmpdir```: Directory to hold temporary output for each uniform simulation. This should have fast I/O and be large. It will be cleaned at the end of the ```compute_bk_randoms.py``` script.
-- ```mcdir```:  Directory to hold temporary Monte Carlo summations. This should be large, and can be removed after ```compute_bk_data.py``` has been run at least once.
-- ```outdir```: Output directory for saving bispectra. Additional auxilliary data including bias and Fisher matrix terms are also stored.
-- ```include_pix```: If true, forward model the effects of pixellation on the density field. (NB: the pixel window function is still removed at leading order in both cases).
-- ```rand_nbar```: If true, compute the background number density from random particles rather than from the survey mask.
-- ```use_qbar```: If false, do not subtract the 'q-bar' term in the bispectrum estimator (for testing only).
+- ```rand_it```: Index of the Gaussian-random simulation to be created and analyzed (for ```compute_bk_randoms.py```).
+- ```sim_no```: Simulation number (for ```compute_bk_data.py```). If analyzing an unlabelled simulation, or observational data, this is set to -1.
+- ```paramfile```: Parameter file specifying various settings, including the survey mask and dimensions. See [paramfiles](../paramfiles) for examples.
 
 ### Data
 
-In the [data/](data) directory, we give the raw unwindowed bispectrum measurements of BOSS, 2048 MultiDark-Patchy simulations and 84 Nseries simmulations. Further details of the input parameters can be found in the file headers. Note that we remove any bin triplets that are not properly corrected for the survey geometry. The original k-binning limits given in the final header.
+In the [data/](data) directory, we give the raw unwindowed bispectrum measurements of BOSS, 2048 MultiDark-Patchy simulations and 84 Nseries simmulations. Further details of the input parameters can be found in the file headers. Note that we remove any bin triplets that are not properly corrected for the survey geometry. The original k-binning limits given in the final header. Note also that these results were run with an earlier version of the code, thus follow slightly different naming conventions.