Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problem with hpx::collectives::exclusive_scan #6636

Open
mppf opened this issue Mar 11, 2025 · 1 comment
Open

problem with hpx::collectives::exclusive_scan #6636

mppf opened this issue Mar 11, 2025 · 1 comment

Comments

@mppf
Copy link

mppf commented Mar 11, 2025

I'm seeing a problem with hpx::collectives::exclusive_scan where it seems to be returning double the output I was expecting.

#include <hpx/config.hpp>
#include <hpx/algorithm.hpp>
#include <hpx/hpx.hpp>
#include <hpx/hpx_init.hpp>
#include <hpx/include/partitioned_vector.hpp>
#include <hpx/modules/program_options.hpp>
#include <hpx/modules/collectives.hpp>

#include <cstddef>
#include <cstdio>
#include <cstdint>
#include <cstdlib>
#include <ctime>
#include <iostream>
#include <random>
#include <string>
#include <vector>
#include <span>



int hpx_main(hpx::program_options::variables_map& vm)
{
  // this is run on all localities (nodes)
  auto myRank = hpx::get_locality_id();
  auto numRanks = hpx::get_num_localities(hpx::launch::sync);

  hpx::distributed::barrier::synchronize();

  int64_t totalHere = myRank + 100;

  int generation = 1;

  hpx::future<int64_t> myGlobalStartF =
    hpx::collectives::exclusive_scan("myScan",
                                     totalHere,
                                     std::plus<int64_t>{},
                                     hpx::collectives::num_sites_arg(),
                                     hpx::collectives::this_site_arg(),
                                     hpx::collectives::generation_arg(generation));


  int64_t myGlobalStart = myGlobalStartF.get();
  if (myRank == 0) {
    myGlobalStart = 0; // avoid odd behavior around rank 0
  }

  std::cout << "node " << myRank << " totalHere " << totalHere << " myGlobalStart " << myGlobalStart << "\n";

  hpx::distributed::barrier::synchronize();

  return hpx::finalize();
}

int main(int argc, char* argv[])
{
    // add command line option which controls the random number generator seed
    using namespace hpx::program_options;
    options_description desc_commandline(
        "Usage: " HPX_APPLICATION_STRING " [options]");

    desc_commandline.add_options()
        ("numElems,n", value<int64_t>(),
            "the data array size to use (default: 128*1024*1024)")
        ("print,p", value<bool>(),
            "print some of the array (default: false)")
        ;

    // run hpx_main on all localities
    std::vector<std::string> const cfg = {"hpx.run_hpx_main!=1"};

    // Initialize and run HPX
    hpx::init_params init_args;
    init_args.desc_cmdline = desc_commandline;
    init_args.cfg = cfg;

    return hpx::init(argc, argv, init_args);
}
$ mpirun -n 2 ./build/scan-bug --hpx:threads 2
node 0 totalHere 100 myGlobalStart 0
node 1 totalHere 101 myGlobalStart 200

I was expecting node 1's myGlobalStart to be 100 because it's an exclusive scan.
If I change to use hpx::collectives::inclusive_scan and then subtract the count, I get the result I was expecting.

I am using HPX v1.10.0 that I build on Ubuntu 24.10 with OpenMPI and HPX built to use it. I am running all MPI ranks on one system (for testing purposes).

@hkaiser
Copy link
Member

hkaiser commented Mar 11, 2025

@mppf Thanks for this report, I'll investigate. I wouldn't put it past us for this to be broken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants