ParquetRecordReader::read_from_row_group - num_records cannot exceed number of rows

[Paul-Folbrecht]

Your sample:

#[derive(ParquetRecordReader)]
struct ACompleteRecord {
    pub a_bool: bool,
    pub a_string: String,
}

pub fn read_some_records() -> Vec<ACompleteRecord> {
  let mut samples: Vec<ACompleteRecord> = Vec::new();
  let file = File::open("some_file.parquet").unwrap();

  let reader = SerializedFileReader::new(file).unwrap();
  let mut row_group = reader.get_row_group(0).unwrap();
  samples.read_from_row_group(&mut *row_group, 1).unwrap();
  samples
}

The docs state

"Read up to num_records records from row_group_reader into self."

But if you pass more than the number of rows in the file, you get an error like

thread 'parquet_reader::test_read_records' panicked at services/src/parquet_reader.rs:10:10:
index out of bounds: the len is 66945 but the index is 66945

Since there's no way to determine the number of rows without iterating once, this is a problem.

[tustvold]

https://arrow.apache.org/rust/parquet/file/reader/trait.RowGroupReader.html#tymethod.metadata

Contains the number of rows

[Paul-Folbrecht]

Thanks. It turns out that the parquets we're processing have bad metadata and so that value is wrong. But, you are correct, no bug here.