It is possible to use threading libraries to read multiple parquet files in parallel

[aavbsouza]

I need to read multiple files in parallel using the arrow cpp dataset API or the Parquet API. Most of the objects used would be constructed from each thread but for instance the default arrow memory pool would be the same. Is this a problem? I would need to also construct a memory pool per thread/task. I was thinking to use something along these lines (with the dataset api or the parquet API).

// code generated by a chatbot
#include <tbb/parallel_for.h>
#include <tbb/blocked_range.h>
#include <arrow/api.h>
#include <arrow/io/api.h>
#include <parquet/arrow/reader.h>
#include <filesystem>
#include <iostream>

void ReadParquetFile(const std::string& path) {
  // 1. Open input file
  std::shared_ptr<arrow::io::ReadableFile> infile;
  PARQUET_ASSIGN_OR_THROW(
      infile, arrow::io::ReadableFile::Open(path));

  // 2. Create Arrow-Parquet reader
  std::unique_ptr<parquet::arrow::FileReader> reader;
  PARQUET_THROW_NOT_OK(
      parquet::arrow::OpenFile(infile,
                               arrow::default_memory_pool(),
                               &reader));

  // 3. Read entire file as a Table
  std::shared_ptr<arrow::Table> table;
  PARQUET_THROW_NOT_OK(reader->ReadTable(&table));

  // Your processing logic here
  std::cout << "Read " << path << " => " 
            << table->num_rows() << " rows\n";
}

void ReadFilesParallel(const std::vector<std::string>& files) {
  tbb::parallel_for(tbb::blocked_range<size_t>(0, files.size()),
                    [&](const tbb::blocked_range<size_t>& r) {
    for (size_t i = r.begin(); i < r.end(); ++i) {
      ReadParquetFile(files[i]);
    }
  });
}

thanks

[felipecrv]

Memory pools are thread-safe and threading is handled by the memory allocator. So if you're using the jemalloc or mimalloc memory pools, these modern allocators will have their own thread-local caches and internal pools.