Skip to content

ARROW-17933: [C++] SparseCOOTensor raises error when created with zero elements #14378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rok wants to merge 3 commits into
base: main
Choose a base branch
from
Open

ARROW-17933: [C++] SparseCOOTensor raises error when created with zero elements #14378

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
aaa4987
First commit
rok Oct 11, 2022
6201e8b
Review feedback
rok Oct 13, 2022
de2440f
Review feedback
rok Oct 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions cpp/src/arrow/sparse_tensor.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ inline Status CheckSparseCOOIndexValidity(const std::shared_ptr<DataType>& type,

RETURN_NOT_OK(internal::CheckSparseIndexMaximumValue(type, shape));

// Indexes with no values are considered valid

@westonpace westonpace Apr 20, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what is happening here (I don't understand much about the tensors so not surprising). Why is it ok if one of the shape elements is zero? I would expect an empty sparse matrix to still have a shape:

>>> scipy.sparse.coo_matrix(numpy.zeros((2,4)), dtype=numpy.float32).shape
(2, 4)

if (std::find(shape.begin(), shape.end(), 0) != shape.end()) {
return Status::OK();
}
if (!internal::IsTensorStridesContiguous(type, shape, strides)) {
return Status::Invalid("SparseCOOIndex indices must be contiguous");
}
Expand Down
50 changes: 50 additions & 0 deletions cpp/src/arrow/sparse_tensor_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1676,4 +1676,54 @@ TEST(TestSparseCSFMatrixForUInt64Index, Make) {
ASSERT_RAISES(Invalid, SparseCSFTensor::Make(dense_tensor, uint64()));
}

//-----------------------------------------------------------------------------
// Create SparseTensors from a dense Tensor with only zeros

template <typename SparseTensorType>
class TestSparseTensorFromDenseBase : public ::testing::Test {
public:
void SetUp() {
shape_ = {0, 12};
dim_names_ = {"foo", "bar"};
dense_values_ = {};
dense_data_ = Buffer::Wrap(dense_values_);
}

protected:
std::vector<int64_t> shape_;
std::vector<std::string> dim_names_;
std::vector<int64_t> dense_values_;
std::shared_ptr<Buffer> dense_data_;
};

template <typename SparseTensorType>
class TestSparseTensorFromDense : public TestSparseTensorFromDenseBase<SparseTensorType> {
};

TYPED_TEST_SUITE_P(TestSparseTensorFromDense);

TYPED_TEST_P(TestSparseTensorFromDense, TestNonZeroLength) {

@pitrou pitrou Oct 13, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but is it the situation that the PR is meant to be fixing?
AFAIU, the problem was not when there were no non-zero values, but when the logical tensor was empty (one of the dimensions in the shape being zero).

@rok rok Oct 13, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. I changed the c++ test so that one dimension is now 0.

using SparseTensorType = TypeParam;

NumericTensor<Int64Type> dense_tensor_ =
NumericTensor<Int64Type>(this->dense_data_, this->shape_, {}, this->dim_names_);
ASSERT_OK_AND_ASSIGN(
auto sparse_tensor_,
SparseTensorType::Make(dense_tensor_, TypeTraits<Int64Type>::type_singleton()));
ASSERT_EQ(sparse_tensor_->non_zero_length(), 0);
ASSERT_EQ(sparse_tensor_->shape(), this->shape_);
ASSERT_EQ(sparse_tensor_->dim_names(), this->dim_names_);
}

REGISTER_TYPED_TEST_SUITE_P(TestSparseTensorFromDense, TestNonZeroLength);

INSTANTIATE_TYPED_TEST_SUITE_P(TestSparseCOOTensor, TestSparseTensorFromDense,
SparseCOOTensor);
INSTANTIATE_TYPED_TEST_SUITE_P(TestSparseCSRMatrix, TestSparseTensorFromDense,
SparseCSRMatrix);
INSTANTIATE_TYPED_TEST_SUITE_P(TestSparseCSCMatrix, TestSparseTensorFromDense,
SparseCSCMatrix);
INSTANTIATE_TYPED_TEST_SUITE_P(TestSparseCSFTensor, TestSparseTensorFromDense,
SparseCSFTensor);

} // namespace arrow
44 changes: 43 additions & 1 deletion python/pyarrow/tests/test_sparse_tensor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@
except ImportError:
sparse = None


tensor_type_pairs = [
('i1', pa.int8()),
('i2', pa.int16()),
Expand Down Expand Up @@ -434,6 +433,23 @@ def test_sparse_coo_tensor_scipy_roundtrip(dtype_str, arrow_type):
assert sparse_tensor.has_canonical_format
assert out_scipy_matrix.has_canonical_format

scipy_matrix = coo_matrix([[0, 0], [0, 0]])

@pitrou pitrou Oct 12, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually create a tensor with zero elements?

@rok rok Oct 12, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, coo_matrix automatically ignores zeros, I'll add an assertion checking for number of elements to make this explicit.

@rok rok Oct 13, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well actually it's:

    scipy_matrix = coo_matrix(([0], ([0], [0])), shape=(2, 4))
    assert scipy_matrix.nnz == 1
    scipy_matrix = coo_matrix([[0, 0], [0, 0]])
    assert scipy_matrix.nnz == 0

@pitrou pitrou Oct 13, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but the shape is all non-zero.

@rok rok Oct 13, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well those are actually different constructors (shape seems to be ignored in this case):

1.
coo_matrix((data, (i, j)), [shape=(M, N)])
to construct from three arrays:
data[:] the entries of the matrix, in any order
i[:] the row indices of the matrix entries
j[:] the column indices of the matrix entries

    scipy_matrix = coo_matrix(([0], ([0], [0])))
    assert scipy_matrix.nnz == 1

2.
coo_matrix(D)
with a dense matrix D

    scipy_matrix = coo_matrix([[0, 0], [0, 0]])
    assert scipy_matrix.nnz == 0

https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.coo_matrix.html

@pitrou pitrou Oct 13, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this doesn't address the issue, does it?

@rok rok Oct 14, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reported issue was due to scipy.coo_matrix returning a sparse matrix with a dimension of zero size for an all zeros dense tensor. This is not an issue if the sparse matrix is created from components. Propose change tests from dense creation path in Python with scipy.coo_matrix, scipy.csr_matrix and sparse.COO. It also tests C++ SparseTensor creation from a dense tensor with one zero-sized dimension for SparseCOOTensor, SparseCSRMatrix, SparseCSRMatrix and SparseCSFTensor.

@rok rok Oct 14, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be missing the point though :D

@rok rok Oct 18, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pitrou ping

@westonpace westonpace Apr 20, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe @rok has the correct test here but I could be wrong. The original issue is about a sparse matrix with a valid non-zero shape but no elements:

>>> scipy.sparse.coo_matrix(numpy.zeros((2,4)), dtype=numpy.float32)
<2x4 sparse matrix of type '<class 'numpy.float32'>'
	with 0 stored elements in COOrdinate format>
>>> numpy.zeros((2,4))
array([[0., 0., 0., 0.],
       [0., 0., 0., 0.]])
>>> numpy.zeros((2,4)).shape
(2, 4)

sparse_tensor = pa.SparseCOOTensor.from_scipy(scipy_matrix,
dim_names=dim_names)
out_scipy_matrix = sparse_tensor.to_scipy()
dense_array = scipy_matrix.toarray()

assert scipy_matrix.has_canonical_format
assert sparse_tensor.has_canonical_format
assert out_scipy_matrix.has_canonical_format

assert scipy_matrix.nnz == 0
assert scipy_matrix.nnz == sparse_tensor.non_zero_length
assert np.array_equal(scipy_matrix.data, out_scipy_matrix.data)
assert np.array_equal(scipy_matrix.row, out_scipy_matrix.row)
assert np.array_equal(scipy_matrix.col, out_scipy_matrix.col)
assert np.array_equal(dense_array, sparse_tensor.to_tensor().to_numpy())


@pytest.mark.skipif(not csr_matrix, reason="requires scipy")
@pytest.mark.parametrize('dtype_str,arrow_type', tensor_type_pairs)
Expand Down Expand Up @@ -464,6 +480,19 @@ def test_sparse_csr_matrix_scipy_roundtrip(dtype_str, arrow_type):
dense_array = sparse_array.toarray()
assert np.array_equal(dense_array, sparse_tensor.to_tensor().to_numpy())

scipy_matrix = csr_matrix([[0, 0], [0, 0]])
sparse_tensor = pa.SparseCSRMatrix.from_scipy(scipy_matrix,
dim_names=dim_names)
out_scipy_matrix = sparse_tensor.to_scipy()
dense_array = scipy_matrix.toarray()

assert scipy_matrix.nnz == 0
assert scipy_matrix.nnz == sparse_tensor.non_zero_length
assert np.array_equal(scipy_matrix.data, out_scipy_matrix.data)
assert np.array_equal(scipy_matrix.indptr, out_scipy_matrix.indptr)
assert np.array_equal(scipy_matrix.indices, out_scipy_matrix.indices)
assert np.array_equal(dense_array, sparse_tensor.to_tensor().to_numpy())


@pytest.mark.skipif(not sparse, reason="requires pydata/sparse")
@pytest.mark.parametrize('dtype_str,arrow_type', tensor_type_pairs)
Expand All @@ -489,3 +518,16 @@ def test_pydata_sparse_sparse_coo_tensor_roundtrip(dtype_str, arrow_type):
assert np.array_equal(sparse_array.coords, out_sparse_array.coords)
assert np.array_equal(sparse_array.todense(),
sparse_tensor.to_tensor().to_numpy())

sparse_array = sparse.COO.from_numpy([[0, 0], [0, 0]])
sparse_tensor = pa.SparseCOOTensor.from_pydata_sparse(sparse_array,
dim_names=dim_names)
out_sparse_array = sparse_tensor.to_pydata_sparse()
dense_array = sparse_array.todense()

assert sparse_array.nnz == 0
assert sparse_array.nnz == sparse_tensor.non_zero_length
assert out_sparse_array.nnz == sparse_tensor.non_zero_length
assert np.array_equal(sparse_array.data, out_sparse_array.data)
assert np.array_equal(sparse_array.coords, out_sparse_array.coords)
assert np.array_equal(dense_array, sparse_tensor.to_tensor().to_numpy())
Loading