Code

Binary quantization#82

Code

Merged

Kerollmops merged 61 commits intomainmeilisearch/arroy:mainfrom binary-quantizationmeilisearch/arroy:binary-quantizationCopy head branch name to clipboardSep 19, 2024

+1,953-622

Commits

Commits on Jul 1, 2024

• First draft

  

  Kerollmops

  authored and

  irevoire

  committed
  3659904

  Copy full SHA for 3659904

Commits on Jul 3, 2024

• prepare for the new distance trait

  irevoire

  committed
  692f1ef

  Copy full SHA for 692f1ef

Commits on Jul 4, 2024

• re-implements display for all kind of nodes taking the distance into account

  irevoire

  committed
  27ed4d4

  Copy full SHA for 27ed4d4

Commits on Jul 8, 2024

• everything compiles and the api is clean, need work on the tests to ensure it works

  irevoire

  committed
  3315465

  Copy full SHA for 3315465

• wip

  irevoire

  committed
  a27a238

  Copy full SHA for a27a238

• fix the formatter and add a few comments

  irevoire

  committed
  3634125

  Copy full SHA for 3634125

• rewrite the iterator of binary quantized slices

  irevoire

  committed
  3576a88

  Copy full SHA for 3576a88

• fix the size of the iterator

  irevoire

  committed
  facc857

  Copy full SHA for facc857

• fix spaces

  irevoire

  committed
  15a13ff

  Copy full SHA for 15a13ff

Commits on Jul 9, 2024

• remove the now useless craft_owned_unaligned_vector_from_f32 function from the distance trait

  irevoire

  committed
  e4847d2

  Copy full SHA for e4847d2

• get rid of the drain

  irevoire

  committed
  ddb2304

  Copy full SHA for ddb2304

• fix the spaces tests

  irevoire

  committed
  903ec0b

  Copy full SHA for 903ec0b

• fix the spaces again

  irevoire

  committed
  ae65345

  Copy full SHA for ae65345

• add a comment explaining the relevancy issue we may encounter

  irevoire

  committed
  9ab541e

  Copy full SHA for 9ab541e

• rename the vector format to vector codec

  irevoire

  committed
  7907da9

  Copy full SHA for 7907da9

Commits on Jul 10, 2024

• fix the normalized distance for the binary quantized euclidean distance

  irevoire

  committed
  47fd2bd

  Copy full SHA for 47fd2bd

• add a first test of relevancy that uses autogenerated vectors

  irevoire

  committed
  4c853b8

  Copy full SHA for 4c853b8

• add an implementation for the binary quantized manhattan distance

  irevoire

  committed
  7f004e8

  Copy full SHA for 7f004e8

• improve lisibility of the relevancy benchmark

  irevoire

  committed
  b6ea560

  Copy full SHA for b6ea560

• implements a basic oversampling

  irevoire

  committed
  0e8fba2

  Copy full SHA for 0e8fba2

Commits on Aug 5, 2024

• compute two_means on non binary quantized distances

  irevoire

  committed
  9f66448

  Copy full SHA for 9f66448

Commits on Aug 6, 2024

• Make two_means return non binary quantized distances

  irevoire

  committed
  732767a

  Copy full SHA for 732767a

Commits on Aug 7, 2024

• add the angular distance

  irevoire

  committed
  54626c1

  Copy full SHA for 54626c1

Commits on Sep 4, 2024

• implement the oversampling

  irevoire

  committed
  6290296

  Copy full SHA for 6290296

Commits on Sep 5, 2024

• make the binary quantized distance quick again

  irevoire

  committed
  d812bd8

  Copy full SHA for d812bd8

Commits on Sep 9, 2024

• provide a specialized method to check if a vector contains only zeros

  irevoire

  committed
  f9e2b63

  Copy full SHA for f9e2b63

• fix the euclidean distance

  irevoire

  committed
  60875eb

  Copy full SHA for 60875eb

• fix the tests

  irevoire

  committed
  8c541ed

  Copy full SHA for 8c541ed

Commits on Sep 10, 2024

• [perf] use an hashmap instead of a roaring bitmap + a vec to store the immutable tree/leafs

  irevoire

  committed
  d75d550

  Copy full SHA for d75d550

• [perf] reduce the number of allocations by pre-allocating the size of the hashmap in advance

  irevoire

  committed
  cefadca

  Copy full SHA for cefadca

• add or fix comments

  irevoire

  committed
  c96e204

  Copy full SHA for c96e204

Commits on Sep 11, 2024

• write an simd version of the code that converts the f32 vectors to simd vectors

  irevoire

  committed
  0945e33

  Copy full SHA for 0945e33

• fix the reminder while storing binary quantized vectors

  irevoire

  committed
  d4ed3e1

  Copy full SHA for d4ed3e1

• re-implement the binary quantized to f32 with SIMD

  irevoire

  committed
  343dbfb

  Copy full SHA for 343dbfb

• add tests

  irevoire

  committed
  1879821

  Copy full SHA for 1879821

Commits on Sep 12, 2024

• push leaf id in a vec instead of a roaring bitmap because it s quicker

  irevoire

  committed
  4d07b35

  Copy full SHA for 4d07b35

• fix the f32 -> binary quantized

  irevoire

  committed
  98cb3c9

  Copy full SHA for 98cb3c9

• improve comments and types

  irevoire

  committed
  1c975e2

  Copy full SHA for 1c975e2

Commits on Sep 13, 2024

• First version of SIMD on x86_64

  Kerollmops

  committed
  f9fea3b

  Copy full SHA for f9fea3b

• Upload the AVX version of to_vec

  Kerollmops

  committed
  1592fe0

  Copy full SHA for 1592fe0

Commits on Sep 14, 2024

• Update the simd aarch64 to_vec_simd function with blend functions

  Kerollmops

  committed
  fc7a72b

  Copy full SHA for fc7a72b

Commits on Sep 16, 2024

• fix the binary quantized conversions

  irevoire

  committed
  b72958a

  Copy full SHA for b72958a

• gate each simd function behind the right arch and provide a fallback method in case no simd instructions are available

  irevoire

  committed
  0f88bfd

  Copy full SHA for 0f88bfd

• fix and improve the large binary quantized test

  irevoire

  committed
  149e338

  Copy full SHA for 149e338

• move the unaligned vector test to their own module

  irevoire

  committed
  7e8ee31

  Copy full SHA for 7e8ee31

• move the node test to the binary_quantized_tests

  irevoire

  committed
  d61b7af

  Copy full SHA for d61b7af

• fix the warnings

  irevoire

  committed
  283eb2c

  Copy full SHA for 283eb2c

• move proptest to the dev dependencies

  irevoire

  committed
  f2f6a7e

  Copy full SHA for f2f6a7e

• fix comment

  irevoire

  committed
  0b21f34

  Copy full SHA for 0b21f34

• fix typo$

  irevoire

  committed
  605b33a

  Copy full SHA for 605b33a

Commits on Sep 17, 2024

• update a test playing with -0.0 and 0.0 since -0.0 doesn't answer the same way to is_positive under x86 and aarch64

  irevoire

  committed
  df654bc

  Copy full SHA for df654bc

• fix a comment

  irevoire

  committed
  be0f600

  Copy full SHA for be0f600

• rename + unroll loop in from_slice_neon

  irevoire

  committed
  2d9eeb2

  Copy full SHA for 2d9eeb2

• update reader::plot with .is_zero

  irevoire

  committed
  8f6eaaa

  Copy full SHA for 8f6eaaa

• fix github lint

  irevoire

  committed
  c4aaa44

  Copy full SHA for c4aaa44

• clippy again

  irevoire

  committed
  9c457fe

  Copy full SHA for 9c457fe

• improve the display implementation of the splitnode

  irevoire

  committed
  af598ca

  Copy full SHA for af598ca

• making the constants in unaligned_vector::binary_quantized more explicits

  irevoire

  committed
  71f6659

  Copy full SHA for 71f6659

• fix the cfg feature gates around the simd function

  irevoire

  committed
  777effb

  Copy full SHA for 777effb

• remove the small relevancy benchmark in favor of the new repository

  irevoire

  committed
  393a5c5

  Copy full SHA for 393a5c5

Commits on Sep 18, 2024

• fix the way we change the distance

  irevoire

  committed
  ddaedd3

  Copy full SHA for ddaedd3