memory allocation error if strings too long

[richierocks]

Here's a large text file with 10 million lines, each containing 100 characters.

x <- paste0(sample(letters, 100, replace = TRUE), collapse = "")
lines <- rep.int(x, 1e7)
writeLines(lines, "bigfile.txt")

If I try to read it (tested on a modern desktop PC with 16GB RAM under Windows 7), R crashes with the error:

lines <- stri_read_lines("bigfile.txt")
## terminate called after throwing an instance of 'std::bad_alloc'
##   what():  std::bad_alloc

For comparison, I can read the file using base-R's readLines and data.table's fread.

lines <- readLines("bigfile.txt")                          # slow but OK
lines <- data.table::fread("bigfile.txt", sep = "\n")[[1]] # faster, also OK

[gagolews]

How large is this file, in bytes?

[richierocks]

Previous comment was a little short on lines; I've updated it to read 10 million (1e7 lines * 100 chars ~= 1GB). You may need to vary the file size a bit to reproduce the error, but stri_read_lines definitely fails before readLines on the (Windows) machines that I've tried.

stri_read_raw works fine on the file; the crash happens in stri_encode.

[gagolews]

OK, thanks for the notice. That's why stri_read_lines is marked as Draft API. I will rewrite the function in pure C++ some day (now it calls too much R code). I can't promise I'll do that soon, as my main priority for stringi_0.5-1 is to add date/time parse/format features. Thanks also for pointing out data.table's fread == I'll try to make this function even faster than that :)

[richierocks]

No pressure, but is this still on the TODO list?

[gagolews]

Sure, but I'm quite busy right now...

[gagolews]

Actually, it's stri_encode()...

> x <- charToRaw(stringi::stri_dup("a", 2**29))
> y <- stringi::stri_encode(x, NULL, "utf-8")
Error in stringi::stri_encode(x, NULL, "utf-8") : 
  memory allocation or access error

[gagolews]

The same with stri_rand_strings():

> x <- stringi::stri_rand_strings(1, 2**29)
Error in stringi::stri_rand_strings(1, 2^29) : 
  memory allocation or access error

[gagolews]

Fix on the way; changing buf size type to size_t:

> x <- stringi::stri_rand_strings(1, 2**31-1)
> stringi::stri_length(x)
[1] 2147483647

[gagolews]

Well.., I'll give it a thought tomorrow again 🤔

x <- paste0(sample(letters, 100, replace = TRUE), collapse = "")
lines <- rep.int(x, 1e7)
writeLines(lines, "bigfile.txt")
lines <- stringi::stri_read_lines("bigfile.txt")
##Error in stri_encode(txt, encoding, "UTF-8") : 
##  Start of codes indicating failure. (U_ILLEGAL_ARGUMENT_ERROR)

[gagolews]

After fixing a few bugs, which were there, encoding conversion for in-memory data of size 672 MB works fine. A few MBs more and we will get the following error (which, at least, is now more informative):

Error in stringi::stri_encode(x, NULL, "utf-8") : 
   memory allocation or access error

I would have to rewrite the whole stri_encode to support larger buffers than this, which I don't think is of high priority (?) Of course, I'm open for discussion, but still the predicted upper limit for raw inputs will be ca. (AVAILABLE_RAM)/4 anyway -- not that much more, so I don't think it's worth the hassle.

A workaround is to open a file connection and read data in mini-batches (say, of 0.5 GBs of size).

[gagolews]

I have created related issues: #395, #396