Data loading & preprocessing pipeline feature

[ageron]

The DataLoader is nice, but if I understand correctly it requires the dataset to fit in memory. For large datasets that don't fit in memory, it would be nice to have an easy way to load & preprocess the data efficiently, similar to TensorFlow's tf.data API. Maybe something like this exists already?

If not, perhaps one option would be to provide custom transducers to make it possible to write things like:

data = (csv_file_paths |> Shuffle(length(csv_file_paths)) |> Interleave(CSV.File; threads=4)
             |> Map(preprocess_sample) |> Shuffle(100_000) |> Batch(32) |> Prefetch(1))

This would load records from multiple files (in random file order), pick 4 randomly, interleave their records, preprocess every record, shuffle records using a 100,000 element buffer, and batch the records with batch size 32, and prefetch 1 batch (so the CPU can prepare the next batch while the GPU is working on the previous batch). Then the data could be used for training.

[pxl-th]

Right now it is possible to use DataLoader with datasets that do not fit into memory, but feels somewhat hacky... or not.
Let's say we have dataset of images, which we want to load on demand.
Then we can define a custom struct subtyping AbstractArray.

struct Dataset{T, N} <: AbstractArray{T, N}
    frame_template::FormatExpr
    targets::AbstractArray
end

Dataset{T}(frame_template, targets) where {T} =
    Dataset{T, 1}(frame_template, targets)

Then defining getindex function that describes how we load one item

function Base.getindex(d::Dataset{T}, i::Int) where {T}
    path = format(d.frame_template, i - 1)
    image = path |> FileIO.load |> Images.channelview .|> T
    image, d.targets[[i]]
end

and how we load mini-batch

function Base.getindex(d::Dataset{T}, ids::Array) where {T}
    x, y = d[ids[1]]
    xs_last_dim = ntuple(i -> Colon(), ndims(x))
    ys_last_dim = ntuple(i -> Colon(), ndims(y))

    xs = Array{T}(undef, size(x)..., length(ids))
    ys = Array{T}(undef, size(y)..., length(ids))

    xs[xs_last_dim..., 1] .= x
    ys[ys_last_dim..., 1] .= y

    for (i, id) in enumerate(ids[2:end])
        x, y = d[id]
        xs[xs_last_dim..., i + 1] .= x
        ys[ys_last_dim..., i + 1] .= y
    end
    xs, ys
end

And some helper functions

Base.IndexStyle(::Type{Dataset}) = IndexLinear()
Base.size(d::Dataset) = (length(d.targets),)
Base.length(d::Dataset) = length(d.targets)

This in some sense mimicks PyTorch's Dataset
and allows to use DataLoader with datasets that do not fit into memory

frame_template = FormatExpr(raw".\frames\frame-{:d}.jpg")  # Template path for images.
targets = load_from_txt(raw".\speed.txt")  # Array of targets

dataset = Dataset{Float32}(frame_template, targets)
loader = Flux.Data.DataLoader(dataset, batchsize=4, shuffle=true)
println("Loader length: $(length(loader))")
for (i, (x, y)) in enumerate(loader)
    i == 10 && break
    println("$i: $(size(x)) $(size(y))")
end

Output of the data from my example:

Loader length: 240
1: (3, 160, 320, 4) (1, 4)
2: (3, 160, 320, 4) (1, 4)
3: (3, 160, 320, 4) (1, 4)
4: (3, 160, 320, 4) (1, 4)
5: (3, 160, 320, 4) (1, 4)
6: (3, 160, 320, 4) (1, 4)
7: (3, 160, 320, 4) (1, 4)
8: (3, 160, 320, 4) (1, 4)
9: (3, 160, 320, 4) (1, 4)

[CarloLucibello]

For the time being, we can just document the interface that a "Dataset" should expose in order to be compatible with the DataLoader. @pxl-th a PR in this direction would be very welcome.

In the longer run, we should definitely consider reimplementing the DataLoader on top of transducers. Transducers are great and come fully packed with features, as @ageron showed.

[ageron]

Thanks for your detailed answer @pxl-th . I'm not sure whether my code example really makes sense, but it's the kind of API I would imagine, largely inspired from TF's tf.data API, with a transducers twist. I'm happy to help if you want.

[darleybarreto]

I wonder if something more idiomatic could be done, like:

# I can call custom now and it will return three objects
@dataset (:train) image,target1,target2 function custom(path_arrays,idx)
    image = # load from the path_arrays[idx] + ...
    target1 = # load from the path_arrays[idx] + ...
    target2 = # load from the path_arrays[idx] + ...
end

or

# add another method to dataset
function dataset(path_arrays,idx, :train)
    image = # load from the path_arrays[idx] + ...
    target1 = # load from the path_arrays[idx] + ...
    target2 = # load from the path_arrays[idx] + ...

    image,target1,target2 
end

And another type called DataSet could be added having an inner field (_data or something like that), which could be the the array of paths (path_arrays) or the true data, depending on the user choice. And the data inside the DataLoader is a DataSet, which samples from the corresponding dataset in :train, :val, :test or any other custom symbol defining a step.

I am not a Julia expert, but I could help implementing it 😄 .

[paritosh5feb]

Hi everyone! I am trying to write a CNN (U-Net) in Flux ML Library of Julia. However the first hurdle that I am facing is, how to load images from folder to train the model. I have searched the Internet extensively for any method, but no luck. All the examples on the internet for CNN in Flux, use library functions to import pre-existing data-sets like MNIST dataset. Actually all the examples, use a library functions to load this dataset. Please someone tell me, how to load custom image dataset from folders to train a CNN model written in Flux Julia. I would be really grateful!

[DhairyaLGandhi]

You can look at UNet.jl

[paritosh5feb]

You can look at UNet.jl
Can you tell me how to load the images from the folder for training the model. Like there was flowfromdirectory function in Keras (python).

[DhairyaLGandhi]

There's a src/dataloader.jl in UNet.jl which talks about loading single images and also batches of images. You can also look at DataSets.jl which helps keep track of directory structures. Basically, you need to have a way to point to the images you want to load and then use packages such as Images.jl, FilelO.jl ImageMagick.jl, ArchGDAL.jl etc to suit the needs of your specific directory structure and file types. Then you can put it into the format Flux expects (see the tutorials on https://fluxml.ai). There's also Metalhead.preprocess but that typically doesn't manage the directory structure and DataSets.jl can be used to mimic the keras function you mentioned.

Most libraries would have a form of load or read function that you can pass a string or IO/IOBuffer type object to load it into Julia arrays.

[paritosh5feb]

Can I use the load_batch function to output x_train, y_train, so that both could be fed in the -
DataLoader(x_train, y_train, batchsize=N) to create the full data_set

[DhairyaLGandhi]

Quite possibly. Its hard to say without knowing how your dataset is structured. The function you mention assumes that the images and labels exist together in a directory, and every directory points to a different class. If you write a function that can generate valid paths to images and their corresponding labels, then you can use load_img too. It might be useful to see how that's written anyway, it's quite easy.

[paritosh5feb]

My Data directory is structured as follows:-
data/train/images - contains images named as follows:- id.png
data/train/masks - contains the corresponding masks named as follows:- id.png
data/train.csv - contains names of all the images in data/train/images

[DhairyaLGandhi]

So you'd use the CSV to generate valid paths to the data and corresponding labels and shoot that to load_img (quite possibly you'll broadcast over a vector of strings or vector of tuple of strings) and load the images and labels like https://github.com/DhairyaLGandhi/UNet.jl/blob/2b8f2393a8bf9895f69bb8493fbae84d7c9f9c35/src/dataloader.jl#L7

You can copy this function and replace this line with one that doesn't append the .tif extension too

[paritosh5feb]

Well, would this function be correct, if I want to load all the images at once and then feed x_train, y_train to DataLoader function:-

function load_data(base_dir, n)
    img_dir = "/images/"
    mask_dir = "/masks/"
    x = zeros(Float32, rsize..., 1, n) # []
    y = zeros(Float32, rsize..., 1, n) # []
    for i in train_df[!, "id"]
        img = load(joinpath(base_dir, img_dir, i))
        mask = load(joinpath(base_dir, mask_dir, i))
        #img = imresize(img, rsize...)
        #mask = imresize(mask, rsize...)
        img = channelview(img)
        mask = channelview(mask)
        #img = reshape(img, rsize..., 1)
        #mask = reshape(mask, rsize..., 1)
        x′ = @view x[:,:,:,i]
        x′ .= img
        y′ = @view x[:,:,:,i]
        y′ .= mask
      end
    x, y
end

[paritosh5feb]

So you'd use the CSV to generate valid paths to the data and corresponding labels and shoot that to load_img (quite possibly you'll broadcast over a vector of strings or vector of tuple of strings) and load the images and labels like https://github.com/DhairyaLGandhi/UNet.jl/blob/2b8f2393a8bf9895f69bb8493fbae84d7c9f9c35/src/dataloader.jl#L7

You can copy this function and replace this line with one that doesn't append the .tif extension too

But the load_img() function will load one image at a time, right. I want to load all the images at once to x_train, and all the masks to y_train. What should I do in that case.