Integrate with TensorFlow.jl

[davidanthoff]

If possible.

[oxinabox]

I'm confused as to what this would even mean.

[davidanthoff]

I think I had the idea that TensorFlow probably needs some data input, and that it would be nice if it could consume an iterable table. But, I have no idea how TensorFlow works, so this might indeed be a nonsense idea :)

[oxinabox]

The main way to get data into TensorFlow is
passing it in via run. where you loadup your data in julia, then pass it as an argument to the tensorflow run.
Which works just fine I assume already.
Eg run(sess, Ycancerrisk, Dict(Xage=df[:age], Xoccupation=df[:occupation]).
This is the only way I've ever loaded data into TensorFlow myself.

The other way is using various things like readers
so that the tensorflow C backend loads the data directly.
I'm not sure how queues fit into the picture.
It might be that there is a place for IterableTables.jl in integrating with queues.

[davidanthoff]

I guess one thing that would work is that one can pass a table directly as the third argument, instead of the Dict, and then it will use the data from the columns by name. For example with your line it would be

run(sess, Ycancerrisk, df)

And the df variable could be any iterable table that has a column Xageand Xoccupation. Would that be useful?

It would also mean that you could pass in a Query.jl result directly, without materializing it first:

q = @from i in df begin
    @select {Xage=i.age, XOccupation=i.occupation}
end
run(sess, Ycancerrisk, q)

We could also enable the piping syntax that I showed at juliacon, so that things like this would work:

df |> run(sess, Ycancerrisk)

df |>
@query(i, begin
    @select {Xage=i.age, XOccupation=i.occupation}
end) |>
run(sess, Ycancerrisk)

I have no idea how useful any of this would be :)

[oxinabox]

And the df variable could be any iterable table that has a column Xageand Xoccupation. Would that be useful?

Maybe,
It would basically come for free, if the feed_dict (eg Dict(Xage=df[:age], Xoccupation=df[:occupation])),
was changed from Associative{Tensor, Any} to be allowed to be Associative{Symbol, Any}.
Which requires Symbols to be resolved to the Tensor nodes they represent.
Which can be done right now, generally.
I have had some general thoughts in this direction about revamping run.
(Probably also with a macro form)

It would also mean that you could pass in a Query.jl result directly, without materializing it first:

If you mean delaying computation via lazy evaluation then it is not possible (without substantial additional magic).
Not too deep behind run is the data collected so that it can be sent over FFI to the tensorflow C/C++ backend.

[davidanthoff]

Ah, I had not looked carefully enough at the Dict there. Another option would be to pass in an iterable table, and the day a list of pairs that map tensor objects to symbols that refer to the columns. Not a huge improvement, but it would save a few characters if typing, I guess.