diff --git a/dev/example/index.html b/dev/example/index.html index 7e464b0..f5ffd84 100644 --- a/dev/example/index.html +++ b/dev/example/index.html @@ -72,4 +72,4 @@ cp(io, io2) # remove the blob -rm(io) +rm(io) diff --git a/dev/index.html b/dev/index.html index 5e27a50..4ecc3cf 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,2 +1,2 @@ -AzStorage · AzStorage
Edit on GitHub

AzStorage

AzStorage is a Julia API for Azure storage. AzStorage provides methods for interacting with Azure containers and blobs. In order to obtain reasonable through-put, I/O is threaded via OpenMP. In the case of writing, Azure block-blobs are used to help organize the threaded I/O.

+AzStorage · AzStorage
Edit on GitHub

AzStorage

AzStorage is a Julia API for Azure storage. AzStorage provides methods for interacting with Azure containers and blobs. In order to obtain reasonable through-put, I/O is threaded via OpenMP. In the case of writing, Azure block-blobs are used to help organize the threaded I/O.

diff --git a/dev/reference/index.html b/dev/reference/index.html index ffb41a6..1101159 100644 --- a/dev/reference/index.html +++ b/dev/reference/index.html @@ -1,18 +1,18 @@ -Reference · AzStorage
Edit on GitHub

Reference

Construct an Azure Container

AzStorage.AzContainer — Type
container = AzContainer("containername"; storageaccount="myacccount", kwargs...)

container is a handle to a new or existing Azure container in the myaccount sorage account. The storage account must already exist.

Additional keyword arguments

  • session=AzSession(;lazy=false,scope=offline_access+openid+https://storage.azure.com/user_impersonation) user credentials (see AzSessions.jl package).
  • nthreads=Sys.CPU_THREADS number of system threads that OpenMP will use to thread I/O.
  • connect_timeout=30 client-side timeout for connecting to the server.
  • read_timeout=10 client-side timeout for receiving the first byte from the server.
  • nretry=10 number of retries to the Azure service (when Azure throws a retryable error) before throwing an error.
  • verbose=0 verbosity flag passed to libcurl.

Notes

The container name can container "/"'s. If this is the case, then the string preceding the first "/" will be the container name, and the string that remains will be pre-pended to the blob names. This allows Azure to present blobs in a pseudo-directory structure.

source

Container methods

AzStorage.containers — Function
containers(;storageaccount="mystorageaccount", session=AzSession(;lazy=false, scope=__OAUTH_SCOPE), nretry=5, verbose=0, connect_timeout=30, read_timeout=10)

list all containers in a given storage account.

source
Base.Filesystem.cp — Function
cp(from..., to...)

copy a blob to a local file, a local file to a blob, or a blob to a blob.

Examples

local file to blob

cp("localfile.txt", AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"; buffersize=2_000_000_000, show_progress=false)

blob to local file

cp(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt", "localfile.txt", buffersize=2_000_000_000, show_progress=false)

buffersize is the memory buffer size (in bytes) used in the copy algorithm, and defaults to 2_000_000_000 bytes (2GB). Note that half of this memory is used to buffer reads, and the other half is used to buffer writes. Set show_progress=true to display a progress bar for the copy operation.

blob to blob

cp(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_in.txt", AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_out.txt")
source
cp(from, to)

copy a blob to a local file, a local file to a blob, or a blob to a blob.

Examples

local file to blob

cp("localfile.txt", open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"); buffersize=2_000_000_000, show_progress=false)

blob to local file

cp(open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"), "localfile.txt"; buffersize=2_000_000_000, show_progress=false)

buffersize is the memory buffer size (in bytes) used in the copy algorithm, and defaults to 2_000_000_000 bytes (2GB). Note that half of this memory is used to buffer reads, and the other half is used to buffer writes. Set show_progress=true to display a progress bar for the copy operation.

blob to blob

cp(open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_in.txt"), open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_out.txt"))
source
cp(container_src, container_dst)

copy container_src::AzContainer and its blobs to container_dst::AzContainer.

source
Base.Filesystem.mkpath — Function
mkpath(container)

create an Azure container from the handle container::AzContainer. If the container already exists, then this is a no-op.

source

Blob methods

Base.Filesystem.cp — Method
cp(from..., to...)

copy a blob to a local file, a local file to a blob, or a blob to a blob.

Examples

local file to blob

cp("localfile.txt", AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"; buffersize=2_000_000_000, show_progress=false)

blob to local file

cp(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt", "localfile.txt", buffersize=2_000_000_000, show_progress=false)

buffersize is the memory buffer size (in bytes) used in the copy algorithm, and defaults to 2_000_000_000 bytes (2GB). Note that half of this memory is used to buffer reads, and the other half is used to buffer writes. Set show_progress=true to display a progress bar for the copy operation.

blob to blob

cp(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_in.txt", AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_out.txt")
source
Base.Filesystem.cp — Method
cp(from, to)

copy a blob to a local file, a local file to a blob, or a blob to a blob.

Examples

local file to blob

cp("localfile.txt", open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"); buffersize=2_000_000_000, show_progress=false)

blob to local file

cp(open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"), "localfile.txt"; buffersize=2_000_000_000, show_progress=false)

buffersize is the memory buffer size (in bytes) used in the copy algorithm, and defaults to 2_000_000_000 bytes (2GB). Note that half of this memory is used to buffer reads, and the other half is used to buffer writes. Set show_progress=true to display a progress bar for the copy operation.

blob to blob

cp(open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_in.txt"), open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_out.txt"))
source
Serialization.deserialize — Function
deserialize(container, "blobname")

read and deserialize from a blob "blobname" in container::AzContainer.

Example

io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.bin")
+Reference · AzStorage
Edit on GitHub
Reference
Construct an Azure Container
AzStorage.AzContainer — Type
container = AzContainer("containername"; storageaccount="myacccount", kwargs...)
container is a handle to a new or existing Azure container in the myaccount sorage account. The storage account must already exist.
Additional keyword arguments
  • session=AzSession(;lazy=false,scope=offline_access+openid+https://storage.azure.com/user_impersonation) user credentials (see AzSessions.jl package).
  • nthreads=Sys.CPU_THREADS number of system threads that OpenMP will use to thread I/O.
  • connect_timeout=30 client-side timeout for connecting to the server.
  • read_timeout=10 client-side timeout for receiving the first byte from the server.
  • nretry=10 number of retries to the Azure service (when Azure throws a retryable error) before throwing an error.
  • verbose=0 verbosity flag passed to libcurl.
Notes
The container name can container "/"'s.  If this is the case, then the string preceding the first "/" will be the container name, and the string that remains will be pre-pended to the blob names.  This allows Azure to present blobs in a pseudo-directory structure.
source
Container methods
AzStorage.containers — Function
containers(;storageaccount="mystorageaccount", session=AzSession(;lazy=false, scope=__OAUTH_SCOPE), nretry=5, verbose=0, connect_timeout=30, read_timeout=10)
list all containers in a given storage account.
source
Base.Filesystem.cp — Function
cp(from..., to...)
copy a blob to a local file, a local file to a blob, or a blob to a blob.
Examples
local file to blob
cp("localfile.txt", AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"; buffersize=2_000_000_000, show_progress=false)
blob to local file
cp(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt", "localfile.txt", buffersize=2_000_000_000, show_progress=false)
buffersize is the memory buffer size (in bytes) used in the copy algorithm, and defaults to 2_000_000_000 bytes (2GB).  Note that half of this memory is used to buffer reads, and the other half is used to buffer writes. Set show_progress=true to display a progress bar for the copy operation.
blob to blob
cp(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_in.txt", AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_out.txt")
source
cp(from, to)
copy a blob to a local file, a local file to a blob, or a blob to a blob.
Examples
local file to blob
cp("localfile.txt", open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"); buffersize=2_000_000_000, show_progress=false)
blob to local file
cp(open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"), "localfile.txt"; buffersize=2_000_000_000, show_progress=false)
buffersize is the memory buffer size (in bytes) used in the copy algorithm, and defaults to 2_000_000_000 bytes (2GB).  Note that half of this memory is used to buffer reads, and the other half is used to buffer writes. Set show_progress=true to display a progress bar for the copy operation.
blob to blob
cp(open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_in.txt"), open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_out.txt"))
source
cp(container_src, container_dst)
copy container_src::AzContainer and its blobs to container_dst::AzContainer.
source
Base.Filesystem.mkpath — Function
mkpath(container)
create an Azure container from the handle container::AzContainer.  If the container already exists, then this is a no-op.
source
Blob methods
Base.Filesystem.cp — Method
cp(from..., to...)
copy a blob to a local file, a local file to a blob, or a blob to a blob.
Examples
local file to blob
cp("localfile.txt", AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"; buffersize=2_000_000_000, show_progress=false)
blob to local file
cp(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt", "localfile.txt", buffersize=2_000_000_000, show_progress=false)
buffersize is the memory buffer size (in bytes) used in the copy algorithm, and defaults to 2_000_000_000 bytes (2GB).  Note that half of this memory is used to buffer reads, and the other half is used to buffer writes. Set show_progress=true to display a progress bar for the copy operation.
blob to blob
cp(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_in.txt", AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_out.txt")
source
Base.Filesystem.cp — Method
cp(from, to)
copy a blob to a local file, a local file to a blob, or a blob to a blob.
Examples
local file to blob
cp("localfile.txt", open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"); buffersize=2_000_000_000, show_progress=false)
blob to local file
cp(open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob.txt"), "localfile.txt"; buffersize=2_000_000_000, show_progress=false)
buffersize is the memory buffer size (in bytes) used in the copy algorithm, and defaults to 2_000_000_000 bytes (2GB).  Note that half of this memory is used to buffer reads, and the other half is used to buffer writes. Set show_progress=true to display a progress bar for the copy operation.
blob to blob
cp(open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_in.txt"), open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "remoteblob_out.txt"))
source
Serialization.deserialize — Function
deserialize(container, "blobname")
read and deserialize from a blob "blobname" in container::AzContainer.
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.bin")
 serialize(io, (rand(10),rand(20)))
-a,b = deserialize(io)
source
deserialize(object)
read and deserialize a blob object::AzObject.  See deserialize(container, "blobname").
source
Base.filesize — Function
filesize(container, "blobname")
Returns the size of the blob "blobname" that is in container::AzContainer
source
filesize(object::AzObject)
Returns the size of the blob corresponding to object::AzObject
source
Base.Filesystem.isfile — Function
isfile(container, "blobname")
Returns true if the blob "object" exists in container::AzContainer.
source
isfile(object::AzObject)
Returns true if the blob corresponding to object exists.
source
Base.Filesystem.joinpath — Function
joinpath(container, blobname) -> AzObject
Create a handle to an Azure blob with the name blobname::String in the Azure storage container: container::AzContainer.
Example:
io = joinpath(AzContainer("mycontainer"; storageaccount="myaccount"), "foo.bin")
-write(io, rand(10))
source
Base.open — Function
open(container, blobname) -> AzObject
Create a handle to an Azure blob with the name blobname::String in the Azure storage container: container::AzContainer.
Example:
io = open(AzContainer("mycontainer"; storageaccount="myaccount"), "foo.bin")
-write(io, rand(10))
source
open(object::AzObject[, mode="w+"]) -> object
This is an identity operation to support compatability with POSIX I/O. It allows for the following equivalence which can be useful in building methods that are agnostic to storage systems:
io = open(joinpath(AzContainer("foo";storageaccount="bar"), "bar")) # Azure blob sorage
+a,b = deserialize(io)
source
deserialize(object)
read and deserialize a blob object::AzObject.  See deserialize(container, "blobname").
source
Base.filesize — Function
filesize(container, "blobname")
Returns the size of the blob "blobname" that is in container::AzContainer
source
filesize(object::AzObject)
Returns the size of the blob corresponding to object::AzObject
source
Base.Filesystem.isfile — Function
isfile(container, "blobname")
Returns true if the blob "object" exists in container::AzContainer.
source
isfile(object::AzObject)
Returns true if the blob corresponding to object exists.
source
Base.Filesystem.joinpath — Function
joinpath(container, blobname) -> AzObject
Create a handle to an Azure blob with the name blobname::String in the Azure storage container: container::AzContainer.
Example:
io = joinpath(AzContainer("mycontainer"; storageaccount="myaccount"), "foo.bin")
+write(io, rand(10))
source
Base.open — Function
open(container, blobname) -> AzObject
Create a handle to an Azure blob with the name blobname::String in the Azure storage container: container::AzContainer.
Example:
io = open(AzContainer("mycontainer"; storageaccount="myaccount"), "foo.bin")
+write(io, rand(10))
source
open(object::AzObject[, mode="w+"]) -> object
This is an identity operation to support compatability with POSIX I/O. It allows for the following equivalence which can be useful in building methods that are agnostic to storage systems:
io = open(joinpath(AzContainer("foo";storageaccount="bar"), "bar")) # Azure blob sorage
 io = open(joinpath("foo", "bar")) # POSIX
 write(io, "hello")
-close(io)
Please note that the 'mode' is for compatability with Base.open and does not have any effect due to the how Azure blob storage works.
source
Base.read — Function
read(container, "blobname", String)
returns the contents of the blob "blobname" in container::AzContainer as a string.
source
read(object, String)
read a string from object::AzObject.
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.txt")
-read(io, String)
source
Base.read! — Function
read!(container, "blobname", data; offset=0)
read from the blob "blobname" in container::AzContainer into data::DenseArray, and where offset specifies a number of bytes in the blob to skip before reading.  This method returns data.  For example,
data = read!(AzContainer("foo";storageaccount="bar"), "baz.bin", Vector{Float32}(undef,10))
source
read!(object, x; offset=0) -> x
read data from object::AzObject into x::DenseArray, and return x.  offset is an integer that can be used to specify the first byte in the object to read. 
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.txt")
-read!(io, Vector{Float64}(undef, 10))
source
DelimitedFiles.readdlm — Function
readdlm(container, "blobname", args...; options...)
Read the data in a delimited blob with the name blobname in container container::AzContainer
source
readdlm(io:AzObject, args...; options...)
return the parsed delimited blob from the io object io::AzObject
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.txt")
-data = readdlm(io)
source
Base.Filesystem.rm — Method
rm(object::AzObject; force=false)
remove the blob corresponding to object::AzObject.  Note that the force keyword argument does not change the behavior of this method.  It is included to match Julia's Base.rm method, allowing the calling code to work on both POSIX and Azure storage.
source
Serialization.serialize — Function
serialize(container, "blobname", data)
Serialize and write data to a blob with the name blobname in container::AzContainer.
Example
container = AzContainer("mycontainer";storageaccount="mystorageaccount")
+close(io)
Please note that the 'mode' is for compatability with Base.open and does not have any effect due to the how Azure blob storage works.
source
Base.read — Function
read(container, "blobname", String)
returns the contents of the blob "blobname" in container::AzContainer as a string.
source
read(object, String)
read a string from object::AzObject.
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.txt")
+read(io, String)
source
Base.read! — Function
read!(container, "blobname", data; offset=0)
read from the blob "blobname" in container::AzContainer into data::DenseArray, and where offset specifies a number of bytes in the blob to skip before reading.  This method returns data.  For example,
data = read!(AzContainer("foo";storageaccount="bar"), "baz.bin", Vector{Float32}(undef,10))
source
read!(object, x; offset=0) -> x
read data from object::AzObject into x::DenseArray, and return x.  offset is an integer that can be used to specify the first byte in the object to read. 
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.txt")
+read!(io, Vector{Float64}(undef, 10))
source
DelimitedFiles.readdlm — Function
readdlm(container, "blobname", args...; options...)
Read the data in a delimited blob with the name blobname in container container::AzContainer
source
readdlm(io:AzObject, args...; options...)
return the parsed delimited blob from the io object io::AzObject
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.txt")
+data = readdlm(io)
source
Base.Filesystem.rm — Method
rm(object::AzObject; force=false)
remove the blob corresponding to object::AzObject.  Note that the force keyword argument does not change the behavior of this method.  It is included to match Julia's Base.rm method, allowing the calling code to work on both POSIX and Azure storage.
source
Serialization.serialize — Function
serialize(container, "blobname", data)
Serialize and write data to a blob with the name blobname in container::AzContainer.
Example
container = AzContainer("mycontainer";storageaccount="mystorageaccount")
 serialize(container, "foo.bin", (rand(10),rand(20)))
-a,b = deserialize(io)
source
serialize(io::AzObject, data)
Serialize and write data to io::AzObject.  See serialize(conainer, blobname, data).
source
Base.write — Function
write(container, "blobname", data::AbstractString; contenttype="text/plain")
Write the string data to a blob with name blobname in container::AzContainer. Optionally, one can specify the content-type of the blob using the contenttype keyword argument. For example: content-type="text/plain",content-type="applicaton/json", etc..
source
write(container, "blobname", data::StridedArray)
Write the array data to a blob with the name blobname in container::AzContainer.
source
write(io::AzObject, data)
write data to io::AzObject.
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.bin")
+a,b = deserialize(io)
source
serialize(io::AzObject, data)
Serialize and write data to io::AzObject.  See serialize(conainer, blobname, data).
source
Base.write — Function
write(container, "blobname", data::AbstractString; contenttype="text/plain")
Write the string data to a blob with name blobname in container::AzContainer. Optionally, one can specify the content-type of the blob using the contenttype keyword argument. For example: content-type="text/plain",content-type="applicaton/json", etc..
source
write(container, "blobname", data::StridedArray)
Write the array data to a blob with the name blobname in container::AzContainer.
source
write(io::AzObject, data)
write data to io::AzObject.
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.bin")
 write(io, rand(10))
-x = read!(io, zeros(10))
source
DelimitedFiles.writedlm — Function
writedlm(container, "blobname", data, args...; options...)
Write the array data to a delimited blob with the name blobname in container container::AzContainer
source
writedlm(io:AzObject, data, args...; options...)
write the array data to io::AzObject
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.txt")
+x = read!(io, zeros(10))
source
DelimitedFiles.writedlm — Function
writedlm(container, "blobname", data, args...; options...)
Write the array data to a delimited blob with the name blobname in container container::AzContainer
source
writedlm(io:AzObject, data, args...; options...)
write the array data to io::AzObject
Example
io = open(AzContainer("mycontainer";storageaccount="mystorageaccount"), "foo.txt")
 writedlm(io, rand(10,10))
-x = readdlm(io)
source

+x = readdlm(io)
source
diff --git a/dev/search/index.html b/dev/search/index.html index b95a4f6..fa77beb 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · AzStorage

Loading search...

    +Search · AzStorage

    Loading search...