Support for Azure Blob Storage as a Destination

[mohamedmeqlad99]

Feature description

I would like to suggest adding Azure Blob Storage as a supported destination in DLT. Azure Blob Storage is a widely used cloud storage solution, especially in enterprises leveraging Microsoft's Azure ecosystem. Supporting it as a destination would greatly enhance DLT's usability for users who rely on Azure for their data storage needs.

Are you a dlt user?

Yes, I use it for fun.

Use case

Use Cases

1. Enterprise Adoption: Many organizations use Azure Blob Storage as their primary data lake or archive solution.
2. Scalability: DLT users could leverage Azure Blob Storage for high-throughput and cost-effective data storage.
3. Integration: Simplifies workflows for users with hybrid environments combining Azure Blob Storage with other Azure services like Azure Data Factory or Synapse Analytics.

Proposed solution

Proposed Implementation

Destination Name:

azure_blob

Required Parameters:

• account_name: Azure Storage Account name.
• container_name: Name of the Azure Blob Storage container.
• sas_token or account_key: Credentials for authentication.
• connection_string (optional): As an alternative for authentication.

Options:

• Support for different blob types (e.g., Block, Append).
• Configurable settings for blob naming conventions.

---

Potential Challenges

1. Authentication:
   • Offering flexibility between SAS tokens, account keys, and Azure AD integration.
2. File Formats:
   • Ensuring compatibility with formats like CSV, JSON, Parquet, etc.
3. Dependency Management:
   • Using SDKs like azure-storage-blob for integration.

---

Resources

• Azure Blob Storage Documentation: Azure Blob Storage
• Python SDK for Azure Storage: azure-storage-blob

---

Additional Note

I would be happy to contribute to the implementation of this feature myself. If the team agrees that this addition aligns with DLT's roadmap, I can start working on it and submit a pull request.

---

Thank you for considering this feature. Let me know if there are any additional details or prerequisites needed to proceed!

Related issues

No response