Understanding Vail Cloud Storage
The Vail application uses cloud storage by seamlessly integrating with various cloud services, allowing you to manage and move data efficiently. The Vail application uses predefined policies and lifecycle rules to automate data migration, replication, and tiering within the cloud. This allows data to be optimally stored, balancing cost and accessibility.
The Vail application provides a unified management interface that allows you to oversee and control your data across different cloud environments. Data is stored in Vail buckets, which can link to cloud storage, enabling flexible and scalable data management. The Vail software automates the process of moving data to different cloud storage tiers based on access frequency and other criteria, which allows for efficient use of cloud resources and cost savings.
Additionally, the Vail application supports object storage within the cloud, making it easy to store and retrieve large volumes of unstructured data. The integration with cloud storage services also enhances data durability and availability, leveraging the cloud's inherent redundancy and disaster recovery capabilities.
The Vail application supports bi-directional synchronization with cloud storage, allowing data to be synchronized between local BlackPearl buckets and cloud buckets. Any data placed in one location is automatically synchronized to the other location, facilitating seamless data access and management across different storage environments. This can be useful for distributed workflows where data needs to be accessed and processed in multiple locations.
The Vail application provides robust security measures to protect data during transfer to cloud targets. All network traffic between Vail nodes and cloud resources is secured using HTTPS with TLS encryption. This ensures that data is protected from unauthorized access during transit.
Vail Application with AWS Cloud Storage
The information below describes the features provided by Amazon AWS cloud storage when used with the Vail application.
| • | Bucket synchronization allows for seamless synchronization between AWS S3 buckets and Vail buckets. Synchronization occurs bi-directionally. This facilitates distributed and accelerated data ingestion by synchronizing data placed in an AWS bucket to a local Vail bucket and vice versa and allows you to apply AWS services to local data without permanently storing it in the cloud. |
| • | The Vail application uses the multipart upload capabilities of AWS S3, which allows large objects to be uploaded in parts, with each part tracked to resume interrupted transfers seamlessly. |
| • | AWS S3 supports immutable objects, preventing deletion during a specified retention period. The Vail application uses object locking so that objects stored in AWS S3 cloud storage comply with specified retention policies. |
| • | AWS IAM policies are used for secure access and permissions management. AWS credentials (access keys and IAM roles) are used to authenticate and authorize access to S3 buckets. |
| • | AWS S3 versioning is required for AWS buckets used with Vail, so that multiple versions of objects are maintained. |
| • | AWS cloud storage supports creating lifecycle rules to transition objects between storage classes, such as moving infrequently accessed data to GLACIER or DEEP_ARCHIVE. |
| • | Data security features include encrypting data using AWS Key Management Service (KMS) for secure key management, and using HTTPS and V4 authentication ensure data is protected in transit. |
Vail Application with Microsoft Azure Cloud Storage Features
The information below describes the features provided by Microsoft Azure cloud storage when used with the Vail application.
| • | Data stored in Azure cloud pools is treated as native Azure objects, maintaining compatibility and allowing seamless access through Azure storage services. |
| • | Azure cloud storage uses Azure block blobs for multipart uploads, determining the method based on object size. Multipart uploads on Azure do not provide a unique identifier. Part IDs must include a unique value in addition to the part number to prevent simultaneous uploads from interfering with each other. |
| • | Vail object locking works with Azure immutable objects, allowing explicit control and expiration settings. The Vail application does not use Azure immutable storage settings directly, it manages immutable objects using special clone deletion processing. |
| • | The Vail application uses the storage container shared secret key for authentication. Credentials must include permissions for deleting blobs and blob versions using Azure RBAC. |
| • | Azure storage does not handle slashes and backslashes as ordinary characters, converting backslashes to slashes when writing blobs. Leading slashes and repeated slashes are ignored or compressed to a single slash respectively. The Vail application adjusts its behavior to accommodate these Azure storage constraints. |
| Note: | Azure storage restricts user metadata keys to C# naming conventions, and unsupported characters in user metadata are not copied to Azure storage. |
Vail Application with Google Cloud Storage
| • | Vail maps AWS storage classes to Google Cloud Storage tiers, but note that GLACIER and DEEP_ARCHIVE storage classes are not supported with Google Cloud Storage. |
| • | The Vail application uses role-based authentication to connect to Google cloud storage. |
| • | Google cloud storage supports multipart uploads using Google SDK, which simplifies and optimizes large uploads, using resumable upload to track the progress. |
| • | Vail object locking works with immutable objects, allowing explicit control and expiration settings. The Vail application recognizes and handles locked objects, but does not use Google immutable storage settings directly. |
| • | Data stored by the Vail application in Google Cloud storage is stored in packs, with each pack assigned a unique identifier. These packs are then listed in the pack list for the corresponding object version in the Vail application. The version ID and ETag are used to validate the data integrity of an object before it is retrieved and reconstructed. |