Open topic with navigation

 

StorCycle Quick Reference

Use the information in this document section to quickly locate information that describes the core functionality and features of the StorCycle solution, as well as workflow examples for typical use case scenarios.

User Role Description

User Roles determine the level of permissions and access within the StorCycle solution

Administrator

Administrator has unrestricted access to all storage locations, projects, configuration and reporting.

Crypto Officer

Administrator privileges plus Crypto Officer role. Can configure encryption options and enable the encryption key.

Storage Manager

Users who can create Scan, Migrate / Store, and Restore jobs.

• Cannot configure settings or storage locations.
• Can restrict access to data with domain integration and group assignments.

Restore User

Domain users who are able to restore data via HTML links from the StorCycle user interface. Administrators can restrict Restore User access to data with domain integration and group assignments.

Storage Locations

Storage Locations are where data will be migrated from and where data will be migrated to.

Source Storage

Source data will be scanned and archived. Supported sources include NAS and S3.

Note: S3 sources can only be migrated to BlackPearl systems, and restore to NAS sources.

Target Storage

Target storage locations where data is archived to. Supported targets include Spectra BlackPearl systems (NAS and Nearline), NAS, S3, AWS, AWS Glacier, and MS Azure.

Projects and Jobs

Each time a Project (Migrate / Store, Scan, or Restore) runs, it creates a job increment. Migrate / Store and Scan jobs can be manually started or set to run on a schedule. Restores can be manually started or set to run at a future date.

Migrate / Store Project
• A Migrate / Store Project is configured to move data from a source to one or many targets.
• Migrate / Store projects can be configured with a reoccurring schedule.
• Age and Size filters can be applied to a Project.
• Each time a Migrate / Store project runs, it creates a Migrate / Store Job (MigrateJob-1, MigrateJob-2, etc.).

Scan Project
• A Scan Project is configured to scan source storage locations.
• Scan Projects can be configured with a recurring schedule.
• Each time a Scan Project runs, it will create a Scan Job (ScanJob-1, ScanJob-2, etc).

Restore Project
• A Restore Project is configured to restore a specific Migrate / Store Job.
• Restore projects can be configured to run now or run at a future date or time.

Job
• Jobs refer to any Migrate / Store, Restore, or Scan project / job, as well as other job types, including Retention Policy Delete, Version Delete, Bucket Ingest, and Database Backup.
• Every Job is an instance of a running project.

StorCycle Features

Features

Description

Applies To

AD/LDAP Server

If enabled, allows users to login to StorCycle using their domain credentials. Required if using Restore Users. AD/LDAP configurations are also used to query the domain to determine Domain Group membership.

Users

Domain Groups

A single domain group can be assigned to Source storage locations and restrict domain users by group membership. Domain groups are independent of domain file permissions and are only used to prevent or allow access to StorCycle jobs.

Sources and Users

Departments

Departments can be assigned to Source and Target Storage locations and be used to track cost and storage savings. Departments have no relation to users.

Sources and Targets

Encryption

StorCycle Single Key encryption is available for all Target Storage locations. When enabled, all data sent to the target is encrypted.

Note: Included as a standard feature in Data Center and Enterprise License levels, available for Administrator and Work Group licenses.

Targets

Versioning

Versioning can be assigned to any new NAS Source Storage location and is then enabled for all projects originating from the source.

Note: The versioning feature enables users to implement targeted backup workflows but does not replace more traditional backup methods. By default, versioning only allows the user to protect file objects.

NAS Sources

SMTP

A Simple Mail Transfer Protocol (SMTP) Server can be configured to enable automatic email alerts for data restores.

Restores and Users

Linked Instances
• An instance of StorCycle can link to one or many other instances of StorCycle on the same network.
• From a single StorCycle instance, users can search for files and projects across other linked instances.

StorCycle

Ransomware Protection
• When using a Spectra BlackPearl NAS target, StorCycle maintains the BlackPearl volume in read-only mode.
• StorCycle automatically takes snapshots of a BlackPearl volume after every migrate job.

BlackPearl NAS Target

REST API

Most actions which can be performed in the user interface are available as API commands. Use REST API automation and custom scripting.

StorCycle

Throttling

Control bandwidth allocation for peak and non-peak times for migrations.

Migrate / Store Jobs

Filters

Use Filters to select what type of data to migrate by size, age, type, include / exclude, etc.

Migrate / Store Jobs

Tagging

Apply metadata tags to migrate / store jobs to enhance search capabilities.

Migrate / Store Jobs

Retention Policy

Retention Policies can be assigned to Target Storage locations to expire data after a specified time has passed.

Targets + Migrate / Store Jobs

Packing

Use TAR or ZIP packing with Target locations to maximize the efficiency of migrations.

Targets

Transparent Access

Replace original source files with HTML or Symbolic links to allow users transparent access to data.

Sources and Migrate / Store Jobs and Users

Bucket Ingest

Ingest BlackPearl Target buckets and integrate existing objects into the StorCycle archive database.

BlackPearl Target and Restores

Scheduling

Migrate, Scan, and Restore jobs can be scheduled to run at a future date and time. Migrate and Scan jobs can be configured to run on a recurring schedule.

Migrate / Store Jobs and Scan Jobs and Restore Jobs

Database Backup

Database backups can be manually initiated or set on a recurring schedule.

StorCycle

Migrate / Store Filters

Use filters to ensure that a Migrate / Store job only includes the data you wish to migrate. Filter out files based on object size, age, or type. You can also include only specific file types or exclude specific directories.

Object Size

Object Size filters can be applied to Migrate / Store jobs, allowing you to migrate all files which are above a size threshold.

• '>1GB' migrates all files which are larger than 1 GB

Object Age

Object Age filters can be applied to Migrate / Store jobs, allowing you to migrate all files which are older than the specified age threshold.

• '>2 Years' migrates all files which are older than 2 years

Include Type

Include Type filters migrates the specified file type.

• '.PNG' migrates PNG file types

Exclude Type

Exclude Type filters omits the specified file type from being migrated.

• '.PNG' migrates any file that is not a PNG file type

Include Directory

Include Directory allows you to include a specific directory or all directories with the specified name.

• '\\production' includes the 'production' folder from the working root directory
• 'production' includes every folder named production, regardless of where it exists in directory structure

Exclude Directory

Exclude Directory allows you to exclude a specific directory or all directories with the specified name.

• '\\production' excludes the 'production' folder from the working root directory
• 'production' excludes every folder named production, regardless of where it exists in directory structure

Windows Environment: Recommended File and System Permissions

In a Windows environment, special attention is required to configure the StorCycle service account, target storage location, and source location permissions.

StorCycle Server

A Service Account must be used to run the StorCycle service.

• Local Administrator (Recommended) or Domain Administrator
• Custom Service Account (Not Recommended): 'Full Control - This Folder, Files, and Subfolders'

Source Storage
• If using a Local or Domain Administrator StorCycle Service account, no special configuration is required
• If using a Custom Owner service account - Source, Target, and Service Account permissions must match to avoid metadata errors

BlackPearl NAS or Windows NAS Target

StorCycle managed shares created specifically for use by StorCycle.

• Local Administrators - Full Control - This Folder, Files, and Subfolders
• SYSTEM - Full Control - This Folder, Files, and Subfolders
• Remove Domain Administrator if using Local Administrator (and any other group)

S3 / Cloud Target

No special configuration required.

• S3 / Cloud Targets include: AWS, AWS Glacier, MS Azure Archive, Standard S3 targets

Spectra's BlackPearl Platform

BlackPearl Platform

Spectra's BlackPearl platform can be configured in multiple combinations to support NAS, S3, and a Nearline gateway. The following workflow configurations use the base BlackPearl images with a NAS, Nearline, and S3 icon, to indicate how the system is configured.

BlackPearl NAS

• Secondary NAS Storage
• Support for Ransomware protection

BlackPearl Nearline Gateway

• Gateway to Tape archive
• Object Storage Disk

BlackPearl S3

• BlackPearl with S3 support

 

BlackPearl Nearline Gateway + BlackPearl NAS

• A BlackPearl platform can be configured to support both NAS and a Nearline gateway

StorCycle + BlackPearl Platform + Tape Library

• A single BlackPearl can be configured for both NAS and Nearline Gateway
• BlackPearl NAS can serve as the Source Location
• Nearline Gateway can serve as Target Location
• Nearline Gateway moves data to tape library

Basic Workflow

A standard StorCycle workflow has the StorCycle solution installed on a virtual machine or dedicated server. Primary storage will mount to the server/VM via CIFS or NFS. Target NAS storage is also mounted on the StorCycle server. S3 Sources/Targets and BlackPearl targets will connect using S3 credentials. Users and Administrators interact with StorCycle through the StorCycle web interface, the API, or by using HTML or Symbolic Links for transparent access to data.

StorCycle can be configured to scan Source Storage locations on a schedule or when triggered. Storage Managers and Administrators can then configure Migrate / Storage jobs to move all files or just a subset of the files based on age, size, and type filters. After files are migrated, the original source files can remain on source storage, be removed, or replaced with HTML or Symbolic links to provide transparent link access to end users.

Both Files and HTML links can be restored from the user interface by restoring specific Migrate / Store jobs, portions of Migrate / Store jobs, or individual files. Restores can move files to the original Source Location or to a new location. Using StorCycle's robust search feature, users can search for files by project name, file name, or tag.

Reference Workflows

Tape + Tape (Ejected)

A single StorCycle project or job can migrate files to multiple storage targets. Using a single BlackPearl system, StorCycle can write to two BlackPearl buckets with one bucket writing a copy to tape, and the other bucket writing and ejecting the tape. A similar workflow can also be accomplished with a single bucket in BlackPearl, using BlackPearl data policies to write two copies with one copy then ejected. Spectra Logic tape libraries support both LTO and TS11xx technologies.

Tape + Disk

Different target types can be used in a single project or job to write a copy of data to a tape library and a second copy to secondary disk.

Disk + Disk

Multiple NAS Target Locations can be used in a single project or job.

Cloud + Disk

StorCycle's S3 Target Locations can be used for standard cloud storage, such as AWS, and also be used for standard S3 appliances. S3 and disk targets can be mixed on a single project or job.

Cloud + Cloud

Multiple S3 Target Locations can be used in a single project or job.

Ransomware Snapshot Protection

When using a BlackPearl NAS target, StorCycle can be configured to automatically maintain the BlackPearl volume in a read-only state. StorCycle also automatically takes volume snapshots at the end of each Migrate / Store job and File / Delete job (retention policy, version delete, etc.)

Cloud (encrypted) + BlackPearl NAS Ransomware + Tape (2 copies, 1 ejected)

StorCycle supports up to 3 Target Locations in a single Project / Job. This example uses StorCycle encryption for the cloud target, a ransomware protected copy on a BlackPearl NAS, and a single copy sent to a BlackPearl bucket which uses a dual-copy data policy (2 copies on tape, one copy ejected).

Versioning Workflow

StorCycle Versioning allows for many different types of workflows. Below is an example of StorCycle Versioning, moving the same source data to two different filters and retention policies. The weekly project is moving all data that is older than 7-days to a secondary NAS target with a 30-day retention policy. The monthly project is moving all data older than 30-days to a cloud target for long-term archive.

Data Protection Strategies

Encryption

Encryption can be enabled for any Target Location. When enabled, every file or object sent to the target will be encrypted using StorCycle controlled single key encryption.

Domain Groups

A single domain group can be applied to a source storage location. The location is then restricted to Domain Users who are members of the group and Administrators.

Versioning

Versioning is assigned to Source Locations. Once enabled, StorCycle makes a version of the source file with each migrate job. Versioning can be configured to:

• Version file on every Migrate / Store job.
• Version only files which have changed since the last job.
• Optionally maintain a certain number of versions on Target Locations.

Ransomware Protection

Ransomware Protection is delivered by using a BlackPearl NAS Target location. When enabled, StorCycle maintains read-only access on the BlackPearl NAS volume and triggers volume snapshots at the end of every job.

Retention Policy

Retention Policies can be assigned to Target Locations. For example, with a retention policy of '30' days, StorCycle deletes archived data from the targets 30 days after the completion of the archive.

 

Transparent Access

Transparent Access for end users can be provided with Symbolic Links or HTML Links. Symbolic Links are supported on NAS targets only and allow users to access the Symbolic Link from the Source location without a data restoration. HTML Links replace original source files with a HTML file that end users can use to access the StorCycle user interface for data restoration or to send a restore request to a StorCycle Administrator.

Performance Strategies

Throttling

Apply job throttling to Migrate / Store jobs to reduce performance during peak work-hours. You can set peak hour schedule and bandwidth caps.

 

Filters

Use filters on Migrate / Store jobs to only migrate specific sized files, files which meet a particular age criteria, specific file types, and other exclusion types. For example, if a single source has many files which are both very large and very small, a job could be configured to move the large files separately and then move the smaller files with packing enabled on a BlackPearl.

Packing

On targets, Packing can be enabled, which will split a Migrate / Store job into several ZIP or TAR packages, minimizing the total objects transferred and improving archive, search, and restore performance.

Scheduling

Scheduling is used to configure a job to run at a future time. This can be used to run a large scan job during off hours. Scheduling is also used to create recurring scan or Migrate / Store jobs. For example, a scan job could be scheduled to run on the weekend, and a recurring Migrate / Store job could use the scan data to move data every Monday.

Professional Services Tuning

Please work with Spectra Logic Professional Services if changes to StorCycle's configuration for performance is required.

Linked Instances

StorCycle can connect to other StorCycle instances on the same network and allow users to search the connected databases for archived files.

In the example below, StorCycle Instance #1 can make a connection to Instance #2 to allow users to search for any file which was archived by either system. When ready to restore data, users will be guided to the user interface of the instance where their data is located. The sources and targets remain independent between the two instances: Source A and Source B can only archive data to S3 / Cloud and BlackPearl NAS. StorCycle Instance #1 can also only be restored to either Source A or Source B.

When StorCycle Instance #1 connects to Instance #2, it is a one-way connection and Instance #2 will not be able to search the archived contents in Instance #1. However, Instance #2 can configure its own connection in order to be able to search for archived files of Instance #1.

StorCycle Master Diagram

The diagram below displays a fully featured workflow utilizing many of the StorCycle features and workflows available. Features in RED are optional.

Restore Users

Storage Managers

Administrator

Crypto Officer

API Integrations
• Requires AD/LDAP configuration.
• Can access HTML links.
• Can restore user interface.
• Access restricted with domain groups.
• Can configure jobs.
• Cannot configure storage locations or settings.
• Access restricted with domain groups.
• Default Administrator.
• Super user with unrestricted access.
• Full Administrator permission.
• Can configure and set encryption.
• Available OpenAPI for integrations.