检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Using OBS+SFS Turbo for Storage Acceleration in AI Scenarios Solution Overview Resource Planning and Costs Process Implementation Procedure FAQs
Managing SFS Turbo+OBS Storage Interworking Overview In scenarios like AI training and inference, high-performance data preprocessing, EDA, rendering, and simulation, you can use SFS Turbo file systems to speed access to your data in OBS buckets.
Configuring Auto Data Export from SFS Turbo to OBS After auto export is configured, checkpoint files periodically written to the SFS Turbo file system during training will be automatically exported to the OBS bucket for long-term storage. The asynchronous auto export does not interrupt
Configuring Network Passthrough Between ModelArts and SFS Turbo Creating an Agency to Authorize ModelArts to Use SFS Turbo Log in to the IAM console as the IAM administrator. In the navigation pane on the left, choose Permissions > Policies/Roles. Configure a custom policy for calling
Configuring the SFS Turbo Data Eviction Policy After an OBS bucket is added as the storage backend of an SFS Turbo HPC file system, you are advised to configure a cold data eviction duration. Once configured, SFS Turbo will automatically delete files that have not been accessed within
Uploading Data to OBS and Preloading the Data to SFS Turbo Uploading Data to OBS An OBS bucket has been created by referring to Creating a Bucket. obsutil has been installed by referring to Downloading and Installing obsutil. Visit the ImageNet official website at http://image-net.org
Configuring SFS Turbo and OBS Interworking SFS Turbo HPC file systems can access objects stored in OBS buckets seamlessly. You can specify an SFS Turbo interworking directory and associate it with an OBS bucket. Log in to the SFS console. In the left navigation pane, choose SFS Turbo
SFS Turbo Security Best Practices SFS Turbo provides scalable, high-performance file storage that can be used for AI training, AI generated content (AIGC), autonomous driving, rendering, EDA simulation, and enterprise NAS applications.
Creating a Training Job Create a ModelArts training job based on the SFS Turbo shared file storage. Log in to the ModelArts console. In the navigation pane, choose Training Management > Training Jobs. Click Create Training Job in the upper right corner. On the displayed page, set
When the storage read/write bandwidth no longer meets the AI training needs, for example, checkpoint saves and loads take a longer time or loading datasets slows down the training, you can expand the file system performance to reduce the data loading time.
Parent topic: Using OBS+SFS Turbo for Storage Acceleration in AI Scenarios
Training Uploading Data to OBS and Preloading the Data to SFS Turbo Creating a Training Job Parent topic: Implementation Procedure
Parent topic: Using OBS+SFS Turbo for Storage Acceleration in AI Scenarios
The powerful compute, storage, and network performance of the AI infrastructure ensure the balanced development of AI compute power.
Parent topic: Using OBS+SFS Turbo for Storage Acceleration in AI Scenarios
Basic Configurations Configuring Network Passthrough Between ModelArts and SFS Turbo Configuring SFS Turbo and OBS Interworking Configuring Auto Data Export from SFS Turbo to OBS Configuring the SFS Turbo Data Eviction Policy Parent topic: Implementation Procedure
Creating Resources This best practice uses a VPC, an SFS Turbo HPC file system, an OBS bucket, and a ModelArts resource pool. To achieve optimal acceleration performance, you are advised to select the same region and AZ for the SFS Turbo HPC file system and ModelArts resource pool
Implementation Procedure Creating Resources Basic Configurations Training Routine O&M Parent topic: Using OBS+SFS Turbo for Storage Acceleration in AI Scenarios
training, autonomous driving, EDA simulation, rendering, enterprise NAS, and HPC web applications AI training, autonomous driving, EDA simulation, rendering, enterprise NAS, and HPC web applications Large-scale AI training, AI models, and AI generated content Large-scale AI training
