检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Exporting Data Introduction to Exporting Data Exporting Data to a New Dataset Exporting Data to OBS
The OBS paths in the debugging code should be replaced with your OBS paths. PyTorch is used to write debugging code in this document. The process is the same for different AI frameworks. You only need to modify some parameters.
If the OBS path to the boot script is obs://bucket-name/app/code/train.py, set the code directory to /bucket-name/app/code/ when creating a job.
Table 1 Required OBS folders Folder Usage obs://test-modelarts/mpi/demo-code/ Stores the MPI boot script and training script file. obs://test-modelarts/mpi/log/ Stores training log files.
If you need to use OBS external storage mounting for custom images in real-time services, create a new directory for OBS data, for example, /obs-mount/. Otherwise, the existing files will be overwritten. You can add, view, and modify the files in the OBS mount directory.
Multi-node multi-card: If the data volume is large (1 TB training data) and the compute is high (four nodes with eight Vnt1 cards), use SFS to store data and a common OBS bucket to store code, and use distributed training.
ModelArts Standard Preparations Configuring Access Authorization for ModelArts Standard Creating and Managing a Workspace Creating an OBS Bucket for ModelArts to Store Data
environment variables Variable Description Example S3_ENDPOINT OBS endpoint - S3_VERIFY_SSL Whether to use SSL to access OBS S3_VERIFY_SSL=0 S3_USE_HTTPS Whether to use HTTPS to access OBS S3_USE_HTTPS=1 Table 5 Environment variables of the PIP source and API Gateway address Variable
# Build a local image and save to local path and OBS ma-cli image build .ma/customize_from_ubuntu_18.04_to_modelarts/Dockerfile --target .
Ensure that the OBS directory you use and ModelArts are in the same region. For details about model package requirements, see Model Package Example. Input and Output Mode The built-in image processing mode cannot be overwritten.
Exporting Data from a ModelArts Dataset Exporting Data from ModelArts to OBS Exporting Data as a New Dataset Parent topic: Preparing and Processing Data
Submitting a Training Job (New Version) Stopping a Training Job Viewing Training Logs OBS-based upload and download Local files or folders can be uploaded to OBS and files or folders can be downloaded from OBS to a local directory.
Code Directory OBS directory for storing the algorithm code. Boot File OBS directory for storing the boot file. This parameter is available only for algorithms created using a preset image. Boot Command Boot command of an algorithm created using a custom image.
Step 2 Preparing Script Files and Uploading Them to OBS Prepare the MPI boot script run_mpi.sh and training script mpi-verification.py and upload them to the obs://test-modelarts/mpi/demo-code/ folder of the OBS bucket.
The value must be a specific directory in an OBS bucket rather than the root directory of an OBS bucket. If type is set to obsfs, the parameter value must be a valid bucket name of the OBS parallel file system. (CCE does not support subdirectory mounting.)
Importing Data Introduction to Data Importing Importing Data from OBS Importing Data from DLI Importing Data from MRS Importing Data from DWS Importing Data from Local Files
Importing Data to a ModelArts Dataset Introduction to Data Importing Importing Data from OBS Importing Data from MRS to ModelArts Importing Data from Local Files Parent topic: Preparing and Processing Data
For a file that exceeds 100 MB but does not exceed 5 GB, upload the file to OBS (an object bucket or a parallel file system), and then download the file from OBS to a notebook instance. After the download is complete, the file is deleted from OBS.
Uploading Files to JupyterLab Scenarios Uploading Files from a Local Path to JupyterLab Cloning an Open-Source Repository in GitHub Uploading OBS Files to JupyterLab Uploading Remote Files to JupyterLab Parent topic: Uploading and Downloading Data in Notebook
obs-copy Copy files or directories between OBS and a local path. ma-job Submit ModelArts jobs and obtain jod details. dli-job Submit DLI spark jobs and obtain jod details.
