检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Interaction between ModelArts and OBS ModelArts does not support encrypted OBS buckets. When creating an OBS bucket, do not enable bucket encryption. ModelArts does not support cross-region access to OBS buckets. Ensure that OBS and ModelArts are in the same region.
ModelArts Standard Preparations Configuring Access Authorization for ModelArts Standard Creating and Managing a Workspace Creating an OBS Bucket for ModelArts to Store Data
If you need to use OBS external storage mounting for custom images in real-time services, create a new directory for OBS data, for example, /obs-mount/. Otherwise, the existing files will be overwritten. You can add, view, and modify the files in the OBS mount directory.
To quickly obtain the latest data in the OBS bucket, click Synchronize Data Source on the Unlabeled tab page of the dataset details page to add the data uploaded using OBS to the dataset.
Multi-node multi-card: If the data volume is large (1 TB training data) and the compute is high (four nodes with eight Vnt1 cards), use SFS to store data and a common OBS bucket to store code, and use distributed training.
Exporting Data from a ModelArts Dataset Exporting Data from ModelArts to OBS Exporting Data as a New Dataset Parent topic: Preparing and Processing Data
environment variables Variable Description Example S3_ENDPOINT OBS endpoint - S3_VERIFY_SSL Whether to use SSL to access OBS S3_VERIFY_SSL=0 S3_USE_HTTPS Whether to use HTTPS to access OBS S3_USE_HTTPS=1 Table 5 Environment variables of the PIP source and API Gateway address Variable
The root directory obs://deep-learning/new/mnist/ is used as an example. The training code file is obs://deep-learning/new/mnist/. The data is stored in obs://deep-learning/new/mnist/mnist_data. Create a training job using a custom image.
# Build a local image and save to local path and OBS ma-cli image build .ma/customize_from_ubuntu_18.04_to_modelarts/Dockerfile --target .
Ensure that the OBS directory you use and ModelArts are in the same region. For details about model package requirements, see Model Package Example. Input and Output Mode The built-in image processing mode cannot be overwritten.
Submitting a Training Job (New Version) Stopping a Training Job Viewing Training Logs OBS-based upload and download Local files or folders can be uploaded to OBS and files or folders can be downloaded from OBS to a local directory.
Code Directory OBS directory for storing the algorithm code. Boot File OBS directory for storing the boot file. This parameter is available only for algorithms created using a preset image. Boot Command Boot command of an algorithm created using a custom image.
Importing Data Import Operation Specifications for Importing Data from an OBS Directory Specifications for Importing the Manifest File Parent topic: Data Management (Old Version to Be Terminated)
Importing Data Introduction to Data Importing Importing Data from OBS Importing Data from DLI Importing Data from MRS Importing Data from DWS Importing Data from Local Files
Step 2 Preparing Script Files and Uploading Them to OBS Prepare the MPI boot script run_mpi.sh and training script mpi-verification.py and upload them to the obs://test-modelarts/mpi/demo-code/ folder of the OBS bucket.
Importing Data to a ModelArts Dataset Introduction to Data Importing Importing Data from OBS Importing Data from MRS to ModelArts Importing Data from Local Files Parent topic: Preparing and Processing Data
The value must be a specific directory in an OBS bucket rather than the root directory of an OBS bucket. If type is set to obsfs, the parameter value must be a valid bucket name of the OBS parallel file system. (CCE does not support subdirectory mounting.)
Uploading Files to JupyterLab Scenarios Uploading Files from a Local Path to JupyterLab Cloning an Open-Source Repository in GitHub Uploading OBS Files to JupyterLab Uploading Remote Files to JupyterLab Parent topic: Uploading and Downloading Data in Notebook
For a file that exceeds 100 MB but does not exceed 5 GB, upload the file to OBS (an object bucket or a parallel file system), and then download the file from OBS to a notebook instance. After the download is complete, the file is deleted from OBS.
Creating an AI Application Importing a Meta Model from a Training Job Importing a Meta Model from a Template Importing a Meta Model from OBS Importing a Meta Model from a Container Image Parent topic: Managing AI Applications
