检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Input Path OBS path to the input data in the batch service. Output Path OBS path to the output data in the batch service. Model Name & Version Name and version of the model used by the batch service. Advanced Log Management This feature is disabled by default.
In this example, the installation package has been uploaded to obs://cnnorth4-test/codes/mox_benchmarks/apex-master/.
For example, the OBS path specified by Code Directory contains model files and the pip-requirements.txt file.
String Error code. error_msg String Error message. file_statistics FileCopyProgress object File replication progress finished_file_count Long Number of files that have been transferred. finished_file_size Long Size of the file that has been transferred, in bytes. import_path String OBS
Critical Table 7 Events for dynamic OBS mounting Event Description Severity DynamicMountStorage The OBS storage is mounted. Major DynamicUnmountStorage The OBS storage is unmounted.
# Define the input OBS object. obs_data = wf.data.OBSPlaceholder(name="obs_placeholder_name", object_type="directory") # Use JobStep to define a training phase, and use OBS to store the output. job_step = wf.steps.JobStep( name="training_job", # Name of a training phase.
ModelArts data management provides the following functions for you to obtain high-quality AI data: Data acquisition Allows you to import data from OBS and MRS. Provides 18+ data augmentation operators to increase data volume for training.
Feature Description Phase Document 1 Access key authorization discontinued Certain ModelArts functions require access to OBS, SWR, and IEF. Before using ModelArts, ensure your account has been authorized to access these services.
Input Path OBS path to the input data in the batch service. Output Path OBS path to the output data in the batch service. AI Application Name & Version Name and version of the AI application used by the batch service. Advanced Log Management This function is disabled by default.
Feature Description Phase Document 1 Access key authorization discontinued Certain ModelArts functions require access to OBS, SWR, and IEF. Before using ModelArts, ensure your account has been authorized to access these services.
Code Directory Select the code directory required for this training job, for example, obs://test-modelarts/ascend/code/ in this case. Boot Command Python boot command of the image, for example, bash ${MA_JOB_DIR}/code/run_torch_ddp_npu.sh in this case.
For example, you can grant access permission on an OBS bucket to a tenant for model management. Internal resource authorization ModelArts inference supports fine-grained permission control.
To quickly obtain the latest data in the OBS bucket, on the All or Unlabeled tab page of the dataset details page, click Synchronize Data Source to add data from OBS to the dataset.
Data mounting If the collected logs are stored on OBS, mount the log data in OBS using the Rclone tool. Download and install Rclone. Configure the credentials required for accessing OBS. # Hard-coded or plaintext AK/SK is risky.
# Define the input OBS object. obs_data = wf.data.OBSPlaceholder(name="obs_placeholder_name", object_type="directory") # Use JobStep to define a training phase, and use OBS to store the output. job_step_1 = wf.steps.JobStep( name="training_job_1", # Name of a training phase
Permanent logs: These logs are dumped to your OBS bucket. When creating a training job, you can enable persistent log saving and set a job log path for dumping. Figure 1 Enabling Persistent Log Saving Real-time logs and historical logs have no difference in content.
Obtain the detailed OBS path next to Output Path, switch to the path and obtain the batch service prediction results, including the prediction result file and the AI application prediction result.
String OBS path of the output data of a batch job req_uri String Inference path of a batch job mapping_type String Mapping type of the input data.
If the output path is an OBS directory, set the path to a value starting with obs://. If labels are changed for label data, perform operations in If Labels Are Changed before running mox.run.
Prerequisites The training job has been successfully executed, and the model has been stored in the OBS directory where the training output is stored (the input parameter is train_url).
