检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Figure 1 Dataset details Log in to the OBS console and locate the directory of the corresponding dataset version from the OBS path obtained in 2 to obtain the labeling result of the dataset. Figure 2 Obtaining the labeling result Parent topic: ModelArts Standard Data Preparation
Preset images Preset image used by a training job Code Directory OBS path to the code directory of a training job You can click Edit Code on the right to edit the training script code in OBS Online Editor.
object:GetObjectAcl obs:object:PutObjectAcl obs:object:PutObject obs:object:GetObject obs:object:DeleteObject obs:object:ModifyObjectMetaData √ √ Modifying a training job PUT /v2/{project_id}/training-jobs/{training_job_id} modelarts:trainJob:update N/A √ √ Deleting a training job
How Do I Rename an OBS File on a ModelArts Notebook Instance? How Do I Use the pandas Library to Process Data in OBS Buckets on a ModelArts Notebook Instance? How Do I Access the OBS Bucket of Another Account from a ModelArts Notebook Instance?
for heterogeneous jobs Table 28 remote Parameter Type Description obs obs object OBS in which data input and output are stored Table 29 obs Parameter Type Description obs_url String OBS URL of the dataset for a training job, for example, /usr/data/ Table 30 outputs Parameter Type
for heterogeneous jobs Table 28 remote Parameter Type Description obs obs object OBS in which data input and output are stored Table 29 obs Parameter Type Description obs_url String OBS URL of the dataset for a training job, for example, /usr/data/ Table 30 outputs Parameter Type
Does ModelArts Use the OBS API to Access OBS Files over an Intranet or the Internet?
Symptom: After the labeled data is uploaded to OBS and synchronized, the data is displayed as unlabeled. Possible causes: Automatic encryption is enabled in the OBS bucket. Solution: Create an OBS bucket and upload data again, or disable bucket encryption and upload data again.
object:PutObject obs:object:GetObject obs:object:GetObjectVersion obs:bucket:HeadBucket obs:object:DeleteObject obs:object:GetObject obs:bucket:CreateBucket obs:bucket:ListBucket modelarts:trainJob:list modelarts:trainJob:update modelarts:trainJobVersion:delete modelarts:pool:list
For example, to use OBS, search for OBS and select OBS OperateAccess. ModelArts training jobs use OBS to forward data. Therefore, the permissions for using OBS are necessary. For permissions of more cloud services, such as SWR, see Table 1.
You can also specify an OBS path, for example, obs://Bucket name/Package name. Local files are also supported. To specify multiple parameters, use --jars jar1 --jars jar2.
Table 1 Required OBS folders Folder Usage obs://test-modelarts/pytorch/demo-code/ Stores the training script. obs://test-modelarts/pytorch/log/ Stores training log files.
bucket:ListAllMybuckets", "obs:bucket:HeadBucket", "obs:bucket:ListBucket", "obs:bucket:GetBucketLocation", "obs:object:GetObject", "obs:object:GetObjectVersion", "obs:object:PutObject
Error 404 If this error is reported when an IAM user creates an instance, the IAM user does not have the permission to access the OBS bucket. Solution Log in to the OBS console using the tenant account and grant access permissions for the OBS bucket to the IAM user.
Take OBS path obs://obs-bucket/training-test/demo-code as an example. The content in the OBS path will be automatically downloaded to ${MA_JOB_DIR}/demo-code in the training container, and demo-code (customizable) is the last-level directory of the OBS path.
Log in to the OBS console using the current account, and search for the OBS buckets, folders, and files in the path to check whether the code directory exists.
Symptom: After the labeled data is uploaded to OBS and synchronized, the data is displayed as unlabeled. Possible causes: Automatic encryption is enabled in the OBS bucket. Solution: Create an OBS bucket and upload data again, or disable bucket encryption and upload data again.
"obs:object:GetObject", "obs:object:GetObjectVersion", "obs:bucket:HeadBucket", "obs:object:DeleteObject", "obs:bucket:CreateBucket", "obs:bucket:ListBucket" ]
Dynamically Mounting OBS Obtaining the Notebook Instances with OBS Storage Mounted Obtain the notebook instances with OBS storage mounted. Dynamically Mounting OBS Dynamically mount OBS to a notebook instance in running state.
Step 2 Preparing the Training Script and Uploading It to OBS Prepare the training script pytorch-verification.py and upload it to the obs://test-modelarts/pytorch/demo-code/ folder of the OBS bucket.
