检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Model Management Permissions Table 1 Fine-grained permissions for model management Permission API Action Related Action IAM Project Enterprise Project Importing a Model POST /v1/{project_id}/models modelarts:model:create obs:bucket:ListAllMybuckets obs:object:PutObjectAcl obs:bucket
Checking Whether the Affected Path Is an OBS Path When using ModelArts, store data in an OBS bucket. However, the OBS path cannot be used to read data during the execution of the training code.
Client Overview of OBS Management How to call the SDK APIs of Object Storage Service (OBS), including the APIs for creating OBS buckets, uploading and downloading files and folders, as well as deleting OBS objects and buckets ModelArts SDK operations: Data Management Training Management
OBS: Import data from an OBS path or a manifest file. Local file: Import local data that has been uploaded to an OBS path. Table Data Sources You can import data from OBS, MRS, and local files. Import Mode There are five modes for importing data to a dataset.
If the folder exists in the OBS bucket, the upload is successful. import moxing as mox mox.file.copy_parallel('/home/ma-user/work/sub_dir_0', 'obs://bucket_name/sub_dir_0') Use OBS or ModelArts SDK to download the file from OBS to the local path.
For details, see How Do I Upload a File from a Notebook Instance to OBS or Download a File from OBS to a Notebook Instance? Parent topic: Code Running Failures
Storage Mounted OBS Storage Mounting Obtaining Details About a Notebook Instance with OBS Storage Mounted Unmounting OBS Storage from a Notebook Instance Querying Supported Images Registering a Custom Image Querying the User Image List Obtaining Details of an Image Deleting an Image
CAUTION: OBS resources for storing data are continuously billed. To stop billing, delete the data stored in OBS. Pay-per-use Yearly/Monthly Creating an OBS bucket is free of charge. You pay only for the storage capacity and duration you actually use.
OBS NOTE: OBS is a whitelist function. To use this function, contact Huawei technical support. When uploading or downloading a large amount of data in the development environment, you can use OBS buckets to transfer data.
Take OBS path obs://obs-bucket/training-test/demo-code as an example. The content in the OBS path will be automatically downloaded to ${MA_JOB_DIR}/demo-code in the training container, and demo-code (customizable) is the last-level directory of the OBS path.
upload Upload local file or OBS object to DLI resources.
OBS buckets and folders for storing data are available. In addition, the OBS buckets and ModelArts are in the same region. OBS parallel file systems are not supported. Select object storage. ModelArts does not support encrypted OBS buckets.
OBS: Import data from an OBS path or a manifest file. Local file: Import local data that has been uploaded to an OBS path. Table Data Sources You can import data by downloading built-in datasets from AI Gallery, or from OBS, DWS, DLI, MRS, and local files.
Take OBS path obs://obs-bucket/training-test/demo-code as an example. The content in the OBS path will be automatically downloaded to ${MA_JOB_DIR}/demo-code in the training container, and demo-code (customizable) is the last-level directory of the OBS path.
Upload images to any OBS directory and import the images from the OBS directory to an existing dataset. Use data source synchronization.
an OBS file. 1 2 3 4 import pandas as pd import moxing as mox with mox.file.File("obs://bucket_name/b.txt", "r") as f: csv = pd.read_csv(f) Use pandas to write an OBS file. 1 2 3 4 5 import pandas as pd import moxing as mox df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]}) with
The storage fee is based on the OBS billing rules.
Table 1 Required OBS folders Folder Description obs://test-modelarts/tensorflow/code/ Stores the training script. obs://test-modelarts/tensorflow/data/ Stores dataset files. obs://test-modelarts/tensorflow/log/ Store training log files.
Table 2 Parameters for initializing TrainingFiles Parameter Mandatory Type Description code_dir Yes String Code directory of a training job, which is an OBS path and must start with obs:/, for example, obs://xx/yy/ boot_file Yes String Boot file of a training job, which must be stored
Table 1 Required OBS folders Folder Usage obs://test-modelarts/tensorflow/code/ Stores the training script. obs://test-modelarts/tensorflow/data/ Stores dataset files. obs://test-modelarts/tensorflow/log/ Store training log files.
