检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Import Modes There are two import modes: OBS path and Manifest file. OBS path: indicates that the dataset to be imported has been stored in an OBS directory in advance. In this case, you need to select an OBS path that you can access.
Storage Mounted OBS Storage Mounting Obtaining Details About a Notebook Instance with OBS Storage Mounted Unmounting OBS Storage from a Notebook Instance Querying Supported Images Registering a Custom Image Obtaining User Image Groups Obtaining Details of an Image Deleting an Image
CAUTION: OBS resources for storing data are continuously billed. To stop billing, delete the data stored in OBS. Pay-per-use Yearly/Monthly Creating an OBS bucket is free of charge. You pay only for the storage capacity and duration you actually use.
OBS NOTE: OBS is a whitelist function. To use this function, contact Huawei technical support. When uploading or downloading a large amount of data in the development environment, you can use OBS buckets to transfer data.
Take OBS path obs://obs-bucket/training-test/demo-code as an example. The content in the OBS path will be automatically downloaded to ${MA_JOB_DIR}/demo-code in the training container, and demo-code (customizable) is the last-level directory of the OBS path.
Take OBS path obs://obs-bucket/training-test/demo-code as an example. The content in the OBS path will be automatically downloaded to ${MA_JOB_DIR}/demo-code in the training container, and demo-code (customizable) is the last-level directory of the OBS path.
upload Upload local file or OBS object to DLI resources.
OBS buckets and folders for storing data are available. In addition, the OBS buckets and ModelArts are in the same region. OBS parallel file systems are not supported. Select object storage. ModelArts does not support encrypted OBS buckets.
OBS: Import data from an OBS path or a manifest file. Local file: Import local data that has been uploaded to an OBS path. Table Data Sources You can import data by downloading built-in datasets from AI Gallery, or from OBS, DWS, DLI, MRS, and local files.
Upload images to any OBS directory and import the images from the OBS directory to an existing dataset. Use data source synchronization.
an OBS file. 1 2 3 4 import pandas as pd import moxing as mox with mox.file.File("obs://bucket_name/b.txt", "r") as f: csv = pd.read_csv(f) Use pandas to write an OBS file. 1 2 3 4 5 import pandas as pd import moxing as mox df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]}) with
As a result, OBS cannot be accessed and images cannot be loaded. You are not allowed to access the target OBS bucket. The OBS bucket or file may be encrypted. The OBS storage class does not allow the parallel file system to process images.
Data Storage How Do I Rename an OBS File? Do Files in /cache Still Exist After a Notebook Instance is Stopped or Restarted? How Do I Avoid a Restart? How Do I Use the pandas Library to Process Data in OBS Buckets?
Enter an existing OBS directory that starts with obs://, for example, obs://obs_bucket_name/folder_name. n_clusters No auto Number of data sample types. The default value is auto.
The storage fee is based on the OBS billing rules.
Table 1 Required OBS folders Folder Description obs://test-modelarts/tensorflow/code/ Stores the training script. obs://test-modelarts/tensorflow/data/ Stores dataset files. obs://test-modelarts/tensorflow/log/ Store training log files.
Table 2 Parameters for initializing TrainingFiles Parameter Mandatory Type Description code_dir Yes String Code directory of a training job, which is an OBS path and must start with obs:/, for example, obs://xx/yy/ boot_file Yes String Boot file of a training job, which must be stored
Table 1 Required OBS folders Folder Description obs://test-modelarts/tensorflow/code/ Stores the training script. obs://test-modelarts/tensorflow/data/ Stores dataset files. obs://test-modelarts/tensorflow/log/ Store training log files.
Figure 1 Dataset details Log in to the OBS console and locate the directory of the corresponding dataset version from the OBS path obtained in 2 to obtain the labeling result of the dataset. Figure 2 Obtaining the labeling result Parent topic: Data Management (Old Version)
After a dataset is deleted, if you need to delete the data in the dataset input and output paths in OBS to release resources, delete the data and the OBS folders on the OBS Console. Procedure In the left navigation pane, choose Data Management > Datasets.
