检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Table 1 Process of building a third-party model dataset Procedure Step Description Reference Importing data to the Pangu platform Creating an import task Import data stored in OBS or local data into the platform for centralized management, facilitating subsequent processing or publishing
Importing data to the Pangu platform Creating an import task Import data stored in OBS into the platform for centralized management, facilitating subsequent processing or publishing.
You can use OBS to import data. For details, see Using OBS Console. To create an import job, do as follows: Log in to ModelArts Studio Large Model Deveopment Platform. In the My Spaces area, click the required workspace.
Check whether the dataset file exists in the original OBS bucket. download obs file failed. Check whether the network is normal and whether data in the OBS bucket can be accessed. Data Evaluation annotate type is invalid.
To configure the OBS access permission, perform the following steps: Log in to ModelArts Studio. Configure OBS access authorization. Method 1: Click here in the pop-up message on the top of the homepage.
Figure 1 Dataset construction flowchart Table 1 Dataset construction process Procedure Step Description Importing data to the Pangu platform Creating an import task Import data stored in OBS or local data into the platform for centralized management, facilitating subsequent processing
Public services, such as Elastic Cloud Server (ECS), Elastic Volume Service (EVS), Object Storage Service (OBS), Virtual Private Cloud (VPC), Elastic IP (EIP), and Image Management Service (IMS), are shared within the same region.
Permanently Deleting a Dataset Function For data uploaded from OBS, you need to delete the associated raw data in OBS when deleting the datasets.
Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited. The following is an example.
# Example of an OBS parameter - key: sensitive_word name: OBS path of the sensitive word dictionary file type: OBS tips: sensitive word dictionary file required: true visible: true default: NLP/system_resource/sensitive_word.csv # OBS path of the default
obs:bucket:HeadBucket obs:bucket:ListAllMyBuckets obs:bucket:ListBucket obs:object:GetObject obs:object:GetObjectAcl obs:object:GetObjectVersion obs:object:GetObjectVersionAcl obs:object:ListMultipartUploadParts Read-only permission on the user's OBS bucket Pangu User Roles Pangu
Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited. Local upload: The size of a single file cannot exceed 10 MB, and the number of files cannot exceed 100. Parent topic: Dataset Format Requirements
Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited.
Related Services OBS PanguLM uses Object Storage Service (OBS) to securely and reliably store data and models at low costs. ModelArts PanguLM uses ModelArts for algorithm training and deployment, helping users quickly create and deploy models.
Import from OBS: The size of a single compressed package cannot exceed 50 GB (only .tar packages are supported). The size of a single file cannot exceed 50 GB. The number of files is not limited.
Meteorology - Ocean data nc, cdf, netcdf, gr, gr1, grb, grib, grb1, grib1, gr2, grb2, and grib2 Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited.
Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited. Local upload: The size of a single file cannot exceed 10 MB, and the number of files cannot exceed 100.
You can use the OBS path to query all raw datasets created based on the path and the subsequent lineage information. - Permanently Deleting a Dataset For data uploaded from OBS, you need to delete the associated raw data in OBS when deleting the datasets.
Constraints: N/A Value range: [1, 1000] Default value: 100 from_path Yes string Definition: Source OBS path. Constraints: Full OBS path of the end tenant.
OBS-based data protection PanguLM works with OBS to store and protect user data. For details, see OBS Data Protection. Parent topic: Security
