AI开发平台MODELARTS-启动推理服务:离线推理
时间：2025-05-19 09:57:50
下载AI开发平台MODELARTS用户手册完整版
复制链接到剪贴板
分享文章到微博
分享文章到朋友圈
离线推理
编辑一个python脚本，脚本内容如下，运行该脚本使用ascend-vllm进行模型离线推理。
from vllm import LLM, SamplingParams
def main():
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
model_path = "/path/to/model"
llm = LLM(model=model_path, tensor_parallel_size=1, max_model_len=8192)
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
if __name__=="__main__":
main()support.huaweicloud.com/bestpractice-modelarts/modelarts_llm_infer_91206.html
