Skip to content

[TTS]Out of memory with fastspeech2_csmsc #4060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DBDXSS opened this issue Apr 16, 2025 · 5 comments
Open

[TTS]Out of memory with fastspeech2_csmsc #4060

DBDXSS opened this issue Apr 16, 2025 · 5 comments
Assignees

Comments

@DBDXSS
Copy link

DBDXSS commented Apr 16, 2025

参考https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/tts3
运行代码,环境配置如下:
python3 -m pip install paddlepaddle-gpu==2.5.2.post117 paddlespeech -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html

显卡配置:H800 80G cuda 11.8

Building prefix dict from the default dictionary ...
[2025-04-16 14:33:00,898] [ DEBUG] init.py:113 - Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
[2025-04-16 14:33:00,899] [ DEBUG] init.py:132 - Loading model from cache /tmp/jieba.cache
Loading model cost 0.505 seconds.
[2025-04-16 14:33:01,403] [ DEBUG] init.py:164 - Loading model cost 0.505 seconds.
Prefix dict has been built successfully.
[2025-04-16 14:33:01,403] [ DEBUG] init.py:166 - Prefix dict has been built successfully.
W0416 14:33:12.098174 5426 interpreter_util.cc:838] range raises an exception paddle::memory::allocation::BadAlloc,


C++ Traceback (most recent call last):

0 run_program_ad_func(std::vector<paddle::Tensor, std::allocatorpaddle::Tensor > const&, std::vector<paddle::Tensor, std::allocatorpaddle::Tensor > const&, std::vector<paddle::Tensor*, std::allocatorpaddle::Tensor* >&, std::vector<paddle::framework::Scope*, std::allocatorpaddle::framework::Scope* >&, std::vector<paddle::Tensor*, std::allocatorpaddle::Tensor* >&, paddle::framework::AttributeMap const&)
1 RunProgramAPI(std::vector<paddle::Tensor, std::allocatorpaddle::Tensor > const&, std::vector<paddle::Tensor, std::allocatorpaddle::Tensor > const&, std::vector<paddle::Tensor*, std::allocatorpaddle::Tensor* >&, std::vector<paddle::framework::Scope*, std::allocatorpaddle::framework::Scope* >&, std::vector<paddle::Tensor*, std::allocatorpaddle::Tensor* >&, bool, paddle::framework::AttributeMap const&)
2 paddle::framework::InterpreterCore::Run(std::vector<std::string, std::allocator<std::string > > const&, bool)
3 paddle::framework::interpreter::BuildOpFuncList(phi::Place const&, paddle::framework::BlockDesc const&, std::set<std::string, std::less<std::string >, std::allocator<std::string > > const&, std::vector<paddle::framework::OpFuncNode, std::allocatorpaddle::framework::OpFuncNode >, paddle::framework::VariableScope, paddle::framework::interpreter::ExecutionConfig const&, bool, bool)
4 void phi::ArangeKernel<long, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor*)
5 long* phi::DeviceContext::Alloc(phi::TensorBase*, unsigned long, bool) const
6 phi::DeviceContext::Impl::Alloc(phi::TensorBase*, phi::Place const&, phi::DataType, unsigned long, bool, bool) const
7 phi::DenseTensor::AllocateFrom(phi::Allocator*, phi::DataType, unsigned long, bool)
8 paddle::memory::allocation::Allocator::Allocate(unsigned long)
9 paddle::memory::allocation::StatAllocator::AllocateImpl(unsigned long)
10 paddle::memory::allocation::Allocator::Allocate(unsigned long)
11 paddle::memory::allocation::Allocator::Allocate(unsigned long)
12 paddle::memory::allocation::Allocator::Allocate(unsigned long)
13 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long)
14 std::string phi::enforce::GetCompleteTraceBackString<std::string >(std::string&&, char const*, int)
15 phi::enforce::GetCurrentTraceBackStringabi:cxx11


Error Message Summary:

ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 7.019809EB memory on GPU 0, 915.187500MB memory has been allocated and available memory is only 78.426208GB.

显示需要7.019809EB,修改batchsize无效

@DBDXSS
Copy link
Author

DBDXSS commented Apr 16, 2025

使用cpu可以运行,设置gpu如上

python3 ${BIN_DIR}/../synthesize_e2e.py
--am=fastspeech2_csmsc
--am_config=fastspeech2_nosil_baker_ckpt_0.4/default.yaml
--am_ckpt=fastspeech2_nosil_baker_ckpt_0.4/snapshot_iter_76000.pdz
--am_stat=fastspeech2_nosil_baker_ckpt_0.4/speech_stats.npy
--voc=pwgan_csmsc
--voc_config=pwg_baker_ckpt_0.4/pwg_default.yaml
--voc_ckpt=pwg_baker_ckpt_0.4/pwg_snapshot_iter_400000.pdz
--voc_stat=pwg_baker_ckpt_0.4/pwg_stats.npy
--lang=zh
--text=${BIN_DIR}/../../assets/sentences.txt
--output_dir=exp/default/test_e2e
--inference_dir=exp/default/inference
--phones_dict=fastspeech2_nosil_baker_ckpt_0.4/phone_id_map.txt
--ngpu=0

@zxcd zxcd self-assigned this Apr 16, 2025
@zxcd
Copy link
Collaborator

zxcd commented Apr 16, 2025

可以在保证gpu显存的情况下尝试使用paddlespeech == develop + paddlepaddle-gpu>=3.0beta2 看看是否能跑

@DBDXSS
Copy link
Author

DBDXSS commented Apr 16, 2025

可以在保证gpu显存的情况下尝试使用paddlespeech == develop + paddlepaddle-gpu>=3.0beta2 看看是否能跑

请问有可参考的下载文档吗

@DBDXSS
Copy link
Author

DBDXSS commented Apr 16, 2025

可以在保证gpu显存的情况下尝试使用paddlespeech == develop + paddlepaddle-gpu>=3.0beta2 看看是否能跑

我测试了下都安装了,输出下面错误,InvalidArgumentError: The 3-th dimension of input tensor (17) must match or be broadcastable to the corresponding dimension (4352512999702696242) in shape. 似乎这也是为什么会爆显存的原因。

@zxcd
Copy link
Collaborator

zxcd commented Apr 21, 2025

${BIN_DIR}/../../assets/sentences.txt 中的文本是默认的文本吗?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants