gpu版本推断示例数据OOM

[planetb113]

直接按照官网推荐的方式搭建环境

paddlespeech asr --lang zh --input zh.wav

报错

/usr/local/lib/python3.7/dist-packages/librosa/core/constantq.py:1059: DeprecationWarning: `np.complex` is a deprecated alias for the builtin `complex`. To silence this warning, use `complex` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.complex128` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  dtype=np.complex,
W0921 17:11:46.739881 21960 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.6, Runtime API Version: 10.2
W0921 17:11:46.742161 21960 gpu_resources.cc:91] device: 0, cuDNN Version: 8.1.
[2022-09-21 17:11:58,727] [   ERROR] - (Fatal) Operator range raises an paddle::memory::allocation::BadAlloc exception.
The exception content is
:ResourceExhaustedError: 

Out of memory error on GPU 0. Cannot allocate 655.142555PB CUDA managed memory on GPU 0, 493.057617MB memory has been allocated.

Please check whether there is any other process using GPU 0.
1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model. 

 (at /paddle/paddle/fluid/memory/allocation/cuda_managed_allocator.cc:81)
. (at /paddle/paddle/fluid/imperative/tracer.cc:307)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/paddlespeech/cli/asr/infer.py", line 308, in infer
    simulate_streaming=cfg.simulate_streaming)
  File "/root/.local/lib/python3.7/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/base.py", line 354, in _decorate_function
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/paddlespeech/s2t/models/u2/u2.py", line 767, in decode
    simulate_streaming=simulate_streaming)
  File "/usr/local/lib/python3.7/dist-packages/paddlespeech/s2t/models/u2/u2.py", line 531, in attention_rescoring
    num_decoding_left_chunks, simulate_streaming)
  File "/usr/local/lib/python3.7/dist-packages/paddlespeech/s2t/models/u2/u2.py", line 417, in _ctc_prefix_beam_search
    simulate_streaming)  # (B, maxlen, encoder_dim)
  File "/usr/local/lib/python3.7/dist-packages/paddlespeech/s2t/models/u2/u2.py", line 221, in _forward_encoder
    num_decoding_left_chunks=num_decoding_left_chunks
  File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/paddlespeech/s2t/modules/encoder.py", line 165, in forward
    masks = make_non_pad_mask(xs_lens).unsqueeze(1)  # (B, 1, L)
  File "/usr/local/lib/python3.7/dist-packages/paddlespeech/s2t/modules/mask.py", line 90, in make_non_pad_mask
    return make_pad_mask(lengths).logical_not()
  File "/usr/local/lib/python3.7/dist-packages/paddlespeech/s2t/modules/mask.py", line 61, in make_pad_mask
    seq_range = paddle.arange(0, max_len, dtype=paddle.int64)
  File "/usr/local/lib/python3.7/dist-packages/paddle/tensor/creation.py", line 567, in arange
    return paddle.fluid.layers.range(start, end, step, dtype, name)
  File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layers/tensor.py", line 1505, in range
    out = _C_ops.range(start, end, step)
SystemError: (Fatal) Operator range raises an paddle::memory::allocation::BadAlloc exception.
The exception content is
:ResourceExhaustedError: 

Out of memory error on GPU 0. Cannot allocate 655.142555PB CUDA managed memory on GPU 0, 493.057617MB memory has been allocated.

Please check whether there is any other process using GPU 0.
1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model. 

 (at /paddle/paddle/fluid/memory/allocation/cuda_managed_allocator.cc:81)
. (at /paddle/paddle/fluid/imperative/tracer.cc:307)

KeyError: 'result'

[yt605155624]

Out of memory error on GPU 0. Cannot allocate 655.142555PB CUDA managed memory on GPU 0, 493.057617MB memory has been allocated.
看样子你的显卡只有 1G 多的显存，是不够用的

[leolle]

要几百PB的内存了，还是显存问题吗