About GPU memory leak

[xurong1981]

Does anyone encounter the following error on "CUDA_ERROR_ILLEGAL_ADDRESS" ?
I have changed multiprocessing to single process, but the same problem happened.

My GPU is GeForce RTX 2080 8GB (driver: 440.33.01), and
tensorflow: 1.12.0
cuda: 9.0
cudnn: 7.5.0

Training command is like this way:
CUDA_VISIBLE_DEVICES=0 python main.py
--model_name=model_roerich
--batch_size=1
--phase=train
--image_size=768
--lr=0.0002
--dsr=0.8
--ptcd=./data/Places2/data_large
--ptad=./data/artist/nicholas-roerich

Finally, the error message are:
tensorflow::CurrentStackTrace()
stream_executor::cuda::CUDADriver::SynchronizeContext(stream_executor::cuda::CudaContext*)
stream_executor::StreamExecutor::SynchronizeAllActivity()
tensorflow::GPUUtil::SyncAll(tensorflow::Device*)
tensorflow::BaseGPUDevice::Sync()

Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)
std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&)


clone

*** End stack trace ***

2020-01-26 00:10:03.045956: E tensorflow/stream_executor/event.cc:34] error destroying CUDA event in context 0x5402c10: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
File "/home/username/.local/share/virtualenvs/adaptive-style-transfer-PbxNnQ9W/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/username/.local/share/virtualenvs/adaptive-style-transfer-PbxNnQ9W/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/username/.local/share/virtualenvs/adaptive-style-transfer-PbxNnQ9W/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: GPU sync failed