@@ -105,7 +105,6 @@ For tags `>=23.03`,
105105[ tritonserver release containers] ( https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver )
106106come with the following cache implementations out of the box:
107107- [ local] ( https://github.com/triton-inference-server/local_cache ) : ` /opt/tritonserver/caches/local/libtritoncache_local.so `
108- - [ redis] ( https://github.com/triton-inference-server/redis_cache ) : ` /opt/tritonserver/caches/redis/libtritoncache_redis.so `
109108
110109With these TRITONCACHE APIs, ` tritonserver ` exposes a new ` --cache-config `
111110CLI flag that gives the user flexible customization of which cache implementation
@@ -125,26 +124,18 @@ When `--cache-config local,size=SIZE` is specified with a non-zero `SIZE`,
125124Triton allocates the requested size in CPU memory and ** shares the
126125cache across all inference requests and across all models** .
127126
128- #### Redis Cache
129-
130- The ` redis ` cache implementation was added along with these TRITONCACHE API
131- changes for users that require a more configurable cache. More
132- ` redis ` cache specific details can be found in the
133- [ redis cache implementation] ( https://github.com/triton-inference-server/redis_cache ) .
134-
135127#### Custom Cache
136128
137129With the new the TRITONCACHE API interface, it is now possible for
138130users to implement their own cache to suit any use-case specific needs.
139131To see the required interface that must be implemented by a cache
140132developer, see the
141133[ TRITONCACHE API header] ( https://github.com/triton-inference-server/core/blob/main/include/triton/core/tritoncache.h ) .
142- The ` local ` and ` redis ` cache implementations may be used as reference
143- implementations.
134+ The ` local ` cache implementation may be used as a reference implementation.
144135
145136Upon successfully developing and building a custom cache, the resulting shared
146137library (ex: ` libtritoncache_<name>.so ` ) must be placed in the cache directory
147- similar to where the ` local ` and ` redis ` cache implementations live . By default,
138+ similar to where the ` local ` cache implementation lives . By default,
148139this directory is ` /opt/tritonserver/caches ` , but a custom directory may be
149140specified with ` --cache-dir ` as needed.
150141
0 commit comments