Add FAQ section on how to avoid excessive LIST load when writing apps

Author: JohnRusk

sig-scalability/configs-and-limits/faq.md

@@ -62,6 +62,31 @@ summarized as:
 - if you can't keep your object size below 100kB, reach out to SIG
   Scalability and discuss the usecase to see how we can make it performant
 
+### How should we code client applications to improve scalability?
+
+As noted above, LIST requests can be particularly expensive. So when working with lists
+that may have more than a few thousand elements, consider these guidelines:
+
+1. When defining a new resource type (new CRD) consider expected numbers 
+of objects that will exist (numbers of CRs).  See guidelines 
+[here](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/95-custom-resource-definitions#scale-targets-for-ga).
+1. If your code needs to hold an up-to-date list of objects in memory,
+avoid repeated LIST calls if possible.  Instead consider using the
+`Informer` classes that are provided in most Kubernetes client 
+libraries.   Informers automatically combine LIST and WATCH functionality
+to efficiently maintain an in-memory collection.
+1. If `Informer`s don't suit your needs, try to use the API Server cache 
+when LISTing.  To use the cache you must supply a `ResourceVersion`. 
+Read the [documentation about ResourceVersions](https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions) carefully to understand how it will affect the 
+freshness of the data you receive.
+1. If you can't use `Informer`s AND you can't use the API Server cache,
+ then be sure to [read large lists in chunks](https://kubernetes.io/docs/reference/using-api/api-concepts/#retrieving-large-results-sets-in-chunks).
+1. Consider the number of instances of your client application which will be running. For instance,
+there is a big difference between having 
+just one controller listing objects, versus having demonsets on every node 
+doing the same thing.  If there will be many instances of your client application
+(either in daemonsets or some other form) you should be particularly careful
+about LIST-related load.
 
 ### How do you setup clusters for scalability testing?