11--- 
2- title : " `uarray`: A Generic Override Framework for Methods" 
2+ title : ' `uarray`: A Generic Override Framework for Methods' 
33author : hameer-abbasi 
44published : April 30, 2019 
55description : ' The problem is, stated simply: How do we use all of the PyData libraries in tandem, moving seamlessly from one to the other, without actually changing the API, or even the imports?' 
66category : [PyData ecosystem] 
77featuredImage :
8-   src : /posts/hello-world-post/blog_hero_var1 .svg 
8+   src : /posts/uarray-intro/blog_feature_var1 .svg 
99  alt : ' An illustration of a brown and a dark brown hand coming towards each other to pass a business card with the logo of Quansight Labs.' 
1010hero :
11-   imageSrc : /posts/hello-world-post /blog_hero_var1.svg 
11+   imageSrc : /posts/uarray-intro /blog_hero_var1.svg 
1212  imageAlt : ' An illustration of a brown hand holding up a microphone, with some graphical elements highlighting the top of the microphone.' 
1313--- 
1414
@@ -31,12 +31,12 @@ GPUs and distributed computing emerged. Also, there were old ideas that
3131couldn't really be used with NumPy's API, such as sparse arrays. To
3232solve these problems, various libraries emerged:
3333
34- -     Dask, for distributed NumPy
35- -     CuPy, for NumPy on Nvidia-branded GPUs.
36- -     PyData/Sparse, a project started to make sparse arrays conform to
37-      the NumPy API
38- -     Xnd, which extends the type system and the universal function
39-      concept found in NumPy
34+ -  Dask, for distributed NumPy
35+ -  CuPy, for NumPy on Nvidia-branded GPUs.
36+ -  PyData/Sparse, a project started to make sparse arrays conform to
37+   the NumPy API
38+ -  Xnd, which extends the type system and the universal function
39+   concept found in NumPy
4040
4141There were yet other libraries that emerged: PyTorch, which mimics NumPy
4242to a certain degree; TensorFlow, which defines its own API; and MXNet,
@@ -53,27 +53,21 @@ Oliphant so eloquently puts it, \"re-writing the world\"?
5353In my mind, the goals are (stated abstractly):
5454
55551 .   Methods that are not tied to a specific implementation.
56- 
57- -    For example ` np.arange ` 
58- 
56+     -  For example ` np.arange ` 
59571 .   Backends that implement these methods.
60- 
61- -    NumPy, Dask, PyTorch are all examples of this.
62- 
58+     -  NumPy, Dask, PyTorch are all examples of this.
63591 .   Coercion of objects to other forms to move between backends.
64- 
65- -    This means converting a NumPy array to a Dask array, and vice versa.
60+     -  This means converting a NumPy array to a Dask array, and vice versa.
6661
6762In addition, we wanted to be able to do this for arbitrary objects. So
6863` dtype ` s, ` ufunc ` s etc. should also be dispatchable and coercible.
6964
7065## The Solution?  
7166
7267With that said, let's dive into ` uarray ` . If you're not interested in
73- the gory details, you can jump down to
74- ` <a href="#how-to-use-it"> ` {=html}this section` </a> ` {=html}.
68+ the gory details, you can jump down to <a  href = " #how-to-use-it"  >this section</a >
7569
76- ```   python 
70+ ``` python 
7771import  uarray as  ua
7872
7973#  Let's ignore this for now
@@ -99,43 +93,67 @@ def myfunc_be2(): # Note that it has exactly the same signature
9993    return  " Strawberry" 
10094``` 
10195
102- ```  python 
96+ <CH.Code  className = " labs-blog-cell-in-out"  >
97+ 
98+ ``` python  in
10399with  ua.set_backend(be1):
104100    print (myfunc())
105101``` 
106102
107-     Potato 
103+ --- 
104+ 
105+ ``` txt  out
106+ Potato 
107+ ``` 
108+ 
109+ </CH.Code >
108110
109- ```  python 
111+ <CH.Code  className = " labs-blog-cell-in-out"  >
112+ 
113+ ``` python  in
110114with  ua.set_backend(be2):
111115    print (myfunc())
112116``` 
113117
114-     Strawberry 
118+ --- 
119+ 
120+ ``` text  out
121+ Strawberry 
122+ ``` 
123+ 
124+ </CH.Code >
115125
116126As we can clearly see: We have already provided a way to do (1) and (2)
117127above. But then we run across the problem: How do we decide between
118128these backends? How do we move between them? Let's go ahead and
119129register both of these backends for permanent use. And see what happens
120130when we want to implement both of their methods!
121131
122- ```   python 
132+ ``` python 
123133ua.register_backend(be1)
124134ua.register_backend(be2)
125135``` 
126136
127- ```  python 
137+ <CH.Code  className = " labs-blog-cell-in-out"  >
138+ 
139+ ``` python  in
128140print (myfunc())
129141``` 
130142
131-     Potato 
143+ --- 
144+ 
145+ ``` text  out
146+ Potato 
147+ ``` 
148+ 
149+ </CH.Code >
132150
133151As we see, we get only the first backend's answer. In general, it's
134152indeterminate what backend will be selected. But, this is a special
135153case: We're not passing arguments in! What if we change one of these to
136154return ` NotImplemented ` ?
137155
138- ```   python 
156+ ``` python 
139157#  We redefine the multimethod so it's new again
140158@ua.create_multimethod (myfunc_rd)
141159def  myfunc ():
@@ -158,38 +176,54 @@ ua.register_backend(be1)
158176ua.register_backend(be2)
159177``` 
160178
161- ```  python 
179+ <CH.Code  className = " labs-blog-cell-in-out"  >
180+ 
181+ ``` python  in
162182with  ua.set_backend(be1):
163183    print (myfunc())
164184``` 
165185
166-     Strawberry 
186+ --- 
187+ 
188+ ``` txt  out
189+ Strawberry 
190+ ``` 
191+ 
192+ </CH.Code >
167193
168194Wait\. .. What? Didn't we just set the first ` Backend ` ? Ahh, but, you
169- see\. .. It's signalling that it has * no *  implementation for ` myfunc ` .
195+ see\. .. It's signalling that it has _ no _  implementation for ` myfunc ` .
170196The same would happen if you simply didn't register one. To force a
171197` Backend ` , we must use ` only=True `  or ` coerce=True ` , the difference will
172198be explained in just a moment.
173199
174- ```  python 
200+ <CH.Code  className = " labs-blog-cell-in-out"  >
201+ 
202+ ``` python  in
175203with  ua.set_backend(be1, only = True ):
176204    print (myfunc())
177205``` 
178206
179-     --------------------------------------------------------------------------- 
180-     BackendNotImplementedError                Traceback (most recent call last) 
181-     <ipython-input-8-ec856cf7c88b> in <module> 
182-           1 with ua.set_backend(be1, only=True): 
183-     ----> 2     print(myfunc()) 
207+ --- 
184208
185-     ~/Quansight/uarray/uarray/backend.py in __call__(self, *args, **kwargs) 
186-         108  
187-         109         if result is NotImplemented: 
188-     --> 110             raise BackendNotImplementedError('No selected backends had an implementation for this method.') 
189-         111  
190-         112         return result 
209+ ``` txt  out
210+ --------------------------------------------------------------------------- 
211+ BackendNotImplementedError                Traceback (most recent call last) 
212+ <ipython-input-8-ec856cf7c88b> in <module> 
213+         1 with ua.set_backend(be1, only=True): 
214+ ----> 2     print(myfunc()) 
215+ 
216+ ~/Quansight/uarray/uarray/backend.py in __call__(self, *args, **kwargs) 
217+     108 
218+     109         if result is NotImplemented: 
219+ --> 110             raise BackendNotImplementedError('No selected backends had an implementation for this method.') 
220+     111 
221+     112         return result 
222+ 
223+ BackendNotImplementedError: No selected backends had an implementation for this method. 
224+ ``` 
191225
192-      BackendNotImplementedError: No selected backends had an implementation for this method. 
226+ </ CH.Code > 
193227
194228Now we are told that no backends had an implementation for this function
195229(which is nice, good error messages are nice!)
@@ -199,7 +233,7 @@ Now we are told that no backends had an implementation for this function
199233Let's say we had two ` Backend ` s. Let's choose the completely useless
200234example of one storing a number as an ` int `  and one as a ` float ` .
201235
202- ```   python 
236+ ``` python 
203237class  Number (ua .DispatchableInstance ):
204238    pass 
205239
@@ -222,56 +256,82 @@ Number.register_convertor(be2, lambda x: str(x))
222256Let's also define a \" catch-all\"  method. This catches all
223257implementations of methods not already registered.
224258
225- ```   python 
259+ ``` python 
226260#  This can be arbitrarily complex
227261def  gen_impl1 (method , args , kwargs , dispatchable_args ):
228262    if  not  all (isinstance (a, Number) and  isinstance (a.value, int ) for  a in  dispatchable_args):
229263        return  NotImplemented 
230-      
264+ 
231265    return  args[0 ]
232266
233267#  This can be arbitrarily complex
234268def  gen_impl2 (method , args , kwargs , dispatchable_args ):
235269    if  not  all (isinstance (a, Number) and  isinstance (a.value, str ) for  a in  dispatchable_args):
236270        return  NotImplemented 
237-      
271+ 
238272    return  args[0 ]
239273
240274be1.register_implementation(None , gen_impl1)
241275be2.register_implementation(None , gen_impl2)
242276``` 
243277
244- ```  python 
278+ <CH.Code  className = " labs-blog-cell-in-out"  >
279+ 
280+ ``` python  in
245281myfunc(' 1'  ) #  This calls the second implementation
246282``` 
247283
248-     '1' 
284+ --- 
285+ 
286+ ``` txt  out
287+ '1' 
288+ ``` 
289+ 
290+ </CH.Code >
291+ 
292+ <CH.Code  className = " labs-blog-cell-in-out"  >
249293
250- ```   python 
294+ ``` python  in 
251295myfunc(1 ) #  This calls the first implementation
252296``` 
253297
254-     1 
298+ --- 
299+ 
300+ ``` txt  out
301+ 1 
302+ ``` 
303+ 
304+ </CH.Code >
255305
256- ```  python 
306+ <CH.Code  className = " labs-blog-cell-in-out"  >
307+ 
308+ ``` python  in
257309myfunc(1.0 ) #  This fails
258310``` 
259311
312+ --- 
313+ 
314+ ``` txt  out
260315    --------------------------------------------------------------------------- 
261316    BackendNotImplementedError                Traceback (most recent call last) 
262317    <ipython-input-13-8431c1275db5> in <module> 
263318    ----> 1 myfunc(1.0) # This fails 
264319
265320    ~/Quansight/uarray/uarray/backend.py in __call__(self, *args, **kwargs) 
266-         108   
321+         108 
267322        109         if result is NotImplemented: 
268323    --> 110             raise BackendNotImplementedError('No selected backends had an implementation for this method.') 
269-         111   
324+         111 
270325        112         return result 
271326
272327    BackendNotImplementedError: No selected backends had an implementation for this method. 
328+ ``` 
273329
274- ```  python 
330+ </CH.Code >
331+ 
332+ <CH.Code  className = " labs-blog-cell-in-out"  >
333+ 
334+ ``` python  in
275335#  But works if we do this:
276336
277337with  ua.set_backend(be1, coerce = True ):
@@ -281,8 +341,14 @@ with ua.set_backend(be2, coerce=True):
281341    print (type (myfunc(1.0 )))
282342``` 
283343
284-     <class 'int'> 
285-     <class 'str'> 
344+ --- 
345+ 
346+ ``` txt  out
347+ <class 'int'> 
348+ <class 'str'> 
349+ ``` 
350+ 
351+ </CH.Code >
286352
287353This may seem like too much work, but remember that it's broken down
288354into a lot of small steps:
@@ -305,15 +371,23 @@ right values from it.
305371` unumpy `  is a set of NumPy-related multimethods built on top of
306372` uarray ` . You can use them as follows:
307373
308- ```  python 
374+ <CH.Code  className = " labs-blog-cell-in-out"  >
375+ 
376+ ``` python  in
309377import  unumpy as  np #  Note the changed import statement
310378from  unumpy.xnd_backend import  XndBackend
311379
312380with  ua.set_backend(XndBackend):
313381    print (type (np.arange(0 , 100 , 1 )))
314382``` 
315383
316-     <class 'xnd.array'> 
384+ --- 
385+ 
386+ ``` txt  out
387+ <class 'xnd.array'> 
388+ ``` 
389+ 
390+ </CH.Code >
317391
318392And, as you can see, we get back an Xnd array when using a NumPy-like
319393API. Currently, there are three back-ends: NumPy, Xnd and PyTorch. The
@@ -324,8 +398,7 @@ We are also working on supporting more of the NumPy API, and dispatching
324398over dtypes.
325399
326400Feel free to browse the source and open issues at:
327- < https://github.com/Quansight-Labs/uarray >  or shoot me an email at
328- ` <a href="mailto:[email protected] "> ` {=html}
[email protected] ` </a> ` {=html}
329- if you want to contact me directly. You can also find the full
330- documentation at < https://uarray.readthedocs.io/en/latest/ > .
401+ https://github.com/Quansight-Labs/uarray  or shoot me an email at
331402
403+ 404+ if you want to contact me directly. You can also find the full documentation at https://uarray.readthedocs.io/en/latest/ .
0 commit comments