ILI9341 Performance Improvements

[espruino-discuss3]

Posted at 2014-11-07 by DaveNI

Hi

I got a 2.2" 240*320 ILI9341 controlled display and have been really pleased with it (considering its low price) but the current ILI9341 module uses a Graphics Callback to display every pixel which is a bit too slow.

I have implemented a new method in the Controller module:

LCD.drawStringBuffered(str, x, y, fontSize, r, g, b)

This creates a buffer for the string being drawn then sends it to the display all at once. It is working (much faster than pixel by pixel) but I am having issues with the colour (wrong colours being displayed). I think it is something to do with the endianness of the 16bit colour data.

The current module uses the following code when writing 16 bit values (e.g. pixel colour) to the SPI:

spi.write(c>>8,c);

Could this be achieved by changing modifying the spi setup to use an order of 'lsb'. (I did try this but nothing was displayed!)

Anyway I'm pleased with how the screen performs & can work around the invalid colors but it would be great if I could get it functioning correctly. My method is as follows:

  LCD.drawStringBuffered = function(str, x, y, fontSize, r, g, b){
	//Create a dummy Graphics object to get size of string in fontSize
   var g = Graphics.createArrayBuffer(1, 1, 1);
	g.setFontVector(fontSize);
	var width = g.stringWidth(str)
	
	var height = fontSize * 1.3; // this works but not sure how height is related to size for vector font
	var x2 = x + width-1;
	var y2 = y + height-1;
	
	//now the width/height is known actual Graghics object can be created
	g = Graphics.createArrayBuffer(width, height, 16);	
    g.setFontVector(fontSize);
	g.setColor(r, g, b);
	g.drawString(str,0,0);
	
	//write buffer to spi
	ce.reset();
    spi.write(0x2A,dc);
    spi.write(x>>8,x,x2>>8,x2);
    spi.write(0x2B,dc);
    spi.write(y>>8,y,y2>>8,y2);
    spi.write(0x2C,dc);
    spi.write(g.buffer);
    ce.set();
	
	//I tried shifting the buffer by 8 bits - some colors correct but others lost
	//offsetBuffer = new Uint8Array(g.buffer, 1);
	//spi.write(offsetBuffer);
  }

[espruino-discuss3]

Posted at 2014-11-07 by @gfwilliams

Thanks - that's a really neat idea!

Unfortunately LSB in SPI is for the bit order, not the byte order. I guess there should really be an option when initialising Graphics, but for now you could actually cheat and set the colour that you want right at the start...

setColor takes either r,g,b or the actual bit representation of the colour, and getColor returns the bit representation. You could do setColor/getColor on the first graphics context, and could then flip the bytes around for the next one (the first context would have to be 16 bit too).

  LCD.drawStringBuffered = function(str, x, y, fontSize, r, g, b){
    //Create a dummy Graphics object to get size of string in fontSize
   var g = Graphics.createArrayBuffer(1, 1, 16);
   g.setColor(r,g,b);
   var col = g.getColor();
   col = ((col>>8)&0xFF) | ((col<<8)&0xFF00); // flip bytes
   //....
    g = Graphics.createArrayBuffer(width, height, 16);  
    g.setColor(col);
    // ...

---

Posted at 2014-11-07 by DaveNI

Thanks - that sounds perfect! I'll try it out this evening and may write a completely new ILI9341 module which is backwardly compatible with the existing one but doesn't use any callbacks.

Setting the background colour could be an issue but I could query the display to get the existing background (probably just one pixel needed assuming the background was from a simple fillRect).

The SPI speed should be the only bottleneck now - clear() takes a second or so but then that is 320 * 240 * 16 = 1,228,800 bits!!

An option when initialising Graphics would definitely be cleaner (some day!) but your suggested method should have very little impact on performance.

---

Posted at 2014-11-08 by @allObjects

Speed paid with memory... or memory paid with speed. @DaveNI, do you have some memory profiling for this approach?

---

Posted at 2014-11-10 by DaveNI

@gfwilliams : I implemented your suggestions at the weekend and am really pleased with the results: Drawing text takes 1/4 of the time taken by the stock ILI9341 module and the colours are now correct - thanks.

I have re-factored my module to make is easier to use but have one question:

Is is possible to get details of the current font from a Graphics object (e.g. is it Bitmap/ Vector/ Custom, vector font size/custom font bitmap etc)? I'm sure this is held within the object but just not surfaced (maybe there a way to access it?).

Without these details I need the user to set the font via a new method: setFont(nameOrSize). This sets the font of the LCD object (using eval to call the setFontNxN() method for custom fonts). I then save the nameOrSize parameter to use when setting the font of my internal buffer Graphics object.

@allObjects : You are 100% correct about memory - even one character in a large font can use a lot of ram. I have taken this into consideration and the connect method now takes an additional parameter: maxBufferSize. If the string requires a buffer larger than this (based on width/height) I draw the characters individually (an individual character can still cause this to be exceeded - if the font is very large).

I will post the code later when fully completed (maybe someone could review the code and if OK add it to the modules library - its a really nice display, just a bit slow at present)

---

Posted at 2014-11-10 by @gfwilliams

Is is possible to get details of the current font from a Graphics object

I'm afraid not. If you want to maintain backwards compatibility you could do:

var g = ....;
g._setFontVector = g.setFontVector;
g.setFontVector = function(s) { this._setFontVector(s); save s somewhere; }

It'd be great if you could contribute something! As you say, it's a bit too slow right now. I wonder if it can be put into a 4 or 8 bit mode? That would at least double the throughput when clearing.

Ideally they'd have a 'fill the current window with colour x' capability, but I never found one :(

---

Posted at 2014-11-10 by DaveNI

Great idea - that would obviously work for setFontBitmap too. I'll have to think about the custom fonts - if I "override" setCustomFont in a similar way maybe I could trap setFont8x12 etc. (its really cool the way custom fonts are implemented - I was impressed)

Although I've programmed for years (vb, delphi, c# ...) I'm new to javascript (very basic knowledge before getting my Espruino - learning a lot)

---

Posted at 2014-11-11 by DaveNI

Thanks Gordon, "Overiding" the functions was just what I needed to make my buffered ILI9341 module backwardly compatible with the existing one.

The module has two additional methods:

setUseBuffer(true/false): If true then use buffering, otherwise use callbacks (default=true)

setMaxBuffer(size): Sets the maximum size of the graphics buffer (number of 16 bit pixels). This defaults to 2000 (i.e. 4kb, similar to the 1bit PCD8544 84*48 display).

The full string is generally written in one go but if the maxBufferSize will be exceeded it is written one character at a time (It may make more sense to have the maxBufferSize in KB).

The buffer is only used during the drawing process but you may wish to lower it if memory is an issue. (NOTE If set too low, a single character in a large font could exceed maxBufferSize, I suppose I should really revert to non-buffered mode if this arises)

I've attached the module (DL_ILI9341.js) and a simple demo comparing buffered with unbuffered.

Hopefully anyone with one of the displays will find it useful - The demo takes 7.6 seconds unbuffered and 1.7 seconds when using buffering. Just set the appropriate pins in onInit().

---

Attachments:

• DL_ILI9341_Test.js
• DL_ILI9341.js

---

Posted at 2014-11-11 by @gfwilliams

Wow, this looks great - thanks!

---

Posted at 2014-11-11 by @allObjects

Interesting:

  //Added "*1" to prevent minifier removing brackets
  function swap16(val) {
    return ((val & 0xFF) << 8) * 1 | ((val >> 8) & 0xFF) * 1;
  }

Gives me hints for solving a similar problem I face in http://forum.espruino.com/conversations/256957 ,
where minifier removed parentheses and Espruino JS choked on their absence... ;-) ...how can you choke on something that is not there... ok: choking on the statement body not greased with parentheses...

var f0 = function() { console.log("f0()"); };
var f1 = function (b) { if (b) for (...) f0(); } ;

in code in Sandbox read with minification on fails (still?) to NOT execute the for statement for

f1();

---

Posted at 2014-11-11 by DaveNI

I have come across something similar to that - its because the minifier identifies that the final semi-colon in a function is superfluous and removes it. The Espruino interpreter chokes if there is no semicolon after a loop (in my case a while loop). I added a "return 1;" statement after the loop and all was fixed. Hope this helps.

---

Posted at 2014-11-11 by DaveNI

Maybe we should compile a list of things to avoid if you want your code to minify correctly (regardless of whether to problem lies with closure or with Espruino)

---

Posted at 2014-11-11 by @gfwilliams

@DaveNI do you have an example of the (minified) code that breaks Espruino?

Things like the loop are probably very easy for me to fix - once I know about them!

---

Posted at 2014-11-11 by DaveNI

@gfwilliams I wasn't suggesting there are many, the interpreter seems extremely "fluent" - I have only come across 3 issues:

The first was when closure decided to insert a labeled statement when I used a break or return within a loop - I can't remember the exact code, but I wouldn't expect labels to be implemented in Espruino and if this does occur it is extremely easy to see within the error message.

The second was as described above - closure identifies that the final semi-colon in a function is superfluous and removes it - Espruino expects a semicolon after a do.. while (and possibly after for loop as @allObjects experienced). The following code is valid but breaks when using simple optimizations

function test(){
       var i=0;
       do{
             console.log(i++);
       }
       while (i < 10);
}
test();

The final problem was with brackets being removed by the minifier - the precedence of operators must be evaluated slightly differently by Espruino. This was probably the most difficult to debug because an error wasn't thrown (correct answer is 256 but returns 0 when minified because outer brackets are removed by minifier )

function swap16(val) {
        return ((val & 0xFF) << 8)  | ((val >> 8) & 0xFF) ;
}
console.log(swap16(1));

I hope this simplified code helps - although minor its the sort of thing that may put beginners off the platform and that would be a pity because it can be extremely enjoyable to work with.

---

Posted at 2014-11-11 by DaveNI

The minifier used by Espruino - Google Closure ( http://closure-compiler.appspot.com/home ) is excellent & I think it makes sense for the IDE to make use of it's API but it is strange at times:

I've just noticed that when I minify the first example above using Simple Optimization but select the "Pretty Print" option the semi-colon isn't removed!

I thought "Pretty Print"only affected white space?

(It still removes the brackets in the second example though.)

---

Posted at 2014-11-12 by @gfwilliams

Thanks - I really want to get these kind of things sorted - as you say, they're pretty minor (and often easy to fix) but they effect the overall impression of the interpreter. If you paste some JS code in, you expect it to work.

The do...while thing is now fixed, and will be in 1v72. It was actually a symptom of a bigger issue, so I'm really glad to have that fixed.

I'm still working on fixing the precedence thing - it's interesting, as I'd believed that & and | had equal precedence, but it turns out they don't. It seems it's even the same in C as well. I guess I'd never come across it because I tend to bracket things if precedence wasn't obvious - you learn something every day :)

---

Posted at 2014-11-12 by DaveNI

Agreed - I tend to use brackets too - I can usually only remember that * / comes before + -

I could never remember all this accurately: Operator_Precedence

---

Posted at 2014-11-12 by @gfwilliams

I've just fixed the precedence issue - in the process I found a way to make the interpreter faster and use a bit less stack too. It would be really handy if people could try out this build and see if anything is horribly broken?

It passes all the tests, but part of me thinks it shouldn't :)

---

Posted at 2014-11-12 by DaveNI

Sounds good - I'm currently at work but will definitely try it later this evening.

---

Posted at 2014-11-12 by @allObjects

Will test this build and see what it does to me. I had the challenge with left-right-shift (<<,>>), added parentheses to overcome the issue, but when minified, it removed them because of precedence they are not needed.

It is great to have all these tools - and their view and implementation of javascript - but if they are not on the same page what language definition is, it becomes Russian roulette whether will work or not. As much I like the 'adamant' - to the error - ;-) reminder to add semicolons in IDE, I'm not expecting the Espruino interpreter to depend on that (I mean, if it means to save a lot - with the definition of 'a lot' by @gfwilliams's judgement - then so be it).

The reason I plea for as much a possible compatibility with JS in the Web world is because of the huge pool of people familiar with the JS world from there. There exists still this perception JS is not a language for serious... and it is complicated and a night mare... even more so among programmers of other languages.

Making it work the way users are, is the best and helps to establish a good reputation.

---

Posted at 2014-11-12 by DaveNI

@gfwilliams I tested 1v72 - it seems fine, I tried a few projects and did not get any errors.

Both issues appear to be fixed and it performs better than 1v71 but still slightly slower than 1v70 (by about 10% as you said earlier).

I really appreciate how quickly you managed to sort these issues out.

---

Posted at 2014-11-13 by DaveNI

@gfwilliams - Further to my reply above I have noticed an issue with 1v72 (it may also happen with 1v71, I haven't had a chance to try). Its regarding the RTC after a hard reset.

Power up Espruino and try this code (it should take 3 and a bit seconds):

function test(){
     for (var i=0;i<10000;i++){
        var j = new Array(10000);
        j.fill(0xFF);
      }
}
var t = getTime();
test();
console.log(getTime()-t);

Now disconnect IDE, hard reset & run again - zero duration!

(I hope the code above has no syntax errors, I cant run it just at present)

---

Posted at 2014-11-13 by @gfwilliams

If you keep querying getTime() I think you'll find it reads the same value all the time?

I've had this - but not specifically after a hard reset.

I think it happens when the device has lost power, but not completely gone to 0v - eg. if you unplug it and re-plug it.

The low speed oscillator stops, but the registers are still set up - so when Espruino starts up it thinks everything is working great and doesn't reset the RTC - but actually the oscillator isn't running any more.

---

Posted at 2014-11-17 by DaveNI

I came across the following link about speeding up SPI and was wondering if something similar could be implemented in Espruino (maybe it already is - write vs send?).

http://developer.mbed.org/users/Sissors/code/BurstSPI/

I thought it could speed up writing large amounts of data to the ILI9341 but not too sure if it is relevant?

---

Posted at 2014-11-17 by @gfwilliams

Yes, that's what SPI.write does already I'm afraid :)

With a bit of work, writes could be made a bit faster by being able to do it in the background, but I think you'll still be disappointed. The best thing to do would be to build the driver into Espruino itself (or to be able to make it from native code).

---

Posted at 2015-08-26 by fms1961

Ok, it's an old thread, but the question is still remaining ...

@DaveNI nice work with your faster display code!

But there is one problem: when writing text, the background is always white, because of the fact that the call of "getPixel" always returns "(255,255,255)". Is there any solution in sight?

---

Posted at 2015-08-27 by @gfwilliams

Maybe just replace getPixel with getBgColor()? I think that should probably have the right effect.

I'd be really interested in seeing an updated one of these that works with the 'paletted mode'. Since all the data is stored in internal memory, just the area that changed could be sent to the display.

I wonder whether this is something that the graphics library could help with (keeping track of the area that changed) and then all the LCD libraries with offscreen buffers could take advantage of it.

---

Posted at 2015-08-27 by fms1961

Unfortunately not ... getPixel tries to achieve the color at the given coordinates x, y, while getBgColor()has no coordinates as parameters at all. The getPixelreturn value is always [255,255,255] not being aware of the selected color format.

I've looked through a dozen sources of different ILI9371drivers in the net - and compared to the implementation the code seems pretty correct. So it's not easy to analyze where the unuseable return values are from. I guess this might be a slightly wrong setting of the display mode, so it returns only the whitecolor.

---

Posted at 2015-08-27 by @gfwilliams

The issue is that the 9341 driver is one-way - it never actually reads anything back from the 9341 at all (in fact on a few devices I've seen I don't think the MISO pin is connected at all) - so it has absolutely no way of getting the current pixel colour.

It'd be much easier to use the current background colour. After all, presumably you as the programmer know what colour the background will be where you're writing text, and can set it accordingly.

---

Posted at 2015-08-27 by fms1961

Well, the ILITEK ILI9341 specifications tells us another story - so you think that there are displays in the wild which do not support all the abilities the driver chip provides? This is bad new ... But I do not understand why there are modules available which provides a getPixel(x,y)function - so the function will not work at all, even in the module atatched here by @DaveNI - this is puzzling. But ok, so I will switch to "plan B" ... ;-)

BTW: did I mention that I love the Espruino? Great piece of work ... tiny, easy to handle, extremely flexible ... it's my IoT favorite ...

P.S.: After reading again also in the teensy forum, Paul Stoffregen is claiming that the readPixelfunction will work correctly ... I'll give it a try with my teensy 3.1 ...

---

Posted at 2015-08-27 by @gfwilliams

Thanks!

I think the ILI9341 itself always work, but you may find that the MISO pin (the output from the ILI9341) isn't connected or even brought out on some displays.

On top of this, the Graphics.createCallback function doesn't allow you to specify a getPixel function at the moment, so it'll never work on Espruino unless you're using one of the boards with LCD support compiled into the chip (like the HY ones). You could write your own getPixel function that doesn't go through Graphics and stick it into @DaveNI's module if you desperately wanted though -but the 'Plan B' is probably more desirable as it's faster (and that's kind of the point of his module).

---

Posted at 2015-08-27 by @allObjects

Check this for reading pixels back. It is correct that the hardware has to be connected of course... ;)

---

Posted at 2018-01-24 by mrQ

hi,
the ILI9341 needs some performance tweak to allow fast visible feedback after user interaction (button press).
imho the paletted driver is no solution, due to it's very slow .flip(). even when combined with .getModified() there remains the trade off between high memory consumption and low color depth. and mostly it is not necessary to keep the display data all the time in memory.

running just a partial display update, with data generated by Graphics.createArrayBuffer() does the job quite fine.
unfortunately the spi.write() blocks execution until everything sent. this prevents interlacing of drawing and spi output - which would bring some noteable performance boost.

is there any possibility to make the spi.write async?
either with a callback (which allows the app to do some buffer/job managerment), or at least with a simple option "please dont block" (so that the app hands over the buffer and forgets it)?

---

Posted at 2018-01-24 by @gfwilliams

Hi - which device are you looking at using this on?

I think realistically having async SPI is some way down the priority list because of the complexity of handling DMA on all the different platforms Espruino is supported for, but yes - it would be a really big help in cases like these.

---

Posted at 2018-01-24 by mrQ

hi - i am using the EspruinoWIFI

---

Posted at 2018-01-24 by @gfwilliams

Have you had any luck pushing higher bit rates on SPI? The display should be able to take data quite quickly, and the Espruino WiFi should be able to push data out at 10MHz or more?

---

Posted at 2018-01-24 by mrQ

i am using 5MHz; when setting to anything higher it still works, but does not work faster. seems that the espruino limits the SPI to 5MHZ?

and yes - i am using the connection plan from the official example http://www.espruino.com/ILI9341

---

Posted at 2018-01-24 by @gfwilliams

I guess it's possible that Espruino struggles to keep the higher speed SPI fully loaded with data. There's an issue open for adding DMA (mainly for non-glitchy neopixels while getting WiFi data), and I just updated it to mention this: espruino/Espruino#1212

I can't promise when I'll have time to do something about it though. At the moment I'm not sure what to suggest - however with the WiFi's extra RAM you could use the paletted version and:

• tweak it to use bigger RAM buffers
• Use Graphics.getModified to get the modified area, then drawImage to draw it into a new Graphics instance of the right size and write just that instance to the screen.

While it would add some delay to the output, it would provide a much smoother, less glitchy update - especially for button presses.

---

Posted at 2018-01-24 by @allObjects

There is just not enough memory available in a regular Espruino envrironment to double the memory the ILI9341 has: 172KB. And that would be needed twice to make a dive and push out changed regions... In other words, this is way beyond the flip-operation-mode resources.

When working on ILI9341 driven displays, the main issue was to have overlay and recreate the are. I resorted to backup the display controller's memory, used the backed up area for some pop-up kind of thing, and then restore the area (see post ). Alternative is redrawing the whole screen if the area is 40+%. The modular UI code I have is enabling this function already. But as everything it is not really snappy. Snappy it would become when using 8..16 parallel data, but that's not really an option, since are talking SPI.