Performance improvement on TA functions

[pegaltier]

The library works well in overall but in terms of performance it has some issue for instance when I load the whole historical data provided by Binance for BNBUSDT in 4h interval I get around 16500 candles and in this case the library take 174 seconds to do the calculation of the squeeze momentum indicator, of course this indicator is a combination of many indicators so I understand it takes a bit of time but maybe not that long.

I suspect its because of two reasons :

1. how the indicators are calculated it has no maximum lookback so the more we advance to recent candles the more it has to calculate the previous bar, for instance by hardcoding a max lookback to 300 in all ocurrence of TechnicalAnalysis class (slice(0).reverse().slice(0, 300) instead of slice(0).reverse()) it takes now 56 seconds to calculate instead of 174 s
2. how the indicators are calculated it redo one complete calculation for each candle, I could suggest instead to refactor the TA functions to use one existing optimized technical indicators library like trading-signals

It could be significantly faster because the candles are streamed and thus calculate only the minimum between each candle

[alaa-eddine]

Hey @pegaltier
Yes I'm aware of this issue, actually the way I'm considering to tackle it is by implementing a cache system that avoids recalculating previous data.
but the problem is not obvious because within the same indicator, you can have functions that rely on the last 4 candles and others that relies on the last 200 (the simplest example is a 50 & 200 SMA cross), and what makes it more challenging, is that many native functions recursively relies on the full candles history.

[pegaltier]

@alaa-eddine i was thinking to simply add an init phase where all the indicators will be initialized in order by the streamed indicators and then the loop will only read the values but it's probably not easy to do and maybe not the best anyway... Do you have plan to work on the library in the following weeks? By the way you have some private key in a config files at the root of the repo!

[alaa-eddine]

@pegaltier Oh thank you for reporting ! fortunatly the secrets were not active but still ! that was a critical leak

As for performance, the suggested solution would not solve the problem and will make the initial load very slow.
I'm not sure to be able to work on this next week, I'm currently busy with a much bigger project that is taking all my time, but will try to address the reported issues here as soon as I have some spare time

[alaa-eddine]

Hey @pegaltier I couldn't push an update to this repo for a while, but I finally got some time to make an update :)
and it's mainly focused on performance optimization.

with this update I observe between x4 to x7 speed improvement, it's very noticable for complex indicators .

The next step will be to implement incremental computation with cache, so you don't need to recalculate an indicator if the input data and parameters are exactly the same.

if you're still using pineTS, I hope that this update help you improve your indicator.

one side effect of this optimization is a loss of precision at the 8th digit after the floating point, (0.00000001), but should not have much impact on the indicators results, I'm still investigating if this is just due to floating numbers representation or an actual loss of precision

[pegaltier]

Hello @alaa-eddine I have installed it and I can confirm there is a significant improvement on performance, I was able to calculate the simple moving average of 69915 candles in half a second! Even if there is still room for improvement compared to others libraries, Pinets now starts to be usable for backtesting. Thank you for this great work! For reference here the benchmark with others javascript library: indicatorts, time: 10ms - @ixjb94/indicators, time: 14ms - ti-engine, time: 47ms - pinets, time: 460ms.

I have seen your notes for future performance optimizations and I have experimented with few of them on my software, here my feedback:
4. Parallel Execution for Independent Indicators (High Impact) -> I have tried the great library called Tinypool which I recommend for sharing indicator calculation on worker threads even if the performance improvement was not that great it was a great experiment.
5. WebAssembly for Compute-Intensive Operations (Very High Impact) -> I have discovered the brand new library called ti-engine which is basically the exact implementation of what you want to achieve.

[alaa-eddine]

Thank you for your detailled feedback @pegaltier I'm happy to see the performance reflected in your benchmarks as well.
The main challenge I have with pineTS is that I need to simulate pinescript timeseries, which involves a lot of overhead (mostly arrays manipulations), and this limits the scope of possible optimizations (WASM is a consideration for the future, workers would benefit if you have indicators running in parallel, but it's not possible within the same indicator because of the timeseries nature).

ti-engine is very good btw, I also used it (actually I used RustTI), but it does not allow porting pinescript indicators, you'll need to rewrite them from scratch and the most interesting ones are really hard to port to RustTI syntax.

The main goal of PineTS is to bring pinescript indicators in a JS environment ... because there are tons of very good community indicators there but locked to tradingview app.

The first step (today) is to use a JS syntax that is very close to the Pinescript language, then in the future, to provide a transpiler that will compile actual pinescript code into PineTS ==> this achieves full portability.

a comparison of current PineTS syntax vs Pinescript for Williams indicator

as you can see, it's almost identical already :)

[alaa-eddine]

hey @pegaltier ,

I released a new update with major refactoring and optimizations,
the optimization is not as dramatic as last time, but it brings another x2 and sometimes x3 compute speed.

If you have some time to give it a try with your same indicator , please tell me if you observe the same enhancement.

[pegaltier]

Hello @alaa-eddine I can confirm that the same calculation of the simple moving average on 69915 candles that was taking 460ms with pinets 0.2.1 is now taking 400ms with the latest release of pinets 0.3.0. Indeed it's not dramatic but it gets better and better. Thanks for your work, looking forward for the next versions!