Path to solve a machine learning problem

[cd155]

@smiths @JacquesCarette While studying machine learning-related topics, I noticed that Drasil may not be far away from solving an optimal problem, such as minimizing mean square errors (MSE), by using gradient descent. Drasil has some knowledge of ODEs, including partial derivatives, which is central in gradient descent. The algorithm runs a for loop, and each time we can use the matrix product to update weights. If Drasil cannot handle matrix products, adding a third-party library, such as numpy.dot, can help.

Solving min(MSE) can help to create a linear regression model, which is widely used in machine learning.

[smiths]

Great to hear from you @cd155! Yes, we could teach Drasil about linear regression in the same way that we taught it about ODEs. We could build on that to teach Drasil about machine learning. We could interface with existing machine learning libraries, rather than generate all the code within Drasil. @JacquesCarette and I have spoken about this in the past. It feels like a very likely direction for future research, but it feels premature to pursue it right now. We have been working with @balacij (current PhD student) to refactor the implementation of Drasil. Building too much knowledge into the current implementation will incur technical debt that we would have trouble paying in the future. 😄

Hopefully we will get to expanding Drasil's knowledge soon. Future students would likely be interested in machine learning projects. If we can remove the drudgery of creating machine learning code, allowing the focus to shift to domain knowledge, we could have an impact in the ML community too. 😄

[cd155]

@smiths Sounds great. One more thing I think is worth pointing out is the gradient descent algorithm. Assume Drasil will have a machine learning study case, and it is a linear regression. It could be solved in closed form, just like we did in the Single pendulum. An alternative is to use the gradient descent algorithm.

In my previous study on the family of greedy algorithms, I found it challenging to encode them in Drasil, as most greedy algorithms require a graph environment. However, the gradient descent algorithm does not require a graph environment. Instead, it requires knowledge of ODEs. In each iteration, the algorithm calculates delta_w and updates w. It involves calculating partial derivatives of w. On top of that, stochastic gradient descent is a variant of gradient descent that incorporates a random factor.

Hope this discussion is helpful.

[smiths]

Thank you for the additional information @cd155. Before adding ML capabilities to Drasil, we would want to do a commonality analysis of the existing software/knowledge. We would then be able to make a plan on how to phase in ML knowledge over time.