Proposal for the initial API

[pavanky]

This library will sit on top of arrayfire. The default implementation will be C++, however C wrappers will be added on top so that the library can be used from other languages.

The optimization function and distance metrics need to be abstract out so the user has a choice of switching them out whenever possible.

Base classes

• Classifier (Model fit ?)
• Neural Networks

Classifiers

• Perceptron
• Naive Bayes classifier
• Support Vector Machines (SVM)
• Logistic Regression
• Softmax Regression
• Multi Layer Perceptron
• Convolutional Neural Networks (CNN)
• Deep Belief Networks
• Nearest Neighbour Search (kNN)

Dimensionality Reduction

• Restricted Boltzman Machines (RBM)
• Autoencoders
• Principal Component Analysis (PCA)
• Linear Discriminant Analysis (LDA)
• Locality Sensitive Hashing (LSH)

Regression Analysis

• Linear Regression
• Genaralized Linear Regression (GLM)

Clustering

• k-Means

Optimization functions

Needs research

[pavanky]

@umar456 @pentschev @jramapuram Any suggestions about what can go in the first go would be greatly appreciated.

[pavanky]

In particular, additions to Regression and clustering would be nice.

[jramapuram]

I think it makes sense to start with the basics, go with:

• Linear Regression
• Logistic Regression (natural extension due to just a kernel swap)
• k-Means
• Standard Multi Layer Perceptron.

And then take it from there! :)

Personally I would like to maybe do some LSTM's & other RNN type models.
However, for these we will need an optimizer. SGD & ADAM should be easy enough to implement

[jramapuram]

Btw for the suite of neural models it would be good to abstract a standard 'Layer' class. This will make the addition of conv layers/etc pretty easy.

[pavanky]

@jramapuram I was just running through the library design via github issues. I came up with this: #17 for the standard Layer class. It might change once the work actually begins.

[pavanky]

I will look into the optimization functions and update the list. Can you create an issue for base RNN class and / or LSTM and discuss what additional work should be done ?

[jramapuram]

Sure thing. For optimization I suggest paralleling something like keras's solution minus maybe the automatic differentiation that theano offers (thats pretty tough to implement I'd imagine).

[futurely]

@zxie has implemented a very elegant neural network library which maybe was based on the general flow graph described in the Stanford course Deep Learning for Natural Language Processing.

A graph based neural network model can be built with Nodes which do the math with Ops. Currently only one model was implemented with nodes. The library also implements some NNet models without using the nodes.

Caffe is also trying to manage the relationship of layers in a computation graph.

[unbornchikken]

I have some experience with recurrent learning models, like RTLR and LSTM. Those coluld be used for tasks like speech recognition and realtime time series predictions, and so. Do you have plans for supporting such functionality aswell?

[jramapuram]

@unbornchikken : #20

[unbornchikken]

Yeah, but that doesn't get icluded in this list above. Yet?

[UniqueFool]

Caffe is also trying to manage the relationship of layers in a computation graph.

FWIW, there is an increasing number of ML algorithms that are doing tree/graph processing to some extent, so that may be another useful thing to have in arrayfire-ml: graph processing utilities for some of the more common algorithms (e.g. breadth-first search implemented in OpenCL)