Solved – How to increase accuracy of simple regression neural network

I'm (very new, and) struggling to improve the accuracy of a simple neural network to predict a synthetic function.

I started from a neural network to predict sin, as described here: Why does this neural network in keras fail so badly?. I got this working perfectly, but I cannot get the solution to adapt to my own function.

I am trying to predict (something very similar to) the angle theta in

cos(theta) = O / H (H is input, O is constant) 

The closes I have gotten is the below, where 'r+' is training points, and 'bo' is the predictions over those same points.

I've tried thousands of neurons, multiple layers, hundreds of thousands of epochs (I'm not at all worried about over-fitting, my first step is to replicate as near as possible the actual input data). I thought over-fitting to a perfect match would be easy – what am I doing wrong???

enter image description here

Test data here: https://1drv.ms/u/s!AgspNAHPoHu6kDQHyOQzkPCt8fop

import math import random import matplotlib.pyplot as plt  import keras from keras.layers import Dense from keras.models import Sequential #from sklearn.preprocessing import MinMaxScaler  r = Sequential()  def setup_nn():     r.add(Dense(5000, input_dim=1, kernel_initializer="uniform", activation="sigmoid"))     #r.add(Dense(200, kernel_initializer="uniform", activation="sigmoid"))     r.add(Dense(units=1, activation="linear"))      sgd = keras.optimizers.SGD(lr=0.0002, momentum=0.0, decay=0.0, nesterov=False)     adam = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-10, decay=0)     r.compile(loss='mean_squared_error', optimizer=sgd, metrics=['accuracy'])  def read_data():     filename = "1d-ik.csv"     with open(filename) as inf:         # Skip header         #next(inf)         colDist = []         colB1Z = []         colB2Z = []         for line in inf:             # Read data, using python, into our features             dist,b1Z,b2Z = line.strip().split(",")             colDist.append(float(dist))             colB1Z.append(float(b1Z) / -80)             colB2Z.append(float(b2Z))          train_X = colDist         train_Y = colB1Z          #y_scaler = MinMaxScaler()         #train_X = y_scaler.fit_transform(train_Y)         train_X = train_X;         return train_X,train_Y  #############################################################################  setup_nn() plt.interactive(False)  X,Y = read_data() r.fit(X, Y, batch_size=10, epochs=5000)  pred_Y = r.predict(X)  plt.plot(X, Y, 'r+') plt.plot(X, pred_Y, 'bo') plt.show() 

So, there turned out to be several steps that significantly improved my result – most of which could have been found by analyzing the working solution (from the linked question) with changes I assumed were better in my code:

1) scaling input (X) to 0-1. In the original code, X is always between 0 – 1, In the code above, input (distance) is from 0-80

2) Removed normalizing of (Y). Contrary to all the advice I have seen – The original code learnt sin(X) * 10 well, but failed learning sin(X). I found even better results when scaling higher (eg cos * 100 worked better than cos * 10)

3) Use the ADAM optimizer. SGD seemed to learn sin ok, but failed on cosine & more complicated graphs

4) More epochs. Sometimes, the ADAM would appear to get stuck, with the error metric even occasionally rising. However, after a while (10-30 epochs) learning would resume and even pick up pace.

The take-away:

Rules of thumb are great, but experimentation is important!

Always work forward from a working solution ONE STEP AT A TIME!!!

Similar Posts:

Rate this post

Leave a Comment