I'm (very new, and) struggling to improve the accuracy of a simple neural network to predict a synthetic function.

I started from a neural network to predict sin, as described here: Why does this neural network in keras fail so badly?. I got this working perfectly, but I cannot get the solution to adapt to my own function.

I am trying to predict (something very similar to) the angle theta in

`cos(theta) = O / H (H is input, O is constant) `

The closes I have gotten is the below, where 'r+' is training points, and 'bo' is the predictions over those same points.

I've tried thousands of neurons, multiple layers, hundreds of thousands of epochs (I'm not at all worried about over-fitting, my first step is to replicate as near as possible the actual input data). I thought over-fitting to a perfect match would be easy – what am I doing wrong???

Test data here: https://1drv.ms/u/s!AgspNAHPoHu6kDQHyOQzkPCt8fop

`import math import random import matplotlib.pyplot as plt import keras from keras.layers import Dense from keras.models import Sequential #from sklearn.preprocessing import MinMaxScaler r = Sequential() def setup_nn(): r.add(Dense(5000, input_dim=1, kernel_initializer="uniform", activation="sigmoid")) #r.add(Dense(200, kernel_initializer="uniform", activation="sigmoid")) r.add(Dense(units=1, activation="linear")) sgd = keras.optimizers.SGD(lr=0.0002, momentum=0.0, decay=0.0, nesterov=False) adam = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-10, decay=0) r.compile(loss='mean_squared_error', optimizer=sgd, metrics=['accuracy']) def read_data(): filename = "1d-ik.csv" with open(filename) as inf: # Skip header #next(inf) colDist = [] colB1Z = [] colB2Z = [] for line in inf: # Read data, using python, into our features dist,b1Z,b2Z = line.strip().split(",") colDist.append(float(dist)) colB1Z.append(float(b1Z) / -80) colB2Z.append(float(b2Z)) train_X = colDist train_Y = colB1Z #y_scaler = MinMaxScaler() #train_X = y_scaler.fit_transform(train_Y) train_X = train_X; return train_X,train_Y ############################################################################# setup_nn() plt.interactive(False) X,Y = read_data() r.fit(X, Y, batch_size=10, epochs=5000) pred_Y = r.predict(X) plt.plot(X, Y, 'r+') plt.plot(X, pred_Y, 'bo') plt.show() `

**Contents**hide

#### Best Answer

So, there turned out to be several steps that significantly improved my result – most of which could have been found by analyzing the working solution (from the linked question) with changes I assumed were better in my code:

1) scaling input (X) to 0-1. In the original code, X is always between 0 – 1, In the code above, input (distance) is from 0-80

2) Removed normalizing of (Y). Contrary to all the advice I have seen – The original code learnt sin(X) * 10 well, but failed learning sin(X). I found even better results when scaling higher (eg cos * 100 worked better than cos * 10)

3) Use the ADAM optimizer. SGD seemed to learn sin ok, but failed on cosine & more complicated graphs

4) More epochs. Sometimes, the ADAM would appear to get stuck, with the error metric even occasionally rising. However, after a while (10-30 epochs) learning would resume and even pick up pace.

The take-away:

Rules of thumb are great, but experimentation is important!

Always work forward from a working solution ONE STEP AT A TIME!!!

### Similar Posts:

- Solved – Network in Network in keras implementation
- Solved – Network in Network in keras implementation
- Solved – Rescaling neural network sigmoid output to give probability of binary classification for a chosen threshold
- Solved – Training a neural network on chess data
- Solved – Training a neural network on chess data