Neural Network from Scratch in C

392 lines, no ML libraries, and a bug that was not where I looked

role: Solo
when: 2026
loc: 392
files: 7

C
gcc
no ML libraries

Source

By the numbers

81.01%

test accuracy

against a 62.4% majority-class baseline

8 to 4 to 1

architecture, seed 42, reproducible

712 / 179

train / test split, 12 features

Why write it in C

Calling model.fit() teaches you an API. It does not teach you what a gradient is doing to a weight. So I wrote a feedforward network in one C file with nothing but libc and libm: forward pass, backpropagation, stochastic gradient descent, ReLU and sigmoid, He and Xavier initialisation, binary cross-entropy. All of it, including the loops underneath.

gcc -Wall -O2 Nn.c -o nn -lm
./nn

No dependencies. 392 lines. A fixed seed, so every run is reproducible.

Three decisions worth defending

The output gradient is collapsed analytically. Chaining the derivative of binary cross-entropy with the derivative of sigmoid produces terms that cancel exactly, leaving

delta = a - y

SGD() computes that directly. It is simpler, it is one fewer place to introduce a sign error, and it is better behaved numerically than applying both derivatives and hoping the floating point cooperates.

ReLU's derivative is taken on the post-activation value. relu_deriv(a) rather than relu_deriv(z). That works because relu(z) > 0 exactly when z > 0, so the two agree everywhere except at the single point z = 0. It saves storing the pre-activations. It would be wrong for sigmoid or tanh, and the code says so.

Initialisation is chosen per layer type, not globally. He variance (2/n_in) for the ReLU hidden layers, Xavier (1/n_in) for the sigmoid output. Normal samples come from a Box-Muller transform in rand_normal(). Using one scheme everywhere is the common shortcut and it quietly costs you convergence speed.

The bug was not where I was looking

The network would not converge. Loss sat flat, or ran to NaN inside a few epochs. My working hypothesis, for longer than I would like, was that I had got backpropagation wrong, because backpropagation was the part I had written myself and trusted least. The workaround at the time was dropping the learning rate to 0.0001, which flattened the symptom and fixed nothing.

Backprop was fine. load_csv() was not, in three separate ways at once.

The labels were wrong. pandas writes an unnamed index column, so in y_train.csv column 0 is the row index and column 1 is the actual label. The training loop was reading column 0. The network was being asked to push the row index, 0 through 711, out through a sigmoid, and then scored on it with binary cross-entropy.

That same index was also being fed in as a feature. x_train.csv carries it too. It ranges 0 to 711 while every real feature sits roughly in [-2, 9], so it dominated every dot product into the first layer. Saturating activations, exploding gradients.

And the header row was training example number one. The parser had no header handling, and the header line happened to parse cleanly as numbers.

The lesson generalises past this bug. The fault was in the part of the pipeline I never questioned, not the part I was least confident about. load_csv() now takes explicit skip_header and skip_first_col flags instead of assuming the file is clean.

Results

Default configuration, 8 to 4 to 1, learning rate 0.01, 100 epochs, seed 42:

Epoch    0: train loss = 0.6823, train acc = 59.55% | test loss = 0.6860, test acc = 58.66%
Epoch   10: train loss = 0.4504, train acc = 81.74% | test loss = 0.4594, test acc = 81.56%
Epoch   20: train loss = 0.4144, train acc = 81.74% | test loss = 0.4468, test acc = 82.68%
Epoch   50: train loss = 0.3955, train acc = 82.87% | test loss = 0.4508, test acc = 81.01%
Epoch   99: train loss = 0.3877, train acc = 83.15% | test loss = 0.4519, test acc = 81.01%

81.01% test accuracy against a 62.4% majority-class baseline. The baseline matters. Any number at or under 62.4% means the network learned nothing and is guessing the majority class.

Test loss bottoms out around epoch 20 at 0.4468 and then drifts up while training loss keeps falling. That is textbook overfitting, and stopping at epoch 20 gets 82.68% instead. The gap is small only because the network is small and has little capacity to overfit with. Early stopping or regularisation is the obvious next thing.

The performance claim, with its caveat

An equivalent network in TensorFlow trains roughly twice as slowly on this problem. I would rather state that with the caveat attached than let it stand on its own.

At this scale, a handful of neurons and a few hundred samples, the difference is almost entirely framework overhead: graph construction and per-step Python dispatch, not arithmetic. My matrix code is plain nested loops with no blocking, no vectorisation and no BLAS. On any workload that actually matters TensorFlow wins comfortably, and it is not close.