Hyeongrok Ryu Engineering Portfolio

AI · Deep Learning · Computer Vision · Discriminative Models

OUTTA Basic — Neural Networks for Function Approximation and Nonlinear Classification

2026-08-01 · updated 2026-08-01 · Hyeongrok Ryu

I compared sigmoid-basis function approximation, spiral decision boundaries, and a stalled classifier to understand network capacity and training failure.

Series
2024 OUTTA AI Bootcamp Basic · 2
Type / level
study-note · beginner
Tools
Python, PyTorch, Matplotlib
01Sigmoid as a basis

A checkpoint in the study sequence for this note.

02Loss and backpropagation

A checkpoint in the study sequence for this note.

03Classifying spiral data

A checkpoint in the study sequence for this note.

04Reading a failed run

A checkpoint in the study sequence for this note.

A compact concept path generated from this post's table of contents.

Sigmoid as a basis

I stopped thinking of a network only as a neuron count and rewrote it as transformations followed by a weighted sum. The activation section in the 71-page neural-network module led me to compare how weights and biases shift the slope and position of each sigmoid.

Target cosine-like function and a poor initial approximation
Before the basis functions were adjusted, the approximation missed the target curvature.
Target periodic function and a closely matching learned approximation
The weighted sigmoid combination reconstructed the target bends.
Ten sigmoid basis functions with different positions and slopes
Simple units become a useful basis when their locations and directions differ.
Several sigmoid curves and the full model output as a thick dashed line
The thin basis curves combine into the dashed model output.

The hidden-layer output became easier to connect to the figures when I wrote

ŷ(x) = Σⱼ vⱼσ(wⱼx + bⱼ) + c

Loss and backpropagation

I measured the distance between the approximation and target with mean squared error and propagated it to every parameter. The stored curve approaches zero near 2,000 epochs; I used the trend and oscillation rather than the epoch count alone.

Training loss decreasing over two thousand function-approximation epochs
The loss retained some early noise while its overall trend fell toward zero.

Classifying spiral data

The three-class spiral made the need for hidden nonlinear layers visible. I passed logits, not softmax probabilities, to CrossEntropyLoss and printed the (batch, 3) output shape first. As training progressed, the boundary bent between the intertwined classes.

Three colored classes intertwined in a spiral dataset
No single straight line can separate these classes.
Spiral-classification loss decreasing over one thousand epochs
Batch noise remained, but the loss trend moved downward.
Spiral-classification accuracy increasing over one thousand epochs
Accuracy improved with substantial per-step variation.
Longer spiral-classification loss curve through six thousand epochs
Extending training did not remove the small oscillations.
Noisy spiral-classification accuracy through six thousand epochs
The width of the curve mattered alongside its peak value.
A neural-network boundary following the three spiral classes
The nonlinear boundary follows the inner and outer turns of the spiral.

Reading a failed run

A second spiral experiment stayed near loss 1.10 and accuracy 1/3, the random level for three classes. I checked whether an activation was missing, whether labels used long, and whether the optimizer held the current model parameters.

A second three-class spiral dataset
Similar-looking inputs still require an independent training check.
Training loss oscillating near 1.10 without improving
The loss stalled near the three-class random baseline.
Accuracy fixed at 0.333 for one thousand epochs
The flat line shows that useful learning never began.
Coarse decision regions that do not follow the spiral geometry
The learned regions fail to represent the intertwined structure.

Minimal classifier

model = torch.nn.Sequential(
    torch.nn.Linear(2, 32),
    torch.nn.Tanh(),
    torch.nn.Linear(32, 32),
    torch.nn.Tanh(),
    torch.nn.Linear(32, 3),
)

logits = model(features)
assert logits.shape == (features.shape[0], 3)
loss = torch.nn.functional.cross_entropy(logits, labels.long())

Checks I kept

I read loss, accuracy, and the decision boundary together. If only one looked good, I returned to the model, labels, and split. The retained outputs cover both a useful nonlinear boundary and a stalled network, which made the contrast more instructive than a success-only summary.

Sources used

  • Artificial Neural Networks — course-pdf; perceptrons, activations, and backpropagation
Publication first-page preview