Alexis Rondeau

Bookmarks

5632 bookmarks · page 8 of 226

Bookmarked from Mathematica @mathemetica

These curves turn linear algebra into intelligence.

Without activation functions a deep network is just one big matrix multiply.

σ(x)=1/(1+e⁻ˣ) squeezes to (0,1); tanh(x) centers at zero; ReLU=max(0,x) zeros the negatives; Leaky ReLU, ELU and Maxout keep a gentle slope so gradients never die.
They are the reason neural nets recognize faces, write code and drive cars.

The simplest nonlinearities often unlock the deepest learning.
Which of these do you reach for first when a new model refuses to train?

Bookmarked from AA @measure_plan

thanks for all the love on this, i'll publish for you to play soon

in the meantime here's a demo of some new mechanics:

  • bending light around a black hole
  • splitting with a prism
  • focusing with a convex lens

more physics - puzzles - levels coming soon :]

Bookmarked from signüll @signulll

we’re using local apple intelligence models a lot these days in our product for a lot of little things & they work incredibly well, especially for personalization & copy.

you can cache these so you create a pretty lovely little experience in subtle places where the app feels tailored than just one to many (all for free!).

it’ll feel cool as hell when they get even smarter.

Bookmarked from Chris McCormick @ChrisJMcCormick

I've started writing my model code as a single, compilable forward_backward function:

1 - No more autograd. The backward math is written out, same as forward.
2 - No torch.nn module abstractions, just matrices and math. (Glorious!)
3 - Attach a .gacc to each parameter to store its gradients.
4 - Intermediate activations are explicitly stashed for use in backward.

The screenshot shows the activation stash for the 24-layer nanochat architecture, where Claude cleverly skipped a few with high footprint and low compute cost.

Most notably, the MLP needs the activation values both pre and post-ReLU, and they're 18GB each! We just store pre, and re-apply ReLU in backward.

(A recent modded-nanogpt PR did something similar-but-different, need to look at it more)

I'm not familiar with the backward math (Claude wrote it all), but I'm excited to gradually build up the same familiarity with it as the forward path.

Saved on Hacker News · SyneRyder on GLM-5.3: Frontier coding with emergent cyber capabilities

> ... Anthropic's Project Glasswing is supposed to find them quite a while ago?

That was my thought too. For all of Anthropic's talk about their "adversaries", it seems Z.AI have been quietly offering fixes for single shot Remote Code Execution flaws in US software (Safari / WebKit) that Apple and Glasswing / Mythos missed, and that Apple would not attribute to GLM.