Bookmarks

If you are not your audience, become your audience.

Everybody knows that you should "build for yourself".

But, sometimes an opportunity so ripe presents itself that you would be foolish to pass up on it.

This is exactly what happened with Umax (9m downloads in <1 year).

I knew nothing about looksmaxxing, so I became a looksmaxxer. Bought my first skincare, vitamin c serum, gua sha, and more.

I consumed the content, hung out in the Reddit forums, and began to think like a looksmaxxer.

A large proportion of our success can be attributed to the Madden style rating screen - the screen that has been repurposed in thousands of other apps too. We would have never come to this idea if we hadn't oriented our life around the niche.

Your probability of success significantly increases with domain expertise. If you don't already have it, speed run it.

A Kafka in the cloud doing 30MB/s costs more than $110,000 a year.

A $1,000 laptop can do 10x that.

Where did we go wrong? 👇

The Cloud. Namely - its absurd networking charges 👎

Let’s break it down simply:

• AWS charges you $0.01/GB for data crossing AZs (but in the same region).
• They charge you on each GB in and out. Meaning each time a GB passes, you pay twice - for the one who sends it (outgoing) and the one who receives it (incoming)
• For a normal Kafka cluster with replication factor of 3 and a read fanout of 3x, you are going to be charged:

• 2x for 2/3rd of the produce throughput
• 4x for 100% of the produce from replicating it
• 6x of 2/3rd of the produce throughput for consumption.

(but it can get a lot worse - read until the end to see)

Simple example:

• 3-broker cluster, each in a separate AZ
• 3 producers, each in a separate AZ
• 3 consumer groups with 3 consumers each, each group with consumers in a separate AZ

The producers are producing 30MB/s in total to the same leader.

2/3 producers are in a different AZ, so 20MB/s of produce traffic is being charged at cross-zone rates. 👌

It’s charged both on the OUT (producer’s side) and IN (broker’s side).

The leader is replicating the full 30MB/s to both of its replicas.

This is again being charged both on the OUT (leader’s side) and IN (follower’s side), for both replication links. (60MB/s)

Then, each of the 3 consumer groups has 3 consumers.

All consumers read from the leader, with 2/3 in a different zone.

This results in 20MB/s of consume traffic charged at cross-zone rates PER GROUP. (60MB/s total)

Again charged both on the OUT (broker’s side) and IN (consumer’s side).

The total amounts to 140MB/s worth of cross-AZ traffic. Charged both ways.

When one MB is $0.00001/s, this means we’re paying $0.0028/s. 🤔

That’s:

• $241 a day 😕
• $7500 a month 😥
• $88,300 a year 🤯

It all goes down the drain on network traffic ALONE. 🔥

What about the hardware?

Quick napkin math assuming:

• 7 day retention
• all of the data is on EBS (not using tiered storage since it's not GA yet)
• keeping 50% of the disk free for operational purpose (don't ask me what happens if we run out of disk)
• the 3 brokers are running modest r4.xlarge instances (kinda overkill but hey, why not)

We'd pay:

• $19,440/yr for the EBS storage
• $6,990/yr for the EC2 instances

That’s right - you’re paying just $26.4k/yr for the hardware and 88.3k for the network (3.3x the hardware)

For a total of $115k/yr. 💸

I’m not even counting load balancer costs, which could be $12k by some quick napkin math too.

How ridiculous is that? 😂

Want it to get more ridiculous?

This calculation assumes you’re hosting your own Kafka cluster in the same AWS account.

💡If you use a managed Kafka provider that’s not AWS, or otherwise just another AWS account, you’re typically connecting to them through a public endpoint.

AWS then charges all traffic at the cross-AZ $0.01/GB rate internet traffic rate, even if it's in the same AZ.

The end result?

$113,000 a year for network costs. 💀

For 30MB/s. (!!!)

btw - 30 MB/s is absolutely nothing for Kafka... 🤡

It is most often network/disk bounded.

Doing 3GB/s is not hard. 👌

The higher throughput you go, the more absurdly large this discrepancy between network and hardware cost becomes.

For example - this exact setup could probably do 3x the traffic (90MB/s), assuming storage space isn't a concern.

Then you'd have:

• $264,000 a year for the cross-AZ rate. 🥲
• $339,000 a year for the internet rate. 💀

Why is this cost (more than 300k a year) and complexity (this calculation) the case when three laptops can run this practically for free?

Where did we go wrong?

Worth Noting:

There are a few optimizations that can be done here:

• consumers can use fetch from follower, which results in free read traffic (no cross-AZ charges) in the first example. But the second example would still be charged internet costs. 🤝

• you can avoid internet costs by VPC-peering or Private Link-ing the two AWS accounts. This is largely what most cloud providers do, otherwise it becomes prohibitively expensive. It can be super complex to do. 🔧

• AWS can give you large discounts (up to 90%+ afaict) on the quoted prices, depending on your usage. It’s unclear what customer gets what discount. 💰

And perhaps the best example - you can use an ingeniously-designed product like WarpStream that eliminates all of this complexity and cost. ⭐️

It's no wonder they got acquired after just 13 months of operation.

🚀im proud to announce my first software release!

ComplexTensor: A PyTorch extension for complex-valued tensor operations. Enables efficient handling of complex numbers in neural networks, supporting quantum-inspired algorithms and advanced signal processing tasks.

you can download it now with pip install complextensor

FEATURES:

1. Dual real-imaginary tensor representation
2. Complex arithmetic operations (addition, multiplication, division)
3. Complex-valued activation functions (ReLU, sigmoid)
4. Gradient computation for complex backpropagation
5. Polar form conversion and operations
6. Complex exponential and logarithmic functions
7. Fourier transform operations (FFT, IFFT)
8. Quantum-inspired operations support
9. Hyperdimensional computing capabilities
10. Efficient memory management for complex tensors
11. Compatibility with existing PyTorch operations
12. Custom autograd functions for complex operations
13. Complex matrix operations (e.g., dot product, matrix multiplication)
14. Support for complex-valued neural network layers
15. Serialization and deserialization of complex tensors
16. Added working superposition and entanglement algorithm

github in the comments!

Ya, it’s crazy how the whole creative process has flipped

The idea "just because you can, doesn’t mean you will" has been stuck in my head all day

AI lets one person be all these different roles at once, which seems like it should lead to a wave of high quality work

But just scroll through your feed...the gap between what people can do and what they actually create is huge.

It's like anyone can pick up a paintbrush, but that doesn’t mean they’ll paint a masterpiece. Tools don't replace taste, vision, or understanding how things fit together.

Also, the idea that now, instead of being a specialist, you’re expected to be a one person creative team is a lot

Just cause you can do it all doesn’t mean you’ll do it all well. It’s a different skill set, learning how to integrate all these pieces together. Visuals, sound, narrative, etc

I feel like the whole AI explosion has yet to raise the bar for the average quality of creative work, it’s just given more people access to make stuff

And the fact that you can generate stuff with them doesn't make you a creative person or an "artist"

That still has to be earned