Newsfeed

In reply to @SpringStreetNYC · May 17, 2025

(Can't share the one I mentioned because it's a confidential conversation from my wife's work)

But I just did another test with a similar interview I conducted with a friend of mine a few years back.

Here I did the analysis "leg-work" myself, so I can speak to the results from the LLMs.

Setup
Chip: Apple M4 Max
Memory: 128 GB
macOS: 15.4.1

Models

• OpenAI Model: ChatGPT 4o (via web, no customizations)
• Local Llama4 Model (via ollama run llama4)

- parameters 108.6B
- context length 10485760
- embedding length 5120
- quantization Q4_K_M

Prompt
"Check this interview I did with Jonathan. Summarize the overall conversation and extract the most important insights. Here's the transcript: <paste of the transcript>"

Transcript
Text input: ~38,400 characters
Input language: English, casual and inquisitive
Tokens: ~9,820

One-Shot Results

ChatGPT 4o:

• The summary is thoughtful and thourough.
• The insights are on-point and at least as good, if not better, than mine.

Here's the first half of the response

```chat

[Prompt + transcript]

# Summary of the Conversation
Jonathan is a photographer who recently shifted his full focus to photography after ending his coaching commitments. He’s been in “training mode” for the past three months and now feels that he’s reached a new level professionally. His days are packed with client shoots—mainly assisting his partner Lindsey with elopements and couple sessions—leaving little time to edit or build his own portfolio.

He currently struggles most with a backlog of unedited shoots (12+ “messy kitchens”), which prevents him from building his own brand and attracting independent clients. The editing process is particularly time-consuming because he is still developing his personal editing style from scratch and refuses to rely on presets that would limit his learning.

Despite this, he’s highly motivated, journaling his learning daily and constantly experimenting. He values the process and wants to do things the right way, not the fast way. He also described moments of imposter syndrome and anxiety when comparing older work to his improving eye.

The conversation evolved into how his editing bottleneck might be supported, leading to a surprising and heartfelt idea: home-cooked meals by grandmas, delivered or picked up, to reduce the burden of cooking and create space for editing while offering emotional nourishment and routine.

t.co/LP4sQHmzIy

```

Llama4:

• The summary is literal, unimaginative and bland.
• The summary only picks up on one small point ("home-cooked meals") but drops the other 80% of topics.
• There is no mention of insights even though clearly required in the prompt

```chat

[Prompt + transcript]

This conversation appears to be a transcribed interview between two individuals. I'll provide a summarized version of the conversation:

The conversation revolves around the idea of having meals prepared by a grandmother-like figure and delivered to one's home. The speaker seems to
appreciate the idea of having home-cooked meals without the hassle of cooking.

The speaker also brings up the idea of wanting to have meals that are home-cooked and delivered to his place. He seems to like that idea a lot. The
idea is that there will be like meals that are prepared and then delivered.

The speaker also seems to be appreciative of having a meal prepared by someone else, delivered at his own home.

There are lot's of conversation going on. There are a lot's of different meals and lot's of different conversation going on

There are different ideas discussed about meals being delivered and speaker like that idea a lot.

The person also seems like to be appreciative. The person seems to like that idea lot.

He seems like, he seems to be appreciative. The idea seems to like.

This conversation seems to be transcript of two people having conversation .There are lot's of ideas discussed about meals and different kind
conversation. There are lot's of different meal's and lot's of different conversation.

>>> What are the most surprising insights?

The conversation appears to be fragmented and open-ended. Some possible insights that can be gathered from this conversation are:

* The speaker seems to appreciate the idea of having home-cooked meals without the hassle of cooking.
* The speaker likes the idea of having meals delivered to their home.
* The conversation is casual and conversational, with no clear structure or topic.

Some possible surprising insights that can be gathered from this conversation are:

* The speaker's appreciation for home-cooked meals and the desire for convenience.
* The speaker's casual and conversational tone.

However, it's worth noting that the conversation is quite fragmented and doesn't provide a clear narrative or specific insights. If you have more
context or information, I'd be happy to try and provide more specific insights.
```

So maybe I'm missing some obvious config?

• ↗ chatgpt.com/share/6828b2e6…