nvidia

/

NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

like 320

Follow

NVIDIA 45.7k

Hey @cmp-nct

Thank you for your feedback.

This appears to be an issue of tokenization or pretraining on rare tokens rather than a limitation in reasoning or memory. I verified that Nemotron-v3 Nano successfully solves the task both with and without thinking when the rare ▣▢▧ tokens are replaced by more common symbols such as -, x, and o.

I did observe a few minor issues in the non-thinking mode: the model sometimes uses a space (' ') instead of \n as a column separator, and it sometimes omits the single - character. Enabling thinking mode, generally eliminates these minor errors.

It's an interesting insight, thanks.
Though it's the first model I've tested in a long time with those issues.
Tokenization issues usually cause mangled UTF8 symbols, however Nemotron is surprisingly consistent - doesn't destroy the tokens just mistakes their locations.

I tested 2 of my visual benchmarks with pure ascii (looks gruesome without utf8) and the model was able to solve the easy one which it previously failed.
However the maze was not solved and it mangled it as well.
Asking it to replace the model with more pleasant characters in utf8 causes the same issue, it places wrong blocks but it keeps the layout size correct.

My guess is that the efficient attention mechanism has a problem with these type of tasks.
Would be awesome if a later update or the next "small" nemotron release could have some evaluation done on such sort of "visual" recalling.

@cmp-nct I had the same behavior with a simple "Reverse this string: .DefaultCellStyle"! It just couldn't find the right tokens to assemble among those in its vocab! It was almost a frustrating torture for it :D
To have some fun and understand why it struggles, I like to paste the several options it's revisiting again and again in https://huggingface.co/spaces/bartar/tokenizers to see what it sees ;) it's really instructive

@cmp-nct I had the same behavior with a simple "Reverse this string: .DefaultCellStyle"! It just couldn't find the right tokens to assemble among those in its vocab! It was almost a frustrating torture for it :D
To have some fun and understand why it struggles, I like to paste the several options it's revisiting again and again in https://huggingface.co/spaces/bartar/tokenizers to see what it sees ;) it's really instructive

It might be related, attention seems to fail on intricate tasks.
Asking it to reverse it fails, that's a PhD level task.
Asking it to split it into chars works :['.', 'D', 'e', 'f', 'a', 'u', 'l', 't', 'C', 'e', 'l', 'l', 's', 't', 'y', 'l', 'e']
But asking it to reverse that (now simple problem) fails again !: ['e', 'l', 'y', 'l', 't', 's', 'l', 'l', 'e', 'C', 't', 'l', 'e', 'u', 'a', 'f', 'e', 'D', '.']

OSS-20B didn't have a problem reversing it correctly but 20B also does not scale context size efficiently.

I really wish it wasn't like that - I don't think the AI can be used for productive tasks. Earth needs open source models, foundation models and good training recipes - I'm really glad to see Nvidia starts giving back a little for the trillions it is earning from it's unique position.

I don't think the AI can be used for productive tasks

I guess such a task isn't actually representative of real world use case if it can call a tool to do it.

20B also does not scale context size efficiently.

I don't really think it's directly link to the number of parameters

But I share the rest with you

I mean look at https://artificialanalysis.ai/leaderboards/models?size_class=tiny%2Csmall%2Cmedium&is_open_weights=open_source
for the AA-LCR benchmark for example, while gpt-oss-20b gets 31%, Apriel-v1.6-15B-Thinker gets 50%