rafiki6's comments

No technical analysis, but all models experience drift eventually.

Pretty fun! A few questions:

1 - are you planning to let people write their own prompts?

2 - when will you share the model names?

Because OpenAI is a single team with a singular focus.

Google is a behemoth with multiple products and a lot of people with opinions who you have to get through to launch a product.

Also, OpenAI has not unseated Google's dominance in search nor do I see this happening.

More specifically validation loss is irrelevant when you can't even sample out of distribution anymore.

You seem to have a serious attitude problem in your responses so this is my last one.

It's propietary company evaluation data, and it's for a specific domain related to software development, a domain that OpenAI is actively attempting to improve performance for.

Anyways enjoy your evening. If you want to actually have a reasonable discussion without being unpleasant I'd be happy to discuss further.

Did I say that?

The OP's post was saying it's somehow able to solve something new. It's showing a severe misunderstanding how how language modelling works.

I never said: - "it's not me it's the test" - "These systems are not just next token predictors"

None of the papers or blogs you've shared offer any points that actually rebutt what I'm saying.

And yes, we will eventually have them work in real time. Can't wait.

I have no idea what is shifting in real time. I formed this opinion of GPT4 by running it through several benchmarks and making adjustments to them, so my view is empirical and it was formed 1 week after it came out.

Your post says nothing of substance because it offers no substantial rebuttal and seems to just attack a position by creating a hand-waved argument without any clear understanding of how parameters in-fact impact a model's outputs.

You also completely missed my point.

Just because it's newly created doesn't mean that the structure of the language and the concepts it represents are actually new.

It's clear that whatever tests he writes cover well established and understood concepts.

This is where I believe people are missing the point. GPT4 is not a general intelligence. It is a highly overfit model, but it's overfit to literally every piece of human knowledge.

Language is humanities way of modelling real world concepts. So GPT is able to leverage the relationships we create through our language to real world concepts. It's just learned all language up until today.

It's an incredible knowledge retrieval machine. It can even mimick how our language is used to conduct reasoning very well.

It can't do this efficiently, nor can it actually stumble upon a new insight because it's not being exposed in real time to the real world.

So, this professors 'new' test is not really new. It's just a test that fundamentally has already been modelled.

The problem is everyone is doing this.

There's no moat anymore. Building some dumb webapp to sell to people to make their lives marginally more convenient is not sustainable model.

So what do you do now? Seems like the only option is to move towards a life and death industry.

But when everyone does this, it's game over.

It's easy to say embrace it, but we aren't that far off from a fully capable developer. People are already stitching task based flows to make it basically build a whole thing and a big thing.

The majority of the software industry is in a lot of trouble and honestly given that it was one of the highest paying industries that's not great for the economy.

What are people supposed to even do? What's next for all these displaced workers?

That's pretty much what GPT is.

That's an interesting take on what to do personally. I've been thinking the best think to do is just to work hard, keep head down, and if the axe falls, expect that it will likely take at least 1 year to land the next role and it won't be an ideal setup and to just ride it out.

Generally speaking no company will willingly canabalize it's current product line for an unproven and premature technology in a new product line.

That's also usually how companies get displaced. I'm not sure we'll be driving any cars from established manufacturers if EVs gain wide adoption and reach a price level that's affordable to the average person.

I don't quite understand the strategy of sitting and waiting on a technology competitive advantage. Surely, to get the technology ramped up and integrated into their manufacturing process isn't an instantaneous thing?

I do agree that Tesla's entire value is in their battery tech. Their 'premium' cars are actually about on par with Korean and Japanese mid-tier vehicles in terms of quality at best, by every objective measure. But their pricing is luxury level simply due to the cost of batteries.

Voice assistants, chat bots etc. are all premature technologies that are dying slow deaths.

The primary reason is quality control. The way these devices are tested can never truly represent the massive variation which would impact their ability to process and parse sound. For example, the wide range of accents for a language like English. The variations in ambient noise in real world environments etc.

Beyond that, generative language models have only recently become powerful, but they need server side processing which is incredibly expensive for the majority of contexts where an AI is useful. Think of call centers. I HATE when companies try to use voice AI in call centers, thinking it's a good way to save money.

Bank Call Center Phone Cal example:

Voice AI: "tell me, how can I help?" Me: "I'd like to request my final statements for a recently closed account." Voice AI: "I'm not sure I heard that correctly" Me: "Statements for a closed account" Voice AI: "Do you want to close an account?" Me: "Statements" Voice AI: "I'm not sure I can help with that, let me get you to a customer care representative. Please enter or say your 16 digit account number"

What was the point of that? The vast majority of customers know how to use online banking to get information at this point. Why did you make me do this? And then, imagine I get disconnected and need to call back. Go through the same process again. The bank may have saved some money (questionable, as they have already outsourced the call center anyway to somewhere cheap), but they've irked me so much, I'm always ready to switch. To bad all banks are the same where I live.

Point being, the tech is too premature, unfinished and hard to build and it offers questionable value.

Voice AI is mostly useful in situations where I need to be handsfree. I think what SoundHound is doing makes the most sense. Sell your Voice AI as an API to manufacturers who build good quality speakers.

Everything else is pointless right now.

I just want to say I find it very enlightening to see my previous post have dual interpretations because I wasn't precise. It really explains why ancient literature can cause so much grief!

To clear it up, I did mean that despite the massive increase in workers, with today's layoffs most of them have not yet reverted back to their pre-pandemic 2019 headcounts.

Where did all of these workers come from? Overall as an industry we did not actually substantially increase our headcount. We're about at par. Why?

- People did retire/leave the industry due to 2020 layoffs/Covid - Big tech, who had the largest "influx" actually just vacuumed people from other non-big tech (e.g. banks, mid-size, startups, etc.) - Much of the headcount growth was actually in non-technology positions, especially HR, marketing, sales etc. because many companies wanted to capitalize on pandemic growth

It seems to me like we'd probably be better off partnering with domain experts in Genomics who want to build software that can be used across the board. Sounds like an interesting opportunity for a business. I'm open to the idea if anyone wants to chat, let me know. I'm SWE but would want to partner with a Genomics Expert.

Honestly, I think people like you are the right kind of people to start private enterprises and bring along professional software engineers to help you build something incredible in the space.

I wish more fields would just start adopting the product/engineer partnership that Software companies have perfected. Engineers are very good at what they do. Product people are very good at what they do. They need each other to build things. Sure, engineers might know enough about product to get by and product people might know enough about coding to get by, but the reason it works is because each one is an expert in what they do and are equal.

Its no different in finance, healthcare, genomics etc. I'd love to work in a setting where I'm paired with an SME product manager in a domain I have no clue about and they respect my work and I respect theirs and we are partners.

This is one of the biggest factors that made software/internet companies explode. They respected people who build software. They didn't need to. A bunch of MBAs could have easily just decided that the best way to run the company was to treat the people building the product as a cost center. Many did. I think that's probably one of the reason for the lack of innovation and down fall in many old tech companies like HP/IBM.

The ones that treated SWEs properly and valued them accordingly, did very well.