kellenmace's comments

I could do this, but it would introduce other cache invalidation issues. Take this sequence of events, for example:

* The user performs a search, reviews the matches, then closes the browser tab.

* The title or description for one or more of the videos on that YouTube channel are updated.

* 45 minutes later, the user remembers that they wanted to search for something else on that same YouTube channel. They return to VideoMentions Search to perform another search.

If I had sent cache-control headers in the first response telling the browser to cache the results of those HTTP requests, they would not be refetched for subsequent requests and the user would potentially be shown incorrect matches.

By fetching the data fresh whenever the user opens VideoMentions Search in a new browser tab, I guarantee that they're getting correct matches based on the titles, descriptions and spoken words that the videos have at the time the search is performed.

So as it's implemented currently, I'm choosing to take a performance hit in exchange for improved accuracy. And I'm comfortable with that tradeoff since searches seem decently speedy to me when I use the tool.

Thanks for your thoughts, and the alternative client-side caching idea!

Hey @atentaten! YouTube doesn't provide a way to search across all of YouTube based on transcriptions/spoken words, unfortunately. That's why specifying a channel is required. Once the user provides a channel URL, then VideoMentions Search can perform an on-the-fly search of the videos on that channel to identify matches.

Thanks for checking out my project!

I have found that this method is unreliable. If I run a Google search for "the "paradox of choice. You have so many choices, how do you know which one to pick? Even when I recommend to newer developers in the React", no YouTube video results are displayed, despite the fact that that exact string of text exists in the transcript of this video: https://www.youtube.com/watch?v=uQntFkK8Z54

That's why I built VideoMentions Search. It pulls in the transcript for every video within the channel and date range you selected and shows you any and all matches.

Also - even when a Google search does work to find a video based on spoken words, all you're given is a link to that video's page. You can't see all the times that keyword was mentioned within the video and click on one of those mentions to jump to that exact moment in the video, like you can with VideoMentions Search.

If Google/YouTube added those capabilities, then VideoMentions Search would be irrelevant. In the meantime though, I think it's a useful free tool for quickly finding spoken word matches within a specific channel.

Thanks for checking it out!

Thank you! I don't do anything fancy at the moment. The matching videos are presented in reverse chronological order. So you'll see the most recent videos that contain the keywords at the top, then older video matches as you scroll down the page. And the user provides the URL of the single YouTube channel they want to search within.

Thanks for checking out my project!

Currently, YouTube does not reliably include spoken word/transcript matches in search results. You would have to manually open each individual video in a browser tab, open the transcript, hit cmd/ctrl+f and search for the words you're interested in. And if you need to search tens/hundreds/thousands of videos, that would take a very long time.

VideoMentions Search allows you to specify the channel URL, keywords and date range, then does all that hard work for you, quickly locating all videos that contain those keywords.

I hope that's helpful. Thanks for checking out my project!

Hey @pks016 - if you recall a new channel URLs that didn't work, can you please reply with them? Then I can look into why and fix that bug.

Thanks for checking out my project!

Thanks for bringing this up, @atleta– I'll look into why that particular channel URL doesn't work.

Thanks @ojiwan! Yeah, that's precisely what the paid offering on the VideoMentions.com homepage is. It currently emails the user once for each matching video, but in the future, I could certainly add support for grouping them and sending weekly digests, like you're describing.

Thanks for checking out my project!

Nice! Glad to be of assistance :)

Nice! Perfect timing, then! Glad you're finding it useful! :)

Cool @happy_pancake! Nice use-case! Is your project still online somewhere?

Hey @jonplackett! as @alex_smart noticed in another comment, I am performing on-the-fly searches on the client.

Some requests can't be sent from the browser directly to YouTube due to the Content Security Policy directives that YouTube has in place, though.

For example, if you try to run this code to fetch a YouTube video page in the browser console from any non-YouTube site (like Hacker News or VideoMentions), you'll see the it errors out:

(async () => { const response = await fetch('https://www.youtube.com/watch?v=irjc1nJ1eJs'); console.log(response); })();

That's why my app uses serverless API endpoints as a middleman. It works like this:

1. Browser fires off a request to the API endpoint.

2. The serverless function Node.js process spins up, fetches the data from YouTube, returns the response, then spins back down.

3. My app takes the video data in the response, saves it in memory, searches it for matches, and re-renders the UI to show the matches, if any.

I built my app to do as much work as possible client-side, and to use serverless function API endpoints for anything that can't be done in-browser.

I replied to @lewisjoe's comment above this one with some more details about tricks I'm employing to make searches fast.

Thanks for checking out my project!

Yep! You nailed it. My app performs an on-the-fly search on the client. I replied to @lewisjoe's comment above this one with some more details about tricks I'm employing to make searches fast.

Thanks for checking out my project!

Nice job on PhraseFinder, @SteveDR! I am indeed doing something similar for my app. The user enters the YouTube Channel URL, the keywords and the date range, then I perform an on-the-fly search to find the matches.

I replied to @lewisjoe's comment above this one with some more details about tricks I'm employing to make searches fast.

Thanks for checking out my project!

Hey @alex_smart- Yep! You're exactly right. My app performs as much of the fetching and searching work as possible on the client, then calls serverless function API endpoints for the few things that can't be done in-browser.

I replied to @lewisjoe's comment above this one with some more details about tricks I'm employing to make searches fast.

Thanks for checking out my project!

Hey @lewisjoe! YouTube doesn't provide a way to search across all of YouTube based on transcriptions/spoken words. My app performs on-the-fly searches, as @alex_smart noticed in another comment.

My app does as much of the fetching and match finding work that it can client-side. For the few things that can't be done client-side, my app calls serverless function API endpoints to fetch the YouTube channel and video data it needs. Here are the tricks I'm using to make it fast:

- As soon as the "Channel URL" field loses focus, I start fetching the most recent 30 videos on that channel in the background. This way, by the time the user enters the keyword and date range, I've already fetched some (maybe even all!) of the data ahead of time, which means less wait time for them.

- Once a specific video's data (title, description, transcript, etc.) has been fetched once, it is saved in memory. All other searches the user performs from that point on will pull the video data from the in-memory cache, if it's there. Otherwise, it will fall back to fetching the video data over the network. This in-memory caching makes subsequent searches within the same date range (or a shorter date range) take <1 second.

- Network requests to fetch video data are processed concurrently rather than one at a time. So the browser fires off as many as it can in parallel to get them all resolved as quickly as possible.

- As soon as any matches are found, the UI updates to show the user. This way, the user can start scrolling through the matches and reviewing them while the search is still in progress– they don't have to wait until it finishes to start interacting with the matches.

Thanks for checking out my project, and for the kind words! I appreciate it.

Welcome! :)

Hey @messe! I built VideoMentions Search with a combination of custom code and a couple NPM packages that are able to scrape publicly accessible YouTube pages to get the channel and videos data the app relies on.

So because it doesn't use the YouTube API at all, the total monthly cost is $0 :)

That's why I'm able to offer free, unrestricted access to the tool and allow folks to perform unlimited searches.

Thanks for checking out my project! I hope you find it useful!

Hey @silax! Good thoughts. Some requests can't be sent from the browser directly to YouTube due to the Content Security Policy directives that YouTube has in place, though.

For example, if you try to run this code to fetch a YouTube video page in the browser console from any non-YouTube site (like Hacker News or VideoMentions), you'll see the it errors out:

(async () => { const response = await fetch('https://www.youtube.com/watch?v=irjc1nJ1eJs'); console.log(response); })();

That's why my app uses serverless API endpoints as a middleman. It works like this:

1. Browser fires off a request to the API endpoint.

2. The serverless function Node.js process spins up, fetches the data from YouTube, returns the response, then spins back down.

3. My app takes the video data in the response, saves it in memory, searches it for matches, and re-renders the UI to show the matches, if any.

I built my app to do as much work as possible client-side, and to use serverless function API endpoints for anything that can't be done in-browser.

Thanks for checking out my project!

Hey @filoeleven! Great- glad to hear you're finding it useful!

Thanks for checking out my project! :)