Stop Wasting Hours: Analyze Any Video with AI
Most people analyze video by sampling frames or reading transcripts, and both approaches miss crucial context. Today I am going to show you the three ways AI can look at videos, the pros and cons of each, and then run a live demo where Gemini understands a silent video of me, down to the equation displayed on my phone.
What you need
- A Gemini API key, created for free in Google AI Studio
- An AI agent to drive the analysis: I use Codex in this demo, but Claude Code works the same way as long as you hand it the API key
- A video file to analyze
1. Level 1: sampling video snapshots (0:15)
The first way AI can look at a video is by looking at snapshots. It takes the video, splits it into images, and looks at one image every couple of seconds to know how things evolve. This is level one for a reason: it is not a very good method, because you lose all the information that lives in the sound.
2. Level 2: transcribing the audio (0:49)
Level two takes the audio from the video and creates the transcript that goes with it. For, I would say, ninety percent of videos, the transcript holds the most information. You can do this with free tools on Mac and PC using speech-to-text models like Whisper from OpenAI or Parakeet from NVIDIA; they work extremely well, and I even use them in my day-to-day on my MacBook.
But this method circles back to the level one problem in reverse: you have everything from the transcript and no visual information. A lot of communication happens through images, and not static images, the evolution over time.
3. Level 3: native video understanding (1:58)
That is where level three comes in: an AI that watches the video the way us humans do. There is only one model out there able to natively ingest video, and that is Gemini, through the API. Gemini looks at the images and hears what is being said at the same time.
4. What Gemini video analysis unlocks (2:24)
This unlocks a lot of use cases. Whenever I upload a YouTube video, Gemini watches it and flags anything secret that leaked on screen. You can repurpose content: point it at an old video, ask when you are in a certain position or spot, and Gemini tags the video so you can cut the sections out. One honest caveat: the timestamping does not work all that well yet and still needs some refinement. For the rest, it is excellent.
5. Hands-on silent video demo (2:58)
To show the true power, I prepared a sixteen-second video of me holding my phone, putting out numbers with my hand, moving my face, and showing an equation on the phone screen. No sound whatsoever. The question is: can the model tell me what happened, and when?
6. Create a test key in Google AI Studio (3:54)
Go into Google AI Studio and click Create an API key. I create one named "test" just for this video, click Create key, and copy it. That is the whole setup.
7. Run the analysis through Codex (4:08)
I drag and drop the video into Codex and tell it what I want:
I'm going to give you an API key for Gemini. What I want you to do is I want you to analyze that video natively. Gemini should be able to tell me what's exactly happening inside of that video.
Then I paste the API key. You can do this with Claude Code, with Codex, really any of these AIs, as long as you give it the API key it needs.
8. Gemini 3.1 Pro results (5:03)
The result, thanks to Gemini 3.1 Pro in this case: exact timestamps of what happens inside the video, despite it having no sound at all. It caught me counting from one on my fingers, lowering my hand, and, most impressively, it read what was on my phone: a very well-known integral I had put up on the screen. That is what AI video analysis with Gemini 3.1 Pro can truly do.
9. The cheaper Gemini option (5:49)
If you need a cheaper version of this, simply use Gemini 3.1 Flash Lite, which works amazingly well. It is what I use for all of my video editing. Take your own videos and start analyzing: recordings to review, content to repurpose, moments to find, edits to automate, all without watching every minute manually.
Pitfalls and tips
- Do not rely on frame sampling or transcripts alone; each one throws away half of what the video communicates.
- Treat Gemini's timestamps as approximate for now; the feature still needs refinement.
- Create a disposable test key in AI Studio for experiments, and drop to Gemini 3.1 Flash Lite when you need scale over maximum quality.
Where to go next
- Turn this into channel strategy with How You Can Grow Your YouTube Channel with Claude.
- Wondering how the Gemini family is doing overall? Watch Gemini 3.6 Flash: Don't Believe the Benchmarks.
