Conversation
DeArrow (https://dearrow.ajay.app) is the crowdsourced database, by the SponsorBlock author, that replaces clickbait titles with community-written honest ones and clickbait thumbnails with a real frame from the video. Requested in #1051. Off by default. An untouched install makes no DeArrow request at all; everything lives under Settings -> DeArrow. The objection in #402 was that reloading titles and thumbnails cannot be done seamlessly on Android. DeArrowBinder is built around that, enforcing three rules at every bind site rather than leaving them to each call site: 1. The original is bound first, synchronously, always. A lookup can never delay a bind or leave a row blank. 2. A replacement is only written if the row still shows the same video, so a slow response for a recycled holder is discarded rather than corrupting the row that replaced it. 3. A cached result is applied with no asynchronous hop, so the visible swap happens once per video and never again. Thumbnails keep the uploader's current image as the Picasso placeholder with noFade(), so a slow or failed DeArrow render shows the old thumbnail rather than an empty box. Any failure at all is a no-op. Lookups use the hash-prefix endpoint /api/branding/{sha256(videoId)[0:4]}, not ?videoID=, so the server sees a bucket shared by ~130 videos and never learns which video is being watched -- the same privacy model this app's SponsorBlock integration already uses. One request also caches the other ~130 videos in the bucket. No new dependency: okhttp, RxJava 3, nanojson and Picasso are already here, and the fetch goes through the existing extractor Downloader so proxy settings are honoured. DeArrowParser has no Android imports, which is what lets the selection rules be covered by plain JVM unit tests. app/src/test was previously empty; it now holds 39 tests whose fixtures are unedited responses captured from the live API, so they fail if the API changes shape rather than passing against a fiction. Two behaviours worth calling out, both matching the browser extension: - A winning submission marked "original" yields no replacement, because that is the community voting to keep what the uploader chose. Promoting the runner-up would show a title the community explicitly did not pick. "Me at the zoo" is the real case and is a test fixture. - Equal votes break deterministically on UUID, so two devices never disagree about which of two tied submissions to show. The only non-mechanical edit to existing code is in VideoDetailFragment.prepareAndHandleInfoIfNeededAfterDelay, which decided "is this page already drawn?" by comparing the displayed title against info.getName(). A replaced title made that comparison always fail and redraw the page on every check, so it now also accepts the DeArrow title for the same video. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
readBucket called parseBucketEntry in a loop, and parseBucketEntry parses the entire response body every time. A bucket holds ~130 videos in about 17 KB, so loading one did ~130 full JSON parses of the whole payload -- quadratic work, repeated for every distinct bucket a screen of results touches. On a real device that was enough to saturate the CPU and produce an "isn't responding" dialog as soon as the feature was switched on. Every unit test was green while this was happening, and the build was clean; only installing the APK and using it surfaced it. parseBucket() now parses the body once and returns branding for every video in it. Two regression tests cover it: the one-pass result must agree entry-for- entry with the old path, and a disabled config must still yield nothing for every video in the bucket. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without this, DeArrow does nothing on most of YouTube. Measured coverage of
community submissions on 2026-09-23: 0/8 for "Typical Gamer GTA 5", 0/8 for
"Fortnite", against 7/8 for Veritasium. So on exactly the content whose
thumbnails are worth replacing, the feature was invisible.
Upstream does not behave that way. From dearrow.ajay.app: "if there are no
submissions, it will format the original title to the user-specified format,
and set a screenshot from a random timestamp as the thumbnail." The default
for that fallback therefore flips from off to on here too.
Two things had to be built, because neither the API nor the thumbnail server
can supply this:
- The branding API returns NOTHING for an unsubmitted video -- it is simply
absent from its hash bucket, so there is no randomTime to read.
DeArrowRandomTime ports alea(videoID), the seeded PRNG the extension and
the server both use, so the timestamp is derived locally. Seeded rather
than random for two reasons: a video must show the same frame every time,
and matching upstream's seed means asking the thumbnail server for the
frame it already has. Verified against the live API -- alea("dQw4w9WgXcQ")
is 0.5678500605281442, exactly the randomTime the API reports.
- The thumbnail server will not generate frames on demand. getThumbnail with
generateNow=true returned HTTP 204 on every attempt across three videos,
five polls each. thumbnailRenderer.ts agrees: it renders locally from the
video stream. DeArrowFrameRenderer does the same with
MediaMetadataRetriever.
Three things that made this silently do nothing, each found only by running it:
- Video-only (adaptive) streams were being filtered out. YouTube serves
almost nothing else, and a frame grab needs no audio, so the candidate
list came back empty every time.
- getVideoStreams() holds only the progressive formats; the adaptive ones
are in getVideoOnlyStreams(). Both are needed.
- MediaMetadataRetriever sends no User-Agent of its own and YouTube refuses
the request, so the app's own User-Agent is passed explicitly.
Cost is bounded: live streams and unknown durations are skipped outright, at
most two renders run at once, frames are cached by video id and scaled to list
size, and every failure is a no-op that leaves the uploader's thumbnail alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes from testing the feature on a real device rather than only in unit tests. DeArrow gets its own icon. It had been reusing ic_sponsor_block_enable, so the two rows in Settings were visually identical. Redrawn from the project's own mark (ajayyy/DeArrow public/icons/logo.svg) as concentric rings, tinted like every other settings icon rather than carrying the brand's blues. Render concurrency 2 -> 6. Each render is dominated by network waiting -- resolving a stream, then range-reading into it -- so a low cap did not save CPU, it serialised latency. It is not the binding constraint on a single screen, since only visible non-live rows ever request a render, but it costs nothing and covers a full screen in one pass. Live streams are deferred rather than shipped half-working. The thumbnail server does render a broadcast's current frame for SOME live videos (generateNow=true returned a real 640x360 frame), but answers HTTP 204 for others, and Picasso turns that into an empty grey box -- strictly worse than the broadcaster's own thumbnail. Repairing it from Picasso's error callback did not reliably fire, so live is skipped outright for now and the row keeps what YouTube gave it. DeArrowLiveThumbnail and its five tests stay in the tree for the follow-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rendering a frame meant resolving a playback URL through the extractor and then
range-reading into the video container — seconds per thumbnail. On a screen of
twelve results only about two landed inside a minute, so almost every row still
showed the uploader's artwork by the time anyone looked at it. The feature passed
its own A/B test and read as dead.
YouTube already publishes what we were going to all that trouble to compute.
Every upload has three automatically-extracted frames at roughly a quarter, half
and three quarters of the way through, served beside the uploader's thumbnail as
plain ~7 KB images with no key, no player request and no deciphering:
https://i.ytimg.com/vi/<videoId>/hq1.jpg (also hq2, hq3)
They are real frames, not crops of the artwork — 0.28-0.34 normalised RMSE away
from hqdefault.jpg, and a similar distance from each other. Which one a video
gets is chosen with the same seeded generator the DeArrow server uses to pick a
timestamp, so the choice is arbitrary but stable: a row that scrolls away and
back does not change picture.
A live broadcast 404s on all three and instead has hq720_live.jpg, 1280x720 and
natively 16:9, holding the current moment of the stream. That is the frame the
extension asks dearrow-thumb.ajay.app to generate, and going to the image host
directly is both faster and far more reliable: that endpoint answered HTTP 204
for every live broadcast tried, on two separate days. It is kept only as a
fallback, behind a check that a real image came back, because handing a 204 to
an image loader paints an empty grey box over a row that had a perfectly good
thumbnail.
The upload frames arrive letterboxed inside a 4:3 box, so the black bars are
detected and removed rather than assumed away — a genuinely 4:3 upload has none,
and cropping it blind would cut the top and bottom off the picture.
DeArrowFrameRenderer is kept as the fallback for videos with no stored frame,
which in practice means a very fresh upload.
Measured on an emulator with a control pass (the same journey run twice with the
feature off, to quantify what moves on its own — 0.000):
live streams 3/3 visible rows replaced at t+0s
uploads 2/2 visible rows replaced at t+0s
60 DeArrow unit tests pass, 7 of them new.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GScc5y1fZP2obD8P3QBtQ
hq720_live.jpg is not a frame. It is present on every live broadcast, it is 1280x720 and natively 16:9, and it is the broadcaster's own thumbnail at 720p — so a row using it had its clickbait replaced by the same clickbait. The measurement that was supposed to catch this did not. Comparing hq720_live against hqdefault reads 0.28-0.40 normalised RMSE, which looks like a real difference and is not: hqdefault is boxed into 4:3 and hq720_live is not, so the whole distance is letterboxing. The two are the same picture. What exposed it was looking at them side by side. What should have raised the alarm days earlier was a live stream whose hq720_live matched its maxresdefault at exactly RMSE 0 — that is what happens when a broadcaster sets no custom thumbnail and YouTube fills both slots from the same source. That zero was recorded and not followed up. Live has no cheap frame source at all: the storyboard sprite sheets uploads carry are absent on a broadcast (confirmed against two), and dearrow-thumb.ajay.app answers HTTP 204 for every live stream tried, now on three separate days. The only thing that yields a real frame is decoding the broadcast at the live edge, so DeArrowFrameRenderer#renderLive does that: resolve the stream, open the smallest rendition, take the frame at time 0 — which on a live playlist is the segment currently being served, since a live playlist holds only a short trailing window. A 240p variant is preferred and costs nothing, because the result is scaled to a list-row thumbnail regardless. Uploads are unaffected and stay on the stored-frame path, which was verified by eye as well as by measurement: hq2.jpg of a GTA video is an in-game cutscene where hqdefault.jpg is the uploader's artwork. Verified on the emulator against a screen of six live broadcasts: every tile is now an in-stream frame — a game interior, a talk-show set, a night scene with emergency lights, a competitor mid-pose — where each was previously promotional artwork. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GScc5y1fZP2obD8P3QBtQ
Reported on a real device: rows went from their thumbnail to a grey box with a play arrow. Cause: the parser synthesised a thumbnail-server URL from `randomTime` for every video that had a bucket entry at all, and the binder handed it to Picasso. dearrow-thumb.ajay.app only serves frames it already holds and answers HTTP 204 for the rest — which an image loader reads as a successful empty response, so it paints its placeholder over a row that a moment earlier had a perfectly good image. A cosmetic feature made the screen strictly worse, and because it only hit videos DeArrow knows about, it looked sporadic rather than broken. This exact failure was already documented in DeArrowLiveFrame's javadoc, found and fixed on the live path in September. The lesson did not get applied to the other path that fetches from the same server, and there was nothing structural to make it: any code holding a thumbnail-server URL could hand it to Picasso. Two changes, so it cannot come back: - The `randomTime` fallback is gone. It existed only because frames could not be produced locally; DeArrowAutoThumbnail now does that instantly, so a URL from the parser is only ever a genuine community submission. This removes the systemic case. - DeArrowImageFetch is the single way an image reaches a view: it fetches the bytes, requires HTTP 200 and a body that actually decodes, and completes empty otherwise. Callers can only receive a real bitmap. Repairing this from an image loader's error callback was tried in September and did not reliably fire. Verified on the emulator against the reported query: the two rows that were grey now show in-video frames, and so does every other row on the screen. 59 DeArrow unit tests pass, including a regression test asserting randomTime never becomes a URL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GScc5y1fZP2obD8P3QBtQ
Every other thumbnail path costs one small image fetch from YouTube's own thumbnail host. A live broadcast has no stored frame anywhere, so the only way to get one is to resolve the stream and briefly open it — a player request plus a media decode for every live row scrolled past. That is a different order of cost from the rest of the feature, and the player request in particular is the call YouTube rate-limits and fingerprints. It should not ride along with "replace thumbnails when nobody submitted one". So it is its own switch, defaulting off, depending on the frame fallback since it is a narrower case of the same behaviour. The setting summary says plainly why. "Live off" is also the default at the API level: the existing DeArrowConfig constructor is kept as an overload that passes false, so a caller has to name the flag to turn it on and no existing call site changes meaning. Separately, the cheap path now runs for EVERY row, including ones the extractor types as live. A finished broadcast keeps its LIVE badge but has been processed like any other upload, so it does have stored frames — branching on stream type before trying them sent every archived stream down the expensive path, and once that path became opt-in, skipped them entirely. Asking for the stored frames is also a better test of "is this actually live" than the type is: a broadcast in progress has none and answers 404. Verified on the emulator, DeArrow on and live left at its default: - the one genuinely live row (LIVE badge, no duration) keeps its artwork - every archived stream on the same screen now shows an in-game frame - ordinary uploads unaffected, still replaced 63 DeArrow unit tests pass, four of them new and pinning that turning on the ordinary frame fallback can never turn on live. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GScc5y1fZP2obD8P3QBtQ
A self-review of the branch returned twelve findings. Ten are fixed here; the remaining two are quality notes recorded on the card. The three worst would all have shipped. MEMORY — three static bitmap caches were bounded in ENTRIES, not bytes. LruCache counts entries unless sizeOf is overridden, and none of them did: 120, 60 and 40 entries of decoded ARGB_8888 is roughly 62 MB, 31 MB and 37 MB. On a 2 GB device, scrolling a long feed could get the app killed — by a feature whose entire job is cosmetic. There is now one DeArrowFrameCache, bounded in bytes at an eighth of the heap shared between its users, overriding sizeOf with getByteCount (not width*height*4, which guesses the config wrong for RGB_565). It also registers a ComponentCallbacks2 trim so the system can reclaim the frames instead of killing the app, and the live path's cache is simply gone — forty entries reserved for a code path that has never once returned an image. LEAK — switchIfEmpty takes a VALUE, so the expensive fallback chain was built on every bind even when the cheap path was about to succeed. Building it is not free: render() and renderLive() register themselves in an inFlight map as a side effect of construction, and a chain that is never subscribed never runs its doFinally. Browsing a few thousand feed items left a few thousand cold Rx chains pinned in an unbounded map. Both branches are now behind Maybe.defer. RACE — on the detail page the replacement was applied BEFORE initThumbnailViews, which is what loads the uploader's thumbnail into the same view, so the image loader painted the clickbait straight over it. It worked only when the DeArrow fetch happened to finish last, and never at all when the frame was already cached. In list rows the same class of bug: writing a bitmap directly does not cancel the loader's in-flight request for that view, so a cache-hit replacement followed by a cache-miss original made the row visibly flip BACK to clickbait. Every write now goes through paint(), which cancels first. COST — the claim that the default costs "one small image fetch, no extractor call" was not true. Only live decoding was behind the expensive-path opt-in; a non-live row whose stored frames 404 — an upload too fresh to have been processed, which is what a subscription feed is made of — fell through to a full extraction plus partial video download, gated only by a switch that defaults on. Both decode paths are now behind the same opt-in, and the setting text says what it actually covers. Also: failures are remembered with a TTL, so scrolling past a video with no frame stops re-requesting it on every rebind; the settings screen's clear-cache action now clears the frames it promises to, not just the titles; the semaphore wait is bounded, so a fast scroll no longer parks dozens of unbounded-pool io threads on it; the settings read is cached behind a preference listener instead of nine resource lookups and eight SharedPreferences reads per row bind on the main thread during a fling; and the letterbox "detection went wrong" guard, which could never fire because the crop cap already bounded it, now tests the thing it meant to. 63 unit tests pass. Re-verified on the emulator: four ordinary uploads still replaced with in-video frames, no grey placeholders, no flip-back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GScc5y1fZP2obD8P3QBtQ
Second review pass. The settings snapshot was being invalidated correctly, but DeArrowParser bakes the title and thumbnail switches into each DeArrowBranding as it parses, and those objects live in a 5000-entry cache. Turning "replace titles" off therefore left every already-seen video still showing its DeArrow title until the app restarted — which reads as the setting being ignored. The preference listener now drops the branding cache and the frame caches too, so the next bind re-parses under the new settings. Also: the class javadoc still claimed settings are read on every lookup rather than cached, which is now the opposite of what the class does, and the dead entry-count constants, unused imports and a javadoc @PARAM for a removed parameter are gone. Verified on the emulator: with thumbnail replacement off, rows show the uploader's artwork; with it on, in-video frames. 63 unit tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019GScc5y1fZP2obD8P3QBtQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.