Ran comprehensive link audit across all 1,113 URLs in arf.json.
Removed nodes where the URL has definitively failed (DNS resolution
failure, connection refused, or HTTP 404). Left in place: .onion
Tor hidden services, 403 bot-blocked sites (site still exists),
5xx server errors (transient), and timeouts.
Audit summary:
- Total URLs checked: 1,113
- OK: 829
- Dead (removed): 113 (DNS failure: 61, connection refused: 8, HTTP 404: 49, minus 5 already deduplicated)
- Redirected (kept, flagged in report): 374
- Slow >10s (kept): 1
Also adds tooling:
- tools/link_checker.py -- async URL checker, writes JSON report
- tools/remove_dead_links.py -- removes confirmed dead nodes from arf.json
- tools/link_audit_report.json -- full audit results
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- JSON lint: validates arf.json is well-formed on every PR/push
- Smoke test: verifies index.html local asset references exist
- Link check: weekly scheduled + master push lychee scan of all 1147 URLs in arf.json
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Here's a complete list of removals/changes with reasoning, broken down by subsection:
Social Networks > Facebook > Search
findmyfbid.com - domain has no A record
FB People Directory - returns an error saying "blocked by our security systems"
netbootcamp.org - domain has no A record
searchisback.com - site is inaccessible, site functionality stopped in 2019-ish
socialsearching.info - site is inaccessible since 2018 (wayback machine)
Facebook Live Map - functionality removed from FB in 2019
Social Networks > Facebook > Analytics
stalkscan.com - domain has no A record
Social Networks > Twitter > Search
Twitter Name Search - functionality removed from Twitter
Twitter User Directory - functionality removed from Twitter
Twitter Search for Live Streaming Video - functionality removed from Twitter
conweets.com - domain is parked and for sale
twitterfall.com - site is not functioning, site functionality stopped in 2023
www.twellow.com - is no longer an OSINT resource
tweetreach.com - site is inaccessible
Moz Profile Search - moved to followerwonk.com
hootsuite.com - page no longer exists, other services are not free
tweetdeck.twitter.com - now part of X Pro and no longer available for free
pdevesian.eu - domain is parked and redirecting
twoogel.com - site is not functioning, site functionality stopped in 2021
Treeverse - in 2023 changed from Twitter to Bluesky, so moving to a new category
Social Networks > Twitter > Pictures
twicsy.com - paid-only site to buy followers
Social Networks > Twitter > Analytics > Profile
tweepsect.com - domain parked and redirecting
Followerwonk Analyze - consolidated and moved to moved to followerwonk.com (already in Twitter > Search)
Followerwonk Compare - consolidated and moved to moved to followerwonk.com (already in Twitter > Search)
klear.com - bought by meltwater.com and no longer offering free service
mentionmapp.com - domain parked
sleepingtime.org - domain has no A record, site inaccessible since early 2023 (wayback machine)
bioischanged.com - no longer functioning
X0rz Tweets_analyzer - downloadable Python tool so tagging it as such
socialbearing.com - site no longer functioning
Social Networks > Twitter > Analytics > Hashtag
tagboard.com - no longer offering free services
trendsmap.com - no longer offering free services
Social Networks > Twitter > Location / Mapping
www.tweetpaths.com - domain parked and redirecting
teachingprivacy.com - domain parked and redirecting
echosec.net - bought by flashpoint.io and no free service offered
mapd.csail.mit.edu - redirects to www.heavy.ai/demos/tweetmap so updating this entry
worldmap.harvard.edu - redirects to worldmap.maps.arcgis.com but tweetmap doesn't exist there
Creepy - downloadable tool, tagging it as such
tweepsmap.com - bought by fedica.com, renaming
geotweet.altervista.org - domain has no A record
www.mapd.com - redirects to heavy.ai which is already updated in above lines
Social Networks > Twitter > Archive / Deleted Tweets
deadbird.site - domain parked and redirecting
Social Networks > Twitter
Twitter Back From The Dead - not OSINT, not updated in 9 years, won't work with current API
Social Networks > Reddit
redditmetrics.com - domain has no A record
Social Networks > LinkedIn
LinkedInt - downloadable tool, tagging as such
ScrapedIn - downloadable tool, tagging as such
InSpy - downloadable tool, tagging as such
Social Networks > TikTok
urlebird.com - correcting minor typo in URL
Social Networks > Other Social Networks
raven - works only for LinkedIn so moving to that category
many links in this section are not OSINT but just links to sites, but I'm leaving them for now
Social Networks > Search
www.perisearch.net - domain is parked and for sale
www.periscope.tv - service discontinued in 2021
app.buzzsumo.com - page no longer exists, other services are not free
searchlr.net - domain is parked and for sale
