diff --git a/Script/README.md b/Script/README.md index c7500ae..1326185 100644 --- a/Script/README.md +++ b/Script/README.md @@ -8,4 +8,17 @@ Happy learning If any bug, missue and logic please let me know with mention the issue also the script was provided -Thanks \ No newline at end of file +Thanks + +# Table Of Content + +- [CNN-News-Scrapping-and-Archive](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/CNN-News-Scrapping-and-Archive) +- [Darkweb-Scrapping](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/Darkweb-Scrapping) +- [Dorking](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/Dorking) +- [Github-Search](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/Github-Search) +- [Linkedin-Script](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/Linkedin-Script) +- [OSM-QUERY-LIST](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/OSM-QUERY-LIST) +- [Red-Team](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/Red-Team) +- [SENTINEL-SAT-SCRIPT-LIST](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/SENTINEL-SAT-SCRIPT-LIST) +- [SOCMINT-Twitter](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/SOCMINT-Twitter) +- [Web-TLD-Enumerate-NS-Check](https://github.com/Jieyab89/OSINT-Cheat-sheet/tree/main/Script/Web-TLD-Enumerate-NS-Check) \ No newline at end of file diff --git a/Script/SOCMINT-Twitter/Readme.md b/Script/SOCMINT-Twitter/Readme.md index 2729079..2bd1b90 100644 --- a/Script/SOCMINT-Twitter/Readme.md +++ b/Script/SOCMINT-Twitter/Readme.md @@ -1,6 +1,41 @@ # Jieyab ft Xquik -Image +Image + +## Features + +- **Multi-source search** — one query fans out to Cookie, Xquik API, + Wayback Machine, and Google CSE in parallel, each result tagged with + which source it came from. +- **10 extraction tools** — tweet search, follower/following explorer, post + (timeline) extractor, article extractor, community posts, tweet replies, + tweet retweeters, geo-tagged search with a map view, Wayback archive + search, and multi-source search. +- **Infinite scroll + inline nested reply threads** — scroll to load more + pages automatically; expand a reply's own replies in place, recursively, + the way X's own UI threads a conversation. +- **Relationship graph** (`/graph`) — a Cytoscape-based node graph. Select a + node to expand its replies, retweeters, posts, followers, or following; + pivot from any tweet straight to its author (no extra request — the data's + already on the tweet); box-select multiple nodes and drag them together; + edges are labeled by relationship (*replied by*, *followed by*, *authored + by*, ...) and a distinct color flags where two different paths through the + graph converge on the same account or tweet. +- **Archive** (`/archives`) — one-click (or auto-) save of a search's full + raw results, including downloaded media, browsable and re-searchable later + independent of whether the source is still reachable. +- **Analytics** (`/analytics`) — sentiment clustering (pro / neutral / con) + over a saved archive, plus who posts most, what's driving the most + engagement, and word frequency — see [Sentiment Analysis](#sentiment-analysis) + below. +- **Account-age forensics** — every result with a numeric X/Twitter ID gets + its account creation date decoded straight from the ID's Snowflake bits + (no extra API call), flagged New / Recent / Established. +- **Security-conscious by default** — strict CSP with per-request nonces, + hardened cookies, a whitelisted SSRF-safe video proxy, per-source request + throttling to protect the logged-in account from rate limits, and + URL-scheme validation everywhere a scraped link is rendered as an image or + embedded in CSS. # Sett up @@ -31,18 +66,109 @@ Check the result in the table 3. Wayback Machine (Cdx API) 4. Goole CSE API (free quota 100 per day u can increase u limit with buy the service) +## Usage + +### Search (`/`) + +Pick a tool from the dropdown, fill in the field it asks for (query, +username, tweet ID, ...), and hit **Run**. Cookie-only tools (marked +`[Cookie]`) always use your logged-in session; everything else lets you +toggle between **Cookie** (your session, no xquik quota used) and **xquik +API** (uses your API key's quota) mode. + +- Scroll down to auto-load more pages on any tool that supports pagination. +- Any card whose post has replies of its own gets an **Expand N replies** + button — click it to thread the conversation inline, as deep as it + actually goes. +- Toggle **Auto Archive** before running a search to save results as you go + (including every scrolled-in page) — see [Archives](#archives-archives). +- **Geo Post Extractor** switches the results view to a map, geocoding each + author's profile location. +- The `↓ JSON` button downloads exactly what's on screen as raw JSON. + +### Graph (`/graph`) + +Same 10 tools, rendered as a node graph instead of a card list. + +- Click a node to inspect its full raw data in the side panel. +- **Expand Replies / Retweets** (tweet or reply nodes), **Expand Posts / + Followers / Following** (user or retweeter nodes), and **View Author + Profile** (tweet/reply nodes — pulls the author out as their own node, + for free) all attach new nodes with a labeled edge showing the + relationship. +- **Shift/Alt/Ctrl + drag** on empty canvas to box-select multiple nodes; + drag any one of them to move the whole selection together. Plain drag + still pans, scroll still zooms. +- A **yellow edge** means two different paths through the graph converged + on the same node — worth a second look. +- **Archive All** / **Dump JSON** export everything currently on canvas. +- Click the **?** button (bottom-left) for the full legend. + +### Archives (`/archives`) + +Browse everything saved from Search or Graph. Pick an archive from the +sidebar to see its full raw results and any downloaded media. Archives +persist independent of whether the original source is still reachable — +useful for content that gets deleted or a session that expires. + +### Analytics (`/analytics`) + +Pick a saved archive to run sentiment clustering and the surrounding +aggregates over it: + +- **Sentiment clustering** — every item with text gets classified **pro** / + **neutral** / **con**, shown as a diverging bar plus per-category tiles + you can click to filter the item list below. +- **Most active accounts** — who shows up most often in that archive. +- **Most engagement** — which items drove the most reply+retweet+like + activity. +- **Word frequency** — a word cloud sized by how often each word appears. +- **Item browser** — every scored item, filterable by sentiment and + searchable by text/author, with the model's confidence (or, in lexicon + fallback mode, the exact matched words) shown per item — a classification + is always inspectable, never a black box. + +## Sentiment Analysis + +Two backends, tried in this order (see `sentiment.py`): + +1. **ML (preferred)** — [`cardiffnlp/twitter-xlm-roberta-base-sentiment`](https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment), + an XLM-RoBERTa model fine-tuned for tweet sentiment across 8 languages + (Arabic, English, French, German, Hindi, Italian, Portuguese, Spanish). + Its underlying pretraining covers roughly 100 languages, so it degrades + gracefully rather than failing outright on a language outside that + fine-tuned set — this is what makes the tool usable for a global + audience, not just Indonesian speakers. Requires `torch` + `transformers` + (see `requirements.txt`) and downloads ~1.1GB of model weights from + Hugging Face the first time it runs. +2. **Lexicon fallback** — a hand-built Indonesian positive/negative word + list with negation handling (e.g. *"tidak bagus"* flips from positive to + negative). Used automatically whenever `torch`/`transformers` aren't + installed, so a lightweight install still has a working — if + Indonesian-only — sentiment feature instead of a hard failure. + +The `/api/analytics/` response always reports which backend +(`"ml"` or `"lexicon"`) produced its results, and the Analytics page's +banner reflects it. Neither backend is a ground-truth classifier — short +text, sarcasm, and irony degrade accuracy either way. Treat results as a +starting point for investigation, not a verdict. + ## Update Note -1. Update infinity scroll and load new data for twitter reply and retweets +1. Update infinity scroll and load new data for each search also in graph 2. Update data corelation 3. Fix business logic flow 4. Monitoring (Soon) 5. MCP (Soon) 6. Add more parameter for enrichment -7. Add no rate limit (throttle) +7. Add no rate limit (throttle) 8. Add Google CSE data source +9. Expand data user profile post, follower and following, reply post, retweet post in graph — +10. Add sentiment analysis for clustering data, pro, neutral, con based on archive data and dump data +11. Add more data source and other parameter (soon) still research +12. Add more detail data source for the context -## Setup +## Settup ```bash pip install -r requirements.txt @@ -50,6 +176,83 @@ pip install -r requirements.txt Edit `config.ini.example` to config.ini +## Installing the ML sentiment model (optional, recommended) + +The Analytics page's sentiment scoring works two ways — see +[Sentiment Analysis](#sentiment-analysis) above. Skipping everything on this +page is completely fine: the app still runs and Analytics still works, just +using the Indonesian-only lexicon instead of the multilingual model. This +section is only for turning on the better (multilingual) one. **You don't +need to know Python or be a developer to follow this — it's copy/paste.** + +**Why a "virtual environment" (venv)?** It's just a private, throwaway +folder for this project's Python packages, kept separate from anything else +Python-related already on your computer. The ML packages (`torch`, +`transformers`) are large and can conflict with other unrelated tools if +installed system-wide — a venv avoids that entirely, and if anything ever +goes wrong, you just delete the `.venv` folder and start over, nothing else +on your machine is touched. + +**1. Open a terminal in this project's folder** (the same folder as +`app.py`). + +**2. Create the venv** (only needs to be done once): + +```bash +python3 -m venv .venv +``` + +**3. Activate it** (needs to be done every time you open a new terminal to +work on this project): + +- macOS / Linux: + ```bash + source .venv/bin/activate + ``` +- Windows (Command Prompt): + ```bat + .venv\Scripts\activate.bat + ``` +- Windows (PowerShell): + ```powershell + .venv\Scripts\Activate.ps1 + ``` + +Your terminal prompt should now start with `(.venv)` — that means it's +active and every `pip install` from here on stays inside this project's +private folder. + +**4. Install everything** (the base app + the ML packages), in this order: + +```bash +pip install --upgrade pip +pip install flask requests twifork +pip install --index-url https://download.pytorch.org/whl/cpu torch +pip install transformers sentencepiece protobuf +``` + +The third line is deliberately its own command — installing `torch` the +plain way can pull down a multi-gigabyte GPU-enabled build depending on your +system, when this app only ever needs the much smaller CPU version. Using +that exact command is what keeps the download small. + +**5. Run the app as usual** (make sure `.venv` is still active — you'll see +`(.venv)` in your prompt): + +```bash +python app.py +``` + +The **first time** you use the Analytics page after this, it will download +the sentiment model itself (~1.1GB) from Hugging Face automatically — this +needs an internet connection and can take a few minutes depending on your +connection, but only happens once. After that, it's cached on your computer +and loads instantly. + +**Next time you come back to work on this project**, you only need step 3 +again (activate) before running the app — steps 1, 2, and 4 are one-time +setup. + ## Run Local Web Server ```bash @@ -66,33 +269,41 @@ Xquik Dashboard Dasboard Home -image +Image Archive -image +Image -image +Image Graph -image +Image Vidio -image +Image Dir Output image -# Help +Sentiment Analysis + +Image + +Image + +Image + +# Help About SnowflakeID -> Twitter userid : https://en.wikipedia.org/wiki/Snowflake_ID About paramater was provided in data and dump with json file type -image +Image Xquik API DOC diff --git a/Script/SOCMINT-Twitter/app.py b/Script/SOCMINT-Twitter/app.py index 3e03083..d94fe66 100644 --- a/Script/SOCMINT-Twitter/app.py +++ b/Script/SOCMINT-Twitter/app.py @@ -10,6 +10,7 @@ import requests as _req from flask import Flask, g, jsonify, render_template, request, Response, stream_with_context, send_from_directory import archive as _archive +import sentiment as _sentiment from xquik_client import XquikClient, XquikError, load_config from wayback_client import wayback_search, WaybackError from google_cse_client import google_cse_search, GoogleCSEError @@ -17,6 +18,7 @@ from id_forensics import enrich_account_age from cookie_client import ( cookie_tweet_search, cookie_follower_explorer, + cookie_following_explorer, cookie_post_extractor, cookie_article_extractor, cookie_community_post_extractor, @@ -31,6 +33,7 @@ config = load_config() MAX_CONCURRENT_REQUESTS = 3 # parallel execution slots ACQUIRE_TIMEOUT = 15 # seconds to wait before returning 429 +MAX_COUNT = 2000 # upper bound on a single page's requested item count _sem = threading.Semaphore(MAX_CONCURRENT_REQUESTS) @@ -47,21 +50,33 @@ _throttle_lock = threading.Lock() _last_call_at: dict = {"cookie": 0.0, "wayback": 0.0} _COOKIE_TOOLS = { - "tweet_search_extractor", "follower_explorer", "post_extractor", + "tweet_search_extractor", "follower_explorer", "following_explorer", "post_extractor", "community_post_extractor", "tweet_replies_extractor", "tweet_retweeters_extractor", "geo_post_extractor", } -def _check_throttle(source: str): - """None if the call may proceed (and starts the next cooldown window); - otherwise the number of seconds still left to wait.""" +def _check_throttle(sources): + """None if the call may proceed (and starts the next cooldown window for + every source in `sources`); otherwise the number of seconds still left to + wait. Checking is atomic across all requested sources — if any one of + them is still cooling down, none of the clocks are touched, so a + rejected multi-source call (e.g. multi_source_search, which hits both + the cookie and wayback clocks) never partially starts a window for the + sources that *did* have room.""" + if isinstance(sources, str): + sources = (sources,) now = time.monotonic() with _throttle_lock: - elapsed = now - _last_call_at[source] - if elapsed < _THROTTLE_SECONDS: - return round(_THROTTLE_SECONDS - elapsed, 1) - _last_call_at[source] = now + wait = 0.0 + for source in sources: + elapsed = now - _last_call_at[source] + if elapsed < _THROTTLE_SECONDS: + wait = max(wait, _THROTTLE_SECONDS - elapsed) + if wait: + return round(wait, 1) + for source in sources: + _last_call_at[source] = now return None # ── Cookie & session security ───────────────────────────────────────────────── @@ -267,30 +282,61 @@ def _filter_by_date(items: list, from_date: str, to_date: str) -> list: return kept -def _multi_source_search(query: str, count: int, from_date: str = "", to_date: str = "") -> list: - # Pagination isn't wired up for multi-source search yet (cookie + wayback - # only, per current scope) — grab just the items, discard the cursor. +def _multi_source_search( + query: str, count: int, from_date: str = "", to_date: str = "", cursor: str | None = None, +) -> tuple[list, str | None]: + """Fans out across every source in parallel. cursor (if given) is an + opaque JSON object of {source: source_cursor} built from a previous + call's returned cursor — each key present in it is a source that still + had more to give, so only those get re-queried. xquik has no pagination + at all (no cursor concept), so it's only ever queried on the first page + (cursor=None); every load-more page after that is cookie/wayback/cse only. + A cursor value that doesn't parse as a JSON object is treated as "no + cursor" (first page) rather than raising — same tolerant-of-garbage-input + posture as the rest of this file's client-supplied-field handling.""" twitter_query = _apply_date_operators(query, from_date, to_date) - jobs = { - "cookie": lambda: cookie_tweet_search(twitter_query, count=count, config=config)[0], - "xquik": lambda: XquikClient(config).tweet_search(twitter_query), - "wayback": lambda: wayback_search(query, count=count, from_date=from_date, to_date=to_date)[0], + + try: + incoming = json.loads(cursor) if cursor else {} + if not isinstance(incoming, dict): + incoming = {} + except (TypeError, ValueError): + incoming = {} + first_page = not incoming + + jobs = {} + if first_page or "cookie" in incoming: + c = incoming.get("cookie") + jobs["cookie"] = lambda c=c: cookie_tweet_search(twitter_query, count=count, config=config, cursor=c) + if first_page: + jobs["xquik"] = lambda: (XquikClient(config).tweet_search(twitter_query), None) + if first_page or "wayback" in incoming: + c = incoming.get("wayback") + jobs["wayback"] = lambda c=c: wayback_search(query, count=count, from_date=from_date, to_date=to_date, cursor=c) + if first_page or "cse" in incoming: + c = incoming.get("cse") # Google has no since:/until: query syntax like Twitter/Wayback do, so # this lane runs unbounded by date — _filter_by_date below keeps # results whose own timestamp it can't verify rather than dropping them. - "cse": lambda: google_cse_search(query, count=count, config=config)[0], - } + jobs["cse"] = lambda c=c: google_cse_search(query, count=count, config=config, cursor=c) + + results = [] + next_cursor_parts = {} with ThreadPoolExecutor(max_workers=len(jobs)) as pool: futures = {key: pool.submit(fn) for key, fn in jobs.items()} - results = [] for key in ("cookie", "xquik", "wayback", "cse"): # deterministic display order + if key not in futures: + continue try: - data = futures[key].result() + data, next_c = futures[key].result() except Exception: continue # a source failing (missing creds, network, ...) shouldn't sink the others results.extend(_tag_source(data, SOURCE_LABELS[key])) + if next_c: + next_cursor_parts[key] = next_c - return _filter_by_date(results, from_date, to_date) + next_cursor = json.dumps(next_cursor_parts) if next_cursor_parts else None + return _filter_by_date(results, from_date, to_date), next_cursor # Whitelist: only proxy Twitter's video CDN to prevent SSRF @@ -325,23 +371,27 @@ def run_tool(): body = request.get_json(silent=True) or {} tool_type = body.get("toolType") mode = body.get("mode", "api") # "api" | "cookie" - count = max(1, min(int(body.get("count", 20)), 200)) + count = max(1, min(int(body.get("count", 20)), MAX_COUNT)) cursor = body.get("cursor") or None # opaque page token from a previous response's nextCursor # Cookie/Wayback calls are throttled to one per 5s per source — checked # up front, before taking a concurrency slot, so a request that's about - # to be rejected doesn't waste one. - throttle_source = None + # to be rejected doesn't waste one. multi_source_search fans out to both + # cookie and wayback internally, so it's checked (and, once it proceeds, + # starts the cooldown) against both clocks at once. + throttle_sources = [] if mode == "cookie" and tool_type in _COOKIE_TOOLS: - throttle_source = "cookie" + throttle_sources = ["cookie"] elif tool_type == "wayback_archive_search": - throttle_source = "wayback" - if throttle_source: - wait = _check_throttle(throttle_source) + throttle_sources = ["wayback"] + elif tool_type == "multi_source_search": + throttle_sources = ["cookie", "wayback"] + if throttle_sources: + wait = _check_throttle(throttle_sources) if wait is not None: return jsonify({ "ok": False, - "error": f"Please wait {wait}s before the next {throttle_source} request — this protects the account from rate limiting.", + "error": f"Please wait {wait}s before the next {'/'.join(throttle_sources)} request — this protects the account from rate limiting.", "retryAfter": wait, }), 429 @@ -368,6 +418,12 @@ def run_tool(): else: data = XquikClient(config).follower_explorer(username) + elif tool_type == "following_explorer": + username = body.get("targetUsername", "") + if mode != "cookie": + return jsonify({"ok": False, "error": "following_explorer requires cookie mode"}), 400 + data, next_cursor = cookie_following_explorer(username, count=count, config=config, cursor=cursor) + elif tool_type == "article_extractor": tweet_id = body.get("targetTweetId", "") if mode == "cookie": @@ -423,7 +479,7 @@ def run_tool(): for label, val in (("dateFrom", from_date), ("dateTo", to_date)): if val and not _valid_date8(val): return jsonify({"ok": False, "error": f"{label} must be an 8-digit date (YYYYMMDD)"}), 400 - data = _multi_source_search(query, count=count, from_date=from_date, to_date=to_date) + data, next_cursor = _multi_source_search(query, count=count, from_date=from_date, to_date=to_date, cursor=cursor) else: return jsonify({"ok": False, "error": f"Unknown toolType: {tool_type}"}), 400 @@ -431,7 +487,7 @@ def run_tool(): # Single choke point: every tool's output passes through here, so the # account-age label, fetch timestamp, and tweet_url all show up # everywhere downstream for free — cards, graph nodes, JSON dump, and - # archives (once the fields are whitelisted in archive.py's _pick_fields). + # archives (which now store the item's full raw shape verbatim). data = enrich_account_age(data) data = _stamp_fetched_at(data) data = _stamp_tweet_url(data) @@ -519,6 +575,22 @@ def archive_list(): return jsonify({"ok": True, "archives": _archive.list_all()}) +# ── Analytics (sentiment / clustering) ────────────────────────────────────── + +@app.route("/analytics") +def analytics_viewer(): + return render_template("analytics.html") + + +@app.route("/api/analytics/") +def analytics_run(archive_id): + results_file = _archive.ARCHIVE_ROOT / archive_id / "results.json" + if not results_file.exists(): + return jsonify({"ok": False, "error": "Archive not found"}), 404 + items = json.loads(results_file.read_text()) + return jsonify({"ok": True, **_sentiment.analyze(items)}) + + if __name__ == "__main__": host = config.get("server", "host", fallback="127.0.0.1") port = config.getint("server", "port", fallback=5000) diff --git a/Script/SOCMINT-Twitter/archive.py b/Script/SOCMINT-Twitter/archive.py index 27d4788..0a0178f 100644 --- a/Script/SOCMINT-Twitter/archive.py +++ b/Script/SOCMINT-Twitter/archive.py @@ -102,24 +102,6 @@ def _download_file(url: str, dest: Path) -> bool: return False -def _pick_fields(item: dict) -> dict: - """Keep only the fields we want to archive.""" - keys = ["id", "text", "full_text", "article_text", "user", "user_id", - "created_at", "reply_count", "retweet_count", "favorite_count", - "view_count", "in_reply_to_tweet_id", "name", "screen_name", - "verified", "is_blue_verified", - "description", "followers_count", "following_count", "tweet_count", - "lat", "lon", "place", - "retweeted_text", "retweeted_by_user", "retweeted_by_name", - "retweeted_by_bio", "retweeted_at", "retweeted_tweet_id", - "iso_date", "original", "statuscode", "mimetype", "length", - "archive_url", "post_title", "post_text", "preview_image", - "result_url", "display_link", "fetched_at", - "source", "account_created", "account_age", "account_age_flag", - "account_age_precision"] - return {k: item[k] for k in keys if k in item and item[k] is not None} - - # ── Core archive runner (runs in background thread) ─────────────────────────── def _run(archive_id: str, tool_type: str, data, query_info: dict) -> None: @@ -139,7 +121,10 @@ def _run(archive_id: str, tool_type: str, data, query_info: dict) -> None: enriched.append(item) continue - record = _pick_fields(item) + # Archive the item's full raw shape (every field the source API gave + # us), not a whitelisted subset — downstream sentiment analysis needs + # the raw record, not just the fields the card UI happens to display. + record = dict(item) record["tweet_url"] = item.get("tweet_url") or build_tweet_url(item) local_media = [] diff --git a/Script/SOCMINT-Twitter/cookie_client.py b/Script/SOCMINT-Twitter/cookie_client.py index f2fcef8..b40a0cd 100644 --- a/Script/SOCMINT-Twitter/cookie_client.py +++ b/Script/SOCMINT-Twitter/cookie_client.py @@ -71,12 +71,78 @@ def _id_str(val) -> str | None: return str(val) if val is not None else None +def _full_text(t: object) -> str | None: + """twikit's `.text` is Twitter's own legacy `full_text` field — despite the + name, X still truncates *that* mid-sentence into a t.co link for anything + past the classic length limit (long-form "Note" tweets, e.g. Premium/Blue + posts). twikit's `.full_text` PROPERTY (a different thing from the legacy + field of the same name) checks the tweet's note_tweet payload first and + returns the real complete text when one exists, falling back to `.text` + itself otherwise — so it's always at least as complete, strictly more so + for long tweets. Swallows a malformed note_tweet shape rather than letting + one tweet's data take down the whole batch.""" + try: + return getattr(t, "full_text", None) + except Exception: + return None + + +def _hashtags(t: object) -> list | None: + """Same note_tweet-aware source as _full_text — a long-form tweet's + hashtags live in the note_tweet entity set, not the legacy entities twikit + falls back to otherwise.""" + try: + tags = getattr(t, "hashtags", None) + return tags or None + except Exception: + return None + + +def _leading_reply_mentions(t: object) -> list | None: + """Tapping "Reply" on X auto-prefixes the compose box with every account + the reply-chain already has tagged — not just the tweet being replied to + — and that prefix is genuinely stored as the literal start of the + reply's own full_text. X's own web/app UI never shows it inline though: + it reads `display_text_range` (the slice of full_text actually meant to + be *shown*) and renders anything before that start index as a separate + "Replying to @x @y" line instead. Skipping this meant our raw `text` + looked like the replier had typed those @mentions themselves — e.g. a + reply that only ever says "Proyek kepentingan, bukan untuk rakyat..." + displayed as if it opened with "@regar_op0sisi @prabowo ...", which is + exactly what looked wrong compared to the tweet on x.com. This only + covers the un-extended legacy text/entities — a long-form Note tweet's + entity indices belong to its own separate note_tweet string, which this + intentionally does not touch rather than risk slicing the wrong string. + Reads twikit's private `_legacy`/`_note_tweet_results` (no public + equivalent exists) — same trade-off already made for `_get_more_replies` + elsewhere in this file. Returns None rather than raising on any + unexpected shape, since this is purely a display aid, never the record + of truth `text` already is.""" + try: + if t._note_tweet_results: + return None + legacy = t._legacy + start = (legacy.get("display_text_range") or [0])[0] + if not start: + return None + names = [] + for m in (legacy.get("entities") or {}).get("user_mentions", []) or []: + idx = m.get("indices") or [None, None] + if idx[0] is not None and idx[1] is not None and idx[1] <= start: + sn = m.get("screen_name") + if sn: + names.append(sn) + return names or None + except Exception: + return None + + def _tweet_to_dict(t: object) -> dict: user_obj = getattr(t, "user", None) d = { "id": _id_str(getattr(t, "id", None)), "created_at": getattr(t, "created_at", None), - "text": getattr(t, "text", None), + "text": _full_text(t) or getattr(t, "text", None), "user": getattr(user_obj, "screen_name", None) if user_obj else None, "user_id": _id_str(getattr(user_obj, "id", None)) if user_obj else None, "user_location": getattr(user_obj, "location", None) if user_obj else None, @@ -85,6 +151,11 @@ def _tweet_to_dict(t: object) -> dict: "name": getattr(user_obj, "name", None) if user_obj else None, "verified": getattr(user_obj, "verified", None) if user_obj else None, "is_blue_verified": getattr(user_obj, "is_blue_verified", None) if user_obj else None, + # Author's avatar/cover/bio — twikit's embedded user object on a tweet + # already carries these, no extra lookup needed. + "user_avatar": getattr(user_obj, "profile_image_url", None) if user_obj else None, + "user_banner": getattr(user_obj, "profile_banner_url", None) if user_obj else None, + "user_bio": getattr(user_obj, "description", None) if user_obj else None, "reply_count": getattr(t, "reply_count", None), "retweet_count": getattr(t, "retweet_count", None), "favorite_count": getattr(t, "favorite_count", None), @@ -94,6 +165,12 @@ def _tweet_to_dict(t: object) -> dict: media = _extract_media(t) if media: d["media"] = media + tags = _hashtags(t) + if tags: + d["hashtags"] = tags + mentions = _leading_reply_mentions(t) + if mentions: + d["reply_to_mentions"] = mentions return d @@ -103,6 +180,8 @@ def _user_to_dict(u: object) -> dict: "name": getattr(u, "name", None), "screen_name": getattr(u, "screen_name", None), "description": getattr(u, "description", None), + "avatar": getattr(u, "profile_image_url", None), + "banner": getattr(u, "profile_banner_url", None), "followers_count": getattr(u, "followers_count", None), "following_count": getattr(u, "following_count", None), "tweet_count": getattr(u, "statuses_count", None), @@ -145,6 +224,16 @@ async def _follower_explorer_async(username: str, auth_token: str, ct0: str, cou return [_user_to_dict(u) for u in followers], _next_cursor(followers) +async def _following_explorer_async(username: str, auth_token: str, ct0: str, count: int, cursor: str | None) -> tuple[list, str | None]: + """Who `username` follows — the other half of follower_explorer. twikit + exposes this as Client.get_user_following, same shape/cursor contract as + get_user_followers, so this mirrors _follower_explorer_async exactly.""" + client = await _make_client(auth_token, ct0) + user = await _resolve_user(client, username) + following = await client.get_user_following(str(user.id), count=count, cursor=cursor) + return [_user_to_dict(u) for u in following], _next_cursor(following) + + async def _post_extractor_async(username: str, auth_token: str, ct0: str, count: int, cursor: str | None) -> tuple[list, str | None]: client = await _make_client(auth_token, ct0) user = await _resolve_user(client, username) @@ -265,6 +354,12 @@ def cookie_follower_explorer(username: str, count: int = 20, config: configparse return asyncio.run(_follower_explorer_async(username, auth, ct0, count, cursor)) +def cookie_following_explorer(username: str, count: int = 20, config: configparser.ConfigParser = None, cursor: str | None = None) -> tuple[list, str | None]: + cfg = config or load_config() + auth, ct0 = _get_creds(cfg) + return asyncio.run(_following_explorer_async(username, auth, ct0, count, cursor)) + + def cookie_post_extractor(username: str, count: int = 20, config: configparser.ConfigParser = None, cursor: str | None = None) -> tuple[list, str | None]: cfg = config or load_config() auth, ct0 = _get_creds(cfg) diff --git a/Script/SOCMINT-Twitter/google_cse_client.py b/Script/SOCMINT-Twitter/google_cse_client.py index bcd9922..be51c17 100644 --- a/Script/SOCMINT-Twitter/google_cse_client.py +++ b/Script/SOCMINT-Twitter/google_cse_client.py @@ -46,6 +46,14 @@ _META_TAG_RE = re.compile(r"]*>", re.IGNORECASE) _ATTR_RE = re.compile(r'''([\w:-]+)\s*=\s*"([^"]*)"|([\w:-]+)\s*=\s*'([^']*)\'''') _TITLE_TAG_RE = re.compile(r"]*>(.*?)", re.IGNORECASE | re.DOTALL) _TWEET_ID_RE = re.compile(r"/status/(\d+)") +_X_HOST_RE = re.compile(r"^https?://(?:www\.)?(?:x|twitter)\.com(?:/|$)", re.IGNORECASE) +_PROFILE_RE = re.compile(r"^https?://(?:www\.)?(?:x|twitter)\.com/([^/?#]+)/?(?:\?.*)?$", re.IGNORECASE) +# Path segments that look like a profile URL shape but aren't a person/org +# account — X's own site-nav pages live at this same depth. +_NON_PROFILE_PATHS = { + "home", "explore", "notifications", "messages", "i", "search", "settings", + "compose", "login", "logout", "signup", "tos", "privacy", "about", +} def _tweet_created_at(url: str) -> str | None: @@ -64,6 +72,26 @@ def _tweet_created_at(url: str) -> str | None: return dt.strftime("%a %b %d %H:%M:%S +0000 %Y") +def _classify_url(url: str) -> str: + """A search result linking to x.com/someone with no other context reads + as "a Twitter profile" whether it's actually a specific tweet, a bare + profile page, or some other X page entirely — this is a real user report: + a result was shown for a keyword match with no way to tell that it was + (or wasn't) an actual tweet permalink. Purely a label derived from the + URL's own shape; never touches the URL/title/snippet themselves. + Returns 'tweet' | 'profile' | 'twitter_other' | 'other'.""" + if not url: + return "other" + if _TWEET_ID_RE.search(url): + return "tweet" + if not _X_HOST_RE.match(url): + return "other" + m = _PROFILE_RE.match(url) + if m and m.group(1).lower() not in _NON_PROFILE_PATHS: + return "profile" + return "twitter_other" + + class GoogleCSEError(Exception): pass @@ -95,13 +123,27 @@ def _row_to_record(item: dict) -> dict: record = {} title = item.get("title") if title: - record["post_title"] = html.unescape(title).strip() + # Kept as serp_title even after _enrich_records below potentially + # overwrites post_title with a live re-fetch — X serves bots a + # generic/gated page for most URLs, so a live fetch of an X link + # often returns less specific content than Google's own SERP/cache + # already had. Without this, that overwrite silently threw away the + # more useful value with no way to get it back. + record["serp_title"] = html.unescape(title).strip() + record["post_title"] = record["serp_title"] snippet = item.get("snippet") if snippet: - record["post_text"] = html.unescape(snippet).strip() + # Google truncates this itself (ends in "…" mid-sentence) — that's + # the SERP snippet as Google's own API hands it back, not something + # this scraper cuts short. Named `description` (not `serp_snippet`) + # to read clearly as "what this page is about" next to post_text + # (the live-fetched og:description, which may or may not agree). + record["description"] = html.unescape(snippet).strip() + record["post_text"] = record["description"] link = item.get("link") if link and _SAFE_URL_RE.match(link): - record["result_url"] = link + record["result_url"] = link + record["content_type"] = _classify_url(link) created_at = _tweet_created_at(link) if created_at: record["created_at"] = created_at # when the post itself was actually made @@ -173,8 +215,13 @@ def _fetch_live_meta(url: str) -> dict: def _enrich_records(records: list[dict]) -> None: - """Mutates each record in place. Runs in parallel — one slow/dead site - shouldn't hold up the rest of the result set.""" + """Mutates each record in place — post_title/post_text become "best known + value," preferring a fresh live fetch over Google's SERP snapshot when + one succeeds. serp_title/description (set in _row_to_record, before this + runs) are never touched here, so Google's original values always survive + even when this overwrites post_title/post_text with something less + useful (X routinely serves bots a generic/gated page). Runs in parallel — + one slow/dead site shouldn't hold up the rest of the result set.""" candidates = [r for r in records if r.get("result_url")] if not candidates: return diff --git a/Script/SOCMINT-Twitter/requirements.txt b/Script/SOCMINT-Twitter/requirements.txt index 0045a81..bd5b8ea 100644 --- a/Script/SOCMINT-Twitter/requirements.txt +++ b/Script/SOCMINT-Twitter/requirements.txt @@ -1,3 +1,7 @@ flask requests -twifork \ No newline at end of file +twifork +torch +transformers +sentencepiece +protobuf \ No newline at end of file diff --git a/Script/SOCMINT-Twitter/sentiment.py b/Script/SOCMINT-Twitter/sentiment.py new file mode 100644 index 0000000..0c03054 --- /dev/null +++ b/Script/SOCMINT-Twitter/sentiment.py @@ -0,0 +1,368 @@ +"""Sentiment clustering (pro / neutral / con) for archived X/Twitter data, +plus the supporting "who's loudest, what's trending, what words dominate" +aggregates — the same shape of dashboard Drone Emprit-style tools give a +dataset. + +Two scoring backends, tried in this order: + +1. ML (preferred) — cardiffnlp/twitter-xlm-roberta-base-sentiment, an + XLM-RoBERTa model fine-tuned for tweet sentiment across 8 languages + (ar/en/fr/de/hi/it/pt/es). Its base pretraining covers ~100 languages, so + it degrades gracefully rather than failing outright on a language outside + that fine-tuning set — this is what makes the tool usable for an + open-source audience that isn't Indonesian-only. Needs torch + + transformers (see requirements.txt) and ~1.1GB of model weights + downloaded from Hugging Face on first use. +2. Lexicon fallback — a hand-built Indonesian positive/negative word list + with basic negation handling ("tidak bagus" flips "bagus" from positive + to negative). Used automatically whenever torch/transformers aren't + installed, so a lightweight install (just requirements.txt's base deps) + still has a working — if Indonesian-only — sentiment feature rather than + a hard failure. Every scored item exposes exactly which backend produced + it (`method` on the analyze() payload), so a classification's provenance + is never ambiguous, and the lexicon path additionally exposes the exact + words that drove its score for full transparency. + +Neither backend is a ground-truth classifier — short text, sarcasm, and +irony all degrade accuracy regardless of approach. Treat results as a +starting point for investigation, not a verdict. +""" + +import re +import threading +from collections import Counter + +# Local fallback + +# ── Sentiment lexicon ─────────────────────────────────────────────────────── +# Indonesian words skew political/social-discourse (matches the kind of +# content this tool actually pulls — keyword searches on public affairs, +# government programs, public figures) as well as general register. + +# Arr data words +# Need to feedback and research to sett the all parameter for each words +# Need to help netizen Indo, with slang and words + +POSITIVE_WORDS = { + "bagus", "baik", "hebat", "keren", "mantap", "mantul", "top", "terbaik", + "sukses", "berhasil", "tepat", "benar", "setuju", "dukung", "dukungan", + "mendukung", "apresiasi", "mengapresiasi", "bangga", "membanggakan", + "senang", "gembira", "puas", "memuaskan", "bersyukur", "syukur", + "alhamdulillah", "semoga", "maju", "kemajuan", "bijak", "bijaksana", + "cerdas", "pintar", "amanah", "jujur", "kejujuran", "adil", "keadilan", + "peduli", "kepedulian", "bermanfaat", "manfaat", "membantu", "bantuan", + "solusi", "prestasi", "berprestasi", "unggul", "keunggulan", "luar biasa", + "joss", "jos", "gas terus", "lanjutkan", "sip", "oke", "aman", "tenang", + "damai", "sejahtera", "kesejahteraan", "makmur", "kemakmuran", "indah", + "cantik", "tampan", "ramah", "sopan", "santun", "hormat", "menghormati", + "salut", "kagum", "mengagumkan", "respect", "terharu", "terinspirasi", + "inspiratif", "positif", "optimis", "optimisme", "harapan", "berharap", + "cinta", "mencintai", "sayang", "suka", "menyukai", "rindu", "kangen", + "gemas", "lucu", "menghibur", "menyenangkan", "menginspirasi", "tegas", + "berani", "keberanian", "kuat", "gigih", "semangat", "bersemangat", + "sukacita", "kompeten", "profesional", "berkualitas", "kualitas", + "elegan", "canggih", "inovatif", "inovasi", "kreatif", "efisien", + "efektif", "transparan", "transparansi", "akuntabel", "akuntabilitas", + "merakyat", "membela rakyat", "pro rakyat", "berpihak pada rakyat", + "terpuji", "membanggakan", "gemilang", "cemerlang", "berkah", "istimewa", +} + +NEGATIVE_WORDS = { + "buruk", "keburukan", "jelek", "gagal", "kegagalan", "bodoh", "tolol", + "goblok", "bego", "dungu", "idiot", "korupsi", "korup", "koruptor", + "bohong", "kebohongan", "hoax", "menipu", "penipu", "penipuan", "tipu", + "curang", "kecurangan", "culas", "zalim", "menzalimi", "kejam", + "kekejaman", "jahat", "kejahatan", "rusak", "merusak", "hancur", + "menghancurkan", "kacau", "mengacaukan", "parah", "memalukan", + "memuakkan", "bejat", "biadab", "tolak", "menolak", "penolakan", + "kecewa", "mengecewakan", "kekecewaan", "marah", "kemarahan", "murka", + "benci", "membenci", "kebencian", "muak", "jijik", "menjijikkan", + "sampah", "anjing", "bangsat", "bajingan", "kampret", "sialan", + "kacung", "boneka", "munafik", "kemunafikan", "pengkhianat", + "mengkhianati", "khianat", "pengkhianatan", "penjajah", "menjajah", + "licik", "licin", "serakah", "keserakahan", "tamak", "otoriter", + "diktator", "fasis", "salah", "kesalahan", "blunder", "konyol", + "ngawur", "ngaco", "absurd", "aneh", "ironis", "ironi", "tragis", + "miris", "prihatin", "keprihatinan", "sedih", "menyedihkan", "susah", + "kesusahan", "sulit", "kesulitan", "sengsara", "menderita", + "penderitaan", "korban", "dizalimi", "ditindas", "menindas", + "penindasan", "kriminal", "pelanggaran", "melanggar", "ilegal", + "pungli", "sogok", "menyuap", "disuap", "suap", "nepotisme", "kkn", + "provokasi", "provokator", "memprovokasi", "fitnah", "memfitnah", + "ancaman", "mengancam", "teror", "intimidasi", "mengintimidasi", + "brutal", "kekerasan", "sadis", "tragedi", "bencana", "krisis", + "darurat", "gawat", "resah", "keresahan", "meresahkan", "cemas", + "kecemasan", "khawatir", "kekhawatiran", "takut", "ketakutan", "geram", + "kesal", "jengkel", "dongkol", "malu", "hina", "menghina", + "penghinaan", "murahan", "norak", "kampungan", "terbelakang", "mundur", + "kemunduran", "ambruk", "bangkrut", "kebangkrutan", "defisit", + "terlilit", "terjerat", "terjebak", "cengeng", "lemah", "kelemahan", + "pengecut", "penakut", "plin-plan", "labil", "egois", "keegoisan", + "sombong", "kesombongan", "angkuh", "arogan", "sok", "songong", + "kurang ajar", "tidak becus", "amburadul", "berantakan", "semrawut", + "menyengsarakan", "represif", "represi", "diskriminasi", + "mendiskriminasi", "rasis", "rasisme", "intoleran", "intoleransi", + "penjilat", "gila", "kontol", "memek", "paok", "stress", "goblog", + "kepala batu", "oon", +} + +# Flips the polarity of a sentiment word found within NEGATION_WINDOW tokens +# after it ("tidak bagus" -> negative even though "bagus" alone is positive). +# Indonesian often puts more distance between the negation and the word it +# actually governs than English does — "ngga ngajarin cara ngmg yg sopan" +# ("doesn't teach how to speak politely") is 5 tokens from negation to the +# sentiment word it negates. A wider window catches more of those at the +# cost of occasionally flipping something the negation wasn't really about; +# lexicon scoring is a heuristic either way, this just picks which failure +# mode to lean toward. +NEGATION_WORDS = {"tidak", "tak", "bukan", "belum", "jangan", "nggak", "ga", "gak", "kagak", "ngga"} +NEGATION_WINDOW = 5 + +# Boosts a sentiment word's weight rather than changing its polarity. +INTENSIFIERS = {"sangat", "banget", "sekali", "sungguh", "amat", "terlalu", "sangatlah"} +INTENSIFIER_MULTIPLIER = 1.5 + +STOPWORDS = { + "yang", "dan", "di", "ke", "dari", "untuk", "dengan", "ini", "itu", "ya", + "nya", "adalah", "akan", "saya", "kamu", "kita", "kami", "mereka", "dia", + "juga", "saja", "sudah", "belum", "atau", "karena", "jika", "kalau", + "agar", "supaya", "pada", "oleh", "dalam", "luar", "atas", "bawah", + "antara", "seperti", "sebagai", "tentang", "bahwa", "namun", "tetapi", + "tapi", "hingga", "sampai", "sejak", "setelah", "sebelum", "ketika", + "saat", "ada", "punya", "milik", "lah", "kah", "pun", "deh", "dong", + "sih", "kok", "loh", "nih", "gitu", "gini", "dsb", "dll", "dst", "yg", + "utk", "dgn", "krn", "gak", "ga", "nggak", "tak", "tidak", "bukan", + "jangan", "apa", "apakah", "siapa", "mengapa", "kenapa", "bagaimana", + "dimana", "kapan", "para", "si", "sang", "an", "kan", "in", "the", "is", + "are", "was", "were", "be", "been", "being", "to", "of", "for", "on", + "with", "as", "by", "at", "an", "a", "rt", +} | NEGATION_WORDS | INTENSIFIERS + +_WORD_RE = re.compile(r"[a-zA-ZÀ-ÿ]+(?:-[a-zA-ZÀ-ÿ]+)?") +_URL_RE = re.compile(r"https?://\S+") +_MENTION_RE = re.compile(r"@\w+") + + +def _tokenize(text: str) -> list[str]: + """Lowercased word tokens with URLs/@mentions stripped first (both would + otherwise pollute the lexicon match and the word cloud with usernames/ + link fragments neither list has any business scoring).""" + if not text: + return [] + cleaned = _URL_RE.sub(" ", text) + cleaned = _MENTION_RE.sub(" ", cleaned) + return [w.lower() for w in _WORD_RE.findall(cleaned)] + + +def score_text(text: str) -> dict: + """Returns {label, score, matches} for one piece of text. label is one of + "pro" / "neutral" / "con". matches lists (word, polarity, weight) for + every lexicon hit, so a classification can be inspected rather than + trusted blindly — a defining trait of a rule-based classifier is that you + CAN see exactly why it decided what it decided.""" + tokens = _tokenize(text) + score = 0.0 + matches = [] + + for i, tok in enumerate(tokens): + polarity = 1 if tok in POSITIVE_WORDS else -1 if tok in NEGATIVE_WORDS else 0 + if polarity == 0: + continue + + weight = 1.0 + # Negation: any negation word in the preceding window flips polarity. + window_start = max(0, i - NEGATION_WINDOW) + if any(t in NEGATION_WORDS for t in tokens[window_start:i]): + polarity = -polarity + # Intensifier: any intensifier immediately before boosts magnitude + # (checked after negation so "tidak sangat bagus" still flips first). + if i > 0 and tokens[i - 1] in INTENSIFIERS: + weight = INTENSIFIER_MULTIPLIER + + contribution = polarity * weight + score += contribution + matches.append({"word": tok, "polarity": "pro" if polarity > 0 else "con", "weight": weight}) + + label = "pro" if score > 0 else "con" if score < 0 else "neutral" + return {"label": label, "score": round(score, 2), "matches": matches} + + +def _item_text(item: dict) -> str: + """The text worth scoring/tokenizing for a given archived record — + varies by which tool produced it (a tweet's own text vs. a Wayback/CSE + page's scraped title+description vs. a bare user's bio).""" + parts = [ + item.get("text"), item.get("full_text"), item.get("article_text"), + item.get("post_title"), item.get("post_text"), item.get("description"), + ] + return " ".join(p for p in parts if p) + + +def _item_author(item: dict) -> str | None: + return item.get("screen_name") or item.get("user") or item.get("name") + + +def _item_engagement(item: dict) -> int: + total = 0 + for k in ("reply_count", "retweet_count", "favorite_count"): + v = item.get(k) + if isinstance(v, (int, float)): + total += v + return total + + +def word_frequencies(items: list[dict], top_n: int = 60) -> list[dict]: + """Word-cloud data: [{word, count}], most frequent first. Stopwords and + single-character tokens are dropped; everything else counts regardless + of whether it happened to be in the sentiment lexicon.""" + counts = Counter() + for item in items: + for tok in _tokenize(_item_text(item)): + if len(tok) < 3 or tok in STOPWORDS: + continue + counts[tok] += 1 + return [{"word": w, "count": c} for w, c in counts.most_common(top_n)] + + +def top_users(items: list[dict], top_n: int = 20) -> list[dict]: + """Who shows up most often across the archive — every record with an + identifiable author counts once, regardless of whether it's a tweet, a + reply, a retweeter entry, or a bare follower/following record. Carries + along the most recently seen avatar/name for that handle so the + dashboard can show a face, not just a bare count.""" + counts: Counter = Counter() + display: dict[str, dict] = {} + for item in items: + handle = item.get("screen_name") or item.get("user") + if not handle: + continue + counts[handle] += 1 + display[handle] = { + "screen_name": handle, + "name": item.get("name"), + "avatar": item.get("avatar") or item.get("user_avatar"), + "verified": item.get("verified"), + "is_blue_verified": item.get("is_blue_verified"), + } + ranked = [] + for handle, count in counts.most_common(top_n): + ranked.append({**display[handle], "count": count}) + return ranked + + +def top_engagement(items: list[dict], top_n: int = 15) -> list[dict]: + """Which posts drove the most reply+retweet+favorite activity — "paling + ramai" (busiest/most-discussed), not just most recent.""" + scored = [(_item_engagement(it), it) for it in items if _item_text(it)] + scored.sort(key=lambda pair: pair[0], reverse=True) + return [{"engagement": score, "item": it} for score, it in scored[:top_n] if score > 0] + + +# ── ML backend (preferred) ─────────────────────────────────────────────────── +# Lazy-loaded: importing torch/transformers and loading ~1.1GB of weights is +# slow, and both packages are optional (requirements.txt notes how to add +# them) — doing this at module import time would slow down every single use +# of this app, including ones that never touch analytics, and would hard-crash +# an install that skipped the ML deps entirely instead of just falling back. +_ML_MODEL_NAME = "cardiffnlp/twitter-xlm-roberta-base-sentiment" +_ML_LABEL_MAP = {"positive": "pro", "negative": "con", "neutral": "neutral"} +_ml_pipeline = None +_ml_unavailable = False # sticky once loading fails — stop retrying every call +_ml_load_lock = threading.Lock() + + +def _get_ml_pipeline(): + global _ml_pipeline, _ml_unavailable + if _ml_pipeline is not None or _ml_unavailable: + return _ml_pipeline + with _ml_load_lock: + if _ml_pipeline is not None or _ml_unavailable: # re-check post-lock + return _ml_pipeline + try: + from transformers import pipeline + _ml_pipeline = pipeline("sentiment-analysis", model=_ML_MODEL_NAME) + except Exception: + # Missing torch/transformers, no internet for the first-time + # model download, out of memory, ... any of these should fall + # back to the lexicon scorer rather than take the whole + # analytics endpoint down. + _ml_unavailable = True + _ml_pipeline = None + return _ml_pipeline + + +def _score_texts_ml(texts: list[str]) -> list[dict] | None: + """Batch-scores every text in one call (far faster on CPU than one + pipeline call per item). Returns None if the model isn't available, so + the caller falls back to the lexicon scorer instead. `score` is signed + (positive for pro, negative for con, 0 for neutral) to match the + lexicon backend's convention; `confidence` carries the model's own + unsigned probability for the label it picked.""" + clf = _get_ml_pipeline() + if clf is None: + return None + raw = clf(texts, truncation=True, batch_size=16) + results = [] + for r in raw: + label = _ML_LABEL_MAP.get(str(r.get("label", "")).lower(), "neutral") + confidence = float(r.get("score", 0.0)) + signed = confidence if label == "pro" else -confidence if label == "con" else 0.0 + results.append({ + "label": label, "score": round(signed, 3), + "confidence": round(confidence, 3), "matches": [], + }) + return results + + +def analyze(items: list[dict]) -> dict: + """Full analytics payload for one archive's worth of raw items.""" + if not isinstance(items, list): + items = [items] + # archive.py's own _run() passes non-dict entries through as-is rather + # than dropping them (see its "if not isinstance(item, dict)" branch), + # so a saved archive can legitimately contain a stray non-dict item — + # every function below assumes dict.get(), so those get filtered here + # once rather than each helper needing its own isinstance guard. + items = [it for it in items if isinstance(it, dict)] + + text_items, texts = [], [] + for item in items: + text = _item_text(item) + if text: # skip e.g. a bare follower/following record with no post text of its own + text_items.append(item) + texts.append(text) + + ml_results = _score_texts_ml(texts) if texts else None + method = "ml" if ml_results is not None else "lexicon" + if ml_results is None: + ml_results = [score_text(t) for t in texts] + + sentiment_counts = {"pro": 0, "neutral": 0, "con": 0} + scored_items = [] + for item, text, result in zip(text_items, texts, ml_results): + sentiment_counts[result["label"]] += 1 + scored_items.append({ + "label": result["label"], + "score": result["score"], + "confidence": result.get("confidence"), + "matches": result.get("matches", []), + "author": _item_author(item), + "text": text, + "item": item, + }) + + total = len(scored_items) + return { + "method": method, + "total_items": len(items), + "total_scored": total, + "sentiment_counts": sentiment_counts, + "sentiment_pct": { + k: round(v / total * 100, 1) if total else 0.0 + for k, v in sentiment_counts.items() + }, + "scored_items": scored_items, + "word_freq": word_frequencies(items), + "top_users": top_users(items), + "top_engagement": top_engagement(items), + } diff --git a/Script/SOCMINT-Twitter/static/js/card_constants.js b/Script/SOCMINT-Twitter/static/js/card_constants.js index ad45db0..e464744 100644 --- a/Script/SOCMINT-Twitter/static/js/card_constants.js +++ b/Script/SOCMINT-Twitter/static/js/card_constants.js @@ -8,7 +8,7 @@ // Arr params const PRIORITY = [ - 'source', 'account_age_flag', 'account_age', 'account_created', + 'source', 'content_type', 'account_age_flag', 'account_age', 'account_created', 'user', 'screen_name', 'name', 'user_id', 'username', 'verified', 'is_blue_verified', 'text', 'full_text', 'article_text', 'post_title', 'post_text', 'content', 'title', 'description', 'bio', @@ -18,9 +18,23 @@ const PRIORITY = [ 'retweeted_by_user', 'retweeted_by_name', 'retweeted_text', 'retweeted_by_bio', 'retweeted_at', 'retweeted_tweet_id', 'lat', 'lon', 'place', 'user_location', 'tweet_url', 'archive_url', 'result_url', 'preview_image', 'display_link', + 'serp_title', 'iso_date', 'original', 'statuscode', 'mimetype', 'length', ]; +// content_type: what kind of X/Twitter page a Google CSE / Wayback result +// actually is — a search hit for a keyword could be a specific tweet, a +// bare profile page, some other X page, or (Google CSE only) a site off X +// entirely. Cookie/Xquik records don't carry this field at all (a tweet +// search result there is unambiguously always a tweet), so it only ever +// shows up for the two sources it's meant to disambiguate. +const CONTENT_TYPE_LABELS = { + tweet: 'Tweet', + profile: 'Profile page', + twitter_other: 'Other X/Twitter page', + other: 'External page (non-X)', +}; + // Which fields link out to a fresh extraction for that tweet — same anchor // behavior whether you're looking at a live result or a saved archive. const DRILLABLE = { diff --git a/Script/SOCMINT-Twitter/templates/_field_glossary.html b/Script/SOCMINT-Twitter/templates/_field_glossary.html index 3d7a37a..1b4081a 100644 --- a/Script/SOCMINT-Twitter/templates/_field_glossary.html +++ b/Script/SOCMINT-Twitter/templates/_field_glossary.html @@ -4,10 +4,13 @@
id / user_id Twitter's internal numeric ID for the tweet / account
user / screen_name The @handle
name Display name
+
avatar / banner Profile photo / cover photo — direct CDN links, cookie mode only. On a tweet/reply record these are the author's photos, named user_avatar/user_banner instead so they don't collide with the tweet's own fields
Content
-
text / full_text / article_text The post's own text
-
description Profile bio
+
text / full_text / article_text The post's own complete text — cookie mode always resolves the real full text even for long-form "Note" tweets that X itself truncates in its legacy API field
+
reply_to_mentions The @handles X auto-prepended to a reply's text when it was composed (everyone already tagged in that conversation) — x.com's own UI hides these from the visible tweet body and shows them as a separate "Replying to" line instead, which is exactly what every card/panel here does with this field too. The raw text itself still contains them untouched
+
hashtags Hashtags used in the post
+
description Profile bio (a user/follower/following record), or Google's own SERP snippet for that result (a CSE record) — which one depends on whether the record is a person or a web result
created_at When the post itself was actually made. Same field, every source: Cookie/Xquik get it straight from the API; Wayback and Google CSE decode it from the tweet id's Snowflake bits whenever the result links to a tweet permalink (absent otherwise — e.g. a profile page or a non-X result, where there's no post id to decode)
fetched_at When this tool pulled the record — same field, every source (Cookie, Xquik, Wayback, Google CSE), so results are comparable side by side. Not to be confused with created_at (the post's own creation time) or iso_date (a Wayback snapshot's capture time)
@@ -47,9 +50,11 @@
statuscode / mimetype / length HTTP status / content type / size of the snapshot
post_title / post_text / preview_image Scraped from the archived page's own meta tags (og:/twitter: tags, or <title>/<meta name="description"> on older captures)
-
Google CSE
-
post_title Result title from Google
-
post_text The page's own og:/twitter:/meta description, fetched live from result_url — not Google's own SERP snippet, which is usually clipped mid-sentence. Falls back to Google's snippet only if that live fetch fails
+
Google CSE / Wayback
+
content_type What kind of X/Twitter page a result actually is — tweet (a specific permalink) / profile (a bare profile page) / twitter_other (some other X page) / other (a non-X site, Google CSE only). Derived purely from the result URL's own shape — a keyword match isn't always a specific tweet, and this is how to tell without opening the link
+
serp_title Google's own SERP result title — always kept as-is, even if post_title below gets replaced by a fresher live fetch
+
description Google's own SERP snippet for this result — Google truncates this itself (ends mid-sentence in "…"), that's not this tool cutting it short. Kept untouched alongside post_text below since a live re-fetch of an X link often gets a less specific, bot-facing page instead
+
post_title / post_text "Best known" title/description — starts as Google's own SERP values, replaced by a live fetch of the page's own og:/twitter:/meta tags when one succeeds. See serp_title/description above for what Google originally had, in case the live fetch made things worse rather than better
result_url The live page Google indexed, shown as its full raw address
display_link The result's domain, as shown in Google's own results
preview_image Thumbnail — from Google's own indexed metadata, or the live page's og:/twitter:image when Google didn't have one
@@ -59,9 +64,9 @@
Graph node types (graph page only)
Search root The diamond node — the query you ran
-
Tweet A tweet/post returned by search or by expanding a node
+
Tweet A tweet/post returned by search, by Expand Posts on a user node, or by expanding another node
Reply A tweet fetched via Expand Replies on a tweet node
-
User / Retweeter A person — from Follower Explorer, or via Expand Retweets on a tweet node
+
User / Retweeter A person — from Follower/Following Explorer, via Expand Retweets/Followers/Following on a node, or via View Author Profile on a tweet/reply (pulled from that tweet's already-fetched author info, no extra request). Once on canvas, select it and Expand Posts/Followers/Following works the same regardless of how it got there
Wayback snapshot An archived-page result, from Wayback Archive Search or the Wayback portion of Multi-Source Search
Web result A Google CSE result, from the Google CSE portion of Multi-Source Search
Viewed Turns a node solid white once you've clicked it — a visual "already looked at this" marker, not part of the underlying data
diff --git a/Script/SOCMINT-Twitter/templates/analytics.html b/Script/SOCMINT-Twitter/templates/analytics.html new file mode 100644 index 0000000..e702a28 --- /dev/null +++ b/Script/SOCMINT-Twitter/templates/analytics.html @@ -0,0 +1,816 @@ + + + + + +Jieyab89 SOCMINT X — Analytics + + + + +
+

Jieyab89 SOCMINT X

+ / + Analytics — sentiment clustering, top actors, word frequency + + +
+ + + +
+ + +
+
Pick an archive on the left to run sentiment clustering, top-actor, and word-frequency analysis on its raw data.
+
+
+ + + + + diff --git a/Script/SOCMINT-Twitter/templates/archive.html b/Script/SOCMINT-Twitter/templates/archive.html index 494a914..9e8a459 100644 --- a/Script/SOCMINT-Twitter/templates/archive.html +++ b/Script/SOCMINT-Twitter/templates/archive.html @@ -31,7 +31,9 @@ display: flex; align-items: center; gap: 12px; - position: relative; + position: sticky; + top: 0; + z-index: 100; } header h1 { font-size: 14px; font-weight: 600; } header .sep { color: var(--border); } @@ -96,6 +98,27 @@ z-index: 251; transition: all 0.15s; } + .back-to-top { + position: fixed; + bottom: 14px; + right: 14px; + width: 36px; + height: 36px; + border-radius: 50%; + background: var(--surface); + border: 1px solid var(--border); + color: #FFFFFF; + font-size: 17px; + cursor: pointer; + z-index: 251; + box-shadow: 0 4px 14px rgba(0,0,0,0.4); + transition: all 0.15s; + display: none; + align-items: center; + justify-content: center; + } + .back-to-top:hover { border-color: var(--accent); } + .back-to-top.visible { display: flex; } .help-toggle:hover { color: var(--text); border-color: var(--accent); } .help-popup { position: fixed; @@ -211,6 +234,7 @@ font-weight: 500; } .ae-meta { font-size: 11px; color: var(--muted); margin-top: 2px; } + .ae-date { font-size: 10.5px; color: var(--muted); opacity: 0.75; margin-top: 2px; } .ae-stats { font-size: 11px; color: var(--muted); display: flex; gap: 8px; margin-top: 3px; } .ae-stat { display: flex; align-items: center; gap: 3px; } .no-archives { @@ -337,6 +361,23 @@ .age-badge.age-recent { color: #f59e0b; border-color: #78350f; background: #1c0e02; } .age-badge.age-established { color: var(--success); border-color: #14532d; background: #052011; } + /* What kind of X/Twitter page a Google CSE / Wayback result actually is — + a keyword match linking to x.com/someone reads as "just a profile" with + no way to tell a tweet permalink apart from a bare profile page or an + unrelated page otherwise. */ + .content-type-badge { + display: inline-block; + font-size: 11px; + font-weight: 500; + padding: 2px 9px; + border-radius: 20px; + border: 1px solid var(--border); + } + .content-type-badge.ct-tweet { color: var(--success); border-color: #14532d; background: #052011; } + .content-type-badge.ct-profile { color: #a78bfa; border-color: #4c3a8f; background: #1e1535; } + .content-type-badge.ct-twitter_other { color: #f59e0b; border-color: #78350f; background: #1c0e02; } + .content-type-badge.ct-other { color: var(--muted); border-color: var(--border); background: var(--bg); } + /* ── Media ── */ .card-media { display: flex; @@ -411,17 +452,24 @@ Graph
Archives + Analytics +