OSINT-Cheat-sheet/Script/SOCMINT-Twitter/Readme.md

Jieyab ft Xquik

Features

• Multi-source search — one query fans out to Cookie, Xquik API, Wayback Machine, and Google CSE in parallel, each result tagged with which source it came from.
• 10 extraction tools — tweet search, follower/following explorer, post (timeline) extractor, article extractor, community posts, tweet replies, tweet retweeters, geo-tagged search with a map view, Wayback archive search, and multi-source search.
• Infinite scroll + inline nested reply threads — scroll to load more pages automatically; expand a reply's own replies in place, recursively, the way X's own UI threads a conversation.
• Relationship graph (/graph) — a Cytoscape-based node graph. Select a node to expand its replies, retweeters, posts, followers, or following; pivot from any tweet straight to its author (no extra request — the data's already on the tweet); box-select multiple nodes and drag them together; edges are labeled by relationship (replied by, followed by, authored by, ...) and a distinct color flags where two different paths through the graph converge on the same account or tweet.
• Archive (/archives) — one-click (or auto-) save of a search's full raw results, including downloaded media, browsable and re-searchable later independent of whether the source is still reachable.
• Analytics (/analytics) — sentiment clustering (pro / neutral / con) over a saved archive, plus who posts most, what's driving the most engagement, and word frequency — see Sentiment Analysis below.
• Account-age forensics — every result with a numeric X/Twitter ID gets its account creation date decoded straight from the ID's Snowflake bits (no extra API call), flagged New / Recent / Established.
• Security-conscious by default — strict CSP with per-request nonces, hardened cookies, a whitelisted SSRF-safe video proxy, per-source request throttling to protect the logged-in account from rate limits, and URL-scheme validation everywhere a scraped link is rendered as an image or embedded in CSS.

Sett up

1. Config your Google api console here, enable and manage API Custom Search by Google

2. Sett the api key in web console Google

Check the result in the table

3. Settings CSE Google to put the cx key

4. Add site want to crawll e.g twitter.com and x.com

Data Source

1. Xquik API (subs there is a price)
2. Cookie (your account cookie session)
3. Wayback Machine (Cdx API)
4. Goole CSE API (free quota 100 per day u can increase u limit with buy the service)

Usage

Search (/)

Pick a tool from the dropdown, fill in the field it asks for (query, username, tweet ID, ...), and hit Run. Cookie-only tools (marked [Cookie]) always use your logged-in session; everything else lets you toggle between Cookie (your session, no xquik quota used) and xquik API (uses your API key's quota) mode.

• Scroll down to auto-load more pages on any tool that supports pagination.
• Any card whose post has replies of its own gets an Expand N replies button — click it to thread the conversation inline, as deep as it actually goes.
• Toggle Auto Archive before running a search to save results as you go (including every scrolled-in page) — see Archives.
• Geo Post Extractor switches the results view to a map, geocoding each author's profile location.
• The ↓ JSON button downloads exactly what's on screen as raw JSON.

Graph (/graph)

Same 10 tools, rendered as a node graph instead of a card list.

• Click a node to inspect its full raw data in the side panel.
• Expand Replies / Retweets (tweet or reply nodes), Expand Posts / Followers / Following (user or retweeter nodes), and View Author Profile (tweet/reply nodes — pulls the author out as their own node, for free) all attach new nodes with a labeled edge showing the relationship.
• Shift/Alt/Ctrl + drag on empty canvas to box-select multiple nodes; drag any one of them to move the whole selection together. Plain drag still pans, scroll still zooms.
• A yellow edge means two different paths through the graph converged on the same node — worth a second look.
• Archive All / Dump JSON export everything currently on canvas.
• Click the ? button (bottom-left) for the full legend.

Archives (/archives)

Browse everything saved from Search or Graph. Pick an archive from the sidebar to see its full raw results and any downloaded media. Archives persist independent of whether the original source is still reachable — useful for content that gets deleted or a session that expires.

Analytics (/analytics)

Pick a saved archive to run sentiment clustering and the surrounding aggregates over it:

• Sentiment clustering — every item with text gets classified pro / neutral / con, shown as a diverging bar plus per-category tiles you can click to filter the item list below.
• Most active accounts — who shows up most often in that archive.
• Most engagement — which items drove the most reply+retweet+like activity.
• Word frequency — a word cloud sized by how often each word appears.
• Item browser — every scored item, filterable by sentiment and searchable by text/author, with the model's confidence (or, in lexicon fallback mode, the exact matched words) shown per item — a classification is always inspectable, never a black box.

Sentiment Analysis

Two backends, tried in this order (see sentiment.py):

1. ML (preferred) — cardiffnlp/twitter-xlm-roberta-base-sentiment, an XLM-RoBERTa model fine-tuned for tweet sentiment across 8 languages (Arabic, English, French, German, Hindi, Italian, Portuguese, Spanish). Its underlying pretraining covers roughly 100 languages, so it degrades gracefully rather than failing outright on a language outside that fine-tuned set — this is what makes the tool usable for a global audience, not just Indonesian speakers. Requires torch + transformers (see requirements.txt) and downloads ~1.1GB of model weights from Hugging Face the first time it runs.
2. Lexicon fallback — a hand-built Indonesian positive/negative word list with negation handling (e.g. "tidak bagus" flips from positive to negative). Used automatically whenever torch/transformers aren't installed, so a lightweight install still has a working — if Indonesian-only — sentiment feature instead of a hard failure.

The /api/analytics/<archive_id> response always reports which backend ("ml" or "lexicon") produced its results, and the Analytics page's banner reflects it. Neither backend is a ground-truth classifier — short text, sarcasm, and irony degrade accuracy either way. Treat results as a starting point for investigation, not a verdict.

Update Note

1. Update infinity scroll and load new data for each search also in graph
2. Update data corelation
3. Fix business logic flow
4. Monitoring (Soon)
5. MCP (Soon)
6. Add more parameter for enrichment
7. Add no rate limit (throttle)
8. Add Google CSE data source
9. Expand data user profile post, follower and following, reply post, retweet post in graph —
10. Add sentiment analysis for clustering data, pro, neutral, con based on archive data and dump data
11. Add more data source and other parameter (soon) still research
12. Add more detail data source for the context

Settup

pip install -r requirements.txt

Edit config.ini.example to config.ini

Installing the ML sentiment model (optional, recommended)

The Analytics page's sentiment scoring works two ways — see Sentiment Analysis above. Skipping everything on this page is completely fine: the app still runs and Analytics still works, just using the Indonesian-only lexicon instead of the multilingual model. This section is only for turning on the better (multilingual) one. You don't need to know Python or be a developer to follow this — it's copy/paste.

Why a "virtual environment" (venv)? It's just a private, throwaway folder for this project's Python packages, kept separate from anything else Python-related already on your computer. The ML packages (torch, transformers) are large and can conflict with other unrelated tools if installed system-wide — a venv avoids that entirely, and if anything ever goes wrong, you just delete the .venv folder and start over, nothing else on your machine is touched.

1. Open a terminal in this project's folder (the same folder as app.py).

2. Create the venv (only needs to be done once):

python3 -m venv .venv

3. Activate it (needs to be done every time you open a new terminal to work on this project):

• macOS / Linux:

  source .venv/bin/activate
  

• Windows (Command Prompt):

  .venv\Scripts\activate.bat
  

• Windows (PowerShell):

  .venv\Scripts\Activate.ps1
  

Your terminal prompt should now start with (.venv) — that means it's active and every pip install from here on stays inside this project's private folder.

4. Install everything (the base app + the ML packages), in this order:

pip install --upgrade pip
pip install flask requests twifork
pip install --index-url https://download.pytorch.org/whl/cpu torch
pip install transformers sentencepiece protobuf

The third line is deliberately its own command — installing torch the plain way can pull down a multi-gigabyte GPU-enabled build depending on your system, when this app only ever needs the much smaller CPU version. Using that exact command is what keeps the download small.

5. Run the app as usual (make sure .venv is still active — you'll see (.venv) in your prompt):

python app.py

The first time you use the Analytics page after this, it will download the sentiment model itself (~1.1GB) from Hugging Face automatically — this needs an internet connection and can take a few minutes depending on your connection, but only happens once. After that, it's cached on your computer and loads instantly.

Next time you come back to work on this project, you only need step 3 again (activate) before running the app — steps 1, 2, and 4 are one-time setup.

Run Local Web Server

python app.py

Open http://127.0.0.1:5000

Results

Xquik Dashboard

Dasboard Home

Archive

Graph

Vidio

Dir Output

Sentiment Analysis

Help

About SnowflakeID -> Twitter userid : https://en.wikipedia.org/wiki/Snowflake_ID

About paramater was provided in data and dump with json file type

Xquik API DOC

Offc doc: https://docs.xquik.com/api-reference/overview

Soon i will check more detail about Twitter or X mechanism and business logic also endpoint API was listed in Mobile and Web

Wayback archive data source

The server connection to the Wayback Machine archive is often down, so try bumping the thread and don't set the throttle too high, and try checking the connection manually using curl.