# Jieyab ft Xquik

# Update Note 1. Update infinity scroll and load new data for each search also in graph 2. Update data corelation 3. Fix business logic flow 4. Monitoring (Soon) 5. MCP (Soon) 6. Add more parameter for enrichment 7. Add no rate limit (throttle) 8. Add Google CSE data source 9. Expand data user profile post, follower and following, reply post, retweet post in graph 10. Add sentiment analysis for clustering data, pro, neutral, con. Based on archive data and dump data 11. Add more data source and other parameter (soon) still research 12. Add more detail data source for the context 13. Auto repair broken archive and sentiment analysis data dump 14. Update rendering data in sentiment analysis ## Features - **Multi-source search** — one query fans out to Cookie, Xquik API, Wayback Machine, and Google CSE in parallel, each result tagged with which source it came from. - **10 extraction tools** — tweet search, follower/following explorer, post (timeline) extractor, article extractor, community posts, tweet replies, tweet retweeters, geo-tagged search with a map view, Wayback archive search, and multi-source search. - **Infinite scroll + inline nested reply threads** — scroll to load more pages automatically; expand a reply's own replies in place, recursively, the way X's own UI threads a conversation. - **Relationship graph** (`/graph`) — a Cytoscape-based node graph. Select a node to expand its replies, retweeters, posts, followers, or following; pivot from any tweet straight to its author (no extra request — the data's already on the tweet); box-select multiple nodes and drag them together; edges are labeled by relationship (*replied by*, *followed by*, *authored by*, ...) and a distinct color flags where two different paths through the graph converge on the same account or tweet. - **Archive** (`/archives`) — one-click (or auto-) save of a search's full raw results, including downloaded media, browsable and re-searchable later independent of whether the source is still reachable. - **Analytics** (`/analytics`) — sentiment clustering (pro / neutral / con) over a saved archive, a Maltego-style social network graph, plus who posts most, what's driving the most engagement, and word frequency — see [Sentiment Analysis](#sentiment-analysis) below. - **Account-age forensics** — every result with a numeric X/Twitter ID gets its account creation date decoded straight from the ID's Snowflake bits (no extra API call), flagged New / Recent / Established. - **Security-conscious by default** — strict CSP with per-request nonces, hardened cookies, a whitelisted SSRF-safe video proxy, per-source request throttling to protect the logged-in account from rate limits, and URL-scheme validation everywhere a scraped link is rendered as an image or embedded in CSS. ## Data Source 1. Xquik API (subs there is a price) 2. Cookie (your account cookie session) 3. Wayback Machine (Cdx API) 4. Goole CSE API (free quota 100 per day u can increase u limit with buy the service) ## Setup **1. Create a virtual environment** (a private folder for this project's Python packages, kept separate from anything else on your machine — only needs doing once): ```bash python3 -m venv .venv ``` **2. Activate it** (do this every time you come back to work on the project): - macOS / Linux: `source .venv/bin/activate` - Windows (Command Prompt): `.venv\Scripts\activate.bat` - Windows (PowerShell): `.venv\Scripts\Activate.ps1` Your terminal prompt should now start with `(.venv)`. **3. Install dependencies:** ```bash pip install -r requirements.txt ``` This installs Flask, the scraping/API clients, and the multilingual sentiment model's packages (`torch`, `transformers`) in one go — see [Sentiment Analysis](#sentiment-analysis) for what that model does. If your machine has no NVIDIA GPU, torch's default download can be a few GB larger than it needs to be; keep it small by running this once first, then installing as above: ```bash pip install --index-url https://download.pytorch.org/whl/cpu torch ``` Whenever the project's dependencies change (new features, updates), just re-run: ```bash pip install -r requirements.txt --upgrade ``` **4. Configure the Google CSE data source** (optional — only needed if you want that source; everything else works without it): 1. Enable "Custom Search API" in the Google API Console. enable

2. Get an API key. g-api

Check the result in the table. g - api result

3. Set up a Programmable Search Engine to get a `cx` key. cx key

4. Add the sites to crawl, e.g. `twitter.com` and `x.com`. add host and domain twitter

**5. Copy `config.ini.example` to `config.ini`** and fill in your keys. **6. Run the app:** ```bash python app.py ``` Open `http://127.0.0.1:5000`. The first time you open the Analytics page, it downloads the sentiment model (~1.1GB) from Hugging Face automatically — needs an internet connection once, then it's cached locally and loads instantly after that. ## Usage ### Search (`/`) Pick a tool from the dropdown, fill in the field it asks for (query, username, tweet ID, ...), and hit **Run**. Cookie-only tools (marked `[Cookie]`) always use your logged-in session; everything else lets you toggle between **Cookie** (your session, no xquik quota used) and **xquik API** (uses your API key's quota) mode. - Scroll down to auto-load more pages on any tool that supports pagination. - Any card whose post has replies of its own gets an **Expand N replies** button — click it to thread the conversation inline, as deep as it actually goes. - Toggle **Auto Archive** before running a search to save results as you go (including every scrolled-in page) — see [Archives](#archives-archives). - **Geo Post Extractor** switches the results view to a map, geocoding each author's profile location. - The `↓ JSON` button downloads exactly what's on screen as raw JSON. ### Graph (`/graph`) Same 10 tools, rendered as a node graph instead of a card list. - Click a node to inspect its full raw data in the side panel. - **Expand Replies / Retweets** (tweet or reply nodes), **Expand Posts / Followers / Following** (user or retweeter nodes), and **View Author Profile** (tweet/reply nodes — pulls the author out as their own node, for free) all attach new nodes with a labeled edge showing the relationship. - **Shift/Alt/Ctrl + drag** on empty canvas to box-select multiple nodes; drag any one of them to move the whole selection together. Plain drag still pans, scroll still zooms. - A **yellow edge** means two different paths through the graph converged on the same node — worth a second look. - **Archive All** / **Dump JSON** export everything currently on canvas. - Click the **?** button (bottom-left) for the full legend. ### Archives (`/archives`) Browse everything saved from Search or Graph. Pick an archive from the sidebar to see its full raw results and any downloaded media. Archives persist independent of whether the original source is still reachable — useful for content that gets deleted or a session that expires. ### Analytics (`/analytics`) Pick a saved archive to run sentiment clustering and the surrounding aggregates over it: - **Sentiment clustering** — every item with text gets classified **pro** / **neutral** / **con**, shown as a diverging bar plus per-category tiles you can click to filter the item list below. - **Social network graph** — click **View as Graph** to see accounts as nodes, sized by how many items they posted in this archive and colored by their own pro/neutral/con mix, Maltego-style. Edges show reply relationships found in the archive; click a node to see its per-account sentiment breakdown. - **Most active accounts** — who shows up most often in that archive. - **Most engagement** — which items drove the most reply+retweet+like activity. - **Word frequency** — a word cloud sized by how often each word appears. - **Item browser** — every scored item, filterable by sentiment and searchable by text/author, with the model's confidence (or, in lexicon fallback mode, the exact matched words) shown per item — a classification is always inspectable, never a black box. ## Sentiment Analysis Two backends, tried in this order (see `sentiment.py`): 1. **ML (preferred)** — [`cardiffnlp/twitter-xlm-roberta-base-sentiment`](https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment), an XLM-RoBERTa model fine-tuned for tweet sentiment across 8 languages (Arabic, English, French, German, Hindi, Italian, Portuguese, Spanish). Its underlying pretraining covers roughly 100 languages, so it degrades gracefully rather than failing outright on a language outside that fine-tuned set — this is what makes the tool usable for a global audience, not just Indonesian speakers. Requires `torch` + `transformers` (see `requirements.txt`) and downloads ~1.1GB of model weights from Hugging Face the first time it runs. 2. **Lexicon fallback** — a hand-built Indonesian positive/negative word list with negation handling (e.g. *"tidak bagus"* flips from positive to negative). Used automatically whenever `torch`/`transformers` aren't installed, so a lightweight install still has a working — if Indonesian-only — sentiment feature instead of a hard failure. The `/api/analytics/` response always reports which backend (`"ml"` or `"lexicon"`) produced its results, and the Analytics page's banner reflects it. Neither backend is a ground-truth classifier — short text, sarcasm, and irony degrade accuracy either way. Treat results as a starting point for investigation, not a verdict. # Results [![Watch the video](https://vumbnail.com/1216677900.jpg)](https://vimeo.com/1216677900) Xquik Dashboard

Dasboard Home