01 — Overview
Overview
a. Project Scope & Limitations
PediaFX is an established forex education and broker analysis platform. Our team was hired to build only the automated price feed scrapper and AI rewriting pipeline, rather than redesigning the entire site. We built a custom WordPress plugin (pediafx-scrapper) that retrieves real-time signals and news, transforms them, and publishes them.
The plugin creates custom post types (pediafx-signal) and injects scraped price variables (e.g. entry, stop loss, target profit, average daily range, and open position ratios) directly into custom fields. By isolating this automated pipeline, the platform is updated regularly with zero impact on the front-end layout.
b. Custom Plugin Architecture
We engineered a hybrid architecture composed of a containerized Scrapy crawler daemon and a WordPress REST API bridge. The crawler extracts raw data from broker systems and macro news hubs, while the pediafx-scrapper plugin registers custom post types and custom fields.
Data communication is secured via HTTP Basic Auth headers using WordPress application passwords. The Scrapy pipeline pushes JSON payloads to the custom endpoint (/wp-json/wp/v2/pediafx-signal), where our plugin intercepts the payload, maps metadata values, and attaches featured media dynamically.
- Designed and developed custom WordPress plugin pediafx-scrapper to handle signal metadata and options mapping
- Implemented Scrapy crawling tasks passing authentication headers to target mobile-app APIs
- Built an automated media upload bridge converting scraped image URLs to WordPress media library IDs
- Configured OpenAI GPT-4o-mini prompts to translate and rewrite macro news summaries into HTML
- Created custom admin setting dashboards managing scraping endpoints and asset parameters
- Established duplicate posts detection checks utilizing local SQLite databases
02 — Problems & Challenges
Problems & Challenges
a. Bypassing Scraper Blocks / Rate Limits
Scraping real-time market data is prone to blocks, particularly when target APIs employ anti-bot systems or require authenticated mobile headers. Standard cURL requests trigger instant rate limits or Cloudflare challenges.
We resolved this block by configuring the Scrapy engine to mirror mobile app requests. Inside the MifxSignalSpider, we set headers mimicking iOS client network libraries: user-agent: MIFXMobile/4 CFNetwork/1498.700.2 Darwin/23.6.0. We built a login sequence (parse_authorize and parse_access_token) that queries authorization endpoints and updates bearer tokens dynamically, bypassing blocks.
b. AI Content Hallucination Control
Utilizing AI to rewrite and translate financial news articles can cause hallucinations. OpenAI models can introduce markdown elements, summarize key price numbers incorrectly, or alter the meaning of trading signals.
We built a strict JSON parsing structure inside the OpenAIPipeline. We engineered a system prompt requiring the gpt-4o-mini model to output content strictly inside a JSON schema: {"title": "...", "content": "..."}. We forced the model to return ONLY raw JSON, which is then parsed by the Python worker, protecting HTML integrity.
c. WP Cron Job Reliability
Relying on standard WordPress cron (wp-cron.php) to trigger scraping scripts was highly unreliable. Since WordPress crons depend on web visits to execute, the site experienced data lag during low-traffic periods.
We solved this by decoupling the scheduler from WordPress. We configured a Dockerized cron daemon on the server that runs Scrapy crawling pipelines on a strict timeline, pushing updates to WordPress independently of web traffic.
03 — Project Goals
Project Goals
a. Real-Time Price Accuracy
Our primary goal was to ensure price data remains updated, updating Average Daily Range (ADR) metrics, open buy/sell ratios, and current sentiment score variables across currency pairs.
By writing concurrent API calls to sentiment and ratio endpoints in mifx_signal.py, the plugin captures and pushes market updates within seconds of their publication.
b. Automating the Content Pipeline
We aimed to eliminate manual translation and content writing tasks. The system translates Indonesian macro overviews into English articles, publishes them, and formats HTML styles automatically.
04 — Design Process
Design Process
a. Scraping Engine Logic
We designed the Scrapy pipeline to run three tasks: (1) MediaUploaderPipeline downloads raw image attachments and uploads them to the WordPress media library, updating HTML image sources; (2) OpenAIPipeline processes news texts through the gpt-4o-mini rewriter; and (3) WordPressArticlePipeline posts the new article.
This architecture keeps tasks decoupled, ensuring that media download errors do not block the AI translation or publishing steps.
b. AI Prompt Engineering & Data Injection
We wrote system prompts to strip outbound links, translate text to English, and keep the original HTML markup layout intact, preserving image tags and block styles.
The parsed text is injected into the WordPress REST API payload, assigning custom categories (Daily Summary, Trading Analysis) and tags automatically.
05 — Usability Testing
Usability Testing
a. API Latency & Execution Time
Running OpenAI requests during scrap campaigns can cause execution timeouts. The original GPT-4 model averaged 8 seconds per call, causing HTTP delays.
We optimized the pipeline by migrating the prompt workload to the gpt-4o-mini model. This reduced average request latency by 65% (down to 2.8 seconds) while maintaining translation and rewriting accuracy.
b. Automated Content Review System
To prevent publishing duplicate articles or repeating api calls, we built an automated database logging check. The system logs processed post IDs in a local SQLite file (db.py), checking for duplicates before executing OpenAI calls.
06 — Prototyping & Handoff
Prototyping & Handoff
a. Plugin Deployment & Configuration Dashboard
To facilitate easy handoff, we developed a settings page (build_options_page) inside the WordPress admin. Administrators can set scraping parameters, update API credentials, and configure target category mappings.
We packaged the settings page and shortcode templates into a clean zip format, complete with step-by-step documentation detailing environment configurations, API key registration, and Docker commands.
07 — Key Takeaways
Key Takeaways
a. Enhancing Legacy Systems with AI
This project proved that legacy WordPress sites can be enhanced with modern AI features by building decoupled Python microservices, bypassing PHP memory bottlenecks.
b. Reliable Data Scraping at Scale
Emulating mobile client behaviors and using structured JSON prompts is critical when scraping data from protected APIs and building reliable automated workflows.