> ## Documentation Index > Fetch the complete documentation index at: https://docs.cafescraper.com/llms.txt > Use this file to discover all available pages before exploring further. # Python Script # 🧩 Python Script Demo ## GitHub Repository Python Script Demo\ Repository:\ [https://github.com/CafeScraper/PythonScirptDemo](https://github.com/CafeScraper/PythonScirptDemo) *** ## Required Files (Project Root Directory) ```text theme={null} ├── main.py ├── requirements.txt ├── input_schema.json ├── README.md ├── sdk.py ├── sdk_pb2.py ├── sdk_pb2_grpc.py ``` *** ## File Overview | File Name | Description | | :------------------------------------------ | :-------------------------------------------------------- | | [**main.py**](http://main.py) | Script entry file (execution entry), must be named `main` | | **requirements.txt** | Python dependency management file | | **input\_schema.json** | UI input form configuration file | | [**README.md**](http://README.md) | Project documentation | | [**sdk.py**](http://sdk.py) | Core SDK functionality | | **sdk\_**[**pb2.py**](http://pd2.py) | Enhanced data processing module | | **sdk\_pb2\_**[**grpc.py**](http://grpc.py) | Network communication module | *** ## ⭐ Core Script SDK ### 📁 SDK File Description The following three SDK files are **mandatory** and must be placed in the script’s root directory: | File Name | Main Function | | :------------------------------------------ | :------------------------------ | | [**sdk.py**](http://sdk.py) | Core functionality module | | **sdk\_**[**pb2.py**](http://pd2.py) | Enhanced data processing module | | **sdk\_pb2\_**[**grpc.py**](http://grpc.py) | Network communication module | These three files together form the script’s **toolbox**, providing all essential capabilities required for crawler execution and interaction with the platform’s backend system. *** ## 🔧 Core Feature Usage Guide ### 1. Environment Parameters **Retrieve configuration passed at script startup** When the script starts, external configuration parameters can be passed in (such as target website URLs, search keywords, etc.). Use the following method to retrieve them: ```python theme={null} # Retrieve all input parameters as a dictionary parameters = SDK.Parameter.get_input_json_dict() # Example return value: # {"website": "example.com", "keyword": "tech news"} ``` **Use case**:\ If you need to collect data from different websites for different tasks, simply pass different parameters without modifying the script code. *** ### 2. Runtime Logging **Record script execution progress** During execution, you can log messages at different levels. These logs will be displayed in the platform UI, making it easier to monitor execution status and troubleshoot issues. ```python theme={null} # Debug-level logs (most detailed) SDK.Log.debug("Connecting to target website...") # Informational logs (normal execution flow) SDK.Log.info("Successfully collected 10 news articles") # Warning logs (non-fatal issues) SDK.Log.warn("Network latency detected, collection speed may be affected") # Error logs (execution failures) SDK.Log.error("Failed to access target website, please check network connection") ``` #### Log Level Explanation * **debug**: Most detailed logs, recommended during development * **info**: Normal execution flow, recommended at key steps * **warn**: Potential issues that do not stop execution * **error**: Errors that require attention *** ### 3. Returning Results **Send collected data back to the platform** Once data is collected, it must be returned to the platform in **two steps**. *** #### Step 1: Define Table Headers (Required) Before pushing any data, define the table structure—similar to defining column headers in Excel. ```text theme={null} headers = [ { "label": "News Title", "key": "title", "format": "text", }, { "label": "Publish Time", "key": "publish_time", "format": "text", }, { "label": "Category", "key": "category", "format": "text", }, ] # Set table headers res = CafeSDK.Result.set_table_header(headers) ``` ##### Field Explanation * **label**: Column name shown to users (recommended to use descriptive names) * **key**: Unique identifier used in code (recommended lowercase with underscores) * **format**: Data type, supported values: * `"text"`: String / text * `"integer"`: Integer * `"boolean"`: Boolean (true / false) * `"array"`: List / array * `"object"`: Dictionary / object *** #### Step 2: Push Data Records One by One ```python theme={null} news_data = [ {"title": "AI Breakthrough", "publish_time": "2023-10-01", "category": "Technology"}, {"title": "Stock Market Today", "publish_time": "2023-10-01", "category": "Finance"}, ] for i, news in enumerate(news_data): obj = { "title": news.get('title'), "publish_time": news.get('publish_time'), "category": news.get('category'), } res = CafeSDK.Result.push_data(obj) SDK.Log.info(f"Pushed record {i + 1}: {news.get('title')}") ``` ##### Important Notes * Header definition and data pushing **can be done in either order** * Data keys **must exactly match** the header keys (case-sensitive) * Data must be pushed **one record at a time** * Logging after each push is recommended *** ## 💡 Complete Code Example ```python theme={null} from sdk import CafeSDK # 1. Get input parameters config = SDK.Parameter.get_input_json_dict() website = config.get("website", "default website") SDK.Log.info(f"Starting collection for website: {website}") # 2. Set table headers headers = [ {"label": "Title", "key": "title", "format": "text"}, {"label": "Time", "key": "publish_time", "format": "text"}, {"label": "Category", "key": "category", "format": "text"}, {"label": "Views", "key": "view_count", "format": "integer"}, ] CafeSDK.Result.set_table_header(headers) # 3. Simulate data collection SDK.Log.info("Collecting data...") collected_data = [ {"title": "Sample News 1", "publish_time": "2023-10-01 10:00", "category": "Tech", "view_count": 1000}, {"title": "Sample News 2", "publish_time": "2023-10-01 11:00", "category": "Finance", "view_count": 500}, ] # 4. Push data SDK.Log.info(f"Collected {len(collected_data)} records, pushing data...") for data in collected_data: obj = { "title": data.get("title"), "publish_time": data.get("publish_time"), "category": data.get("category"), "view_count": data.get("view_count", 0), } CafeSDK.Result.push_data(obj) # 5. Done SDK.Log.info("Data collection task completed!") ``` *** ## ⭐ Script Entry File ([main.py](http://main.py)) ### 💡 Example ```python theme={null} #!/usr/bin/env python3 # -*- coding: utf-8 -*- import asyncio import os from sdk import CafeSDK async def run(): """ Simplified Instagram Reel data collection example Retains only core SDK calls and proxy configuration """ try: # 1. Get input parameters input_json_dict = CafeSDK.Parameter.get_input_json_dict() CafeSDK.Log.debug(f"Input parameters: {input_json_dict}") # 2. Proxy configuration proxy_domain = "proxy-inner.cafescraper.com:6000" try: proxy_auth = os.environ.get("PROXY_AUTH") CafeSDK.Log.info(f"Proxy auth info: {proxy_auth}") except Exception as e: CafeSDK.Log.error(f"Failed to get proxy auth: {e}") proxy_auth = None proxy_url = f"socks5://{proxy_auth}@{proxy_domain}" if proxy_auth else None CafeSDK.Log.info(f"Proxy URL: {proxy_url}") # 3. Business logic url = input_json_dict.get('url') CafeSDK.Log.info(f"Processing URL: {url}") result = { "url": url, "status": "success", "data": { "title": "Sample Title", "content": "Sample Content", } } # 4. Push result CafeSDK.Result.push_data(result) # 5. Set headers if table output is required headers = [ {"label": "URL", "key": "url", "format": "text"}, {"label": "Status", "key": "status", "format": "text"}, ] CafeSDK.Result.set_table_header(headers) CafeSDK.Log.info("Script execution completed") except Exception as e: CafeSDK.Log.error(f"Execution error: {e}") CafeSDK.Result.push_data({ "error": str(e), "error_code": "500", "status": "failed" }) raise if __name__ == "__main__": asyncio.run(run()) ``` *** ## Automated Data Collection Script: Operation & Principles ### 1. Script Overview This is a **Script** for an automated tool that works like a **“digital employee.”**\ It automatically opens specified web pages (such as social media pages), extracts required information, and organizes it into structured tables. *** ### 2. Execution Workflow The process consists of four main stages: 1. **Receive Instructions**\ Input parameters such as target URLs and data limits 2. **Stealth Preparation**\ Automatic proxy configuration for overseas or restricted websites 3. **Automated Execution**\ Navigate pages and extract titles, content, images, etc. 4. **Result Reporting**\ Push structured data and generate table headers automatically *** ## ⭐ Python Dependency Management (requirements.txt) This file lists all third-party Python packages required to run the script.\ The system automatically installs all dependencies specified in this file. ### Example ```text theme={null} attrs==25.4.0 beautifulsoup4==4.14.2 certifi==2025.10.5 cffi==2.0.0 charset-normalizer==3.4.4 click==8.3.0 colorama==0.4.6 cssselect==1.3.0 DataRecorder==3.6.2 DownloadKit==2.0.7 DrissionPage==4.1.1.2 et_xmlfile filelock ``` *** ## ❗ Important Notes ### 1. Versioning * Packages with versions (e.g. `beautifulsoup4==4.14.2`) will be installed exactly as specified * Packages without versions will install the latest available version ### 2. Installation * Dependencies are installed automatically * Installation time depends on network speed and package size * Errors will be displayed if installation fails ### 3. Ensuring Proper Execution * **grpcio** and **protobuf** must be included (required by the SDK) * All third-party libraries must be listed * Core dependencies should use fixed versions * Update dependencies regularly for security and stability *** ## FAQ **Q: Why specify versions?**\ A: To ensure consistent behavior across development, testing, and production environments. **Q: What if I don’t specify a version?**\ A: The latest version will be installed, which may cause compatibility issues. **Q: How do I add new dependencies?**\ A: Add a new line to this file and re-upload the ZIP package. **Q: What if installation fails?**\ A: Check network connectivity or package mirrors, or contact the system administrator. *** **Last Updated**:\ Always update this dependency list whenever script functionality changes to reflect new dependencies.