Issue with Extracting URLs Using ScrapeGraphAI in Flask Application

[deejay99]

Hi everyone,

I'm currently working on a Flask application that uses ScrapeGraphAI to scrape web pages. While my application successfully scrapes pages and extracts content, I only get NA's back for URLs of listings on scraped pages. Interestingly, the official demo of ScrapeGraphAI can extract URLs of listings on pages without any issues. I'm hoping someone here might be able to help me identify and resolve the problem.

eg on {"source":"https://park-immobilien.ch/kaufen", "prompt":"List me the first 10 listings incl. urls"}

My Setup
Here's a brief overview of my setup:

Framework: Flask
Deployment Platform: Local and Heroku - both having same problem

Code Snippet
Below is the relevant part of my app.py:

from flask import Flask, request, jsonify
from scrapegraphai.graphs import SmartScraperGraph
import os
from dotenv import load_dotenv

load_dotenv()

app = Flask(__name__)

@app.route('/scrape', methods=['POST'])
def scrape():
    data = request.json
    prompt = data.get('prompt', 'List me all the articles')
    source = data.get('source', 'https://perinim.github.io/projects')

    graph_config = {
        "llm": {
            "api_key": os.getenv("OPENAI_API_KEY"),
            "model": "gpt-4o",
        },
    }

    smart_scraper_graph = SmartScraperGraph(
        prompt=prompt,
        source=source,
        config=graph_config
    )

    result = smart_scraper_graph.run()
    return jsonify(result)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

and then

curl -X POST http://localhost:5000/scrape -H "Content-Type: application/json" -d '{"source":"https://park-immobilien.ch/kaufen", "prompt":"List me the first 10 listings incl. their URL"}'

[deejay99]

Solved with new update

[VinciGit00]

Idk man