Using SmartScraperGraph in a JupyterNotebook or within async functions raises RuntimeError: asyncio.run() cannot be called from a running event loop

[dsanmart]

Describe the bug
When using the SmartScraperGraph in environments that already have an active event loop, such as Jupyter Notebooks or within other async functions, the following error occurs: RuntimeError: asyncio.run() cannot be called from a running event loop
This error is caused by the asyncio.run() function being invoked inside the ScrapeGraphAI library's internals (e.g., within the document loaders). This results in a conflict when an active event loop is already running.

To Reproduce
Steps to reproduce the behavior:

1. Use SmartScraperGraph in a Jupyter Notebook or any environment where an event loop is already running (such as inside another async function).
2. Create a SmartScraperGraph instance and call the run() method.
3. Observe the RuntimeError.

Expected behavior
SmartScraperGraph should work seamlessly in environments where an event loop is already running. It should await async calls directly or provide alternatives that are compatible with environments that have active event loops (like Jupyter Notebooks or async frameworks).

Additional context

• The error originates from calls to asyncio.run() inside the library, particularly when handling web sources (e.g., in scrapegraphai/docloaders/chromium.py).
• A potential fix could involve checking if an event loop is already running and using await instead of asyncio.run() in such cases. I would be happy to make a PR with this implementation.

[VinciGit00]

Pleas at the example notebook in the readme

[dsanmart]

Thanks @VinciGit00 for the quick response and the nest_asyncio suggestion. While it resolves the issue in Jupyter Notebooks, I believe it's more of a temporary workaround than a proper fix.

• nest_asyncio is a patch, not a fix: It allows nested event loops, but it's not ideal for environments with multiple concurrent async tasks. It can introduce complexity and conflicts, especially in larger applications.

• Proper Async integration will help the framework: Native async support without blocking the event loop is essential. The asyncio.run() call in

  Scrapegraph-ai/scrapegraphai/docloaders/chromium.py

  Line 111 in 27ae896

  html_content = asyncio.run(scraping_fn(url))

  blocks the current event loop, which is contrary to the nature of asynchronous execution. This will negate the benefits of async IO, which is inefficient for apps with concurrent tasks.

Many developers work in fully async environments (especially in scraping pipelines xd) and expect libraries to behave consistently. I think native async support would enhance the usability and adoption of your library across diverse projects.

[VinciGit00]

ok, can you suggest the modification in a pr?