mgnipy.V2.query_executor module

mgnipy.V2.query_executor module#

class mgnipy.V2.query_executor.QueryExecutor(query_set)[source]#

Bases: object

Parameters:

query_set (QuerySet)

async abulk_fetch(limit=1000, *, pages=None, safety=False, hide_progress=False)[source]#

Fetch pages in bulk asynchronously.

Example

>>> # Async bulk fetch (doctest skipped)
>>> executor = QueryExecutor(qs)
>>> import asyncio
>>> asyncio.run(executor.abulk_fetch(limit=50))
Parameters:
async afirst()[source]#

Asynchronously retrieve the first page of metadata for the current resource and parameters. Same as preview() but returns the raw dictionary instead of a DataFrame.

Return type:

dict

async aget()[source]#

Async alternative to fetch the next page.

Return type:

The next page dict or None when iteration is complete.

Example

>>> # Async fetch via helper (doctest skipped)
>>> import asyncio
>>> executor = QueryExecutor(qs)
>>> asyncio.run(executor.aget())
async apage(page_num, client=None)[source]#

Async fetch for a single page.

Example

>>> import asyncio
>>> executor = QueryExecutor(qs)
>>> asyncio.run(executor.apage(1))
Parameters:
Return type:

dict [int , list [dict ]] | None

bulk_fetch(limit=1000, *, pages=None, safety=False, hide_progress=False)[source]#

Fetch pages in bulk synchronously.

Example

>>> # Bulk fetch usage (doctest skipped)
>>> executor = QueryExecutor(qs)
>>> executor.bulk_fetch(limit=50)
Parameters:
continue_iterator(start_page=None)[source]#

  • Continue iterating from a given page or next batch after pages_limit

  • For resuming after hitting the page limit

Parameters:

start_page (int , optional) – The page number to start from. If None, starts from the next page after the current limit.

Examples

>>> # Continue from a specific page
>>> executor = QueryExecutor(qs)
>>> executor.continue_iterator(start_page=50)
>>> # Continue from next batch after previous pages
>>> executor.continue_iterator()
first()[source]#

Retrieve the first page of metadata for the current resource and parameters. Same as preview() but returns the raw dictionary instead of a DataFrame.

Return type:

dict

get()[source]#

Alternative to getting the next page of results.

Return type:

The next page dict or None when iteration is complete.

Example

>>> # Fetch next page via helper (doctest skipped)
>>> executor = QueryExecutor(qs)
>>> executor.get()
property last_successful_page: int | None #
async map_with_concurrency(items, worker, hide_progress=False)[source]#

Map a worker function over a list of items with controlled concurrency. In plain English, it is a “process these things in parallel, but not too many at once” helper.

Example

>>> # Map worker over pages with concurrency (doctest skipped)
>>> executor = QueryExecutor(qs)
>>> pages = [1,2,3]
>>> results = await executor.map_with_concurrency(
...     items=pages,
...     worker=lambda p: executor.apage(p),
... )
Parameters:

hide_progress (bool )

page(page_num, client=None)[source]#

Retrieve a specific page of metadata for the current resource and parameters. This method allows the user to retrieve metadata one page at a time, which can be useful for previewing data or for manual pagination control.

Parameters:

  • page_num (int ) – The page number to retrieve (1-based index).

  • client (Client, optional) – An optional MGnify API client instance to use for the request. If None, a new client will be initialized.

Returns:

A dictionary containing the metadata from the specified page of results, or None if the page is not found.

Return type:

Optional[dict [int , list [dict ]]]

Examples

>>> # Fetch a single page (doctest skipped)
>>> executor = QueryExecutor(qs)
>>> executor.page(1)
property progress#
query_setups(request_num=None, **httpx_kwargs)[source]#
Parameters:

request_num (int | None)

Return type:

dict [dict [str , Any ]]

reset_iterator()[source]#

Reset the iterator to start from the beginning.

Example

>>> executor = QueryExecutor(qs)
>>> executor.reset_iterator()
resume()[source]#

Resume iteration from the page after the last successful one.

Example

>>> executor = QueryExecutor(qs)
>>> executor.resume()
set_counts()[source]#

Helper method to set the count and num_requests attributes based on the current parameters and endpoint.

Example

>>> # Populate qs.count and qs.num_requests (doctest skipped)
>>> executor = QueryExecutor(qs)
>>> executor.set_counts()
Contents