mgnipy.V2.datasets.taxonomic module

Contents

mgnipy.V2.datasets.taxonomic module#

class DWCTaxaMGazine(downloads, config=None, *, client=None, mgnify_studies=None, mgnify_analyses=None, mgnify_runs=None, mgnify_samples=None, mgnify_assemblies=None, biosamples_metadata=None, obs=None)[source]#

Bases: MGazine

A special MGazine for handling Darwin Core (DwC) ready taxonomic datasets.

This class extends MGazine providing additional functionality for working with taxonomic assignment data and metadata from MGnify that has been formated in a standard Darwin Core (`DwC`_) format.

There are methods for merging the feature matrices (X) and getting the taxonomic label metadata (i.e., taxonomic ranks). Additionally the data can be converted into DataFrames, polars.DataFrame or pandas.DataFrame or anndata.AnnData .

:param See MGazine for parameters:

Parameters:
taxonomic_metadata[source]#

The taxonomic metadata as either a polars.DataFrame or a :class:`pandas.DataFrame, depending on the specified engine. The taxonomic ranks are split into separate columns.

Type:

pl.DataFrame | pd.DataFrame

Parameters:

df_engine (Literal ['polars', 'pandas'])

Return type:

DataFrame | DataFrame

X#

The feature matrix (X) from the merged taxonomic datasets - counts. The feature matrix contains the non-taxonomic columns from the merged dataset.

Type:

pl.DataFrame | pd.DataFrame

to_anndata#

Converts the taxonomic metadata and feature matrix into an anndata.AnnData dataframe. The taxonomic ranks are stored in the obs attribute of the AnnData object, and the feature matrix X() is sent to the ad.Anndata.X() property.

Type:

ad.AnnData

runs_accessions#

A list of run accessions from the merged taxonomic datasets. This property retrieves the list of run accessions from the merged taxonomic datasets from the ‘RunID’ column.

Type:

list

see also :class:`MGazine` for additional attributes and methods.
.. _DwC
Type:

https://dwc.tdwg.org/

async aclose()#
async adownload(to_dir, alias=None, *, url=None, filename=None, overwrite=False, hide_progress=False)#

Asynchronously download a file from an alias or URL.

Parameters:

  • to_dir (DirectoryPath) – Directory where the file will be saved.

  • alias (str or None, optional) – Download alias known to this MGazine instance.

  • url (str or None, optional) – Direct URL to fetch. Either alias or url must be provided.

  • filename (str or None, optional) – Filename to use for the saved file. When omitted the alias is used.

  • httpx_aclient (httpx.AsyncClient, optional) – Optional httpx.AsyncClient to use for the HTTP request.

  • overwrite (bool , optional) – If False and the destination file already exists the download is skipped. When True the existing file will be overwritten.

  • hide_progress (bool , optional) – Disable the progress bar when True.

Raises:

ValueError – If neither alias nor url is provided.

Examples

downloads = [ … { … “alias”: “example.txt”, … “url”: “http://ex/x ”, … “file_type”: “txt”, … }] mg = MGazine(downloads) await mg.adownload(“download_to_here”, alias=”example.txt”) # doctest: +SKIP

async adownload_all(to_dir, overwrite=False, hide_progress=False)#

Asynchronously download all files known to this MGazine.

Parameters:

  • to_dir (DirectoryPath) – Directory where the files will be saved.

  • overwrite (bool , optional) – Passed to adownload to control overwriting behavior.

  • hide_progress (bool , optional) – Disable progress bars when True.

Notes

This helper creates a single async HTTP client and schedules concurrent adownload calls for all aliases.

Examples

>>> downloads = [
...     {"alias": "example.txt", "url": "http://ex/x", "file_type": "txt"},
...     {"alias": "example2.fasta.gz", "url": "http://ex/x2", "file_type": "fasta"},
... ]
>>> mg = MGazine(downloads)
>>> await mg.adownload_all("download_to_here")
property aliases: list [str ]#

Return a list of all download aliases.

Example

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x"}]
>>> MGazine(downloads).aliases
['example.txt']
append_biosamples_metadata(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_analyses(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_assemblies(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_runs(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_samples(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_studies(value)#
Parameters:

value (dict [str , Any ])

append_obs(value)#
Parameters:

value (dict [str , Any ])

property async_httpx_client: AsyncClient#

Get the asynchronous httpx client instance from the AuthenticatedClient.

Returns:

The asynchronous httpx client instance.

Return type:

httpx.AsyncClient

property available_metadata_sets: list [str ]#

Return a list of available metadata sets in the MGazine.

This property checks which metadata sets (e.g., studies, analyses, runs, samples, assemblies, biosamples) are non-empty and returns their names as a list.

Returns:

A list of names of non-empty metadata sets available in the MGazine.

Return type:

list of str

Examples

>>> mg = MGazine(downloads)
>>> mg.available_metadata_sets
['mgnify_studies', 'mgnify_analyses', 'mgnify_runs']
property biosamples_metadata: ResultsHandler#
by_downloads_col(col)#

Group downloads by a specified column in the downloads dataframe.

Parameters:

col (str ) – The column name to group by.

Returns:

A dictionary where keys are unique values from the specified column and values are lists of download dictionaries.

Return type:

dict

Raises:

ValueError – If the specified column is not present in the downloads dataframe.

close()#
download(to_dir, alias=None, *, url=None, filename=None, overwrite=False, hide_progress=False)#

Download a file by its alias or URL.

Download a file from an alias or URL to a local directory.

Parameters:

  • to_dir (DirectoryPath) – Directory where the file will be saved.

  • alias (str or None, optional) – Download alias known to this MGazine instance. When provided the corresponding URL from the instance’s downloads list is used.

  • url (str or None, optional) – Direct URL to fetch. Either alias or url must be provided.

  • filename (str or None, optional) – Filename to use for the saved file. When omitted the alias is used.

  • overwrite (bool , optional) – If False and the destination file already exists the download is skipped. When True the existing file will be overwritten.

  • hide_progress (bool , optional) – Disable the progress bar when True.

Raises:

ValueError – If neither alias nor url is provided.

Examples

mg = MGazine(downloads) # doctest: +SKIP mg.download(“download_to_here”, alias=”example.txt”) # doctest: +SKIP

download_all(to_dir, hide_progress=False, overwrite=False)#

Download all files known to this MGazine instance.

Parameters:

  • to_dir (DirectoryPath) – Directory where the files will be saved.

  • hide_progress (bool , optional) – Disable per-file and overall progress bars when True.

  • overwrite (bool , optional) – Passed to download to control overwriting behavior.

Notes

This helper calls download for each alias present in the instance’s downloads list.

Examples

>>> downloads = [
...     {"alias": "example.txt", "url": "http://ex/x", "file_type": "txt"},
...     {"alias": "example2.fasta.gz", "url": "http://ex/x2", "file_type": "fasta"},
... ]
>>> mg = MGazine(downloads)
>>> mg.download_all("download_to_here")
downloads_df(**pd_kwargs)#

The downloads as a DataFrame.

This returns a pandas.DataFrame of all downloads. The dataframe should contain columns such as alias, url and file_type (TODO pandera).

Parameters:

pd_kwargs (dict ) – Additional keyword arguments to pass to the pandas.DataFrame constructor.

Returns:

A DataFrame containing the downloads information

Return type:

pd.DataFrame

Examples

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x", "file_type": "txt"}]
>>> mag = MGazine(downloads)
>>> df = mag.downloads_df(index=["boop"])
property httpx_client: Client#

Get the synchronous httpx client instance from the AuthenticatedClient.

Returns:

The synchronous httpx client instance.

Return type:

httpx.Client

lazy_concat(aliases=None, urls=None, how='vertical_relaxed', **pl_kwargs)#

Return a concatenated Polars LazyFrame of the datasets corresponding to the provided aliases or URLs.

Parameters:

  • aliases (list [str ] or None, optional) – List of download aliases to stream and concatenate. If provided, this takes precedence over urls.

  • urls (list [str ] or None, optional) – List of download URLs to stream and concatenate. Used only if aliases is not provided.

  • how (str , optional) – Concatenation method. Options include ‘vertical’, ‘horizontal’, ‘vertical_relaxed’, etc. See Polars documentation for details.

  • **pl_kwargs – Additional keyword arguments to pass to the Polars concatenation function.

Returns:

A Polars LazyFrame representing the concatenated datasets.

Return type:

pl.LazyFrame

property lazy_merged: LazyFrame | None #

Return the current lazy merged Polars LazyFrame if available.

Returns:

The current lazy merged Polars LazyFrame, or None if not set.

Return type:

pl.LazyFrame or None

list_pipeline_version()#

A list of unique pipeline versions in the MGazine.

Returns:

A list of unique pipeline versions extracted from the downloads.

Return type:

list of str

Examples

>>> downloads = [
...     {"alias": "example.txt", "url": "http://ex/x", "pipeline_version": 'v4_1'},
...     {"alias": "example2.txt", "url": "http://ex/x2", "pipeline_version": 'v5'},
... ]
>>> MGazine(downloads).list_pipeline_version()
['v4_1', 'v5']
list_short_descriptions()#

A list of unique short descriptions of the downloads.

The unique short descriptions in the given column

Returns:

A list of unique short descriptions extracted from the downloads.

Return type:

list of str

Examples

>>> downloads = [
...     {"alias": "example.txt", "short_description": "shortdesc1"},
...     {"alias": "boo.txt", "short_description": "shortdesc1"},
...     {"alias": "example2.txt", "short_description": "shortdesc2"},
... ]
>>> MGazine(downloads).list_short_descriptions()
['shortdesc1', 'shortdesc2']
load()[source]#

Lazy load taxonomic datasets.

This method lazily loads and attempts to merge all the datasets contained in url_list(). Lazy loads as a polars.LazyFrame which can then be accessed via property lazy_merged()). Doesnt return anything.

Return type:

None

property long_short_mapping: dict [str , str ]#

Returns the long to short taxonomic rank mapping based on the short description of the dataset.

Returns:

A dictionary mapping the long taxonomic rank names (e.g., “Superkingdom”) to their corresponding short prefixes (e.g., “sk”).

Return type:

dict [str , str ]

property mgnify_analyses: MGnifyMetadata#
property mgnify_assemblies: MGnifyMetadata#
property mgnify_runs: MGnifyMetadata#
property mgnify_samples: MGnifyMetadata#
property mgnify_studies: MGnifyMetadata#
property obs: ResultsHandler#
obs_metadata(df_engine='pandas', expand_nested_dicts=True, drop_duplicates=False, how='left', coalesce=True, for_runs=None, index_col_name='_mgnipy_runs_accs')#
Parameters:
Return type:

DataFrame | DataFrame

renew_client()#

Init a new client instance and replace the existing one. This is useful if the current client has been closed or is no longer valid, allowing for a fresh start with a new HTTP client session.

property runs_accessions: list #

The list of run accessions from the merged taxonomic datasets.

Notes

  • This property retrieves the list of run accessions from the merged taxonomic datasets.

  • If the run accessions have already been computed and cached, it returns the cached value.

  • Otherwise, it attempts to compute the run accessions by selecting the “RunID” column from the merged dataset and collecting it into a list.

property short_desc: str #

The short description of the MGazine.

This property returns the FIRST short description of the MGazine, which is derived from the downloads. If multiple short descriptions are present, a warning is issued.

status()#

Print the status of the MGnipy client, including the type of client and whether the synchronous and asynchronous httpx client sessions are open.

Return type:

None

stream(*, alias=None, url=None, chunksize=None, max_skip=5, **kwargs)#

Streams a single download based on its alias or url.

If chunksize is specified then iterators of dataframes or strings will be returned; otherwise the full data will be returned as a single object.

Supported formats and their handlers#

param alias:

The alias of the download to stream.

type alias:

Optional[str]

param url:

The url of the download to stream.

type url:

Optional[HttpUrl]

param chunksize:

The size of the chunks to read from the stream.

type chunksize:

Optional[int]

param max_skip:

The maximum number of rows to skip before raising an error. Default is 5.

type max_skip:

int, optional

param **kwargs:

Additional keyword arguments to pass to the streamer function.

returns:

The streamer result for the resolved alias or url.

rtype:

Any

Parameters:

  • alias (str | None)

  • url (HttpUrl | None)

  • chunksize (int | None)

  • max_skip (int )

Return type:

Any

stream_biom(url, **skbio_kwargs)#

Stream a biom file from a URL using scikit-bio’s read function. Refer there for more info.

Parameters:

  • url (str ) – The URL to the biom file to stream.

  • **skbio_kwargs – Additional keyword arguments passed to skbio.io.read(), such as into and verify.

Returns:

A generator yielding scikit-bio Sequence objects parsed from the biom file.

Return type:

Generator

stream_fasta(url, **skbio_kwargs)#

Stream a FASTA file from a URL using scikit-bio’s read function. Refer there for more info.

Parameters:

  • url (str ) – The URL to the FASTA file to stream.

  • **skbio_kwargs – Additional keyword arguments passed to skbio.io.read(), such as into and verify.

Returns:

A generator yielding scikit-bio Sequence objects parsed from the FASTA file.

Return type:

Generator

stream_gff(url, **skbio_kwargs)#

Stream a GFF file from a URL using scikit-bio’s read function. Refer there for more info.

Parameters:

  • url (str ) – The URL to the GFF file to stream.

  • **skbio_kwargs – Additional keyword arguments passed to skbio.io.read(), such as into and verify.

Returns:

A generator yielding scikit-bio Sequence objects parsed from the GFF file.

Return type:

Generator

stream_gzipped(url, chunksize=None, decode=False, encoding='utf-8', errors='replace', **httpx_kwargs)#

Stream a gzipped HTTP resource and present a file-like interface.

When chunksize is None the entire compressed payload is fetched and decompressed into memory. When chunksize is provided a streaming file-like object is returned.

Parameters:
Return type:

bytes | str | BufferedReader | TextIOWrapper

stream_html(url, **web_kwargs)#

Open an HTML URL in the default web browser.

Parameters:

  • url (str ) – The URL to open in the web browser.

  • **web_kwargs – Additional keyword arguments passed to webbrowser.open(), such as new and autoraise.

Returns:

True if the URL was opened successfully, False otherwise.

Return type:

bool

stream_json(url, chunksize=None, **httpx_kwargs)#
Parameters:
Return type:

dict | Generator

stream_jsonl(url, orient=None, chunksize=None, df_engine='pandas', **df_kwargs)#
Parameters:

  • url (str )

  • orient (Literal ['records', 'split', 'index', 'columns', 'values', 'table'] | None)

  • chunksize (int | None)

  • df_engine (Literal ['pandas', 'polars'] | None)

Return type:

dict

stream_pandas(url, sep='\t', chunksize=None, max_skip=5, low_memory=False, **pd_kwargs)#

Read a TSV from a URL or local file with resilient header handling.

The helper will retry with increasing skiprows when pandas raises a ParserError (useful for files with extra header lines). When chunksize is provided an iterator is returned.

Parameters:

  • url (str ) – The URL or local file path to read the TSV from.

  • sep (str ) – The delimiter to use (default is tab).

  • chunksize (int or None) – If an integer is provided, returns an iterator that yields DataFrames of that many rows. If None, returns a single DataFrame.

  • max_skip (int ) – The maximum number of lines to skip when trying to parse the TSV.

  • **pd_kwargs – Additional keyword arguments passed to pd.read_csv.

  • low_memory (bool )

Returns:

A DataFrame containing the TSV data, or an iterator yielding DataFrames if chunksize is specified.

Return type:

pd.DataFrame or TextFileReader

Raises:

  • ValueError – If chunksize is not a positive integer or None.

  • RuntimeError – If the TSV cannot be parsed after skipping up to max_skip lines.

  • Pandas ParserError – If the TSV cannot be parsed due to a format error (after retries).

stream_polars(url, sep='\t', chunksize=None, max_skip=5, low_memory=False, **pl_kwargs)#

Read a TSV from a URL or local file into a Polars DataFrame with resilient header handling.

The helper will retry with increasing skip_rows when Polars raises an error (useful for files with extra header lines). When chunksize is provided an iterator is returned.

Parameters:

  • url (str ) – The URL or local file path to read the TSV from.

  • sep (str ) – The delimiter to use (default is tab).

  • chunksize (int or None) – If an integer is provided, returns an iterator that yields DataFrames of that many rows. If None, returns a single DataFrame.

  • max_skip (int ) – The maximum number of lines to skip when trying to parse the TSV.

  • **pl_kwargs – Additional keyword arguments passed to pl.read_csv.

  • low_memory (bool )

Returns:

A Polars DataFrame containing the TSV data, or an iterator yielding DataFrames if chunksize is specified.

Return type:

pl.DataFrame or Iterator[pl.DataFrame]

Raises:

  • ValueError – If chunksize is not a positive integer or None.

  • RuntimeError – If the TSV cannot be parsed after skipping up to max_skip lines.

  • Polars Error – If the TSV cannot be parsed due to a format error (after retries).

stream_tree(url, **skbio_kwargs)#
Parameters:

url (str )

Return type:

Generator

stream_txt(url, chunksize=None, **httpx_kwargs)#

Stream a plain-text resource. When chunksize is None the full text is returned as a string. When chunksize is an integer the function yields lists of lines.

Parameters:

  • url (str ) – The URL to stream the text from.

  • chunksize (int or None) – If an integer is provided, yields lists of lines of that size. If None, yields the entire text as a single string.

  • httpx_client (httpx.Client, optional) – An optional httpx.Client to use for the request. If None, a new client will be created for the request.

  • **httpx_kwargs – Additional keyword arguments passed to the httpx.Client.request() method

Returns:

The full text as a string if chunksize is None, or a generator yielding lists of lines if chunksize is an integer.

Return type:

str or Generator

taxonomic_metadata(df_engine='pandas')[source]#

Gets the taxonomic metadata.

Prepares the taxonomic metadata DataFrame by splitting the taxonomy string into separate columns for each taxonomic rank.

Parameters:

df_engine (Literal["polars", "pandas"], optional) – The DataFrame engine to use for the output. If “polars” is specified, a polars.DataFrame is returned; if “pandas” is specified, a pandas.DataFrame is returned.

Return type:

DataFrame | DataFrame

to_pandas(**pd_kwargs)#
Return type:

DataFrame

to_polars()#
Return type:

DataFrame

property url_dict: dict [str , dict ]#

Return mapping of alias to URL for all downloads.

Returns:

Dictionary mapping alias -> url (or None when no url is available).

Return type:

dict

Examples

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x"}]
>>> MGazine(downloads).url_dict
{'example.txt': 'http://ex/x'}
property url_list#

Return a list of all download URLs.

Examples

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x"}]
>>> MGazine(downloads).urls
['http://ex/x']
property urls: list [str | None ]#

Return a list of all download URLs. Same as url_list().

Examples

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x"}]
>>> MGazine(downloads).urls
['http://ex/x']
class TaxaMGazine(downloads, config=None, *, client=None, mgnify_studies=None, mgnify_analyses=None, mgnify_runs=None, mgnify_samples=None, mgnify_assemblies=None, biosamples_metadata=None, obs=None)[source]#

Bases: MGazine

A special MGazine for handling taxonomic datasets.

This class extends MGazine providing additional functionality for working with taxonomic assignment data and metadata from MGnify.

There are methods for merging the feature matrices (X) and getting the taxonomic label metadata (i.e., taxonomic ranks). Additionally the data can be converted into DataFrames, polars.DataFrame or pandas.DataFrame or anndata.AnnData .

:param See MGazine for parameters:

Parameters:
taxonomic_metadata[source]#

The taxonomic metadata as either a polars.DataFrame or a :class:`pandas.DataFrame, depending on the specified engine. The taxonomic ranks are split into separate columns.

Type:

pl.DataFrame | pd.DataFrame

Parameters:
Return type:

DataFrame | DataFrame

X[source]#

The feature matrix (X) from the merged taxonomic datasets - counts. The feature matrix contains the non-taxonomic columns from the merged dataset.

Type:

pl.DataFrame | pd.DataFrame

Parameters:

df_engine (Literal ['polars', 'pandas'])

Return type:

DataFrame | DataFrame

to_anndata[source]#

Converts the taxonomic metadata and feature matrix into an anndata.AnnData dataframe. The taxonomic ranks are stored in the obs attribute of the AnnData object, and the feature matrix X() is sent to the ad.Anndata.X() property.

Type:

ad.AnnData

Parameters:

drop_duplicates (bool )

Return type:

AnnData

runs_accessions#

A list of run accessions from the merged taxonomic datasets. This property retrieves the list of run accessions from the merged taxonomic datasets from the ‘RunID’ column.

Type:

list

see also :class:`MGazine` for additional attributes and methods.
.. _DwC
Type:

https://dwc.tdwg.org/

X(df_engine='pandas')[source]#

Gets the feature matrix (X) from the merged taxonomic datasets.

Parameters:

df_engine (Literal["polars", "pandas"], optional) – The DataFrame engine to use for the output. If “polars” is specified, a polars.DataFrame is returned; if “pandas” is specified, a pandas.DataFrame is returned.

Returns:

The feature matrix (X) containing the non-taxonomic columns from the merged taxonomic datasets

Return type:

pl.DataFrame or pd.DataFrame

async aclose()#
async adownload(to_dir, alias=None, *, url=None, filename=None, overwrite=False, hide_progress=False)#

Asynchronously download a file from an alias or URL.

Parameters:

  • to_dir (DirectoryPath) – Directory where the file will be saved.

  • alias (str or None, optional) – Download alias known to this MGazine instance.

  • url (str or None, optional) – Direct URL to fetch. Either alias or url must be provided.

  • filename (str or None, optional) – Filename to use for the saved file. When omitted the alias is used.

  • httpx_aclient (httpx.AsyncClient, optional) – Optional httpx.AsyncClient to use for the HTTP request.

  • overwrite (bool , optional) – If False and the destination file already exists the download is skipped. When True the existing file will be overwritten.

  • hide_progress (bool , optional) – Disable the progress bar when True.

Raises:

ValueError – If neither alias nor url is provided.

Examples

downloads = [ … { … “alias”: “example.txt”, … “url”: “http://ex/x ”, … “file_type”: “txt”, … }] mg = MGazine(downloads) await mg.adownload(“download_to_here”, alias=”example.txt”) # doctest: +SKIP

async adownload_all(to_dir, overwrite=False, hide_progress=False)#

Asynchronously download all files known to this MGazine.

Parameters:

  • to_dir (DirectoryPath) – Directory where the files will be saved.

  • overwrite (bool , optional) – Passed to adownload to control overwriting behavior.

  • hide_progress (bool , optional) – Disable progress bars when True.

Notes

This helper creates a single async HTTP client and schedules concurrent adownload calls for all aliases.

Examples

>>> downloads = [
...     {"alias": "example.txt", "url": "http://ex/x", "file_type": "txt"},
...     {"alias": "example2.fasta.gz", "url": "http://ex/x2", "file_type": "fasta"},
... ]
>>> mg = MGazine(downloads)
>>> await mg.adownload_all("download_to_here")
property aliases: list [str ]#

Return a list of all download aliases.

Example

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x"}]
>>> MGazine(downloads).aliases
['example.txt']
append_biosamples_metadata(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_analyses(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_assemblies(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_runs(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_samples(value)#
Parameters:

value (dict [str , Any ])

append_mgnify_studies(value)#
Parameters:

value (dict [str , Any ])

append_obs(value)#
Parameters:

value (dict [str , Any ])

property async_httpx_client: AsyncClient#

Get the asynchronous httpx client instance from the AuthenticatedClient.

Returns:

The asynchronous httpx client instance.

Return type:

httpx.AsyncClient

property available_metadata_sets: list [str ]#

Return a list of available metadata sets in the MGazine.

This property checks which metadata sets (e.g., studies, analyses, runs, samples, assemblies, biosamples) are non-empty and returns their names as a list.

Returns:

A list of names of non-empty metadata sets available in the MGazine.

Return type:

list of str

Examples

>>> mg = MGazine(downloads)
>>> mg.available_metadata_sets
['mgnify_studies', 'mgnify_analyses', 'mgnify_runs']
property biosamples_metadata: ResultsHandler#
by_downloads_col(col)#

Group downloads by a specified column in the downloads dataframe.

Parameters:

col (str ) – The column name to group by.

Returns:

A dictionary where keys are unique values from the specified column and values are lists of download dictionaries.

Return type:

dict

Raises:

ValueError – If the specified column is not present in the downloads dataframe.

close()#
download(to_dir, alias=None, *, url=None, filename=None, overwrite=False, hide_progress=False)#

Download a file by its alias or URL.

Download a file from an alias or URL to a local directory.

Parameters:

  • to_dir (DirectoryPath) – Directory where the file will be saved.

  • alias (str or None, optional) – Download alias known to this MGazine instance. When provided the corresponding URL from the instance’s downloads list is used.

  • url (str or None, optional) – Direct URL to fetch. Either alias or url must be provided.

  • filename (str or None, optional) – Filename to use for the saved file. When omitted the alias is used.

  • overwrite (bool , optional) – If False and the destination file already exists the download is skipped. When True the existing file will be overwritten.

  • hide_progress (bool , optional) – Disable the progress bar when True.

Raises:

ValueError – If neither alias nor url is provided.

Examples

mg = MGazine(downloads) # doctest: +SKIP mg.download(“download_to_here”, alias=”example.txt”) # doctest: +SKIP

download_all(to_dir, hide_progress=False, overwrite=False)#

Download all files known to this MGazine instance.

Parameters:

  • to_dir (DirectoryPath) – Directory where the files will be saved.

  • hide_progress (bool , optional) – Disable per-file and overall progress bars when True.

  • overwrite (bool , optional) – Passed to download to control overwriting behavior.

Notes

This helper calls download for each alias present in the instance’s downloads list.

Examples

>>> downloads = [
...     {"alias": "example.txt", "url": "http://ex/x", "file_type": "txt"},
...     {"alias": "example2.fasta.gz", "url": "http://ex/x2", "file_type": "fasta"},
... ]
>>> mg = MGazine(downloads)
>>> mg.download_all("download_to_here")
downloads_df(**pd_kwargs)#

The downloads as a DataFrame.

This returns a pandas.DataFrame of all downloads. The dataframe should contain columns such as alias, url and file_type (TODO pandera).

Parameters:

pd_kwargs (dict ) – Additional keyword arguments to pass to the pandas.DataFrame constructor.

Returns:

A DataFrame containing the downloads information

Return type:

pd.DataFrame

Examples

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x", "file_type": "txt"}]
>>> mag = MGazine(downloads)
>>> df = mag.downloads_df(index=["boop"])
property httpx_client: Client#

Get the synchronous httpx client instance from the AuthenticatedClient.

Returns:

The synchronous httpx client instance.

Return type:

httpx.Client

lazy_concat(aliases=None, urls=None, how='vertical_relaxed', **pl_kwargs)#

Return a concatenated Polars LazyFrame of the datasets corresponding to the provided aliases or URLs.

Parameters:

  • aliases (list [str ] or None, optional) – List of download aliases to stream and concatenate. If provided, this takes precedence over urls.

  • urls (list [str ] or None, optional) – List of download URLs to stream and concatenate. Used only if aliases is not provided.

  • how (str , optional) – Concatenation method. Options include ‘vertical’, ‘horizontal’, ‘vertical_relaxed’, etc. See Polars documentation for details.

  • **pl_kwargs – Additional keyword arguments to pass to the Polars concatenation function.

Returns:

A Polars LazyFrame representing the concatenated datasets.

Return type:

pl.LazyFrame

property lazy_merged: LazyFrame | None #

Return the current lazy merged Polars LazyFrame if available.

Returns:

The current lazy merged Polars LazyFrame, or None if not set.

Return type:

pl.LazyFrame or None

list_pipeline_version()#

A list of unique pipeline versions in the MGazine.

Returns:

A list of unique pipeline versions extracted from the downloads.

Return type:

list of str

Examples

>>> downloads = [
...     {"alias": "example.txt", "url": "http://ex/x", "pipeline_version": 'v4_1'},
...     {"alias": "example2.txt", "url": "http://ex/x2", "pipeline_version": 'v5'},
... ]
>>> MGazine(downloads).list_pipeline_version()
['v4_1', 'v5']
list_short_descriptions()#

A list of unique short descriptions of the downloads.

The unique short descriptions in the given column

Returns:

A list of unique short descriptions extracted from the downloads.

Return type:

list of str

Examples

>>> downloads = [
...     {"alias": "example.txt", "short_description": "shortdesc1"},
...     {"alias": "boo.txt", "short_description": "shortdesc1"},
...     {"alias": "example2.txt", "short_description": "shortdesc2"},
... ]
>>> MGazine(downloads).list_short_descriptions()
['shortdesc1', 'shortdesc2']
load()[source]#
Return type:

None

property long_short_mapping: dict [str , str ]#

Returns the long to short taxonomic rank mapping based on the short description of the dataset.

Returns:

A dictionary mapping the long taxonomic rank names (e.g., “Superkingdom”) to their corresponding short prefixes (e.g., “sk”).

Return type:

dict [str , str ]

property mgnify_analyses: MGnifyMetadata#
property mgnify_assemblies: MGnifyMetadata#
property mgnify_runs: MGnifyMetadata#
property mgnify_samples: MGnifyMetadata#
property mgnify_studies: MGnifyMetadata#
property obs: ResultsHandler#
obs_metadata(df_engine='pandas', expand_nested_dicts=True, drop_duplicates=False, how='left', coalesce=True, for_runs=None, index_col_name='_mgnipy_runs_accs')#
Parameters:
Return type:

DataFrame | DataFrame

renew_client()#

Init a new client instance and replace the existing one. This is useful if the current client has been closed or is no longer valid, allowing for a fresh start with a new HTTP client session.

property runs_accessions: list #
property short_desc: str #

The short description of the MGazine.

This property returns the FIRST short description of the MGazine, which is derived from the downloads. If multiple short descriptions are present, a warning is issued.

status()#

Print the status of the MGnipy client, including the type of client and whether the synchronous and asynchronous httpx client sessions are open.

Return type:

None

stream(*, alias=None, url=None, chunksize=None, max_skip=5, **kwargs)#

Streams a single download based on its alias or url.

If chunksize is specified then iterators of dataframes or strings will be returned; otherwise the full data will be returned as a single object.

Supported formats and their handlers#

param alias:

The alias of the download to stream.

type alias:

Optional[str]

param url:

The url of the download to stream.

type url:

Optional[HttpUrl]

param chunksize:

The size of the chunks to read from the stream.

type chunksize:

Optional[int]

param max_skip:

The maximum number of rows to skip before raising an error. Default is 5.

type max_skip:

int, optional

param **kwargs:

Additional keyword arguments to pass to the streamer function.

returns:

The streamer result for the resolved alias or url.

rtype:

Any

Parameters:

  • alias (str | None)

  • url (HttpUrl | None)

  • chunksize (int | None)

  • max_skip (int )

Return type:

Any

stream_biom(url, **skbio_kwargs)#

Stream a biom file from a URL using scikit-bio’s read function. Refer there for more info.

Parameters:

  • url (str ) – The URL to the biom file to stream.

  • **skbio_kwargs – Additional keyword arguments passed to skbio.io.read(), such as into and verify.

Returns:

A generator yielding scikit-bio Sequence objects parsed from the biom file.

Return type:

Generator

stream_fasta(url, **skbio_kwargs)#

Stream a FASTA file from a URL using scikit-bio’s read function. Refer there for more info.

Parameters:

  • url (str ) – The URL to the FASTA file to stream.

  • **skbio_kwargs – Additional keyword arguments passed to skbio.io.read(), such as into and verify.

Returns:

A generator yielding scikit-bio Sequence objects parsed from the FASTA file.

Return type:

Generator

stream_gff(url, **skbio_kwargs)#

Stream a GFF file from a URL using scikit-bio’s read function. Refer there for more info.

Parameters:

  • url (str ) – The URL to the GFF file to stream.

  • **skbio_kwargs – Additional keyword arguments passed to skbio.io.read(), such as into and verify.

Returns:

A generator yielding scikit-bio Sequence objects parsed from the GFF file.

Return type:

Generator

stream_gzipped(url, chunksize=None, decode=False, encoding='utf-8', errors='replace', **httpx_kwargs)#

Stream a gzipped HTTP resource and present a file-like interface.

When chunksize is None the entire compressed payload is fetched and decompressed into memory. When chunksize is provided a streaming file-like object is returned.

Parameters:
Return type:

bytes | str | BufferedReader | TextIOWrapper

stream_html(url, **web_kwargs)#

Open an HTML URL in the default web browser.

Parameters:

  • url (str ) – The URL to open in the web browser.

  • **web_kwargs – Additional keyword arguments passed to webbrowser.open(), such as new and autoraise.

Returns:

True if the URL was opened successfully, False otherwise.

Return type:

bool

stream_json(url, chunksize=None, **httpx_kwargs)#
Parameters:
Return type:

dict | Generator

stream_jsonl(url, orient=None, chunksize=None, df_engine='pandas', **df_kwargs)#
Parameters:

  • url (str )

  • orient (Literal ['records', 'split', 'index', 'columns', 'values', 'table'] | None)

  • chunksize (int | None)

  • df_engine (Literal ['pandas', 'polars'] | None)

Return type:

dict

stream_pandas(url, sep='\t', chunksize=None, max_skip=5, low_memory=False, **pd_kwargs)#

Read a TSV from a URL or local file with resilient header handling.

The helper will retry with increasing skiprows when pandas raises a ParserError (useful for files with extra header lines). When chunksize is provided an iterator is returned.

Parameters:

  • url (str ) – The URL or local file path to read the TSV from.

  • sep (str ) – The delimiter to use (default is tab).

  • chunksize (int or None) – If an integer is provided, returns an iterator that yields DataFrames of that many rows. If None, returns a single DataFrame.

  • max_skip (int ) – The maximum number of lines to skip when trying to parse the TSV.

  • **pd_kwargs – Additional keyword arguments passed to pd.read_csv.

  • low_memory (bool )

Returns:

A DataFrame containing the TSV data, or an iterator yielding DataFrames if chunksize is specified.

Return type:

pd.DataFrame or TextFileReader

Raises:

  • ValueError – If chunksize is not a positive integer or None.

  • RuntimeError – If the TSV cannot be parsed after skipping up to max_skip lines.

  • Pandas ParserError – If the TSV cannot be parsed due to a format error (after retries).

stream_polars(url, sep='\t', chunksize=None, max_skip=5, low_memory=False, **pl_kwargs)#

Read a TSV from a URL or local file into a Polars DataFrame with resilient header handling.

The helper will retry with increasing skip_rows when Polars raises an error (useful for files with extra header lines). When chunksize is provided an iterator is returned.

Parameters:

  • url (str ) – The URL or local file path to read the TSV from.

  • sep (str ) – The delimiter to use (default is tab).

  • chunksize (int or None) – If an integer is provided, returns an iterator that yields DataFrames of that many rows. If None, returns a single DataFrame.

  • max_skip (int ) – The maximum number of lines to skip when trying to parse the TSV.

  • **pl_kwargs – Additional keyword arguments passed to pl.read_csv.

  • low_memory (bool )

Returns:

A Polars DataFrame containing the TSV data, or an iterator yielding DataFrames if chunksize is specified.

Return type:

pl.DataFrame or Iterator[pl.DataFrame]

Raises:

  • ValueError – If chunksize is not a positive integer or None.

  • RuntimeError – If the TSV cannot be parsed after skipping up to max_skip lines.

  • Polars Error – If the TSV cannot be parsed due to a format error (after retries).

stream_tree(url, **skbio_kwargs)#
Parameters:

url (str )

Return type:

Generator

stream_txt(url, chunksize=None, **httpx_kwargs)#

Stream a plain-text resource. When chunksize is None the full text is returned as a string. When chunksize is an integer the function yields lists of lines.

Parameters:

  • url (str ) – The URL to stream the text from.

  • chunksize (int or None) – If an integer is provided, yields lists of lines of that size. If None, yields the entire text as a single string.

  • httpx_client (httpx.Client, optional) – An optional httpx.Client to use for the request. If None, a new client will be created for the request.

  • **httpx_kwargs – Additional keyword arguments passed to the httpx.Client.request() method

Returns:

The full text as a string if chunksize is None, or a generator yielding lists of lines if chunksize is an integer.

Return type:

str or Generator

taxonomic_metadata(fill_na='NA', df_engine='pandas', hide_index=True)[source]#
Parameters:
Return type:

DataFrame | DataFrame

to_anndata(drop_duplicates=True, **anndata_kwargs)[source]#

Converts the taxonomic metadata to an AnnData object. The taxonomic ranks are stored in the obs attribute of the AnnData object.

Parameters:

  • **anndata_kwargs – Additional keyword arguments to pass to the AnnData constructor.

  • drop_duplicates (bool )

Returns:

An AnnData object containing the taxonomic metadata in the obs attribute.

Return type:

ad.AnnData

to_pandas(**pd_kwargs)#
Return type:

DataFrame

to_polars()#
Return type:

DataFrame

property url_dict: dict [str , dict ]#

Return mapping of alias to URL for all downloads.

Returns:

Dictionary mapping alias -> url (or None when no url is available).

Return type:

dict

Examples

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x"}]
>>> MGazine(downloads).url_dict
{'example.txt': 'http://ex/x'}
property url_list#

Return a list of all download URLs.

Examples

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x"}]
>>> MGazine(downloads).urls
['http://ex/x']
property urls: list [str | None ]#

Return a list of all download URLs. Same as url_list().

Examples

>>> downloads = [{"alias": "example.txt", "url": "http://ex/x"}]
>>> MGazine(downloads).urls
['http://ex/x']
long_short_mapper(desc, mapping=None)[source]#
Parameters:
Return type:

dict [str , str ]

prep_obs(df, tax_col, long_short_mapping, fill_na='NA')[source]#

Prepares the taxonomy DataFrame by splitting the taxonomy string into separate columns for each taxonomic rank.

Parameters:

  • df (pl.DataFrame) – A Polars DataFrame containing a column named ‘taxonomy’ with taxonomic classifications in a semicolon-separated format.

  • tax_col (Literal["taxonomy", "#SampleID"]) – The name of the column in the DataFrame that contains the taxonomy string to be split.

  • long_short_mapping (dict [str , str ]) – A dictionary mapping the long taxonomic rank names (e.g., “Superkingdom”) to their corresponding short prefixes (e.g., “sk”). This is used to clean the taxonomic rank values by stripping the short prefixes.

  • fill_na (Any, default="NA") – The value to use for filling empty strings or null values in the taxonomic rank columns after stripping the short prefixes. If not provided, it defaults to “NA”.

Returns:

A Polars DataFrame with separate columns for each taxonomic rank based on the taxonomy ranks defined in the constants.

Return type:

pl.DataFrame

Contents