Getting all Wastewater study datasets

Here we demonstrate how mgnipy can be used to build a cross-study taxonomic dataset for a given biome with rich sample metadata from MGnify and BioSamples in a few lines of code.

First starting the session with a MGnipy client

from mgnipy import MGnipy

# configure session 
MG = MGnipy(cache_dir='wwtp')

# selecting the studies resource
studies_resource = MG.studies

# helper to see accepted search params for endpoint
studies_resource.describe_endpoint()

List all studies analysed by MGnify

MGnify studies inherit directly from studies (or projects) in ENA.

Supported parameters:
- order: ListMgnifyStudiesOrderType0 | None | Unset
- biome_lineage: None | str | Unset The lineage to match, including all descendant biomes
- has_analyses_from_pipeline: None | PipelineVersions | Unset If set, will only show studies with analyses from the specified MGnify pipeline version
- search: None | str | Unset Search within study titles and accessions
- page: int | Unset Default: 1.
- page_size: int | None | Unset

now using mgnifier to build and then execute the query set

# preparing query set
wwtp_studies = studies_resource(biome_lineage='root:Engineered:Wastewater')
# helper to preview the query set prior to fetch
wwtp_studies.explain()

https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=1
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=2
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=3
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=4
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=5
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=6
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=7
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=8

# now actually executing queries
async with MG: 
    # get all 8 pages of records from the list endpoint
    await wwtp_studies.aget_all()
    # enrich the records with study metadata from detail endpoint
    await wwtp_studies.aenrich_details()

# taking a look at metdata so far 
wwtp_studies.metadata.to_pandas(expand_nested_dicts=True).head()

	accession	ena_accessions	title	updated_at	downloads	first_accession	biome__biome_name	biome__lineage
0	MGYS00004949	[ERP112636, PRJEB30205]	EMG produced TPA metagenomics assembly of the ...	2026-05-28T15:46:59.833000+00:00	[{'file_type': 'tsv', 'download_type': 'Taxono...	ERP112636	Activated Sludge	root:Engineered:Wastewater:Activated Sludge
1	MGYS00001670	[ERP015835, PRJEB14205]	metagenome of the anaerobic digested sludge be...	2026-05-28T15:46:51.557000+00:00	[{'file_type': 'tsv', 'download_type': 'Taxono...	ERP015835	Water and sludge	root:Engineered:Wastewater:Water and sludge
2	MGYS00006558	[SRP033648, PRJNA230567]	Systems Biology of Lipid Accumulating Organisms	2026-05-28T15:47:02.215000+00:00	[{'file_type': 'tsv', 'download_type': 'Taxono...	SRP033648	Wastewater	root:Engineered:Wastewater
3	MGYS00001566	[ERP021742, PRJEB19684]	Black Water Project	2026-05-28T15:46:51.633000+00:00	[{'file_type': 'tsv', 'download_type': 'Taxono...	ERP021742	Wastewater	root:Engineered:Wastewater
4	MGYS00004898	[ERP112583, PRJEB30152]	EMG produced TPA metagenomics assembly of the ...	2026-05-28T15:46:59.966000+00:00	[{'file_type': 'tsv', 'download_type': 'Taxono...	ERP112583	Industrial wastewater	root:Engineered:Wastewater:Industrial wastewater

exploring the MGazine of datasets for the studies

# getting the magazine of datasets
MZ = wwtp_studies.datasets
# filtering to taxonomic of interest
filtered_MZ = MZ['v4_1']['Taxonomic assignments SSU']
# with helpers 
taxo_mz = filtered_MZ.taxonomic 

TaxaMGazine containing:
- MGnify pipeline versions: ['v4_1']
- Number of downloads: 114
- Short descriptions: ['Taxonomic assignments SSU']
- Nonempty metadata sets: .mgnify_studies
-----------------------
Next steps: Use `.load()` to initialize.

# lazy loading the datasets
taxo_mz.load()

optionally we use MGnetizers to collect additional information from MGnify given accessions

# collecting run/assembly metadata for given accessions
run_accs = [x for x in taxo_mz.runs_accessions if not x.startswith('ERZ')]
assembly_accs = [x for x in taxo_mz.runs_accessions if x.startswith('ERZ')]
# init mgnetizer to collect 
mnet_run = MG.mgnetizer(resource='run', all_ids=run_accs)
mnet_assembly = MG.mgnetizer(resource='assembly', all_ids=assembly_accs)
# now executing the requests to the detail endpoints
async with MG: 
    await mnet_run.aenrich(limit=None)
    await mnet_assembly.aenrich(limit=None)

# passing the additional metadata to the Mgazine of taxonomic datasets
taxo_mz.mgnify_runs = mnet_run.metadata.to_list() + mnet_assembly.metadata.to_list()

optionally can also use BioSampler helper to collect additional sample metadata from BioSamples

# # the sample accessions to use
# sample_ids = taxo_mz.mgnify_runs.to_pandas()['sample_accession'].unique()
# # init biosampler to collect
# bios = MG.biosampler(sample_ids)
# # actually executing the requests
# async with MG: 
#     await bios.aenrich(limit=None)
# # passing the matadata back to MGazine
# taxo_mz.biosamples_metadata = bios.metadata.to_list(drop_duplicates=True)
# # taking a look 
# print(taxo_mz)

from the TaxaMGazine we can get an annotated dataframe with the observation metadata and taxonomic metadata (i.e., taxonomic ranks)

# convert to annotated dataframe
an_df = taxo_mz.to_anndata()
# demo adding a layer with filled in zeros 
an_df.layers['filled_zeros'] = an_df.to_df().fillna(0)
# exporting to h5ad file 
an_df.obs = an_df.obs.astype(str) #workaround for h5ad export issue with mixed types in obs
an_df.write_h5ad('wwtp_biome.h5ad')

as an example here we demo how to use the dataset in a new script:

import anndata as ad 
# read in data
back = ad.read_h5ad('wwtp_biome.h5ad')
# check it out
back

AnnData object with n_obs × n_vars = 1411 × 11604
    obs: 'experiment_type', 'instrument_model', 'instrument_platform', 'sample_accession', 'study_accession', 'updated_at', 'run_accession', 'reads_study_accession', 'assembly_study_accession', 'assembler_name', 'assembler_version', 'status', 'sample__accession', 'sample__ena_accessions', 'sample__sample_title', 'sample__biome', 'sample__updated_at', 'study__accession', 'study__ena_accessions', 'study__title', 'study__updated_at', 'study__biome.biome_name', 'study__biome.lineage'
    var: 'Superkingdom', 'Kingdom', 'Phylum', 'Class', 'Order', 'Family', 'Genus', 'Species'
    layers: 'filled_zeros'