Getting all Wastewater study datasets#
Here we demonstrate how mgnipy can be used to build a cross-study taxonomic dataset for a given biome with rich sample metadata from MGnify and BioSamples in a few lines of code.
# uncomment if colab
# !pip install mgnipy
# !pip install plotly-express
# !pip install anndata
MGnipy: Init session#
First starting the session with a MGnipy client
from mgnipy import MGnipy
# configure session
MG = MGnipy(cache_dir='wwtp')
# selecting the studies resource
studies_resource = MG.studies
# helper to see accepted search params for endpoint
studies_resource.describe_endpoint()
List all studies analysed by MGnify
MGnify studies inherit directly from studies (or projects) in ENA.
Supported parameters:
- order: ListMgnifyStudiesOrderType0 | None | Unset
- biome_lineage: None | str | Unset The lineage to match, including all descendant biomes
- has_analyses_from_pipeline: None | PipelineVersions | Unset If set, will only show studies with analyses from the specified MGnify pipeline version
- search: None | str | Unset Search within study titles and accessions
- page: int | Unset Default: 1.
- page_size: int | None | Unset
MGnifier: Build and execute queries#
now using mgnifier to build and then execute the query set
# preparing query set
wwtp_studies = studies_resource(biome_lineage='root:Engineered:Wastewater')
# helper to preview the query set prior to fetch
wwtp_studies.explain()
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=1
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=2
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=3
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=4
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=5
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=6
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=7
https://www.ebi.ac.uk/metagenomics/api/v2/studies?biome_lineage=root%3AEngineered%3AWastewater&page=8
# now actually executing queries
async with MG:
# get all 8 pages of records from the list endpoint
await wwtp_studies.aget_all()
# enrich the records with study metadata from detail endpoint
await wwtp_studies.aenrich_details()
# taking a look at metdata so far
wwtp_studies.metadata.to_pandas(expand_nested_dicts=True).head()
MGazine: Filtering and loading study datasets#
exploring the MGazine of datasets for the studies
# getting the magazine of datasets
MZ = wwtp_studies.datasets
# filtering to taxonomic of interest
filtered_MZ = MZ['Taxonomic assignments SSU']
# keeping latest pipeline version if multiple output files
dedupe_downloads: list[dict] = (
filtered_MZ.downloads_df()
.sort_values(by='pipeline_version', ascending=False)
.drop_duplicates(subset='accession', keep='first')
).to_dict(orient='records')
# assign back to the filtered_MZ object
filtered_MZ.downloads = dedupe_downloads
# with taxonomic helpers
taxo_mz = filtered_MZ.taxonomic
TaxaMGazine containing:
- MGnify pipeline versions: ['v4', 'v4_1', 'v5']
- Number of downloads: 123
- Short descriptions: ['Taxonomic assignments SSU']
- Nonempty metadata sets: .mgnify_studies
-----------------------
Next steps: Use `.load()` to initialize.
# lazy loading the datasets
taxo_mz.load()
MGnetizer: Collecting metadata from MGnify#
optionally we use MGnetizers to collect additional information from MGnify given accessions
# collecting run/assembly metadata for given accessions
run_accs = [x for x in taxo_mz.runs_accessions if not x.startswith('ERZ')]
assembly_accs = [x for x in taxo_mz.runs_accessions if x.startswith('ERZ')]
# init mgnetizer to collect
mnet_run = MG.mgnetizer(resource='run', all_ids=run_accs)
mnet_assembly = MG.mgnetizer(resource='assembly', all_ids=assembly_accs)
# now executing the requests to the detail endpoints
async with MG:
await mnet_run.aenrich(limit=None)
await mnet_assembly.aenrich(limit=None)
studies_as_dict = {x['accession']: x for x in wwtp_studies.search_results.to_list()}
# passing the additional metadata to the Mgazine of taxonomic datasets
taxo_mz.mgnify_runs = mnet_run.metadata.to_list() + (
# some cleaning of assembly metadata to match the run metadata format
mnet_assembly.metadata.to_pandas()
.dropna(how='all', axis=1)
.rename(columns={'assembly_study_accession': 'study_accession'})
.assign(study=lambda df: df['study_accession'].map(studies_as_dict))
).to_dict(orient='records')
# taking a look
taxo_mz.mgnify_runs.to_pandas(expand_nested_dicts=True).head()
| experiment_type | instrument_model | instrument_platform | accession | sample_accession | study_accession | updated_at | run_accession | status | sample__accession | sample__ena_accessions | sample__sample_title | sample__biome | sample__updated_at | study__accession | study__ena_accessions | study__title | study__updated_at | study__biome.biome_name | study__biome.lineage | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Amplicon | 454 GS FLX | LS454 | DRR046685 | SAMD00041314 | MGYS00005741 | NaN | NaN | NaN | SAMD00041314 | [SAMD00041314, DRS050256] | WL10_018 | NaN | 2026-04-30T16:33:28.429000+00:00 | MGYS00005741 | [DRP003823, PRJDB4240] | WL Reactor published in ME | 2026-05-28T15:47:00.074000+00:00 | Activated Sludge | root:Engineered:Wastewater:Activated Sludge |
| 1 | Amplicon | 454 GS FLX | LS454 | DRR046686 | SAMD00041315 | MGYS00005741 | NaN | NaN | NaN | SAMD00041315 | [SAMD00041315, DRS050257] | WL10_021 | NaN | 2026-04-30T16:33:27.744000+00:00 | MGYS00005741 | [DRP003823, PRJDB4240] | WL Reactor published in ME | 2026-05-28T15:47:00.074000+00:00 | Activated Sludge | root:Engineered:Wastewater:Activated Sludge |
| 2 | Amplicon | 454 GS FLX | LS454 | DRR046690 | SAMD00041319 | MGYS00005741 | NaN | NaN | NaN | SAMD00041319 | [SAMD00041319, DRS050261] | WL10_030 | NaN | 2026-04-30T16:33:27.052000+00:00 | MGYS00005741 | [DRP003823, PRJDB4240] | WL Reactor published in ME | 2026-05-28T15:47:00.074000+00:00 | Activated Sludge | root:Engineered:Wastewater:Activated Sludge |
| 3 | Amplicon | 454 GS FLX | LS454 | DRR046689 | SAMD00041318 | MGYS00005741 | NaN | NaN | NaN | SAMD00041318 | [SAMD00041318, DRS050260] | WL10_028 | NaN | 2026-04-30T16:32:55.063000+00:00 | MGYS00005741 | [DRP003823, PRJDB4240] | WL Reactor published in ME | 2026-05-28T15:47:00.074000+00:00 | Activated Sludge | root:Engineered:Wastewater:Activated Sludge |
| 4 | Amplicon | 454 GS FLX | LS454 | DRR046693 | SAMD00041322 | MGYS00005741 | NaN | NaN | NaN | SAMD00041322 | [SAMD00041322, DRS050264] | WL10_037 | NaN | 2026-04-30T16:33:25.641000+00:00 | MGYS00005741 | [DRP003823, PRJDB4240] | WL Reactor published in ME | 2026-05-28T15:47:00.074000+00:00 | Activated Sludge | root:Engineered:Wastewater:Activated Sludge |
BioSampler: Collecting metadata from BioSamples#
optionally can also use BioSampler helper to collect additional sample metadata from BioSamples
MGazine.to_anndata(): Saving taxa count matrix with metadata#
from the TaxaMGazine we can get an annotated dataframe with the observation metadata and taxonomic metadata (i.e., taxonomic ranks)
# convert to annotated dataframe
an_df = taxo_mz.to_anndata()
# demo adding a layer with filled in zeros
an_df.layers['filled_zeros'] = an_df.to_df().fillna(0)
# exporting to h5ad file
an_df.obs = an_df.obs.astype(str) #workaround for h5ad export issue with mixed types in obs
an_df.write_h5ad('wwtp_biome.h5ad')
In the above we curated a wastewater biome dataset using MGnify API v2.
Bonus: Loading in the AnnData dataframe#
as an example here we demo how to use the dataset in a new script:
import anndata as ad
# read in data
back: ad.AnnData = ad.read_h5ad('wwtp_biome.h5ad')
print(back)
AnnData object with n_obs × n_vars = 1875 × 12120
obs: 'experiment_type', 'instrument_model', 'instrument_platform', 'sample_accession', 'study_accession', 'updated_at', 'run_accession', 'status', 'sample__accession', 'sample__ena_accessions', 'sample__sample_title', 'sample__biome', 'sample__updated_at', 'study__accession', 'study__ena_accessions', 'study__title', 'study__updated_at', 'study__biome.biome_name', 'study__biome.lineage'
var: 'Superkingdom', 'Kingdom', 'Phylum', 'Class', 'Order', 'Family', 'Genus', 'Species'
layers: 'filled_zeros'
How to filter annotated df by obs (sample) metadata#
# filtering out even more samples with no metadata
an_df_filt: ad.AnnData = back[back.obs['study_accession'] != 'None']
# now how many studies?
print("Number of studies with sample metadata: ", an_df_filt.obs['study_accession'].nunique())
Number of studies with sample metadata: 116
How to filter annotated df by var (taxa) metadata#
# pruning var to species level
to_species: ad.AnnData = an_df_filt[:, an_df_filt.var['Species']!='NA']
# filtering out samples with no species level
has_species_level_info: ad.AnnData = to_species[~to_species.to_df().isna().all(axis=1)]
# now how many studies?
print("Number of studies with species level information: ", has_species_level_info.obs['study_accession'].nunique())
Number of studies with species level information: 109
Adding more obs metadata columns#
Number of samples after deduplication: 1654
Number of studies after deduplication: 109