Collecting more metadata#
In MGni.py there are helpers for collecting metdata from MGnify or BioSamples for a list of MGnify accessions.
On this page we will learn how to:
Collect metadata from MGnify using
mgnipy.collect.MGnetizerCollect metadata from BioSamples using
mgnipy.collect.BioSampler
This is especially useful if you already know the list of MGnify items that you would like the detailed metadata for such as a list of study accessions. Additionally, when you already have a MGnify dataset of samples and would like to get more metadata starting from the Run accessions which we will demonstrate below.
# uncomment below if colab
#!pip install mgnipy
We’ll pick up from the previous page where we had downloaded the “ERP014435_GO-slim_abundances_v3.0.tsv” dataset from MGnify.
import pandas as pd
# read in GO-slim abundances file
df_go = pd.read_csv("downloads/ERP014435_GO-slim_abundances_v3.0.tsv", sep="\t")
# get run accessions as a list
run_ids = df_go.columns[3:].to_list()
# sanity check
df_go.head()
The MGnetizer#
The run accessions/ids can be passed to a MGnetizer to collect their detailed metadata. MGnetizer’s are a lot like MGnifiers:
they can be accessed as attributes from MGnipy client, inheriting the configuration
they build the set of queries lazily which you can explore via
.explain()before executing them
from mgnipy import MGnipy
# init client
MG = MGnipy(cache_dir=None)
# init mgnetizer
mnet = MG.mgnetizer(resource="run", all_ids=run_ids)
# check out query set
mnet.explain()
now actually executing the above with .enrich() or .aenrich()
with mnet:
mnet.enrich()
again we can access the metadata via .metadata
# as df
run_md = mnet.metadata.to_pandas(expand_nested_dicts=False)
# check it out
run_md.head()
The BioSampler#
The above sample accessions can be passed to a BioSampler to collect even more metadata from the BioSamples database.
BioSamplers can also be accessed from the MGnipy instance, inheriting config.
bios = MG.biosampler(sample_ids=run_md["sample_accession"].to_list())
print(bios)
BioSampler with 14 sample_ids.
Progress: 0 ids.
Cache directory: None
Note: if wanting to pass runs accessions above instead (e.g., MG.biosampler(sample_ids=run_ids)) then .enrich(incl_ena=True)
now that we have built the queries we can execute them
with bios:
bios.enrich()
bios.metadata.to_pandas(expand_nested_dicts=False).head()
From here of course you can take over to merge the sets of MGnify metadata and Biosamples metadata.
However mgnipy has a helper class that combines a MGnify dataset with its metadata:
MTG MGic (the) Gatherer#
The MGic gatherer (MTG) takes a dataset as pandas or polars dataframe and MGnify or BioSamples metadata and combines them into a single object.
MTG can be used to enrich the dataset with metadata, and to convert the dataset into different formats such as pandas, polars, or anndata.
MTG = MG.mtg(
dataset=df_go,
var_cols=["description", "category"],
var_index="GO",
obs_index="name_of_your_chosing",
# mgnify_runs=mnet.metadata.to_list() #can pass here or assign the sets later
)
# can assign the sets at any time after init
MTG.mgnify_runs = mnet.metadata.to_list()
MTG.biosamples_metadata = bios.metadata.to_list()
# info
print(MTG)
MTG containing:
- Dataset type: <class 'pandas.core.frame.DataFrame'>
- var_cols: ['description', 'category']
- var_index: 'GO'
- obs_index: 'name_of_your_chosing'
- Nonempty metadata sets: .mgnify_runs, .biosamples_metadata
Example 1. to polars#
# the original but as a polars df
# MTG.to_polars()
# the feature matrix
# MTG.X(df_engine="polars") # default is pandas
# the features metadata
# MTG.var_metadata(df_engine="polars")
# the obs (samples) metadata
MTG.obs_metadata(df_engine="polars")
Example 2. to anndata#
as an annotated dataframe which keeps data matrices aligned with the corresponding metadata – even when transforming the data so that there are added matrix layers.
# to anndata object
an_df = MTG.to_anndata()
# the feature matrix
# an_df.to_df() # or an_df.X
# the features metadata
# an_df.var
# the obs (samples) metadata
an_df.obs
# exporting to h5ad file
fname = "example_collectors.h5ad"
an_df.obs = an_df.obs.astype(
str
) # workaround for h5ad export issue with mixed types in obs
an_df.write_h5ad(fname)
import anndata as ad
# read in data
back = ad.read_h5ad(fname)
# check it out
back
AnnData object with n_obs × n_vars = 14 × 116
obs: 'experiment_type', 'instrument_model', 'instrument_platform', 'sample_accession', 'study_accession', 'sample__accession', 'sample__ena_accessions', 'sample__sample_title', 'sample__biome', 'sample__updated_at', 'study__accession', 'study__ena_accessions', 'study__title', 'study__updated_at', 'study__biome.biome_name', 'study__biome.lineage', 'GivenID', 'RunID', 'SRA accession', 'name', 'taxid', 'ENA-CHECKLIST', 'ENA-FIRST-PUBLIC', 'ENA-LAST-UPDATE', 'External Id', 'INSDC center name', 'INSDC first public', 'INSDC last update', 'INSDC status', 'Submitter Id', 'collection date', 'depth', 'environment (biome)', 'environment (feature)', 'environment (material)', 'geographic location (country and/or sea)', 'geographic location (elevation)', 'geographic location (latitude)', 'geographic location (longitude)', 'investigation type', 'organism', 'project name', 'scientific_name', 'sequencing method', 'soil environmental package', 'title', 'description'
var: 'description', 'category'
Wrap Up:#
We started with only a MGnify dataset that included a list of run accessions.
This page was a quick start demonstration of:
✅ Using
MGnetizers to collect metadata from MGnify✅ Collecting even more metadata from BioSamples with
BioSampler✅ Merging the dataset with the rich metadata using
MTG