Get a MGnify List

Here we demonstrate the basic usability of MGni.py to see what records/items (e.g., biomes) are available in a given resource (e.g., the Biomes endpoint of the MGnify API v2 )

After clicking the “Activate Notebook” button you can run the cells in this browser. Alternatively, you can also click on the 🚀 to launch in colab or binder.

Activate Notebook

---

🎯 The Goal: Get a list of MGnify Biomes

The GOLD ecosystem classifications organize environmental samples into a hierarchical taxonomy of biome types—from broad categories like “Engineered” to specific environments like “Plant rhizosphere.”

This demo will show you how to:

1. Prepare queries — Learn different ways to initialize and configure your API requests using MGnipy or direct proxies

2. Preview before fetching — Use filtering and preview methods (preview, dry_run, explain) to confirm your query before retrieving results

3. Fetch results — Execute requests using iterative get(), specific page(), or get_all() methods (sync or async)

4. Monitor progress — Track your requests and check completion status

By the end, we hope you’ll be comfortable querying the MGnify resource – or specifically the biomes resource at least

# uncomment below if colab
#!pip install mgnipy

We can initiate using mgnipy.MGnipy or proxies.Biomes

🖍️ The start: Preparing queries

Option 1. mgnipy.MGnipy

The MGnipy client offers a unified interface to access various MGnify API endpoints, including biomes. This approach is convenient if you want to manage multiple types of queries or resources through a single client object.

• Instantiate MGnipy to configure your API access and manage requests.

• Use .biomes to create a biome query with your desired parameters.

• Use list_parameters() to see all available filters and options.

• The filter() method allows you to refine your query further.

• The explain() method previews the constructed API URLs and the first few results.

This method has an additional helper function to list and describe available resources

💡 Tip: See Configuration page for more setup details 🛠.

from mgnipy import MGnipy

# init
MG = MGnipy(
    # configuration
    cache_dir=None,  # set to None to disable caching, or specify a directory for caching
)

# access proxy
biomes = MG.biomes

# checking it out
print(biomes)

<class 'mgnipy.V2.proxies.biomes.Biomes'> for 'biomes' resource
- Endpoint: 'mgnipy.emgapi_v2_client.api.miscellaneous.list_mgnify_biomes'
- Params: {}
- Child resource: 'biome'

In the print we can see that we have not initiated any query parameters.

If you would like to know what params are supported for the endpoint there is a helper method you can use: .list_supported_params()

# if not sure what kwargs suupported
print("Supported kwargs for biomes: ", biomes.list_supported_params())

Supported kwargs for biomes:  ['biome_lineage', 'max_depth', 'page', 'page_size']

also like describe_resources() there is a describe_endpoint() with even more info about the endpoint based on the openapi.json spec

biomes.describe_endpoint()

List all biomes

List all biomes in the MGnify database.

Supported parameters:
- biome_lineage: None | str | Unset The lineage to match, including all descendant biomes
- max_depth: int | None | Unset Maximum depth of the biome lineage to include, e.g. `root` is 1 and `root:Host-Associated:Human` is level 3
- page: int | Unset Default: 1.
- page_size: int | None | Unset

Let’s add some search params via .filter()

biomes = biomes.filter(
    page_size=5,
    max_depth=6,
)

# check it out again
print(biomes)

<class 'mgnipy.V2.proxies.biomes.Biomes'> for 'biomes' resource
- Endpoint: 'mgnipy.emgapi_v2_client.api.miscellaneous.list_mgnify_biomes'
- Params: {'page_size': 5, 'max_depth': 6}
- Child resource: 'biome'

Great we can see that the query string (i.e., after ?s) has been updated with our given parameters

👓 Previewing your requests

There is an optional but recommended step to

• .preview() the first page of results as a pandas.DataFrame, or

• .dry_run() to print the number of pages and records to request

• .explain() to print the planned request urls

before .get()ting all the result pages.

# checking out first 5 request urls to be made
biomes.explain(head=5)
# or
# biomes.dry_run()
# or
biomes.preview()

https://www.ebi.ac.uk/metagenomics/api/v2/biomes?max_depth=6&page=1&page_size=5
https://www.ebi.ac.uk/metagenomics/api/v2/biomes?max_depth=6&page=2&page_size=5
https://www.ebi.ac.uk/metagenomics/api/v2/biomes?max_depth=6&page=3&page_size=5
https://www.ebi.ac.uk/metagenomics/api/v2/biomes?max_depth=6&page=4&page_size=5
https://www.ebi.ac.uk/metagenomics/api/v2/biomes?max_depth=6&page=5&page_size=5

	biome_name	biome_lineage
0	root	root
1	Control	root:Control
2	Engineered	root:Engineered
3	Biogas plant	root:Engineered:Biogas plant
4	Wet fermentation	root:Engineered:Biogas plant:Wet fermentation

📨 Carry out requests to list endpoints

If happy with the plan, proceed with the async or sync get requests.

There are multiple options:

• .get() or .aget() like next() iteratively carries out one page/request at a time per call. Returning the page dict or None when iteration is complete

• page() or .apage() pass specific page_num

• .get_all() or aget_all() fetch the pages in bulk sync or asynchronously

Option 1. .get() iteratively

For a demo of this we will make the first 5 requests.

# getting first 5
with MG: # or biomes 
    for _ in range(5):
        biomes.get()

For each option there is an async option

async with MG: 
    for _ in range(5):
        await biomes.aget()

and you can take a look at the results as you go 😀 :

# by page, e.g. page 5
biomes.search_results.results[5]

Show code cell output

Hide code cell output

[{'biome_name': 'Activated sludge',
  'lineage': 'root:Engineered:Bioremediation:Terephthalate:Wastewater:Activated sludge'},
 {'biome_name': 'Bioreactor',
  'lineage': 'root:Engineered:Bioremediation:Terephthalate:Wastewater:Bioreactor'},
 {'biome_name': 'Tetrachloroethylene and derivatives',
  'lineage': 'root:Engineered:Bioremediation:Tetrachloroethylene and derivatives'},
 {'biome_name': 'Chloroethene',
  'lineage': 'root:Engineered:Bioremediation:Tetrachloroethylene and derivatives:Chloroethene'},
 {'biome_name': 'Bioreactor',
  'lineage': 'root:Engineered:Bioremediation:Tetrachloroethylene and derivatives:Chloroethene:Bioreactor'}]

# or by records, first 2 records
biomes.search_results.to_list()[:2]
# or via .records iterator
#list(biomes.search_results.records)[:2]

[{'biome_name': 'root', 'lineage': 'root'},
 {'biome_name': 'Control', 'lineage': 'root:Control'}]

Specific to the biomes, results can also be visualized as a tree “print” “hshow” or “vshow”

biomes.show_tree()

Show code cell output

Hide code cell output

root
├── Control
└── Engineered
    ├── Biogas plant
    │   └── Wet fermentation
    ├── Bioreactor
    │   └── Continuous culture
    │       ├── Marine intertidal flat sediment inoculum
    │       │   └── Wadden Sea-Germany
    │       └── Marine sediment inoculum
    │           └── Wadden Sea-Germany
    ├── Bioremediation
    │   ├── Hydrocarbon
    │   │   └── Benzene
    │   │       └── Bioreactor
    │   ├── Metal
    │   ├── Persistent organic pollutants (POP)
    │   ├── Polycyclic aromatic hydrocarbons
    │   ├── Terephthalate
    │   │   └── Wastewater
    │   │       ├── Activated sludge
    │   │       └── Bioreactor
    │   └── Tetrachloroethylene and derivatives
    │       ├── Chloroethene
    │       │   └── Bioreactor
    │       └── Tetrachloroethylene
    │           └── Bioreactor
    ├── Biotransformation
    │   ├── Microbial enhanced oil recovery
    │   ├── Microbial solubilization of coal
    │   └── Mixed alcohol bioreactor
    ├── Built environment
    ├── Food production
    │   ├── Dairy products
    │   ├── Fermented beverages
    │   ├── Fermented seafood
    │   ├── Fermented vegetables
    │   └── Silage fermentation
    ├── Industrial production
    │   └── Engineered product
    ├── Lab enrichment
    │   ├── Defined media
    │   │   ├── Aerobic media
    │   │   ├── Anaerobic media
    │   │   └── Marine media
    │   │       └── Algoconsortia
    │   └── Undefined media
    ├── Lab Synthesis
    │   └── Genetic cross
    └── Modeled

Option 2. get a specific page()

• Will make the request and also returns the items/records in a list like above.

• When calling page() on an alrady completed request, the api call is not repeated and instead the output is a page from the cache

with MG: 
    biomes.page(3)

Option 3. get_all() of all requests (with safety limits)

can handle multiple requests via

• specifying a list of pages to .get_all(pages=<list_of_pages>)

• or by not specifying pages you can continually call on the method which will let the bulk fetch handle the batching whilst considering limit=<num_items

Especially before fetching in bulk we should take a look at the total number of requests/pages.

# let's first checkout num requests
print("Number of requests:", biomes.num_requests)
# or better yet do a dry_run
biomes.dry_run()

Number of requests: 99
Planning the API call with params:
{'page_size': 5, 'max_depth': 6}
Total requests to make: 99
Total records to retrieve: 492

Now we can get some data sync or async:

# synchronously fetch up to 50 pages
with MG: 
    biomes.get_all(limit=50)

# and async
async with MG: 
    await biomes.aget_all(limit=50)

⏳ Checking progress

As we saw earlier in the notebook we can take a look at results as we go along. For a concise update on progress you can use .progress

biomes.progress

Retrieved pages: 100%|████████████████████| 99/99

# no cache for this isntance but we can clear anywahys
biomes.clear_cache()

# also check clients closed
MG.status()

Client or AuthenticatedClient type: Client
HTTP client open: False
Async client open: False