Don’t Let Conda Eat Your Hard Drive

In the event you’re an Anaconda person, that conda environments make it easier to handle package deal dependencies, keep away from compatibility conflicts, and share your initiatives with others. Sadly, they will additionally take over your pc’s exhausting drive.

I write plenty of pc tutorials and to maintain them organized, every has a devoted folder construction full with a Conda Environment. This labored nice at first, however quickly my pc’s efficiency degraded, and I observed that my SSD was filling up. At one level I had solely 13 GB free.

Conda helps handle this drawback by storing downloaded package deal recordsdata in a single “cache” (pkgs_dirs). If you set up a package deal, conda checks for it within the package deal cache earlier than downloading. If not discovered, conda will obtain and extract the package deal and hyperlink the recordsdata to the energetic surroundings. As a result of the cache is “shared,” completely different environments can use the identical downloaded recordsdata with out duplication.

As a result of conda caches each downloaded package deal, pkgs_dirs can develop to many gigabytes. And whereas conda hyperlinks to shared packages within the cache, there’s nonetheless a have to retailer some packages within the surroundings folder. That is primarily to keep away from model conflicts, the place completely different environments want completely different variations of the identical dependency (a package deal required to run one other package deal).

As well as, massive, compiled binaries like OpenCV could require full copies within the surroundings’s listing, and every surroundings requires a duplicate of the Python interpreter (at 100–200 MB). All these points can bloat conda environments to a number of gigabytes.

On this Fast Success Knowledge Science undertaking, we’ll take a look at some methods for decreasing the storage necessities for conda environments, together with these saved in default places and devoted folders.

---

Reminiscence Administration Strategies

Under are some Memory Management methods that may make it easier to cut back conda’s storage footprint in your machine. We’ll focus on every in flip.

1. Cache cleansing
2. Sharing task-based environments
3. Archiving with surroundings and specs recordsdata
4. Archiving environments with conda-pack
5. Storing environments on an exterior drive
6. Relocating the package deal cache
7. Utilizing digital environments (venv)

1. Cleansing the Bundle Cache

Cleansing the package deal cache is the primary and best step for releasing up reminiscence. Even after deleting environments, conda retains the associated package deal recordsdata within the cache. You’ll be able to unlock house by eradicating these unused packages and their related tarballs (compressed package deal recordsdata), logs, index caches (metadata saved in conda), and momentary recordsdata.

Conda permits an elective “dry run” to see how a lot reminiscence can be reclaimed. You’ll need to run this from both the terminal or Anaconda Immediate in your base surroundings:

conda clear --all --dry-run

To commit, run:

conda clear --all

Right here’s how this seems to be on my machine:

This course of trimmed a wholesome 6.28 GB and took a number of minutes to run.

2. Sharing Job-based Environments

Creating just a few environments for specialised duties — like pc imaginative and prescient or geospatial work — is extra reminiscence environment friendly than utilizing devoted environments for every undertaking. These environments would come with primary packages plus ones for the precise activity (corresponding to OpenCV, scikit-image, and PIL for pc imaginative and prescient).

A bonus of this method is which you can simply preserve all of the packages updated and hyperlink the environments to a number of initiatives. Nonetheless, this received’t work if some initiatives require completely different variations of the shared packages.

3. Archiving with Setting and Specs Information

In the event you don’t have sufficient storage websites or need to protect legacy initiatives effectively, think about using surroundings or specs recordsdata. These small recordsdata file an surroundings’s contents, permitting you to rebuild it later.

Saving conda environments on this method reduces their measurement on disk from gigabytes to a couple kilobytes. After all, you’ll must recreate the surroundings to make use of it. So, you’ll need to keep away from this system should you incessantly revisit initiatives that hyperlink to the archived environments.

NOTE: Think about using Mamba, a drop-in alternative for conda, for quicker rebuilds. Because the docs say, “If conda, Mamba!”

Utilizing Setting Information: An environmental file is a small file that lists all of the packages and variations put in in an surroundings, together with these put in utilizing Python’s package deal installer (pip). This helps you each restore an surroundings and share it with others.

The surroundings file is written in YAML (.yml), a human-readable data-serialization format for knowledge storage. To generate an surroundings file, it’s essential to activate after which export the surroundings. Right here’s find out how to make a file for an surroundings named my_env:

 conda activate my_env
 conda env export > my_env.yml

You’ll be able to identify the file any legitimate filename however watch out as an present file with the identical identify can be overwritten.

By default, the surroundings file is written to the person listing. Right here’s a truncated instance of the file’s contents:

identify: C:Usershannaquick_successfed_hikesfed_env
channels:
  - defaults
  - conda-forge
dependencies:
  - asttokens=2.0.5=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - blas=1.0=mkl
  - bottleneck=1.3.4=py310h9128911_0
  - brotli=1.0.9=ha925a31_2
  - bzip2=1.0.8=he774522_0
  - ca-certificates=2022.4.26=haa95532_0
  - certifi=2022.5.18.1=py310haa95532_0
  - colorama=0.4.4=pyhd3eb1b0_0
  - cycler=0.11.0=pyhd3eb1b0_0
  - debugpy=1.5.1=py310hd77b12b_0
  - decorator=5.1.1=pyhd3eb1b0_0
  - entrypoints=0.4=py310haa95532_0

  ------SNIP------

Now you can take away your conda surroundings and reproduce it once more with this file. To take away an surroundings, first deactivate it after which run the take away command (the place ENVNAME is the identify of your surroundings):

conda deactivate
conda take away -n ENVNAME --all

If the conda surroundings exists exterior of Anaconda’s default envs folder, then embody the listing path to the surroundings, as so:

conda take away -p PATHENVNAME --all

Word that this archiving approach will solely work completely should you proceed to make use of the identical working system, corresponding to Home windows or macOS. It is because fixing for dependencies can introduce packages which may not be suitable throughout platforms.

To revive a conda surroundings utilizing a file, run the next, the place my_env represents your conda surroundings identify and surroundings.yml represents your surroundings file:

 conda env create -n my_env -f directorypathtoenvironment.yml

You can even use the surroundings file to recreate the surroundings in your D: drive. Simply present the brand new path when utilizing the file. Right here’s an instance:

conda create --prefix D:my_envsmy_new_env --file surroundings.yml

For extra on surroundings recordsdata, together with find out how to manually produce them, go to the docs.

Utilizing Specs Information: In the event you haven’t put in any packages utilizing pip, you should utilize a specs file to breed a conda surroundings on the identical working system. To create a specification file, activate an surroundings, corresponding to my_env, and enter the next command:

 conda checklist --explicit > exp_spec_list.txt

This produces the next output, truncated for brevity:

 # This file could also be used to create an surroundings utilizing:
 # $ conda create --name  --file 
 # platform: win-64
 @EXPLICIT
 https://conda.anaconda.org/conda-forge/win-64/ca-certificates-202x.xx.x-h5b45459_0.tar.bz2
 https://conda.anaconda.org/conda-forge/noarch/tzdata-202xx-he74cb21_0.tar.bz2

------snip------

Word that the --explicit flag ensures that the focused platform is annotated within the file, on this case, # platform: win-64 within the third line.

Now you can take away the surroundings as described within the earlier part.

To re-create my_env utilizing this textual content file, run the next with a correct listing path:

conda create -n my_env -f directorypathtoexp_spec_list.txt

4. Archiving Environments with conda-pack

The conda-pack command allows you to archive a conda surroundings earlier than eradicating it. It packs all the surroundings right into a compressed archive with the extension: .tar.gz. It’s helpful for backing up, sharing, and transferring environments with out the necessity to reinstall packages.

The next command will protect an surroundings however take away it out of your system (the place my_env represents the identify of your surroundings):

conda set up -c conda-forge conda-pack
conda pack -n my_env -o my_env.tar.gz

To revive the surroundings later run this command:

mkdir my_env && tar -xzf my_env.tar.gz -C my_env

This system received’t save as a lot reminiscence because the textual content file possibility. Nonetheless, you received’t have to re-download packages when restoring an surroundings, which suggests it may be used with out web entry.

5. Storing Environments on an Exterior Drive

By default, conda shops all environments in a default location. For Home windows, that is below the …anaconda3envs folder. You’ll be able to see these environments by working the command conda data --envs in a immediate window or terminal. Right here’s the way it seems to be on my C: drive (it is a truncated view):

Utilizing a Single Environments Folder: In case your system helps an exterior or secondary drive, you possibly can configure conda to retailer environments there to unlock house in your main disk. Right here’s the command; you’ll have to substitute your particular path:

conda config --set envs_dirs /path/to/exterior/drive

In the event you enter a path to your D drive, corresponding to D:conda_envs, conda will create new environments at this location.

This system works effectively when your exterior drive is a quick SSD and once you’re storing packages with massive dependencies, like TensorFlow. The draw back is slower efficiency. In case your OS and notebooks stay on the first drive, it’s possible you’ll expertise some learn/write latency when working Python.

As well as, some OS settings could energy down idle exterior drives, including a delay after they spin again up. Instruments like Jupyter could battle to find conda environments if the drive letter modifications, so that you’ll need to use a hard and fast drive letter and be sure that the right kernel paths are set.

Utilizing A number of Setting Folders: As a substitute of utilizing a single envs_dirs listing for all environments, you possibly can retailer every surroundings inside its respective undertaking folder. This allows you to retailer the whole lot associated to a undertaking in a single place.

For instance, suppose you might have a undertaking in your Home windows D: drive in a folder known as D:projectsgeospatial. To position the undertaking’s conda surroundings on this folder, loaded with ipykernel for JupyterLab, you’d run:

conda create -p D:projectsgeospatialenv ipykernel

After all, you possibly can name env one thing extra descriptive, like geospatial_env.

As with the earlier instance, environments saved on a distinct disk could cause efficiency points.

Particular Word on JupyterLab: Relying on the way you launch JupyterLab, its default habits could also be to open in your person listing (corresponding to, C:Usersyour_user_name). Since its file browser is restricted to the listing from which it’s launched, you received’t see directories on different drives like D:. There are a lot of methods to deal with this, however one of many easiest is to launch JupyterLab from the D: drive.

For instance, in Anaconda Immediate, kind:

D:

adopted by:

jupyter lab

Now, it is possible for you to to choose from kernels on the D: drive.

For extra choices on altering JupyterLab’s working listing, ask an AI about “find out how to change Jupyter’s default working listing” or “find out how to create a Symlink to D: in your person folder.”

Transferring Current Environments: It’s best to by no means manually transfer a conda surroundings, corresponding to by chopping and pasting to a brand new location. It is because conda depends on inner paths and metadata that may change into invalid with location modifications.

As a substitute, you must clone present environments to a different drive. It will duplicate the surroundings, so that you’ll have to manually take away it from its unique location.

Within the following instance, we use the --clone flag to supply a precise copy of a C: drive surroundings (known as my_env) on the D: drive:

conda create -p D:new_envsmy_env --clone C:pathtooldenv

NOTE: Contemplate exporting your surroundings to a YAML file (as described in Part 3 above) earlier than cloning. This lets you recreate the surroundings if one thing goes flawed with the clone process.

Now, once you run conda env checklist, you’ll see the surroundings listed in each the C: and D: drives. You’ll be able to take away the outdated surroundings by working the next command within the base surroundings:

conda take away --name my_env --all -y

Once more, latency points could have an effect on these setups should you’re working throughout two disks.

You might be questioning, is it higher to maneuver a conda surroundings utilizing an surroundings (YAML) file or to make use of--clone? The brief reply is that --clone is one of the best and quickest possibility for transferring an surroundings to a distinct drive on the identical machine. An surroundings file is finest for recreating the identical surroundings on a completely different machine. Whereas the file ensures a constant surroundings throughout completely different methods, it may take for much longer to run, particularly with massive environments.

6. Relocating the Bundle Cache

In case your main drive is low on house, you possibly can transfer the package deal cache to a bigger exterior or secondary drive utilizing this command:

conda config --set pkgs_dirs D:conda_pkgs

On this instance, packages at the moment are saved on the D drive (D:conda_pkgs) as an alternative of the default location.

In the event you’re working in your main drive and each drives are SSD, then latency points shouldn’t be important. Nonetheless, if one of many drives is a slower HDD, you possibly can expertise slowdowns when creating or updating environments. If D: is an exterior drive related by USB, you may even see important slowdowns for big environments.

You’ll be able to mitigate a few of these points by retaining the package deal cache (pkgs_dirs) and incessantly used environments on the quicker SSD, and different environments on the slower HDD.

One final thing to think about is backups. Major drives could have routine backups scheduled however secondary or exterior drives could not. This places you vulnerable to dropping all of your environments.

7. Utilizing Digital Environments

In case your undertaking doesn’t require conda’s intensive package deal administration system for dealing with heavy dependencies (like TensorFlow or GDAL), you possibly can considerably cut back disk utilization with a Python digital surroundings (venv). This represents a light-weight various to a conda surroundings.

To create a venv named my_env, run the next command:

This kind of surroundings has a small base set up. A minimal conda surroundings takes up about 200 MB and contains a number of utilities, corresponding to conda, pip, setuptools, and so forth. A venv is way lighter, with a minimal set up measurement of solely 5–10 MB.

Conda additionally caches package deal tarballs in pkgs_dirs. These tarballs can develop to a number of GBs over time. As a result of venv installs packages instantly into the surroundings, no further copies are preserved.

Basically, you’ll need to take into account venv once you solely want primary Python packages like NumPy, pandas, or Scikit-learn. Packages for which conda is strongly beneficial, like Geopandas, ought to nonetheless be positioned in a conda surroundings. In the event you use plenty of environments, you’ll in all probability need to stick to conda and profit from its package deal linking.

Yow will discover particulars on find out how to activate and use Python digital environments within the venv docs.

---

Recap

Excessive influence/low disruption reminiscence administration methods for conda environments embody cleansing the package deal cache and storing little-used environments as YAML or textual content recordsdata. These strategies can save many gigabytes of reminiscence whereas retaining Anaconda’s default listing construction.

Different excessive influence strategies embody transferring the package deal cache and/or conda environments to a secondary or exterior drive. It will resolve reminiscence issues however could introduce latency points, particularly if the brand new drive is a gradual HDD or makes use of a USB connection.

For easy environments, you should utilize a Python digital surroundings (venv) as a light-weight various to conda.