Running MLflow Projects on Azure Databricks | by Invisible Guru Jii | Mar, 2025

MLflow is an open-source platform designed to streamline the machine studying (ML) lifecycle. Amongst its core functionalities, MLflow Tasks enable customers to package deal ML code in a reproducible format, making it straightforward to share and deploy fashions throughout varied environments.

Azure Databricks, a managed Apache Spark platform, gives scalable computing energy to run MLflow Tasks effectively. This information explores learn how to execute MLflow Tasks on Azure Databricks, highlighting new options launched in MLflow 2.14.

An MLflow Challenge is a structured format that packages machine studying code, guaranteeing reproducibility and ease of execution. It usually consists of:

• MLproject File: A YAML file defining entry factors and dependencies.
• Setting Specification: Both conda.yaml, python_env.yaml, or Docker configuration.
• Entry Factors: Scripts (.py or .sh) executed with specified parameters.

title: MyProject
entry_points:
most important:
parameters:
data_file: path
regularization: {sort: float, default: 0.1}
command: "python practice.py --data_file {data_file} --regularization {regularization}"

MLflow integrates seamlessly with Azure Databricks, enabling scalable execution of ML workflows. With Databricks Runtime 13.0 ML and later, MLflow Tasks now run completely as Databricks Spark Jobs, utilizing a brand new structured format.

This format is required when working MLflow Tasks on Databricks job clusters.

Key Issues:

• The MLproject file should outline both:
• databricks_spark_job.python_file
• entry_points
• If each or neither are outlined, MLflow raises an error.

title: MyDatabricksProject
databricks_spark_job:
python_file: "practice.py"
parameters: ["--param1", "value1"]
python_libraries:
- mlflow==2.14
- scikit-learn

title: MyDatabricksSparkProject
databricks_spark_job:
python_libraries:
- mlflow==2.14
- numpy
entry_points:
most important:
parameters:
script_name: {sort: string, default: practice.py}
model_name: {sort: string, default: my_model}
command: "python {script_name} {model_name}"

To run the mission, execute:

mlflow run . -b databricks --backend-config cluster-spec.json 
-P script_name=practice.py -P model_name=model123 
--experiment-id 

To execute an MLflow Challenge on Azure Databricks, use the command:

mlflow run  -b databricks --backend-config 

Instance Cluster Specification:

{
"spark_version": "13.0.x-scala2.12",
"num_workers": 2,
"node_type_id": "Standard_DS3_v2"
}

To put in dependencies on employee nodes:

{
"new_cluster": {
"spark_version": "13.0.x-scala2.12",
"num_workers": 2,
"node_type_id": "Standard_DS3_v2"
},
"libraries": [
{ "pypi": { "package": "tensorflow" } },
{ "pypi": { "package": "mlflow" } }
]
}

To allow SparkR, set up and import it inside your MLflow Challenge:

if (file.exists("/databricks/spark/R/pkg")) {
set up.packages("/databricks/spark/R/pkg", repos = NULL)
} else {
set up.packages("SparkR")
}
library(SparkR)
sparkR.session()

This instance reveals learn how to create an experiment, run the MLflow tutorial mission on an Azure Databricks cluster, view the job run output, and look at the run within the experiment.

1. Set up MLflow utilizing pip set up mlflow.
2. Set up and configure the Databricks CLI. The Databricks CLI authentication mechanism is required to run jobs on an Azure Databricks cluster.

1. Within the workspace, choose Create > MLflow Experiment.
2. Within the Identify area, enter Tutorial.
3. Click on Create. Notice the Experiment ID. On this instance, it’s 14622565.

The next steps arrange the MLFLOW_TRACKING_URI atmosphere variable and run the mission, recording the coaching parameters, metrics, and the skilled mannequin to the experiment famous within the previous step:

1. Set the MLFLOW_TRACKING_URI atmosphere variable to the Azure Databricks workspace.

export MLFLOW_TRACKING_URI=databricks

2. Run the MLflow tutorial mission, coaching a wine model. Substitute with the Experiment ID you famous within the previous step.

mlflow run https://github.com/mlflow/mlflow#examples/sklearn_elasticnet_wine -b databricks --backend-config cluster-spec.json --experiment-id 

=== Fetching mission from https://github.com/mlflow/mlflow#examples/sklearn_elasticnet_wine into /var/folders/kc/l20y4txd5w3_xrdhw6cnz1080000gp/T/tmpbct_5g8u ===
=== Importing mission to DBFS path /dbfs/mlflow-experiments//projects-code/16e66ccbff0a4e22278e4d73ec733e2c9a33efbd1e6f70e3c7b47b8b5f1e4fa3.tar.gz ===
=== Completed importing mission to /dbfs/mlflow-experiments//projects-code/16e66ccbff0a4e22278e4d73ec733e2c9a33efbd1e6f70e3c7b47b8b5f1e4fa3.tar.gz ===
=== Working entry level most important of mission https://github.com/mlflow/mlflow#examples/sklearn_elasticnet_wine on Databricks ===
=== Launched MLflow run as Databricks job run with ID 8651121. Getting run standing web page URL... ===
=== Verify the run's standing at https://#job//run/1 ===

3. Copy the URL https://#job//run/1 within the final line of the MLflow run output.

1. Open the URL you copied within the previous step in a browser to view the Azure Databricks job run output:

1. Navigate to the experiment in your Azure Databricks workspace.

2. Click on the experiment.

3. To show run particulars, click on a hyperlink within the Date column.

You possibly can view logs out of your run by clicking the Logs hyperlink within the Job Output area.

For some instance MLflow initiatives, see the MLflow App Library, which comprises a repository of ready-to-run initiatives aimed toward making it straightforward to incorporate ML performance into your code.