[mlflow series 3] mlflow upgrade

Time:2020-11-22

background

mlflowThe update iteration speed is still very fast. On average, a large version is updated every month. As of November 1, it has been updated to version 1.11.0
Let’s seemlflow releaseYou can see that as early as 1.10.0, it provides better feature support for model registry, as well as the ability to logically delete experiments,
However, these features are not available in mlflow 1.4.0, especially the feature of deleting experiments. If there are many experiments, the experiments we see are disorderly, which is not convenient for us to manage, so we upgrade mlflow

Upgrade and preparation

Before referenceBuilding and using mlflowFirst, we establish the CONDA environment of mlflow 1.4.0 and mlflow 1.11.0
Suppose that you have established the corresponding CONDA environment, and they are mlflow-1.4.0 and mlflow-1.11.0, execute:

conda activate mlflow-1.11.0

reference resourcesmlflow db upgrade, execute

mlflow db upgrade mysql://user:[email protected]:port/db
For example: mlflow DB upgrade mysql://root:[email protected]/mlflow

among

noun explain
user The user name of the database
passwd Password for the database
host The host address of the database
port The port of the database can be omitted if it is 3306 by default
db Database of database

If the execution is successful, you will see the following output information:

2020/11/02 10:24:50 INFO mlflow.store.db.utils: Updating database tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 2b4d017a5e9b -> cfd24bdc0731, Update run status constraint with killed
INFO  [alembic.runtime.migration] Running upgrade cfd24bdc0731 -> 0a8213491aaa, drop_duplicate_killed_constraint
WARNI [0a8213491aaa_drop_duplicate_killed_constraint_py] Failed to drop check constraint. Dropping check constraints may not be supported by your SQL database. Exception content: (MySQLdb._exceptions.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'CHECK status' at line 1")
[SQL: ALTER TABLE runs DROP CHECK status]
(Background on this error at: http://sqlalche.me/e/f405)
INFO  [alembic.runtime.migration] Running upgrade 0a8213491aaa -> 728d730b5ebd, add registered model tags table
INFO  [alembic.runtime.migration] Running upgrade 728d730b5ebd -> 27a6a02d2cf1, add model version tags table
INFO  [alembic.runtime.migration] Running upgrade 27a6a02d2cf1 -> 84291f40a231, add run_link to model_version

If you execute it again in mlflow 1.4.0:

mlflow server \
      --backend-store-uri mysql://root:[email protected]/mlflow \
      --host 0.0.0.0 -p 5002 \
      --default-artifact-root s3://mlflow

You will report an error:

2020/11/02 10:25:41 ERROR mlflow.cli: Error initializing backend store
2020/11/02 10:25:41 ERROR mlflow.cli: Detected out-of-date database schema (found version 84291f40a231, but expected 2b4d017a5e9b). Take a backup of your database, then run 'mlflow db upgrade <database_uri>' to migrate your database to the latest schema. NOTE: schema migration may result in database downtime - please consult your database's documentation for more detail.
Traceback (most recent call last):
  File "/Users/ljh/opt/miniconda3/envs/mlflow-1.4.0-dev/lib/python3.6/site-packages/mlflow/cli.py", line 263, in server
    initialize_backend_stores(backend_store_uri, default_artifact_root)
  File "/Users/ljh/opt/miniconda3/envs/mlflow-1.4.0-dev/lib/python3.6/site-packages/mlflow/server/handlers.py", line 97, in initialize_backend_stores
    _get_tracking_store(backend_store_uri, default_artifact_root)
  File "/Users/ljh/opt/miniconda3/envs/mlflow-1.4.0-dev/lib/python3.6/site-packages/mlflow/server/handlers.py", line 83, in _get_tracking_store
    _tracking_store = _tracking_store_registry.get_store(store_uri, artifact_root)
  File "/Users/ljh/opt/miniconda3/envs/mlflow-1.4.0-dev/lib/python3.6/site-packages/mlflow/tracking/_tracking_service/registry.py", line 37, in get_store
    return builder(store_uri=store_uri, artifact_uri=artifact_uri)
  File "/Users/ljh/opt/miniconda3/envs/mlflow-1.4.0-dev/lib/python3.6/site-packages/mlflow/server/handlers.py", line 54, in _get_sqlalchemy_store
    return SqlAlchemyStore(store_uri, artifact_uri)
  File "/Users/ljh/opt/miniconda3/envs/mlflow-1.4.0-dev/lib/python3.6/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 99, in __init__
    mlflow.store.db.utils._verify_schema(self.engine)
  File "/Users/ljh/opt/miniconda3/envs/mlflow-1.4.0-dev/lib/python3.6/site-packages/mlflow/store/db/utils.py", line 52, in _verify_schema
    "more detail." % (current_rev, head_revision))
mlflow.exceptions.MlflowException: Detected out-of-date database schema (found version 84291f40a231, but expected 2b4d017a5e9b). Take a backup of your database, then run 'mlflow db upgrade <database_uri>' to migrate your database to the latest schema. NOTE: schema migration may result in database downtime - please consult your database's documentation for more detail.

This indicates that the upgrade was successful

At this time, it is executed in the CONDA environment of mlflow 1.11.0

 mlflow server \
      --backend-store-uri mysql://root:[email protected]/mlflow \
      --host 0.0.0.0 -p 5003 \
      --default-artifact-root s3://mlflow

You can see the page normally, so the upgrade of mlflow from 1.4.0 to 1.11.0 is completed

matters needing attention

If it is an online operation, the database should be backed up first, because the upgrade may not guarantee the success of the upgrade. If the upgrade fails, it can be directly recovered from the backup database or referencedFailure handlingProcessing