<h1 class="display-5 link-body-emphasis mb-1">MLOps and comparison of different models</h1> <p class="blog-post-meta">January 1, 2021 by <a href="/about">Irfan TOOR</a></p>

Toor_Irfan_3_article_de_blog_012023 pdf

  • Irfan Toor (OpenClassrooms | Projet No. 7)
  • keywords : MLOps, MLFlow, traçage, organisation, modèles m/l, optimisation, hyper-paramètres, versioning, autolog, grid-search, reproduction, erreurs, package, partage, model simple, LSTM, BERT, transfer-learning, comparaison, embeddings, word2vec, Glove.6B, TweetEval, roBERTa, sentiment, sst-2, analysis

A B S T R A C T

"Je crois qu'on ne perdure pas si on perd les repères."
-- David de Rothschild

Artificial Intelligence has the potential to change the way humans interact and behave with the existing or the new phenomenas of the world. The automatic collection of data, it's cleaning, processing or feature engineering and the ever increasing compute-power are improving the existing models at an exponential rate. More inclined we are to this ever changing environment and it's associated dependencies, which are changing at a phenomenal rate, more we are prone to the risks associated with finding ourself in the unknown lands and the territories of the machine control, thereby falling prey to this self invented trap and to an ultimate oblivion.

AI is not a new technology, which is taking the world by storm but its the logical evolution of what we have already been creating through machine learning models. We, the humans, have understood after the experience of millenniums that keeping track of whatever we are doing is fundamental to our boundless potential, flawless evolution and behemoth success in every domain we touch. So, its quite natural that we transfer and apply the knowledge and the concepts learned in one domain to another, which could not have been possible without tracking every bit of our experiments -- be it psychology or science.

The concepts of DevOps have found their way into the M/L world, not only facilitating the versioning, logging and tracking of models but are extended into keeping track of the data, the parameters and the performance of these models as well. This concept of tracking and managing all of the life-cycle of M/L models, the associated data, their versioning, performance and parameter tracking and even staging from development to deployment is handled by Machine/Learning Operations i.e MLOps. We will discuss about MLOps, its deployment, and its usage through the comparison of three models namely: Simple, LSTM and BERT.


1. I N T R O D U C T I O N

1.1 MLOps - Experiment tracking

What is MLOps?

MLOps is to Machine Learning models, as DevOps is to development. Since the requirements for Machine Learning are more than that of the Development, the MLOps have more to manage and handle as compared to DevOps. Its not about keeping track of the code, the dependencies or staging alone, but that of the data versioning, hyper-parameters, performance metrics also.

Furthermore the life cycles of the models, their different versions, development, staging and deployment etc. are all managed by a combination of tools, protocols, policies and practices known as MLOps.

How an MLOps paradigme helps us?

MLOps paradigm gives us the following advantages:

  • For reproducibility of errors or the state of models
  • Organisation of different models, data, hyper-parameters and the results etc.
  • Optimisation of models/hyper-parameters
  • Automatic logging
  • Consistance between the standards of logging

Why the other classic techniques like spread-sheets are not so effective?

  • Possibility of incorporating errors due to its manual logging nature.
  • No standards have so far been established regarding the spread-sheet logging
  • Different or inconsistant logging is prone to erroneous results.
  • Problems of interpretability of different columns or values might arise over time.

1.2 MLFlow - Manage the complete life-cycle of your M/L Models

What is MLFlow?

MLFlow helps in managing the complete life cycle of an M/L Project.

In a nutshell MLFlow is the combination of tools or techniques to give the AI or M/L Engineers, the possibility to not only manage their models, and their data, but to easily fine tune these models and play with different versions and are capable to be deployed at different places with different policies or parameters.

It helps track the results by not only automatically logging the metrics, the parameters and even the models but it gives you the flexibility to compare different models on the basis of different parameters or performance, not only in the tabular form but in the graphical format as well.

It can be easily integrated into any existing models easily and provides with a web interface, where the different M/L models can be comprehensively analyzed and compared to other models. Recently, certain M/L Model integration code has also been included in mlflow, facilitating the auto-logging of a lot of regression or logistic models.

The four components offered by MLFlow:

  • MLflow Tracking -- Record and query experiments i.e the code, data, parameters and the results
  • MLflow Projects -- Packaging ML code, reproducible to share, transfer to diff stages
  • MLflow Models -- Deploying models from ML Libraries in diverse environments
  • Model Registry -- Central model repository to manage versioning, annotation and discovery

    Note : MLFlow is library-agnostic

What is MLFlow Tracking?

MLFlow tracking includes but is not limited to following components :

  • Organisation of Experiments
    • Logging the result of different runs
    • Tracking API
  • Track of different M/L models :
    • Source Code source versioning
    • Trained Model
    • Data and its versioning
    • Associated Artifacts / Metadata
  • Automatic logging :
    • Scikit-learn
    • Keras
    • Pytorch
    • XGBoost
    • LightBost
  • Hyper-Parameters
  • Evaluation/Performance Metrics
  • Storage
    • Backend store : Filesystem / Sqlite db etc.
    • Artifacts storage : Amazon S3, , Azure Blob Storage, Google Cloud, FTP etc.

For comparing different models, it helps with :

  • Tabular comparaison
  • Graphical comparison
  • Effect of parameters on performance

What is MLFlow Projects?

MLFlow Projects is a convention for projects and data storage, so that the other data scientists can interact or use your code managed by MLFlow :

  • The root directory is named after the project's name
  • An entry point is defined with a .sh or .py file. These entry points do not have any parameters by default.
  • While using API, the parameters can be passed to the entry point.
  • Helps in building multi step flows:
    • The data science code be modularized and version so that different teams can work on different branches at the same time
    • Hyper-parameter tuning can be achieved through multiple runs locally or even in cloud environments for taking benefits of parallel computing, thus helping in finding the best performing models
    • Cross-validation and grid search can be used directly by providing different ranges of hyper-parameters, to find the best parameters for a specific kind of a data.

What is an MLFlow Model?

MLFlow model is a standard format for packaging M/L Models, so that these models can be used with a lot of other available tools and can be made available to public to be used in their projects, not having to be dependent upon the associated dependencies in their usage.

MLFlow Model keeps stores the models and the related environment information in the following directory structure:

model_alpha_beta_zetta/
├── MLmodel
├── model.pk
├── conda.yaml
├── python_env.yaml
└── requirements.txt

This model can be used by :

```sh
(.venv) % mlflow models serve model_alpha_beta_zetta
```

Some interesting fields concerning the management of projects consist of signatures, inputs and the runtimes etc.

What is an MLFlow Registry?

MLFlow Registry is a central repository for the storage of models, a set of APIs and UI to help manage the life cycle of your ML Models. It provides model annotation, their lineage (i.e. a kind of traceability of experiments to different runs with different parameters to different models), version control and their staging from development to deployment.

MLFlow Registry provides you with :

  • Web UI : Registering / Using the model
  • API Workflow : Adding/Registering, Fetching, Serving, Staging, Listing and Searching etc.

2. D E P L O Y M E N T

How to install MLFlow?

mlflow can be installed in your python workflow by using :

```sh
(.venv) % pip install mlflow
```

Note :

  • As of this writing the latest version of mlflow available for mac is 2.1.1
  • Certain libraries might need a downgrading to function properly with mlflow

How to run the MLFlow Server?

MLFlow server instance can be initiated with the following command locally:

```sh
(.venv) % mlflow server -p 8000

[2023-02-20 19:13:42 +0100] [58273] [INFO] Starting gunicorn 20.1.0
[2023-02-20 19:13:42 +0100] [58273] [INFO] Listening at: http://127.0.0.1:8000 (58273)
[2023-02-20 19:13:42 +0100] [58273] [INFO] Using worker: sync
[2023-02-20 19:13:42 +0100] [58274] [INFO] Booting worker with pid: 58274
[2023-02-20 19:13:42 +0100] [58275] [INFO] Booting worker with pid: 58275
[2023-02-20 19:13:42 +0100] [58276] [INFO] Booting worker with pid: 58276
[2023-02-20 19:13:42 +0100] [58277] [INFO] Booting worker with pid: 58277
```

_Note : -p helps defining the localhost port, since it uses 5000 by default, which might pose a problem of connectivity with a few versions of MacOS_

Note that the server launches multiple listening threads, to load-balance any incoming requests. After launching this server you can access its Web UI, by accessing http://127.0.0.1:8000, using your navigator i.e. Safari or Firefox etc.

MLFlow Experiments :

MLFlow is capable of managing and grouping the results in different experiments, so that different projects or teams can be separated at experimentation levels easily.

alt=mlflow-exp

MLFlow Model :

Each model can be tracked. It's description, parameters, metrics, tags and artifacts can be tracked. As an example the artifacts of the model can be seen in the next figures.

lt=model deliteful-colt alt=confusion matrix alt=recall curve alt:roc curve

MLFlow Grid-Search :

We can select all of the models created and grouped as sub models, while doing a grid search, and launch do a comparative analysis of their parameters as well as their results.

![alt=g](/img/mlops/grid-search.pnrid searchg)

MLFlow Comparaison :

Different runs can be compared graphically, their metrics and run details can also be compared in tabular form.

alt=comparing 5 runs alt=comparing 6 runs alt=comparing 26 runs alt=comparing metrics alt=comparing run details

MLFlow Models :

The models can be registered so that these can be easily staged for production.

alt=registered models alt=delightful-colt alt=registered-v1 alt=registered-v1-inputs

Model lineage helps tracking the model and their respective parameters and results etc.

alt=model delightful-colt alt=deliteful-colt parameters alt=delightful-colot metrics alt=delightful-colt artifacts


3. C O M P A R A I S O N

alt=comparison


4. P R E D I C T I O N S

4.1 Simple Model

alt=simple predictions

4.2 Model roBERTa (TweetEval)

alt=advanced predictions


5. C O N C L U S I O N

Its a complex and time consuming process to first choose a model, then adding the training layers according to the type of data and then optimizing the model. There are no particular standards defining the vector lengths of embeddings, but models are seemingly trying to stick to the de-facto and/or the precedence established by the pioneers of the models in different categories.

In the mean time it is important to note that the https://huggingface.co/ is an exceptional resource for the pre-trained models or embeddings, for eliminating the time and huge computing resources required to train the models. These pertained models and embeddings can be fine-tuned with our own data to handle a specific task.


6. D E C L A R A T I O N

I am not affiliated to any of the organisations, tools or technologies, referred or used while doing the experiments, with an intended direct or indirect financial interest, and I have tried my level best to be rational and unbiased regarding the comparison of the discussed models.


7. A C K N O W L E D G E M E N T S

I thank twitter for the availability of the dataset, all of the people involved in the production and availability of python, the m/l libraries specially Numpy, Pandas, Scikit-learn, Tensorflow, Keras, MLFlow, LSTM, Huggingface for making available the NLP models like BERT etc. and twitter for the availability of the tweets so that a comparison of models for a sentiment analysis could be done.

I personally thank the team behind OpenClassrooms for adding the innovative references to the courses and specially to my mentor Mr. Panayotis Papoutsis for his invaluable guidelines and advices.


8. R E F E R E N C E

About

Irfan TOOR

Irfan TOOR is an Engineer (Civil + AI), and an Innovator, passionate about the Information Technology, Digital Transformation, Cyberspace and Cybersecurity.

He obtained his first degree in Civil Engineering from UET (1993), and obtained his second degree of Engineering in Artificial Intelligence (Master 2) from OC (2023).

In his own words: "I love coding, optimizing and at times re-inventing."
— Irfan TOOR

continue ...

Recent posts