16. Evaluation of a National Economic Model - An Abstract Concept and Design

Evaluation of National Economic Model - An Abstract Concept and Design

At the onset, I would like to emphasize that this is a very high level abstract conceptual idea involving the use of ML (Machine Learning) models to evaluate a giant entity like the national economy.

The field of AI/ML is now being used in almost all spheres of life. How best to use this technology when tackling a juggernaut like a national economy?

New ML models involving not millions, but billions of parameters are being created and deployed in production for various purposes. My mind vaguely recollect a recent data where it is suggested that on an average, a large global organization with a team of 300 data science associates (including analysts, engineers, scientists, SMEs) may evaluate about 1000-1200 models a year and only about probably 10-15 of them make it to the production.

I think , a ML model to evaluate a national economy may contain many sub-models and may need good integration logics with the encompassing model.

I am presenting below a very crude and raw-cut parameter/model list of a national economy.

In the representation above, some may be parameters, some-may be sub-models and some others can be just critical Hyperparameters.

In machine learning a hyperparameter is a parameter that can be set in order to define any configurable part of a model’s learning process. Hyperparameters can be classified as either model hyperparameters (such as the topology and size of a neural network) or algorithm hyperparameters (such as the learning rate and the batch size of an optimizer). These are named hyperparameters in contrast to parameters, which are characteristics that the model learns from the data.

I think, to build a gigantic ML model representing national economy the process may look something like how I have outlined below.

1. Set the mission, vision and milestones for the project/program and define the MVP and project success criteria - be reasonable at the pilot attempt.

2. Preferably use SAFe (Scaled Agile Framework) for the program.

3. Identify the needs and boundaries of the main program and the pilot projects.

4. Identify the first round of Project Directors, Program/Project Managers, Engineers, Analysts, SMEs, Data Scientists, Statisticians and Mathematicians needed to identify the early raw models and sub-models along with the pipeline design and program executing/operating parameters.

5. Create the Program structure consisting of project teams, governance team and program management team.

6. Create a repeatable project structure and process to be followed by each project team with defined goals with appropriate milestones and definitions of done for requirements.

7. Create program and project teams with assigned G&Os. (A ML model like the model for a national economy may need may different sub-models where each sub-models may contain hundreds if not thousands of parameters). If possible, start identifying the early hyperparameters for the encompassing model.

8. Conceptually design and test how the integration of various sub-models is going to be achieved procedurally, process-wise and technically. Start with a raw design and improve as you go.

9. Create the sample production pipeline for the ML model and test it.

10. All necessary technical infrastructure and software should be available for the team.

11. Technical processes related to meta-data repositories, data catalogue, data governance. data security and data obsoletion should be evolved

12. Each team targeting a sub-model should perfect the techniques on –

a. Data acquisition

b. Data Cleansing

c. Data Transformation

d. Feature Extraction

e. Feature Selection

f. Feature Iteration

13. The integrations of sub-models are expected to be hard-to-tackle-beasts and will need involvement of high-level SMEs.

14. Processes around model documentation, testing, cataloguing, retirement etc., should be established and should be integrated with the production pipeline.

15. Model testing includes, testing of sub-models, model-integration testing and testing of encompassing models. It also includes model evaluation and performance testing.

16. The deployment of the model may be designed to happen in 3 stages – pre-production, production and post-production. (Evaluate the suitable deployment mechanisms like the multi-service deployment, Blue-Green Deployment, Canary Deployment etc.,)

Next, how can we use data coming out of the model runs?

This is a model which is encapsulating an ultra-complex gigantic system. Hence the inputs can be complex – it may not be some point values but ranges with probabilities. Similarly, it is expected that the recommendations may present several scenarios with assumptions again each with a range and probability. We may need brainstorming by SMEs before we interpret the results and create action plans (SMART Goals).

We can be very innovative with respect to the arrangement of models and sub-models to the extent that, in a proposed 3-layered model consisting of base layer, integration layer and results layer, the last 2 layers - i.e the integration layer and results layer can be expected to dynamically evolve as the execution progresses !

If necessary, we can expect the execution to pause for analysis and critical inputs from SME after the execution of the base layer. Optionally a similar pause can be arranged post execution of the integration layer models for analysis and critical inputs. To tackle complexity, an elemental layer can also be considered before the base layer.

The various scenarios that is expected to be tested using this encompassing, configurable (at layer level and at model/sub-model levels in that layer), dynamic (at layer level and at model/sub-model levels in that layer), decoupled (across layer and in-between models in a layer) model are:

Given a group of inputs and controls , what can be the values of a group of target output parameters
Given a group of outputs and controls, what can be the values of a group of target input parameters
Given a group of inputs, expected output parameters and controls , what can be the values of a single target output parameter
Given a group of outputs, expected input parameters and controls , what can be the values of a single target input parameter
Given a group of outputs and inputs, what can be the optimal settings for a group of controls
Given a group of outputs, inputs and some expected controls, what can be the optimal settings for a single target control parameter

The executive team should charter the program team with the action outline – may be to identify the optimal interest rate cut or to favourably increase the balance of payments etc., Please note that the launch of the production model for the first time may take more time but the subsequent changes can be done in a reasonable time frame. The model will also be expected to have a great deal of configurability and decoupling along with the ability for quick execution post changes. The team, then should identify the dynamic parts, quasi-static parts and static parts and work on the encompassing model system. It should have access to top-quality data.

The final output may be a report for the executing team consisting of –

- Range of results and probabilities

- Assumptions for each of the results

- Constraints expected for each of the results

- Confidence level and probability of success towards achieving the goal set for each result

- Information on Risks and Rewards

- Contingency measures if the risks materialize

- Time expected to be available to implement the measures

- Expiry time for each of the results before fresh exercises are needed again

- Information on any collateral damages

References:

https://www.udemy.com/course/deployment-of-machine-learning-models