Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not all valid YAML files are supported by Autosubmit at the moment #15

Open
LuiggiTenorioK opened this issue May 31, 2023 · 17 comments
Open

Not all valid YAML files are supported by Autosubmit at the moment #15

LuiggiTenorioK opened this issue May 31, 2023 · 17 comments
Assignees
@dbeltrankyl
Labels
testing

Comments

@LuiggiTenorioK
Copy link
Member

In GitLab by @kinow on May 31, 2023, 12:50

( This is a template for bug reports. Please fill in the relevant information and delete the rest. )

Hello @dbeltrankyl and @kinow,

Autosubmit Version

Latest & latest autosubmit4-config-parser

Expid affected(If applicable)

NA

Which task has issues? Where is the log(If applicable)

NA

Summary

See linked ticket in the parser: https://earth.bsc.es/gitlab/ces/autosubmit4-config-parser/-/issues/3

Not all valid YAML files are supported by Autosubmit. This could be an issue, for example, if users are assuming they can use existing YAML configuration files, and if those files use something like an array with dictionaries.

Found while working on the RO-Crate issue this morning autosubmit#929

Steps to reproduce

Use anything like this in an Autosubmit configuration:

ROCRATE:
  INPUT:
    - name: "DATELIST"
      value: "%EXPERIMENT.DATELIST%"
      additionalType: "String"
      valueRequired: "True"

What is the current bug behavior?

(autosubmit4) kinow@ranma:~/Development/python/workspace/autosubmit4-config-parser$ autosubmit archive --rocrate a008 && unzip -p ~/autosubmit/a008/rocrate.zip ro-crate-metadata.json > ~/autosubmit/a008/ro-crate-metadata.json && subl ~/autosubmit/a008/ro-crate-metadata.json
Autosubmit is running with 4.0.0b
[ERROR] Trace: unhashable type: 'dict'
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/autosubmit/bin/autosubmit", line 36, in main
    Autosubmit.parse_args()
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 713, in parse_args
    return Autosubmit.rocrate(args.expid)
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 4231, in rocrate
    as_configuration.reload(True)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1382, in reload
    custom_conf_pre = self.load_custom_config_section({}, filenames_to_load["PRE"])
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1338, in load_custom_config_section
    current_data_pre,current_data_post = self.load_custom_config(current_data, filenames_to_load)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1305, in load_custom_config
    current_data_pre,current_data_post = self.load_config_folder(copy.deepcopy(current_data),filename)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 612, in load_config_folder
    return self.load_custom_config(current_data, filenames_to_load)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1317, in load_custom_config
    current_data_pre = self.unify_conf(current_data_pre,self.load_custom_config_section(copy.deepcopy(current_data), filenames_to_load_level["PRE"]))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1338, in load_custom_config_section
    current_data_pre,current_data_post = self.load_custom_config(current_data, filenames_to_load)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1305, in load_custom_config
    current_data_pre,current_data_post = self.load_config_folder(copy.deepcopy(current_data),filename)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 612, in load_config_folder
    return self.load_custom_config(current_data, filenames_to_load)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1310, in load_custom_config
    current_data = self.substitute_dynamic_variables(self.unify_conf(self.substitute_dynamic_variables(current_data), self.load_config_file(current_data,filename)))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 583, in load_config_file
    return self.unify_conf(current_folder_data,self.deep_normalize(new_file.data))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 657, in unify_conf
    current_data = self.deep_update(current_data, new_data)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 490, in deep_update
    tmp = self.deep_update(unified_config.get(key, {}), val)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 494, in deep_update
    if current_list != set(val):
TypeError: unhashable type: 'dict'
 [CRITICAL] Noneunhashable type: 'dict'
Unhandled error: If you see this message, please report it in Autosubmit's GitLab project

The linked issue in autosubmit4-config-parser has more information.

What is the expected correct behavior?

Either support every valid YAML file, or document & warn users about what is supported or not.

Relevant logs and/or screenshots(if applicable)

Done above, and also in the linked issue in autosubmit4-config-parser.

Any other relevant information(if applicable)

()
@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on May 31, 2023, 12:50

To test some YAML, one can use https://www.yamllint.com/, e.g.

image

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on May 31, 2023, 12:55

Note too that in CWL it's quite common to have this kind of dict/objects in lists, e.g. https://www.commonwl.org/user_guide/topics/expressions.html

image

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on May 31, 2023, 12:58

Here's another example.

I went to the ScriptEngine website, and copied their Hello Planet example:

- do:
  - base.context:
      planet: Earth
  - base.echo:
      msg: "Hello, {{planet}}!"

Validated in YAML Lint

image

And saved that file to my conf/scriptengine.yml. Then removed the previous invalid block (the one that I used for this issue, so that it would not raise an issue), and re-executed my command. That ScriptEngine YAML also causes the workflow to fail to create/run/etc if it's found in the conf folder:

image

ScriptEngine is used in EC-Earth 4, I think. So we should probably support their YAML syntax, or let users know up ahead if they will encounter any issues (and maybe give them some workaround instructions too).

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @dbeltrankyl on Jun 1, 2023, 08:56

Yes, right now, not all yaml files are available due to the post-processing that Autosubmit does.

It is a bit tricky to support all formats due to the placeholder substitution mechanism and the fact that there Can be duplicated variables ( in different files ) while yaml standard doesn't allow it

related to autosubmit#1000

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on Jun 5, 2023, 23:50

mentioned in merge request autosubmit!317

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on Jul 4, 2023, 13:26

mentioned in issue python-amip-reader#5

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @dbeltrankyl on Sep 21, 2023, 11:51

Hello,

There were some fixes for other issues regarding this in the last weeks, maybe it is worth checking if it is working for the rocrate issue

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on Sep 21, 2023, 14:21

Thanks for the heads up Dani! I'm almost on my way to the airport to return to Barcelona. My inbox is more or less under control, so I should be able to start working on issues again next week. Will try to have a look at this one, and check if the ScriptEngine YAML files can be parsed by AS, as well as the RO-Crate YAML files & that merge request.

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on Sep 27, 2023, 10:22

Hi @dbeltrankyl ,

I searched the ScriptEngine documentation, and copied the very first YAML I found, here https://scriptengine.readthedocs.io/en/latest/concepts.html

- base.context:
    planet: Earth
- base.echo:
    msg: "Hello, {{planet}}!"

That file can be parsed by online YAML tools, e.g. https://yaml-online-parser.appspot.com/?url=http%3A%2F%2F

image

But if I create a dummy experiment, using the latest Autosubmit code from master, and copy this YAML file somewhere like conf/se.yml, then autosubmit create fails with:

(autosubmit4) kinow@ranma:~/autosubmit$ autosubmit create a006
/home/kinow/mambaforge/envs/autosubmit4/bin/autosubmit:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('autosubmit==4.0.93')
[ERROR] Trace: 'list' object has no attribute 'get'
 [CRITICAL] Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/autosubmit/bin/autosubmit", line 37, in main
    Autosubmit.parse_args()
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 655, in parse_args
    Autosubmit._init_logs(args, args.logconsole, args.logfile, expid)
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 797, in _init_logs
    as_conf.reload(force_load=True)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1387, in reload
    starter_conf = self.unify_conf(starter_conf, self.load_config_file(starter_conf, Path(filename)))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 586, in load_config_file
    if new_file.data.get("DEFAULT", 7000).get("CUSTOM_CONFIG", None) is not None:
AttributeError: 'list' object has no attribute 'get'
'list' object has no attribute 'get'
Unhandled error: If you see this message, please report it in Autosubmit's GitLab project

I checked out the latest main branch from autosubmit4-config-parser, and re-installed it on my venv with pip install -e .. Tried again, and got

(autosubmit4) kinow@ranma:~/Development/python/workspace/autosubmit4-config-parser$ autosubmit create a006
/home/kinow/mambaforge/envs/autosubmit4/bin/autosubmit:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('autosubmit==4.0.93')
[ERROR] Trace: 'list' object has no attribute 'get'
 [CRITICAL] Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/autosubmit/bin/autosubmit", line 37, in main
    Autosubmit.parse_args()
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 655, in parse_args
    Autosubmit._init_logs(args, args.logconsole, args.logfile, expid)
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 797, in _init_logs
    as_conf.reload(force_load=True)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1395, in reload
    starter_conf = self.unify_conf(starter_conf, self.load_config_file(starter_conf, Path(filename)))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 589, in load_config_file
    if new_file.data.get("DEFAULT", 7000).get("CUSTOM_CONFIG", None) is not None:
AttributeError: 'list' object has no attribute 'get'
'list' object has no attribute 'get'
Unhandled error: If you see this message, please report it in Autosubmit's GitLab project

Same error, I believe.

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on Feb 1, 2024, 12:51

This issue might be important again, now for the fondos/phenomena project, where they will need a YAML file for their workflow & application. We will have to check if they need/want to use any feature of YAML that is not supported by AS4 config model (the sooner we identify that, the merrier -- cc @ctena & Carmen [couldn't find her user])

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @dbeltrankyl on Feb 27, 2024, 12:53

mentioned in issue digital-twins/de_340/workflow#235

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on Mar 5, 2024, 11:10

mentioned in issue autosubmit#1240

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @kinow on Mar 12, 2024, 15:45

mentioned in issue otintopr/visualization#9

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @dbeltrankyl on Aug 13, 2024, 10:53

mentioned in merge request autosubmit4-config-parser!9

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @dbeltrankyl on Aug 19, 2024, 13:05

All code is done in the asconfigparser side https://earth.bsc.es/gitlab/es/autosubmit4-config-parser/-/merge_requests/9

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @dbeltrankyl on Aug 19, 2024, 13:07

moved from autosubmit#1045

@LuiggiTenorioK
Copy link
Member Author

In GitLab by @dbeltrankyl on Aug 19, 2024, 13:07

Moved from autosubmit repo to asconfigparser

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dbeltrankyl dbeltrankyl

Labels
testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@LuiggiTenorioK @dbeltrankyl