How to acces the pipeline's ConfigLoader?

[mjmare]

From the docs (https://kedro.readthedocs.io/en/stable/kedro_project_setup/configuration.html#local-and-base-configuration-environments) I know I can create a ConfigLoader. But how does one get the current ConfigLoader, ie the one set for the currently running pipeline?
For some more context: I want to access some credentials to initialise a hook.

In Kedro 0.17 I did something like this:

from kedro.framework.session import get_current_session
       :
       :
        session = get_current_session()
        context = session.load_context()
        credentials = context.config_loader.get('*credentials.yml')
        rb_token = credentials['rollbar']['token']
        rb_environment = 'development' if context.env == 'local' else 'production'
        rollbar.init(rb_token,
                     environment=rb_environment,
                     )
        

But get_current_session has gone away in 0.18.0. So what is the Kedro-esque way of getting the current ConfigLoader?

[noklam]

In the coming 0.18.1 release, a new hook after_context_created will expose the context object and it contains the information like env and config_loader.

#1458 There is a workaround in this issue for 0.18.0.

[mjmare]

Thanks. Workaround works!
Thanks also for the importing tip. It has bitten me in the past.

[giuliolovisotto]

Hi @noklam. What is the correct way to use the hook to access context from one of the nodes during a kedro run (I'm on kedro==0.18.3? I have something like the following:

# My node function
def get_annotations(image_dataset: Dict[str, Callable]) -> List[Dict]:
    # what I am currently doing
    conf_source = str(Path.cwd() / settings.CONF_SOURCE)
    conf_loader = ConfigLoader(conf_source=conf_source, env="local")
    username = conf_loader.get("credentials.yml").get("username")
    # do something which includes username
    ...

Now that get_current_session() has been removed, is there a way to do this that does not require specifying the conf_source path with Path.cwd()?

Thanks for the help.

[noklam]

Accessing context from a node defies the purpose of having a context. This shouldn't be needed to access data. Why do you need to read credentials manually? Would a custom dataset be more appropriate?

[giuliolovisotto]

I must admit my setup may be strange and I'm new here (I found out about kedro recently and I really like it, I am trying to use it on a toy example). What I'd like to do in the node get_annotations is to use a remote API (Google Cloud) to do the processing of the image_dataset, then save the output. The init of the API requires credentials, including username, that's why I was trying to fetch these inside the node - would you do this differently? Thanks.

[noklam]

The IO Save and Load should be done by a DataSet and the processing logic will remains in the node.

https://kedro.readthedocs.io/en/stable/extend_kedro/custom_datasets.html

You can refer to other datasets implementation. If you are using GCS it would be supported by native Kedro datasets, otherwise you may need to implement the connection to the corresponding API.