BioNeMo 2: Accumule TB history for the clients. Clients recording TB history locally from trainer:

[holgerroth]

          Suggestion regarding accumulation TB history for the clients. Clients recording TB history locally from `trainer`:

from tensorboard.backend.event_processing.event_accumulator import EventAccumulator

def get_tensorboard_history(trainer):
    # Get the log directory from trainer's logger
    log_dir = trainer.logger.experiment.log_dir
    
    # Create an EventAccumulator object
    event_acc = EventAccumulator(log_dir)
    event_acc.Reload()  # Load all data
    
    # Get available tags (metrics)
    tags = event_acc.Tags()['scalars']
    
    # Extract history for each metric
    history = {}
    for tag in tags:
        events = event_acc.Scalars(tag)
        # Each event has attributes: wall_time, step, value
        history[tag] = [(e.step, e.value) for e in events]
    
    return history

And then after sharing through Shareable object you create a TB writer to accumulate them all:

tb_writer = SummaryWriter('continuous_logs')
for shareable in shareables:
    metrics_history = shareable['history']
    # Write each metric to the continuous log
    for metric_name, values in metrics_history.items():
        for step, value in values:
            global_step = (current_round * max_steps_per_round) + step
            writer.add_scalar(f'continuous/{metric_name}',  value, global_step=global_step)

Originally posted by @farhadrgh in #3095 (comment)