Skip to content

The frequency of the predicted results of the finetuned model cannot be aligned #209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SoarAnyway opened this issue Apr 16, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@SoarAnyway
Copy link

SoarAnyway commented Apr 16, 2025

Describe the bug
I try to write moirai-moe finetune code, finetune sucessfully, both the training loss and validation loss were reduced relatively low,but The frequency of the predicted results of the finetuned model is wrong, I don't know if the problem lies in the finetuning stage or the prediction stage

the picture of train loss

Image

the picture of predict result

Image

To Reproduce
Fine tuned loss function still using Moirai,in function train_transform_map I fix PatchCrop

finetune model yaml set as

_target_: uni2ts.model.moirai_moe.MoiraiMoEFinetune
module:
  _target_: uni2ts.model.moirai_moe.MoiraiMoEModule.from_pretrained
  pretrained_model_name_or_path: Salesforce/moirai-moe-1.0-R-small
module_kwargs:
  _target_: builtins.dict
  distr_output:
    _target_: uni2ts.distribution.MixtureOutput
    components:
      - _target_: uni2ts.distribution.StudentTOutput
      - _target_: uni2ts.distribution.NormalFixedScaleOutput
      - _target_: uni2ts.distribution.NegativeBinomialOutput
      - _target_: uni2ts.distribution.LogNormalOutput
  d_model: 384
  num_layers: 6
  patch_sizes: ${as_tuple:[16]}
  max_seq_len: 2800
  attn_dropout_p: 0.0
  dropout_p: 0.0
  scaling: true
min_patches: 2
min_mask_ratio: 0.15
max_mask_ratio: 0.5
max_dim: 128
loss_func:
  _target_: uni2ts.loss.packed.PackedNLLLoss
val_metric:
  - _target_: uni2ts.loss.packed.PackedMSELoss
  - _target_: uni2ts.loss.packed.PackedNRMSELoss
    normalize: absolute_target_squared
lr: 5e-4
weight_decay: 1e-1
beta1: 0.9
beta2: 0.98
num_training_steps: ${mul:${trainer.max_epochs},${train_dataloader.num_batches_per_epoch}}
num_warmup_steps: 0

predict model init as

device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
model_one = MoiraiMoEForecast.load_from_checkpoint(model_path, 
                                            prediction_length=192,
        context_length=5*96,
        patch_size=16,
        num_samples=20,
        target_dim=1,
        feat_dynamic_real_dim=len(feature_cols),
        past_feat_dynamic_real_dim=0,).to(device)
@SoarAnyway SoarAnyway added the bug Something isn't working label Apr 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant