Repos are not initialized if configured via JSON

[jules2689]

Describe the bug
A clear and concise description of what the bug is.

I'm not sure if this is a bug or a feature request, but ultimately landed on considering it a bug for the purpose of this issue because it seems that backrest encourages the use of config.json, especially when restoring.

In my case, I am using Ansible to programmatically set up my computer. In my scripts I am generating a valid config.json file that includes repos and plans - Backrest has no problem detecting these, but leaves them uninitialized. This leads to checks failing, and backups cannot run. I can fix this by running init in the Run Command mode, but this is not ideal - im unsure if there are missing options (e.g. the opts variable on this line https://github.com/garethgeorge/backrest/blob/b3402a18d2026a2b5998ecdae5a9802f7b3c844a/pkg/restic/restic.go#L116C64-L116C68)

In an ideal world, backrest would loop through the repos on boot and initialize any of them that are not initialized (perhaps this can be a config in config.json!)

To Reproduce
Steps to reproduce the behavior:

1. Fill out a config.json with repos and plans for a new instance
2. Launch instance
   3.Go to repo and click "check now"
3. See error like unable to open config file: stat ...

Expected behavior

Repos should either:

• initialize on boot if uninitialized (ideal)
• Offer a 1 click button to initialize on the repo page if uninitialized

Platform Info

• OS and Architecture: Docker compose image garethgeorge/backrest:latest
• Backrest Version 1.6.2

[garethgeorge]

Hi, unfortunately there's a foot gun here irrespective of how I approach this

• If backrest implicitly initializes repos, it's possible for an accidental change to the config or a changed bind-mount to result in implicitly creating a new repo. This can lead to data loss with a user silently thinking their backups are going to some other destination.
• If backrest does not implicitly initialize repos, as you say, it's a harder to copy around configs or pragmatically initialize backrest.

I think a one-click init button could make sense as a nice to add, in the mean time I'll update the docs to recommend that if adding repos by config they should be already initialized OR can be initialized via run command.

Since backrest is backup software I've opted to be very conservative here and prefer making the user take action whenever there's ambiguity -- but this does mean there are more times that a user might be presented with an error that requires intervention.

[jules2689]

That makes sense @garethgeorge ... I assume I can just run a bare init and nothing else is needed? 😄

[garethgeorge]

Hmm. This is interesting actually -- with the latest update (1.7.0) there's a new concept of a repo GUID which means any added repo must have a random unique ID specified in its configuration.

Using init through backrest's add repo flow will populate this for you (from the repo's config) but running init externally won't. I'll likely need to submit some sort of patch to fix this for you elsewise you're likely to get an error when attempting to load your config.

You can work around the problem by simply generating a 256 bit (64 char hex) random ID in the GUID field, but it really is something that backrest should be doing for you.

tl;dr not sure that there was a problem here when you asked, I think there might be now. I'll include this in the set of fixes for 1.7.1.

[jules2689]

Appreciated! Id be fine with generating a random ID, but ideally Id just be able to provide a config like this on boot and backrest does whats needed:

{
  "modno":  1,
  "version":  4,
  "instance":  "minipc",
  "repos":  [
    {
      "id":  "s3-minipc",
      "uri":  "s3:https://s3.ca-east-006.backblazeb2.com/...",
      "password":  "...",
      "prunePolicy":  {
        "schedule":  {
          "cron":  "5 4 * * 0",
          "clock":  "CLOCK_LAST_RUN_TIME"
        },
        "maxUnusedPercent":  10
      },
      "checkPolicy":  {
        "schedule":  {
          "cron":  "0 0 1 * *",
          "clock":  "CLOCK_LAST_RUN_TIME"
        },
        "readDataSubsetPercent":  0
      },
      "commandPrefix":  {}
    },
    {
      "id":  "raid_array",
      "uri":  "/raid_array",
      "password":  "12345",
      "prunePolicy":  {
        "schedule":  {
          "cron":  "0 */6 * * *",
          "clock":  "CLOCK_LAST_RUN_TIME"
        },
        "maxUnusedPercent":  10
      },
      "checkPolicy":  {
        "schedule":  {
          "cron":  "0 0 1 * *",
          "clock":  "CLOCK_LAST_RUN_TIME"
        },
        "readDataSubsetPercent":  0
      },
      "commandPrefix":  {}
    }
  ],
  "plans":  [
    {
      "id":  "s3-minipc",
      "repo":  "s3-minipc",
      "paths":  [
        "/wd_2tb"
      ],
      "schedule":  {
        "cron":  "0 * * * *",
        "clock":  "CLOCK_LOCAL"
      },
      "retention":  {
        "policyTimeBucketed":  {
          "hourly":  24,
          "daily":  7,
          "monthly":  1
        }
      },
      "hooks":  [
        {
          "conditions":  [
            "CONDITION_UNKNOWN",
            "CONDITION_ANY_ERROR",
            "CONDITION_SNAPSHOT_ERROR",
            "CONDITION_SNAPSHOT_WARNING",
            "CONDITION_SNAPSHOT_SKIPPED"
          ],
          "actionShoutrrr":  {
            "shoutrrrUrl":  "ntfy://...",
            "template":  "{{ .Summary }}"
          }
        }
      ]
    },
    {
      "id":  "raid_array",
      "repo":  "raid_array",
      "paths":  [
        "/wd_2tb"
      ],
      "schedule":  {
        "cron":  "0 * * * *",
        "clock":  "CLOCK_LOCAL"
      },
      "retention":  {
        "policyTimeBucketed":  {
          "hourly":  24,
          "daily":  30,
          "monthly":  12
        }
      },
      "hooks":  [
        {
          "conditions":  [
            "CONDITION_UNKNOWN",
            "CONDITION_ANY_ERROR",
            "CONDITION_SNAPSHOT_ERROR",
            "CONDITION_SNAPSHOT_WARNING",
            "CONDITION_SNAPSHOT_SKIPPED"
          ],
          "actionShoutrrr":  {
            "shoutrrrUrl":  "ntfy://...",
            "template":  "{{ .Summary }}"
          }
        }
      ]
    }
  ],
  "auth":  {
    "disabled":  true,
    "users":  [
      {
        "name":  "jules2689",
        "passwordBcrypt":  "12345"
      }
    ]
  }
}

[garethgeorge]

This wound up being surprisingly painful but I think important to address. The solution I've come up with for the time being is a new autoInitialize property that can be set on repos that backrest should automatically create.

The PR is #650 . In a followup backrest will use a similar flow to automatically populate the GUID for config entries missing it.