Restore progress

[kstewart]

Similar to #20, do something similar to periodically update the progress of a restore.

[ncdc]

Idea: store per-restore log file to ???. Add ark restore log command to retrieve it.

[nrb]

We now have the ark restore logs command, but it retrieves finished logs vs providing a progress update.

Pulling over some details from the comments on #20, it would be possible to store the number of items in a backup on the backup metadata, which would help with restoring when the entire backup is being restored, though doing a selective restore would still need to do some filtering.

I like the idea of some generic 'progress' struct that applies to both backups and restores.

[ncdc]

Is it worth considering using Kubernetes events for this? They have a pretty short TTL, but could be used on the fly to see what's going on. They could be used in addition to status field updates, perhaps.

[nrb]

That would make sense to me. A restore/backup could generate an EventSeries for that restore and external processes could look for that series to be in the Finished state to know when to stop monitoring.

Depending on how much information we want to report and when in the restore process, we could potentially used the Related field to link to the most recent object restored.

[ncdc]

It wouldn't work quite like that. The events could contain data such as starting to restore pods, starting to restore pod/foo, finished restoring pod/foo, finished restoring pods, and so on. A watcher that wants to know when a restore is done would look at the restore's status.phase.

An alternative that doesn't use events is to periodically update some field(s) on the restore's status related to progress, as described in #20. I am always concerned, however, about unbounded lists and maps in our data types, given the etcd per item size limit. (The unbounded bits here would be # of api groups and resources.)

[nrb]

Is the Related field not serialized when the event is sent? This is the one I'm referring to specifically. It looks somewhat new, as does the EventSeries type.

I don't think we should store an explicit list for progress for the exact reason you mentioned.

[jhamilton1]

Adding a suggestion to this issue. Can the "ark get restore " command output that a non kube-system namespace did not restore because the namespace already existed and contained the pods to be restored?

[ncdc]

@jhamilton1 we wouldn't show that in ark get restore as that's meant to be a high-level summary that lists the restores and their statuses. We could put it in ark describe restore. If we do that, it would go as a per-namespace error (which we already have support for). I think this RFE belongs in #691 as part of the timeout discussion (after we exceed the timeout, record an error).

[jhamilton1]

@ncdc okay I will make a note on #691