Replace the backup disk with a soft_delete flag on object storage disks

[joelynch]

No description provided.

[tilman-aiven]

soft_delete is silent in both directions

• nothing in the log when it engages
• no error when it is set where nothing reads it

After create_directories we log when it engages

LOG_INFO(
    getLogger("registerDiskObjectStorage"),
    "Disk `{}`: soft delete enabled, deletion markers under {}",
    disk_name, markers_path);

This makes the invariant auditable in an incident: "was this disk soft-deleting?" becomes one grep instead of an inference from the stack shape.

Reject it where it is ignored

/// `soft_delete` is honoured only by the object storage disk that owns the blobs (see
/// `wrapIfSoftDelete`). A disk layer above it cannot implement soft delete: it would
/// delegate the removal downwards and the blob would be physically unlinked. Reject the
/// flag rather than accept a config that silently does nothing.
void rejectSoftDeleteFlag(
    const Poco::Util::AbstractConfiguration & config,
    const String & config_prefix,
    const String & name,
    const String & disk_type)
{
    if (!config.getBool(config_prefix + ".soft_delete", false))
        return;

    throw Exception(
        ErrorCodes::BAD_ARGUMENTS,
        "Disk `{}` of type `{}` does not support `soft_delete`. Set it on the object storage disk "
        "that holds the blobs, not on a layer above it",
        name, disk_type);
}