Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGES/+migrate-push-repository.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Added a `migrate` endpoint on push repositories that converts a legacy `ContainerPushRepository`
into a `ContainerRepository` in place. The repository primary key and version history are
preserved; only the repository type changes.
24 changes: 23 additions & 1 deletion docs/user/guides/push-image.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,36 @@ X-Frame-Options: SAMEORIGIN

!!! note

Content is pushed to a push repository type. A push repository does not support mirroring of the
New registry pushes create a regular `ContainerRepository`. Older deployments may still have
legacy `ContainerPushRepository` instances. A push repository does not support mirroring of
remote content via the Pulp API. Trying to push content with the same name as an existing
"regular" repository will fail.

!!! note

Rollback to the previous repository versions is not possible with a push repository. Its latest version will always be served.

## Migrating legacy push repositories

Legacy push repositories can be converted to regular container repositories with the `migrate`
action. Migration changes the repository type in place: the repository primary key, name,
metadata, content, version history, and distribution association are preserved. Registry paths
and tags stay the same. The repository href path changes from `container-push` to `container`;
the converted repository is returned in the task result.

```bash
http POST $BASE_ADDR/pulp/api/v3/repositories/container/container-push/$PUSH_REPO_PK/migrate/
```

After migration, the repository is listed under container repositories instead of push
repositories. You can continue to push and pull through the same registry path, and you gain
standard repository operations such as sync and version management.

!!! warning

Do not push or upload content to a repository while it is being migrated. In-flight blob
uploads tied to the push repository can fail when the repository type changes.

!!! warning

Image that has been pulled from a registry and then subsequently pushed to another registy can lead to the blobs digest change.
Expand Down
6 changes: 6 additions & 0 deletions pulp_container/app/serializers.py
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,12 @@ class Meta:
model = models.ContainerPushRepository


class MigratePushRepositorySerializer(ValidateFieldsMixin, serializers.Serializer):
"""
Serializer for migrating a push repository to a container repository in place.
"""


class ContainerRemoteSerializer(RemoteSerializer):
"""
A Serializer for ContainerRemote.
Expand Down
1 change: 1 addition & 0 deletions pulp_container/app/tasks/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from .download_image_data import aadd_and_remove, download_image_data # noqa
from .builder import build_image_from_containerfile, build_image # noqa
from .migrate_push_repository import migrate_push_repository # noqa
from .recursive_add import recursive_add_content # noqa
from .recursive_remove import recursive_remove_content # noqa
from .sign import sign # noqa
Expand Down
133 changes: 133 additions & 0 deletions pulp_container/app/tasks/migrate_push_repository.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
from django.contrib.contenttypes.models import ContentType
from django.db import connection, transaction

from pulpcore.plugin.models import CreatedResource
from pulpcore.plugin.models.role import GroupRole, UserRole

from pulp_container.app.models import ContainerPushRepository, ContainerRepository
from pulp_container.app.serializers import ContainerRepositorySerializer


def migrate_push_repository(push_repository_pk):
"""
Convert a ContainerPushRepository into a ContainerRepository in place.

Swaps the multi-table-inheritance child row and updates `pulp_type` on the
parent repository so the primary key is preserved. Repository versions,
distributions, and other FKs to `core.Repository` remain valid.

Args:
push_repository_pk (str): The primary key for the push repository to migrate.
"""
with transaction.atomic():
with connection.cursor() as cursor:
# First statement in the transaction: protect related reads/writes
# (pending M2Ms, role GFKs) that are not covered by row locks alone.
cursor.execute("SET TRANSACTION ISOLATION LEVEL SERIALIZABLE")

# FOR UPDATE on the MTI join locks both the push child row and the parent
# core_repository row for the duration of this transaction.
push_repository = ContainerPushRepository.objects.select_for_update().get(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the select_for_update is supposed to protect here, but why not play it extra safe...
OTOH, does this "select for update" the base Repository row too?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about adding cursor.execute("set transaction isolation level serializable") as the very first statement in this transaction?

It could be the single one measure to protect all the reads happening here, including the GenericForeignKeys.

It may be a hit on performance, and could lead to SerializationErrors but for this kind of once in a lifetime migration operation it's probably a good tradeoff.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, select_for_update locks both parents and childs row. Added the SET transaction level.

pk=push_repository_pk
)
# Sanity check only: pulp_type should still be container-push
if push_repository.pulp_type != ContainerPushRepository.get_pulp_type():
raise RuntimeError(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a sanity check, since we expect the transition down below to happen in a transaction.
Maybe with a comment we can prevent future readers to panic.

f"Repository {push_repository_pk} has pulp_type "
f"{push_repository.pulp_type!r}, expected a push repository."
)
if ContainerRepository.objects.filter(pk=push_repository_pk).exists():
raise RuntimeError(
f"Container repository child row already exists for {push_repository_pk}."
)

signing_service_id = push_repository.manifest_signing_service_id

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one "normal" field we have to transfer from the child table, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct.

push_ct = ContentType.objects.get_for_model(ContainerPushRepository)
container_ct = ContentType.objects.get_for_model(ContainerRepository)

with connection.cursor() as cursor:
cursor.execute(
"""
UPDATE core_repository
SET pulp_type = %s
WHERE pulp_id = %s AND pulp_type = %s
""",
[
ContainerRepository.get_pulp_type(),
push_repository_pk,
ContainerPushRepository.get_pulp_type(),
],
)
if cursor.rowcount != 1:
raise RuntimeError(
f"Failed to update pulp_type for repository {push_repository_pk}."
)

cursor.execute(
"""
INSERT INTO container_containerrepository
(repository_ptr_id, manifest_signing_service_id)
VALUES (%s, %s)
""",
[push_repository_pk, signing_service_id],
)

cursor.execute(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hah, I guess it wouldn't be dramatic to loose the pending items accidentally, but yes, good call too.

Reminds me that "pending content" should probably be a pulpcore thing by now.

"""
INSERT INTO container_containerrepository_pending_blobs
(containerrepository_id, blob_id)
SELECT containerpushrepository_id, blob_id
FROM container_containerpushrepository_pending_blobs
WHERE containerpushrepository_id = %s
""",
[push_repository_pk],
)
cursor.execute(
"""
INSERT INTO container_containerrepository_pending_manifests
(containerrepository_id, manifest_id)
SELECT containerpushrepository_id, manifest_id
FROM container_containerpushrepository_pending_manifests
WHERE containerpushrepository_id = %s
""",
[push_repository_pk],
)

# Pending M2M FKs are ON DELETE NO ACTION at the DB level (not CASCADE),
# so remove them explicitly before deleting the push child.
cursor.execute(
"""
DELETE FROM container_containerpushrepository_pending_blobs
WHERE containerpushrepository_id = %s
""",
[push_repository_pk],
)
cursor.execute(
"""
DELETE FROM container_containerpushrepository_pending_manifests
WHERE containerpushrepository_id = %s
""",
[push_repository_pk],
)
cursor.execute(
"""
DELETE FROM container_containerpushrepository
WHERE repository_ptr_id = %s
""",
[push_repository_pk],
)

UserRole.objects.filter(content_type=push_ct, object_id=str(push_repository_pk)).update(
content_type=container_ct
)
GroupRole.objects.filter(content_type=push_ct, object_id=str(push_repository_pk)).update(
content_type=container_ct
)
Comment on lines +120 to +125

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call!


container_repository = ContainerRepository.objects.get(pk=push_repository_pk)

CreatedResource(content_object=container_repository).save()

return ContainerRepositorySerializer(
instance=container_repository, context={"request": None}
).data
36 changes: 35 additions & 1 deletion pulp_container/app/viewsets.py
Original file line number Diff line number Diff line change
Expand Up @@ -1135,7 +1135,7 @@ class ContainerPushRepositoryViewSet(
],
},
{
"action": ["update", "partial_update", "set_label", "unset_label"],
"action": ["update", "partial_update", "set_label", "unset_label", "migrate"],
"principal": "authenticated",
"effect": "allow",
"condition_expression": [
Expand All @@ -1154,6 +1154,40 @@ class ContainerPushRepositoryViewSet(
}
LOCKED_ROLES = {}

@extend_schema(
description=(
"Trigger an asynchronous task to convert this push repository into a "
"container repository in place. The repository primary key and version "
"history are preserved; only the repository type changes."
),
summary="Migrate push repository to container repository",
request=serializers.MigratePushRepositorySerializer,
responses={202: AsyncOperationResponseSerializer},
)
@action(
detail=True, methods=["post"], serializer_class=serializers.MigratePushRepositorySerializer
)
def migrate(self, request, pk):
"""
Create a task which converts a push repository into a container repository in place.
"""
repository = self.get_object()

serializer = serializers.MigratePushRepositorySerializer(
data=request.data, context={"request": request}
)
serializer.is_valid(raise_exception=True)

distributions = models.ContainerDistribution.objects.filter(repository=repository)
exclusive_resources = [repository, *distributions]

result = dispatch(
tasks.migrate_push_repository,
exclusive_resources=exclusive_resources,
kwargs={"push_repository_pk": str(repository.pk)},
)
return OperationPostponedResponse(result, request)

@extend_schema(
description=(
"Trigger an asynchronous task to remove a manifest and all its associated "
Expand Down
Loading
Loading