Skip to content

Conversation

@kairoaraujo
Copy link
Member

@kairoaraujo kairoaraujo commented Oct 14, 2024

Improve bootstrap performance for hash bins (large number such as 16384 PyPI/RubyGems)

Before : Added 2048 hash bins in 119.36732602119446 seconds
Current: Added 2048 hash bins in 1.8565280437469482 seconds

PyPI (PEP 458) and RubyGems size creation:
Added 16384 hash bins in 15.13554310798645 seconds

Fix query to bump empty target files

the actual crud query fails if there is not target files (artifacts) in
the database (TargetFiles) and do not return the the Target Role.

This commmit fixes this query using isouter

https://docs.sqlalchemy.org/en/20/orm/queryguide/query.html#sqlalchemy.orm.Query.join.params.isouter

Refactor: update_snapshot.

Split update_snapshot in a functional way for:

  • updating targets roles for hashbin delegated roles
  • updating targets roles for custom delegated roles

TODO:

Improve the performance and use the celery multi-threading

   repository-service-tuf-worker-1  | [2024-10-14 06:00:00,132: DEBUG/ForkPoolWorker-9] PRE_RUN: bump_online_roles
   repository-service-tuf-worker-1  | [2024-10-14 06:00:00,417: DEBUG/ForkPoolWorker-9] Checking for expired hash bin roles (16384 bins)
   repository-service-tuf-worker-1  | [2024-10-14 06:01:26,330: DEBUG/ForkPoolWorker-9] Total 16384 expired roles in 85.92996311187744s
   repository-service-tuf-worker-1  | [2024-10-14 06:04:07,217: INFO/ForkPoolWorker-9] Task app.repository_service_tuf_worker[bump_online_roles] succeeded in 247.09811807099322s: True
  • is_expired: ~90s
  • _update_targetfiles_hashbin_delegated_roles: ~250s
  • _update_targetfiles_custom_delegated_roles (Nice to Have)

Kairo Araujo and others added 6 commits October 11, 2024 12:10
Before : Added 2048 hash bins in 119.36732602119446 seconds
Current: Added 2048 hash bins in 1.8565280437469482 seconds

PyPI (PEP 458) and RubyGems size creation:

Added 16384 hash bins in 15.13554310798645 seconds

Signed-off-by: Kairo Araujo <kairo.araujo@testifysec.com>
the actual crud query fails if there is not target files (artifacts) in
the database (TargetFiles) and do not return the the Target Role.

This commmit fixes this query using isouter

https://docs.sqlalchemy.org/en/20/orm/queryguide/query.html#sqlalchemy.orm.Query.join.params.isouter

Signed-off-by: Kairo Araujo <kairo.araujo@testifysec.com>
Refactor the `update_snapshot`.

Split `update_snapshot` in a functional way for:
  - updating targets roles for hashbin delegated roles
  - updating targets roles for custom delegated roles

TODO:
Improve the performance and use the celery multi-threading
  - `is_expired`
  - `_update_targetfiles_hashbin_delegated_roles`
  - `_update_targetfiles_custom_delegated_roles` (Nice to Have)

Signed-off-by: Kairo Araujo <kairo.araujo@testifysec.com>
Code works, but requires refactoring and structure

Signed-off-by: Kairo Araujo <kairo.araujo@testifysec.com>
Signed-off-by: Kairo Araujo <kairo.araujo@testifysec.com>
Signed-off-by: Kairo de Araujo <kairo@dearaujo.nl>
@kairoaraujo kairoaraujo force-pushed the perform-use-multithread-writing-files branch from 18658b9 to 11e49c6 Compare October 20, 2024 09:11
Signed-off-by: Kairo de Araujo <kairo@dearaujo.nl>
@kairoaraujo kairoaraujo force-pushed the perform-use-multithread-writing-files branch from 11e49c6 to 78b9cb6 Compare October 20, 2024 09:21
Signed-off-by: Kairo de Araujo <kairo@dearaujo.nl>
@kairoaraujo kairoaraujo force-pushed the perform-use-multithread-writing-files branch from d913b14 to 3059daf Compare October 23, 2024 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant