Skip to content

Commit 75f364d

Browse files
luciasanchezbella01nubtron
luciasanchezbella01
and
nubtron
authored
Package Size Analyzer (AI-5021) (#20128)
* basic status reporting * integration into ddev * clean commented code * add diff mode * add diff mode * final diff version * final diff version * Added timeline mode and uncompressed sizes (#5025) * Test ddev size status in GHA (by hijacking the slapr workflow) * Timeline mode * Try to fix job summaries * Try fixing the job summaries again * CI size status integration on master * types added * change --help * fix * fix typing * fix typing * fix types * fix types * fix typing * fix typing * fix typing * fix typing * fix tests for Windows * Fix tests * fix windows tests * Fix tests * Fix tests * fix windows tests * Final visualizations * Changelog * CI images integration and fixed typing * user errors and dep and int versions * user errors and dep and int versions * user errors and dep and int versions * fix timeline error * Adding version, and json and markdown formats * Fix test * simplify code * final fixes * fix ddev windows * fix lint * testing ddev tests on windows * test * test * test * test * test * test * test * test * test * test * test * test * fixes * test gha * test gha * test gha * fixes * fixes * fix * correct comments * correct comments * Change comments * Change number to constant * Fix a comment * Rerun checks * Fix versions --------- Co-authored-by: Enrico Donnici <enrico.donnici@datadoghq.com>
1 parent f135a6e commit 75f364d

18 files changed

+3694
-1
lines changed

‎.github/workflows/measure-disk-usage.yml

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
name: Measure Disk Usage
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
env:
8+
PYTHON_VERSION: "3.12"
9+
10+
jobs:
11+
measure-disk-usage:
12+
runs-on: ubuntu-22.04
13+
steps:
14+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
15+
with:
16+
fetch-depth: 0
17+
- name: Set up Python ${{ env.PYTHON_VERSION }}
18+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
19+
with:
20+
python-version: ${{ env.PYTHON_VERSION }}
21+
- name: Install ddev
22+
run: |
23+
pip install -e ./datadog_checks_dev[cli]
24+
pip install -e ./ddev
25+
26+
- name: Configure ddev
27+
run: |
28+
ddev config set repos.core .
29+
ddev config set repo core
30+
- name: Measure disk usage (uncompressed)
31+
run: |
32+
mkdir -p status_visualizations
33+
ddev size status --csv > size-uncompressed.csv
34+
ddev size status --save_to_png_path status_visualizations/uncompressed.png > size-uncompressed.txt
35+
cat size-uncompressed.txt
36+
echo "# Size (uncompressed)" >> $GITHUB_STEP_SUMMARY
37+
echo '```' >> $GITHUB_STEP_SUMMARY
38+
cat size-uncompressed.txt >> $GITHUB_STEP_SUMMARY
39+
echo '```' >> $GITHUB_STEP_SUMMARY
40+
41+
- name: Measure disk usage (compressed)
42+
run: |
43+
mkdir -p status_visualizations
44+
ddev size status --csv --compressed > size-compressed.csv
45+
ddev size status --compressed --save_to_png_path status_visualizations/compressed.png > size-compressed.txt
46+
cat size-compressed.txt
47+
echo "# Size (compressed)" >> $GITHUB_STEP_SUMMARY
48+
echo '```' >> $GITHUB_STEP_SUMMARY
49+
cat size-compressed.txt >> $GITHUB_STEP_SUMMARY
50+
echo '```' >> $GITHUB_STEP_SUMMARY
51+
52+
53+
- name: Measure disk usage differences from last commit (uncompressed)
54+
if: false # Disabled for now: size difference can be misleading due to dependencies not being built in the same PR
55+
run: |
56+
mkdir -p diff_visualizations
57+
BEFORE=$(git rev-parse HEAD^)
58+
AFTER=$(git rev-parse HEAD)
59+
ddev size diff $BEFORE $AFTER --csv > diff-uncompressed.csv
60+
ddev size diff $BEFORE $AFTER --save_to_png_path diff_visualizations/diff-uncompressed-linux.png > diff-uncompressed.txt
61+
cat diff-uncompressed.txt
62+
echo "# Size diff (uncompressed)" >> $GITHUB_STEP_SUMMARY
63+
echo '```' >> $GITHUB_STEP_SUMMARY
64+
cat diff-uncompressed.txt >> $GITHUB_STEP_SUMMARY
65+
echo '```' >> $GITHUB_STEP_SUMMARY
66+
67+
- name: Measure disk usage differences from last commit (compressed)
68+
if: false # Disabled for now: size difference can be misleading due to dependencies not being built in the same PR
69+
run: |
70+
mkdir -p diff_visualizations
71+
BEFORE=$(git rev-parse HEAD^)
72+
AFTER=$(git rev-parse HEAD)
73+
ddev size diff $BEFORE $AFTER --compressed --csv > diff-compressed.csv
74+
ddev size diff $BEFORE $AFTER --compressed --save_to_png_path diff_visualizations/diff-compressed-linux.png > diff-compressed.txt
75+
cat diff-compressed.txt
76+
echo "# Size diff (compressed)" >> $GITHUB_STEP_SUMMARY
77+
echo '```' >> $GITHUB_STEP_SUMMARY
78+
cat diff-compressed.txt >> $GITHUB_STEP_SUMMARY
79+
echo '```' >> $GITHUB_STEP_SUMMARY
80+
81+
- name: Upload file sizes (uncompressed)
82+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
83+
with:
84+
name: size-uncompressed.csv
85+
path: size-uncompressed.csv
86+
if-no-files-found: error
87+
88+
- name: Upload file sizes (compressed)
89+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
90+
with:
91+
name: size-compressed.csv
92+
path: size-compressed.csv
93+
if-no-files-found: error
94+
95+
- name: Upload file sizes diff (uncompressed)
96+
if: false # Disabled for now: size difference can be misleading due to dependencies not being built in the same PR
97+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
98+
with:
99+
name: diff-uncompressed.csv
100+
path: diff-uncompressed.csv
101+
if-no-files-found: error
102+
103+
- name: Upload file sizes diff (compressed)
104+
if: false # Disabled for now: size difference can be misleading due to dependencies not being built in the same PR
105+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
106+
with:
107+
name: diff-compressed.csv
108+
path: diff-compressed.csv
109+
if-no-files-found: error
110+
111+
- name: Upload status PNGs
112+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
113+
with:
114+
name: size-visuals
115+
path: status_visualizations/
116+
if-no-files-found: error
117+
118+
- name: Upload diff PNGs
119+
if: false # Disabled for now: size difference can be misleading due to dependencies not being built in the same PR
120+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
121+
with:
122+
name: diff-visuals
123+
path: diff_visualizations/
124+
if-no-files-found: error
125+

‎.github/workflows/slapr.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,4 +27,4 @@ jobs:
2727
SLAPR_EMOJI_APPROVED: "approved2"
2828
SLAPR_EMOJI_CHANGES_REQUESTED: "changes_requested"
2929
SLAPR_EMOJI_MERGED: "merged"
30-
SLAPR_EMOJI_CLOSED: "closed"
30+
SLAPR_EMOJI_CLOSED: "closed"

‎ddev/changelog.d/20128.added

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Added new commands to track and analyze size changes in integrations and dependencies:
2+
- **`ddev size status`**: Shows current sizes of all modules.
3+
- **`ddev size diff [COMMIT_BEFORE] [COMMIT_AFTER]`**: Compares size changes between two commits.
4+
- **`ddev size timeline {integration | dependency} [INTEGRATION_NAME/DEPENDENCY_NAME]`**: Visualizes the size evolution of a module over time.

‎ddev/pyproject.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ dependencies = [
4040
"tomli-w",
4141
"tomlkit",
4242
"tqdm",
43+
"requests",
44+
"matplotlib",
45+
"squarify"
4346
]
4447
dynamic = ["version"]
4548

‎ddev/src/ddev/cli/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
from ddev.cli.env import env
1919
from ddev.cli.meta import meta
2020
from ddev.cli.release import release
21+
from ddev.cli.size import size
2122
from ddev.cli.status import status
2223
from ddev.cli.test import test
2324
from ddev.cli.validate import validate
@@ -149,6 +150,7 @@ def ddev(
149150
ddev.add_command(status)
150151
ddev.add_command(test)
151152
ddev.add_command(validate)
153+
ddev.add_command(size)
152154

153155
__management_command = os.environ.get('PYAPP_COMMAND_NAME', '')
154156
if __management_command:

‎ddev/src/ddev/cli/size/__init__.py

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# (C) Datadog, Inc. 2022-present
2+
# All rights reserved
3+
# Licensed under a 3-clause BSD style license (see LICENSE)
4+
5+
import click
6+
7+
from ddev.cli.size.diff import diff
8+
from ddev.cli.size.status import status
9+
from ddev.cli.size.timeline import timeline
10+
11+
12+
@click.group()
13+
def size():
14+
"""
15+
Analyze the download size of integrations and dependencies in various modes.
16+
17+
This command provides tools to inspect the current status, compare commits and monitor size changes of modules
18+
across different commits, platforms, and Python versions.
19+
20+
"""
21+
22+
pass
23+
24+
25+
size.add_command(status)
26+
size.add_command(diff)
27+
size.add_command(timeline)
28+
29+
if __name__ == "__main__":
30+
size()

0 commit comments

Comments
 (0)