Benchmarking setup #2366

2025-06-14T00:44:12+08:00

zachgoll commented

2025-06-14 00:44:12 +08:00

(Migrated from github.com)

This PR sets up derailed_benchmarks so that we can run benchmarks locally against various endpoints and get a nicely formatted report, by-commit printed to /tmp/[endpoint].txt. For example, a benchmark run against /budgets/jun-2025 for an authenticated demo user using the "default" data from rake demo_data:default:

======================================================================
📊 BENCHMARK RUN - 2025-06-14 15:16:55 (d5caf922)
======================================================================

| Type | IPS | Deviation | Time/Iteration | Iterations | Total Time |
|------|-----|-----------|----------------|------------|------------|
| COLD | 0.607 | ± 0.00%     | 1.65 s/i       | 1.000      | 1.647418s  |
| WARM | 0.778 | ± 0.00%     | 1.28 s/i       | 24.000     | 31.059852s |

Note about setup: Benchmarking results may vary based on the data in the DB. The best way to benchmark is setting up a production-mirror with anonymized / scrubbed data, which is a lengthy process that we cannot provide in an OSS context. The best alternative we have is synthetic demo data, which can be generated via rake demo_data:default. Benchmarks can still be run on this small demo family, but time/iteration will be much smaller and performance issues caused by inefficient queries running on large datasets will not show up in the results.

This PR sets up `derailed_benchmarks` so that we can run benchmarks locally against various endpoints and get a nicely formatted report, by-commit printed to `/tmp/[endpoint].txt`. For example, a benchmark run against `/budgets/jun-2025` for an authenticated demo user using the "default" data from `rake demo_data:default`: ```md ====================================================================== 📊 BENCHMARK RUN - 2025-06-14 15:16:55 (d5caf922) ====================================================================== | Type | IPS | Deviation | Time/Iteration | Iterations | Total Time | |------|-----|-----------|----------------|------------|------------| | COLD | 0.607 | ± 0.00% | 1.65 s/i | 1.000 | 1.647418s | | WARM | 0.778 | ± 0.00% | 1.28 s/i | 24.000 | 31.059852s | ``` **Note about setup**: Benchmarking results may vary based on the data in the DB. The best way to benchmark is setting up a production-mirror with anonymized / scrubbed data, which is a lengthy process that we cannot provide in an OSS context. The best alternative we have is synthetic demo data, which can be generated via `rake demo_data:default`. Benchmarks can still be run on this small demo family, but time/iteration will be much smaller and performance issues caused by inefficient queries running on large datasets will not show up in the results.

cursor[bot] (Migrated from github.com) reviewed 2025-06-14 23:34:34 +08:00

cursor[bot] (Migrated from github.com) left a comment

Bug: Removed Method Causes Scenario Error

The generate_multi_currency_data! method attempts to call generate_for_scenario(:multi_currency, family_names). This method, along with the entire scenario-based architecture, was removed during the refactoring of the Demo::Generator class, leading to a NoMethodError when generate_multi_currency_data! is invoked.

app/models/demo/generator.rb#L114-L120

4fe41d44b2/app/models/demo/generator.rb (L114-L120)

Fix in Cursor

Was this report helpful? Give feedback by reacting with 👍 or 👎

<details open> <summary><h3>Bug: Removed Method Causes Scenario Error</h3></summary> The `generate_multi_currency_data!` method attempts to call `generate_for_scenario(:multi_currency, family_names)`. This method, along with the entire scenario-based architecture, was removed during the refactoring of the `Demo::Generator` class, leading to a `NoMethodError` when `generate_multi_currency_data!` is invoked. <p></p> <details> <summary><code>app/models/demo/generator.rb#L114-L120</code></summary> https://github.com/maybe-finance/maybe/blob/4fe41d44b21574812f9ef4d62844004ec67fb84f/app/models/demo/generator.rb#L114-L120 </details> <a href="https://cursor.com/open?data=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImJ1Z2JvdC12MSJ9.eyJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7ImJ1ZyI6ImV5SnNiMk5oZEdsdmJuTWlPbHQ3SW1acGJHVWlPaUpoY0hBdmJXOWtaV3h6TDJSbGJXOHZaMlZ1WlhKaGRHOXlMbkppSWl3aWMzUmhjblJNYVc1bElqb3hNVFFzSW1WdVpFeHBibVVpT2pFeU1Dd2lZMjlrWlV4cGJtVnpJanBiWFgxZExDSnBaQ0k2SWpaalpESTNaREkyTFdWbFpEZ3ROREEwTlMwNU4yRTJMV0U0WldVMU5XSmpPVEptTUNJc0ltUmxjMk55YVhCMGFXOXVJam9pVkdobElHQm5aVzVsY21GMFpWOXRkV3gwYVY5amRYSnlaVzVqZVY5a1lYUmhJV0FnYldWMGFHOWtJR0YwZEdWdGNIUnpJSFJ2SUdOaGJHd2dZR2RsYm1WeVlYUmxYMlp2Y2w5elkyVnVZWEpwYnlnNmJYVnNkR2xmWTNWeWNtVnVZM2tzSUdaaGJXbHNlVjl1WVcxbGN5bGdMaUJVYUdseklHMWxkR2h2WkN3Z1lXeHZibWNnZDJsMGFDQjBhR1VnWlc1MGFYSmxJSE5qWlc1aGNtbHZMV0poYzJWa0lHRnlZMmhwZEdWamRIVnlaU3dnZDJGeklISmxiVzkyWldRZ1pIVnlhVzVuSUhSb1pTQnlaV1poWTNSdmNtbHVaeUJ2WmlCMGFHVWdZRVJsYlc4Nk9rZGxibVZ5WVhSdmNtQWdZMnhoYzNNc0lHeGxZV1JwYm1jZ2RHOGdZU0JnVG05TlpYUm9iMlJGY25KdmNtQWdkMmhsYmlCZ1oyVnVaWEpoZEdWZmJYVnNkR2xmWTNWeWNtVnVZM2xmWkdGMFlTRmdJR2x6SUdsdWRtOXJaV1F1SWl3aVkyOXVabWxrWlc1alpTSTZNQzQyTXpBMU1UazJPVGszTXpZMU9URTJMQ0pqWVhSbFoyOXllU0k2SWt4UFIwbERYMEpWUnlKOSIsImJyYW5jaCI6InphY2hnb2xsL2JlbmNobWFya2luZy1zZXR1cC12MiJ9LCJpYXQiOjE3NDk5MTUyNzQsImV4cCI6MTc1MDUyMDA3NH0.g59Vq5p7fb6uK7Ro4xqV2BX9ltCe3CckISS9mSPFjl9f5yKxoXNu5Bfxjt_vdgAcqMJlh3Pe4VWdh5j8yW5Aplx5iCoGrEz1MNs_GqSTU3XnSbmS7TuA6Fg7x4ZoMeDI09eZDHFMCLUg2q3HWHcJhNu5YRQncFrytCwBZkbPeET0IvCSqOSnNcc9E1WQlh-2yHQICMbA_rO1juEU4OAXQI1sDAA6i5j6faRDqf6r6KEB5jXgqUAyS-iYvfTMas3srRchhwZ7Mgx4NX_EoTTlfbwGGHe_ETB_Qfxpu93Dl2iP-pii-i-wZ9jSQ2QIFojGZAOtyK_8aMax4VeDH5DYmw">Fix in Cursor</a> </details> --- _Was this report helpful? Give feedback by reacting with 👍 or 👎_

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: gavin/maybe#2366