Twenty surfaces, measured against live production

Executive summary

The platform is stable and getting busier. What follows is not a list of outages — it is a set of places where the system is working exactly as configured, and the configuration is not what anyone intended.

Where it is strong

Where it is not

Two revenue-assurance findings that no error dashboard can see. 31 of 233 merchants labelled messaging-only are running loyalty mechanics — Fire and Flower awarded 66.6M points in 90 days — because platform_type gates four call sites in the entire application, none of them on the API or the loyalty path. Separately, 79 billing organisations sent 26.8M messages in 90 days against $897k of collectible unpaid invoices — enterprise accounts billed off platform are excluded from that figure.

Observability remains configured to hide its own failures. New Relic records zero error events across 345,845 HTTP 500s. Rollbar discards RecordNotFound outright and files ActionView::Template::Error as a warning on the strength of a comment that is no longer true. The wallet project — a live consumer surface serving 1.98M requests a week — reports nothing at all.

The highest-volume failure is still unattributable, but the class around it is not. The transaction-date defect held at 23,112–27,550/day, unchanged across both measurements and still logged without merchant or POS identity. But duplicate-member errors — now the platform's largest error class at 24,968/day — are attributable today via enrollment_change_source, which had been assumed impossible.

The public API contract is narrower than a raw endpoint count suggests, and sharper within its real scope. The documentation was enumerated completely. pos and general are the two gateways public by design; across their 93 endpoints only 23 are documented — 24.7%. 35 undocumented endpoints mutate data — twelve delete records or move member balances, including DELETE /members/{} and PUT /members/{}/reset_balance. Every one of the 24 API reference pages declares a non-production host, with a live "Try It" explorer pointed at it.

One partner is generating 350,000 rejected requests a day, and is now named. A single API key — lmlp-leaflogix, issued 2022 and never rotated — accounts for 100% of missing-header rejections on the busiest POS endpoint, 29.3% of that partner's own calls to it. Seven other partners transact on the same endpoint with zero such failures. The traffic is rejected at the gateway, so it never reached an application metric and nobody reported it in over thirty days.

Two safety nets are not catching anything. A dead-letter queue holds 722 undelivered POS customer-sync messages that nothing has ever consumed, its oldest message pinned at the four-day retention cap — so failures are being destroyed on a rolling basis. All 87 queues, and all 38 dead-letter queues, sit below the fourteen-day maximum. And there is not a single CloudWatch alarm on any SQS queue, which is the complete explanation for why nobody knew. Separately, 31 alarms have received no data at all, 25 of them for between 165 and 1,274 days.

Webhooks are noisier in reputation than in fact, and blinder than expected. Firing is guarded at three levels and volume is proportionate to business activity — the apparent excess is POS location fanout, and 73% of the delivery lambda's traffic turned out to be inbound callbacks rather than outbound webhooks. But 0 of 816 endpoints can receive a change-set: the dirty state is discarded before the payload is built, so every receiver must blind-write the full record. That is the mechanism behind the known full-object clobber.

One recurring defect produced two fresh instances during the gap. Two audiences fail daily on date_trunc('1', …) — the exact misconfiguration diagnosed on 2026-08-08. The shape is still writable, so it recurs roughly every three to six days.

What to do first

The web tier

The strongest surface measured, and the one whose cost is most concentrated.

The 4xx are structural, not incidental

404 and 401 together are 14.35% of all traffic, and neither is a defect. The 404s are overwhelmingly the POS lookup-then-create protocol: a partner asks whether a member exists, is told no, and creates one. The 401s are dominated by two endpoints known to be broken since the August endpoint census — api/pos/v1/batches/create (829,971 in 7 days) and api/wallet/auth/v1/remember_me/authenticate (320,896) — plus routine unauthenticated probing.

Where the compute actually goes

The API contract

The deployed gateway and the Rails application still disagree, and the gap is wide, and one long-standing assumption about it turns out to be false.

Of the 369 deployed endpoints, 305 proxy to Rails and 64 resolve to Lambda, SQS or mock integrations. Of those 305, 276 match a Rails route and 29 do not — they are deployed, reachable, and land on nothing.

Destructive endpoints are still deployed and undocumented

DELETE /members/{} and PUT /members/{}/reset_balance both remain live on the pos gateway as proxies to Rails, and both have matching Rails routes. Both require an API key. That key is authentication, not authorisation: it establishes that the caller is a valid partner and then grants the whole stage. All 93 endpoints on the two public gateways carry authType = NONE — no Lambda authoriser, no IAM, no per-method scope — and none of the six usage plans throttles per method. A destructive endpoint is configured identically to a read endpoint; the only thing separating them is knowing the path.

The published documentation

The published documentation was enumerated completely — 48 pages, all HTTP 200, via the readme.io site manifest and its embedded OpenAPI fragments. Every claim below comes from the published source text. The headline earlier estimate is overturned — not because it was too harsh, but because it was too generous.

The docs were enumerated completely rather than sampled: readme.io publishes a machine-readable site manifest at llms.txt, and each reference page embeds a full OpenAPI 3.1 fragment. All 48 pages returned HTTP 200 — no login wall, no rate limiting. One page (docs/template-api) is absent from the manifest and reachable only by following an inbound link.

The host claim, refuted in the wrong direction

Scope: two gateways are public by design

pos and general are the externally documented gateways. The other ten are consumed by internal services — native app, wallet, brands, callbacks, tablet, join, reports, netsuite, analytics and shortlinks — and are either active internal surfaces or superseded. They are not expected to carry public documentation, and their absence from readme.io is correct, not a gap.

API Key ID: wd6n3j71og
HTTP Method: GET, Resource Path: /pos/v1/members/38685148
Method request headers: {x-datadog-sampling-priority=1, CloudFront-Viewer-Country=US,
  x-datadog-origin=rum, CloudFront-Forwarded-Proto=https, User-Agent=Amazon CloudFront, ...}
Request parameter validation failed. Missing parameters: [AUTH-TOKEN]
Method completed with status: 400
Usage Plan check succeeded for API Key **********************************mmcaK9

The gateway estate

Twelve APIs, unchanged in a year of drift, still almost entirely uninstrumented — but the traffic they reject has fallen by nearly half.

The identifier map is unchanged since early August — no API added, none removed — and the orphaned h2dn0titad log group still exists with no API behind it. Only 2 of 12 have access logging, and metricsEnabled is false on all 12, so no per-route metric or alarm exists anywhere in the estate. That is unchanged.

Requests rejected before they reach Rails

Analytics rejections are entirely POST /v1/native_app/events with an API key not associated with a usage plan. The pos rejections are all Missing parameters: [AUTH-TOKEN]. Callbacks has effectively stopped rejecting — two rejections in a full 24-hour window, against 447 a day when last measured.

Where POS writes actually go

Three of the 41 pos endpoints bypass the monolith entirely:

Unchanged, along with the two general endpoints that resolve to SQS (auto_campaigns/trigger and message_template_logs). A further 17 callbacks endpoints route to the same Lambda and 12 reports endpoints to an Athena handler.

Background jobs

The job fleet is reliable. The measurement of it needed correcting, and the corrected figures are below.

Success rate 99.955%, roughly 6.36M job starts a day, and the books balance (start − done − fail = −3). -secondary carries 92% of all jobs, which confirms the inverted environment naming: the environment that serves almost no web traffic runs nearly all the background work.

Per-worker attribution is still impossible

Fail lines read, verbatim:

I, [2026-08-16T21:58:50.223153 #51824]  INFO -- : fail

No class, no JID, no elapsed time — only a PID. Per-worker execution counts, success rates and durations remain unobtainable from this source. The fix is written and waiting on the observability-quick-fixes branch, which sets a JSON formatter and level: :info; the latter also removes the ~92% DEBUG-SQL volume that makes a 30-day retention unaffordable today.

The transaction-date defect — and a class that is no longer blind

The platform's most persistent error is unchanged. But the error class around it turns out to be attributable today, which narrows the biggest standing recommendation considerably.

Steady since early August

Still logged as a bare string with no merchant, no location and no POS type:

[merchant_location] Validation failed: Transaction date can not be greater than created date

The rescue site (pos_merchant_location_concern.rb, ~line 175) logs only the exception object and is annotated # do nothing. The top standing recommendation — add merchant_id and pos_type here — remains unimplemented.

The error league table has changed at the top

Duplicate-email failures have tripled and are now the platform's largest single error class.

Swallowing rescues

The code census was run as an AST walk rather than a text search, so the counts are exact. What is new is the discovery that some of these rescues do not merely lose a log line — they can persist wrong data.

The census, re-measured

Re-run as a Ruby Ripper.sexp AST walk over all 2,350 .rb files in app/ and lib/, rather than a text search. Zero parse failures.

A naive text search returns 644. The single difference is a commented-out logger call in headset_integration/location_service.rb:43, which the AST correctly excludes. Two directory-level movements since 4 August are both explained by git history: one worker logging call was removed in the Lightspeed token-refresh refactor, and one was added in the new enroll/home_controller.rb.

Swallowing rescues

The structural consequence, now counted

When a rescue swallows inside a Sidekiq perform body, the job records as succeeded — no retry, no dead-letter queue, no alert. The subset can be counted precisely. 42 of the 208 swallowing sites sit directly inside a perform method, across 33 worker files, with zero ambiguous attributions.

The transaction-date rescue, quoted

The site behind the platform's most persistent error, verbatim:

    begin
      visit.save!
      if visit.persisted? && pos.try(:visit_details_included?, o)
        visit.visit_details = pos.create_visit_details(visit, o.dig("line_items"))
      end
    rescue ActiveRecord::RecordInvalid => e
      logger.tagged("merchant_location") do
        logger.info "trying to save a visit"
        logger.error e
      end
      # do nothing
      # its just sidekiq trying to save the same visit in multiple threads

It still swallows, and it logs no merchant id, no location, no pos type and no order identifier — only a literal tag and the exception. That is why ~25,000 failures a day cannot be attributed to anyone.

Swallowed exceptions that corrupt data

Twelve of the 208 sites sit in app/models/member_profile.rb. They look like safe retry loops and are not — the retry is guarded, so when the counter is exhausted control falls off the end of the rescue and the exception is absorbed:

      member_profile.save!
    rescue ActiveRecord::StatementInvalid => e
      logger.tagged('member_profile') do
        logger.error 'Trying to create a member_profile'
        logger.error e
      end
      if (retries -= 1).positive?
        sleep rand(0.1..0.8) # to avoid db lock due to sidekiq race conditions
        retry
      end

save! is the bang form, chosen precisely so it raises on failure. On the terminal path that raise is absorbed, the method returns normally, and the caller cannot distinguish "profile created" from "profile never persisted." The condition this code was written for — lock contention between concurrent Sidekiq jobs, per its own comment — is exactly the condition under which every retry fails and the loss is silent.

POS vendor health

Strong across the board, with one location that has quietly gone dark.

Roughly 341,000 visits a day across 36 active vendors. Every vendor with meaningful volume delivered on the last full day measured; none stopped.

One location has quietly gone dark

cova location 26112 (Alpha Cannabis, merchant 11637) logged 17,795 failed 401s in seven days while visits landed normally, and was deliberately excluded from the 8 August cleanup on that basis.

Re-measured: zero visits since 2026-05-20, roughly three months, while the location remains enabled and actively scheduled for polling. The errors that were correctly dismissed now sit alongside no ingestion at all.

Two vendor-collapse alerts, both false

• teamworks (−89.6%) — merchant 36627 loaded 463,769 backfilled rows on 4 August whose transaction_date values span June and July. Measured by transaction_date, daily activity is flat.
• import (−52.9%) — a manual bulk-load channel that fires on isolated days. Its mean is meaningless, and it is not a vendor.

Merchant status and unprocessed churn

lost 3,863 · active 732 · pending 158 · suspended 56 · onboarding 12. Stable. Lost outnumbers active better than five to one, so any platform-wide error count that does not filter on merchant status is dominated by dead accounts.

129 billing-active merchants have sent zero visits in 60 days (245 silent, less the 116 that never transacted); roughly 30 have been silent for over two years. Four dormant accounts flagged earlier have not recovered: merchants 10370 (last visit 2024-12-16), 13977 (2025-02-01), 12143 (2026-01-09) and 17351 (2026-04-01).

One cluster is worth a targeted check: four dutchiepos merchants — The Vault Spokane (2066), Silvana (2358), Lake Stevens (2853) and Herbn Elements (3650) — went silent on exactly 2026-05-21, each retaining one enabled location. dutchiepos overall is healthy, so this is per-account rather than a vendor problem.

Partners: is it them or us?

This is answerable for one of the two largest error classes and not the other. One of the two can now be answered — and the answer points at us more than at them.

The question matters commercially: when a POS integration produces hundreds of thousands of errors a week, the response differs entirely depending on whether the partner is sending bad data or springbig is mishandling good data. That reading concluded honestly that an honest split was not computable. That is now half true.

What can be attributed today

Duplicate-member failures — the platform's largest error class at 24,968 a day — carry opt_event_data.enrollment_change_source on the retry line. Over a four-hour window they resolve cleanly:

But attribution is not the same as fault

A duplicate-member error means the platform tried to create a member who already exists. That can be the partner re-sending, or it can be springbig's own matching logic failing to recognise an existing member. The member-identity code path is the one carrying twelve swallowing rescues that null out failed fields and retry — so a member record whose email or mobile was silently dropped on a previous write will not match on the next one, and will present as a duplicate.

What still cannot be attributed

The transaction-date defect — roughly 25,000 failures a day — remains genuinely blind. Its rescue site logs a literal tag and an exception object with no merchant, location, POS type or order identifier. It is now the only major error class in that state, which makes the instrumentation recommendation both smaller in scope and harder to defer.

APM: what New Relic sees, and what it is configured not to

Every transaction is instrumented. No error is. This is a settings problem, not a tooling problem, and it is the cheapest item on the list.

Compute grew about 30% while request volume grew about 7%. Work per request is rising faster than traffic — not yet a problem, but the trend is worth watching, and api/pos/v1/members/index at 338 hours a week is where it concentrates.

Error reporting

Rollbar's active item count more than doubled in twelve days. That is not necessarily more breakage — but nobody is triaging it, and the highest-volume items carry no stack trace.

84 of the 166 rails items (51%) are new since 4 August, and 118 of 166 (71%) fired within the last three days. This is a live surface, not a stale backlog. The new items are numerous but individually low-volume — a broad front rather than one runaway error.

The wallet project's silence is now proven rather than assumed: its token authenticates normally and returns zero for active, resolved and muted alike. A live consumer surface serving 1.98M requests a week reports nothing at all. The frontend project shows 0 resolved and 0 muted against 115 active — nobody is working it.

Suppression is in code, not in the console

config/initializers/rollbar.rb discards or downgrades whole exception classes:

'ActiveRecord::RecordNotFound' => 'ignore',
'ActionController::RoutingError' => 'ignore',
'ActionView::Template::Error' => 'warning', # this is just CS people / ActiveAdmin

Both rules are individually defensible and collectively blinding. The RecordNotFound rule cannot distinguish a bot probing a URL from a member whose link stopped working — both raise the same class and both are dropped. The ActionView::Template::Error comment is an assumption that has outlived its context: the same class now fires on customer-facing enrol pages and on the API. The Reward 48176 deletion produced 194 occurrences of it between 13 and 14 August, all filed as warnings, resolving to app/views/api/web/v1/extension_rewards/_reward.jbuilder:1.

New clusters since 4 August

• PG::InvalidTextRepresentation — 1,355 occurrences in a tight burst on 10 August that started and stopped the same day.
• PG::QueryCanceled and ConnectionNotEstablished — ~220 occurrences 12–14 August against a single host, the known Aurora replica-conflict signature.
• ActionView::Template::Error — ~194 occurrences from 13 August, the Reward 48176 deletion, still filed as warnings.
• Frontend: vue-query hooks can only be used inside setup() accounts for 397 of 464 new occurrences.

Native apps, and why frontend errors are unreadable

The sharpest earlier claim on this surface is confirmed emphatically. Two earlier counts are not reproducible from the underlying data and are withdrawn. And one dataset previously recorded as unavailable turns out to hold 410 million rows.

Not one readable stack trace

The top 15 frontend items account for 264,516 occurrences — 99.2% of all active frontend volume. One instance was fetched from each and every frame examined.

Every frame resolves to a content-hashed bundle and a mangled identifier:

assets/index-mm3f9i1-.js :28:1996  [XMLHttpRequest.S]
assets/index.afe282a1.js :16:1994  [RB]
assets/index-BjBsRhPE.js :14:1401  [Y4]

Three independent signals of source-map resolution were checked across all frames — .vue/.ts extensions, webpack:/ or /src/ paths, and Rollbar's code field, which populates only when a map resolves. Zero hits on all three. Rollbar is receiving raw minified frames and resolving nothing.

This is not a noise problem

Classification was done on stack-frame filenames rather than titles — extension protocol prefixes and known third-party CDN hosts — so it does not depend on error text. Of the top 45 items by volume, 35 originate on springbig.io itself and none on an extension or a third party. The inbox is almost entirely first-party signal that cannot be read.

Two counts that cannot be reproduced

Figures of 198 mobile app projects and 318 cloud projects have circulated internally. Neither is reproducible from the underlying data.

One related figure is reconciled rather than disputed: 568 active non-demo merchants sit under an organisation with a native app; the 594 quoted elsewhere in this session is the same measure including 26 demo accounts.

Mobile is instrumented for behaviour and not at all for failure

Mobile telemetry has been described internally as not retained anywhere queryable. It is: springbig_production_logs.native_app_events holds 410,602,054 rows spanning January 2023 to today, live. It appears in Postgres as a foreign table and is unreadable there, but queries directly against Redshift.

The wallet reports nothing, and it is a live consumer surface

The wallet Rollbar project is enabled, its token is valid, and it returns zero items in every status — active, resolved, muted and suspended alike. It has eight registered environments including production, and Rollbar creates an environment only when events arrive, so it reported at some point and stopped.

The edge

A full sweep of all 1,799 zones — not a sample — gives the first complete picture of edge traffic. Nearly a quarter of it is a 4xx, and the plan tier prevents us from saying why.

The 4xx figure is dominated by 3,653,786 403s — 20.02% of all edge traffic. That should not be read as 3.65 million security events: a 403 at the edge is frequently a member-not-found rather than a genuine authorization failure. Distinguishing the two requires firewall event data, which is denied on the Free plan — re-tested at both zone and account level, still code=authz.

Cache remains effectively unused at 4.15% of requests, against a 59.3% byte hit rate — meaning a few large static assets cache well while nearly all dynamic and API traffic does not. Workers remain the well-instrumented layer at a 0.015% error rate, with two exceptions: wallet-3-testing-amplify fails 100% of its 85 invocations and wallet-3-testing-api 69.6% of 125. Volume is negligible, so these read as broken test workers rather than customer impact — but they fail silently.

Several low-volume zones fail on nearly every request and nobody is watching them: springbig.ai (99.3% 4xx), sbtest101.com (99.4% 5xx), stash-board.com (88.3% 5xx), springbig-wallet-alpha.click (72.1% 5xx).

The lambda estate and its alarms

The estate is healthy on invocations. What matters here is quieter: a large fraction of the alarm estate is not protecting anything, and one whole service has no alarms at all.

Error rates, measured as ERROR lines per invocation

Ratios exceed 100% because batch handlers emit many error lines per invocation — this is a lines-per-invocation measure, not a failure rate, and it is not comparable to the earlier 58.7%. analytics_process_incoming_events remains rank one with the identical error string quoted earlier.

The alarm estate

The structural point — CloudWatch notifies on state transition, so a stuck ALARM is invisible — remains architecturally true. But the specific instance it cited is not reproducible today: only one alarm is in ALARM and it tripped eleven minutes before measurement.

Provenance

Only 2 of 49 functions run a deprecated runtime (both python3.9, both silent CDK helpers), and seven have been unmodified for over a year — including test_lambda, a test function deployed to production, silent for 370 days and still provisioned.

The "one repo hosts four lambdas" finding is confirmed: the pos-request-service repository deploys pos_request_runner, pos_cache_runner, pos_response_runner and pos_service_request as four separate CloudFormation stacks. Nothing in the function list reveals this — only the stack-name prefix does.

Dead-letter queues

This was called "a delete timer wearing a safety-net label" and found one instance. It is not one instance. It is every queue in the account.

Every dead-letter queue in this account is a four-day delete timer. Not one queue uses the maximum retention, and the nineteen send-request queues at one day are tighter still.

722 messages are being destroyed on a rolling basis

Fourteen days of metrics on the larger queue show NumberOfMessagesReceived = 0 and NumberOfMessagesDeleted = 0 every single day — nothing has ever consumed it. Yet its depth oscillates between 571 and 722 while the oldest message sits pinned at exactly the four-day retention cap.

The "dead queue still provisioned" finding is confirmed literally: dead_queue exists with zero messages, alongside shoryuken_legacy_jobs and its DLQ.

Sending identity: 10DLC registration

The registration chain is in good shape, and the gap that exists is specific and actionable.

Of 652 active non-demo merchants, 499 are flagged ten_dlc_enabled and expected to send SMS. Tracing the full brand → campaign → number chain for those 499:

464 of 499 hold the complete chain. The 36 that do not have a valid brand and campaign and are missing only the number — a single, specific, fixable link rather than a registration failure.

Live numbers held by accounts that should not have them

The 341 numbers held by seven lost merchants are the clearest reclaim candidates — a departed account holding live 10DLC numbers is both a recurring cost and unnecessary carrier-registration surface.

Data-integrity notes from the same pass: 39 brands are orphaned (no merchant points at them), 2 active campaigns hang off an inactive brand, and all 652 active merchants carry a non-null legacy dlc_campaigns JSON column running parallel to the normalised tables — a possible dual source of truth that was not audited.

Revenue assurance

Two findings that belong together. Both describe a commercial arrangement and a running system that disagree, and neither appears on any monitoring surface — because in both cases every request succeeds.

Loyalty mechanics on messaging-only accounts

Of 233 active, non-demo merchants labelled platform_type = messaging_only, 31 are running loyalty mechanics in the last 90 days.

These are not recent reclassifications. Read against PaperTrail — up to 2,393 versions per merchant — every one has held messaging_only with loyalty = true for its entire recorded history. One merchant, High Ties, did move from loyalty_and_messaging on 2026-03-18, after which its points programme genuinely stopped (redemptions fell from 46,296 to 1) while its offers programme continued. 231 of the 233 messaging-only merchants carry loyalty = true.

Why it is possible: the enforcement gap

The capability boundary exists in code and is almost entirely unwired. require_platform! is defined twice and called four times, all in two legacy server-rendered controllers. The api/web/v1 concern defines its own copy that nothing calls.

Messages sent against unpaid invoices

Measured at billing-organisation level over twelve months, excluding zero-amount invoices:

The collectible cohort sent 26,805,228 messages in 90 days. Largest exposures: Story Cannabis $180,460 (7.7M messages), Beyond Hello $135,047 (11.8M messages), Spiritleaf $133,509, Lume Co. $96,621, FIKA $75,010 (2.4M messages), Sessions Cannabis $51,454 (1.2M messages).

The backlog is persistent rather than recent — unpaid invoices appear in every one of the last twelve months, between 33 and 114 per month.

Audiences

Two audiences are failing right now, both with the defect closed on 8 August, because the configuration shape that causes it is still writable.

Both carry failure_message = NULL, which is itself diagnostic: the Node Lambda writes state directly to Postgres and never runs the Rails setter that populates the message. A null message with state 3 means the query died in Redshift.

71620 — merchant 38389

variant: greater_than
value: 1
secondary_variant: month     # datepart in the WRONG key
secondary_value: '1'

71378 — merchant 3092

variant: greater_than
value: 1
secondary_value: '1'         # datepart MISSING entirely

Working audiences on the same event type put the datepart in secondary_value. Because ParseQueryService#render substitutes an empty string for any variable it cannot resolve, the misconfiguration does not raise — it renders structurally broken SQL that travels through SQS and dies far from its cause. Captured from /aws/lambda/audience_build_members:

date_trunc('1', transaction_details.transaction_date AT TIME ZONE 'UTC' AT TIME ZONE 'America/Denver')
ERROR: Invalid datetime part for DATE_TRUNC()

Both audiences fail every day: 71378 on eight of the last seven days' runs (10, 11, 12, 13, 14, 15 and twice on 10 August), 71620 on four. Twelve failedBuildsForDlq records over seven days, every one the same signature.

Webhooks: firing frequency and payload completeness

Two hypotheses were tested — that webhooks fire too often, and that they carry incomplete data. The second is confirmed. The first is largely refuted, and the reason it looked true is itself the more useful finding.

"Fires too often" — refuted as stated

The emission path is guarded at three independent levels, and the measured rate is close to the floor. webhook_member_concern.rb:32-34 checks an attribute allowlist, so a bare touch or an updated_at-only save emits nothing:

changed_values = previous_changes.keys.map(&:to_sym)
deliver_webhook(:updated) if saved_changes? && (WEBHOOK_PERMITTED_ATTRIBUTES & changed_values).any?

A second gate at enqueue time returns early unless a subscribed endpoint exists, and EventWorker carries unique_for: 60.seconds. Measured over two hours against real business activity:

"Carries incomplete data" — confirmed, but not by being sparse

Payloads are large: 29 to 32 fields, including points balance (ghetto_balance), consent state (allowed_email, allowed_sms, allowed_loyalty) and tier (current_milestone_tiers). What is missing is not bulk. It is meaning.

This is the documented mechanism behind the known Cova full-object-PUT clobber: a receiver that cannot tell what changed must write everything back, and a partial write erases what it omits.

Sampled production payloads

Every sampled payload carried current_milestone_tiers: {spend: null, point: null, visit: null} — tier data is structurally present and empty in practice. None contained any change indicator, confirming the code reading empirically.

Ten declared events have no subscribers

Of 13 declared event types, only three have any subscriber: member_updated (816 endpoints), member_created (811) and member_deleted (1). The remaining ten — campaign, message template, template image and reward grant events — still execute their callbacks and a database existence check on every relevant save, for zero recipients. visit_created and mailchimp_created are declared and never emitted at all.

Endpoint hygiene

Four endpoints point at production.api.sringbig.technology — a misspelled domain that cannot resolve.

What to change

Confidence and known limits

Every figure in this document is a measurement, and measurements have error bars. This section records where the numbers are firm, where they are estimates, and the specific ways this investigation got things wrong before catching them — because a reader deciding what to act on needs to know which is which.

Corrections made during the investigation

Seven working conclusions were wrong and were caught by a second check. They are listed because each one nearly reached this document as fact, and because they share a pattern worth naming.

• A controller label was read as a page identity. New Relic groups by controller action, so /favicon.ico — swallowed by a catch-all route — was counted as enroll/members/edit. A 99.6% failure rate looked like members locked out of their wallets. The nginx access log, which records the requested URL, showed 85.6% were browser icon requests and zero were member wallet pages.
• A deploy was nearly blamed for it. Those failures appeared to begin on 11 August, matching a commit that touched that controller. New Relic retains eight days; the apparent onset was the retention edge.
• A malformed query returned a confident zero. http.statusCode is numeric in this account, so = '500' matched nothing.
• A live table was declared dead. invoice_items was filtered on transaction_date, which is null table-wide; the live column is created_at.
• An unpaid total was inflated 4.7×. Organisation-level invoices are attributed to every child merchant, so seven sibling accounts each showed the same $450,783.
• A YAML parser was pointed at JSON. versions.object is JSON; the first pass returned null for every field across 9,771 rows and would have read as "no history".
• An index was assumed absent. reward_grants has a merchant_id index, so a workaround built to avoid a sequential scan was unnecessary.

Figures that are firm

Counts drawn directly from a full population, with no sampling or extrapolation: the endpoint and route inventories; the code census (an AST walk over 2,350 files, zero parse failures); merchant, audience, webhook-endpoint and billing counts from Postgres; the Cloudflare zone sweep across all 1,799 zones; Rollbar active-item counts, fully paginated.

Figures that carry caveats

What could not be measured

• No GCP credential is reachable from this environment, so the cloud-project estate is not measurable here.
• Mobile crash telemetry does not exist in any reachable source — not in Rollbar, not in the 410.6M-row mobile event table. The apps are instrumented for behaviour and not for failure.
• Carrier-side 10DLC registration is not in this database, so the 931 campaigns flagged not-submitted cannot be adjudicated from here.
• Delivery success for 91 webhook endpoints that bypass the delivery Lambda is unmeasured.
• Two Rails-side structural claims — 7 routes at missing controller classes, 28 unrouted controllers — were not re-checked and are carried from the earlier measurement.
• Whether the member-identity rescues cause the duplicate-member errors is a hypothesis, not a finding. It needs a targeted trace of individual records across successive writes.

Method

Constraints that shape these numbers

• New Relic retains eight days. No onset earlier than 2026-08-08 can be established from it, and per-day TIMESERIES results proved unreliable in this account — discrete windowed counts were used instead and cross-checked against aggregates.
• Non-production applications must be excluded. 90% of raw 5xx volume is Beta-environment health checks that return 500 by design. Every figure here filters to appName = 'springBIG'.
• Rollbar caps total_count at 1,000, so resolved and unique counts are floors. Rate-limit warnings in the frontend project mean its occurrence total is also a floor.
• Cloudflare's account rollup undercounts by 40% against a full per-zone sweep, for reasons that could not be established. The per-zone sweep is used throughout, and no traffic delta between readings is published because the baselines may differ in derivation.
• Firewall, bot-score and origin-latency data remain unavailable on the Cloudflare Free plan — re-tested and still denied at both zone and account level.
• No GCP credential is reachable from this environment, so the cloud-project estate is not measurable here.
• The published documentation was enumerated completely via the readme.io site manifest and its embedded OpenAPI fragments — 48 pages, all HTTP 200. Two Rails-side structural claims (7 routes at missing controller classes, 28 unrouted controllers) were not re-checked and remain at their 4 August values.
• Sunday effect. 16 August was a Sunday. Where a figure could be distorted by weekend seasonality it was measured across full days and normalised against total log volume; the transaction-date rate is flat at ~19,500–20,600 errors per million log lines across the window.