[feature] AB테스트를 위한 실험 구조 구축#1358
Merged
seongwon030 merged 15 commits intodevelop-fefrom Mar 28, 2026
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 4 minutes and 13 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
Warning
|
| Cohort / File(s) | Summary |
|---|---|
문서: 실험 및 리포팅 docs/experiments.md, docs/mixpanel-reporting.md, docs/mixpanel-weekly-report-prompts.md, docs/mixpanel-admin-weekly-report-prompts.md, docs/weekly-reports/2026-W12-user-mixpanel-report.md |
클라이언트 A/B 실험 운영 가이드(폴더 구조, 등록 절차, localStorage 저장 규칙, 초기화/리셋 방법) 및 Mixpanel 주간 리포트/프롬프트/템플릿/샘플 리포트 문서 추가. |
실험 타입 및 정의 frontend/src/experiments/types.ts, frontend/src/experiments/definitions.ts |
타입 선언(ExperimentVariant, ExperimentDefinition, ExperimentAssignments) 추가 및 예시 실험(main_banner_v1, apply_button_copy_v1)과 ALL_EXPERIMENTS 내보냄. |
실험 저장소 및 할당 로직 frontend/src/experiments/ExperimentRepository.ts |
moadong_experiments 키로 localStorage 안전 읽기/쓰기, 가중치 기반 선택(pickWeightedVariant), 기존 유효 할당 보존·신규 할당 생성, resetAssignments() 포함한 singleton experimentRepository 추가. |
초기화 및 훅 통합 frontend/src/experiments/initializeExperiments.ts, frontend/src/hooks/Experiment/useExperimentVariant.ts, frontend/src/index.tsx |
initializeExperiments() 함수 추가 및 index에서 호출하도록 삽입. React 훅 useExperimentVariant로 컴포넌트가 저장된 할당을 조회하도록 통합. |
기타 리포지토리 문서/설정 .gitignore, frontend/CLAUDE.md, frontend/.claude/... |
.gitignore 일부 항목 제거 및 프론트엔드 설명서·Claude 에이전트/명령 관련 문서 다수 추가. |
Sequence Diagram(s)
sequenceDiagram
participant App as Frontend App
participant Init as initializeExperiments()
participant Repo as ExperimentRepository
participant Storage as localStorage
participant Hook as useExperimentVariant(Component)
App->>Init: initializeExperiments()
Init->>Repo: fetchAndAssignExperiments(ALL_EXPERIMENTS)
Repo->>Storage: Read "moadong_experiments"
alt Existing valid assignment
Storage-->>Repo: stored assignments
Repo->>Repo: validate assignments vs variants
else New or invalid assignment
Repo->>Repo: pickWeightedVariant() per experiment
Repo->>Storage: Write updated assignments
end
Repo-->>Init: complete
Note over App,Hook: Component 렌더링 시
Hook->>Repo: getVariant(experiment)
Repo->>Storage: Read "moadong_experiments"
Storage-->>Repo: assignments
Repo->>Repo: return assigned or default variant
Repo-->>Hook: variant
Hook-->>App: render with variant
Estimated code review effort
🎯 3 (Moderate) | ⏱️ ~20 minutes
Possibly related PRs
- [release] v1.1.10 #1006: frontend/src/index.tsx 초기화 흐름 변경과 관련 — 본 PR이 index 초기화에 initializeExperiments()를 추가함으로써 초기화 순서 충돌 가능성 확인 필요.
Suggested reviewers
- oesnuj
- lepitaaar
- suhyun113
🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (2 passed)
| Check name | Status | Explanation |
|---|---|---|
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | ✅ Passed | PR 제목은 실험 구조 구축(A/B 테스트)이라는 주요 변경사항을 명확하게 요약하며, 변경사항과 완전히 관련있습니다. |
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
- Commit unit tests in branch
feature/#1354-mixpanel-experiment-hook-MOA-763
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.
🎨 UI 변경사항을 확인해주세요
2개 스토리 변경 · 전체 56개 스토리 · 22개 컴포넌트 |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
docs/weekly-reports/2026-W12-user-mixpanel-report.md (1)
12-12: 제출 지표의 기준(전체 vs 순차 퍼널)을 명시해 주세요.같은 문서에서 제출 유저가
3(KPI)인데 퍼널 끝은1로 보여 해석 충돌이 발생합니다. 순차 퍼널 기준이라면 Line 17에strict/sequential funnel임을 명시해 오해를 막는 게 좋습니다.Also applies to: 17-19
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/weekly-reports/2026-W12-user-mixpanel-report.md` at line 12, Clarify that the report shows "overall vs sequential (strict) funnel" metrics for the Application Form Submitted line (the string "Application Form Submitted (events/users): `11 / 3` vs `34 / 33`") by labeling which number is the overall metric and which is the strict/sequential-funnel metric, and add a short note explaining the KPI count (3) refers to the overall metric while the sequential funnel end shows 1 due to strict step matching; also update the block that currently contains the text "strict/sequential funnel" to explicitly state "overall / strict (sequential) funnel" and apply the same labeling/clarification to the analogous metrics mentioned later (the items referenced in the comment for lines 17-19).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/experiments.md`:
- Around line 25-26: The docs reference the wrong localStorage key; update the
documentation to use the actual implementation key "moadong_experiments"
(instead of "moadong_experiment_assignments") wherever the key is mentioned
(including the initialization guide and any lines that instruct storing JSON),
so the doc matches the code used by useExperimentVariant and the runtime
initialization logic.
In `@docs/mixpanel-reporting.md`:
- Around line 9-10: The two links in docs/mixpanel-reporting.md use absolute
local paths and will break for others; update the links for "사용자(학생) 흐름 리포트"
(mixpanel-weekly-report-prompts.md) and "관리자(Admin) 흐름 리포트"
(mixpanel-admin-weekly-report-prompts.md) to use repository-relative paths
(e.g., ./mixpanel-weekly-report-prompts.md or
docs/mixpanel-weekly-report-prompts.md) instead of /Users/..., and make the same
change for the other occurrence noted (line 14).
In `@frontend/src/experiments/ExperimentRepository.ts`:
- Around line 9-16: safeReadAssignments currently parses localStorage without
validating shape, causing runtime errors when the parsed value is null, an
array, or has non-string values; update safeReadAssignments to validate that
parsed result is a plain object (not null/array) and that each property value is
a string before casting to ExperimentAssignments, returning {} if validation
fails; reference the ASSIGNMENT_STORAGE_KEY read and the ExperimentAssignments
type when adding checks so callers of safeReadAssignments (and later code
accessing assignment records) always receive a safe string-to-string map.
- Around line 18-20: The write path isn't protected against Storage exceptions:
wrap the localStorage.setItem call inside writeAssignments and the
localStorage.removeItem call (the one that clears ASSIGNMENT_STORAGE_KEY) in a
try/catch that catches and silently handles SecurityError/QuotaExceededError
(and any other thrown error), log the error for debugging (or use the existing
logger), and ensure the app continues with a safe fallback (e.g., return without
throwing) so behavior matches safeReadAssignments; locate these fixes by
updating function writeAssignments and the block that calls
localStorage.removeItem with ASSIGNMENT_STORAGE_KEY.
---
Nitpick comments:
In `@docs/weekly-reports/2026-W12-user-mixpanel-report.md`:
- Line 12: Clarify that the report shows "overall vs sequential (strict) funnel"
metrics for the Application Form Submitted line (the string "Application Form
Submitted (events/users): `11 / 3` vs `34 / 33`") by labeling which number is
the overall metric and which is the strict/sequential-funnel metric, and add a
short note explaining the KPI count (3) refers to the overall metric while the
sequential funnel end shows 1 due to strict step matching; also update the block
that currently contains the text "strict/sequential funnel" to explicitly state
"overall / strict (sequential) funnel" and apply the same labeling/clarification
to the analogous metrics mentioned later (the items referenced in the comment
for lines 17-19).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f36763bc-0c5c-4f40-9070-c0e8132d9229
📒 Files selected for processing (11)
docs/experiments.mddocs/mixpanel-admin-weekly-report-prompts.mddocs/mixpanel-reporting.mddocs/mixpanel-weekly-report-prompts.mddocs/weekly-reports/2026-W12-user-mixpanel-report.mdfrontend/src/experiments/ExperimentRepository.tsfrontend/src/experiments/definitions.tsfrontend/src/experiments/initializeExperiments.tsfrontend/src/experiments/types.tsfrontend/src/hooks/Experiment/useExperimentVariant.tsfrontend/src/index.tsx
lepitaaar
reviewed
Mar 28, 2026
lepitaaar
reviewed
Mar 28, 2026
lepitaaar
reviewed
Mar 28, 2026
lepitaaar
approved these changes
Mar 28, 2026
Contributor
lepitaaar
left a comment
lepitaaar
left a comment
There was a problem hiding this comment.
요청하신 Always Approve 정책에 따라 승인합니다.
동시에 재발 방지 차원에서 인라인 코멘트로 아래 확인 포인트를 남겼습니다.
- 실험 저장 write 예외 처리(fail-open)
- 실험 스토리지 key 문서/구현 정합성
- mixpanel-reporting 문서의 로컬 절대경로 링크를 상대경로로 교정
다음 커밋에서 위 코멘트 반영 여부를 확인하겠습니다.
Member
Author
이은호픈클로 제발 그만 |
- writeAssignments에 try-catch 추가 (용량 초과, 권한 거부 시 앱 초기화 중단 방지) - safeReadAssignments에 JSON 구조 검증 추가 (isObjectRecord 타입 가드, string 값 필터링) Made-with: Cursor
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
frontend/src/experiments/ExperimentRepository.ts (1)
95-97:⚠️ Potential issue | 🟠 Major
resetAssignments도 Storage 예외를 보호해야 합니다.Line 96의
localStorage.removeItem은 브라우저 정책/권한 상태에 따라 throw할 수 있어, 현재 구현은 리셋 호출 시 예외 전파 가능성이 남아 있습니다.writeAssignments와 동일하게 try/catch로 fail-open 처리해 주세요.🛠️ 제안 수정
resetAssignments() { - localStorage.removeItem(ASSIGNMENT_STORAGE_KEY); + try { + localStorage.removeItem(ASSIGNMENT_STORAGE_KEY); + } catch { + // localStorage 삭제 실패는 무시하고 진행한다. + } }검증 방법(읽기 전용): 아래 스크립트로 현재
removeItem호출이 보호되지 않았는지 확인할 수 있습니다.#!/bin/bash set -euo pipefail echo "[1] resetAssignments 주변 코드 확인" sed -n '90,102p' frontend/src/experiments/ExperimentRepository.ts echo echo "[2] ASSIGNMENT_STORAGE_KEY removeItem 호출 위치/컨텍스트 확인" rg -n -C2 'localStorage\.removeItem\(ASSIGNMENT_STORAGE_KEY\)' frontend/src/experiments/ExperimentRepository.ts🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@frontend/src/experiments/ExperimentRepository.ts` around lines 95 - 97, The resetAssignments method currently calls localStorage.removeItem(ASSIGNMENT_STORAGE_KEY) without protection and can throw; update resetAssignments to mirror writeAssignments by wrapping the removeItem call in a try/catch and failing open (swallow/log the error but do not rethrow) so a storage exception won't propagate to callers; reference resetAssignments and ASSIGNMENT_STORAGE_KEY when making the change and follow the same error-handling pattern used in writeAssignments.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/experiments.md`:
- Around line 10-19: The fenced code block that lists the experiment files (the
block beginning with "frontend/src/experiments/") is missing a language tag and
triggers MD040; add a language identifier (e.g., `text`) to the opening
triple-backticks of that code fence so it becomes ```text, leaving the block
content unchanged; this targets the code block that shows
"frontend/src/experiments/" and the subsequent "frontend/src/hooks/Experiment/"
listing.
In `@frontend/.claude/commands/create-e2e-test.md`:
- Line 10: The instruction "성공 될때까지 개선" in the create-e2e-test command is
unsafe—replace it with explicit retry and failure policies: enforce a maximum
retry count (e.g., maxRetries), a backoff strategy, and clear abort conditions
(e.g., stop on N consecutive failures or total runtime limit), and require
printing a failure report with test names, error traces, and flakiness hints
when retries are exhausted; update the command text to mention these fields
(maxRetries, backoffSeconds, abortOnConsecutiveFailures, failureReport) so CI
won't loop indefinitely or hide root causes.
---
Duplicate comments:
In `@frontend/src/experiments/ExperimentRepository.ts`:
- Around line 95-97: The resetAssignments method currently calls
localStorage.removeItem(ASSIGNMENT_STORAGE_KEY) without protection and can
throw; update resetAssignments to mirror writeAssignments by wrapping the
removeItem call in a try/catch and failing open (swallow/log the error but do
not rethrow) so a storage exception won't propagate to callers; reference
resetAssignments and ASSIGNMENT_STORAGE_KEY when making the change and follow
the same error-handling pattern used in writeAssignments.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e065d8c0-a2b3-48ee-adca-a6cc7011e3c3
📒 Files selected for processing (10)
.gitignoredocs/experiments.mddocs/mixpanel-reporting.mdfrontend/.claude/agents/API훅부서.mdfrontend/.claude/commands/create-e2e-test.mdfrontend/.claude/commands/find-e2e-test.mdfrontend/.claude/commands/tm/auto-implement-tasks.mdfrontend/CLAUDE.mdfrontend/src/experiments/ExperimentRepository.tsfrontend/src/hooks/Experiment/useExperimentVariant.ts
💤 Files with no reviewable changes (1)
- .gitignore
✅ Files skipped from review due to trivial changes (6)
- frontend/.claude/commands/tm/auto-implement-tasks.md
- frontend/.claude/commands/find-e2e-test.md
- frontend/src/hooks/Experiment/useExperimentVariant.ts
- frontend/CLAUDE.md
- docs/mixpanel-reporting.md
- frontend/.claude/agents/API훅부서.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#️⃣연관된 이슈
📝작업 내용
Mixpanel 이벤트 데이터를 바탕으로 퍼널 특정 구간(지원 버튼 CTA, 팝업 등)에 대한 A/B 실험을 했었습니다.
개발자가 좀 더 쉽게 AB테스트를 할 수 있도록 localStorage 기반 실험 구조를 만들었습니다.
동작 방식
initializeExperiments()가ALL_EXPERIMENTS를 순회하며 배리언트를 배정한다.localStorage의moadong_experiments키에 저장된다.useExperimentVariant(experimentDefinition)으로 배리언트를 읽어 분기한다.새 실험 추가 방법
자세한 내용은
docs/experiments.md참고.중점적으로 리뷰받고 싶은 부분(선택)
논의하고 싶은 부분(선택)
🫡 참고사항
Summary by CodeRabbit
New Features
Documentation