Skip to content

Commit 570bbad

Browse files
AHS12AHS12
authored
Merge pull request #11 from Innovix-Matrix-Systems/laravel-12-upgrade
feat(data-processing): add background job system for import/export
2 parents 506ebba + 81b1637 commit 570bbad

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+2248
-68
lines changed

‎README.md‎

Lines changed: 21 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# IMS Laravel API Starter
22

3-
A production-ready Laravel API starter kit with enterprise-grade features. Built-in authentication, RBAC, comprehensive API documentation, and advanced observability tools. Designed for rapid development with clean architecture patterns.
3+
A production-ready Laravel API starter kit with enterprise-grade features. Built-in authentication, RBAC, comprehensive API documentation, advanced observability tools, and production-ready testing infrastructure. Designed for rapid development with clean architecture patterns.
44

55
<p align="center">
66
<img src="image1.png" alt="Observability Dashboard" width="100%" />
@@ -17,14 +17,20 @@ A production-ready Laravel API starter kit with enterprise-grade features. Built
1717
## Features
1818

1919
- 🔐 **Laravel Sanctum** - Secure API authentication with personal access tokens
20+
- 📱 **Multi-Device Login** - Device-specific token management with logout capabilities
21+
- 🔢 **OTP Authentication** - Configurable phone-based OTP login flow with rate limiting
2022
- 🔑 **RBAC System** - Role-based access control with permissions and roles
2123
- 📚 **API Documentation** - Scalar, Swagger UI, OpenAPI with Postman compatibility
2224
- 📊 **Observability** - Telescope, Pulse, Health with unified dashboard
2325
- 🛠️ **Clean Architecture** - Repository pattern, DTOs, service layer
24-
- 💾 **Data Management** - User management, data export, media library, backups
25-
- 🌍 **Internationalization** - Multi-language support (Bangla & English)
26+
- 💾 **Data Management** - User management, data export/import, media library, backups
27+
- 📤 **Background Import/Export** - Queue-based bulk user data processing with Excel/CSV support
28+
- 📊 **Job Tracking** - Real-time monitoring of background jobs with progress tracking
29+
- 🧹 **Automated Cleanup** - Scheduled cleanup of completed jobs and temporary files
30+
- 🌍 **Internationalization** - Multi-language support (English, Bengali), ability to add more as needed
2631
- 🐳 **Docker Support** - Complete containerized development environment
2732
- ⚡ **Development Tools** - Code generators, IDE helpers, Git hooks
33+
- 🧪 **Production-Ready Testing** - Pest PHP with Mockery, comprehensive feature/unit tests, queue testing, DTO validation
2834

2935
## Quick Start
3036

@@ -63,20 +69,15 @@ A production-ready Laravel API starter kit with enterprise-grade features. Built
6369
### Default Credentials
6470
- **Super Admin**: superadmin@ims.com / 123456
6571

72+
**Development Environment Compatibility:** Works seamlessly with modern development tools including [Laravel Herd](https://herd.laravel.com/) (blazing fast native Laravel environment), [FlyEnv](https://www.flyenv.com/) (all-in-one full-stack environment), and [Laragon](https://laragon.org/) (lightweight Windows development environment)
73+
6674
## 📖 Documentation
6775

68-
### 🌐 [GitHub Wiki](https://github.com/Innovix-Matrix-Systems/ims-laravel-api-starter/wiki)
76+
Our comprehensive documentation covers everything from setup to advanced features. Learn about background job processing, OTP authentication configuration, API endpoints, monitoring tools, and deployment strategies. Whether you're setting up for the first time or scaling for production, our detailed guides provide step-by-step instructions and best practices.
77+
78+
🌐 **[GitHub Wiki](https://github.com/Innovix-Matrix-Systems/ims-laravel-api-starter/wiki)**
6979

70-
Comprehensive documentation covering:
71-
- **Features Overview** - Complete feature breakdown
72-
- **Quick Start Guide** - Step-by-step setup instructions
73-
- **API Documentation** - Interactive API docs and testing
74-
- **Observability Guide** - Monitoring and debugging tools
75-
- **Project Structure** - Directory organization and architecture
76-
- **Running Tests** - Testing guidelines and commands
77-
- **Backup System** - Application and database backup
78-
- **Docker Guide** - Container development setup
79-
- **Extra Information** - Additional development tools
80+
*The [`docs/`](docs/) folder contains a local mirror of the wiki for offline access.*
8081

8182
### API Documentation
8283

@@ -95,13 +96,18 @@ Comprehensive documentation covering:
9596

9697
```bash
9798
# Code generation
98-
php artisan make:crud Product
99+
php artisan make:crud Product // All necessary skeleton files
99100
php artisan make:dto ProductDTO
100101
php artisan make:service Product/ProductService
102+
php artisan make:repo Product/ProductRepository
103+
// and many more, check wiki!
101104

102105
# Code quality
103106
php artisan pint
104107
php artisan optimize:clear
108+
php artisan ide-helper:generate
109+
php artisan ide-helper:models -N
110+
// and many more, check wiki!
105111
```
106112

107113
## License

‎app/Console/Commands/CleanupCompletedDataProcessingJobsCommand.php‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
<?php
2+
3+
namespace App\Console\Commands;
4+
5+
use App\Enums\DataProcessingJobStatus;
6+
use App\Models\DataProcessingJob;
7+
use Illuminate\Console\Command;
8+
use Illuminate\Support\Carbon;
9+
use Illuminate\Support\Facades\Storage;
10+
11+
class CleanupCompletedDataProcessingJobsCommand extends Command
12+
{
13+
/**
14+
* The name and signature of the console command.
15+
*
16+
* @var string
17+
*/
18+
protected $signature = 'data-processing:cleanup-completed
19+
{--days=30 : Number of days to keep completed jobs}
20+
{--dry-run : Show what would be deleted without actually deleting}';
21+
22+
/**
23+
* The console command description.
24+
*
25+
* @var string
26+
*/
27+
protected $description = 'Cleanup completed data processing jobs older than specified days';
28+
29+
/** Execute the console command. */
30+
public function handle(): int
31+
{
32+
$days = (int) $this->option('days');
33+
$dryRun = $this->option('dry-run');
34+
35+
if ($days < 1) {
36+
$this->error('Days must be a positive number');
37+
38+
return self::FAILURE;
39+
}
40+
41+
$cutoffDate = Carbon::now()->subDays($days);
42+
43+
$query = DataProcessingJob::where('status', DataProcessingJobStatus::COMPLETED)
44+
->where('completed_at', '<', $cutoffDate);
45+
46+
$count = $query->count();
47+
48+
if ($count === 0) {
49+
$this->info('No completed jobs found to cleanup');
50+
51+
return self::SUCCESS;
52+
}
53+
54+
if ($dryRun) {
55+
$this->info("Would delete {$count} completed jobs older than {$days} days");
56+
57+
// Show sample jobs that would be deleted
58+
$sampleJobs = $query->limit(5)->get(['job_id', 'type', 'completed_at']);
59+
if ($sampleJobs->isNotEmpty()) {
60+
$this->table(
61+
['Job ID', 'Type', 'Completed At'],
62+
$sampleJobs->map(fn ($job) => [
63+
$job->job_id,
64+
$job->type->value,
65+
$job->completed_at->format('Y-m-d H:i:s'),
66+
])
67+
);
68+
}
69+
70+
return self::SUCCESS;
71+
}
72+
73+
$this->info("Found {$count} completed jobs to cleanup (older than {$days} days)");
74+
75+
$progressBar = $this->output->createProgressBar($count);
76+
$progressBar->start();
77+
78+
$deletedCount = 0;
79+
$batchSize = 100;
80+
81+
do {
82+
$jobsToDelete = $query->limit($batchSize)->get();
83+
84+
if ($jobsToDelete->isEmpty()) {
85+
break;
86+
}
87+
88+
foreach ($jobsToDelete as $job) {
89+
// Delete associated files if they exist
90+
if ($job->file_path && Storage::disk('public')->exists($job->file_path)) {
91+
try {
92+
Storage::disk('public')->delete($job->file_path);
93+
} catch (\Exception $e) {
94+
$this->warn("Could not delete file: {$job->file_path}");
95+
}
96+
}
97+
98+
$job->delete();
99+
$deletedCount++;
100+
$progressBar->advance();
101+
}
102+
} while ($jobsToDelete->count() === $batchSize);
103+
104+
$progressBar->finish();
105+
$this->newLine();
106+
107+
$this->info("Successfully deleted {$deletedCount} completed jobs");
108+
109+
return self::SUCCESS;
110+
}
111+
}

‎app/DTOs/DataProcessingJob/DataProcessingJobDTO.php‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
<?php
2+
3+
namespace App\DTOs\DataProcessingJob;
4+
5+
use App\Enums\DataProcessingJobStatus;
6+
use App\Enums\DataProcessingJobType;
7+
use App\Models\DataProcessingJob;
8+
use Illuminate\Http\Request;
9+
10+
class DataProcessingJobDTO
11+
{
12+
public function __construct(
13+
public readonly ?string $jobId,
14+
public readonly ?DataProcessingJobType $type,
15+
public readonly ?DataProcessingJobStatus $status,
16+
public readonly ?string $entityType,
17+
public readonly ?string $filePath,
18+
public readonly ?string $fileName,
19+
public readonly ?string $originalFileName,
20+
public readonly ?array $filters,
21+
public readonly ?array $errors,
22+
public readonly ?int $processedRows,
23+
public readonly ?int $successCount,
24+
public readonly ?int $errorCount,
25+
public readonly ?string $errorMessage,
26+
public readonly ?int $userId,
27+
public readonly ?\DateTime $startedAt,
28+
public readonly ?\DateTime $completedAt,
29+
) {}
30+
31+
public static function fromRequest(Request $request): self
32+
{
33+
return new self(
34+
$request->input('job_id'),
35+
$request->input('type') ? DataProcessingJobType::from($request->input('type')) : null,
36+
$request->input('status') ? DataProcessingJobStatus::from($request->input('status')) : null,
37+
$request->input('entity_type'),
38+
$request->input('file_path'),
39+
$request->input('file_name'),
40+
$request->input('original_file_name'),
41+
$request->input('filters') ? json_decode($request->input('filters'), true) : null,
42+
$request->input('errors') ? json_decode($request->input('errors'), true) : null,
43+
$request->input('processed_rows'),
44+
$request->input('success_count'),
45+
$request->input('error_count'),
46+
$request->input('error_message'),
47+
$request->input('user_id'),
48+
$request->input('started_at') ? new \DateTime($request->input('started_at')) : null,
49+
$request->input('completed_at') ? new \DateTime($request->input('completed_at')) : null,
50+
);
51+
}
52+
53+
public static function fromArray(array $data): self
54+
{
55+
return new self(
56+
$data['job_id'] ?? null,
57+
isset($data['type']) ? ($data['type'] instanceof DataProcessingJobType ? $data['type'] : DataProcessingJobType::from($data['type'])) : null,
58+
isset($data['status']) ? ($data['status'] instanceof DataProcessingJobStatus ? $data['status'] : DataProcessingJobStatus::from($data['status'])) : null,
59+
$data['entity_type'] ?? null,
60+
$data['file_path'] ?? null,
61+
$data['file_name'] ?? null,
62+
$data['original_file_name'] ?? null,
63+
$data['filters'] ?? null,
64+
$data['errors'] ?? null,
65+
$data['processed_rows'] ?? null,
66+
$data['success_count'] ?? null,
67+
$data['error_count'] ?? null,
68+
$data['error_message'] ?? null,
69+
$data['user_id'] ?? null,
70+
isset($data['started_at']) ? ($data['started_at'] instanceof \DateTime ? $data['started_at'] : new \DateTime($data['started_at'])) : null,
71+
isset($data['completed_at']) ? ($data['completed_at'] instanceof \DateTime ? $data['completed_at'] : new \DateTime($data['completed_at'])) : null,
72+
);
73+
}
74+
75+
public static function fromModel(DataProcessingJob $model): self
76+
{
77+
return new self(
78+
$model->job_id,
79+
$model->type,
80+
$model->status,
81+
$model->entity_type,
82+
$model->file_path,
83+
$model->file_name,
84+
$model->original_file_name,
85+
$model->filters,
86+
$model->errors,
87+
$model->processed_rows,
88+
$model->success_count,
89+
$model->error_count,
90+
$model->error_message,
91+
$model->user_id,
92+
$model->started_at,
93+
$model->completed_at,
94+
);
95+
}
96+
97+
public function toArray(): array
98+
{
99+
return array_filter([
100+
'job_id' => $this->jobId,
101+
'type' => $this->type?->value,
102+
'status' => $this->status?->value,
103+
'entity_type' => $this->entityType,
104+
'file_path' => $this->filePath,
105+
'file_name' => $this->fileName,
106+
'original_file_name' => $this->originalFileName,
107+
'filters' => $this->filters,
108+
'errors' => $this->errors,
109+
'processed_rows' => $this->processedRows,
110+
'success_count' => $this->successCount,
111+
'error_count' => $this->errorCount,
112+
'error_message' => $this->errorMessage,
113+
'user_id' => $this->userId,
114+
'started_at' => $this->startedAt?->format('Y-m-d H:i:s'),
115+
'completed_at' => $this->completedAt?->format('Y-m-d H:i:s'),
116+
], fn ($value) => ! is_null($value));
117+
}
118+
}

‎app/DTOs/User/UserDTO.php‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,21 @@ public static function fromRequest(Request $request, ?User $existing = null, ?ar
3636
);
3737
}
3838

39+
public static function fromArray(array $data): self
40+
{
41+
return new self(
42+
$data['id'] ?? null,
43+
$data['first_name'] ?? null,
44+
$data['last_name'] ?? null,
45+
$data['name'] ?? null,
46+
$data['email'] ?? null,
47+
$data['password'] ?? null,
48+
$data['phone'] ?? null,
49+
isset($data['is_active']) ? (bool) $data['is_active'] : null,
50+
isset($data['roles']) && is_array($data['roles']) ? $data['roles'] : null,
51+
);
52+
}
53+
3954
public function toArray(): array
4055
{
4156
return array_filter([

‎app/DTOs/User/UserFilterDTO.php‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,4 +26,16 @@ public static function fromRequest(Request $request): self
2626
perPage: $request->input('per_page', config('constant.DEFAULT_PAGINATION_ITEM_COUNT', 10)),
2727
);
2828
}
29+
30+
public static function fromArray(array $data): self
31+
{
32+
return new self(
33+
search: $data['search'] ?? null,
34+
isActive: $data['is_active'] ?? null,
35+
roleName: $data['role_name'] ?? null,
36+
orderBy: $data['order_by'] ?? 'created_at',
37+
orderDirection: $data['order_direction'] ?? 'desc',
38+
perPage: $data['per_page'] ?? config('constant.DEFAULT_PAGINATION_ITEM_COUNT', 10),
39+
);
40+
}
2941
}

‎app/Enums/DataProcessingJobStatus.php‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<?php
2+
3+
namespace App\Enums;
4+
5+
enum DataProcessingJobStatus: string
6+
{
7+
case PENDING = 'pending';
8+
case PROCESSING = 'processing';
9+
case COMPLETED = 'completed';
10+
case FAILED = 'failed';
11+
}

‎app/Enums/DataProcessingJobType.php‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<?php
2+
3+
namespace App\Enums;
4+
5+
enum DataProcessingJobType: string
6+
{
7+
case IMPORT = 'import';
8+
case EXPORT = 'export';
9+
}

0 commit comments

Comments
 (0)