"""Импорт месячных JSON из report-tool/ в WorkLog.""" from __future__ import annotations import calendar import json import re from dataclasses import dataclass, field from datetime import date, datetime from pathlib import Path from django.contrib.auth import get_user_model from django.db import transaction from apps.it_manager.models import WorkLog from apps.locations.models.base import Location, organization_object_letter from apps.locations.models.pharmacies import Pharmacy from core.models.directories import Directory Employee = get_user_model() UA_LOWER_TO_RU = {'і': 'и', 'ї': 'и', 'є': 'е', 'ґ': 'г'} STAFF_LAST_NAMES = { 'babich': ('бабич',), 'aprodov': ('апродов',), 'khizhniak': ('хижняк', 'хіжняк'), 'kogutenko': ('когутенко',), } DIRECTION_TO_WORK_TYPE = { 'prro': 'software', 'terminal': 'installation', 'print': 'repair', 'network': 'network', 'cctv': 'installation', 'pc': 'repair', 'install': 'installation', 'access': 'software', 'infra': 'maintenance', } OBJ_CODE_RE = re.compile(r'^([АAПPТTГGЛL])(\d+)$', re.IGNORECASE) IMPORT_MARKER = 'import:report-tool:{month_id}' def normalize_name(value: str) -> str: value = (value or '').strip().lower() return ''.join(UA_LOWER_TO_RU.get(c, c) for c in value) def default_report_tool_root() -> Path: # apps/it_manager/report_import.py → apms/ return Path(__file__).resolve().parents[3] / 'report-tool' def load_company(root: Path) -> dict: return json.loads((root / 'company.json').read_text(encoding='utf-8')) def load_month_json(path: Path) -> dict: return json.loads(path.read_text(encoding='utf-8')) def parse_task_date(value: str, year: int, month: int) -> date | None: value = (value or '').strip() if not value: return None for fmt in ('%d.%m.%Y', '%d.%m.%y', '%Y-%m-%d'): try: return datetime.strptime(value, fmt).date() except ValueError: continue m = re.fullmatch(r'(\d{1,2})/(\d{1,2})/(\d{2,4})', value) if m: mm, dd, yy = int(m.group(1)), int(m.group(2)), int(m.group(3)) if yy < 100: yy += 2000 try: return date(yy, mm, dd) except ValueError: return None m = re.fullmatch(r'(\d{1,2})\.(\d{1,2})', value) if m: try: return date(year, int(m.group(2)), int(m.group(1))) except ValueError: return None return None def month_end(year: int, month: int) -> date: return date(year, month, calendar.monthrange(year, month)[1]) def classify_work_type_code(text: str, directions: list[dict]) -> str: low = (text or '').lower() for direction in directions: for kw in direction.get('keywords') or []: if kw.lower() in low: return DIRECTION_TO_WORK_TYPE.get(direction.get('id'), 'other') return 'other' @dataclass class ImportStats: created: int = 0 skipped: int = 0 missing_employees: list[str] = field(default_factory=list) unresolved_objects: list[str] = field(default_factory=list) unmatched_phrases: list[str] = field(default_factory=list) months: list[str] = field(default_factory=list) class CatalogMatchError(Exception): """Не все фразы сматчились в справочник — импорт запрещён.""" def __init__(self, phrases: list[str]): self.phrases = phrases sample = ', '.join(phrases[:15]) more = f' … (+{len(phrases) - 15})' if len(phrases) > 15 else '' super().__init__( f'Нет матча в справочнике для {len(phrases)} фраз: {sample}{more}. ' f'Дополните work-import-aliases.json и проверьте ' f'manage.py match_report_tool_works.' ) class LocationResolver: def __init__(self): self._by_prefix_num: dict[tuple[str, int], Location] = {} self._by_num: dict[int, list[Location]] = {} self._office: Location | None = None self._warehouse: Location | None = None self._vendors_by_name: dict[str, Location] = {} self._built = False def build(self): if self._built: return self._built = True for pharmacy in Pharmacy.objects.select_related('organization').filter(is_active=True): num = pharmacy.pharmacy_number if num is None: continue letter = organization_object_letter(pharmacy.organization) if letter: self._by_prefix_num[(letter[0].lower(), int(num))] = pharmacy self._by_num.setdefault(int(num), []).append(pharmacy) self._office = Location.objects.filter(location_type='office', is_active=True).order_by('id').first() self._warehouse = Location.objects.filter(location_type='warehouse', is_active=True).order_by('id').first() self._vendors_by_name = {} for loc in Location.objects.filter(location_type='vendor', is_active=True): key = normalize_name(loc.name or '') if key: self._vendors_by_name[key] = loc def resolve(self, object_code: str) -> Location | None: self.build() raw = (object_code or '').strip() if not raw: return None upper = raw.upper().replace(' ', '') if upper in ('ОФИС', 'ОФІС', 'OFFICE'): return self._office if upper in ('СКЛАД',): return self._warehouse vendor = self._vendors_by_name.get(normalize_name(raw)) if vendor is not None: return vendor m = OBJ_CODE_RE.fullmatch(raw.replace(' ', '')) if not m: return None letter = m.group(1).upper().translate(str.maketrans('APTGL', 'АПТГЛ')) num = int(m.group(2)) letter_l = letter.lower() hit = self._by_prefix_num.get((letter_l, num)) if hit: return hit # fallback: единственная аптека с таким номером candidates = self._by_num.get(num) or [] if len(candidates) == 1: return candidates[0] return None def resolve_employee(staff_id: str, staff_name: str, username_map: dict[str, str]) -> Employee | None: if staff_id in username_map: return Employee.objects.filter(username=username_map[staff_id]).first() last_names = STAFF_LAST_NAMES.get(staff_id) if last_names: for emp in Employee.objects.filter(is_active=True): if normalize_name(emp.last_name) in {normalize_name(n) for n in last_names}: return emp # по фамилии из полного имени parts = (staff_name or '').split() if parts: target = normalize_name(parts[0]) for emp in Employee.objects.filter(is_active=True): if normalize_name(emp.last_name) == target: return emp return None def ensure_employee(staff_id: str, staff_name: str) -> Employee: parts = (staff_name or staff_id).split() last_name = parts[0] if parts else staff_id first_name = parts[1] if len(parts) > 1 else staff_id username = f'it.{staff_id}' emp, created = Employee.objects.get_or_create( username=username, defaults={ 'last_name': last_name, 'first_name': first_name, 'email': f'{username}@company.local', 'hire_date': date(2020, 1, 1), 'user_location_type': 'office', 'mattermost_sync_enabled': False, 'ad_sync_enabled': False, 'is_active': True, }, ) if created: emp.set_unusable_password() emp.save() from django.contrib.auth.models import Group it_group, _ = Group.objects.get_or_create(name='IT') emp.groups.add(it_group) return emp def work_type_id(code: str, cache: dict[str, int]) -> int: if code not in cache: pk = Directory.objects.filter( directory_type__code='worklog_type', code=code, ).values_list('id', flat=True).first() if not pk: pk = Directory.objects.filter( directory_type__code='worklog_type', code='other', ).values_list('id', flat=True).first() cache[code] = pk return cache[code] def import_marker(month_id: str) -> str: return IMPORT_MARKER.format(month_id=month_id) def iter_month_files(root: Path, months: list[str] | None) -> list[Path]: data_dir = root / 'data' files = sorted(data_dir.glob('????-??.json')) if months: wanted = set(months) files = [p for p in files if p.stem in wanted] return files def catalog_work_pk(code: str, cache: dict[str, int]) -> int: if code not in cache: pk = Directory.objects.filter( directory_type__code='it_work', code=code, is_active=True, ).exclude(code='_meta').values_list('id', flat=True).first() if not pk: raise CatalogMatchError([f'(нет Directory it_work code={code})']) cache[code] = pk return cache[code] @transaction.atomic def import_report_tool( *, root: Path | None = None, months: list[str] | None = None, username_map: dict[str, str] | None = None, create_missing_employees: bool = False, dry_run: bool = False, require_catalog: bool = True, mapping_path: Path | None = None, aliases_path: Path | None = None, ) -> ImportStats: """Импорт data/YYYY-MM.json → WorkLog с обязательным catalog_work. require_catalog=True (по умолчанию): все фрагменты должны сматчиться exact по mapping+aliases; иначе CatalogMatchError, записи не пишутся. """ from apps.it_manager.work_import_match import ( audit_month_files, build_phrase_index, match_system, split_work_text, ) root = root or default_report_tool_root() username_map = username_map or {} locations = LocationResolver() catalog_cache: dict[str, int] = {} stats = ImportStats() index = build_phrase_index(mapping_path=mapping_path, aliases_path=aliases_path) if require_catalog: audit = audit_month_files(root / 'data', index, months=months) if audit.unmatched: phrases = [ p for p, _ in sorted( audit.unmatched_freq.items(), key=lambda x: (-x[1], x[0].casefold()), ) ] stats.unmatched_phrases = phrases raise CatalogMatchError(phrases) for path in iter_month_files(root, months): month = load_month_json(path) month_id = month.get('id') or path.stem year = int(month['year']) month_num = int(month['month']) marker = import_marker(month_id) stats.months.append(month_id) default_date = month_end(year, month_num) for emp_data in month.get('employees') or []: staff_id = emp_data.get('id') or '' staff_name = emp_data.get('name') or staff_id employee = resolve_employee(staff_id, staff_name, username_map) if employee is None: if create_missing_employees: if dry_run: employee = None else: employee = ensure_employee(staff_id, staff_name) else: if staff_id not in stats.missing_employees: stats.missing_employees.append(staff_id or staff_name) continue for task in emp_data.get('tasks') or []: work = (task.get('work') or '').strip() if not work: continue obj = (task.get('object') or '').strip() log_date = parse_task_date(task.get('date') or '', year, month_num) or default_date location = locations.resolve(obj) if obj else None if obj and location is None and obj.upper() not in stats.unresolved_objects: stats.unresolved_objects.append(obj) for part in split_work_text(work): hit = index.match(part) if hit is None: # require_catalog уже отсёк; на всякий случай raise CatalogMatchError([part]) _create_or_skip( stats=stats, dry_run=dry_run, employee=employee, log_date=log_date, location=location, catalog_work_id=catalog_work_pk(hit.target_id, catalog_cache), details=hit.details, description=part, marker=marker, ) for system in emp_data.get('systems') or []: name = (system.get('name') or '').strip() work = (system.get('work') or '').strip() if not name or not work: continue hit = match_system(index, name, work) if hit is None: raise CatalogMatchError([f'{name}: {work}']) description = f'{name}: {work}' _create_or_skip( stats=stats, dry_run=dry_run, employee=employee, log_date=default_date, location=locations.resolve('ОФИС'), catalog_work_id=catalog_work_pk(hit.target_id, catalog_cache), details=hit.details, description=description, marker=marker, ) return stats def _create_or_skip( *, stats, dry_run, employee, log_date, location, catalog_work_id, details, description, marker, ): if employee is None: # dry-run без создания сотрудников stats.created += 1 return details = details or {} exists = WorkLog.objects.filter( technician=employee, date=log_date, catalog_work_id=catalog_work_id, description=description, result=marker, ).exists() if exists: stats.skipped += 1 return if dry_run: stats.created += 1 return WorkLog.objects.create( date=log_date, technician=employee, location=location, catalog_work_id=catalog_work_id, details=details, work_type_id=None, description=description, result=marker, hours_spent=None, ) stats.created += 1