context-aware bim annotations

date: March 25, 2021

Problem: Construction issues get reported verbally, in scattered notes, through disconnected photos. "The pipe on the second floor near the south wall has a problem." Which pipe? Which wall? What problem? Traditional issue tracking loses spatial context. Photos exist separately from BIM models. Notes reference locations ambiguously. Weeks later, finding the exact element someone reported requires detective work.

Solution: AR-BIM enables precise spatial anchoring. Point at an element, add an annotation, capture the exact view. The issue stays attached to that specific BIM object forever. The camera position, view direction, and photo get stored with the annotation. Anyone reviewing later sees exactly what the annotator saw.

Annotation Data Structure

Each annotation packages multiple data types: text content (subject, message, deadline), assignment data (recipient and CC list for workflow routing), and spatial data (camera position, direction, field of view, and a photo).

The structure captures both human-readable description and machine-readable spatial data. Text for understanding. Camera data for precise positioning. The photo provides visual evidence that survives even if the physical condition changes.

Camera Data Capture

The camera data enables view recreation. Store view parameters at annotation time, later position a virtual camera identically. Four values fully specify camera state: viewPoint (position in world space), direction (forward vector), upVector (orientation), and fieldOfView (viewing angle).

public CameraData GetCameraData(Camera cam)
{
    return new CameraData
    {
        viewPoint = new float[] { cam.transform.position.x, 
            cam.transform.position.y, cam.transform.position.z },
        direction = new float[] { cam.transform.forward.x, 
            cam.transform.forward.y, cam.transform.forward.z },
        upVector = new float[] { cam.transform.up.x, 
            cam.transform.up.y, cam.transform.up.z },
        fieldOfView = (int)cam.fieldOfView
    };
}

Recreating the view is straightforward: set position from viewPoint, build rotation from direction and upVector using Quaternion.LookRotation, apply fieldOfView. The reviewer sees precisely what the annotator saw.

Photo Capture and Encoding

Visual documentation accompanies text descriptions. The system captures the camera feed at annotation time: copy framebuffer to texture with ReadPixels, compress to JPEG at 75% quality, Base64 encode for JSON transport, clean up the temporary texture.

The result embeds directly in JSON. One POST request uploads text, camera data, and image together. Quality 75% balances file size and visual fidelity. Construction photos don't need pixel-perfect accuracy. Issues must be visible. Lower quality reduces bandwidth on construction site networks.

Element Association

Annotations attach to specific BIM elements by ID. Raycasting determines which element the user is looking at:

public void SelectElement()
{
    Ray ray = mainCamera.ScreenPointToRay(new Vector3(
        Screen.width / 2, Screen.height / 2, 0));
    
    if (Physics.Raycast(ray, out hit, Mathf.Infinity, selectableLayerMask))
    {
        BimMetaData metadata = hit.collider.GetComponent<BimMetaData>();
        if (metadata != null)
            annotationUI.SetCurrentElementID(metadata.ElementID);
    }
}

The pattern is common in FPS games: replace gun with selection, same concept. A UI crosshair shows users where they're aiming. The center of screen becomes the selection point. Tap confirm, element gets selected, annotation dialog opens with element ID pre-populated.

UI Flow

The annotation workflow is multi-stage:

User points at BIM object, raycast determines which element. Take photo captures the camera feed showing the element. User confirms the photo shows the issue. Fill form with subject, message, deadline, and recipient. Validation ensures required fields are present and photo is attached. Upload POSTs to the API endpoint. Confirmation shows success or failure. Reset clears the form for the next annotation.

The save button stays disabled until a photo is captured. This prevents invalid submissions. After upload, the form clears automatically so each annotation starts fresh.

API Integration

Annotations POST to a REST endpoint as JSON. The coroutine execution ensures network operations don't block the UI. The endpoint receives element ID in the URL path and annotation data in the request body. The backend associates the annotation with the BIM element in the database, making it queryable by element, by project, by user, or by date.

Workflow Details

Deadlines use ISO 8601 format (2024-12-17T14:30:00Z) for consistency across systems. Minute-level precision is sufficient for construction deadlines. Consistent format prevents parsing errors across client/server boundaries.

Annotations route to responsible parties via employee dropdown. The dropdown shows employee names while storing IDs internally. The backend routes notifications to the appropriate person. A CC list enables additional notifications without changing the primary recipient.

Multiple validation points prevent invalid submissions: save button disabled without photo, required fields checked before submission, element ID validation, HTTP response code checking. Defensive programming provides defaults for empty fields rather than failing.

Why Spatial Anchoring Matters

Traditional issue reports have this problem: "Second floor mechanical room, pipe near north wall has corrosion." Which mechanical room? Which pipe? Which wall is north? The description depends on shared mental models. Different people interpret differently.

Spatial annotation eliminates ambiguity:

{
  "elementId": 4523,
  "subject": "Corrosion detected",
  "message": "Red-brown discoloration on pipe surface",
  "camera": {
    "viewPoint": [45.2, 12.8, -23.4],
    "direction": [0.707, -0.1, 0.707],
    "fieldOfView": 60
  }
}

Element 4523 is a specific pipe in the BIM model, no ambiguity. Camera data enables exact view recreation. The photo provides visual evidence.

Later, another person views the annotation. The system loads the BIM model, highlights element 4523, positions the camera from annotation data, shows the photo alongside the current view. The reviewer sees exactly what the annotator saw. Same element, same angle. If conditions changed, comparison is immediate and obvious.

This transforms issue tracking from text-based to spatially-anchored. Issues don't float in ambiguous space. They attach to specific BIM elements with precise viewing context.

Use Cases

Quality inspections document defects with photo and exact location. Change requests show what needs modification attached to the element. Progress tracking records completion state per element. Safety observations flag hazards attached to specific locations. Client walkthroughs capture feedback spatially referenced.

All leverage the same mechanism: element ID + camera data + photo + text. The spatial anchor makes everything precise.

Performance Considerations

Photo capture is expensive. ReadPixels forces GPU-CPU synchronization. The solution: only capture when user explicitly requests, wait for end of frame to ensure rendering is complete, then process JPEG encoding asynchronously where possible. This isolates the capture cost to explicit user action, preventing frame drops during normal AR usage.

Why This Enables Collaboration

AR-BIM without annotations is single-player. You see the BIM overlay, walk around, inspect. Useful, but isolated.

Annotations make it multiplayer. One person identifies an issue. Another reviews from the same perspective. A third person verifies the fix. All see identical spatial context.

The camera data plus photo combination means remote review works. The reviewer doesn't need to be on site. They see the annotator's view remotely, understand context, make decisions.

Result: AR transforms from visualization tool into collaborative platform. Field observations stay spatially anchored to exact BIM elements. No ambiguity. No lost context. Just precise, spatial issue tracking.