context-aware bim annotations

date: March 25, 2021

Problem: Construction issues get reported verbally, in scattered notes, through disconnected photos. "The pipe on the second floor near the south wall has a problem." Which pipe? Which wall? Traditional issue tracking loses spatial context. Weeks later, finding the exact element someone reported requires detective work.

Solution: AR-BIM enables precise spatial anchoring. Point at an element, add an annotation, capture the exact view. The issue stays attached to that specific BIM object. Camera position, view direction, and photo get stored with the annotation.

Camera data capture

The structure captures both human-readable description and machine-readable spatial data. Four values fully specify camera state: viewPoint (position), direction (forward vector), upVector (orientation), and fieldOfView.

public CameraData GetCameraData(Camera cam)
{
    return new CameraData
    {
        viewPoint = new float[] { cam.transform.position.x, 
            cam.transform.position.y, cam.transform.position.z },
        direction = new float[] { cam.transform.forward.x, 
            cam.transform.forward.y, cam.transform.forward.z },
        upVector = new float[] { cam.transform.up.x, 
            cam.transform.up.y, cam.transform.up.z },
        fieldOfView = (int)cam.fieldOfView
    };
}

Recreating the view: set position from viewPoint, build rotation using Quaternion.LookRotation, apply fieldOfView. The reviewer sees precisely what the annotator saw.

Photo capture

The system captures the camera feed at annotation time: copy framebuffer to texture with ReadPixels, compress to JPEG at 75% quality, Base64 encode for JSON transport. The result embeds directly in JSON. One POST request uploads text, camera data, and image together. Construction photos don't need pixel-perfect accuracy. Issues must be visible. Lower quality reduces bandwidth on construction site networks.

Photo capture is expensive. ReadPixels forces GPU-CPU synchronization. The solution: capture only when the user explicitly requests, wait for end of frame to ensure rendering is complete. This isolates the cost to explicit user action, preventing frame drops during normal AR usage.

Element association

Raycasting determines which BIM element the user is looking at:

public void SelectElement()
{
    Ray ray = mainCamera.ScreenPointToRay(new Vector3(
        Screen.width / 2, Screen.height / 2, 0));
    
    if (Physics.Raycast(ray, out hit, Mathf.Infinity, selectableLayerMask))
    {
        BimMetaData metadata = hit.collider.GetComponent<BimMetaData>();
        if (metadata != null)
            annotationUI.SetCurrentElementID(metadata.ElementID);
    }
}

A UI crosshair shows where users are aiming. Tap confirm, element gets selected, annotation dialog opens with element ID pre-populated.

Workflow

The save button stays disabled until a photo is captured. This prevents invalid submissions. Annotations route to responsible parties via employee dropdown, showing employee names while storing IDs internally. The backend routes notifications to the appropriate person. Deadlines use ISO 8601 format for consistency across client/server boundaries. After upload, the form clears automatically so each annotation starts fresh.

Spatial anchoring in practice

A stored annotation eliminates the ambiguity of text-based reporting:

{
  "elementId": 4523,
  "subject": "Corrosion detected",
  "message": "Red-brown discoloration on pipe surface",
  "camera": {
    "viewPoint": [45.2, 12.8, -23.4],
    "direction": [0.707, -0.1, 0.707],
    "fieldOfView": 60
  }
}

Element 4523 is a specific pipe in the BIM model. Camera data enables exact view recreation. When another person reviews, the system positions the camera from annotation data and shows the photo alongside the current view. Same element, same angle. If conditions changed, comparison is immediate.

All use cases rely on the same mechanism: element ID + camera data + photo + text. Quality inspections, change requests, progress tracking, safety observations. The spatial anchor makes everything precise.

Why this enables collaboration

Annotations make AR-BIM multiplayer. One person identifies an issue. Another reviews from the same perspective. A third verifies the fix. All see identical spatial context.

The camera data plus photo combination means remote review works. The reviewer doesn't need to be on site. They see the annotator's view, understand context, make decisions.

Result: AR transforms from visualization tool into collaborative platform. Field observations stay spatially anchored to exact BIM elements, eliminating the ambiguity of text-based reporting.