Data Flow

This document describes how event data flows through Atlas from source to display.

Overview

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Config    │───▶│  Adapters   │───▶│   Context   │───▶│     UI      │
│ map.config  │    │ Fetch data  │    │ State mgmt  │    │  Rendering  │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘

App Initialization

Atlas uses a unified initialization flow that loads data and initializes the map in parallel:

┌──────────────────────────────────────────────────────────────────────┐
│                          App Initialization                           │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   ┌─────────────┐                     ┌─────────────┐                │
│   │ DataProvider│                     │  MapEngine  │                │
│   │ isLoading   │     (parallel)      │  mapReady   │                │
│   └──────┬──────┘                     └──────┬──────┘                │
│          │                                   │                       │
│          ▼                                   ▼                       │
│   ┌─────────────┐                     ┌─────────────┐                │
│   │ Fetch data  │                     │ Mapbox init │                │
│   │ from source │                     │ onLoad()    │                │
│   └──────┬──────┘                     └──────┬──────┘                │
│          │                                   │                       │
│          └──────────────┬────────────────────┘                       │
│                         ▼                                            │
│              ┌──────────────────┐                                    │
│              │  appReady = true │                                    │
│              │  Fade in content │                                    │
│              └──────────────────┘                                    │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

The unified loader displays a single "Initializing..." message while both processes complete:

// page.tsx
const { isLoading } = useData();
const [mapReady, setMapReady] = useState(false);
const appReady = !isLoading && mapReady;

// Loader fades out, content fades in when appReady is true
<MapEngine onMapReady={() => setMapReady(true)} ... />

This approach provides:

Parallel loading — Data fetches while map initializes
Unified UX — Single loading state instead of sequential messages
Smooth transition — CSS fade-out/fade-in when ready

See LoaderConfig for customization options.

1. Configuration Loading

On application startup:

// map.config.ts is loaded
const config = await import('@/../map.config').then(m => m.default);

// Config is validated
const validatedConfig = validateConfig(config);

// Merged with defaults
const finalConfig = mergeConfig(validatedConfig);

2. Adapter Factory

Adapters are created dynamically via a factory function:

// Factory creates the appropriate adapter based on type
function createAdapter(source: DataSourceConfig): DataAdapter {
  switch (source.adapter) {
    case 'static':
      return new StaticAdapter(source);
    case 'supabase':
      return new SupabaseAdapter(source);
    case 'api':
      return new ApiAdapter(source);
    default:
      throw new Error(`Unknown adapter type: ${source.adapter}`);
  }
}

Adapter Interface

All adapters implement:

interface DataAdapter {
  readonly id: string;
  readonly name: string;
  
  getMapEvents(filters?: MapEventFilters): Promise<MapEvent[]>;
  getMapEventById(id: string): Promise<MapEvent | null>;
  getFilterOptions(): Promise<FilterOptions>;
  clearCache?(): void;
}

3. Data Fetching

When the app loads or sources change, adapters are created and fetched in one operation:

// Create adapters and fetch from all enabled sources
const fetchPromises = enabledSources.map(async (source) => {
  const adapter = createAdapter(source);
  const mapEvents = await adapter.getMapEvents();
  
  // Tag each event with its source ID for filtering/attribution
  return mapEvents.map((evt) => ({
    ...evt,
    dataSourceId: source.id,
  }));
});

// Wait for all to complete (partial failures allowed)
const results = await Promise.allSettled(fetchPromises);

// Combine successful results
const mapEvents = results
  .filter(r => r.status === 'fulfilled')
  .flatMap(r => r.value);

The dataSourceId tag allows events to be filtered by source and enables proper color/attribution in the UI.

Error Handling

// Partial failures are handled gracefully
const failed = results.filter(r => r.status === 'rejected');
const succeeded = results.filter(r => r.status === 'fulfilled');

// Only set error if ALL sources fail
if (failed.length > 0 && succeeded.length === 0) {
  setError('All data sources failed to load');
} else if (failed.length > 0) {
  // Log warning for partial failures
  console.warn(`${failed.length} source(s) failed to load`);
}

4. Data Normalization

Raw events are transformed to display format using toMapEventDisplay:

function toMapEventDisplay(mapEvent: MapEvent): MapEventDisplay {
  const sensitivity = getMapEventSensitivity(mapEvent);
  const showSensitivity = sensitivity && sensitivity !== 'none';
  const eventDate = new Date(mapEvent.temporal.date);
  const now = new Date();
  const ageInDays = Math.floor((now.getTime() - eventDate.getTime()) / (1000 * 60 * 60 * 24));

  return {
    ...mapEvent,
    // Extract coordinates for map rendering
    coordinates: [mapEvent.location.longitude, mapEvent.location.latitude],
    // Add sensitivity display properties (only if set and not 'none')
    sensitivityColor: showSensitivity ? LOCATION_SENSITIVITY_COLORS[sensitivity] : undefined,
    sensitivityLabel: showSensitivity ? LOCATION_SENSITIVITY_LABELS[sensitivity] : undefined,
    // Add formatted date
    formattedDate: eventDate.toLocaleDateString('en-US', {
      year: 'numeric',
      month: 'long',
      day: 'numeric',
    }),
    // Calculate age for filtering/sorting
    ageInDays,
  };
}

5. Context State

Data is stored in React context:

interface DataContextValue {
  // Data
  mapEvents: MapEventDisplay[];
  filteredMapEvents: MapEventDisplay[];
  isLoading: boolean;
  error: Error | null;

  // Data sources
  dataSources: DataSourceConfig[];
  enabledSourceIds: string[];
  toggleSource: (sourceId: string) => void;

  // Filters
  filters: MapEventFilters;
  setFilters: (filters: MapEventFilters | ((prev: MapEventFilters) => MapEventFilters)) => void;
  updateFilter: <K extends keyof MapEventFilters>(key: K, value: MapEventFilters[K]) => void;
  resetFilters: () => void;

  // Filter options (derived from data)
  filterOptions: {
    countries: string[];
    siteTypes: string[];
    locationSensitivities: LocationSensitivity[];
    eventTypes: string[];
    // ... more options
  };

  // Selection
  selectedMapEvent: MapEventDisplay | null;
  setSelectedMapEvent: (mapEvent: MapEventDisplay | null) => void;

  // Timeline
  currentDate: Date | null;
  setCurrentDate: (date: Date | null) => void;
  dateRange: { min: Date; max: Date } | null;

  // Actions
  refresh: () => Promise<void>;
}

6. Filtering Pipeline

When filters change, the filterMapEvents utility is called:

// Filter pipeline in data-context.tsx
const filteredMapEvents = useMemo(() => {
  // First filter by enabled data sources
  const sourceFiltered = mapEvents.filter(i => 
    enabledSourceIds.includes(i.dataSourceId)
  );
  
  // Then apply all filters via the filterMapEvents utility
  return filterMapEvents(sourceFiltered, filters, currentDate);
}, [mapEvents, filters, enabledSourceIds, currentDate]);

The filterMapEvents utility uses modular filter checks:

// filterMapEvents signature
export function filterMapEvents(
  mapEvents: MapEventDisplay[],
  filters: MapEventFilters,
  currentDate: Date | null = null
): MapEventDisplay[]

// Internally uses modular checks:
// - passesLocationFilters(mapEvent, filters)
// - passesClassificationFilters(mapEvent, filters)
// - passesEvidenceFilters(mapEvent, filters)
// - passesInvestigationFilters(mapEvent, filters)
// - passesQuickFilters(mapEvent, filters)
// - passesObjectFilters(mapEvent, filters)
// - passesTimelineFilter(mapEvent, currentDate)

7. Component Consumption

Components access data via hooks:

function MapEngine() {
  const { filteredMapEvents, selectedMapEvent } = useData();
  
  return (
    <Map>
      {filteredMapEvents.map(mapEvent => (
        <Marker key={mapEvent.id} mapEvent={mapEvent} />
      ))}
    </Map>
  );
}

8. User Interactions

User actions update context state:

// Select an event
const handleMarkerClick = (mapEvent: MapEventDisplay) => {
  setSelectedMapEvent(mapEvent);
  flyTo(mapEvent.location);
};

// Update filters
const handleFilterChange = (key: string, value: any) => {
  updateFilter(key, value);
};

// Toggle data source
const handleSourceToggle = (sourceId: string) => {
  toggleSource(sourceId);
};

Data Flow Diagram

User Action
    │
    ▼
┌─────────────────────────────────────────────────┐
│                  DataContext                     │
│                                                  │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐    │
│  │  Events  │──▶│ Filters  │──▶│ Filtered │    │
│  │   Raw    │   │  State   │   │  Events  │    │
│  └──────────┘   └──────────┘   └──────────┘    │
│                      │                          │
└──────────────────────│──────────────────────────┘
                       │
    ┌──────────────────┼──────────────────┐
    │                  │                  │
    ▼                  ▼                  ▼
┌────────┐        ┌────────┐        ┌────────┐
│  Map   │        │ Feed   │        │Timeline│
│ Engine │        │        │        │        │
└────────┘        └────────┘        └────────┘

Performance Optimizations

Memoization

// Expensive computations are memoized
const filteredMapEvents = useMemo(() => {
  return applyFilters(mapEvents, filters);
}, [mapEvents, filters]);

// Callbacks are stable
const updateFilter = useCallback((key, value) => {
  setFilters(prev => ({ ...prev, [key]: value }));
}, []);

Lazy Loading

// Data is fetched on mount, not import
useEffect(() => {
  fetchMapEvents();
}, [enabledSources]);

Virtualization

For large datasets, the event feed uses virtualization:

// Only render visible items
<VirtualList
  items={filteredMapEvents}
  itemHeight={80}
  renderItem={(mapEvent) => <FeedItem mapEvent={mapEvent} />}
/>

Debugging

Enable debug logging:

// In browser console
localStorage.setItem('atlas-debug', 'true');

This logs:

Data fetches and their results
Filter changes and computed results
Selection events
Render cycles

Data Flow

This document describes how event data flows through Atlas from source to display.

Overview

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Config    │───▶│  Adapters   │───▶│   Context   │───▶│     UI      │
│ map.config  │    │ Fetch data  │    │ State mgmt  │    │  Rendering  │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘

App Initialization

Atlas uses a unified initialization flow that loads data and initializes the map in parallel:

┌──────────────────────────────────────────────────────────────────────┐
│                          App Initialization                           │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   ┌─────────────┐                     ┌─────────────┐                │
│   │ DataProvider│                     │  MapEngine  │                │
│   │ isLoading   │     (parallel)      │  mapReady   │                │
│   └──────┬──────┘                     └──────┬──────┘                │
│          │                                   │                       │
│          ▼                                   ▼                       │
│   ┌─────────────┐                     ┌─────────────┐                │
│   │ Fetch data  │                     │ Mapbox init │                │
│   │ from source │                     │ onLoad()    │                │
│   └──────┬──────┘                     └──────┬──────┘                │
│          │                                   │                       │
│          └──────────────┬────────────────────┘                       │
│                         ▼                                            │
│              ┌──────────────────┐                                    │
│              │  appReady = true │                                    │
│              │  Fade in content │                                    │
│              └──────────────────┘                                    │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

The unified loader displays a single "Initializing..." message while both processes complete:

// page.tsx
const { isLoading } = useData();
const [mapReady, setMapReady] = useState(false);
const appReady = !isLoading && mapReady;

// Loader fades out, content fades in when appReady is true
<MapEngine onMapReady={() => setMapReady(true)} ... />

This approach provides:

Parallel loading — Data fetches while map initializes
Unified UX — Single loading state instead of sequential messages
Smooth transition — CSS fade-out/fade-in when ready

See LoaderConfig for customization options.

1. Configuration Loading

On application startup:

// map.config.ts is loaded
const config = await import('@/../map.config').then(m => m.default);

// Config is validated
const validatedConfig = validateConfig(config);

// Merged with defaults
const finalConfig = mergeConfig(validatedConfig);

2. Adapter Factory

Adapters are created dynamically via a factory function:

// Factory creates the appropriate adapter based on type
function createAdapter(source: DataSourceConfig): DataAdapter {
  switch (source.adapter) {
    case 'static':
      return new StaticAdapter(source);
    case 'supabase':
      return new SupabaseAdapter(source);
    case 'api':
      return new ApiAdapter(source);
    default:
      throw new Error(`Unknown adapter type: ${source.adapter}`);
  }
}

Adapter Interface

All adapters implement:

interface DataAdapter {
  readonly id: string;
  readonly name: string;
  
  getMapEvents(filters?: MapEventFilters): Promise<MapEvent[]>;
  getMapEventById(id: string): Promise<MapEvent | null>;
  getFilterOptions(): Promise<FilterOptions>;
  clearCache?(): void;
}

3. Data Fetching

When the app loads or sources change, adapters are created and fetched in one operation:

// Create adapters and fetch from all enabled sources
const fetchPromises = enabledSources.map(async (source) => {
  const adapter = createAdapter(source);
  const mapEvents = await adapter.getMapEvents();
  
  // Tag each event with its source ID for filtering/attribution
  return mapEvents.map((evt) => ({
    ...evt,
    dataSourceId: source.id,
  }));
});

// Wait for all to complete (partial failures allowed)
const results = await Promise.allSettled(fetchPromises);

// Combine successful results
const mapEvents = results
  .filter(r => r.status === 'fulfilled')
  .flatMap(r => r.value);

The dataSourceId tag allows events to be filtered by source and enables proper color/attribution in the UI.

Error Handling

// Partial failures are handled gracefully
const failed = results.filter(r => r.status === 'rejected');
const succeeded = results.filter(r => r.status === 'fulfilled');

// Only set error if ALL sources fail
if (failed.length > 0 && succeeded.length === 0) {
  setError('All data sources failed to load');
} else if (failed.length > 0) {
  // Log warning for partial failures
  console.warn(`${failed.length} source(s) failed to load`);
}

4. Data Normalization

Raw events are transformed to display format using toMapEventDisplay:

function toMapEventDisplay(mapEvent: MapEvent): MapEventDisplay {
  const sensitivity = getMapEventSensitivity(mapEvent);
  const showSensitivity = sensitivity && sensitivity !== 'none';
  const eventDate = new Date(mapEvent.temporal.date);
  const now = new Date();
  const ageInDays = Math.floor((now.getTime() - eventDate.getTime()) / (1000 * 60 * 60 * 24));

  return {
    ...mapEvent,
    // Extract coordinates for map rendering
    coordinates: [mapEvent.location.longitude, mapEvent.location.latitude],
    // Add sensitivity display properties (only if set and not 'none')
    sensitivityColor: showSensitivity ? LOCATION_SENSITIVITY_COLORS[sensitivity] : undefined,
    sensitivityLabel: showSensitivity ? LOCATION_SENSITIVITY_LABELS[sensitivity] : undefined,
    // Add formatted date
    formattedDate: eventDate.toLocaleDateString('en-US', {
      year: 'numeric',
      month: 'long',
      day: 'numeric',
    }),
    // Calculate age for filtering/sorting
    ageInDays,
  };
}

5. Context State

Data is stored in React context:

interface DataContextValue {
  // Data
  mapEvents: MapEventDisplay[];
  filteredMapEvents: MapEventDisplay[];
  isLoading: boolean;
  error: Error | null;

  // Data sources
  dataSources: DataSourceConfig[];
  enabledSourceIds: string[];
  toggleSource: (sourceId: string) => void;

  // Filters
  filters: MapEventFilters;
  setFilters: (filters: MapEventFilters | ((prev: MapEventFilters) => MapEventFilters)) => void;
  updateFilter: <K extends keyof MapEventFilters>(key: K, value: MapEventFilters[K]) => void;
  resetFilters: () => void;

  // Filter options (derived from data)
  filterOptions: {
    countries: string[];
    siteTypes: string[];
    locationSensitivities: LocationSensitivity[];
    eventTypes: string[];
    // ... more options
  };

  // Selection
  selectedMapEvent: MapEventDisplay | null;
  setSelectedMapEvent: (mapEvent: MapEventDisplay | null) => void;

  // Timeline
  currentDate: Date | null;
  setCurrentDate: (date: Date | null) => void;
  dateRange: { min: Date; max: Date } | null;

  // Actions
  refresh: () => Promise<void>;
}

6. Filtering Pipeline

When filters change, the filterMapEvents utility is called:

// Filter pipeline in data-context.tsx
const filteredMapEvents = useMemo(() => {
  // First filter by enabled data sources
  const sourceFiltered = mapEvents.filter(i => 
    enabledSourceIds.includes(i.dataSourceId)
  );
  
  // Then apply all filters via the filterMapEvents utility
  return filterMapEvents(sourceFiltered, filters, currentDate);
}, [mapEvents, filters, enabledSourceIds, currentDate]);

The filterMapEvents utility uses modular filter checks:

// filterMapEvents signature
export function filterMapEvents(
  mapEvents: MapEventDisplay[],
  filters: MapEventFilters,
  currentDate: Date | null = null
): MapEventDisplay[]

// Internally uses modular checks:
// - passesLocationFilters(mapEvent, filters)
// - passesClassificationFilters(mapEvent, filters)
// - passesEvidenceFilters(mapEvent, filters)
// - passesInvestigationFilters(mapEvent, filters)
// - passesQuickFilters(mapEvent, filters)
// - passesObjectFilters(mapEvent, filters)
// - passesTimelineFilter(mapEvent, currentDate)

7. Component Consumption

Components access data via hooks:

function MapEngine() {
  const { filteredMapEvents, selectedMapEvent } = useData();
  
  return (
    <Map>
      {filteredMapEvents.map(mapEvent => (
        <Marker key={mapEvent.id} mapEvent={mapEvent} />
      ))}
    </Map>
  );
}

8. User Interactions

User actions update context state:

// Select an event
const handleMarkerClick = (mapEvent: MapEventDisplay) => {
  setSelectedMapEvent(mapEvent);
  flyTo(mapEvent.location);
};

// Update filters
const handleFilterChange = (key: string, value: any) => {
  updateFilter(key, value);
};

// Toggle data source
const handleSourceToggle = (sourceId: string) => {
  toggleSource(sourceId);
};

Data Flow Diagram

User Action
    │
    ▼
┌─────────────────────────────────────────────────┐
│                  DataContext                     │
│                                                  │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐    │
│  │  Events  │──▶│ Filters  │──▶│ Filtered │    │
│  │   Raw    │   │  State   │   │  Events  │    │
│  └──────────┘   └──────────┘   └──────────┘    │
│                      │                          │
└──────────────────────│──────────────────────────┘
                       │
    ┌──────────────────┼──────────────────┐
    │                  │                  │
    ▼                  ▼                  ▼
┌────────┐        ┌────────┐        ┌────────┐
│  Map   │        │ Feed   │        │Timeline│
│ Engine │        │        │        │        │
└────────┘        └────────┘        └────────┘

Performance Optimizations

Memoization

// Expensive computations are memoized
const filteredMapEvents = useMemo(() => {
  return applyFilters(mapEvents, filters);
}, [mapEvents, filters]);

// Callbacks are stable
const updateFilter = useCallback((key, value) => {
  setFilters(prev => ({ ...prev, [key]: value }));
}, []);

Lazy Loading

// Data is fetched on mount, not import
useEffect(() => {
  fetchMapEvents();
}, [enabledSources]);

Virtualization

For large datasets, the event feed uses virtualization:

// Only render visible items
<VirtualList
  items={filteredMapEvents}
  itemHeight={80}
  renderItem={(mapEvent) => <FeedItem mapEvent={mapEvent} />}
/>

Debugging

Enable debug logging:

// In browser console
localStorage.setItem('atlas-debug', 'true');

This logs:

Data fetches and their results
Filter changes and computed results
Selection events
Render cycles

Data Flow

Overview

App Initialization

1. Configuration Loading

2. Adapter Factory

Adapter Interface

3. Data Fetching

Error Handling

4. Data Normalization

5. Context State

6. Filtering Pipeline

7. Component Consumption

8. User Interactions

Data Flow Diagram

Performance Optimizations

Memoization

Lazy Loading

Virtualization

Debugging

On this page

Data Flow

Overview

App Initialization

1. Configuration Loading

2. Adapter Factory

Adapter Interface

3. Data Fetching

Error Handling

4. Data Normalization

5. Context State

6. Filtering Pipeline

7. Component Consumption

8. User Interactions

Data Flow Diagram

Performance Optimizations

Memoization

Lazy Loading

Virtualization

Debugging

On this page