# IFrameStream Refactoring Plan - Template Elimination

## Session 10 - 2025-12-20

### Problem Statement
The `IFrameStreamWrapper<SPEC, ...>` template has parameter `SPEC` hardcoded to 9 at compile time, causing v3 frame files to be read with wrong `StreamRef` structure (16 bytes instead of 8 bytes). This breaks `test_memory_buffer_ro` for v3 files.

**Root Cause**: Template instantiation happens at compile time, but frame version is only known at runtime when reading the file header.

### Key Insight
After Universal namespace migration, **all** frame object types (FrAdcData, FrameH, FrProcData, etc.) are identical across versions - they all use `Universal::` namespace. The **only** version-specific dependency is the `StreamRef` wire format:
- v3-v4: `StreamRef<INT_4U, INT_2U, INT_2U>` = 8 bytes (4+2+2)
- v5+: `StreamRef<INT_8U, INT_4U, INT_4U>` = 16 bytes (8+4+4)

Runtime version detection already exists via:
```cpp
frameSpecInfo(FrameSpec::SpecInfo(version));  // Sets up version-specific objects
ReferenceStreamRef() -> returns correct StreamRef* for detected version
```

### Solution Strategy
**Incremental Approach**: Add new auto-detecting class alongside existing template, verify it works everywhere, then remove old template code.

---

## Phase 1: Create IFrameStreamAuto Class

### File: `src/Common/FrameStreamWrapper.hh`

**Add NEW class after existing template** (don't modify template yet):

```cpp
//=====================================================================
// Existing template - KEEP UNCHANGED for now
//=====================================================================
template <DECL>
class IFrameStreamWrapper : public IFrameStream {
    // ... existing implementation unchanged ...
};

//=====================================================================
// NEW: Auto-detecting version (no template, no version parameter)
//=====================================================================
class IFrameStreamAuto : public IFrameStream
{
public:
    typedef INT_4U frame_offset_type;

    // All types use Universal namespace
    typedef boost::shared_ptr<Universal::FrameH>    frame_h_type;
    typedef boost::shared_ptr<Universal::FrAdcData> fr_adc_data_type;
    typedef boost::shared_ptr<Universal::FrEvent>   fr_event_type;
    typedef INT_4U                                  fr_event_offset_type;
    typedef boost::shared_ptr<Universal::FrProcData> fr_proc_data_type;
    typedef boost::shared_ptr<Universal::FrRawData>  fr_raw_data_type;
    typedef boost::shared_ptr<Universal::FrSerData>  fr_ser_data_type;
    typedef boost::shared_ptr<Universal::FrSimData>  fr_sim_data_type;
    typedef boost::shared_ptr<Universal::FrSimEvent> fr_sim_event_type;
    typedef INT_4U fr_sim_event_offset_type;

    typedef IFrameStream::buffer_type buffer_type;

    //------------------------------------------------------------------
    // Constructors - ALWAYS auto-detect (Version=0 to base class)
    //------------------------------------------------------------------
    IFrameStreamAuto(buffer_type* Buffer);
    IFrameStreamAuto(bool AutoDelete, buffer_type* Buffer);

    // NO Version parameter - auto-detection is mandatory

    //------------------------------------------------------------------
    // All Read methods inherited from base IFrameStream
    // Return types already use Universal:: from base class implementation
    //------------------------------------------------------------------
};
```

**Implementation in same file** (after class definition):

```cpp
inline IFrameStreamAuto::IFrameStreamAuto(buffer_type* Buffer)
    : IFrameStream(Buffer, 0)  // Version=0 enables auto-detection
{
}

inline IFrameStreamAuto::IFrameStreamAuto(bool AutoDelete, buffer_type* Buffer)
    : IFrameStream(AutoDelete, Buffer, 0)  // Version=0 enables auto-detection
{
}
```

**Build Test**: Verify library compiles with new class added.

---

## Phase 2: Add Test Typedef in Version9

### File: `src/Version9/FrameStream.hh`

**Add second typedef alongside existing one**:

```cpp
namespace FrameCPP {
namespace Version_9 {
    // Existing template-based typedef - KEEP for now
    typedef Common::IFrameStreamWrapper<9,
                                        Universal::FrAdcData,
                                        Universal::FrameH,
                                        Universal::FrProcData,
                                        Universal::FrSerData,
                                        Universal::FrSimData,
                                        Universal::FrRawData,
                                        Universal::FrEvent,
                                        Universal::FrSimEvent>
        IFrameStream;

    // NEW: Auto-detecting version for testing
    typedef Common::IFrameStreamAuto IFrameStreamAuto;

} // namespace Version_9
} // namespace FrameCPP
```

**Build Test**: Verify library compiles with typedef added.

---

## Phase 3: Update test_memory_buffer_ro

### File: `test/test_frameCPP_cpp.cc`

**Current code** (around line 390-407):
```cpp
ReadOnlyMemoryBuffer mb;
mb.pubsetbuf(buf.get(), stat_buf.st_size);

IFrameStream ifs(false, &mb, 0);  // Uses template with SPEC=9
```

**NEW code - use auto-detecting class**:
```cpp
ReadOnlyMemoryBuffer mb;
mb.pubsetbuf(buf.get(), stat_buf.st_size);

// Test new auto-detecting class (no version parameter needed!)
FrameCPP::Version_9::IFrameStreamAuto ifs(false, &mb);
```

**Test Commands**:
```bash
cd /Users/efm/build/LDAS/ldas-tools/u/framecpp-prefix/src/framecpp-build
cmake --build .
./test/test_frameCPP_cpp --run_test=test_memory_buffer_ro
```

**Expected Result**: Test PASSES for v9 (confirms new class works for known-good case).

---

## Phase 4: Test with Version 3 Frame Files (Critical!)

**MUST DO THIS IMMEDIATELY AFTER PHASE 3**

### Purpose
Verify the new `IFrameStreamAuto` class fixes the v3 MemoryBuffer reading bug - this is the **primary reason** for the refactoring.

### Test File Location
```bash
# Find v3 test file
ls /Users/efm/build/LDAS/ldas-tools/u/framecpp-prefix/src/framecpp-build/src/Utilities/Z-R_std_test_frame_ver3-*.gwf
```

### Test Commands
```bash
cd /Users/efm/build/LDAS/ldas-tools/u/framecpp-prefix/src/framecpp-build

# Run test_memory_buffer_ro - should now PASS for v3
./test/test_frameCPP_cpp --run_test=test_memory_buffer_ro --log_level=message

# Check for v3-specific output
grep "file source version: 3" Testing/Temporary/LastTest.log

# Verify no StreamRef errors
grep "Unable to map from the stream class id" Testing/Temporary/LastTest.log || echo "SUCCESS: No StreamRef errors!"
```

### Expected Results

**Before (with template)**:
```
test_memory_buffer_ro: file source version: 3 - starting iteration
test_memory_buffer_ro: file source version: 3: Caught an exception:
  Unable to map from the stream class id (7) to internal class id
  at position: 17839861 [streamref: class: -2 instance: 0 length: 16]
*** 1 failure is detected in the test module
```

**After (with IFrameStreamAuto)**:
```
test_memory_buffer_ro: file source version: 3 - starting iteration
test_memory_buffer_ro: file source version: 3: firstAdc count: 229
test_memory_buffer_ro: file source version: 3: FrProcData count: 1
test_memory_buffer_ro: file source version: 3 - completed iteration
✅ Test PASSES
```

### Debug Verification
If test passes, verify StreamRef structure is correct:

```bash
# Add debug output to test to show StreamRef size
# Expected: v3 uses 8-byte StreamRef (not 16-byte)
```

### Success Criteria
- ✅ test_memory_buffer_ro PASSES for v3 file (no exceptions)
- ✅ v3 FrProcData is read successfully
- ✅ No "Unable to map" errors in log
- ✅ Correct 8-byte StreamRef structure used

### If Phase 4 Fails
**STOP and debug** - this is the core bug we're fixing. Possible issues:
- Version detection not working (check `frameSpecInfo()` calls)
- StreamRef still using wrong structure (check `ReferenceStreamRef()`)
- File vs MemoryBuffer difference (check buffer initialization)

Do NOT proceed to Phase 5 until v3 works!

---

## Phase 5: Verify All Usage Points

### Current Usage Locations (14 files)

**Version Namespace Typedefs** (add IFrameStreamAuto to each):
- [ ] `src/Version5/FrameStream.hh`
- [ ] `src/Version6/FrameStream.hh`
- [ ] `src/Version7/FrameStream.hh`
- [ ] `src/Version8/FrameStream.hh`
- [x] `src/Version9/FrameStream.hh` (done in Phase 2)

**Public API**:
- [ ] `src/OOInterface/IFrameStream.hh` - Check if changes needed
- [ ] `src/OOInterface/IFrameStreamPlan.hh` - Check if plan class needs update
- [ ] `src/Common/FrameStreamPlanWrapper.hh` - Check if wrapper needs update

**Utilities** (may have custom usage):
- [ ] `src/Utilities/FrDumpObjects.cc` - Has custom FrameStream class (likely ignore)
- [ ] `src/Utilities/framecpp_inspector.cc` - Check usage
- [ ] `src/Utilities/framecmp.cc` - Check usage

**Tests**:
- [x] `test/test_frameCPP_cpp.cc` - Modified in Phase 3
- [ ] `test/FrQuery.cc` - Check usage

**Build Files**:
- [ ] `src/OOInterface/CMakeLists.txt` - Likely just comments

**Action for each**: Add `typedef Common::IFrameStreamAuto IFrameStreamAuto;` to version namespaces, verify compilation.

---

## Phase 6: Comprehensive Testing

**Prerequisites**: Phases 3, 4, and 5 all successful

### Test Matrix

| Test Name | Input Type | Versions Tested | Expected Result | Status |
|-----------|------------|-----------------|-----------------|--------|
| test_memory_buffer_ro | MemoryBuffer | v3 (critical!) | PASS (was FAIL) | TODO |
| test_memory_buffer_ro | MemoryBuffer | v4, v6, v7, v8, v9 | PASS | TODO |
| test_standard | File | v9 | PASS | TODO |
| test_small_frames | File | v9 | PASS | TODO |
| test_read_events | File | v9 | PASS | TODO |
| Full test suite | All | All | 462/462 PASS | TODO |

### Test Commands
```bash
cd /Users/efm/build/LDAS/ldas-tools/u/framecpp-prefix/src/framecpp-build

# Test each version's MemoryBuffer reading
./test/test_frameCPP_cpp --run_test=test_memory_buffer_ro --log_level=message

# All frameCPP tests
ctest -R test_frameCPP -V

# Full test suite (462 tests)
ctest

# Verify no regressions
ctest --rerun-failed
```

### Version-Specific Verification
```bash
# Check which versions were tested
grep "file source version:" Testing/Temporary/LastTest.log | sort -u

# Expected output (all versions):
# file source version: 3
# file source version: 4
# file source version: 6
# file source version: 7
# file source version: 8
# file source version: 9
```

### Success Criteria
- ✅ test_memory_buffer_ro PASSES for **v3** (was FAILING - critical fix!)
- ✅ test_memory_buffer_ro PASSES for v4, v6-v9 (no regression)
- ✅ All 4 test_frameCPP_cpp tests PASS
- ✅ Full test suite: 462/462 tests PASS
- ✅ No performance regression (< 5% slower)
- ✅ No memory leaks detected

### Performance Baseline
Record timings for comparison:
```bash
time ./test/test_frameCPP_cpp --run_test=test_memory_buffer_ro
# Baseline with template: ~0.12s
# With IFrameStreamAuto: ~0.12s (should be similar)
```

---

## Phase 7: Migrate Version Typedefs (After Phase 6 Passes)

**ONLY PROCEED IF PHASE 6 IS 100% SUCCESSFUL**

### Files: `src/Version{5,6,7,8,9}/FrameStream.hh`

**Change each version's typedef** (one at a time, test between changes):

**Before**:
```cpp
typedef Common::IFrameStreamWrapper<9, Universal::FrAdcData, ...> IFrameStream;
```

**After**:
```cpp
typedef Common::IFrameStreamAuto IFrameStream;
```

**Migration Order**:
1. Version9 (most tested)
2. Version8
3. Version7
4. Version6
5. Version5

**After each change**: Run full test suite, verify utilities work.

---

## Phase 8: Update Public API Typedef (After Phase 7)

### File: `src/OOInterface/IFrameStream.hh`

**Current**:
```cpp
namespace FrameCPP {
    typedef Version::IFrameStream IFrameStream;
    // ... rest of IFrameFStream class ...
}
```

**Analysis**: This typedef automatically picks up the new definition through the Version namespace alias. **No change needed** - verify it works.

**Test**: Compile all utilities, run integration tests.

---

## Phase 9: Remove Old Template (Final Step)

**ONLY AFTER**:
- ✅ All version typedefs migrated (Phase 7)
- ✅ Public API verified working (Phase 8)
- ✅ Full test suite passes for 1+ week
- ✅ All utilities tested on production data

### File: `src/Common/FrameStreamWrapper.hh`

**Actions**:
1. Rename `IFrameStreamAuto` → `IFrameStreamWrapper`
2. Remove old template definition entirely
3. Update all remaining typedefs to use non-template name

**Verification**:
- Full rebuild from clean state
- All 462 tests pass
- All utilities compile and run
- Check for any external dependencies (framecpp-swig, etc.)

---

## Phase 10: Fix Error Message Bug (Independent)

**Can be done anytime** - independent of refactoring

### File: `src/Common/FrameStream.cc:2408`

**Current**:
```cpp
msg << " [streamref: class: " << retval->GetClass()
    << " instance: " << retval->GetInstance()
    << " length: " << retval->GetLength() << " ]";
```

**Fixed**:
```cpp
msg << " [streamref: class: " << retval->GetStreamClass()  // Changed!
    << " instance: " << retval->GetInstance()
    << " length: " << retval->GetLength() << " ]";
```

**Why**: `GetClass()` returns FSI_COMMON_ELEMENTS (-2), but we want the actual class field value from the StreamRef structure for debugging.

**Expected output change**:
- Before: `streamref: class: -2 instance: 0 length: 16`
- After: `streamref: class: 7 instance: -2 length: 16`  (correct!)

---

## Rollback Procedures

### If Phase 1-2 Fails
- Remove new class definition
- Remove new typedef
- No impact on existing code

### If Phase 3 Fails
- Revert test changes (v9 not working)
- Keep new class for future attempts
- Investigate why auto-detection failed

### If Phase 4 Fails
**CRITICAL** - This is the main bug we're fixing!
- Debug StreamRef structure selection
- Check version detection logic
- Compare file vs MemoryBuffer paths
- Do NOT proceed until v3 works

### If Phase 5-6 Fails
- Revert typedef changes
- Keep new class and test changes
- Analyze test failures before proceeding

### If Phase 7+ Fails
- Revert individual typedef changes
- Return to last known-good state
- Template still available as fallback

---

## Implementation Checklist

### Phase 1: Create IFrameStreamAuto
- [ ] Add class definition to FrameStreamWrapper.hh
- [ ] Add inline constructor implementations
- [ ] Build library
- [ ] Verify no compilation errors

### Phase 2: Add Test Typedef
- [ ] Update Version9/FrameStream.hh
- [ ] Build library
- [ ] Verify typedef resolves correctly

### Phase 3: Update Test
- [ ] Modify test_frameCPP_cpp.cc
- [ ] Build test suite
- [ ] Run test_memory_buffer_ro
- [ ] Verify v9 PASSES (known-good baseline)

### Phase 4: Test Version 3 Files (CRITICAL!)
- [ ] Run test_memory_buffer_ro after Phase 3
- [ ] Verify v3 iteration PASSES (no exceptions)
- [ ] Check log for "file source version: 3 - completed"
- [ ] Confirm no "Unable to map" errors
- [ ] **STOP if this fails** - debug before proceeding

### Phase 5: Add Typedefs to All Versions
- [ ] Version5/FrameStream.hh
- [ ] Version6/FrameStream.hh
- [ ] Version7/FrameStream.hh
- [ ] Version8/FrameStream.hh
- [ ] Build and verify each

### Phase 6: Comprehensive Testing
- [ ] Run test_memory_buffer_ro (all versions v3-v9)
- [ ] Run all test_frameCPP_cpp tests
- [ ] Run full ctest suite (462 tests)
- [ ] Test utilities: framecpp_dump_objects, framecmp, etc.
- [ ] Record performance baseline
- [ ] Document any failures

### Phase 7: Migrate Typedefs (One at a Time)
- [ ] Version9: Change typedef, test, commit
- [ ] Version8: Change typedef, test, commit
- [ ] Version7: Change typedef, test, commit
- [ ] Version6: Change typedef, test, commit
- [ ] Version5: Change typedef, test, commit

### Phase 8: Verify Public API
- [ ] Check IFrameStream.hh works unchanged
- [ ] Test IFrameFStream usage
- [ ] Test utilities compilation
- [ ] Integration tests

### Phase 9: Remove Template (Final)
- [ ] Wait 1+ week of testing
- [ ] Rename IFrameStreamAuto → IFrameStreamWrapper
- [ ] Remove old template definition
- [ ] Full rebuild and test
- [ ] Update documentation

### Phase 10: Fix Error Message
- [ ] Change GetClass() to GetStreamClass()
- [ ] Test error message output
- [ ] Verify improved debugging info

---

## Key Files Reference

### Modified Files (Implementation)
- `src/Common/FrameStreamWrapper.hh` - Add IFrameStreamAuto class
- `src/Version9/FrameStream.hh` - Add IFrameStreamAuto typedef (Phase 2)
- `src/Version{5,6,7,8}/FrameStream.hh` - Add typedefs (Phase 4)
- `test/test_frameCPP_cpp.cc` - Use new class (Phase 3)

### Modified Files (Later Phases)
- `src/Version{5,6,7,8,9}/FrameStream.hh` - Change typedef (Phase 6)
- `src/Common/FrameStreamWrapper.hh` - Remove template (Phase 8)
- `src/Common/FrameStream.cc` - Fix error message (Phase 9)

### Files to Monitor
- `src/OOInterface/IFrameStream.hh` - Public API
- `src/OOInterface/IFrameStreamPlan.hh` - Plan wrapper
- `src/Common/FrameStreamPlanWrapper.hh` - Plan template
- All utility files in `src/Utilities/`
- All test files

---

## Technical Details

### Version Detection Flow
1. Constructor receives `Version=0` (auto-detect mode)
2. `IFrameStream` base constructor calls `frameSpecInfo(FrameSpec::SpecInfo(0))`
3. In `loadFrHeader()`, reads file header to get actual version
4. Calls `frameSpecInfo(FrameSpec::SpecInfo(actualVersion))` to set up correct objects
5. `ReferenceStreamRef()` now returns correct version-specific StreamRef structure
6. All subsequent StreamRef reads use correct 8-byte or 16-byte layout

### Why This Works
- Base class `IFrameStream` already supports `Version=0` for auto-detection
- `FrameSpec::SpecInfo(version)` maintains registry of version-specific objects
- Each version has pre-instantiated `StreamRef<LT,CT,IT,SPEC_VERSION>` object
- Runtime polymorphism via `StreamRefInterface*` allows correct dispatch
- Universal namespace means all other frame objects are version-agnostic

### Why Template Was Wrong
- Template `SPEC=9` hardcoded at compile time
- Constructor `Version` parameter only affected `frameSpecInfo()`, not StreamRef structure
- Template instantiation created `StreamRef<INT_8U,INT_4U,INT_4U,9>` always
- v3 files need `StreamRef<INT_4U,INT_2U,INT_2U,3>` → mismatch!
- Reading 16 bytes at wrong offsets causes class field corruption

---

## Risk Assessment

### Low Risk
✅ New class added alongside existing code (Phases 1-2)
✅ Test-only changes initially (Phase 3)
✅ Incremental typedef migration (Phase 6)
✅ Template removal only after extensive validation (Phase 8)

### Medium Risk
⚠️ Version namespace typedef changes affect public API (Phase 6)
⚠️ Utilities may have assumptions about template types
⚠️ External dependencies (framecpp-swig) may need updates

### Mitigation
- Test each phase thoroughly before proceeding
- Maintain template code until final phase
- Document all changes for rollback
- Monitor external dependencies

---

## Success Criteria

### Must Have
- ✅ test_memory_buffer_ro passes for v3 files
- ✅ All 462 tests pass with new implementation
- ✅ No performance regression (< 5% overhead)
- ✅ Simplified codebase (remove ~500 lines of template code)

### Nice to Have
- ✅ Improved error messages (Phase 9)
- ✅ Updated CLAUDE.md documentation
- ✅ Migration guide for external users
- ✅ Performance benchmarks documented

---

## Notes for Future Sessions

### Session Continuity
- Check this file for current phase
- Review checklist for completion status
- Run relevant tests before proceeding
- Update checklist after each step

### Common Issues
- **Build errors**: Check template vs non-template typedef usage
- **Test failures**: Verify Version=0 auto-detection working
- **Performance**: Monitor StreamRef creation overhead
- **External deps**: Check framecpp-swig compatibility

### Questions to Resolve
- [ ] Does IFrameStreamPlan need similar refactoring?
- [ ] Are there other templates with same pattern?
- [ ] Should Version parameter be deprecated entirely?
- [ ] What about write streams (OFrameStream)?

---

## Session 11 - 2025-12-22

### ✅ **BUG FIXED - Simpler Root Cause Found!**

The v3 MemoryBuffer reading bug was **NOT** caused by the template SPEC parameter as originally hypothesized. The actual root cause was much simpler:

### Actual Root Cause
**File**: `src/Common/FrameStream.cc`
**Line**: 2322-2323 (in `advanceToFrame()` method)

**Bug**: Code was calling `streamref->GetClass()` instead of `streamref->GetStreamClass()` when checking for FrSH/FrSE structures.

```cpp
// WRONG (original code):
if ( ( streamref->GetClass( ) == 1 ) ||
     ( streamref->GetClass( ) == 2 ) )

// CORRECT (fixed):
if ( ( streamref->GetStreamClass( ) == FrameSpec::Info::FSI_FR_SH ) ||
     ( streamref->GetStreamClass( ) == FrameSpec::Info::FSI_FR_SE ) )
```

### Why This Broke v3 Files
- `GetClass()` returns the **FSI ID** (internal class identifier) = -2 for StreamRef
- `GetStreamClass()` returns the **stream class ID** (from file) = 1 for FrSH, 2 for FrSE
- The check always failed, so FrSH structures were **never processed** in `advanceToFrame()`
- For v3 files (no FrTOC), the FrSH for FrEndOfFile was skipped
- Stream class ID 7 → FSI_FR_END_OF_FILE mapping was never registered
- Reading FrEndOfFile threw exception: "Unable to map from the stream class id (7)"

### The Fix
**File**: `src/Common/FrameStream.cc:2322-2323`

Changed from:
```cpp
if ( ( streamref->GetClass( ) == 1 ) ||
     ( streamref->GetClass( ) == 2 ) )
```

To:
```cpp
if ( ( streamref->GetStreamClass( ) == FrameSpec::Info::FSI_FR_SH ) ||
     ( streamref->GetStreamClass( ) == FrameSpec::Info::FSI_FR_SE ) )
```

**Additional Improvement**: Replaced hardcoded constants `1` and `2` with named constants `FSI_FR_SH` and `FSI_FR_SE`.

### Verification
✅ **test_memory_buffer_ro now PASSES for all versions (v3-v9)**
- v3: FrSH for FrEndOfFile is now processed correctly
- v3: Stream class ID 7 → FSI_FR_END_OF_FILE mapping registered
- v3: FrEndOfFile read successfully, no exceptions
- v4-v9: No regression, all continue to work

### Debug Output Added (Temporary)
Added comprehensive debug output to track:
- `init()`: FrSH structure reading during initialization
- `readStreamRef()`: Stream class ID mapping attempts
- `readObject()`: FrSH processing and registration

**These debug outputs should be removed before committing.**

### Impact on Refactoring Plan
The template refactoring described in Phases 1-10 is **still valuable** for:
- Code simplification (remove ~500 lines of template code)
- Better architecture (runtime vs compile-time version handling)
- Future maintainability

However, it is **NOT required** to fix the v3 MemoryBuffer bug. The one-line fix in Session 11 completely resolves the issue.

**Recommendation**: Commit the bug fix separately, then consider the refactoring as a future enhancement.

---

**Last Updated**: 2025-12-22 (Session 11)
**Current Status**: ✅ v3 MemoryBuffer bug FIXED with one-line change
**Next Action**: Remove debug output, commit fix, optionally proceed with refactoring plan
