ob-elixir/tasks/06-test-suite.md

Task 06: Comprehensive Test Suite

Phase: 1 - Core (MVP) Priority: High Estimated Time: 2-3 hours Dependencies: Tasks 01-05 (All Core Tasks)

Objective

Create a comprehensive test suite that covers all implemented functionality, including unit tests, integration tests, and org-buffer tests.

Prerequisites

• All Phase 1 tasks completed
• Basic functionality working

Background

A good test suite should:

1. Test each function in isolation (unit tests)
2. Test the integration with org-mode (integration tests)
3. Be runnable in CI/CD (batch mode tests)
4. Provide good coverage of edge cases

Steps

Step 1: Organize test file structure

Create the test directory structure:

test/
├── test-ob-elixir.el           # Main test file (loads all)
├── test-ob-elixir-core.el      # Core execution tests
├── test-ob-elixir-vars.el      # Variable handling tests
├── test-ob-elixir-results.el   # Result formatting tests
├── test-ob-elixir-errors.el    # Error handling tests
└── test-ob-elixir-org.el       # Org integration tests

Step 2: Create main test file

test/test-ob-elixir.el:

;;; test-ob-elixir.el --- Tests for ob-elixir -*- lexical-binding: t; -*-

;;; Commentary:

;; Main test file that loads all test modules.
;; Run with: make test
;; Or: emacs -batch -l ert -l test/test-ob-elixir.el -f ert-run-tests-batch-and-exit

;;; Code:

(require 'ert)

;; Add source directory to load path
(let ((dir (file-name-directory (or load-file-name buffer-file-name))))
  (add-to-list 'load-path (expand-file-name ".." dir))
  (add-to-list 'load-path dir))

;; Load the package
(require 'ob-elixir)

;; Load test modules
(require 'test-ob-elixir-core)
(require 'test-ob-elixir-vars)
(require 'test-ob-elixir-results)
(require 'test-ob-elixir-errors)
(require 'test-ob-elixir-org)

;;; Test Helpers

(defvar ob-elixir-test--elixir-available
  (executable-find ob-elixir-command)
  "Non-nil if Elixir is available for testing.")

(defmacro ob-elixir-test-with-elixir (&rest body)
  "Execute BODY only if Elixir is available."
  `(if ob-elixir-test--elixir-available
       (progn ,@body)
     (ert-skip "Elixir not available")))

(defmacro ob-elixir-test-with-temp-org-buffer (&rest body)
  "Execute BODY in a temporary org-mode buffer."
  `(with-temp-buffer
     (org-mode)
     (ob-elixir--ensure-org-babel-loaded)
     ,@body))

(defun ob-elixir--ensure-org-babel-loaded ()
  "Ensure org-babel is loaded with Elixir support."
  (require 'org)
  (require 'ob)
  (org-babel-do-load-languages
   'org-babel-load-languages
   '((elixir . t))))

;;; Smoke Test

(ert-deftest ob-elixir-test-smoke ()
  "Basic smoke test - package loads and Elixir is available."
  (should (featurep 'ob-elixir))
  (should (fboundp 'org-babel-execute:elixir))
  (should (boundp 'org-babel-default-header-args:elixir)))

(provide 'test-ob-elixir)
;;; test-ob-elixir.el ends here

Step 3: Create core execution tests

test/test-ob-elixir-core.el:

;;; test-ob-elixir-core.el --- Core execution tests -*- lexical-binding: t; -*-

;;; Code:

(require 'ert)
(require 'ob-elixir)

;;; Command Tests

(ert-deftest ob-elixir-test-command-exists ()
  "Test that the Elixir command is configured."
  (should (stringp ob-elixir-command))
  (should (not (string-empty-p ob-elixir-command))))

(ert-deftest ob-elixir-test-command-executable ()
  "Test that the Elixir command is executable."
  (skip-unless (executable-find ob-elixir-command))
  (should (executable-find ob-elixir-command)))

;;; Basic Execution Tests

(ert-deftest ob-elixir-test-execute-simple-value ()
  "Test simple value evaluation."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "2" (ob-elixir--execute "1 + 1" 'value))))

(ert-deftest ob-elixir-test-execute-simple-output ()
  "Test simple output evaluation."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "hello" (ob-elixir--execute "IO.puts(\"hello\")" 'output))))

(ert-deftest ob-elixir-test-execute-multiline ()
  "Test multiline code execution."
  (skip-unless (executable-find ob-elixir-command))
  (let ((code "x = 10\ny = 20\nx + y"))
    (should (equal "30" (ob-elixir--execute code 'value)))))

(ert-deftest ob-elixir-test-execute-function-def ()
  "Test function definition and call."
  (skip-unless (executable-find ob-elixir-command))
  (let ((code "
defmodule Test do
  def double(x), do: x * 2
end
Test.double(21)"))
    (should (equal "42" (ob-elixir--execute code 'value)))))

(ert-deftest ob-elixir-test-execute-enum ()
  "Test Enum module usage."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "15" 
                 (ob-elixir--execute "Enum.sum([1, 2, 3, 4, 5])" 'value))))

(ert-deftest ob-elixir-test-execute-pipe ()
  "Test pipe operator."
  (skip-unless (executable-find ob-elixir-command))
  (let ((code "[1, 2, 3] |> Enum.map(&(&1 * 2)) |> Enum.sum()"))
    (should (equal "12" (ob-elixir--execute code 'value)))))

;;; Data Type Tests

(ert-deftest ob-elixir-test-execute-list ()
  "Test list result."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "[1, 2, 3]" (ob-elixir--execute "[1, 2, 3]" 'value))))

(ert-deftest ob-elixir-test-execute-tuple ()
  "Test tuple result."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "{:ok, 42}" (ob-elixir--execute "{:ok, 42}" 'value))))

(ert-deftest ob-elixir-test-execute-map ()
  "Test map result."
  (skip-unless (executable-find ob-elixir-command))
  (let ((result (ob-elixir--execute "%{a: 1, b: 2}" 'value)))
    (should (string-match-p "%{" result))
    (should (string-match-p "a:" result))
    (should (string-match-p "b:" result))))

(ert-deftest ob-elixir-test-execute-string ()
  "Test string result."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "\"hello world\"" 
                 (ob-elixir--execute "\"hello world\"" 'value))))

(ert-deftest ob-elixir-test-execute-atom ()
  "Test atom result."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal ":ok" (ob-elixir--execute ":ok" 'value))))

(ert-deftest ob-elixir-test-execute-boolean ()
  "Test boolean result."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "true" (ob-elixir--execute "true" 'value)))
  (should (equal "false" (ob-elixir--execute "false" 'value))))

(ert-deftest ob-elixir-test-execute-nil ()
  "Test nil result."
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "nil" (ob-elixir--execute "nil" 'value))))

;;; Wrapper Tests

(ert-deftest ob-elixir-test-wrap-for-value ()
  "Test value wrapper generation."
  (let ((wrapped (ob-elixir--wrap-for-value "1 + 1")))
    (should (string-match-p "result = " wrapped))
    (should (string-match-p "IO\\.puts" wrapped))
    (should (string-match-p "inspect" wrapped))))

(provide 'test-ob-elixir-core)
;;; test-ob-elixir-core.el ends here

Step 4: Create variable tests

test/test-ob-elixir-vars.el:

;;; test-ob-elixir-vars.el --- Variable handling tests -*- lexical-binding: t; -*-

;;; Code:

(require 'ert)
(require 'ob-elixir)

;;; Type Conversion Tests

(ert-deftest ob-elixir-test-convert-nil ()
  (should (equal "nil" (ob-elixir--elisp-to-elixir nil))))

(ert-deftest ob-elixir-test-convert-true ()
  (should (equal "true" (ob-elixir--elisp-to-elixir t))))

(ert-deftest ob-elixir-test-convert-integer ()
  (should (equal "42" (ob-elixir--elisp-to-elixir 42)))
  (should (equal "-10" (ob-elixir--elisp-to-elixir -10)))
  (should (equal "0" (ob-elixir--elisp-to-elixir 0))))

(ert-deftest ob-elixir-test-convert-float ()
  (should (equal "3.14" (ob-elixir--elisp-to-elixir 3.14)))
  (should (equal "-2.5" (ob-elixir--elisp-to-elixir -2.5))))

(ert-deftest ob-elixir-test-convert-string ()
  (should (equal "\"hello\"" (ob-elixir--elisp-to-elixir "hello")))
  (should (equal "\"\"" (ob-elixir--elisp-to-elixir ""))))

(ert-deftest ob-elixir-test-convert-string-escaping ()
  (should (equal "\"say \\\"hi\\\"\"" 
                 (ob-elixir--elisp-to-elixir "say \"hi\"")))
  (should (equal "\"line1\\nline2\"" 
                 (ob-elixir--elisp-to-elixir "line1\nline2")))
  (should (equal "\"tab\\there\"" 
                 (ob-elixir--elisp-to-elixir "tab\there"))))

(ert-deftest ob-elixir-test-convert-symbol ()
  (should (equal ":foo" (ob-elixir--elisp-to-elixir 'foo)))
  (should (equal ":ok" (ob-elixir--elisp-to-elixir 'ok)))
  (should (equal ":error" (ob-elixir--elisp-to-elixir 'error))))

(ert-deftest ob-elixir-test-convert-list ()
  (should (equal "[1, 2, 3]" (ob-elixir--elisp-to-elixir '(1 2 3))))
  (should (equal "[]" (ob-elixir--elisp-to-elixir '())))
  (should (equal "[\"a\", \"b\"]" (ob-elixir--elisp-to-elixir '("a" "b")))))

(ert-deftest ob-elixir-test-convert-nested-list ()
  (should (equal "[[1, 2], [3, 4]]" 
                 (ob-elixir--elisp-to-elixir '((1 2) (3 4))))))

(ert-deftest ob-elixir-test-convert-vector ()
  (should (equal "{1, 2, 3}" (ob-elixir--elisp-to-elixir [1 2 3]))))

(ert-deftest ob-elixir-test-convert-mixed ()
  (should (equal "[1, \"two\", :three]" 
                 (ob-elixir--elisp-to-elixir '(1 "two" three)))))

;;; Variable Assignment Tests

(ert-deftest ob-elixir-test-var-assignments-single ()
  (let ((params '((:var . ("x" . 5)))))
    (should (equal '("x = 5") 
                   (org-babel-variable-assignments:elixir params)))))

(ert-deftest ob-elixir-test-var-assignments-multiple ()
  (let ((params '((:var . ("x" . 5))
                  (:var . ("y" . 10)))))
    (let ((assignments (org-babel-variable-assignments:elixir params)))
      (should (= 2 (length assignments)))
      (should (member "x = 5" assignments))
      (should (member "y = 10" assignments)))))

(ert-deftest ob-elixir-test-var-assignments-string ()
  (let ((params '((:var . ("name" . "Alice")))))
    (should (equal '("name = \"Alice\"") 
                   (org-babel-variable-assignments:elixir params)))))

(ert-deftest ob-elixir-test-var-assignments-list ()
  (let ((params '((:var . ("data" . (1 2 3))))))
    (should (equal '("data = [1, 2, 3]") 
                   (org-babel-variable-assignments:elixir params)))))

;;; Execution with Variables

(ert-deftest ob-elixir-test-execute-with-var ()
  (skip-unless (executable-find ob-elixir-command))
  (let* ((params '((:var . ("x" . 10))))
         (var-lines (org-babel-variable-assignments:elixir params))
         (full-body (concat (mapconcat #'identity var-lines "\n")
                            "\nx * 2")))
    (should (equal "20" (ob-elixir--execute full-body 'value)))))

(ert-deftest ob-elixir-test-execute-with-list-var ()
  (skip-unless (executable-find ob-elixir-command))
  (let* ((params '((:var . ("nums" . (1 2 3 4 5)))))
         (var-lines (org-babel-variable-assignments:elixir params))
         (full-body (concat (mapconcat #'identity var-lines "\n")
                            "\nEnum.sum(nums)")))
    (should (equal "15" (ob-elixir--execute full-body 'value)))))

(ert-deftest ob-elixir-test-execute-with-string-var ()
  (skip-unless (executable-find ob-elixir-command))
  (let* ((params '((:var . ("name" . "World"))))
         (var-lines (org-babel-variable-assignments:elixir params))
         (full-body (concat (mapconcat #'identity var-lines "\n")
                            "\n\"Hello, #{name}!\"")))
    (should (equal "\"Hello, World!\"" 
                   (ob-elixir--execute full-body 'value)))))

(provide 'test-ob-elixir-vars)
;;; test-ob-elixir-vars.el ends here

Step 5: Create result formatting tests

test/test-ob-elixir-results.el:

;;; test-ob-elixir-results.el --- Result formatting tests -*- lexical-binding: t; -*-

;;; Code:

(require 'ert)
(require 'ob-elixir)

;;; Parsing Tests

(ert-deftest ob-elixir-test-parse-simple-list ()
  (should (equal '(1 2 3) (ob-elixir--table-or-string "[1, 2, 3]"))))

(ert-deftest ob-elixir-test-parse-nested-list ()
  (should (equal '((1 2) (3 4)) 
                 (ob-elixir--table-or-string "[[1, 2], [3, 4]]"))))

(ert-deftest ob-elixir-test-parse-empty-list ()
  (should (equal '() (ob-elixir--table-or-string "[]"))))

(ert-deftest ob-elixir-test-parse-tuple ()
  (should (equal '(1 2 3) (ob-elixir--table-or-string "{1, 2, 3}"))))

(ert-deftest ob-elixir-test-parse-scalar-number ()
  (should (equal "42" (ob-elixir--table-or-string "42"))))

(ert-deftest ob-elixir-test-parse-scalar-atom ()
  (should (equal ":ok" (ob-elixir--table-or-string ":ok"))))

(ert-deftest ob-elixir-test-parse-scalar-string ()
  (should (equal "\"hello\"" (ob-elixir--table-or-string "\"hello\""))))

(ert-deftest ob-elixir-test-parse-empty ()
  (should (null (ob-elixir--table-or-string "")))
  (should (null (ob-elixir--table-or-string "  "))))

;;; Table Sanitization Tests

(ert-deftest ob-elixir-test-sanitize-nil-values ()
  (let ((ob-elixir-nil-to 'hline))
    (should (equal '((1 hline) (hline 2))
                   (ob-elixir--sanitize-table '((1 nil) (nil 2)))))))

(ert-deftest ob-elixir-test-sanitize-nested ()
  (let ((ob-elixir-nil-to 'hline))
    (should (equal '((1 2) (3 4))
                   (ob-elixir--sanitize-table '((1 2) (3 4)))))))

(ert-deftest ob-elixir-test-sanitize-simple ()
  (should (equal '(1 2 3)
                 (ob-elixir--sanitize-table '(1 2 3)))))

;;; Integration Tests

(ert-deftest ob-elixir-test-full-result-list ()
  (skip-unless (executable-find ob-elixir-command))
  (let ((result (ob-elixir--table-or-string
                 (ob-elixir--execute "[1, 2, 3]" 'value))))
    (should (equal '(1 2 3) result))))

(ert-deftest ob-elixir-test-full-result-table ()
  (skip-unless (executable-find ob-elixir-command))
  (let ((result (ob-elixir--table-or-string
                 (ob-elixir--execute "[[1, 2], [3, 4]]" 'value))))
    (should (equal '((1 2) (3 4)) result))))

(provide 'test-ob-elixir-results)
;;; test-ob-elixir-results.el ends here

Step 6: Create error handling tests

test/test-ob-elixir-errors.el:

;;; test-ob-elixir-errors.el --- Error handling tests -*- lexical-binding: t; -*-

;;; Code:

(require 'ert)
(require 'ob-elixir)

;;; Error Detection Tests

(ert-deftest ob-elixir-test-detect-runtime-error ()
  (let ((output "** (RuntimeError) something went wrong"))
    (should (ob-elixir--detect-error output))))

(ert-deftest ob-elixir-test-detect-compile-error ()
  (let ((output "** (CompileError) test.exs:1: undefined function"))
    (should (ob-elixir--detect-error output))))

(ert-deftest ob-elixir-test-detect-no-error ()
  (should-not (ob-elixir--detect-error "42"))
  (should-not (ob-elixir--detect-error "[1, 2, 3]"))
  (should-not (ob-elixir--detect-error ":ok")))

(ert-deftest ob-elixir-test-error-type-runtime ()
  (let* ((output "** (RuntimeError) test error")
         (info (ob-elixir--detect-error output)))
    (should (eq 'runtime (plist-get info :type)))))

(ert-deftest ob-elixir-test-error-type-compile ()
  (let* ((output "** (CompileError) syntax error")
         (info (ob-elixir--detect-error output)))
    (should (eq 'compile (plist-get info :type)))))

;;; Error Execution Tests

(ert-deftest ob-elixir-test-runtime-error-no-signal ()
  (skip-unless (executable-find ob-elixir-command))
  (let ((ob-elixir-signal-errors nil))
    (let ((result (ob-elixir--execute "raise \"test\"" 'value)))
      (should (string-match-p "RuntimeError" result)))))

(ert-deftest ob-elixir-test-runtime-error-signal ()
  (skip-unless (executable-find ob-elixir-command))
  (let ((ob-elixir-signal-errors t))
    (should-error (ob-elixir--execute "raise \"test\"" 'value)
                  :type 'ob-elixir-runtime-error)))

(ert-deftest ob-elixir-test-compile-error ()
  (skip-unless (executable-find ob-elixir-command))
  (let ((ob-elixir-signal-errors nil))
    (let ((result (ob-elixir--execute "def incomplete(" 'value)))
      (should (string-match-p "\\(SyntaxError\\|TokenMissingError\\)" result)))))

(provide 'test-ob-elixir-errors)
;;; test-ob-elixir-errors.el ends here

Step 7: Create org integration tests

test/test-ob-elixir-org.el:

;;; test-ob-elixir-org.el --- Org integration tests -*- lexical-binding: t; -*-

;;; Code:

(require 'ert)
(require 'org)
(require 'ob)
(require 'ob-elixir)

;;; Helper Functions

(defun ob-elixir-test--execute-src-block (code &optional header-args)
  "Execute CODE as an Elixir src block with HEADER-ARGS."
  (with-temp-buffer
    (org-mode)
    (insert (format "#+BEGIN_SRC elixir%s\n%s\n#+END_SRC"
                    (if header-args (concat " " header-args) "")
                    code))
    (goto-char (point-min))
    (forward-line 1)
    (org-babel-execute-src-block)))

;;; Basic Org Tests

(ert-deftest ob-elixir-test-org-simple ()
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "2" (ob-elixir-test--execute-src-block "1 + 1"))))

(ert-deftest ob-elixir-test-org-with-var ()
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "20" (ob-elixir-test--execute-src-block "x * 2" ":var x=10"))))

(ert-deftest ob-elixir-test-org-results-output ()
  (skip-unless (executable-find ob-elixir-command))
  (should (equal "hello" 
                 (ob-elixir-test--execute-src-block 
                  "IO.puts(\"hello\")" 
                  ":results output"))))

(ert-deftest ob-elixir-test-org-results-value ()
  (skip-unless (executable-find ob-elixir-command))
  (should (equal '(1 2 3) 
                 (ob-elixir-test--execute-src-block 
                  "[1, 2, 3]" 
                  ":results value"))))

(ert-deftest ob-elixir-test-org-results-verbatim ()
  (skip-unless (executable-find ob-elixir-command))
  (should (stringp (ob-elixir-test--execute-src-block 
                    "[1, 2, 3]" 
                    ":results verbatim"))))

;;; Table Tests

(ert-deftest ob-elixir-test-org-table-result ()
  (skip-unless (executable-find ob-elixir-command))
  (let ((result (ob-elixir-test--execute-src-block "[[1, 2], [3, 4]]")))
    (should (equal '((1 2) (3 4)) result))))

(provide 'test-ob-elixir-org)
;;; test-ob-elixir-org.el ends here

Step 8: Update Makefile

EMACS ?= emacs
BATCH = $(EMACS) -Q -batch -L . -L test

.PHONY: all compile test test-unit test-integration lint clean

all: compile test

compile:
	$(BATCH) -f batch-byte-compile ob-elixir.el

test: test-unit

test-unit:
	$(BATCH) -l ert \
		-l test/test-ob-elixir.el \
		-f ert-run-tests-batch-and-exit

test-integration:
	$(BATCH) -l ert \
		-l org -l ob \
		-l test/test-ob-elixir.el \
		-f ert-run-tests-batch-and-exit

lint:
	$(BATCH) --eval "(require 'package)" \
		--eval "(package-initialize)" \
		--eval "(unless (package-installed-p 'package-lint) \
		         (package-refresh-contents) \
		         (package-install 'package-lint))" \
		-l package-lint \
		-f package-lint-batch-and-exit ob-elixir.el

clean:
	rm -f *.elc test/*.elc

Acceptance Criteria

• All test files created and organized
• make test runs all tests
• Tests cover core execution, variables, results, and errors
• Org integration tests work
• Tests can run in CI (batch mode)
• Test coverage is comprehensive (major code paths)

Test Coverage Goals

Component	Tests	Coverage
Type conversion	12+ tests	All Elisp types
Execution	10+ tests	Value/output, types
Variables	8+ tests	All var scenarios
Results	8+ tests	Parsing, tables
Errors	6+ tests	Detection, signaling
Org integration	6+ tests	Full workflow

Files Created

• test/test-ob-elixir.el - Main test file
• test/test-ob-elixir-core.el - Core tests
• test/test-ob-elixir-vars.el - Variable tests
• test/test-ob-elixir-results.el - Result tests
• test/test-ob-elixir-errors.el - Error tests
• test/test-ob-elixir-org.el - Org integration tests

References

• docs/02-testing-emacs-elisp.md
• ERT Manual