ob-elixir/tasks/08-mix-project-support.md

Task 08: Mix Dependencies Support

Phase: 3 - Mix Integration Priority: High Estimated Time: 3-4 hours Dependencies: Task 07 (Session Support) or Phase 1 complete

Objective

Implement Mix dependencies support so users can specify dependencies directly in the org file using a special #+BEGIN_DEPS block. ob-elixir will automatically create a temporary Mix project with those dependencies, cache it for reuse, and use it as the execution context for subsequent Elixir code blocks.

Prerequisites

• Phase 1 complete (or Phase 2 for session+deps)
• Understanding of Mix build tool
• Network access for fetching dependencies

Background

The Problem

When writing Elixir code in org-mode, users often need access to external libraries (e.g., JSON parsing, HTTP clients, custom libraries). Without Mix project support, users are limited to Elixir's standard library.

The Solution

Instead of requiring users to point to an existing Mix project, ob-elixir will:

1. Parse dependency specifications from a special #+BEGIN_DEPS block
2. Create a temporary Mix project with those dependencies
3. Cache the project based on a hash of the dependencies (for fast re-evaluation)
4. Execute code blocks within that project context
5. Clean up temporary projects when Emacs exits

Example Usage

* Using External Dependencies

#+BEGIN_DEPS elixir
[
  {:jason, "~> 1.4"},
  {:prova, git: "git@gitlab.com:babel-upm/makina/prova.git"}
]
#+END_DEPS

#+BEGIN_SRC elixir
# Both jason and prova are available
Jason.encode!(%{hello: "world"})
#+END_SRC

#+BEGIN_SRC elixir :session deps-session
# Session also has access to deps
Prova.some_function()
#+END_SRC

* Different Dependencies Section

#+BEGIN_DEPS elixir
[{:httpoison, "~> 2.0"}]
#+END_DEPS

#+BEGIN_SRC elixir
# Now using httpoison instead of the previous deps
HTTPoison.get!("https://example.com")
#+END_SRC

Design Decisions

Aspect	Decision	Rationale
Block format	#+BEGIN_DEPS elixir ... #+END_DEPS	Special block, not confused with executable code
Deps syntax	Elixir list: [{:pkg, "~> 1.0"}]	Natural for Elixir users, supports all dep formats
Scope	All subsequent blocks until next deps block	Simple mental model
Caching	Hash-based	Fast re-evaluation, no redundant compilation
Multiple deps blocks	Later overrides	Allows different sections with different deps
Cleanup	Auto on Emacs exit	No orphaned temp projects
Elixir version	System default	Keep it simple, no version management

Steps

Step 1: Add configuration and infrastructure

;;; Mix Dependencies Configuration

(defcustom ob-elixir-mix-command "mix"
  "Command to run Mix."
  :type 'string
  :group 'ob-elixir
  :safe #'stringp)

(defcustom ob-elixir-deps-cache-dir
  (expand-file-name "ob-elixir" (or (getenv "XDG_CACHE_HOME") "~/.cache"))
  "Directory for caching temporary Mix projects.

Each unique set of dependencies gets its own subdirectory,
named by the hash of the dependencies."
  :type 'directory
  :group 'ob-elixir)

(defvar ob-elixir--deps-projects (make-hash-table :test 'equal)
  "Hash table mapping deps-hash to project directory path.")

(defvar ob-elixir--cleanup-registered nil
  "Whether the cleanup hook has been registered.")

Step 2: Implement deps block parsing

;;; Deps Block Parsing

(defconst ob-elixir--deps-block-regexp
  "^[ \t]*#\\+BEGIN_DEPS[ \t]+elixir[ \t]*\n\\(\\(?:.*\n\\)*?\\)[ \t]*#\\+END_DEPS"
  "Regexp matching a deps block.
Group 1 captures the deps content.")

(defun ob-elixir--find-deps-for-position (pos)
  "Find the most recent deps block before POS.

Returns the deps content as a string, or nil if no deps block found."
  (save-excursion
    (goto-char pos)
    (let ((found nil))
      ;; Search backward for deps blocks
      (while (and (not found)
                  (re-search-backward ob-elixir--deps-block-regexp nil t))
        (when (< (match-end 0) pos)
          (setq found (match-string-no-properties 1))))
      found)))

(defun ob-elixir--normalize-deps (deps-string)
  "Normalize DEPS-STRING for consistent hashing.

Removes comments, extra whitespace, and normalizes formatting."
  (let ((normalized deps-string))
    ;; Remove Elixir comments
    (setq normalized (replace-regexp-in-string "#.*$" "" normalized))
    ;; Normalize whitespace
    (setq normalized (replace-regexp-in-string "[ \t\n]+" " " normalized))
    ;; Trim
    (string-trim normalized)))

(defun ob-elixir--hash-deps (deps-string)
  "Compute SHA256 hash of DEPS-STRING for caching."
  (secure-hash 'sha256 (ob-elixir--normalize-deps deps-string)))

Step 3: Implement temp Mix project creation

;;; Temporary Mix Project Management

(defconst ob-elixir--mix-exs-template
  "defmodule ObElixirTemp.MixProject do
  use Mix.Project

  def project do
    [
      app: :ob_elixir_temp,
      version: \"0.1.0\",
      elixir: \"~> 1.14\",
      start_permanent: false,
      deps: deps()
    ]
  end

  def application do
    [extra_applications: [:logger]]
  end

  defp deps do
    %s
  end
end
"
  "Template for temporary Mix project mix.exs file.
%s is replaced with the deps list.")

(defun ob-elixir--ensure-cache-dir ()
  "Ensure the cache directory exists."
  (unless (file-directory-p ob-elixir-deps-cache-dir)
    (make-directory ob-elixir-deps-cache-dir t)))

(defun ob-elixir--get-deps-project (deps-string)
  "Get or create a Mix project for DEPS-STRING.

Returns the project directory path, creating and initializing
the project if necessary."
  (let* ((deps-hash (ob-elixir--hash-deps deps-string))
         (cached (gethash deps-hash ob-elixir--deps-projects)))
    (if (and cached (file-directory-p cached))
        cached
      ;; Create new project
      (ob-elixir--create-deps-project deps-hash deps-string))))

(defun ob-elixir--create-deps-project (deps-hash deps-string)
  "Create a new temporary Mix project for DEPS-STRING.

DEPS-HASH is used as the directory name.
Returns the project directory path."
  (ob-elixir--ensure-cache-dir)
  (ob-elixir--register-cleanup)
  
  (let* ((project-dir (expand-file-name deps-hash ob-elixir-deps-cache-dir))
         (mix-exs-path (expand-file-name "mix.exs" project-dir))
         (lib-dir (expand-file-name "lib" project-dir)))
    
    ;; Create project structure
    (make-directory project-dir t)
    (make-directory lib-dir t)
    
    ;; Write mix.exs
    (with-temp-file mix-exs-path
      (insert (format ob-elixir--mix-exs-template deps-string)))
    
    ;; Write a placeholder module
    (with-temp-file (expand-file-name "ob_elixir_temp.ex" lib-dir)
      (insert "defmodule ObElixirTemp do\nend\n"))
    
    ;; Fetch dependencies
    (let ((default-directory project-dir))
      (message "ob-elixir: Fetching dependencies...")
      (let ((output (shell-command-to-string
                     (format "%s deps.get 2>&1" ob-elixir-mix-command))))
        (when (string-match-p "\\*\\*.*error\\|Could not" output)
          (error "ob-elixir: Failed to fetch dependencies:\n%s" output)))
      
      ;; Compile
      (message "ob-elixir: Compiling dependencies...")
      (let ((output (shell-command-to-string
                     (format "%s compile 2>&1" ob-elixir-mix-command))))
        (when (string-match-p "\\*\\*.*[Ee]rror" output)
          (error "ob-elixir: Failed to compile dependencies:\n%s" output))))
    
    ;; Cache and return
    (puthash deps-hash project-dir ob-elixir--deps-projects)
    (message "ob-elixir: Dependencies ready")
    project-dir))

Step 4: Implement execution with deps context

;;; Execution with Dependencies

(defun ob-elixir--execute-with-deps (body result-type deps-string)
  "Execute BODY with dependencies from DEPS-STRING.

RESULT-TYPE is 'value or 'output."
  (let* ((project-dir (ob-elixir--get-deps-project deps-string))
         (default-directory project-dir)
         (tmp-file (org-babel-temp-file "ob-elixir-" ".exs"))
         (code (if (eq result-type 'value)
                   (ob-elixir--wrap-for-value body)
                 body)))
    
    ;; Write code to temp file
    (with-temp-file tmp-file
      (insert code))
    
    ;; Execute with mix run
    (let ((command (format "%s run --no-compile %s 2>&1"
                           ob-elixir-mix-command
                           (shell-quote-argument tmp-file))))
      (ob-elixir--process-result
       (shell-command-to-string command)))))

(defun ob-elixir--start-session-with-deps (buffer-name session-name deps-string)
  "Start a new IEx session with dependencies in BUFFER-NAME.

SESSION-NAME is used for the process name.
DEPS-STRING contains the dependencies specification."
  (let* ((project-dir (ob-elixir--get-deps-project deps-string))
         (buffer (get-buffer-create buffer-name))
         (default-directory project-dir)
         (process-environment (cons "TERM=dumb" process-environment)))
    
    (with-current-buffer buffer
      ;; Start IEx with Mix project
      (make-comint-in-buffer
       (format "ob-elixir-%s" session-name)
       buffer
       ob-elixir-iex-command
       nil
       "-S" "mix")
      
      ;; Wait for initial prompt
      (ob-elixir--wait-for-prompt buffer 30)  ; Longer timeout for deps loading
      
      ;; Configure IEx for programmatic use
      (ob-elixir--configure-session buffer)
      
      buffer)))

Step 5: Update main execute function

Modify org-babel-execute:elixir to check for deps context:

(defun org-babel-execute:elixir (body params)
  "Execute a block of Elixir code with org-babel.

BODY is the Elixir code to execute.
PARAMS is an alist of header arguments."
  (let* ((session (cdr (assq :session params)))
         (result-type (cdr (assq :result-type params)))
         (result-params (cdr (assq :result-params params)))
         ;; Find deps for this block's position
         (deps-string (ob-elixir--find-deps-for-position (point)))
         ;; Expand body with variable assignments
         (full-body (org-babel-expand-body:generic
                     body params
                     (org-babel-variable-assignments:elixir params)))
         (result (cond
                  ;; Session mode with deps
                  ((and session (not (string= session "none")) deps-string)
                   (ob-elixir--evaluate-in-session-with-deps
                    session full-body result-type deps-string))
                  ;; Session mode without deps
                  ((and session (not (string= session "none")))
                   (ob-elixir--evaluate-in-session session full-body result-type))
                  ;; Non-session with deps
                  (deps-string
                   (ob-elixir--execute-with-deps full-body result-type deps-string))
                  ;; Plain execution
                  (t
                   (ob-elixir--execute full-body result-type)))))
    (org-babel-reassemble-table
     (org-babel-result-cond result-params
       result
       (ob-elixir--table-or-string result))
     (org-babel-pick-name (cdr (assq :colname-names params))
                          (cdr (assq :colnames params)))
     (org-babel-pick-name (cdr (assq :rowname-names params))
                          (cdr (assq :rownames params))))))

Step 6: Implement session support with deps

(defvar ob-elixir--session-deps (make-hash-table :test 'equal)
  "Hash table mapping session names to their deps-hash.
Used to ensure session deps consistency.")

(defun ob-elixir--get-or-create-session-with-deps (name deps-string params)
  "Get or create an IEx session NAME with DEPS-STRING."
  (let* ((deps-hash (ob-elixir--hash-deps deps-string))
         (buffer-name (format "*ob-elixir:%s*" name))
         (existing (get-buffer buffer-name))
         (existing-deps (gethash name ob-elixir--session-deps)))
    
    ;; Check if existing session has different deps
    (when (and existing existing-deps (not (string= existing-deps deps-hash)))
      (message "ob-elixir: Session %s has different deps, recreating..." name)
      (ob-elixir-kill-session name)
      (setq existing nil))
    
    (if (and existing (org-babel-comint-buffer-livep existing))
        existing
      ;; Create new session
      (let ((buffer (ob-elixir--start-session-with-deps buffer-name name deps-string)))
        (puthash name deps-hash ob-elixir--session-deps)
        buffer))))

(defun ob-elixir--evaluate-in-session-with-deps (session body result-type deps-string)
  "Evaluate BODY in SESSION with DEPS-STRING context."
  (let* ((buffer (ob-elixir--get-or-create-session-with-deps session deps-string nil))
         (code (if (eq result-type 'value)
                   (ob-elixir--session-wrap-for-value body)
                 body))
         (eoe-indicator ob-elixir--eoe-marker)
         (full-body (concat code "\nIO.puts(\"" eoe-indicator "\")"))
         output)
    
    (unless buffer
      (error "Failed to create Elixir session with deps: %s" session))
    
    (setq output
          (org-babel-comint-with-output
              (buffer eoe-indicator t full-body)
            (ob-elixir--send-command buffer full-body)))
    
    (ob-elixir--clean-session-output output result-type)))

Step 7: Implement cleanup

;;; Cleanup

(defun ob-elixir--register-cleanup ()
  "Register cleanup hook if not already registered."
  (unless ob-elixir--cleanup-registered
    (add-hook 'kill-emacs-hook #'ob-elixir-cleanup-deps-projects)
    (setq ob-elixir--cleanup-registered t)))

(defun ob-elixir-cleanup-deps-projects ()
  "Delete all temporary Mix projects created by ob-elixir.

Called automatically on Emacs exit."
  (interactive)
  (maphash
   (lambda (_hash project-dir)
     (when (and project-dir (file-directory-p project-dir))
       (condition-case err
           (delete-directory project-dir t)
         (error
          (message "ob-elixir: Failed to delete %s: %s" project-dir err)))))
   ob-elixir--deps-projects)
  (clrhash ob-elixir--deps-projects)
  (message "ob-elixir: Cleaned up temporary Mix projects"))

(defun ob-elixir-list-deps-projects ()
  "List all cached dependency projects."
  (interactive)
  (if (= (hash-table-count ob-elixir--deps-projects) 0)
      (message "No cached dependency projects")
    (with-output-to-temp-buffer "*ob-elixir deps projects*"
      (princ "Cached ob-elixir dependency projects:\n\n")
      (maphash
       (lambda (hash dir)
         (princ (format "  %s\n    -> %s\n" (substring hash 0 12) dir)))
       ob-elixir--deps-projects))))

Step 8: Add tests

Create test/test-ob-elixir-deps.el:

;;; test-ob-elixir-deps.el --- Deps block tests -*- lexical-binding: t; -*-

(require 'ert)
(require 'ob-elixir)

;;; Parsing Tests

(ert-deftest ob-elixir-test-deps-block-parsing ()
  "Test finding deps blocks in buffer."
  (with-temp-buffer
    (insert "#+BEGIN_DEPS elixir\n[{:jason, \"~> 1.4\"}]\n#+END_DEPS\n\n")
    (insert "#+BEGIN_SRC elixir\nJason.encode!(%{})\n#+END_SRC\n")
    (goto-char (point-max))
    (let ((deps (ob-elixir--find-deps-for-position (point))))
      (should deps)
      (should (string-match-p ":jason" deps)))))

(ert-deftest ob-elixir-test-deps-block-override ()
  "Test that later deps blocks override earlier ones."
  (with-temp-buffer
    (insert "#+BEGIN_DEPS elixir\n[{:jason, \"~> 1.4\"}]\n#+END_DEPS\n\n")
    (insert "#+BEGIN_SRC elixir\ncode1\n#+END_SRC\n\n")
    (insert "#+BEGIN_DEPS elixir\n[{:httpoison, \"~> 2.0\"}]\n#+END_DEPS\n\n")
    (let ((pos (point)))
      (insert "#+BEGIN_SRC elixir\ncode2\n#+END_SRC\n")
      (let ((deps (ob-elixir--find-deps-for-position (+ pos 10))))
        (should deps)
        (should (string-match-p ":httpoison" deps))
        (should-not (string-match-p ":jason" deps))))))

(ert-deftest ob-elixir-test-no-deps-block ()
  "Test behavior when no deps block exists."
  (with-temp-buffer
    (insert "#+BEGIN_SRC elixir\n1 + 1\n#+END_SRC\n")
    (should (null (ob-elixir--find-deps-for-position (point))))))

(ert-deftest ob-elixir-test-deps-hash-consistency ()
  "Test that same deps produce same hash."
  (let ((deps1 "[{:jason, \"~> 1.4\"}]")
        (deps2 "[{:jason,   \"~> 1.4\"}]")  ; extra space
        (deps3 "[{:httpoison, \"~> 2.0\"}]"))
    (should (string= (ob-elixir--hash-deps deps1)
                     (ob-elixir--hash-deps deps2)))
    (should-not (string= (ob-elixir--hash-deps deps1)
                         (ob-elixir--hash-deps deps3)))))

(ert-deftest ob-elixir-test-normalize-deps ()
  "Test deps normalization."
  (should (string= (ob-elixir--normalize-deps "[ {:a, \"1\"} ]")
                   (ob-elixir--normalize-deps "[{:a,\"1\"}]")))
  ;; Comments are stripped
  (should (string= (ob-elixir--normalize-deps "[{:a, \"1\"}] # comment")
                   (ob-elixir--normalize-deps "[{:a, \"1\"}]"))))

;;; Integration Tests (require network and mix)

(ert-deftest ob-elixir-test-deps-project-creation ()
  "Test creating a temporary Mix project with deps."
  (skip-unless (and (executable-find ob-elixir-mix-command)
                    (executable-find ob-elixir-command)))
  (let ((ob-elixir-deps-cache-dir (make-temp-file "ob-elixir-test-" t))
        (deps "[{:jason, \"~> 1.4\"}]"))
    (unwind-protect
        (let ((project-dir (ob-elixir--get-deps-project deps)))
          (should (file-directory-p project-dir))
          (should (file-exists-p (expand-file-name "mix.exs" project-dir)))
          (should (file-directory-p (expand-file-name "deps/jason" project-dir))))
      ;; Cleanup
      (ob-elixir-cleanup-deps-projects)
      (delete-directory ob-elixir-deps-cache-dir t))))

(ert-deftest ob-elixir-test-deps-execution ()
  "Test executing code with deps."
  (skip-unless (and (executable-find ob-elixir-mix-command)
                    (executable-find ob-elixir-command)))
  (let ((ob-elixir-deps-cache-dir (make-temp-file "ob-elixir-test-" t))
        (deps "[{:jason, \"~> 1.4\"}]"))
    (unwind-protect
        (let ((result (ob-elixir--execute-with-deps
                       "Jason.encode!(%{a: 1})"
                       'value
                       deps)))
          (should (string-match-p "\"a\"" result)))
      ;; Cleanup
      (ob-elixir-cleanup-deps-projects)
      (delete-directory ob-elixir-deps-cache-dir t))))

(ert-deftest ob-elixir-test-deps-caching ()
  "Test that deps projects are cached and reused."
  (skip-unless (executable-find ob-elixir-mix-command))
  (let ((ob-elixir-deps-cache-dir (make-temp-file "ob-elixir-test-" t))
        (deps "[{:jason, \"~> 1.4\"}]"))
    (unwind-protect
        (let ((project1 (ob-elixir--get-deps-project deps))
              (project2 (ob-elixir--get-deps-project deps)))
          ;; Should return same directory
          (should (string= project1 project2))
          ;; Should only have one entry in hash table
          (should (= 1 (hash-table-count ob-elixir--deps-projects))))
      ;; Cleanup
      (ob-elixir-cleanup-deps-projects)
      (delete-directory ob-elixir-deps-cache-dir t))))

(provide 'test-ob-elixir-deps)

Step 9: Test in org buffer

Create test cases in test.org:

* Deps Block Tests

** Basic deps usage

#+BEGIN_DEPS elixir
[{:jason, "~> 1.4"}]
#+END_DEPS

#+BEGIN_SRC elixir
Jason.encode!(%{hello: "world", number: 42})
#+END_SRC

** Multiple dependencies

#+BEGIN_DEPS elixir
[
  {:jason, "~> 1.4"},
  {:decimal, "~> 2.0"}
]
#+END_DEPS

#+BEGIN_SRC elixir
num = Decimal.new("3.14159")
Jason.encode!(%{pi: Decimal.to_string(num)})
#+END_SRC

** Git dependencies

#+BEGIN_DEPS elixir
[
  {:prova, git: "git@gitlab.com:babel-upm/makina/prova.git"}
]
#+END_DEPS

#+BEGIN_SRC elixir
# Use the prova library
Prova.hello()
#+END_SRC

** Session with deps

#+BEGIN_DEPS elixir
[{:jason, "~> 1.4"}]
#+END_DEPS

#+BEGIN_SRC elixir :session json-session
data = %{users: [%{name: "Alice"}, %{name: "Bob"}]}
#+END_SRC

#+BEGIN_SRC elixir :session json-session
Jason.encode!(data, pretty: true)
#+END_SRC

** Deps override

#+BEGIN_DEPS elixir
[{:httpoison, "~> 2.0"}]
#+END_DEPS

#+BEGIN_SRC elixir
# Jason no longer available here, httpoison is
{:ok, resp} = HTTPoison.get("https://httpbin.org/get")
resp.status_code
#+END_SRC

** No deps (plain execution)

#+BEGIN_SRC elixir
# This block has no deps context (before any deps block or in different file)
Enum.map(1..5, &(&1 * 2))
#+END_SRC

Acceptance Criteria

• #+BEGIN_DEPS elixir block is recognized and parsed
• Deps content is extracted correctly (Elixir list syntax)
• Temp Mix project is created with correct mix.exs
• mix deps.get runs automatically
• mix compile runs automatically
• Subsequent elixir blocks can use the dependencies
• Projects are cached by deps hash (identical deps = same project)
• Cache reuse is fast (no re-compilation)
• Sessions work with deps context (iex -S mix)
• Session maintains deps consistency (recreates if deps change)
• Later deps blocks override earlier ones for subsequent code blocks
• Auto-cleanup on Emacs exit works
• ob-elixir-cleanup-deps-projects command available
• ob-elixir-list-deps-projects command available
• Proper error handling for:
  • Invalid dependency specs
  • Network failures during deps.get
  • Compilation errors
• All tests pass

Header Arguments Reference

This feature does not add new header arguments. Instead, it uses special blocks:

Block	Format	Description
#+BEGIN_DEPS elixir	Elixir list	Dependencies for subsequent code blocks

Supported dependency formats (same as Mix):

# Hex package
{:package_name, "~> 1.0"}

# Git repository  
{:package_name, git: "https://github.com/user/repo.git"}
{:package_name, git: "git@github.com:user/repo.git", tag: "v1.0"}
{:package_name, git: "...", branch: "main"}

# Path (for local development)
{:package_name, path: "../local_package"}

# With options
{:package_name, "~> 1.0", only: :dev}
{:package_name, "~> 1.0", runtime: false}

Troubleshooting

Dependencies not found

Ensure the deps block is before the code block:

#+BEGIN_DEPS elixir    <-- Must come first
[{:jason, "~> 1.4"}]
#+END_DEPS

#+BEGIN_SRC elixir     <-- Code block after deps
Jason.encode!(%{})
#+END_SRC

Network errors during deps.get

Check network connectivity. The error message will include Mix output.

Compilation errors

Check that dependency versions are compatible. Look at the error output for details.

Slow first execution

The first execution with new deps will be slow (fetching + compiling). Subsequent executions with the same deps use the cached project.

Cleanup

To manually clean up cached projects:

M-x ob-elixir-cleanup-deps-projects

To see what's cached:

M-x ob-elixir-list-deps-projects

Files Modified

• ob-elixir.el - Add deps block support
• test/test-ob-elixir-deps.el - Add deps tests

Implementation Notes

Why not use header arguments?

The deps list can be quite long and complex (multiple deps, git URLs, options). A dedicated block is more readable and maintainable than cramming it into a header argument.

Why auto-cleanup?

Temporary projects can consume significant disk space (compiled deps). Auto-cleanup on exit prevents orphaned projects while still allowing cache reuse during a session.

Why hash-based caching?

Computing a hash of the normalized deps string is fast and ensures that:

1. Identical deps always use the same cached project
2. Any change in deps (version, options, etc.) creates a new project
3. No false cache hits from similar-looking deps

References

• Mix Documentation
• Mix.Project deps