# Task 08: Mix Dependencies Support **Phase**: 3 - Mix Integration **Priority**: High **Estimated Time**: 3-4 hours **Dependencies**: Task 07 (Session Support) or Phase 1 complete ## Objective Implement Mix dependencies support so users can specify dependencies directly in the org file using a special `#+BEGIN_DEPS` block. ob-elixir will automatically create a temporary Mix project with those dependencies, cache it for reuse, and use it as the execution context for subsequent Elixir code blocks. ## Prerequisites - Phase 1 complete (or Phase 2 for session+deps) - Understanding of Mix build tool - Network access for fetching dependencies ## Background ### The Problem When writing Elixir code in org-mode, users often need access to external libraries (e.g., JSON parsing, HTTP clients, custom libraries). Without Mix project support, users are limited to Elixir's standard library. ### The Solution Instead of requiring users to point to an existing Mix project, ob-elixir will: 1. Parse dependency specifications from a special `#+BEGIN_DEPS` block 2. Create a temporary Mix project with those dependencies 3. Cache the project based on a hash of the dependencies (for fast re-evaluation) 4. Execute code blocks within that project context 5. Clean up temporary projects when Emacs exits ### Example Usage ```org * Using External Dependencies #+BEGIN_DEPS elixir [ {:jason, "~> 1.4"}, {:prova, git: "git@gitlab.com:babel-upm/makina/prova.git"} ] #+END_DEPS #+BEGIN_SRC elixir # Both jason and prova are available Jason.encode!(%{hello: "world"}) #+END_SRC #+BEGIN_SRC elixir :session deps-session # Session also has access to deps Prova.some_function() #+END_SRC * Different Dependencies Section #+BEGIN_DEPS elixir [{:httpoison, "~> 2.0"}] #+END_DEPS #+BEGIN_SRC elixir # Now using httpoison instead of the previous deps HTTPoison.get!("https://example.com") #+END_SRC ``` ## Design Decisions | Aspect | Decision | Rationale | |--------|----------|-----------| | Block format | `#+BEGIN_DEPS elixir ... #+END_DEPS` | Special block, not confused with executable code | | Deps syntax | Elixir list: `[{:pkg, "~> 1.0"}]` | Natural for Elixir users, supports all dep formats | | Scope | All subsequent blocks until next deps block | Simple mental model | | Caching | Hash-based | Fast re-evaluation, no redundant compilation | | Multiple deps blocks | Later overrides | Allows different sections with different deps | | Cleanup | Auto on Emacs exit | No orphaned temp projects | | Elixir version | System default | Keep it simple, no version management | ## Steps ### Step 1: Add configuration and infrastructure ```elisp ;;; Mix Dependencies Configuration (defcustom ob-elixir-mix-command "mix" "Command to run Mix." :type 'string :group 'ob-elixir :safe #'stringp) (defcustom ob-elixir-deps-cache-dir (expand-file-name "ob-elixir" (or (getenv "XDG_CACHE_HOME") "~/.cache")) "Directory for caching temporary Mix projects. Each unique set of dependencies gets its own subdirectory, named by the hash of the dependencies." :type 'directory :group 'ob-elixir) (defvar ob-elixir--deps-projects (make-hash-table :test 'equal) "Hash table mapping deps-hash to project directory path.") (defvar ob-elixir--cleanup-registered nil "Whether the cleanup hook has been registered.") ``` ### Step 2: Implement deps block parsing ```elisp ;;; Deps Block Parsing (defconst ob-elixir--deps-block-regexp "^[ \t]*#\\+BEGIN_DEPS[ \t]+elixir[ \t]*\n\\(\\(?:.*\n\\)*?\\)[ \t]*#\\+END_DEPS" "Regexp matching a deps block. Group 1 captures the deps content.") (defun ob-elixir--find-deps-for-position (pos) "Find the most recent deps block before POS. Returns the deps content as a string, or nil if no deps block found." (save-excursion (goto-char pos) (let ((found nil)) ;; Search backward for deps blocks (while (and (not found) (re-search-backward ob-elixir--deps-block-regexp nil t)) (when (< (match-end 0) pos) (setq found (match-string-no-properties 1)))) found))) (defun ob-elixir--normalize-deps (deps-string) "Normalize DEPS-STRING for consistent hashing. Removes comments, extra whitespace, and normalizes formatting." (let ((normalized deps-string)) ;; Remove Elixir comments (setq normalized (replace-regexp-in-string "#.*$" "" normalized)) ;; Normalize whitespace (setq normalized (replace-regexp-in-string "[ \t\n]+" " " normalized)) ;; Trim (string-trim normalized))) (defun ob-elixir--hash-deps (deps-string) "Compute SHA256 hash of DEPS-STRING for caching." (secure-hash 'sha256 (ob-elixir--normalize-deps deps-string))) ``` ### Step 3: Implement temp Mix project creation ```elisp ;;; Temporary Mix Project Management (defconst ob-elixir--mix-exs-template "defmodule ObElixirTemp.MixProject do use Mix.Project def project do [ app: :ob_elixir_temp, version: \"0.1.0\", elixir: \"~> 1.14\", start_permanent: false, deps: deps() ] end def application do [extra_applications: [:logger]] end defp deps do %s end end " "Template for temporary Mix project mix.exs file. %s is replaced with the deps list.") (defun ob-elixir--ensure-cache-dir () "Ensure the cache directory exists." (unless (file-directory-p ob-elixir-deps-cache-dir) (make-directory ob-elixir-deps-cache-dir t))) (defun ob-elixir--get-deps-project (deps-string) "Get or create a Mix project for DEPS-STRING. Returns the project directory path, creating and initializing the project if necessary." (let* ((deps-hash (ob-elixir--hash-deps deps-string)) (cached (gethash deps-hash ob-elixir--deps-projects))) (if (and cached (file-directory-p cached)) cached ;; Create new project (ob-elixir--create-deps-project deps-hash deps-string)))) (defun ob-elixir--create-deps-project (deps-hash deps-string) "Create a new temporary Mix project for DEPS-STRING. DEPS-HASH is used as the directory name. Returns the project directory path." (ob-elixir--ensure-cache-dir) (ob-elixir--register-cleanup) (let* ((project-dir (expand-file-name deps-hash ob-elixir-deps-cache-dir)) (mix-exs-path (expand-file-name "mix.exs" project-dir)) (lib-dir (expand-file-name "lib" project-dir))) ;; Create project structure (make-directory project-dir t) (make-directory lib-dir t) ;; Write mix.exs (with-temp-file mix-exs-path (insert (format ob-elixir--mix-exs-template deps-string))) ;; Write a placeholder module (with-temp-file (expand-file-name "ob_elixir_temp.ex" lib-dir) (insert "defmodule ObElixirTemp do\nend\n")) ;; Fetch dependencies (let ((default-directory project-dir)) (message "ob-elixir: Fetching dependencies...") (let ((output (shell-command-to-string (format "%s deps.get 2>&1" ob-elixir-mix-command)))) (when (string-match-p "\\*\\*.*error\\|Could not" output) (error "ob-elixir: Failed to fetch dependencies:\n%s" output))) ;; Compile (message "ob-elixir: Compiling dependencies...") (let ((output (shell-command-to-string (format "%s compile 2>&1" ob-elixir-mix-command)))) (when (string-match-p "\\*\\*.*[Ee]rror" output) (error "ob-elixir: Failed to compile dependencies:\n%s" output)))) ;; Cache and return (puthash deps-hash project-dir ob-elixir--deps-projects) (message "ob-elixir: Dependencies ready") project-dir)) ``` ### Step 4: Implement execution with deps context ```elisp ;;; Execution with Dependencies (defun ob-elixir--execute-with-deps (body result-type deps-string) "Execute BODY with dependencies from DEPS-STRING. RESULT-TYPE is 'value or 'output." (let* ((project-dir (ob-elixir--get-deps-project deps-string)) (default-directory project-dir) (tmp-file (org-babel-temp-file "ob-elixir-" ".exs")) (code (if (eq result-type 'value) (ob-elixir--wrap-for-value body) body))) ;; Write code to temp file (with-temp-file tmp-file (insert code)) ;; Execute with mix run (let ((command (format "%s run --no-compile %s 2>&1" ob-elixir-mix-command (shell-quote-argument tmp-file)))) (ob-elixir--process-result (shell-command-to-string command))))) (defun ob-elixir--start-session-with-deps (buffer-name session-name deps-string) "Start a new IEx session with dependencies in BUFFER-NAME. SESSION-NAME is used for the process name. DEPS-STRING contains the dependencies specification." (let* ((project-dir (ob-elixir--get-deps-project deps-string)) (buffer (get-buffer-create buffer-name)) (default-directory project-dir) (process-environment (cons "TERM=dumb" process-environment))) (with-current-buffer buffer ;; Start IEx with Mix project (make-comint-in-buffer (format "ob-elixir-%s" session-name) buffer ob-elixir-iex-command nil "-S" "mix") ;; Wait for initial prompt (ob-elixir--wait-for-prompt buffer 30) ; Longer timeout for deps loading ;; Configure IEx for programmatic use (ob-elixir--configure-session buffer) buffer))) ``` ### Step 5: Update main execute function Modify `org-babel-execute:elixir` to check for deps context: ```elisp (defun org-babel-execute:elixir (body params) "Execute a block of Elixir code with org-babel. BODY is the Elixir code to execute. PARAMS is an alist of header arguments." (let* ((session (cdr (assq :session params))) (result-type (cdr (assq :result-type params))) (result-params (cdr (assq :result-params params))) ;; Find deps for this block's position (deps-string (ob-elixir--find-deps-for-position (point))) ;; Expand body with variable assignments (full-body (org-babel-expand-body:generic body params (org-babel-variable-assignments:elixir params))) (result (cond ;; Session mode with deps ((and session (not (string= session "none")) deps-string) (ob-elixir--evaluate-in-session-with-deps session full-body result-type deps-string)) ;; Session mode without deps ((and session (not (string= session "none"))) (ob-elixir--evaluate-in-session session full-body result-type)) ;; Non-session with deps (deps-string (ob-elixir--execute-with-deps full-body result-type deps-string)) ;; Plain execution (t (ob-elixir--execute full-body result-type))))) (org-babel-reassemble-table (org-babel-result-cond result-params result (ob-elixir--table-or-string result)) (org-babel-pick-name (cdr (assq :colname-names params)) (cdr (assq :colnames params))) (org-babel-pick-name (cdr (assq :rowname-names params)) (cdr (assq :rownames params)))))) ``` ### Step 6: Implement session support with deps ```elisp (defvar ob-elixir--session-deps (make-hash-table :test 'equal) "Hash table mapping session names to their deps-hash. Used to ensure session deps consistency.") (defun ob-elixir--get-or-create-session-with-deps (name deps-string params) "Get or create an IEx session NAME with DEPS-STRING." (let* ((deps-hash (ob-elixir--hash-deps deps-string)) (buffer-name (format "*ob-elixir:%s*" name)) (existing (get-buffer buffer-name)) (existing-deps (gethash name ob-elixir--session-deps))) ;; Check if existing session has different deps (when (and existing existing-deps (not (string= existing-deps deps-hash))) (message "ob-elixir: Session %s has different deps, recreating..." name) (ob-elixir-kill-session name) (setq existing nil)) (if (and existing (org-babel-comint-buffer-livep existing)) existing ;; Create new session (let ((buffer (ob-elixir--start-session-with-deps buffer-name name deps-string))) (puthash name deps-hash ob-elixir--session-deps) buffer)))) (defun ob-elixir--evaluate-in-session-with-deps (session body result-type deps-string) "Evaluate BODY in SESSION with DEPS-STRING context." (let* ((buffer (ob-elixir--get-or-create-session-with-deps session deps-string nil)) (code (if (eq result-type 'value) (ob-elixir--session-wrap-for-value body) body)) (eoe-indicator ob-elixir--eoe-marker) (full-body (concat code "\nIO.puts(\"" eoe-indicator "\")")) output) (unless buffer (error "Failed to create Elixir session with deps: %s" session)) (setq output (org-babel-comint-with-output (buffer eoe-indicator t full-body) (ob-elixir--send-command buffer full-body))) (ob-elixir--clean-session-output output result-type))) ``` ### Step 7: Implement cleanup ```elisp ;;; Cleanup (defun ob-elixir--register-cleanup () "Register cleanup hook if not already registered." (unless ob-elixir--cleanup-registered (add-hook 'kill-emacs-hook #'ob-elixir-cleanup-deps-projects) (setq ob-elixir--cleanup-registered t))) (defun ob-elixir-cleanup-deps-projects () "Delete all temporary Mix projects created by ob-elixir. Called automatically on Emacs exit." (interactive) (maphash (lambda (_hash project-dir) (when (and project-dir (file-directory-p project-dir)) (condition-case err (delete-directory project-dir t) (error (message "ob-elixir: Failed to delete %s: %s" project-dir err))))) ob-elixir--deps-projects) (clrhash ob-elixir--deps-projects) (message "ob-elixir: Cleaned up temporary Mix projects")) (defun ob-elixir-list-deps-projects () "List all cached dependency projects." (interactive) (if (= (hash-table-count ob-elixir--deps-projects) 0) (message "No cached dependency projects") (with-output-to-temp-buffer "*ob-elixir deps projects*" (princ "Cached ob-elixir dependency projects:\n\n") (maphash (lambda (hash dir) (princ (format " %s\n -> %s\n" (substring hash 0 12) dir))) ob-elixir--deps-projects)))) ``` ### Step 8: Add tests Create `test/test-ob-elixir-deps.el`: ```elisp ;;; test-ob-elixir-deps.el --- Deps block tests -*- lexical-binding: t; -*- (require 'ert) (require 'ob-elixir) ;;; Parsing Tests (ert-deftest ob-elixir-test-deps-block-parsing () "Test finding deps blocks in buffer." (with-temp-buffer (insert "#+BEGIN_DEPS elixir\n[{:jason, \"~> 1.4\"}]\n#+END_DEPS\n\n") (insert "#+BEGIN_SRC elixir\nJason.encode!(%{})\n#+END_SRC\n") (goto-char (point-max)) (let ((deps (ob-elixir--find-deps-for-position (point)))) (should deps) (should (string-match-p ":jason" deps))))) (ert-deftest ob-elixir-test-deps-block-override () "Test that later deps blocks override earlier ones." (with-temp-buffer (insert "#+BEGIN_DEPS elixir\n[{:jason, \"~> 1.4\"}]\n#+END_DEPS\n\n") (insert "#+BEGIN_SRC elixir\ncode1\n#+END_SRC\n\n") (insert "#+BEGIN_DEPS elixir\n[{:httpoison, \"~> 2.0\"}]\n#+END_DEPS\n\n") (let ((pos (point))) (insert "#+BEGIN_SRC elixir\ncode2\n#+END_SRC\n") (let ((deps (ob-elixir--find-deps-for-position (+ pos 10)))) (should deps) (should (string-match-p ":httpoison" deps)) (should-not (string-match-p ":jason" deps)))))) (ert-deftest ob-elixir-test-no-deps-block () "Test behavior when no deps block exists." (with-temp-buffer (insert "#+BEGIN_SRC elixir\n1 + 1\n#+END_SRC\n") (should (null (ob-elixir--find-deps-for-position (point)))))) (ert-deftest ob-elixir-test-deps-hash-consistency () "Test that same deps produce same hash." (let ((deps1 "[{:jason, \"~> 1.4\"}]") (deps2 "[{:jason, \"~> 1.4\"}]") ; extra space (deps3 "[{:httpoison, \"~> 2.0\"}]")) (should (string= (ob-elixir--hash-deps deps1) (ob-elixir--hash-deps deps2))) (should-not (string= (ob-elixir--hash-deps deps1) (ob-elixir--hash-deps deps3))))) (ert-deftest ob-elixir-test-normalize-deps () "Test deps normalization." (should (string= (ob-elixir--normalize-deps "[ {:a, \"1\"} ]") (ob-elixir--normalize-deps "[{:a,\"1\"}]"))) ;; Comments are stripped (should (string= (ob-elixir--normalize-deps "[{:a, \"1\"}] # comment") (ob-elixir--normalize-deps "[{:a, \"1\"}]")))) ;;; Integration Tests (require network and mix) (ert-deftest ob-elixir-test-deps-project-creation () "Test creating a temporary Mix project with deps." (skip-unless (and (executable-find ob-elixir-mix-command) (executable-find ob-elixir-command))) (let ((ob-elixir-deps-cache-dir (make-temp-file "ob-elixir-test-" t)) (deps "[{:jason, \"~> 1.4\"}]")) (unwind-protect (let ((project-dir (ob-elixir--get-deps-project deps))) (should (file-directory-p project-dir)) (should (file-exists-p (expand-file-name "mix.exs" project-dir))) (should (file-directory-p (expand-file-name "deps/jason" project-dir)))) ;; Cleanup (ob-elixir-cleanup-deps-projects) (delete-directory ob-elixir-deps-cache-dir t)))) (ert-deftest ob-elixir-test-deps-execution () "Test executing code with deps." (skip-unless (and (executable-find ob-elixir-mix-command) (executable-find ob-elixir-command))) (let ((ob-elixir-deps-cache-dir (make-temp-file "ob-elixir-test-" t)) (deps "[{:jason, \"~> 1.4\"}]")) (unwind-protect (let ((result (ob-elixir--execute-with-deps "Jason.encode!(%{a: 1})" 'value deps))) (should (string-match-p "\"a\"" result))) ;; Cleanup (ob-elixir-cleanup-deps-projects) (delete-directory ob-elixir-deps-cache-dir t)))) (ert-deftest ob-elixir-test-deps-caching () "Test that deps projects are cached and reused." (skip-unless (executable-find ob-elixir-mix-command)) (let ((ob-elixir-deps-cache-dir (make-temp-file "ob-elixir-test-" t)) (deps "[{:jason, \"~> 1.4\"}]")) (unwind-protect (let ((project1 (ob-elixir--get-deps-project deps)) (project2 (ob-elixir--get-deps-project deps))) ;; Should return same directory (should (string= project1 project2)) ;; Should only have one entry in hash table (should (= 1 (hash-table-count ob-elixir--deps-projects)))) ;; Cleanup (ob-elixir-cleanup-deps-projects) (delete-directory ob-elixir-deps-cache-dir t)))) (provide 'test-ob-elixir-deps) ``` ### Step 9: Test in org buffer Create test cases in `test.org`: ```org * Deps Block Tests ** Basic deps usage #+BEGIN_DEPS elixir [{:jason, "~> 1.4"}] #+END_DEPS #+BEGIN_SRC elixir Jason.encode!(%{hello: "world", number: 42}) #+END_SRC ** Multiple dependencies #+BEGIN_DEPS elixir [ {:jason, "~> 1.4"}, {:decimal, "~> 2.0"} ] #+END_DEPS #+BEGIN_SRC elixir num = Decimal.new("3.14159") Jason.encode!(%{pi: Decimal.to_string(num)}) #+END_SRC ** Git dependencies #+BEGIN_DEPS elixir [ {:prova, git: "git@gitlab.com:babel-upm/makina/prova.git"} ] #+END_DEPS #+BEGIN_SRC elixir # Use the prova library Prova.hello() #+END_SRC ** Session with deps #+BEGIN_DEPS elixir [{:jason, "~> 1.4"}] #+END_DEPS #+BEGIN_SRC elixir :session json-session data = %{users: [%{name: "Alice"}, %{name: "Bob"}]} #+END_SRC #+BEGIN_SRC elixir :session json-session Jason.encode!(data, pretty: true) #+END_SRC ** Deps override #+BEGIN_DEPS elixir [{:httpoison, "~> 2.0"}] #+END_DEPS #+BEGIN_SRC elixir # Jason no longer available here, httpoison is {:ok, resp} = HTTPoison.get("https://httpbin.org/get") resp.status_code #+END_SRC ** No deps (plain execution) #+BEGIN_SRC elixir # This block has no deps context (before any deps block or in different file) Enum.map(1..5, &(&1 * 2)) #+END_SRC ``` ## Acceptance Criteria - [ ] `#+BEGIN_DEPS elixir` block is recognized and parsed - [ ] Deps content is extracted correctly (Elixir list syntax) - [ ] Temp Mix project is created with correct `mix.exs` - [ ] `mix deps.get` runs automatically - [ ] `mix compile` runs automatically - [ ] Subsequent elixir blocks can use the dependencies - [ ] Projects are cached by deps hash (identical deps = same project) - [ ] Cache reuse is fast (no re-compilation) - [ ] Sessions work with deps context (`iex -S mix`) - [ ] Session maintains deps consistency (recreates if deps change) - [ ] Later deps blocks override earlier ones for subsequent code blocks - [ ] Auto-cleanup on Emacs exit works - [ ] `ob-elixir-cleanup-deps-projects` command available - [ ] `ob-elixir-list-deps-projects` command available - [ ] Proper error handling for: - Invalid dependency specs - Network failures during `deps.get` - Compilation errors - [ ] All tests pass ## Header Arguments Reference This feature does not add new header arguments. Instead, it uses special blocks: | Block | Format | Description | |-------|--------|-------------| | `#+BEGIN_DEPS elixir` | Elixir list | Dependencies for subsequent code blocks | Supported dependency formats (same as Mix): ```elixir # Hex package {:package_name, "~> 1.0"} # Git repository {:package_name, git: "https://github.com/user/repo.git"} {:package_name, git: "git@github.com:user/repo.git", tag: "v1.0"} {:package_name, git: "...", branch: "main"} # Path (for local development) {:package_name, path: "../local_package"} # With options {:package_name, "~> 1.0", only: :dev} {:package_name, "~> 1.0", runtime: false} ``` ## Troubleshooting ### Dependencies not found Ensure the deps block is before the code block: ```org #+BEGIN_DEPS elixir <-- Must come first [{:jason, "~> 1.4"}] #+END_DEPS #+BEGIN_SRC elixir <-- Code block after deps Jason.encode!(%{}) #+END_SRC ``` ### Network errors during deps.get Check network connectivity. The error message will include Mix output. ### Compilation errors Check that dependency versions are compatible. Look at the error output for details. ### Slow first execution The first execution with new deps will be slow (fetching + compiling). Subsequent executions with the same deps use the cached project. ### Cleanup To manually clean up cached projects: ``` M-x ob-elixir-cleanup-deps-projects ``` To see what's cached: ``` M-x ob-elixir-list-deps-projects ``` ## Files Modified - `ob-elixir.el` - Add deps block support - `test/test-ob-elixir-deps.el` - Add deps tests ## Implementation Notes ### Why not use header arguments? The deps list can be quite long and complex (multiple deps, git URLs, options). A dedicated block is more readable and maintainable than cramming it into a header argument. ### Why auto-cleanup? Temporary projects can consume significant disk space (compiled deps). Auto-cleanup on exit prevents orphaned projects while still allowing cache reuse during a session. ### Why hash-based caching? Computing a hash of the normalized deps string is fast and ensures that: 1. Identical deps always use the same cached project 2. Any change in deps (version, options, etc.) creates a new project 3. No false cache hits from similar-looking deps ## References - [Mix Documentation](https://hexdocs.pm/mix/Mix.html) - [Mix.Project deps](https://hexdocs.pm/mix/Mix.Tasks.Deps.html)