Bumped version

[Dev] Version updates 2024-09-12 (#388 )
* Updated some deps * Updated phoenix
2024-09-12 14:39:07 -07:00 · 2024-09-12 13:04:49 -07:00 · 2024-09-12 12:38:12 -07:00 · 2024-09-10 14:16:10 -07:00 · 2024-09-10 11:44:57 -07:00 · 2024-09-10 11:42:10 -07:00
174 changed files with 5167 additions and 1571 deletions
@@ -0,0 +1,12 @@
+{
+  "name": "Pinchflat Dev",
+  "dockerComposeFile": "../docker-compose.yml",
+  "service": "phx",
+  "workspaceFolder": "/app",
+  "shutdownAction": "stopCompose",
+  "customizations": {
+    "vscode": {
+      "extensions": ["phoenixframework.phoenix", "JakeBecker.elixir-ls", "esbenp.prettier-vscode"]
+    }
+  }
+}
@@ -1,3 +1,5 @@
+# TODO: figure out why my vscode extension doesn't respect the formatter.exs file
+# if it's in a subdirectory
 [
  import_deps: [:ecto, :ecto_sql, :phoenix],
  subdirectories: ["priv/*/migrations"],
@@ -0,0 +1,34 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: '[Triage] <your title here> '
+labels: triage
+assignees: kieraneglin
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Diagnostic info**
+
+<!-- Go to Config > App Info > Copy Diagnostic Info and paste that here -->
+
+**Additional context**
+
+<!-- Go to Config > App Info > Download Logs and attach them, if applicable -->
+
+Add any other context about the problem here.
@@ -0,0 +1,19 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: '[FR] <your title here>'
+labels: feature request
+assignees: kieraneglin
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. It's too complicated to [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
@@ -0,0 +1,7 @@
+---
+name: Other
+about: For everything else
+title: ''
+labels: ''
+assignees: kieraneglin
+---
@@ -11,6 +11,12 @@ on:
        options:
          - 'linux/amd64'
          - 'linux/amd64,linux/arm64'
+      docker_tags:
+        type: string
+        description: 'Docker Tags'
+        required: true
+        default: 'dev'
+
  push:
    branches:
      - master
@@ -41,7 +47,8 @@ jobs:
          # All non-release actions will be tagged as `dev` (ie: push, workflow_dispatch)
          tags: |
            type=ref,event=tag
-            type=raw,value=dev,enable=${{ github.event_name != 'release' }}
+            type=raw,value=dev,enable=${{ github.event_name != 'release' && github.event_name != 'workflow_dispatch' }}
+            type=raw,value=${{ inputs.docker_tags }},enable=${{ github.event_name == 'workflow_dispatch' }}
          flavor: |
            latest=auto

@@ -65,10 +72,10 @@ jobs:
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and Push
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
        with:
          context: .
-          file: ./selfhosted.Dockerfile
+          file: ./docker/selfhosted.Dockerfile
          # If the event is a release, use the release_arch, otherwise use the
          # platforms input if present, falling back to dev_arch
          platforms: ${{ github.event_name == 'release' && env.release_arch || (github.event.inputs.platforms || env.dev_arch) }}
@@ -16,24 +16,28 @@ jobs:
    if: "! contains(toJSON(github.event.commits.*.message), '[skip ci]')"
    env:
      COMPOSE_FILE: ./docker-compose.ci.yml
+      MIX_ENV: test

    steps:
      - name: Checkout code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4

      - name: Pull prebuilt images
        run: docker compose pull

-      - name: Setup Docker layer caching
-        uses: jpribyl/action-docker-layer-caching@v0.1.1
-        continue-on-error: true
-        with:
-          key: ci-docker-cache-{hash}
-          restore-keys: |
-            ci-docker-cache-
-            layer-ci-docker-cache-
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3

-      - name: Build and Run Docker image
+      - name: Build docker image
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: ./docker/dev.Dockerfile
+          load: true
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Run Docker image
        run: docker compose up --detach

      # NOTE: All exec commands use the -T flag to compensate for
@@ -43,8 +47,7 @@ jobs:
      # See https://github.com/actions/runner/issues/241 and https://github.com/docker/compose/issues/8537
      - name: Install Elixir and JS deps
        run: |
-          docker compose exec -T phx yarn install && cd assets && yarn install && cd ..
-          docker compose exec -T phx mix deps.get
+          docker compose exec -T phx mix deps.get && yarn install && cd assets && yarn install && cd ..

      - name: Create and Migrate database
        run: |
@@ -23,3 +23,11 @@ alias Pinchflat.Metadata.MetadataFileHelpers
 alias Pinchflat.SlowIndexing.FileFollowerServer

 Pinchflat.Release.check_file_permissions()
+
+defmodule IexHelpers do
+  def restart do
+    :init.restart()
+  end
+end
+
+import IexHelpers
@@ -1,3 +1 @@
 assets/vendor/
-deps/
-_build/
@@ -17,6 +17,7 @@
 [![](https://img.shields.io/github/license/kieraneglin/pinchflat?style=for-the-badge&color=ee512b)](LICENSE)
 [![](https://img.shields.io/github/v/release/kieraneglin/pinchflat?style=for-the-badge)](https://github.com/kieraneglin/pinchflat/releases)
 [![](https://img.shields.io/github/actions/workflow/status/kieraneglin/pinchflat/lint_and_test.yml?style=for-the-badge)](#)
+[![](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode&style=for-the-badge)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/kieraneglin/pinchflat)

 </div>

@@ -31,6 +32,7 @@
  - [Unraid](#unraid)
  - [Portainer](#portainer)
  - [Docker](#docker)
+  - [Environment Variables](#environment-variables)
 - [Username and Password (authentication)](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password)
 - [Frequently asked questions](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions)
 - [Documentation](https://github.com/kieraneglin/pinchflat/wiki)
@@ -70,8 +72,8 @@ If it doesn't work for your use case, please make a feature request! You can als

 ## Screenshots

-<img src="priv/static/images/app-form-screenshot.png" alt="Pinchflat screenshot" width="700" />
-<img src="priv/static/images/app-screenshot.png" alt="Pinchflat screenshot" width="700" />
+<img src="priv/static/images/app-form-screenshot.jpg" alt="Pinchflat screenshot" width="700" />
+<img src="priv/static/images/app-screenshot.jpg" alt="Pinchflat screenshot" width="700" />

 ## Installation

@@ -90,7 +92,7 @@ Docker Compose file:
 version: '3'
 services:
  pinchflat:
-    image: keglin/pinchflat:latest
+    image: ghcr.io/kieraneglin/pinchflat:latest
    environment:
      # Set the timezone to your local timezone
      - TZ=America/New_York
@@ -107,7 +109,7 @@ services:
 2. Prepare the docker image in one of the two ways below:
   - **From GHCR:** `docker pull ghcr.io/kieraneglin/pinchflat:latest`
     - NOTE: also available on Docker Hub at `keglin/pinchflat:latest`
-   - **Building locally:** `docker build . --file selfhosted.Dockerfile -t ghcr.io/kieraneglin/pinchflat:latest`
+   - **Building locally:** `docker build . --file docker/selfhosted.Dockerfile -t ghcr.io/kieraneglin/pinchflat:latest`
 3. Run the container:

 ```bash
@@ -141,6 +143,21 @@ As pointed out in [#137](https://github.com/kieraneglin/pinchflat/issues/137), S

 If you change this setting and it works well for you, please leave a comment on [#137](https://github.com/kieraneglin/pinchflat/issues/137)! Doubly so if it does _not_ work well.

+### Environment variables
+
+| Name                        | Required? | Default                   | Notes                                                                                                                           |
+| --------------------------- | --------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
+| `TZ`                        | No        | `UTC`                     | Must follow IANA TZ format                                                                                                      |
+| `LOG_LEVEL`                 | No        | `debug`                   | Can be set to `info` but `debug` is strongly recommended                                                                        |
+| `BASIC_AUTH_USERNAME`       | No        |                           | See [authentication docs](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password)                                  |
+| `BASIC_AUTH_PASSWORD`       | No        |                           | See [authentication docs](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password)                                  |
+| `EXPOSE_FEED_ENDPOINTS`     | No        | `false`                   | See [RSS feed docs](https://github.com/kieraneglin/pinchflat/wiki/Podcast-RSS-Feeds)                                            |
+| `ENABLE_IPV6`               | No        | `false`                   | Setting to _any_ non-blank value will enable IPv6                                                                               |
+| `JOURNAL_MODE`              | No        | `wal`                     | Set to `delete` if your config directory is stored on a network share (not recommended)                                         |
+| `TZ_DATA_DIR`               | No        | `/etc/elixir_tzdata_data` | The container path where the timezone database is stored                                                                        |
+| `BASE_ROUTE_PATH`           | No        | `/`                       | The base path for route generation. Useful when running behind certain reverse proxies                                          |
+| `YT_DLP_WORKER_CONCURRENCY` | No        | `2`                       | The number of concurrent workers that use `yt-dlp` _per queue_. Set to 1 if you're getting IP limited, otherwise don't touch it |
+
 ## EFF donations

 Prior to 2024-05-10, a portion of all donations were given to the [Electronic Frontier Foundation](https://www.eff.org/). Now, the app doesn't accept donations that go to me personally and instead directs you straight to the EFF. [Here](https://github.com/kieraneglin/pinchflat/issues/234) are some people that have generously donated.
@@ -22,8 +22,8 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-Light.woff2'), url('/fonts/satoshi/Satoshi-Light.woff'),
-    url('/fonts/satoshi/Satoshi-Light.ttf');
+  src: url('../fonts/satoshi/Satoshi-Light.woff2'), url('../fonts/satoshi/Satoshi-Light.woff'),
+    url('../fonts/satoshi/Satoshi-Light.ttf');
  font-weight: 300;
  font-display: swap;
  font-style: normal;
@@ -31,8 +31,9 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-LightItalic.woff2'),
-    url('/fonts/satoshi/Satoshi-LightItalic.woff'), url('/fonts/satoshi/Satoshi-LightItalic.ttf');
+  src: url('../fonts/satoshi/Satoshi-LightItalic.woff2'),
+    url('../fonts/satoshi/Satoshi-LightItalic.woff'),
+    url('../fonts/satoshi/Satoshi-LightItalic.ttf');
  font-weight: 300;
  font-display: swap;
  font-style: italic;
@@ -40,8 +41,8 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-Regular.woff2'), url('/fonts/satoshi/Satoshi-Regular.woff'),
-    url('/fonts/satoshi/Satoshi-Regular.ttf');
+  src: url('../fonts/satoshi/Satoshi-Regular.woff2'), url('../fonts/satoshi/Satoshi-Regular.woff'),
+    url('../fonts/satoshi/Satoshi-Regular.ttf');
  font-weight: 400;
  font-display: swap;
  font-style: normal;
@@ -49,8 +50,8 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-Italic.woff2'), url('/fonts/satoshi/Satoshi-Italic.woff'),
-    url('/fonts/satoshi/Satoshi-Italic.ttf');
+  src: url('../fonts/satoshi/Satoshi-Italic.woff2'), url('../fonts/satoshi/Satoshi-Italic.woff'),
+    url('../fonts/satoshi/Satoshi-Italic.ttf');
  font-weight: 400;
  font-display: swap;
  font-style: italic;
@@ -58,8 +59,8 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-Medium.woff2'), url('/fonts/satoshi/Satoshi-Medium.woff'),
-    url('/fonts/satoshi/Satoshi-Medium.ttf');
+  src: url('../fonts/satoshi/Satoshi-Medium.woff2'), url('../fonts/satoshi/Satoshi-Medium.woff'),
+    url('../fonts/satoshi/Satoshi-Medium.ttf');
  font-weight: 500;
  font-display: swap;
  font-style: normal;
@@ -67,8 +68,9 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-MediumItalic.woff2'),
-    url('/fonts/satoshi/Satoshi-MediumItalic.woff'), url('/fonts/satoshi/Satoshi-MediumItalic.ttf');
+  src: url('../fonts/satoshi/Satoshi-MediumItalic.woff2'),
+    url('../fonts/satoshi/Satoshi-MediumItalic.woff'),
+    url('../fonts/satoshi/Satoshi-MediumItalic.ttf');
  font-weight: 500;
  font-display: swap;
  font-style: italic;
@@ -76,8 +78,8 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-Bold.woff2'), url('/fonts/satoshi/Satoshi-Bold.woff'),
-    url('/fonts/satoshi/Satoshi-Bold.ttf');
+  src: url('../fonts/satoshi/Satoshi-Bold.woff2'), url('../fonts/satoshi/Satoshi-Bold.woff'),
+    url('../fonts/satoshi/Satoshi-Bold.ttf');
  font-weight: 700;
  font-display: swap;
  font-style: normal;
@@ -85,8 +87,8 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-BoldItalic.woff2'), url('/fonts/satoshi/Satoshi-BoldItalic.woff'),
-    url('/fonts/satoshi/Satoshi-BoldItalic.ttf');
+  src: url('../fonts/satoshi/Satoshi-BoldItalic.woff2'),
+    url('../fonts/satoshi/Satoshi-BoldItalic.woff'), url('../fonts/satoshi/Satoshi-BoldItalic.ttf');
  font-weight: 700;
  font-display: swap;
  font-style: italic;
@@ -94,8 +96,9 @@

@font-face {
  font-family: 'Satoshi';
-  src: url('/fonts/satoshi/Satoshi-BlackItalic.woff2'),
-    url('/fonts/satoshi/Satoshi-BlackItalic.woff'), url('/fonts/satoshi/Satoshi-BlackItalic.ttf');
+  src: url('../fonts/satoshi/Satoshi-BlackItalic.woff2'),
+    url('../fonts/satoshi/Satoshi-BlackItalic.woff'),
+    url('../fonts/satoshi/Satoshi-BlackItalic.ttf');
  font-weight: 900;
  font-display: swap;
  font-style: italic;
@@ -29,7 +29,7 @@ window.Alpine = Alpine
 Alpine.start()

 let csrfToken = document.querySelector("meta[name='csrf-token']").getAttribute('content')
-let liveSocket = new LiveSocket('/live', Socket, {
+let liveSocket = new LiveSocket(document.body.dataset.socketPath, Socket, {
  params: { _csrf_token: csrfToken },
  dom: {
    onBeforeElUpdated(from, to) {
@@ -1,20 +1,20 @@
-window.setTabIndex = (index) => {
-  window.location.hash = `tab-${index}`
+window.setTabByName = (tabName) => {
+  window.location.hash = `tab-${tabName}`

-  return index
+  return tabName
 }

 // The conditionals and currIndex stuff ensures that
 // the tab index is always set to 0 if the hash is empty
 // AND other hash values are ignored
-window.getTabIndex = (currIndex) => {
+window.getTabFromHash = (currentTabName, defaultTabName) => {
  if (window.location.hash === '' || window.location.hash === '#') {
-    return 0
+    return defaultTabName
  }

  if (window.location.hash.startsWith('#tab-')) {
-    return parseInt(window.location.hash.replace('#tab-', ''))
+    return window.location.hash.replace('#tab-', '')
  }

-  return currIndex
+  return currentTabName
 }
@@ -26,7 +26,8 @@ config :pinchflat,
  basic_auth_password: "",
  expose_feed_endpoints: false,
  file_watcher_poll_interval: 1000,
-  timezone: "UTC"
+  timezone: "UTC",
+  base_route_path: "/"

 config :pinchflat, Pinchflat.Repo,
  journal_mode: :wal,
@@ -40,7 +41,8 @@ config :pinchflat, PinchflatWeb.Endpoint,
  adapter: Phoenix.Endpoint.Cowboy2Adapter,
  render_errors: [
    formats: [html: PinchflatWeb.ErrorHTML, json: PinchflatWeb.ErrorJSON],
-    layout: false
+    root_layout: {PinchflatWeb.Layouts, :root},
+    layout: {PinchflatWeb.Layouts, :app}
  ],
  pubsub_server: Pinchflat.PubSub,
  live_view: [signing_salt: "/t5878kO"]
@@ -56,16 +58,6 @@ config :pinchflat, Oban,
       {"0 1 * * *", Pinchflat.Downloading.MediaRetentionWorker},
       {"0 2 * * *", Pinchflat.Downloading.MediaQualityUpgradeWorker}
     ]}
-  ],
-  # TODO: consider making this an env var or something?
-  queues: [
-    default: 10,
-    fast_indexing: 6,
-    media_indexing: 2,
-    media_collection_indexing: 2,
-    media_fetching: 2,
-    local_metadata: 8,
-    remote_metadata: 4
  ]

 # Configures the mailer
@@ -100,8 +92,8 @@ config :tailwind,
  ]

 # Configures Elixir's Logger
-config :logger, :console,
-  format: "$time $metadata[$level] $message\n",
+config :logger, :default_formatter,
+  format: "$date $time $metadata[$level] | $message\n",
  metadata: [:request_id]

 # Use Jason for JSON parsing in Phoenix
@@ -67,7 +67,7 @@ config :pinchflat, PinchflatWeb.Endpoint,
 config :pinchflat, dev_routes: true

 # Do not include metadata nor timestamps in development logs
-config :logger, :console, format: "[$level] $message\n"
+config :logger, :default_formatter, format: "[$level] $message\n"

 # Set a higher stacktrace during development. Avoid configuring such
 # in production as building large stacktraces may be expensive.
@@ -40,18 +40,33 @@ config :pinchflat, Pinchflat.Repo,
    Path.join([:code.priv_dir(:pinchflat), "repo", "extensions", "sqlean-linux-#{system_arch}", "sqlean"])
  ]

+# Some users may want to increase the number of workers that use yt-dlp to improve speeds
+# Others may want to decrease the number of these workers to lessen the chance of an IP ban
+{yt_dlp_worker_count, _} = Integer.parse(System.get_env("YT_DLP_WORKER_CONCURRENCY", "2"))
+
+config :pinchflat, Oban,
+  queues: [
+    default: 10,
+    fast_indexing: 6,
+    media_collection_indexing: yt_dlp_worker_count,
+    media_fetching: yt_dlp_worker_count,
+    remote_metadata: yt_dlp_worker_count,
+    local_data: 8
+  ]
+
 if config_env() == :prod do
  config_path = "/config"
  db_path = System.get_env("DATABASE_PATH", Path.join([config_path, "db", "pinchflat.db"]))
  log_path = System.get_env("LOG_PATH", Path.join([config_path, "logs", "pinchflat.log"]))
  metadata_path = System.get_env("METADATA_PATH", Path.join([config_path, "metadata"]))
  extras_path = System.get_env("EXTRAS_PATH", Path.join([config_path, "extras"]))
-
  # For running PF as a podcast host on self-hosted environments
  expose_feed_endpoints = String.length(System.get_env("EXPOSE_FEED_ENDPOINTS", "")) > 0
-
  # For testing alternate journal modes (see issue #137)
  journal_mode = String.to_existing_atom(System.get_env("JOURNAL_MODE", "wal"))
+  # For running PF in a subdirectory via a reverse proxy
+  base_route_path = System.get_env("BASE_ROUTE_PATH", "/")
+  enable_ipv6 = String.length(System.get_env("ENABLE_IPV6", "")) > 0

  config :logger, level: String.to_existing_atom(System.get_env("LOG_LEVEL", "debug"))

@@ -65,7 +80,8 @@ if config_env() == :prod do
    dns_cluster_query: System.get_env("DNS_CLUSTER_QUERY"),
    expose_feed_endpoints: expose_feed_endpoints,
    timezone: System.get_env("TIMEZONE") || System.get_env("TZ") || "UTC",
-    log_path: log_path
+    log_path: log_path,
+    base_route_path: base_route_path

  config :tzdata, :data_dir, System.get_env("TZ_DATA_DIR", "/etc/elixir_tzdata_data")

@@ -105,9 +121,10 @@ if config_env() == :prod do
      # Set it to  {0, 0, 0, 0, 0, 0, 0, 1} for local network only access.
      # See the documentation on https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html
      # for details about using IPv6 vs IPv4 and loopback vs public addresses.
-      ip: {0, 0, 0, 0},
+      ip: if(enable_ipv6, do: {0, 0, 0, 0, 0, 0, 0, 0}, else: {0, 0, 0, 0}),
      port: String.to_integer(System.get_env("PORT") || "4000")
    ],
+    url: [path: base_route_path],
    secret_key_base: secret_key_base

  config :pinchflat, :logger, [
@@ -1,65 +0,0 @@
-ARG ELIXIR_VERSION=1.16.2
-ARG OTP_VERSION=26.2.2
-ARG DEBIAN_VERSION=bookworm-20240130
-ARG DEV_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
-
-FROM ${DEV_IMAGE}
-
-ARG TARGETPLATFORM
-RUN echo "Building for ${TARGETPLATFORM:?}"
-
-# Install debian packages
-RUN apt-get update -qq
-RUN apt-get install -y inotify-tools curl git openssh-client jq \
-  python3 python3-setuptools python3-wheel python3-dev pipx \
-  python3-mutagen locales procps build-essential
-
-# Install ffmpeg
-RUN export FFMPEG_DOWNLOAD=$(case ${TARGETPLATFORM:-linux/amd64} in \
-    "linux/amd64")   echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz"   ;; \
-    "linux/arm64")   echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linuxarm64-gpl.tar.xz" ;; \
-    *)               echo ""        ;; esac) && \
-    curl -L ${FFMPEG_DOWNLOAD} --output /tmp/ffmpeg.tar.xz && \
-    tar -xf /tmp/ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
-    tar -xf /tmp/ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffprobe"
-
-# Install nodejs
-RUN curl -sL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh
-RUN bash nodesource_setup.sh
-RUN apt-get install nodejs
-RUN npm install -g yarn
-
-# Install baseline Elixir packages
-RUN mix local.hex --force
-RUN mix local.rebar --force
-
-# Download and update YT-DLP
-RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
-RUN chmod a+rx /usr/local/bin/yt-dlp
-RUN yt-dlp -U
-
-# Install Apprise
-RUN export PIPX_HOME=/opt/pipx && \
-    export PIPX_BIN_DIR=/usr/local/bin && \
-    pipx install apprise
-
-# Set the locale
-RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
-ENV LANG en_US.UTF-8
-ENV LANGUAGE en_US:en
-ENV LC_ALL en_US.UTF-8
-
-# Create app directory and copy the Elixir projects into it.
-WORKDIR /app
-COPY . ./
-
-# Needs permissions to be updated AFTER the copy step
-RUN chmod +x ./docker-run.dev.sh
-
-# Install Elixir deps
-# RUN mix archive.install github hexpm/hex branch latest
-RUN mix deps.get
-# Gives us iex shell history
-ENV ERL_AFLAGS="-kernel shell_history enabled"
-
-EXPOSE 4008
@@ -1,13 +1,15 @@
-version: '3'
 services:
  phx:
    build:
      context: .
-      dockerfile: dev.Dockerfile
+      dockerfile: ./docker/dev.Dockerfile
    environment:
      - MIX_ENV=test
    volumes:
      - '.:/app'
+      # These lines ensure the deps can be saved as build artifacts for caching
+      - '/app/deps'
+      - '/app/_build'
    ports:
      - '4008:4008'
    command: tail -F /dev/null
@@ -1,14 +1,12 @@
-version: '3'
 services:
  phx:
    build:
      context: .
-      dockerfile: dev.Dockerfile
+      dockerfile: ./docker/dev.Dockerfile
    volumes:
      - '.:/app'
    ports:
      - '4008:4008'
-    command:
-      - ./docker-run.dev.sh
+    command: bash -c "chmod +x docker/docker-run.dev.sh && docker/docker-run.dev.sh"
    stdin_open: true
    tty: true
@@ -0,0 +1,65 @@
+ARG ELIXIR_VERSION=1.17.0
+ARG OTP_VERSION=26.2.5
+ARG DEBIAN_VERSION=bookworm-20240612-slim
+ARG DEV_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
+
+FROM ${DEV_IMAGE}
+
+ARG TARGETPLATFORM
+RUN echo "Building for ${TARGETPLATFORM:?}"
+
+# Install debian packages
+RUN apt-get update -qq && \
+  apt-get install -y inotify-tools curl git openssh-client jq \
+    python3 python3-setuptools python3-wheel python3-dev pipx \
+    python3-mutagen locales procps build-essential graphviz zsh
+
+# Install ffmpeg
+RUN export FFMPEG_DOWNLOAD=$(case ${TARGETPLATFORM:-linux/amd64} in \
+    "linux/amd64")   echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz"   ;; \
+    "linux/arm64")   echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linuxarm64-gpl.tar.xz" ;; \
+    *)               echo ""        ;; esac) && \
+    curl -L ${FFMPEG_DOWNLOAD} --output /tmp/ffmpeg.tar.xz && \
+    tar -xf /tmp/ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
+    tar -xf /tmp/ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffprobe"
+
+# Install nodejs and Yarn
+RUN curl -sL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh && \
+  bash nodesource_setup.sh && \
+  apt-get install -y nodejs && \
+  npm install -g yarn && \
+  # Install baseline Elixir packages
+  mix local.hex --force && \
+  mix local.rebar --force && \
+  # Download and update YT-DLP
+  curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp && \
+  chmod a+rx /usr/local/bin/yt-dlp && \
+  yt-dlp -U && \
+  # Install Apprise
+  export PIPX_HOME=/opt/pipx && \
+  export PIPX_BIN_DIR=/usr/local/bin && \
+  pipx install apprise && \
+  # Set up ZSH tools
+  chsh -s $(which zsh) && \
+  sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
+
+# Set the locale
+RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
+ENV LANG=en_US.UTF-8
+ENV LANGUAGE=en_US:en
+ENV LC_ALL=en_US.UTF-8
+
+WORKDIR /app
+
+COPY mix.exs mix.lock ./
+# Install Elixir deps
+# NOTE: this has to be before the bulk copy to ensure that deps are cached
+RUN MIX_ENV=dev mix deps.get && MIX_ENV=dev mix deps.compile
+RUN MIX_ENV=test mix deps.get && MIX_ENV=test mix deps.compile
+
+COPY . ./
+
+# Gives us iex shell history
+ENV ERL_AFLAGS="-kernel shell_history enabled"
+
+EXPOSE 4008
@@ -2,12 +2,12 @@

 set -e

-# Ensure the app's deps are installed
+echo "\nInstalling Elixir deps..."
 mix deps.get

-# Install JS deps
-echo "\nInstalling JS..."
-cd assets && yarn install
+# Install both project-level and assets-level JS dependencies
+echo "\nInstalling JS deps..."
+yarn install && cd assets && yarn install
 cd ..

 # Potentially Set up the database
@@ -1,8 +1,8 @@
 # Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian
 # instead of Alpine to avoid DNS resolution issues in production.
-ARG ELIXIR_VERSION=1.16.2
-ARG OTP_VERSION=26.2.2
-ARG DEBIAN_VERSION=bookworm-20240130-slim
+ARG ELIXIR_VERSION=1.17.0
+ARG OTP_VERSION=26.2.5
+ARG DEBIAN_VERSION=bookworm-20240612-slim

 ARG BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
 ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"
@@ -27,10 +27,10 @@ RUN apt-get update -y && \
    # Hex and Rebar
    mix local.hex --force && \
    mix local.rebar --force && \
-    # FFmpeg
+    # FFmpeg (latest build that doesn't cause an illegal instruction error for some users - see #347)
    export FFMPEG_DOWNLOAD=$(case ${TARGETPLATFORM:-linux/amd64} in \
-    "linux/amd64")   echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz"   ;; \
-    "linux/arm64")   echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linuxarm64-gpl.tar.xz" ;; \
+    "linux/amd64")   echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/autobuild-2024-07-30-14-10/ffmpeg-N-116468-g0e09f6d690-linux64-gpl.tar.xz"   ;; \
+    "linux/arm64")   echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/autobuild-2024-07-30-14-10/ffmpeg-N-116468-g0e09f6d690-linuxarm64-gpl.tar.xz" ;; \
    *)               echo ""        ;; esac) && \
    curl -L ${FFMPEG_DOWNLOAD} --output /tmp/ffmpeg.tar.xz && \
    tar -xf /tmp/ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/local/bin/ "ffmpeg" && \
@@ -1,4 +0,0 @@
- Write media datbase ID as metadata/to file/whatever so it gives us an option to retroactively match media to the DB down the line. Useful if someone moves the media without informing the UI
-  - Use a UUID for the media database ID (or at least alongside it)
- Look into this and its recommended plugins https://hexdocs.pm/ex_check/readme.html
- Add output template option for the source's friendly name
@@ -24,7 +24,7 @@ defmodule Pinchflat.Application do
      PinchflatWeb.Endpoint
    ]

-    :ok = Oban.Telemetry.attach_default_logger()
+    attach_oban_telemetry()
    Logger.add_handlers(:pinchflat)

    # See https://hexdocs.pm/elixir/Supervisor.html
@@ -40,4 +40,11 @@ defmodule Pinchflat.Application do
    PinchflatWeb.Endpoint.config_change(changed, removed)
    :ok
  end
+
+  defp attach_oban_telemetry do
+    events = [[:oban, :job, :start], [:oban, :job, :stop], [:oban, :job, :exception]]
+
+    :ok = Oban.Telemetry.attach_default_logger()
+    :telemetry.attach_many("job-telemetry-broadcast", events, &PinchflatWeb.Telemetry.job_state_change_broadcast/4, [])
+  end
 end
@@ -31,6 +31,7 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
  """
  @impl true
  def init(state) do
+    ensure_tmpfile_directory()
    reset_executing_jobs()
    create_blank_yt_dlp_files()
    create_blank_user_script_file()
@@ -39,6 +40,14 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
    {:ok, state}
  end

+  defp ensure_tmpfile_directory do
+    tmpfile_dir = Application.get_env(:pinchflat, :tmpfile_directory)
+
+    if !File.exists?(tmpfile_dir) do
+      File.mkdir_p!(tmpfile_dir)
+    end
+  end
+
  # If a node cannot gracefully shut down, the currently executing jobs get stuck
  # in the "executing" state. This is a problem because the job runner will not
  # pick them up again
@@ -4,6 +4,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
  """

  alias Pinchflat.Sources
+  alias Pinchflat.Settings
  alias Pinchflat.Sources.Source
  alias Pinchflat.Media.MediaItem
  alias Pinchflat.Downloading.OutputPathBuilder
@@ -73,6 +74,9 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
        {{:download_auto_subs, true}, %{download_subs: true}} ->
          acc ++ [:write_auto_subs]

+        {{:download_auto_subs, true}, %{embed_subs: true}} ->
+          acc ++ [:write_auto_subs]
+
        {{:embed_subs, true}, %{preferred_resolution: pr}} when pr != :audio ->
          acc ++ [:embed_subs]

@@ -121,27 +125,26 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
  end

  defp quality_options(media_profile) do
-    video_codec_option = fn res ->
-      [format_sort: "res:#{res},+codec:avc:m4a", remux_video: "mp4"]
-    end
-
-    audio_format_precedence = [
-      "bestaudio[ext=m4a]",
-      "bestaudio[ext=mp3]",
-      "bestaudio",
-      "best[ext=m4a]",
-      "best[ext=mp3]",
-      "best"
-    ]
+    vcodec = Settings.get!(:video_codec_preference)
+    acodec = Settings.get!(:audio_codec_preference)
+    container = media_profile.media_container

    case media_profile.preferred_resolution do
      # Also be aware that :audio disabled all embedding options for subtitles
-      :audio -> [:extract_audio, format: Enum.join(audio_format_precedence, "/")]
-      :"360p" -> video_codec_option.("360")
-      :"480p" -> video_codec_option.("480")
-      :"720p" -> video_codec_option.("720")
-      :"1080p" -> video_codec_option.("1080")
-      :"2160p" -> video_codec_option.("2160")
+      :audio ->
+        [:extract_audio, format_sort: "+acodec:#{acodec}", audio_format: container || "best"]
+
+      resolution_atom ->
+        {resolution_string, _} =
+          resolution_atom
+          |> Atom.to_string()
+          |> Integer.parse()
+
+        [
+          # Since Plex doesn't support reading metadata from MKV
+          remux_video: container || "mp4",
+          format_sort: "res:#{resolution_string},+codec:#{vcodec}:#{acodec}"
+        ]
    end
  end

@@ -204,10 +207,8 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
      "source_collection_id" => source.collection_id,
      "source_collection_name" => source.collection_name,
      "source_collection_type" => to_string(source.collection_type),
-      "media_upload_date_index" =>
-        media_item_with_preloads.upload_date_index
-        |> to_string()
-        |> String.pad_leading(2, "0")
+      "media_playlist_index" => pad_int(media_item_with_preloads.playlist_index),
+      "media_upload_date_index" => pad_int(media_item_with_preloads.upload_date_index)
    }
  end

@@ -224,6 +225,12 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
    |> build_output_path(media_item_with_preloads)
  end

+  defp pad_int(integer, count \\ 2, padding \\ "0") do
+    integer
+    |> to_string()
+    |> String.pad_leading(count, padding)
+  end
+
  defp base_directory do
    Application.get_env(:pinchflat, :media_directory)
  end
@@ -85,10 +85,15 @@ defmodule Pinchflat.Downloading.DownloadingHelpers do
  """
  def kickoff_redownload_for_existing_media(%Source{} = source) do
    MediaQuery.new()
-    |> MediaQuery.for_source(source)
-    |> MediaQuery.with_media_downloaded_at()
-    |> MediaQuery.where_download_not_prevented()
-    |> MediaQuery.where_not_culled()
+    |> MediaQuery.require_assoc(:media_profile)
+    |> where(
+      ^dynamic(
+        [m, s, mp],
+        ^MediaQuery.for_source(source) and
+          ^MediaQuery.downloaded() and
+          not (^MediaQuery.download_prevented())
+      )
+    )
    |> Repo.all()
    |> Enum.map(&MediaDownloadWorker.kickoff_with_task/1)
  end
@@ -4,7 +4,7 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
  use Oban.Worker,
    queue: :media_fetching,
    unique: [period: :infinity, states: [:available, :scheduled, :retryable, :executing]],
-    tags: ["media_item", "media_fetching"]
+    tags: ["media_item", "media_fetching", "show_in_dashboard"]

  require Logger

@@ -39,17 +39,14 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
    - `quality_upgrade?`: re-downloads media, including the video. Does not force download
      if the source is set to not download media

-  Returns :ok | {:ok, %MediaItem{}} | {:error, any, ...any}
+  Returns :ok | {:error, any, ...any}
  """
  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"id" => media_item_id} = args}) do
    should_force = Map.get(args, "force", false)
    is_quality_upgrade = Map.get(args, "quality_upgrade?", false)

-    media_item =
-      media_item_id
-      |> Media.get_media_item!()
-      |> Repo.preload(:source)
+    media_item = fetch_and_run_prevent_download_user_script(media_item_id)

    # If the source or media item is set to not download media, perform a no-op unless forced
    if (media_item.source.download_media && !media_item.prevent_download) || should_force do
@@ -62,6 +59,20 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
    Ecto.StaleEntryError -> Logger.info("#{__MODULE__} discarded: media item #{media_item_id} stale")
  end

+  # If a user script exists and, when run, returns a non-zero exit code, prevent this and all future downloads
+  # of the media item.
+  defp fetch_and_run_prevent_download_user_script(media_item_id) do
+    media_item = Media.get_media_item!(media_item_id)
+
+    {:ok, media_item} =
+      case run_user_script(:media_pre_download, media_item) do
+        {:ok, _, exit_code} when exit_code != 0 -> Media.update_media_item(media_item, %{prevent_download: true})
+        _ -> {:ok, media_item}
+      end
+
+    Repo.preload(media_item, :source)
+  end
+
  defp download_media_and_schedule_jobs(media_item, is_quality_upgrade, should_force) do
    overwrite_behaviour = if should_force || is_quality_upgrade, do: :force_overwrites, else: :no_force_overwrites
    override_opts = [overwrite_behaviour: overwrite_behaviour]
@@ -74,9 +85,9 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
            media_redownloaded_at: get_redownloaded_at(is_quality_upgrade)
          })

-        :ok = run_user_script(updated_media_item)
+        run_user_script(:media_downloaded, updated_media_item)

-        {:ok, updated_media_item}
+        :ok

      {:recovered, _} ->
        {:error, :retry}
@@ -112,9 +123,9 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do

  # NOTE: I like this pattern of using the default value so that I don't have to
  # define it in config.exs (and friends). Consider using this elsewhere.
-  defp run_user_script(media_item) do
+  defp run_user_script(event, media_item) do
    runner = Application.get_env(:pinchflat, :user_script_runner, UserScriptRunner)

-    runner.run(:media_downloaded, media_item)
+    runner.run(event, media_item)
  end
 end
@@ -79,12 +79,13 @@ defmodule Pinchflat.Downloading.MediaDownloader do
      |> MetadataParser.parse_for_media_item()
      |> Map.merge(%{
        media_downloaded_at: DateTime.utc_now(),
+        culled_at: nil,
        nfo_filepath: determine_nfo_filepath(media_with_preloads, parsed_json),
        metadata: %{
          # IDEA: might be worth kicking off a job for this since thumbnail fetching
          # could fail and I want to handle that in isolation
          metadata_filepath: MetadataFileHelpers.compress_and_store_metadata_for(media_with_preloads, parsed_json),
-          thumbnail_filepath: MetadataFileHelpers.download_and_store_thumbnail_for(media_with_preloads, parsed_json)
+          thumbnail_filepath: MetadataFileHelpers.download_and_store_thumbnail_for(media_with_preloads)
        }
      })

@@ -105,8 +106,9 @@ defmodule Pinchflat.Downloading.MediaDownloader do

  defp download_with_options(url, item_with_preloads, output_filepath, override_opts) do
    {:ok, options} = DownloadOptionBuilder.build(item_with_preloads, override_opts)
+    runner_opts = [output_filepath: output_filepath, use_cookies: item_with_preloads.source.use_cookies]

-    YtDlpMedia.download(url, options, output_filepath: output_filepath)
+    YtDlpMedia.download(url, options, runner_opts)
  end

  defp recoverable_errors do
@@ -4,7 +4,7 @@ defmodule Pinchflat.Downloading.MediaQualityUpgradeWorker do
  use Oban.Worker,
    queue: :media_fetching,
    unique: [period: :infinity, states: [:available, :scheduled, :retryable, :executing]],
-    tags: ["media_item", "media_fetching"]
+    tags: ["media_item", "media_fetching", "show_in_dashboard"]

  require Logger

@@ -23,10 +23,10 @@ defmodule Pinchflat.Downloading.MediaQualityUpgradeWorker do
  """
  @impl Oban.Worker
  def perform(%Oban.Job{}) do
-    redownloadable_media = Media.list_redownloadable_media_items()
-    Logger.info("Redownloading #{length(redownloadable_media)} media items")
+    upgradable_media = Media.list_upgradeable_media_items()
+    Logger.info("Redownloading #{length(upgradable_media)} media items")

-    Enum.each(redownloadable_media, fn media_item ->
+    Enum.each(upgradable_media, fn media_item ->
      MediaDownloadWorker.kickoff_with_task(media_item, %{quality_upgrade?: true})
    end)
  end
@@ -2,12 +2,15 @@ defmodule Pinchflat.Downloading.MediaRetentionWorker do
  @moduledoc false

  use Oban.Worker,
-    queue: :local_metadata,
+    queue: :local_data,
    unique: [period: :infinity, states: [:available, :scheduled, :retryable, :executing]],
-    tags: ["media_item", "local_metadata"]
+    tags: ["media_item", "local_data"]
+
+  use Pinchflat.Media.MediaQuery

  require Logger

+  alias Pinchflat.Repo
  alias Pinchflat.Media

  @doc """
@@ -20,14 +23,51 @@ defmodule Pinchflat.Downloading.MediaRetentionWorker do
  """
  @impl Oban.Worker
  def perform(%Oban.Job{}) do
-    cullable_media = Media.list_cullable_media_items()
+    cull_cullable_media_items()
+    delete_media_items_from_before_cutoff()
+
+    :ok
+  end
+
+  defp cull_cullable_media_items do
+    cullable_media =
+      MediaQuery.new()
+      |> MediaQuery.require_assoc(:source)
+      |> where(^MediaQuery.cullable())
+      |> Repo.all()
+
    Logger.info("Culling #{length(cullable_media)} media items past their retention date")

    Enum.each(cullable_media, fn media_item ->
+      # Setting `prevent_download` does what it says on the tin, but `culled_at` is purely informational.
+      # We don't actually do anything with that in terms of queries and it gets set to nil if the media item
+      # gets re-downloaded.
      Media.delete_media_files(media_item, %{
        prevent_download: true,
        culled_at: DateTime.utc_now()
      })
    end)
  end
+
+  defp delete_media_items_from_before_cutoff do
+    deletable_media =
+      MediaQuery.new()
+      |> MediaQuery.require_assoc(:source)
+      |> where(^MediaQuery.deletable_based_on_source_cutoff())
+      |> Repo.all()
+
+    Logger.info("Deleting #{length(deletable_media)} media items that are from before the source cutoff")
+
+    Enum.each(deletable_media, fn media_item ->
+      # Note that I'm not setting `prevent_download` on the media_item here.
+      # That's because cutoff_date can easily change and it's a valid behavior to re-download older
+      # media items if the cutoff_date changes.
+      # Download is ultimately prevented because `MediaQuery.pending()` only returns media items
+      # from after the cutoff date (among other things), so it's not like the media will just immediately
+      # be re-downloaded.
+      Media.delete_media_files(media_item, %{
+        culled_at: DateTime.utc_now()
+      })
+    end)
+  end
 end
@@ -54,7 +54,12 @@ defmodule Pinchflat.Downloading.OutputPathBuilder do
      "season_from_date" => "%(upload_date>%Y)S",
      "season_episode_from_date" => "s%(upload_date>%Y)Se%(upload_date>%m%d)S",
      "season_episode_index_from_date" => "s%(upload_date>%Y)Se%(upload_date>%m%d)S{{ media_upload_date_index }}",
-      "artist_name" => "%(artist,creator,uploader,uploader_id)S"
+      "artist_name" => "%(artist,creator,uploader,uploader_id)S",
+      "static_season__episode_by_index" => "Season 1/s01e{{ media_playlist_index }}",
+      "static_season__episode_by_date" => "Season 1/s01e%(upload_date>%y%m%d)S",
+      "season_by_year__episode_by_date" => "Season %(upload_date>%Y)S/s%(upload_date>%Y)Se%(upload_date>%m%d)S",
+      "season_by_year__episode_by_date_and_index" =>
+        "Season %(upload_date>%Y)S/s%(upload_date>%Y)Se%(upload_date>%m%d)S{{ media_upload_date_index }}"
    }
  end
 end
@@ -13,12 +13,13 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
  alias Pinchflat.Media
  alias Pinchflat.Sources.Source
  alias Pinchflat.FastIndexing.YoutubeRss
+  alias Pinchflat.FastIndexing.YoutubeApi
  alias Pinchflat.Downloading.DownloadingHelpers

  alias Pinchflat.YtDlp.Media, as: YtDlpMedia

  @doc """
-  Fetches new media IDs from a source's YouTube RSS feed, indexes them, and kicks off downloading
+  Fetches new media IDs for a source from YT's API or RSS, indexes them, and kicks off downloading
  tasks for any pending media items. See comments in `FastIndexingWorker` for more info on the
  order of operations and how this fits into the indexing process.

@@ -26,7 +27,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
  downloaded_.
  """
  def kickoff_download_tasks_from_youtube_rss_feed(%Source{} = source) do
-    {:ok, media_ids} = YoutubeRss.get_recent_media_ids_from_rss(source)
+    {:ok, media_ids} = get_recent_media_ids(source)
    existing_media_items = list_media_items_by_media_id_for(source, media_ids)
    new_media_ids = media_ids -- Enum.map(existing_media_items, & &1.media_id)

@@ -47,17 +48,27 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
    Enum.filter(maybe_new_media_items, & &1)
  end

+  # If possible, use the YouTube API to fetch media IDs. If that fails, fall back to the RSS feed.
+  # If the YouTube API isn't set up, just use the RSS feed.
+  defp get_recent_media_ids(source) do
+    with true <- YoutubeApi.enabled?(),
+         {:ok, media_ids} <- YoutubeApi.get_recent_media_ids(source) do
+      {:ok, media_ids}
+    else
+      _ -> YoutubeRss.get_recent_media_ids(source)
+    end
+  end
+
  defp list_media_items_by_media_id_for(source, media_ids) do
    MediaQuery.new()
-    |> MediaQuery.for_source(source)
-    |> MediaQuery.with_media_ids(media_ids)
+    |> where(^dynamic([mi], ^MediaQuery.for_source(source) and mi.media_id in ^media_ids))
    |> Repo.all()
  end

  defp create_media_item_from_media_id(source, media_id) do
    url = "https://www.youtube.com/watch?v=#{media_id}"

-    case YtDlpMedia.get_media_attributes(url) do
+    case YtDlpMedia.get_media_attributes(url, use_cookies: source.use_cookies) do
      {:ok, media_attrs} ->
        Media.create_media_item_from_backend_attrs(source, media_attrs)

@@ -4,7 +4,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingWorker do
  use Oban.Worker,
    queue: :fast_indexing,
    unique: [period: :infinity, states: [:available, :scheduled, :retryable]],
-    tags: ["media_source", "fast_indexing"]
+    tags: ["media_source", "fast_indexing", "show_in_dashboard"]

  require Logger

@@ -0,0 +1,92 @@
+defmodule Pinchflat.FastIndexing.YoutubeApi do
+  @moduledoc """
+  Methods for interacting with the YouTube API for fast indexing
+  """
+
+  require Logger
+
+  alias Pinchflat.Settings
+  alias Pinchflat.Sources.Source
+  alias Pinchflat.Utils.FunctionUtils
+  alias Pinchflat.FastIndexing.YoutubeBehaviour
+
+  @behaviour YoutubeBehaviour
+
+  @doc """
+  Determines if the YouTube API is enabled for fast indexing by checking
+  if the user has an API key set
+
+  Returns boolean()
+  """
+  @impl YoutubeBehaviour
+  def enabled?(), do: is_binary(api_key())
+
+  @doc """
+  Fetches the recent media IDs from the YouTube API for a given source.
+
+  Returns {:ok, [binary()]} | {:error, binary()}
+  """
+  @impl YoutubeBehaviour
+  def get_recent_media_ids(%Source{} = source) do
+    api_response =
+      source
+      |> determine_playlist_id()
+      |> do_api_request()
+
+    case api_response do
+      {:ok, parsed_json} -> get_media_ids_from_response(parsed_json)
+      {:error, reason} -> {:error, reason}
+    end
+  end
+
+  # The UC prefix is for channels which won't work with this API endpoint. Swapping
+  # the prefix to UU will get us the playlist that represents the channel's uploads
+  defp determine_playlist_id(%{collection_id: c_id}) do
+    String.replace_prefix(c_id, "UC", "UU")
+  end
+
+  defp do_api_request(playlist_id) do
+    Logger.debug("Fetching recent media IDs from YouTube API for playlist: #{playlist_id}")
+
+    playlist_id
+    |> construct_api_endpoint()
+    |> http_client().get(accept: "application/json")
+    |> case do
+      {:ok, response} ->
+        Phoenix.json_library().decode(response)
+
+      {:error, reason} ->
+        Logger.error("Failed to fetch YouTube API: #{inspect(reason)}")
+        {:error, reason}
+    end
+  end
+
+  defp get_media_ids_from_response(parsed_json) do
+    parsed_json
+    |> Map.get("items", [])
+    |> Enum.map(fn item ->
+      item
+      |> Map.get("contentDetails", %{})
+      |> Map.get("videoId", nil)
+    end)
+    |> Enum.reject(&is_nil/1)
+    |> Enum.uniq()
+    |> FunctionUtils.wrap_ok()
+  end
+
+  defp api_key do
+    Settings.get!(:youtube_api_key)
+  end
+
+  defp construct_api_endpoint(playlist_id) do
+    api_base = "https://youtube.googleapis.com/youtube/v3/playlistItems"
+    property_type = "contentDetails"
+    max_results = 50
+
+    "#{api_base}?part=#{property_type}&maxResults=#{max_results}&playlistId=#{playlist_id}&key=#{api_key()}"
+  end
+
+  defp http_client do
+    Application.get_env(:pinchflat, :http_client, Pinchflat.HTTP.HTTPClient)
+  end
+end
@@ -0,0 +1,11 @@
+defmodule Pinchflat.FastIndexing.YoutubeBehaviour do
+  @moduledoc """
+  This module defines the behaviour for clients that interface with YouTube
+  for the purpose of fast indexing.
+  """
+
+  alias Pinchflat.Sources.Source
+
+  @callback enabled?() :: boolean()
+  @callback get_recent_media_ids(%Source{}) :: {:ok, [String.t()]} | {:error, String.t()}
+end
@@ -1,18 +1,31 @@
 defmodule Pinchflat.FastIndexing.YoutubeRss do
  @moduledoc """
-  Methods for interacting with YouTube RSS feeds
+  Methods for interacting with YouTube RSS feeds for fast indexing
  """

  require Logger

  alias Pinchflat.Sources.Source
+  alias Pinchflat.FastIndexing.YoutubeBehaviour
+
+  @behaviour YoutubeBehaviour
+
+  @doc """
+  Determines if the YouTube RSS feed is enabled for fast indexing. Used to satisfy
+  the `YoutubeBehaviour` behaviour.
+
+  Returns true
+  """
+  @impl YoutubeBehaviour
+  def enabled?(), do: true

  @doc """
  Fetches the recent media IDs from a YouTube RSS feed for a given source.

  Returns {:ok, [binary()]} | {:error, binary()}
  """
-  def get_recent_media_ids_from_rss(%Source{} = source) do
+  @impl YoutubeBehaviour
+  def get_recent_media_ids(%Source{} = source) do
    Logger.debug("Fetching recent media IDs from YouTube RSS feed for source: #{source.collection_id}")

    case http_client().get(rss_url_for_source(source)) do
@@ -21,9 +21,11 @@ defmodule Pinchflat.HTTP.HTTPClient do
  """
  @impl HTTPBehaviour
  def get(url, headers \\ [], opts \\ []) do
+    headers = parse_headers(headers)
+
    case :httpc.request(:get, {url, headers}, [], opts) do
      {:ok, {{_version, 200, _reason_phrase}, _headers, body}} ->
-        {:ok, body}
+        {:ok, to_string(body)}

      {:ok, {{_version, status_code, reason_phrase}, _headers, _body}} ->
        {:error, "HTTP request failed with status code #{status_code}: #{reason_phrase}"}
@@ -32,4 +34,8 @@ defmodule Pinchflat.HTTP.HTTPClient do
        {:error, "HTTP request failed: #{reason}"}
    end
  end
+
+  defp parse_headers(headers) do
+    Enum.map(headers, fn {k, v} -> {to_charlist(k), to_charlist(v)} end)
+  end
 end
@@ -63,15 +63,14 @@ defmodule Pinchflat.Lifecycle.Notifications.SourceNotifications do

  defp pending_media_item_count(source) do
    MediaQuery.new()
-    |> MediaQuery.for_source(source)
-    |> MediaQuery.where_pending_download()
+    |> MediaQuery.require_assoc(:media_profile)
+    |> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.pending()))
    |> Repo.aggregate(:count)
  end

  defp downloaded_media_item_count(source) do
    MediaQuery.new()
-    |> MediaQuery.for_source(source)
-    |> MediaQuery.with_media_filepath()
+    |> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.downloaded()))
    |> Repo.aggregate(:count)
  end

@@ -12,6 +12,7 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
  @behaviour UserScriptCommandRunner

  @event_types [
+    :media_pre_download,
    :media_downloaded,
    :media_deleted
  ]
@@ -22,24 +23,25 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do

  This function will succeed in almost all cases, even if the user script command
  failed - this is because I don't want bad scripts to stop the whole process.
-  If something fails, it'll be logged.
+  If something fails, it'll be logged and returned BUT the tuple will always
+  start with {:ok, ...}.

  The only things that can cause a true failure are passing in an invalid event
  type or if the passed data cannot be encoded into JSON - both indicative of
  failures in the development process.

-  Returns :ok
+  Returns {:ok, :no_executable} | {:ok, output, exit_code}
  """
  @impl UserScriptCommandRunner
  def run(event_type, encodable_data) when event_type in @event_types do
    case backend_executable() do
      {:ok, :no_executable} ->
-        :ok
+        {:ok, :no_executable}

      {:ok, executable_path} ->
        {:ok, encoded_data} = Phoenix.json_library().encode(encodable_data)

-        {_output, _exit_code} =
+        {output, exit_code} =
          CliUtils.wrap_cmd(
            executable_path,
            [to_string(event_type), encoded_data],
@@ -47,7 +49,7 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
            logging_arg_override: "[suppressed]"
          )

-        :ok
+        {:ok, output, exit_code}
    end
  end

@@ -62,7 +64,7 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
    if FilesystemUtils.exists_and_nonempty?(filepath) do
      {:ok, filepath}
    else
-      Logger.warning("User scripts lifecyle file either not present or is empty. Skipping.")
+      Logger.info("User scripts lifecyle file either not present or is empty. Skipping.")

      {:ok, :no_executable}
    end
@@ -25,26 +25,15 @@ defmodule Pinchflat.Media do
  end

  @doc """
-  Returns a list of media_items that are cullable based on the retention period
-  of the source they belong to.
+  Returns a list of media_items that are upgradeable based on the redownload delay
+  of the media_profile their source belongs to. In this context, upgradeable means
+  that it's been long enough since upload that the video may be in a higher quality
+  or have better sponsorblock segments (or similar).

-  Returns [%MediaItem{}, ...]
-  """
-  def list_cullable_media_items do
-    MediaQuery.new()
-    |> MediaQuery.with_media_filepath()
-    |> MediaQuery.where_past_retention_period()
-    |> MediaQuery.where_culling_not_prevented()
-    |> Repo.all()
-  end
+  The logic is that a media_item is past_redownload_delay if the media_item's uploaded_at is
+  at least redownload_delay_days ago AND `media_downloaded_at` - `redownload_delay_days`
+  is before the media_item's `uploaded_at`.

-  @doc """
-  Returns a list of media_items that are redownloadable based on the redownload delay
-  of the media_profile their source belongs to.
-
-  The logic is that a media_item is past_redownload_delay if the media_item's
-  upload_date is at least redownload_delay_days ago AND
-  `media_downloaded_at` - `redownload_delay_days` is before the media_item's `upload_date`.
  This logic grabs media that we've recently downloaded AND is recently uploaded, but
  doesn't grab media that we've recently downloaded and was uploaded a long time ago.
  This also makes things work as expected when downloading media from a source for the
@@ -52,20 +41,16 @@ defmodule Pinchflat.Media do

  Returns [%MediaItem{}, ...]
  """
-  def list_redownloadable_media_items do
+  def list_upgradeable_media_items do
    MediaQuery.new()
-    |> MediaQuery.with_media_downloaded_at()
-    |> MediaQuery.where_download_not_prevented()
-    |> MediaQuery.where_not_culled()
-    |> MediaQuery.where_media_not_redownloaded()
-    |> MediaQuery.where_past_redownload_delay()
+    |> MediaQuery.require_assoc(:media_profile)
+    |> where(^MediaQuery.upgradeable())
    |> Repo.all()
  end

  @doc """
  Returns a list of pending media_items for a given source, where
-  pending means the `media_filepath` is `nil` AND the media_item
-  matches satisfies `MediaQuery.where_pending_download`. You
+  pending means the media_item satisfies `MediaQuery.pending`. You
  should really check out that function if you need to know more
  because it has a lot going on.

@@ -73,15 +58,14 @@ defmodule Pinchflat.Media do
  """
  def list_pending_media_items_for(%Source{} = source) do
    MediaQuery.new()
-    |> MediaQuery.for_source(source)
-    |> MediaQuery.where_pending_download()
+    |> MediaQuery.require_assoc(:media_profile)
+    |> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.pending()))
    |> Repo.all()
  end

  @doc """
  For a given media_item, tells you if it is pending download. This is defined as
-  the media_item having a `media_filepath` of `nil` and matching the format selection
-  rules of the parent media_profile.
+  the media_item satisfying `MediaQuery.pending` which you should really check out.

  Intentionally does not take the `download_media` setting of the source into account.

@@ -91,8 +75,8 @@ defmodule Pinchflat.Media do
    media_item = Repo.preload(media_item, source: :media_profile)

    MediaQuery.new()
-    |> MediaQuery.with_id(media_item.id)
-    |> MediaQuery.where_pending_download()
+    |> MediaQuery.require_assoc(:media_profile)
+    |> where(^dynamic([m, s, mp], m.id == ^media_item.id and ^MediaQuery.pending()))
    |> Repo.exists?()
  end

@@ -141,18 +125,23 @@ defmodule Pinchflat.Media do

  Unlike `create_media_item`, this will attempt an update if the media_item
  already exists. This is so that future indexing can pick up attributes that
-  we may not have asked for in the past (eg: upload_date)
+  we may not have asked for in the past (eg: uploaded_at)

  Returns {:ok, %MediaItem{}} | {:error, %Ecto.Changeset{}}
  """
  def create_media_item_from_backend_attrs(source, media_attrs_struct) do
    attrs = Map.merge(%{source_id: source.id}, Map.from_struct(media_attrs_struct))
+    # Some fields should only be set on insert and not on update.
+    fields_to_drop_on_update = [:playlist_index]

    %MediaItem{}
    |> MediaItem.changeset(attrs)
    |> Repo.insert(
      on_conflict: [
-        set: Map.to_list(attrs)
+        set:
+          attrs
+          |> Map.drop(fields_to_drop_on_update)
+          |> Map.to_list()
      ],
      conflict_target: [:source_id, :media_id]
    )
@@ -182,7 +171,7 @@ defmodule Pinchflat.Media do

    if delete_files do
      {:ok, _} = do_delete_media_files(media_item)
-      :ok = run_user_script(:media_deleted, media_item)
+      run_user_script(:media_deleted, media_item)
    end

    # Should delete these no matter what
@@ -205,7 +194,7 @@ defmodule Pinchflat.Media do

    Tasks.delete_tasks_for(media_item)
    {:ok, _} = do_delete_media_files(media_item)
-    :ok = run_user_script(:media_deleted, media_item)
+    run_user_script(:media_deleted, media_item)

    update_media_item(media_item, Map.merge(filepath_attrs, addl_attrs))
  end
@@ -18,6 +18,8 @@ defmodule Pinchflat.Media.MediaItem do
  alias Pinchflat.Media.MediaItemsSearchIndex

  @allowed_fields [
+    # these fields are only captured on index
+    :playlist_index,
    # these fields are captured on indexing (and again on download)
    :title,
    :media_id,
@@ -26,7 +28,7 @@ defmodule Pinchflat.Media.MediaItem do
    :livestream,
    :source_id,
    :short_form_content,
-    :upload_date,
+    :uploaded_at,
    :upload_date_index,
    :duration_seconds,
    # these fields are captured only on download
@@ -51,7 +53,7 @@ defmodule Pinchflat.Media.MediaItem do
    livestream
    media_id
    source_id
-    upload_date
+    uploaded_at
    short_form_content
  )a

@@ -69,9 +71,10 @@ defmodule Pinchflat.Media.MediaItem do
    field :short_form_content, :boolean, default: false
    field :media_downloaded_at, :utc_datetime
    field :media_redownloaded_at, :utc_datetime
-    field :upload_date, :date
+    field :uploaded_at, :utc_datetime
    field :upload_date_index, :integer, default: 0
    field :duration_seconds, :integer
+    field :playlist_index, :integer, default: 0

    field :media_filepath, :string
    field :media_size_bytes, :integer
@@ -130,7 +133,31 @@ defmodule Pinchflat.Media.MediaItem do
    ~w(__meta__ __struct__ metadata tasks media_items_search_index)a
  end

-  defp update_upload_date_index(%{changes: changes} = changeset) when is_map_key(changes, :upload_date) do
+  # Run it on new records no matter what. The method we delegate to
+  # will handle the case where `uploaded_at` is `nil`
+  defp update_upload_date_index(%{data: %{id: nil}} = changeset) do
+    do_update_upload_date_index(changeset)
+  end
+
+  # For the update case, we only want to recalculate if the day itself has changed.
+  # For instance, this is useful in the migration from `upload_date` to `uploaded_at`
+  defp update_upload_date_index(%{changes: changes} = changeset) when is_map_key(changes, :uploaded_at) do
+    old_uploaded_at = changeset.data.uploaded_at
+    new_uploaded_at = get_change(changeset, :uploaded_at)
+    upload_dates_match = DateTime.to_date(old_uploaded_at) == DateTime.to_date(new_uploaded_at)
+
+    if upload_dates_match do
+      changeset
+    else
+      do_update_upload_date_index(changeset)
+    end
+  end
+
+  # If the record is persisted and the `uploaded_at` field is not being changed,
+  # we don't need to recalculate the index.
+  defp update_upload_date_index(changeset), do: changeset
+
+  defp do_update_upload_date_index(%{changes: changes} = changeset) when is_map_key(changes, :uploaded_at) do
    source_id = get_field(changeset, :source_id)
    source = Sources.get_source!(source_id)
    # Channels should count down from 99, playlists should count up from 0
@@ -142,8 +169,7 @@ defmodule Pinchflat.Media.MediaItem do

    current_max =
      MediaQuery.new()
-      |> MediaQuery.for_source(source_id)
-      |> MediaQuery.where_uploaded_on_date(changes.upload_date)
+      |> where(^dynamic([mi], ^MediaQuery.upload_date_matches(changes.uploaded_at) and ^MediaQuery.for_source(source)))
      |> Repo.aggregate(aggregator, :upload_date_index)

    case current_max do
@@ -152,7 +178,7 @@ defmodule Pinchflat.Media.MediaItem do
    end
  end

-  defp update_upload_date_index(changeset), do: changeset
+  defp do_update_upload_date_index(changeset), do: changeset

  defimpl Jason.Encoder, for: MediaItem do
    def encode(value, opts) do
@@ -23,138 +23,30 @@ defmodule Pinchflat.Media.MediaQuery do
    end
  end

-  # Prefixes:
-  # - for_* - belonging to a certain record
-  # - join_* - for joining on a certain record
-  # - with_*, where_* - for filtering based on full, concrete attributes
-  # - matching_* - for filtering based on partial attributes (e.g. LIKE, regex, full-text search)
-  #
-  # Suffixes:
-  # - _for - the arg passed is an association record
-
-  # NOTE: that dyanmic query approach kinda rocked - should refactor in future
-
  def new do
    MediaItem
  end

-  def for_source(query, source_id) when is_integer(source_id) do
-    where(query, [mi], mi.source_id == ^source_id)
-  end
+  def for_source(source_id) when is_integer(source_id), do: dynamic([mi], mi.source_id == ^source_id)
+  def for_source(source), do: dynamic([mi], mi.source_id == ^source.id)

-  def for_source(query, source) do
-    where(query, [mi], mi.source_id == ^source.id)
-  end
+  def downloaded, do: dynamic([mi], not is_nil(mi.media_filepath))
+  def download_prevented, do: dynamic([mi], mi.prevent_download == true)
+  def culling_prevented, do: dynamic([mi], mi.prevent_culling == true)
+  def redownloaded, do: dynamic([mi], not is_nil(mi.media_redownloaded_at))
+  def upload_date_matches(other_date), do: dynamic([mi], fragment("date(?) = date(?)", mi.uploaded_at, ^other_date))

-  def join_sources(query) do
-    from(mi in query, join: s in assoc(mi, :source), as: :sources)
-  end
-
-  def where_past_retention_period(query) do
-    query
-    |> require_assoc(:source)
-    |> where(
+  def upload_date_after_source_cutoff do
+    dynamic(
      [mi, source],
-      fragment("""
-      IFNULL(retention_period_days, 0) > 0 AND
-      DATETIME('now', '-' || retention_period_days || ' day') > media_downloaded_at
-      """)
+      is_nil(source.download_cutoff_date) or
+        fragment("date(?) >= ?", mi.uploaded_at, source.download_cutoff_date)
    )
  end

-  def where_past_redownload_delay(query) do
-    query
-    |> require_assoc(:source)
-    |> require_assoc(:media_profile)
-    |> where(
-      [_mi, _source, _media_profile],
-      # Returns media items where the upload_date is at least redownload_delay_days ago AND
-      # downloaded_at minus the redownload_delay_days is before the upload date
-      fragment("""
-        IFNULL(redownload_delay_days, 0) > 0 AND
-        DATETIME('now', '-' || redownload_delay_days || ' day') > upload_date AND
-        DATETIME(media_downloaded_at, '-' || redownload_delay_days || ' day') < upload_date
-      """)
-    )
-  end
-
-  def where_culling_not_prevented(query) do
-    where(query, [mi], mi.prevent_culling == false)
-  end
-
-  def where_not_culled(query) do
-    where(query, [mi], is_nil(mi.culled_at))
-  end
-
-  def where_media_not_redownloaded(query) do
-    where(query, [mi], is_nil(mi.media_redownloaded_at))
-  end
-
-  def with_id(query, id) do
-    where(query, [mi], mi.id == ^id)
-  end
-
-  def with_media_ids(query, media_ids) do
-    where(query, [mi], mi.media_id in ^media_ids)
-  end
-
-  def with_media_downloaded_at(query) do
-    where(query, [mi], not is_nil(mi.media_downloaded_at))
-  end
-
-  def with_media_filepath(query) do
-    where(query, [mi], not is_nil(mi.media_filepath))
-  end
-
-  def with_no_media_filepath(query) do
-    where(query, [mi], is_nil(mi.media_filepath))
-  end
-
-  def with_upload_date_after_source_cutoff(query) do
-    query
-    |> require_assoc(:source)
-    |> where([mi, source], is_nil(source.download_cutoff_date) or mi.upload_date >= source.download_cutoff_date)
-  end
-
-  def where_uploaded_on_date(query, date) do
-    where(query, [mi], mi.upload_date == ^date)
-  end
-
-  def where_download_not_prevented(query) do
-    where(query, [mi], mi.prevent_download == false)
-  end
-
-  def matching_source_title_regex(query) do
-    query
-    |> require_assoc(:source)
-    |> where(
-      [mi, source],
-      is_nil(source.title_filter_regex) or fragment("regexp_like(?, ?)", mi.title, source.title_filter_regex)
-    )
-  end
-
-  def matching_search_term(query, nil), do: query
-
-  def matching_search_term(query, term) do
-    from(mi in query,
-      join: mi_search_index in assoc(mi, :media_items_search_index),
-      where: fragment("media_items_search_index MATCH ?", ^term),
-      select_merge: %{
-        matching_search_term:
-          fragment("""
-            coalesce(snippet(media_items_search_index, 0, '[PF_HIGHLIGHT]', '[/PF_HIGHLIGHT]', '...', 20), '') ||
-            ' ' ||
-            coalesce(snippet(media_items_search_index, 1, '[PF_HIGHLIGHT]', '[/PF_HIGHLIGHT]', '...', 20), '')
-          """)
-      },
-      order_by: [desc: fragment("rank")]
-    )
-  end
-
-  def with_format_matching_profile_preference(query) do
-    query
-    |> require_assoc(:media_profile)
-    |> where(
+  def format_matching_profile_preference do
+    dynamic(
+      [mi, source, media_profile],
      fragment("""
        CASE
          WHEN shorts_behaviour = 'only' AND livestream_behaviour = 'only' THEN
@@ -176,22 +68,97 @@ defmodule Pinchflat.Media.MediaQuery do
    )
  end

-  def where_pending_download(query) do
-    query
-    |> where_download_not_prevented()
-    |> with_no_media_filepath()
-    |> with_upload_date_after_source_cutoff()
-    |> with_format_matching_profile_preference()
-    |> matching_source_title_regex()
+  def matches_source_title_regex do
+    dynamic(
+      [mi, source],
+      is_nil(source.title_filter_regex) or fragment("regexp_like(?, ?)", mi.title, source.title_filter_regex)
+    )
  end

-  def where_pending_or_downloaded(query) do
-    query
-    |> where_pending_download()
-    |> or_where([mi], not is_nil(mi.media_downloaded_at))
+  def meets_min_and_max_duration do
+    dynamic(
+      [mi, source],
+      (is_nil(source.min_duration_seconds) or fragment("duration_seconds >= ?", source.min_duration_seconds)) and
+        (is_nil(source.max_duration_seconds) or fragment("duration_seconds <= ?", source.max_duration_seconds))
+    )
  end

-  defp require_assoc(query, identifier) do
+  def past_retention_period do
+    dynamic(
+      [mi, source],
+      fragment("""
+        IFNULL(retention_period_days, 0) > 0 AND
+        DATETIME('now', '-' || retention_period_days || ' day') > media_downloaded_at
+      """)
+    )
+  end
+
+  def past_redownload_delay do
+    dynamic(
+      [mi, source, media_profile],
+      # Returns media items where the uploaded_at is at least redownload_delay_days ago AND
+      # downloaded_at minus the redownload_delay_days is before the upload date
+      fragment("""
+        IFNULL(redownload_delay_days, 0) > 0 AND
+        DATETIME('now', '-' || redownload_delay_days || ' day') > uploaded_at AND
+        DATETIME(media_downloaded_at, '-' || redownload_delay_days || ' day') < uploaded_at
+      """)
+    )
+  end
+
+  def cullable do
+    dynamic(
+      [mi, source],
+      ^downloaded() and
+        ^past_retention_period() and
+        not (^culling_prevented())
+    )
+  end
+
+  def deletable_based_on_source_cutoff do
+    dynamic(
+      [mi, source],
+      ^downloaded() and
+        not (^upload_date_after_source_cutoff()) and
+        not (^culling_prevented())
+    )
+  end
+
+  def pending do
+    dynamic(
+      [mi],
+      not (^downloaded()) and
+        not (^download_prevented()) and
+        ^upload_date_after_source_cutoff() and
+        ^format_matching_profile_preference() and
+        ^matches_source_title_regex() and
+        ^meets_min_and_max_duration()
+    )
+  end
+
+  def upgradeable do
+    dynamic(
+      [mi, source],
+      ^downloaded() and
+        not (^download_prevented()) and
+        not (^redownloaded()) and
+        ^past_redownload_delay()
+    )
+  end
+
+  def matches_search_term(nil), do: dynamic([mi], true)
+
+  def matches_search_term(term) do
+    escaped_term = clean_search_term(term)
+
+    # Matching on `term` instead of `escaped_term` because the latter can mangle empty strings
+    case String.trim(term) do
+      "" -> dynamic([mi], true)
+      _ -> dynamic([mi], fragment("media_items_search_index MATCH ?", ^escaped_term))
+    end
+  end
+
+  def require_assoc(query, identifier) do
    if has_named_binding?(query, identifier) do
      query
    else
@@ -199,6 +166,10 @@ defmodule Pinchflat.Media.MediaQuery do
    end
  end

+  defp do_require_assoc(query, :media_items_search_index) do
+    from(mi in query, join: s in assoc(mi, :media_items_search_index), as: :media_items_search_index)
+  end
+
  defp do_require_assoc(query, :source) do
    from(mi in query, join: s in assoc(mi, :source), as: :source)
  end
@@ -208,4 +179,48 @@ defmodule Pinchflat.Media.MediaQuery do
    |> require_assoc(:source)
    |> join(:inner, [mi, source], mp in assoc(source, :media_profile), as: :media_profile)
  end
+
+  # This needs to be a non-dynamic query because it alone should control things like
+  # ordering and `snippets` for full-text search
+  def matching_search_term(query, nil), do: query
+
+  def matching_search_term(query, term) do
+    escaped_term = clean_search_term(term)
+
+    from(mi in query,
+      join: mi_search_index in assoc(mi, :media_items_search_index),
+      where: fragment("media_items_search_index MATCH ?", ^escaped_term),
+      select_merge: %{
+        matching_search_term:
+          fragment("""
+            coalesce(snippet(media_items_search_index, 0, '[PF_HIGHLIGHT]', '[/PF_HIGHLIGHT]', '...', 20), '') ||
+            ' ' ||
+            coalesce(snippet(media_items_search_index, 1, '[PF_HIGHLIGHT]', '[/PF_HIGHLIGHT]', '...', 20), '')
+          """)
+      },
+      order_by: [desc: fragment("rank")]
+    )
+  end
+
+  # SQLite's FTS5 is very picky about what it will accept as a search term.
+  # To that end, we need to clean up the search term before passing it to the
+  # MATCH clause.
+  # This method:
+  #   - Trims leading and trailing whitespace
+  #   - Collapses multiple spaces into a single space
+  #   - Removes quote characters
+  #   - Wraps any word in quotes (must happen after the double quote replacement)
+  #
+  # This allows for works with apostrophes and quotes to be searched for correctly
+  defp clean_search_term(nil), do: ""
+  defp clean_search_term(""), do: ""
+
+  defp clean_search_term(term) do
+    term
+    |> String.trim()
+    |> String.replace(~r/\s+/, " ")
+    |> String.split(~r/\s+/)
+    |> Enum.map(fn str -> String.replace(str, ~s("), "") end)
+    |> Enum.map_join(" ", fn str -> ~s("#{str}") end)
+  end
 end
@@ -11,6 +11,8 @@ defmodule Pinchflat.Metadata.MetadataFileHelpers do

  alias Pinchflat.Utils.FilesystemUtils

+  alias Pinchflat.YtDlp.Media, as: YtDlpMedia
+
  @doc """
  Returns the directory where metadata for a database record should be stored.

@@ -47,55 +49,44 @@ defmodule Pinchflat.Metadata.MetadataFileHelpers do
  Returns {:ok, map()} | {:error, any}
  """
  def read_compressed_metadata(filepath) do
-    {:ok, json} = File.open(filepath, [:read, :compressed], &IO.read(&1, :all))
+    {:ok, json} = File.open(filepath, [:read, :compressed], &IO.read(&1, :eof))

    Phoenix.json_library().decode(json)
  end

  @doc """
  Downloads and stores a thumbnail for a media item, returning the filepath.
-  Chooses the highest quality thumbnail available (preferring jpg). Returns
-  nil if no thumbnails are available.
+  Chooses the highest quality thumbnail available and converts it to a JPG
+
+  Returns nil if no thumbnail is available or if yt-dlp encounters an error

  Returns binary() | nil
  """
-  def download_and_store_thumbnail_for(database_record, metadata_map) do
-    thumbnails =
-      (metadata_map["thumbnails"] || [])
-      # Give it a low preference if the `preference` key doesn't exist
-      |> Enum.map(&Map.put_new(&1, "preference", -1000))
-      # Give it a low preference if image isn't a jpg
-      |> Enum.map(fn t ->
-        preference_weight = if String.ends_with?(t["url"], ".jpg"), do: t["preference"], else: t["preference"] - 1000
+  def download_and_store_thumbnail_for(media_item_with_preloads) do
+    yt_dlp_filepath = generate_filepath_for(media_item_with_preloads, "thumbnail.%(ext)s")
+    real_filepath = generate_filepath_for(media_item_with_preloads, "thumbnail.jpg")
+    command_opts = [output: yt_dlp_filepath]
+    addl_opts = [use_cookies: media_item_with_preloads.source.use_cookies]

-        Map.put(t, "preference", preference_weight)
-      end)
-
-    case Enum.sort_by(thumbnails, & &1["preference"], :desc) do
-      [thumbnail_map | _] ->
-        thumbnail_url = thumbnail_map["url"]
-        filepath = generate_filepath_for(database_record, Path.basename(thumbnail_url))
-        thumbnail_blob = fetch_thumbnail_from_url(thumbnail_url)
-
-        :ok = FilesystemUtils.write_p!(filepath, thumbnail_blob)
-
-        filepath
-
-      _ ->
-        nil
+    case YtDlpMedia.download_thumbnail(media_item_with_preloads.original_url, command_opts, addl_opts) do
+      {:ok, _} -> real_filepath
+      _ -> nil
    end
  end

  @doc """
  Parses an upload date from the YYYYMMDD string returned in yt-dlp metadata
-  and returns a Date struct.
+  and returns a DateTime struct, appending a time of 00:00:00Z.

-  Returns Date.t()
+  Returns DateTime.t()
  """
  def parse_upload_date(upload_date) do
    <<year::binary-size(4)>> <> <<month::binary-size(2)>> <> <<day::binary-size(2)>> = upload_date

-    Date.from_iso8601!("#{year}-#{month}-#{day}")
+    case DateTime.from_iso8601("#{year}-#{month}-#{day}T00:00:00Z") do
+      {:ok, datetime, _} -> datetime
+      _ -> raise "Invalid upload date: #{upload_date}"
+    end
  end

  @doc """
@@ -135,11 +126,19 @@ defmodule Pinchflat.Metadata.MetadataFileHelpers do
    end
  end

-  defp fetch_thumbnail_from_url(url) do
-    http_client = Application.get_env(:pinchflat, :http_client, Pinchflat.HTTP.HTTPClient)
-    {:ok, body} = http_client.get(url, [], body_format: :binary)
+  @doc """
+  Attempts to determine the season and episode number from a media filepath.

-    body
+  Returns {:ok, {binary(), binary()}} | {:error, :indeterminable}
+  """
+  def season_and_episode_from_media_filepath(media_filepath) do
+    # matches s + 1 or more digits + e + 1 or more digits (case-insensitive)
+    season_episode_regex = ~r/s(\d+)e(\d+)/i
+
+    case Regex.scan(season_episode_regex, media_filepath) do
+      [[_, season, episode] | _] -> {:ok, {season, episode}}
+      _ -> {:error, :indeterminable}
+    end
  end

  defp generate_filepath_for(database_record, filename) do
@@ -8,6 +8,8 @@ defmodule Pinchflat.Metadata.MetadataParser do
  and not have it, ya know?
  """

+  alias Pinchflat.YtDlp.Media, as: YtDlpMedia
+
  @doc """
  Parses the given JSON response from yt-dlp and returns a map of
  the needful media_item attributes, along with anything needed for
@@ -24,15 +26,12 @@ defmodule Pinchflat.Metadata.MetadataParser do
  end

  defp parse_media_metadata(metadata) do
+    Map.merge(
+      Map.from_struct(YtDlpMedia.response_to_struct(metadata)),
      %{
-      media_id: metadata["id"],
-      title: metadata["title"],
-      original_url: metadata["original_url"],
-      description: metadata["description"],
-      media_filepath: metadata["filepath"],
-      livestream: !!metadata["was_live"],
-      duration_seconds: metadata["duration"] && round(metadata["duration"])
+        media_filepath: metadata["filepath"]
      }
+    )
  end

  defp parse_subtitle_metadata(metadata) do
@@ -6,8 +6,8 @@ defmodule Pinchflat.Metadata.NfoBuilder do

  import Pinchflat.Utils.XmlUtils, only: [safe: 1]

-  alias Pinchflat.Metadata.MetadataFileHelpers
  alias Pinchflat.Utils.FilesystemUtils
+  alias Pinchflat.Metadata.MetadataFileHelpers

  @doc """
  Builds an NFO file for a media item (read: single "episode") and
@@ -15,12 +15,12 @@ defmodule Pinchflat.Metadata.NfoBuilder do

  Returns the filepath of the NFO file.
  """
-  def build_and_store_for_media_item(filepath, metadata) do
-    nfo = build_for_media_item(metadata)
+  def build_and_store_for_media_item(nfo_filepath, metadata) do
+    nfo = build_for_media_item(nfo_filepath, metadata)

-    FilesystemUtils.write_p!(filepath, nfo)
+    FilesystemUtils.write_p!(nfo_filepath, nfo)

-    filepath
+    nfo_filepath
  end

  @doc """
@@ -37,10 +37,15 @@ defmodule Pinchflat.Metadata.NfoBuilder do
    filepath
  end

-  defp build_for_media_item(metadata) do
+  defp build_for_media_item(nfo_filepath, metadata) do
    upload_date = MetadataFileHelpers.parse_upload_date(metadata["upload_date"])
+    # NOTE: the filepath here isn't the path of the media item, it's the path that
+    # the NFO should be saved to. This works because the NFO's path is the same as
+    # the media's path, just with a different extension. If this ever changes I'll
+    # need to pass in the media item's path as well.
+    {season, episode} = determine_season_and_episode_number(nfo_filepath, upload_date)
+
    # Cribbed from a combination of the Kodi wiki, ytdl-nfo, and ytdl-sub.
-    # WHO NEEDS A FANCY XML PARSER ANYWAY?!
    """
    <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
    <episodedetails>
@@ -49,8 +54,8 @@ defmodule Pinchflat.Metadata.NfoBuilder do
      <uniqueid type="youtube" default="true">#{safe(metadata["id"])}</uniqueid>
      <plot>#{safe(metadata["description"])}</plot>
      <aired>#{safe(upload_date)}</aired>
-      <season>#{safe(upload_date.year)}</season>
-      <episode>#{Calendar.strftime(upload_date, "%m%d")}</episode>
+      <season>#{safe(season)}</season>
+      <episode>#{episode}</episode>
      <genre>YouTube</genre>
    </episodedetails>
    """
@@ -67,4 +72,11 @@ defmodule Pinchflat.Metadata.NfoBuilder do
    </tvshow>
    """
  end
+
+  defp determine_season_and_episode_number(filepath, upload_date) do
+    case MetadataFileHelpers.season_and_episode_from_media_filepath(filepath) do
+      {:ok, {season, episode}} -> {season, episode}
+      {:error, _} -> {upload_date.year, Calendar.strftime(upload_date, "%m%d")}
+    end
+  end
 end
@@ -20,29 +20,49 @@ defmodule Pinchflat.Metadata.SourceImageParser do
  def store_source_images(base_directory, source_metadata) do
    (source_metadata["thumbnails"] || [])
    |> Enum.filter(&(&1["filepath"] != nil))
-    |> select_useful_images()
+    |> select_useful_images(source_metadata)
    |> Enum.map(&move_image(&1, base_directory))
    |> Enum.into(%{})
  end

-  defp select_useful_images(images) do
+  defp select_useful_images(images, source_metadata) do
    labelled_images =
-      Enum.reduce(images, [], fn image_map, acc ->
+      Enum.reduce(images, %{}, fn image_map, acc ->
        case image_map do
-          %{"id" => "avatar_uncropped"} ->
-            acc ++ [{:poster, :poster_filepath, image_map["filepath"]}]
-
-          %{"id" => "banner_uncropped"} ->
-            acc ++ [{:fanart, :fanart_filepath, image_map["filepath"]}]
-
-          _ ->
-            acc
+          %{"id" => "avatar_uncropped"} -> put_image_key(acc, :poster, image_map["filepath"])
+          %{"id" => "banner_uncropped"} -> put_image_key(acc, :fanart, image_map["filepath"])
+          _ -> acc
        end
      end)

    labelled_images
-    |> Enum.concat([{:banner, :banner_filepath, determine_best_banner(images)}])
-    |> Enum.filter(fn {_, _, tmp_filepath} -> tmp_filepath end)
+    |> add_fallback_poster(source_metadata)
+    |> put_image_key(:banner, determine_best_banner(images))
+    |> Enum.filter(fn {_key, attrs} -> attrs.current_filepath end)
+  end
+
+  # If a poster is set, short-circuit and return the images as-is
+  defp add_fallback_poster(%{poster: _} = images, _), do: images
+
+  # If a poster is NOT set, see if we can find a suitable image to use as a fallback
+  defp add_fallback_poster(images, source_metadata) do
+    case source_metadata["entries"] do
+      nil -> images
+      [] -> images
+      [first_entry | _] -> add_poster_from_entry_thumbnail(images, first_entry)
+    end
+  end
+
+  defp add_poster_from_entry_thumbnail(images, entry) do
+    thumbnail =
+      (entry["thumbnails"] || [])
+      |> Enum.reverse()
+      |> Enum.find(& &1["filepath"])
+
+    case thumbnail do
+      nil -> images
+      _ -> put_image_key(images, :poster, thumbnail["filepath"])
+    end
  end

  defp determine_best_banner(images) do
@@ -58,12 +78,22 @@ defmodule Pinchflat.Metadata.SourceImageParser do
    Map.get(best_candidate || %{}, "filepath")
  end

-  defp move_image({filename, source_attr_name, tmp_filepath}, base_directory) do
-    extension = Path.extname(tmp_filepath)
-    final_filepath = Path.join([base_directory, "#{filename}#{extension}"])
+  defp move_image({_key, attrs}, base_directory) do
+    extension = Path.extname(attrs.current_filepath)
+    final_filepath = Path.join([base_directory, "#{attrs.final_filename}#{extension}"])

-    FilesystemUtils.cp_p!(tmp_filepath, final_filepath)
+    FilesystemUtils.cp_p!(attrs.current_filepath, final_filepath)

-    {source_attr_name, final_filepath}
+    {attrs.attribute_name, final_filepath}
+  end
+
+  defp put_image_key(map, key, image) do
+    attribute_atom = String.to_existing_atom("#{key}_filepath")
+
+    Map.put(map, key, %{
+      attribute_name: attribute_atom,
+      final_filename: to_string(key),
+      current_filepath: image
+    })
  end
 end
@@ -3,7 +3,7 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorker do

  use Oban.Worker,
    queue: :remote_metadata,
-    tags: ["media_source", "source_metadata", "remote_metadata"],
+    tags: ["media_source", "source_metadata", "remote_metadata", "show_in_dashboard"],
    max_attempts: 3

  require Logger
@@ -77,10 +77,8 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorker do

  defp fetch_source_metadata_and_images(series_directory, source) do
    metadata_directory = MetadataFileHelpers.metadata_directory_for(source)
-    tmp_output_path = "#{tmp_directory()}/#{StringUtils.random_string(16)}/source_image.%(ext)S"
-    opts = [:write_all_thumbnails, convert_thumbnails: "jpg", output: tmp_output_path]

-    {:ok, metadata} = MediaCollection.get_source_metadata(source.original_url, opts)
+    {:ok, metadata} = fetch_metadata_for_source(source)
    metadata_image_attrs = SourceImageParser.store_source_images(metadata_directory, metadata)

    if source.media_profile.download_source_images && series_directory do
@@ -94,7 +92,9 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorker do

  defp determine_series_directory(source) do
    output_path = DownloadOptionBuilder.build_output_path_for(source)
-    {:ok, %{filepath: filepath}} = MediaCollection.get_source_details(source.original_url, output: output_path)
+    runner_opts = [output: output_path]
+    addl_opts = [use_cookies: source.use_cookies]
+    {:ok, %{filepath: filepath}} = MediaCollection.get_source_details(source.original_url, runner_opts, addl_opts)

    case MetadataFileHelpers.series_directory_from_media_filepath(filepath) do
      {:ok, series_directory} -> series_directory
@@ -110,6 +110,20 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorker do
    end
  end

+  defp fetch_metadata_for_source(source) do
+    tmp_output_path = "#{tmp_directory()}/#{StringUtils.random_string(16)}/source_image.%(ext)S"
+    base_opts = [convert_thumbnails: "jpg", output: tmp_output_path]
+
+    opts =
+      if source.collection_type == :channel do
+        base_opts ++ [:write_all_thumbnails, playlist_items: 0]
+      else
+        base_opts ++ [:write_thumbnail, playlist_items: 1]
+      end
+
+    MediaCollection.get_source_metadata(source.original_url, opts, use_cookies: source.use_cookies)
+  end
+
  defp tmp_directory do
    Application.get_env(:pinchflat, :tmpfile_directory)
  end
@@ -27,9 +27,8 @@ defmodule Pinchflat.Podcasts.PodcastHelpers do
    limit = Keyword.get(opts, :limit, 1_000)

    MediaQuery.new()
-    |> MediaQuery.for_source(source)
-    |> MediaQuery.with_media_filepath()
-    |> order_by(desc: :upload_date)
+    |> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.downloaded()))
+    |> order_by(desc: :uploaded_at)
    |> Repo.maybe_limit(limit)
    |> Repo.all()
    |> Enum.filter(fn media_item -> File.exists?(media_item.media_filepath) end)
@@ -7,7 +7,6 @@ defmodule Pinchflat.Podcasts.RssFeedBuilder do

  import Pinchflat.Utils.XmlUtils, only: [safe: 1]

-  alias Pinchflat.Utils.DatetimeUtils
  alias Pinchflat.Podcasts.PodcastHelpers
  alias PinchflatWeb.Router.Helpers, as: Routes

@@ -83,7 +82,7 @@ defmodule Pinchflat.Podcasts.RssFeedBuilder do
      <title>#{safe(media_item.title)}</title>
      <link>#{safe(media_item.original_url)}</link>
      <description>#{safe(media_item.description)}</description>
-      <pubDate>#{generate_upload_date(media_item)}</pubDate>
+      <pubDate>#{Calendar.strftime(media_item.uploaded_at, @datetime_format)}</pubDate>
      <itunes:duration>#{media_item.duration_seconds}</itunes:duration>
      <enclosure
        url="#{media_stream_path(url_base, media_item)}"
@@ -133,12 +132,6 @@ defmodule Pinchflat.Podcasts.RssFeedBuilder do
    end
  end

-  defp generate_upload_date(media_item) do
-    media_item.upload_date
-    |> DatetimeUtils.date_to_datetime()
-    |> Calendar.strftime(@datetime_format)
-  end
-
  defp podcast_route(action, params) do
    Routes.podcast_path(PinchflatWeb.Endpoint, action, params)
  end
@@ -27,7 +27,9 @@ defmodule Pinchflat.Profiles.MediaProfile do
    shorts_behaviour
    livestream_behaviour
    preferred_resolution
+    media_container
    redownload_delay_days
+    marked_for_deletion_at
  )a

  @required_fields ~w(name output_path_template)a
@@ -63,7 +65,10 @@ defmodule Pinchflat.Profiles.MediaProfile do
    # See `build_format_clauses` in the Media context for more.
    field :shorts_behaviour, Ecto.Enum, values: ~w(include exclude only)a, default: :include
    field :livestream_behaviour, Ecto.Enum, values: ~w(include exclude only)a, default: :include
-    field :preferred_resolution, Ecto.Enum, values: ~w(2160p 1080p 720p 480p 360p audio)a, default: :"1080p"
+    field :preferred_resolution, Ecto.Enum, values: ~w(4320p 2160p 1080p 720p 480p 360p audio)a, default: :"1080p"
+    field :media_container, :string, default: nil
+
+    field :marked_for_deletion_at, :utc_datetime

    has_many :sources, Source

@@ -0,0 +1,38 @@
+defmodule Pinchflat.Profiles.MediaProfileDeletionWorker do
+  @moduledoc false
+
+  use Oban.Worker,
+    queue: :local_data,
+    tags: ["media_profiles", "local_data"]
+
+  require Logger
+
+  alias __MODULE__
+  alias Pinchflat.Profiles
+
+  @doc """
+  Starts the profile deletion worker. Does not attach it to a task like `kickoff_with_task/2`
+  since deletion also cancels all tasks for the profile
+
+  Returns {:ok, %Task{}} | {:error, %Ecto.Changeset{}}
+  """
+  def kickoff(profile, job_args \\ %{}, job_opts \\ []) do
+    %{id: profile.id}
+    |> Map.merge(job_args)
+    |> MediaProfileDeletionWorker.new(job_opts)
+    |> Oban.insert()
+  end
+
+  @doc """
+  Deletes a profile and optionally deletes its files
+
+  Returns :ok
+  """
+  @impl Oban.Worker
+  def perform(%Oban.Job{args: %{"id" => profile_id} = args}) do
+    delete_files = Map.get(args, "delete_files", false)
+    profile = Profiles.get_media_profile!(profile_id)
+
+    Profiles.delete_media_profile(profile, delete_files: delete_files)
+  end
+end
@@ -11,12 +11,17 @@ defmodule Pinchflat.Settings.Setting do
    :pro_enabled,
    :yt_dlp_version,
    :apprise_version,
-    :apprise_server
+    :apprise_server,
+    :video_codec_preference,
+    :audio_codec_preference,
+    :youtube_api_key
  ]

  @required_fields ~w(
    onboarding
    pro_enabled
+    video_codec_preference
+    audio_codec_preference
  )a

  schema "settings" do
@@ -25,6 +30,10 @@ defmodule Pinchflat.Settings.Setting do
    field :yt_dlp_version, :string
    field :apprise_version, :string
    field :apprise_server, :string
+    field :youtube_api_key, :string
+
+    field :video_codec_preference, :string
+    field :audio_codec_preference, :string
  end

  @doc false
@@ -4,7 +4,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorker do
  use Oban.Worker,
    queue: :media_collection_indexing,
    unique: [period: :infinity, states: [:available, :scheduled, :retryable]],
-    tags: ["media_source", "media_collection_indexing"]
+    tags: ["media_source", "media_collection_indexing", "show_in_dashboard"]

  require Logger

@@ -28,7 +28,7 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
  """
  def kickoff_indexing_task(%Source{} = source, job_args \\ %{}, job_opts \\ []) do
    Tasks.delete_pending_tasks_for(source, "FastIndexingWorker")
-    Tasks.delete_pending_tasks_for(source, "MediaCollectionIndexingWorker")
+    Tasks.delete_pending_tasks_for(source, "MediaCollectionIndexingWorker", include_executing: true)

    MediaCollectionIndexingWorker.kickoff_with_task(source, job_args, job_opts)
  end
@@ -94,7 +94,8 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
    {:ok, pid} = FileFollowerServer.start_link()

    handler = fn filepath -> setup_file_follower_watcher(pid, filepath, source) end
-    result = MediaCollection.get_media_attributes_for_collection(source.original_url, file_listener_handler: handler)
+    runner_opts = [file_listener_handler: handler, use_cookies: source.use_cookies]
+    result = MediaCollection.get_media_attributes_for_collection(source.original_url, runner_opts)

    FileFollowerServer.stop(pid)

@@ -27,6 +27,7 @@ defmodule Pinchflat.Sources.Source do
    series_directory
    index_frequency_minutes
    fast_index
+    use_cookies
    download_media
    last_indexed_at
    original_url
@@ -35,6 +36,9 @@ defmodule Pinchflat.Sources.Source do
    title_filter_regex
    media_profile_id
    output_path_template_override
+    marked_for_deletion_at
+    min_duration_seconds
+    max_duration_seconds
  )a

  # Expensive API calls are made when a source is inserted/updated so
@@ -72,6 +76,7 @@ defmodule Pinchflat.Sources.Source do
    field :collection_type, Ecto.Enum, values: [:channel, :playlist]
    field :index_frequency_minutes, :integer, default: 60 * 24
    field :fast_index, :boolean, default: false
+    field :use_cookies, :boolean, default: false
    field :download_media, :boolean, default: true
    field :last_indexed_at, :utc_datetime
    # Only download media items that were published after this date
@@ -81,12 +86,17 @@ defmodule Pinchflat.Sources.Source do
    field :title_filter_regex, :string
    field :output_path_template_override, :string

+    field :min_duration_seconds, :integer
+    field :max_duration_seconds, :integer
+
    field :series_directory, :string
    field :nfo_filepath, :string
    field :poster_filepath, :string
    field :fanart_filepath, :string
    field :banner_filepath, :string

+    field :marked_for_deletion_at, :utc_datetime
+
    belongs_to :media_profile, MediaProfile

    has_one :metadata, SourceMetadata, on_replace: :update
@@ -112,9 +122,12 @@ defmodule Pinchflat.Sources.Source do
    |> dynamic_default(:custom_name, fn cs -> get_field(cs, :collection_name) end)
    |> dynamic_default(:uuid, fn _ -> Ecto.UUID.generate() end)
    |> validate_required(required_fields)
+    |> validate_title_regex()
+    |> validate_min_and_max_durations()
    |> validate_number(:retention_period_days, greater_than_or_equal_to: 0)
    # Ensures it ends with `.{{ ext }}` or `.%(ext)s` or similar (with a little wiggle room)
    |> validate_format(:output_path_template_override, MediaProfile.ext_regex(), message: "must end with .{{ ext }}")
+    |> validate_format(:original_url, youtube_channel_or_playlist_regex(), message: "must be a channel or playlist URL")
    |> cast_assoc(:metadata, with: &SourceMetadata.changeset/2, required: false)
    |> unique_constraint([:collection_id, :media_profile_id, :title_filter_regex], error_key: :original_url)
  end
@@ -128,7 +141,7 @@ defmodule Pinchflat.Sources.Source do
  @doc false
  def fast_index_frequency do
    # minutes
-    15
+    10
  end

  @doc false
@@ -141,6 +154,33 @@ defmodule Pinchflat.Sources.Source do
    ~w(__meta__ __struct__ metadata tasks media_items)a
  end

+  def youtube_channel_or_playlist_regex do
+    # Validate that the original URL is not a video URL
+    # Also matches if the string does NOT contain youtube.com or youtu.be. This preserves my tenuous support
+    # for non-youtube sources.
+    ~r<^(?:(?!youtube\.com/(watch|shorts|embed)|youtu\.be).)*$>
+  end
+
+  defp validate_title_regex(%{changes: %{title_filter_regex: regex}} = changeset) when is_binary(regex) do
+    case Ecto.Adapters.SQL.query(Repo, "SELECT regexp_like('', ?)", [regex]) do
+      {:ok, _} -> changeset
+      _ -> add_error(changeset, :title_filter_regex, "is invalid")
+    end
+  end
+
+  defp validate_title_regex(changeset), do: changeset
+
+  defp validate_min_and_max_durations(changeset) do
+    min_duration = get_change(changeset, :min_duration_seconds)
+    max_duration = get_change(changeset, :max_duration_seconds)
+
+    case {min_duration, max_duration} do
+      {min, max} when is_nil(min) or is_nil(max) -> changeset
+      {min, max} when min >= max -> add_error(changeset, :max_duration_seconds, "must be greater than minumum duration")
+      _ -> changeset
+    end
+  end
+
  defimpl Jason.Encoder, for: Source do
    def encode(value, opts) do
      value
@@ -0,0 +1,38 @@
+defmodule Pinchflat.Sources.SourceDeletionWorker do
+  @moduledoc false
+
+  use Oban.Worker,
+    queue: :local_data,
+    tags: ["sources", "local_data"]
+
+  require Logger
+
+  alias __MODULE__
+  alias Pinchflat.Sources
+
+  @doc """
+  Starts the source deletion worker. Does not attach it to a task like `kickoff_with_task/2`
+  since deletion also cancels all tasks for the source
+
+  Returns {:ok, %Task{}} | {:error, %Ecto.Changeset{}}
+  """
+  def kickoff(source, job_args \\ %{}, job_opts \\ []) do
+    %{id: source.id}
+    |> Map.merge(job_args)
+    |> SourceDeletionWorker.new(job_opts)
+    |> Oban.insert()
+  end
+
+  @doc """
+  Deletes a source and optionally deletes its files
+
+  Returns :ok
+  """
+  @impl Oban.Worker
+  def perform(%Oban.Job{args: %{"id" => source_id} = args}) do
+    delete_files = Map.get(args, "delete_files", false)
+    source = Sources.get_source!(source_id)
+
+    Sources.delete_source(source, delete_files: delete_files)
+  end
+end
@@ -127,7 +127,7 @@ defmodule Pinchflat.Sources do
    Tasks.delete_tasks_for(source)

    MediaQuery.new()
-    |> MediaQuery.for_source(source)
+    |> where(^MediaQuery.for_source(source))
    |> Repo.all()
    |> Enum.each(fn media_item ->
      Media.delete_media_item(media_item, delete_files: delete_files)
@@ -180,7 +180,9 @@ defmodule Pinchflat.Sources do
  end

  defp add_source_details_to_changeset(source, changeset) do
-    case MediaCollection.get_source_details(changeset.changes.original_url) do
+    use_cookies = Ecto.Changeset.get_field(changeset, :use_cookies)
+
+    case MediaCollection.get_source_details(changeset.changes.original_url, [], use_cookies: use_cookies) do
      {:ok, source_details} ->
        add_source_details_by_collection_type(source, changeset, source_details)

@@ -12,20 +12,26 @@ defmodule Pinchflat.Sources.SourcesQuery do

  alias Pinchflat.Sources.Source

-  # Prefixes:
-  # - for_* - belonging to a certain record
-  # - join_* - for joining on a certain record
-  # - with_* - for filtering based on full, concrete attributes
-  # - matching_* - for filtering based on partial attributes (e.g. LIKE, regex, full-text search)
-  #
-  # Suffixes:
-  # - _for - the arg passed is an association record
+  # This allows the module to be aliased and query methods to be used
+  # all in one go
+  # usage: use Pinchflat.Sources.SourcesQuery
+  defmacro __using__(_opts) do
+    quote do
+      import Ecto.Query, warn: false
+
+      alias unquote(__MODULE__)
+    end
+  end

  def new do
    Source
  end

-  def for_media_profile(query, media_profile) do
-    where(query, [s], s.media_profile_id == ^media_profile.id)
+  def for_media_profile(media_profile_id) when is_integer(media_profile_id) do
+    dynamic([s], s.media_profile_id == ^media_profile_id)
+  end
+
+  def for_media_profile(media_profile) do
+    dynamic([s], s.media_profile_id == ^media_profile.id)
  end
 end
@@ -53,20 +53,6 @@ defmodule Pinchflat.Tasks do
    )
  end

-  @doc """
-  Returns the list of pending tasks for a given record type and ID. Optionally allows you to specify
-  which worker to include.
-
-  Returns [%Task{}, ...]
-  """
-  def list_pending_tasks_for(record, worker_name \\ nil) do
-    list_tasks_for(
-      record,
-      worker_name,
-      [:available, :scheduled, :retryable]
-    )
-  end
-
  @doc """
  Gets a single task.

@@ -127,13 +113,13 @@ defmodule Pinchflat.Tasks do

  @doc """
  Deletes all tasks attached to a given record, cancelling any attached jobs.
-  Optionally allows you to specify which worker to include.
+  Optionally allows you to specify which worker and job states to include.

  Returns :ok
  """
-  def delete_tasks_for(record, worker_name \\ nil) do
+  def delete_tasks_for(record, worker_name \\ nil, job_states \\ Oban.Job.states()) do
    record
-    |> list_tasks_for(worker_name)
+    |> list_tasks_for(worker_name, job_states)
    |> Enum.each(&delete_task/1)
  end

@@ -143,10 +129,12 @@ defmodule Pinchflat.Tasks do

  Returns :ok
  """
-  def delete_pending_tasks_for(record, worker_name \\ nil) do
-    record
-    |> list_pending_tasks_for(worker_name)
-    |> Enum.each(&delete_task/1)
+  def delete_pending_tasks_for(record, worker_name \\ nil, opts \\ []) do
+    include_executing = Keyword.get(opts, :include_executing, false)
+    base_job_states = [:available, :scheduled, :retryable]
+    job_states = if include_executing, do: base_job_states ++ [:executing], else: base_job_states
+
+    delete_tasks_for(record, worker_name, job_states)
  end

  @doc """
@@ -0,0 +1,45 @@
+defmodule Pinchflat.Tasks.TasksQuery do
+  @moduledoc """
+  Query helpers for the Tasks context.
+
+  These methods are made to be one-ish liners used
+  to compose queries. Each method should strive to do
+  _one_ thing. These don't need to be tested as
+  they are just building blocks for other functionality
+  which, itself, will be tested.
+  """
+  import Ecto.Query, warn: false
+
+  alias Pinchflat.Tasks.Task
+
+  # This allows the module to be aliased and query methods to be used
+  # all in one go
+  # usage: use Pinchflat.Tasks.TasksQuery
+  defmacro __using__(_opts) do
+    quote do
+      import Ecto.Query, warn: false
+
+      alias unquote(__MODULE__)
+    end
+  end
+
+  def new do
+    Task
+  end
+
+  def join_job(query) do
+    query
+    |> join(:left, [t], j in assoc(t, :job))
+    |> preload([t, j], job: j)
+  end
+
+  def in_state(states) when is_list(states) do
+    dynamic([t, j], j.state in ^states)
+  end
+
+  def in_state(state), do: in_state([state])
+
+  def has_tag(tag) do
+    dynamic([t, j], ^tag in j.tags)
+  end
+end
@@ -24,11 +24,12 @@ defmodule Pinchflat.Utils.CliUtils do
  def wrap_cmd(command, args, passthrough_opts \\ [], opts \\ []) do
    wrapper_command = Path.join(:code.priv_dir(:pinchflat), "cmd_wrapper.sh")
    actual_command = [command] ++ args
+    command_opts = set_command_opts() ++ passthrough_opts
    logging_arg_override = Keyword.get(opts, :logging_arg_override, Enum.join(args, " "))

    Logger.info("[command_wrapper]: #{command} called with: #{logging_arg_override}")

-    {output, status} = System.cmd(wrapper_command, actual_command, passthrough_opts)
+    {output, status} = System.cmd(wrapper_command, actual_command, command_opts)
    log_cmd_result(command, logging_arg_override, status, output)

    {output, status}
@@ -81,4 +82,13 @@ defmodule Pinchflat.Utils.CliUtils do

    Logger.log(log_level, log_message)
  end
+
+  defp set_command_opts do
+    # This resolves an issue where yt-dlp would attempt to write to a read-only directory
+    # if you scanned a new video with `--windows-filenames` enabled. Hopefully can be removed
+    # in the future.
+    [
+      cd: Application.get_env(:pinchflat, :tmpfile_directory)
+    ]
+  end
 end
@@ -1,17 +0,0 @@
-defmodule Pinchflat.Utils.DatetimeUtils do
-  @moduledoc """
-  Utility methods for working with dates and datetimes
-  """
-
-  @doc """
-  Converts a Date to a DateTime
-
-  Returns %DateTime{}
-  """
-  def date_to_datetime(date) do
-    date
-    |> Date.to_gregorian_days()
-    |> Kernel.*(86_400)
-    |> DateTime.from_gregorian_seconds()
-  end
-end
@@ -13,4 +13,27 @@ defmodule Pinchflat.Utils.NumberUtils do
    |> max(minimum)
    |> min(maximum)
  end
+
+  @doc """
+  Converts a number to a human readable byte size. Can take a precision
+  option to specify the number of decimal places to round to.
+
+  Returns {integer(), String.t()}
+  """
+  def human_byte_size(number, opts \\ [])
+  def human_byte_size(nil, opts), do: human_byte_size(0, opts)
+
+  def human_byte_size(number, opts) do
+    precision = Keyword.get(opts, :precision, 2)
+    suffixes = ["B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB"]
+    base = 1024
+
+    Enum.reduce_while(suffixes, {number / 1.0, "B"}, fn suffix, {value, _} ->
+      if value < base do
+        {:halt, {Float.round(value, precision), suffix}}
+      else
+        {:cont, {value / base, suffix}}
+      end
+    end)
+  end
 end
@@ -18,6 +18,8 @@ defmodule Pinchflat.YtDlp.CommandRunner do
    - :output_filepath - the path to save the output to. If not provided, a temporary
      file will be created and used. Useful for if you need a reference to the file
      for a file watcher.
+    - :use_cookies - if true, will add a cookie file to the command options. Will not
+      attach a cookie file if the user hasn't set one up.

  Returns {:ok, binary()} | {:error, output, status}.
  """
@@ -28,7 +30,7 @@ defmodule Pinchflat.YtDlp.CommandRunner do

    output_filepath = generate_output_filepath(addl_opts)
    print_to_file_opts = [{:print_to_file, output_template}, output_filepath]
-    user_configured_opts = cookie_file_options()
+    user_configured_opts = cookie_file_options(addl_opts)
    # These must stay in exactly this order, hence why I'm giving it its own variable.
    all_opts = command_opts ++ print_to_file_opts ++ user_configured_opts ++ global_options()
    formatted_command_opts = [url] ++ CliUtils.parse_options(all_opts)
@@ -78,7 +80,14 @@ defmodule Pinchflat.YtDlp.CommandRunner do
    ]
  end

-  defp cookie_file_options do
+  defp cookie_file_options(addl_opts) do
+    case Keyword.get(addl_opts, :use_cookies) do
+      true -> add_cookie_file()
+      _ -> []
+    end
+  end
+
+  defp add_cookie_file do
    base_dir = Application.get_env(:pinchflat, :extras_directory)
    filename_options_map = %{cookies: "cookies.txt"}

@@ -10,7 +10,7 @@ defmodule Pinchflat.YtDlp.Media do
    :original_url,
    :livestream,
    :short_form_content,
-    :upload_date,
+    :uploaded_at,
    :duration_seconds
  ]

@@ -21,8 +21,9 @@ defmodule Pinchflat.YtDlp.Media do
    :original_url,
    :livestream,
    :short_form_content,
-    :upload_date,
-    :duration_seconds
+    :uploaded_at,
+    :duration_seconds,
+    :playlist_index
  ]

  alias __MODULE__
@@ -36,9 +37,9 @@ defmodule Pinchflat.YtDlp.Media do
  Returns {:ok, map()} | {:error, any, ...}.
  """
  def download(url, command_opts \\ [], addl_opts \\ []) do
-    opts = [:no_simulate] ++ command_opts
+    all_command_opts = [:no_simulate] ++ command_opts

-    with {:ok, output} <- backend_runner().run(url, opts, "after_move:%()j", addl_opts),
+    with {:ok, output} <- backend_runner().run(url, all_command_opts, "after_move:%()j", addl_opts),
         {:ok, parsed_json} <- Phoenix.json_library().decode(output) do
      {:ok, parsed_json}
    else
@@ -46,17 +47,31 @@ defmodule Pinchflat.YtDlp.Media do
    end
  end

+  @doc """
+  Downloads a thumbnail for a single piece of media. Usually used for
+  downloading thumbnails for internal use
+
+  Returns {:ok, ""} | {:error, any, ...}.
+  """
+  def download_thumbnail(url, command_opts \\ [], addl_opts \\ []) do
+    all_command_opts = [:no_simulate, :skip_download, :write_thumbnail, convert_thumbnail: "jpg"] ++ command_opts
+
+    # NOTE: it doesn't seem like this command actually returns anything in `after_move` since
+    # we aren't downloading the main media file
+    backend_runner().run(url, all_command_opts, "after_move:%()j", addl_opts)
+  end
+
  @doc """
  Returns a map representing the media at the given URL.

-  Returns {:ok, [map()]} | {:error, any, ...}.
+  Returns {:ok, %Media{}} | {:error, any, ...}.
  """
-  def get_media_attributes(url) do
+  def get_media_attributes(url, addl_opts \\ []) do
    runner = Application.get_env(:pinchflat, :yt_dlp_runner)
    command_opts = [:simulate, :skip_download]
    output_template = indexing_output_template()

-    case runner.run(url, command_opts, output_template) do
+    case runner.run(url, command_opts, output_template, addl_opts) do
      {:ok, output} ->
        output
        |> Phoenix.json_library().decode!()
@@ -70,9 +85,11 @@ defmodule Pinchflat.YtDlp.Media do

  @doc """
  Returns the output template for yt-dlp's indexing command.
+
+  NOTE: playlist_index is really only useful for playlists that will never change their order.
  """
  def indexing_output_template do
-    "%(.{id,title,was_live,webpage_url,description,aspect_ratio,duration,upload_date})j"
+    "%(.{id,title,live_status,webpage_url,description,aspect_ratio,duration,upload_date,timestamp,playlist_index})j"
  end

  @doc """
@@ -87,10 +104,11 @@ defmodule Pinchflat.YtDlp.Media do
      title: response["title"],
      description: response["description"],
      original_url: response["webpage_url"],
-      livestream: !!response["was_live"],
+      livestream: !!response["live_status"] && response["live_status"] != "not_live",
      duration_seconds: response["duration"] && round(response["duration"]),
      short_form_content: response["webpage_url"] && short_form_content?(response),
-      upload_date: response["upload_date"] && MetadataFileHelpers.parse_upload_date(response["upload_date"])
+      uploaded_at: response["upload_date"] && parse_uploaded_at(response),
+      playlist_index: response["playlist_index"] || 0
    }
  end

@@ -106,10 +124,22 @@ defmodule Pinchflat.YtDlp.Media do
      #
      # These don't fail if duration or aspect_ratio are missing
      # due to Elixir's comparison semantics
-      response["duration"] <= 60 && response["aspect_ratio"] < 0.8
+      response["duration"] <= 60 && response["aspect_ratio"] <= 0.85
    end
  end

+  defp parse_uploaded_at(%{"timestamp" => ts} = response) when is_number(ts) do
+    case DateTime.from_unix(ts) do
+      {:ok, datetime} -> datetime
+      _ -> MetadataFileHelpers.parse_upload_date(response["upload_date"])
+    end
+  end
+
+  # This field is needed before inserting into the database, but absence
+  # of this field should fail at insert-time rather than here
+  defp parse_uploaded_at(%{"upload_date" => nil}), do: nil
+  defp parse_uploaded_at(response), do: MetadataFileHelpers.parse_upload_date(response["upload_date"])
+
  defp backend_runner do
    # This approach lets us mock the command for testing
    Application.get_env(:pinchflat, :yt_dlp_runner)
@@ -25,15 +25,17 @@ defmodule Pinchflat.YtDlp.MediaCollection do
    # the first video has not released yet (ie: is a premier). We don't care about
    # available formats since we're just getting the media details
    command_opts = [:simulate, :skip_download, :ignore_no_formats_error, :no_warnings]
+    use_cookies = Keyword.get(addl_opts, :use_cookies, false)
    output_template = YtDlpMedia.indexing_output_template()
    output_filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
    file_listener_handler = Keyword.get(addl_opts, :file_listener_handler, false)
+    runner_opts = [output_filepath: output_filepath, use_cookies: use_cookies]

    if file_listener_handler do
      file_listener_handler.(output_filepath)
    end

-    case runner.run(url, command_opts, output_template, output_filepath: output_filepath) do
+    case runner.run(url, command_opts, output_template, runner_opts) do
      {:ok, output} ->
        parsed_lines =
          output
@@ -64,7 +66,7 @@ defmodule Pinchflat.YtDlp.MediaCollection do

  Returns {:ok, map()} | {:error, any, ...}.
  """
-  def get_source_details(source_url, addl_opts \\ []) do
+  def get_source_details(source_url, command_opts \\ [], addl_opts \\ []) do
    # `ignore_no_formats_error` is necessary because yt-dlp will error out if
    # the first video has not released yet (ie: is a premier). We don't care about
    # available formats since we're just getting the source details
@@ -75,10 +77,10 @@ defmodule Pinchflat.YtDlp.MediaCollection do
      playlist_end: 1
    ]

-    command_opts = default_opts ++ addl_opts
+    all_command_opts = default_opts ++ command_opts
    output_template = "%(.{channel,channel_id,playlist_id,playlist_title,filename})j"

-    with {:ok, output} <- backend_runner().run(source_url, command_opts, output_template),
+    with {:ok, output} <- backend_runner().run(source_url, all_command_opts, output_template, addl_opts),
         {:ok, parsed_json} <- Phoenix.json_library().decode(output) do
      {:ok, format_source_details(parsed_json)}
    else
@@ -99,13 +101,24 @@ defmodule Pinchflat.YtDlp.MediaCollection do
  as a compressed blob for possible future use. That's why it's not getting formatted like
  `get_source_details/1`

+  ! IMPORTANT ! - you'll always want to set `playlist_items: int` in `addl_opts.
+  This is great if you want to also return details about the videos in the playlists,
+  but it should be set in all cases to not over-fetch data.
+  For channels you should usually set this to 0 since channels return all the
+  metadata we need without needing to fetch the videos. On the other hand, playlists
+  don't return very useful images so you can set this to 1 to get the first video's
+  images, for instance.
+
  Returns {:ok, map()} | {:error, any, ...}.
  """
-  def get_source_metadata(source_url, addl_opts \\ []) do
-    opts = [playlist_items: 0] ++ addl_opts
+  def get_source_metadata(source_url, command_opts, addl_opts \\ []) do
+    # This only validates that the `playlist_items` key is present. It's otherwise unused
+    _playlist_items = Keyword.fetch!(command_opts, :playlist_items)
+
+    all_command_opts = [:skip_download] ++ command_opts
    output_template = "playlist:%()j"

-    with {:ok, output} <- backend_runner().run(source_url, opts, output_template),
+    with {:ok, output} <- backend_runner().run(source_url, all_command_opts, output_template, addl_opts),
         {:ok, parsed_json} <- Phoenix.json_library().decode(output) do
      {:ok, parsed_json}
    else
@@ -19,6 +19,7 @@ defmodule PinchflatWeb.CoreComponents do
  import PinchflatWeb.Gettext

  alias Phoenix.LiveView.JS
+  alias PinchflatWeb.CustomComponents.TextComponents

  @doc """
  Renders a modal.
@@ -644,7 +645,11 @@ defmodule PinchflatWeb.CoreComponents do
      <li :for={{k, v} <- @iterable_attributes} class="mb-2 w-2/3">
        <strong><%= k %>:</strong>
        <code class="inline-block text-sm font-mono text-gray p-0.5 mx-0.5">
+          <%= if is_binary(v) && URI.parse(v).scheme && URI.parse(v).scheme =~ "http" do %>
+            <TextComponents.inline_link href={v}><%= v %></TextComponents.inline_link>
+          <% else %>
            <%= v %>
+          <% end %>
        </code>
      </li>
    </ul>
@@ -6,25 +6,32 @@ defmodule PinchflatWeb.CustomComponents.TabComponents do
  Takes a list of tabs and renders them in a tabbed layout.
  """
  slot :tab, required: true do
+    attr :id, :string, required: true
    attr :title, :string, required: true
  end

  slot :tab_append, required: false

  def tabbed_layout(assigns) do
+    assigns = Map.put(assigns, :first_tab_id, hd(assigns.tab).id)
+
    ~H"""
    <div
-      x-data="{ openTab: getTabIndex(0), activeClasses: 'text-meta-5 border-meta-5', inactiveClasses: 'border-transparent' }"
-      @hashchange.window="openTab = getTabIndex(openTab)"
+      x-data={"{
+        openTab: getTabFromHash('#{@first_tab_id}', '#{@first_tab_id}'),
+        activeClasses: 'text-meta-5 border-meta-5',
+        inactiveClasses: 'border-transparent'
+      }"}
+      @hashchange.window={"openTab = getTabFromHash(openTab, '#{@first_tab_id}')"}
      class="w-full"
    >
      <header class="flex flex-col md:flex-row md:justify-between border-b border-strokedark">
        <div class="flex flex-wrap gap-5 sm:gap-10">
          <a
-            :for={{tab, idx} <- Enum.with_index(@tab)}
+            :for={tab <- @tab}
            href="#"
-            @click.prevent={"openTab = setTabIndex(#{idx})"}
-            x-bind:class={"openTab === #{idx} ? activeClasses : inactiveClasses"}
+            @click.prevent={"openTab = setTabByName('#{tab.id}')"}
+            x-bind:class={"openTab === '#{tab.id}' ? activeClasses : inactiveClasses"}
            class="border-b-2 py-4 w-full sm:w-fit text-sm font-medium hover:text-meta-5 md:text-base"
          >
            <span class="text-xl"><%= tab.title %></span>
@@ -35,7 +42,7 @@ defmodule PinchflatWeb.CustomComponents.TabComponents do
        </div>
      </header>
      <div class="mt-4 min-h-60">
-        <div :for={{tab, idx} <- Enum.with_index(@tab)} x-show={"openTab === #{idx}"} class="font-medium leading-relaxed">
+        <div :for={tab <- @tab} x-show={"openTab === '#{tab.id}'"} class="font-medium leading-relaxed">
          <%= render_slot(tab) %>
        </div>
      </div>
@@ -2,7 +2,8 @@ defmodule PinchflatWeb.CustomComponents.TableComponents do
  @moduledoc false
  use Phoenix.Component

-  alias PinchflatWeb.CoreComponents
+  import PinchflatWeb.CoreComponents
+  import PinchflatWeb.CustomComponents.TextComponents

  @doc """
  Renders a table component with the given rows and columns.
@@ -36,7 +37,7 @@ defmodule PinchflatWeb.CustomComponents.TableComponents do
        </tr>
      </thead>
      <tbody>
-        <tr :for={row <- @rows} class="border-b border-boxdark hover:border-strokedark">
+        <tr :for={row <- @rows} class="hover:bg-strokedark">
          <td
            :for={col <- @col}
            class={[
@@ -76,11 +77,13 @@ defmodule PinchflatWeb.CustomComponents.TableComponents do
            phx-click={@page_number != 1 && "page_change"}
            phx-value-direction="dec"
          >
-            <CoreComponents.icon name="hero-chevron-left" />
+            <.icon name="hero-chevron-left" />
          </span>
        </li>
        <li>
-          <span class="mx-2">Page <%= @page_number %> of <%= @total_pages %></span>
+          <span class="mx-2">
+            Page <.localized_number number={@page_number} /> of <.localized_number number={@total_pages} />
+          </span>
        </li>
        <li>
          <span
@@ -92,7 +95,7 @@ defmodule PinchflatWeb.CustomComponents.TableComponents do
            phx-click={@page_number != @total_pages && "page_change"}
            phx-value-direction="inc"
          >
-            <CoreComponents.icon name="hero-chevron-right" />
+            <.icon name="hero-chevron-right" />
          </span>
        </li>
      </ul>
@@ -94,4 +94,28 @@ defmodule PinchflatWeb.CustomComponents.TextComponents do
    <time><%= Calendar.strftime(Timex.Timezone.convert(@datetime, @timezone), @format) %></time>
    """
  end
+
+  @doc """
+  Renders a localized number using the Intl.NumberFormat API, falling back to the raw number if needed
+  """
+  attr :number, :any, required: true
+
+  def localized_number(assigns) do
+    ~H"""
+    <span x-data x-text={"Intl.NumberFormat().format(#{@number})"}><%= @number %></span>
+    """
+  end
+
+  @doc """
+  Renders a word with a suffix if the count is not 1
+  """
+  attr :word, :string, required: true
+  attr :count, :integer, required: true
+  attr :suffix, :string, default: "s"
+
+  def pluralize(assigns) do
+    ~H"""
+    <%= @word %><%= if @count == 1, do: "", else: @suffix %>
+    """
+  end
 end
@@ -68,7 +68,7 @@ defmodule PinchflatWeb.Layouts do
        </span>
      </span>

-      <ul x-bind:class="selected ? 'block' :'hidden'">
+      <ul x-cloak x-show="selected">
        <li :for={menu <- @submenu} class="text-bodydark2">
          <.sidebar_link icon={menu[:icon]} text={menu[:text]} href={menu[:href]} target={menu[:target]} class="pl-10" />
        </li>
@@ -1,4 +1,4 @@
-<header class="sticky top-0 z-999 flex h-20 w-full bg-white drop-shadow-1 dark:bg-boxdark dark:drop-shadow-none">
+<header class="sticky top-0 z-999 flex min-h-20 w-full bg-white drop-shadow-1 dark:bg-boxdark dark:drop-shadow-none">
  <div class="flex flex-grow items-center justify-between lg:justify-end px-4 py-4 shadow-2 md:px-6 2xl:px-11">
    <div class="flex items-center gap-2 sm:gap-4 lg:hidden w-2/6">
      <section class="pr-1">
@@ -22,7 +22,7 @@
            type="text"
            name="q"
            value={@params["q"]}
-            placeholder="Type to search..."
+            placeholder="Search all media..."
            class="w-full bg-transparent pl-9 pr-4 border-0 focus:ring-0 focus:outline-none"
          />
        </form>
@@ -19,6 +19,7 @@
      onboarding: #{Settings.get!(:onboarding)}
    }"}
    class="dark text-bodydark bg-boxdark-2"
+    data-socket-path={Path.join(Application.get_env(:pinchflat, :base_route_path), "/live")}
  >
    <%= @inner_content %>

@@ -1,14 +1,7 @@
 defmodule PinchflatWeb.ErrorHTML do
  use PinchflatWeb, :html

-  # If you want to customize your error pages,
-  # uncomment the embed_templates/1 call below
-  # and add pages to the error directory:
-  #
-  #   * lib/pinchflat_web/controllers/error_html/404.html.heex
-  #   * lib/pinchflat_web/controllers/error_html/500.html.heex
-  #
-  # embed_templates "error_html/*"
+  embed_templates "error_html/*"

  # The default is to render a plain text page based on
  # the template name. For example, "404.html" becomes
@@ -0,0 +1,3 @@
+<section>
+  <h2 class="text-title-md2 font-bold text-white">404 (not found)</h2>
+</section>
@@ -0,0 +1,29 @@
+<section>
+  <h2 class="text-title-md2 font-bold text-white">Internal Server Error</h2>
+  <p class="text-body-md text-white mt-2">
+    This shouldn't happen! Please make a
+    <.inline_link href="https://github.com/kieraneglin/pinchflat/issues/new/choose">GitHub issue</.inline_link>
+    with the following information:
+  </p>
+
+  <ul class="list-disc ml-8 mb-8">
+    <li>What you were doing when you saw this page</li>
+    <li>
+      Your system details and logs from
+      <.inline_link href={~p"/app_info"}>app info</.inline_link>
+    </li>
+    <li>All the information in the textarea below (use select all + copy)</li>
+  </ul>
+  <textarea class="w-full min-h-96 font-mono inline-block rounded-lg" readonly>
+**Status**:
+`<%= if Map.has_key?(assigns, :status), do: @status, else: "" %>`
+
+**Reason**:
+`<%= if Map.has_key?(assigns, :reason), do: inspect(@reason), else: "" %>`
+
+**Stacktrace**:
+```
+  <%= if Map.has_key?(assigns, :stack), do: Exception.format_stacktrace(@stack), else: "" %>
+```
+  </textarea>
+</section>
@@ -85,6 +85,7 @@ defmodule PinchflatWeb.MediaItems.MediaItemController do
          |> put_resp_header("accept-ranges", "bytes")
          |> put_resp_header("content-range", "bytes #{start_pos}-#{end_pos}/#{file_size}")
          |> put_resp_header("content-length", to_string(length))
+          |> put_resp_header("content-disposition", "inline; filename=\"#{media_item.title}\"")
          |> send_file(206, media_item.media_filepath, start_pos, length)

        {:error, :invalid_range} ->
@@ -92,8 +93,10 @@ defmodule PinchflatWeb.MediaItems.MediaItemController do

          conn
          |> put_resp_content_type(mime_type)
-          |> put_resp_header("content-length", to_string(file_size))
          |> put_resp_header("accept-ranges", "bytes")
+          |> put_resp_header("content-range", "bytes 0-#{file_size - 1}/#{file_size}")
+          |> put_resp_header("content-length", to_string(file_size))
+          |> put_resp_header("content-disposition", "inline; filename=\"#{media_item.title}\"")
          |> send_file(200, media_item.media_filepath)
      end
    else
@@ -18,7 +18,7 @@ defmodule PinchflatWeb.MediaItems.MediaItemHTML do
  def media_type(media_item) do
    case Path.extname(media_item.media_filepath) do
      ext when ext in [".mp4", ".webm", ".mkv"] -> :video
-      ext when ext in [".mp3", ".m4a"] -> :audio
+      ext when ext in [".mp3", ".m4a", ".opus"] -> :audio
      _ -> :unknown
    end
  end
@@ -23,7 +23,7 @@
        <.actions_dropdown media_item={@media_item} />
      </:tab_append>

-      <:tab title="Media">
+      <:tab title="Media" id="media">
        <div class="flex flex-col gap-10 dark:text-white">
          <%= if media_file_exists?(@media_item) do %>
            <section class="grid grid-cols-1 xl:grid-cols-2 xl:gap-6 mt-6">
@@ -31,11 +31,19 @@
                <.media_preview media_item={@media_item} />
              </div>
              <aside class="mt-4 xl:mt-0">
-                <div>Uploaded: <%= @media_item.upload_date %></div>
-                <div :if={URI.parse(@media_item.original_url).scheme =~ "http"}>
+                <div>Uploaded: <%= DateTime.to_date(@media_item.uploaded_at) %></div>
+                <div>
+                  <span :if={URI.parse(@media_item.original_url).scheme =~ "http"}>
                    <.subtle_link href={@media_item.original_url} target="_blank">Open Original</.subtle_link>
+                    <span class="mx-2">or</span>
+                  </span>
+                  <span>
+                    <.subtle_link href={~p"/media/#{@media_item.uuid}/stream"} target="_blank">
+                      Open Local Stream
+                    </.subtle_link>
+                  </span>
                </div>
-                <div class="mt-4 text-bodydark">
+                <div :if={@media_item.description} class="mt-4 text-bodydark">
                  <.break_on_newline text={@media_item.description} />
                </div>
              </aside>
@@ -52,7 +60,7 @@
          </section>
        </div>
      </:tab>
-      <:tab title="Tasks">
+      <:tab title="Tasks" id="tasks">
        <%= if match?([_|_], @media_item.tasks) do %>
          <.table rows={@media_item.tasks} table_class="text-black dark:text-white">
            <:col :let={task} label="Worker">
@@ -1,16 +1,16 @@
 defmodule PinchflatWeb.MediaProfiles.MediaProfileController do
  use PinchflatWeb, :controller
-
-  import Ecto.Query, warn: false
+  use Pinchflat.Sources.SourcesQuery

  alias Pinchflat.Repo
  alias Pinchflat.Profiles
-  alias Pinchflat.Sources.SourcesQuery
  alias Pinchflat.Profiles.MediaProfile
+  alias Pinchflat.Profiles.MediaProfileDeletionWorker

  def index(conn, _params) do
    media_profiles =
      MediaProfile
+      |> where([mp], is_nil(mp.marked_for_deletion_at))
      |> order_by(asc: :name)
      |> Repo.all()

@@ -43,7 +43,7 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileController do

    sources =
      SourcesQuery.new()
-      |> SourcesQuery.for_media_profile(media_profile)
+      |> where(^SourcesQuery.for_media_profile(media_profile))
      |> order_by(asc: :custom_name)
      |> Repo.all()

@@ -72,19 +72,15 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileController do
  end

  def delete(conn, %{"id" => id} = params) do
-    delete_files = Map.get(params, "delete_files", false)
+    # This awkward comparison converts the string to a boolean
+    delete_files = Map.get(params, "delete_files", "") == "true"
    media_profile = Profiles.get_media_profile!(id)
-    {:ok, _media_profile} = Profiles.delete_media_profile(media_profile, delete_files: delete_files)

-    flash_message =
-      if delete_files do
-        "Media profile, its sources, and its files deleted successfully."
-      else
-        "Media profile and its sources deleted successfully. Files were not deleted."
-      end
+    {:ok, _} = Profiles.update_media_profile(media_profile, %{marked_for_deletion_at: DateTime.utc_now()})
+    MediaProfileDeletionWorker.kickoff(media_profile, %{delete_files: delete_files})

    conn
-    |> put_flash(:info, flash_message)
+    |> put_flash(:info, "Media Profile deletion started. This may take a while to complete.")
    |> redirect(to: ~p"/media_profiles")
  end

@@ -23,6 +23,7 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do

  def friendly_resolution_options do
    [
+      {"8k", "4320p"},
      {"4k", "2160p"},
      {"1080p", "1080p"},
      {"720p", "720p"},
@@ -52,12 +53,24 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
    ]
  end

-  def custom_output_template_options do
+  def media_center_custom_output_template_options do
+    %{
+      season_by_year__episode_by_date: "<code>Season YYYY/sYYYYeMMDD</code>",
+      season_by_year__episode_by_date_and_index:
+        "same as the above but it handles dates better. <strong>This is the recommended option</strong>",
+      static_season__episode_by_index:
+        "<code>Season 1/s01eXX</code> where <code>XX</code> is the video's position in the playlist. Only recommended for playlists (not channels) that don't change",
+      static_season__episode_by_date:
+        "<code>Season 1/s01eYYMMDD</code>. Recommended for playlists that might change or where order isn't important"
+    }
+  end
+
+  def other_custom_output_template_options do
    %{
      upload_day: nil,
      upload_month: nil,
      upload_year: nil,
-      upload_yyyy_mm_dd: "the upload date in the format YYYY-MM-DD",
+      upload_yyyy_mm_dd: "the upload date in the format <code>YYYY-MM-DD</code>",
      source_custom_name: "the name of the sources that use this profile",
      source_collection_id: "the YouTube ID of the sources that use this profile",
      source_collection_name:
@@ -65,9 +78,11 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
      source_collection_type: "the collection type of the sources using this profile. Either 'channel' or 'playlist'",
      artist_name: "the name of the artist with fallbacks to other uploader fields",
      season_from_date: "alias for upload_year",
-      season_episode_from_date: "the upload date formatted as sYYYYeMMDD",
+      season_episode_from_date: "the upload date formatted as <code>sYYYYeMMDD</code>",
      season_episode_index_from_date:
-        "the upload date formatted as sYYYYeMMDDII where II is an index to prevent date collisions"
+        "the upload date formatted as <code>sYYYYeMMDDII</code> where <code>II</code> is an index to prevent date collisions",
+      media_playlist_index:
+        "the place of the media item in the playlist. Do not use with channels. May not work if the playlist is updated"
    }
  end

@@ -76,7 +91,6 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
      id
      ext
      title
-      fulltitle
      uploader
      channel
      upload_date
@@ -98,7 +112,7 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
  end

  defp media_center_output_template do
-    "/shows/{{ source_custom_name }}/Season {{ season_from_date }}/{{ season_episode_index_from_date }} - {{ title }}.{{ ext }}"
+    "/shows/{{ source_custom_name }}/{{ season_by_year__episode_by_date_and_index }} - {{ title }}.{{ ext }}"
  end

  defp audio_output_template do
@@ -23,9 +23,8 @@
        <:col :let={media_profile} label="Preferred Resolution">
          <%= media_profile.preferred_resolution %>
        </:col>
-        <:col :let={media_profile} label="" class="flex place-content-evenly">
-          <.icon_link href={~p"/media_profiles/#{media_profile.id}"} icon="hero-eye" class="mx-1" />
-          <.icon_link href={~p"/media_profiles/#{media_profile.id}/edit"} icon="hero-pencil-square" class="mx-1" />
+        <:col :let={media_profile} label="" class="flex justify-end">
+          <.icon_link href={~p"/media_profiles/#{media_profile.id}/edit"} icon="hero-pencil-square" class="mr-4" />
        </:col>
      </.table>
    </div>
@@ -1,4 +1,10 @@
-<.simple_form :let={f} for={@changeset} action={@action}>
+<.simple_form
+  :let={f}
+  for={@changeset}
+  action={@action}
+  x-data="{ advancedMode: !!JSON.parse(localStorage.getItem('advancedMode')) }"
+  x-init="$watch('advancedMode', value => localStorage.setItem('advancedMode', JSON.stringify(value)))"
+>
  <.error :if={@changeset.action}>
    Oops, something went wrong! Please check the errors below.
  </.error>
@@ -30,10 +36,14 @@
      </.input>
    </section>

-    <h3 class="mt-8 text-2xl text-black dark:text-white">
+    <section class="flex justify-between items-center mt-8">
+      <h3 class="text-2xl text-white">
        General Options
      </h3>
-
+      <span class="cursor-pointer hover:underline" x-on:click="advancedMode = !advancedMode">
+        Editing Mode: <span x-text="advancedMode ? 'Advanced' : 'Standard'"></span>
+      </span>
+    </section>
    <section x-data="{ 
        presets: { 
          default: 'Default',
@@ -84,22 +94,22 @@
      />
    </section>

-    <section x-data="{ presets: { default: false, media_center: false, audio: false, archiving: false } }">
-      <.input
-        field={f[:download_auto_subs]}
-        type="toggle"
-        label="Download Autogenerated Subtitles"
-        help="Prefers normal subs but will download autogenerated if needed. Requires 'Download Subtitles' to be enabled"
-        x-init="$watch('selectedPreset', p => p && (enabled = presets[p]))"
-      />
-    </section>
-
    <section x-data="{ presets: { default: true, media_center: true, audio: false, archiving: true } }">
      <.input
        field={f[:embed_subs]}
        type="toggle"
        label="Embed Subtitles"
-        help="Downloads and embeds subtitles in the media file itself, if supported. Uneffected by 'Download Subtitles' (recommended)"
+        help="Downloads and embeds subtitles in the media file itself, if supported. Uneffected by 'Download Subtitles'"
+        x-init="$watch('selectedPreset', p => p && (enabled = presets[p]))"
+      />
+    </section>
+
+    <section x-data="{ presets: { default: false, media_center: false, audio: false, archiving: false } }">
+      <.input
+        field={f[:download_auto_subs]}
+        type="toggle"
+        label="Use Autogenerated Subtitles"
+        help="Prefers normal subs with 'Download Subtitles' or 'Embed Subtitles' but will use autogenerated subs if needed."
        x-init="$watch('selectedPreset', p => p && (enabled = presets[p]))"
      />
    </section>
@@ -183,7 +193,7 @@
        options={friendly_format_type_options()}
        type="select"
        label="Include Livestreams"
-        help="Excludes media that comes from a past livestream"
+        help="How to handle past livestreams"
        x-init="$watch('selectedPreset', p => p && ($el.value = presets[p]))"
      />
    </section>
@@ -203,6 +213,16 @@
      />
    </section>

+    <section x-show="advancedMode">
+      <.input
+        field={f[:media_container]}
+        type="text"
+        label="Media Container"
+        placeholder="mp4"
+        help="Don't change this if you're going to consume media via Plex. Leave blank for default"
+      />
+    </section>
+
    <section x-data="{ presets: { default: null, media_center: 1, audio: null, archiving: 1 } }">
      <.input
        field={f[:redownload_delay_days]}
@@ -1,7 +1,7 @@
 <%!-- The heex HTML formatter is really struggling with this file - I apologize in advance --%>
 <aside>
  <h2 class="text-xl font-bold mb-2">Output Template Syntax</h2>
-  <section class="ml-2 md:ml-4 mb-4 max-w-prose">
+  <section class="ml-2 md:ml-4 mb-4 max-w-prose overflow-auto">
    <p>When generating an output template, you have 3 options for syntax:</p>
    <ul class="list-disc list-inside ml-2 md:ml-5">
      <li>
@@ -46,7 +46,7 @@
  </section>

  <h2 class="text-xl font-bold mb-2">Template Options</h2>
-  <section class="ml-2 md:ml-4 mb-4">
+  <section class="ml-2 md:ml-4 mb-4 overflow-auto">
    <p class="max-w-prose">
      Any single-word <code class="text-sm">yt-dlp</code>
      option
@@ -56,16 +56,23 @@
      can be used with the curly braced liquid-style syntax.
      This is just a list of the most common options as well as some custom aliases
    </p>
-    <h3 class="text-lg font-bold mb-2">Custom Aliases</h3>
-    <ul class="list-disc list-inside ml-2 md:ml-5">
-      <li :for={{k, v} <- custom_output_template_options()}>
+    <h3 class="text-lg font-bold my-2">Media Center Custom Aliases</h3>
+    <ul class="list-disc list-inside mb-4 ml-2 md:ml-5 max-w-prose">
+      <li :for={{k, v} <- media_center_custom_output_template_options()} class="mt-1">
        <.inline_code>{{ <%= k %> }}</.inline_code>
-        <span :if={v}>- <%= v %></span>
+        <span :if={v}>- <%= html_escape({:safe, v}) %></span>
+      </li>
+    </ul>
+    <h3 class="text-lg font-bold mb-2">Other Custom Aliases</h3>
+    <ul class="list-disc list-inside mb-4 ml-2 md:ml-5 max-w-prose">
+      <li :for={{k, v} <- other_custom_output_template_options()} class="mt-1">
+        <.inline_code>{{ <%= k %> }}</.inline_code>
+        <span :if={v}>- <%= html_escape({:safe, v}) %></span>
      </li>
    </ul>
    <h3 class="text-lg font-bold mb-2">Common Options</h3>
    <ul class="list-disc list-inside ml-2 md:ml-5">
-      <li :for={opt <- common_output_template_options()}>
+      <li :for={opt <- common_output_template_options()} class="mt-1">
        <.inline_code>{{ <%= opt %> }}</.inline_code>
      </li>
    </ul>
@@ -23,13 +23,13 @@
        <.actions_dropdown media_profile={@media_profile} />
      </:tab_append>

-      <:tab title="Media Profile">
+      <:tab title="Media Profile" id="media-profile">
        <div class="flex flex-col gap-10 text-white">
          <h3 class="font-bold text-xl mt-6">Raw Attributes</h3>
          <.list_items_from_map map={Map.from_struct(@media_profile)} />
        </div>
      </:tab>
-      <:tab title="Sources">
+      <:tab title="Sources" id="sources">
        <.table rows={@sources} table_class="text-black dark:text-white">
          <:col :let={source} label="Name">
            <.subtle_link href={~p"/sources/#{source.id}"}>
@@ -40,9 +40,8 @@
          <:col :let={source} label="Should Download?">
            <.icon name={if source.download_media, do: "hero-check", else: "hero-x-mark"} />
          </:col>
-          <:col :let={source} label="" class="flex place-content-evenly">
-            <.icon_link href={~p"/sources/#{source.id}"} icon="hero-eye" class="mx-1" />
-            <.icon_link href={~p"/sources/#{source.id}/edit"} icon="hero-pencil-square" class="mx-1" />
+          <:col :let={source} label="" class="flex justify-end">
+            <.icon_link href={~p"/sources/#{source.id}/edit"} icon="hero-pencil-square" class="mr-4" />
          </:col>
        </.table>
      </:tab>
@@ -20,14 +20,14 @@ defmodule PinchflatWeb.Pages.PageController do
  end

  defp render_home_page(conn) do
+    downloaded_media_items = where(MediaQuery.new(), ^MediaQuery.downloaded())
+
    conn
    |> render(:home,
      media_profile_count: Repo.aggregate(MediaProfile, :count, :id),
      source_count: Repo.aggregate(Source, :count, :id),
-      media_item_count:
-        MediaQuery.new()
-        |> MediaQuery.with_media_downloaded_at()
-        |> Repo.aggregate(:count, :id)
+      media_item_size: Repo.aggregate(downloaded_media_items, :sum, :media_size_bytes),
+      media_item_count: Repo.aggregate(downloaded_media_items, :count, :id)
    )
  end

@@ -1,5 +1,23 @@
 defmodule PinchflatWeb.Pages.PageHTML do
  use PinchflatWeb, :html

+  alias Pinchflat.Utils.NumberUtils
+
  embed_templates "page_html/*"
+
+  attr :media_filesize, :integer, required: true
+
+  def readable_media_filesize(assigns) do
+    {num, suffix} = NumberUtils.human_byte_size(assigns.media_filesize, precision: 2)
+
+    assigns =
+      Map.merge(assigns, %{
+        num: num,
+        suffix: suffix
+      })
+
+    ~H"""
+    <.localized_number number={@num} /> <%= @suffix %>
+    """
+  end
 end
@@ -6,7 +6,7 @@ defmodule Pinchflat.Pages.HistoryTableLive do
  alias Pinchflat.Utils.NumberUtils
  alias PinchflatWeb.CustomComponents.TextComponents

-  @limit 10
+  @limit 5

  def render(%{records: []} = assigns) do
    ~H"""
@@ -22,7 +22,9 @@ defmodule Pinchflat.Pages.HistoryTableLive do
    <div>
      <span class="mb-4 flex items-center">
        <.icon_button icon_name="hero-arrow-path" class="h-10 w-10" phx-click="reload_page" tooltip="Refresh" />
-        <span class="ml-2">Showing <%= length(@records) %> of <%= @total_record_count %></span>
+        <span class="ml-2">
+          Showing <.localized_number number={length(@records)} /> of <.localized_number number={@total_record_count} />
+        </span>
      </span>
      <div class="max-w-full overflow-x-auto">
        <.table rows={@records} table_class="text-white">
@@ -32,7 +34,7 @@ defmodule Pinchflat.Pages.HistoryTableLive do
            </.subtle_link>
          </:col>
          <:col :let={media_item} label="Upload Date">
-            <%= media_item.upload_date %>
+            <%= DateTime.to_date(media_item.uploaded_at) %>
          </:col>
          <:col :let={media_item} label="Indexed At">
            <%= format_datetime(media_item.inserted_at) %>
@@ -97,7 +99,8 @@ defmodule Pinchflat.Pages.HistoryTableLive do

  defp generate_base_query do
    MediaQuery.new()
-    |> MediaQuery.where_pending_or_downloaded()
+    |> MediaQuery.require_assoc(:media_profile)
+    |> where(^dynamic(^MediaQuery.downloaded() or ^MediaQuery.pending()))
    |> order_by(desc: :id)
  end

@@ -1,33 +1,54 @@
-<div class="grid grid-cols-1 gap-4 md:grid-cols-3">
-  <div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
-    <a href={~p"/media_profiles"} class="mt-4 flex flex-col items-center justify-center">
-      <span class="text-md font-medium">Media Profile(s)</span>
+<div class="grid grid-cols-1 gap-4 md:grid-cols-4">
+  <div class="flex flex-col justify-center rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
+    <a href={~p"/media_profiles"} class="flex flex-col items-center py-2">
+      <span class="text-md font-medium">
+        Media <.pluralize count={@media_profile_count} word="Profile" />
+      </span>
      <h4 class="text-title-md font-bold text-white">
-        <%= @media_profile_count %>
+        <.localized_number number={@media_profile_count} />
      </h4>
    </a>
  </div>
-  <div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
-    <a href={~p"/sources"} class="mt-4 flex flex-col items-center justify-center">
-      <span class="text-md font-medium">Source(s)</span>
+  <div class="flex flex-col justify-center rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
+    <a href={~p"/sources"} class="flex flex-col items-center py-2">
+      <span class="text-md font-medium">
+        <.pluralize count={@source_count} word="Source" />
+      </span>
      <h4 class="text-title-md font-bold text-white">
-        <%= @source_count %>
+        <.localized_number number={@source_count} />
      </h4>
    </a>
  </div>
-  <div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
-    <span class="mt-4 flex flex-col items-center justify-center">
-      <span class="text-md font-medium">Downloaded Media</span>
+  <div class="flex flex-col justify-center rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
+    <span class="flex flex-col items-center py-2">
+      <span class="text-md font-medium">
+        <.pluralize count={@media_item_count} word="Download" />
+      </span>
      <h4 class="text-title-md font-bold text-white">
-        <%= @media_item_count %>
+        <.localized_number number={@media_item_count} />
+      </h4>
+    </span>
+  </div>
+  <div class="flex flex-col justify-center rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
+    <span class="flex flex-col items-center py-2">
+      <span class="text-md font-medium">Library Size</span>
+      <h4 class="text-title-md font-bold text-white">
+        <.readable_media_filesize media_filesize={@media_item_size} />
      </h4>
    </span>
  </div>
 </div>

 <div class="rounded-sm border shadow-default border-strokedark bg-boxdark mt-4 p-5">
-  <span class="text-2xl font-medium mb-4">History</span>
+  <span class="text-2xl font-medium mb-4">Media History</span>
  <section class="mt-6">
    <%= live_render(@conn, Pinchflat.Pages.HistoryTableLive) %>
  </section>
 </div>
+
+<div class="rounded-sm border shadow-default border-strokedark bg-boxdark mt-4 p-5">
+  <span class="text-2xl font-medium mb-4">Active Tasks</span>
+  <section class="mt-6 min-h-80">
+    <%= live_render(@conn, Pinchflat.Pages.JobTableLive) %>
+  </section>
+</div>
@@ -0,0 +1,97 @@
+defmodule Pinchflat.Pages.JobTableLive do
+  use PinchflatWeb, :live_view
+  use Pinchflat.Tasks.TasksQuery
+
+  alias Pinchflat.Repo
+  alias Pinchflat.Tasks.Task
+  alias PinchflatWeb.CustomComponents.TextComponents
+
+  def render(%{tasks: []} = assigns) do
+    ~H"""
+    <div class="mb-4 flex items-center">
+      <p>Nothing Here!</p>
+    </div>
+    """
+  end
+
+  def render(assigns) do
+    ~H"""
+    <div class="max-w-full overflow-x-auto">
+      <.table rows={@tasks} table_class="text-white">
+        <:col :let={task} label="Task">
+          <%= worker_to_task_name(task.job.worker) %>
+        </:col>
+        <:col :let={task} label="Subject">
+          <.subtle_link href={task_to_link(task)}>
+            <%= StringUtils.truncate(task_to_record_name(task), 35) %>
+          </.subtle_link>
+        </:col>
+        <:col :let={task} label="Attempt No.">
+          <%= task.job.attempt %>
+        </:col>
+        <:col :let={task} label="Started At">
+          <%= format_datetime(task.job.attempted_at) %>
+        </:col>
+      </.table>
+    </div>
+    """
+  end
+
+  def mount(_params, _session, socket) do
+    PinchflatWeb.Endpoint.subscribe("job:state")
+
+    {:ok, assign(socket, tasks: get_tasks())}
+  end
+
+  def handle_info(%{topic: "job:state", event: "change"}, socket) do
+    {:noreply, assign(socket, tasks: get_tasks())}
+  end
+
+  defp get_tasks do
+    TasksQuery.new()
+    |> TasksQuery.join_job()
+    |> where(^TasksQuery.in_state("executing"))
+    |> where(^TasksQuery.has_tag("show_in_dashboard"))
+    |> order_by([t, j], desc: j.attempted_at)
+    |> Repo.all()
+    |> Repo.preload([:media_item, :source])
+  end
+
+  defp worker_to_task_name(worker) do
+    final_module_part =
+      worker
+      |> String.split(".")
+      |> Enum.at(-1)
+
+    map_worker_to_task_name(final_module_part)
+  end
+
+  defp map_worker_to_task_name("FastIndexingWorker"), do: "Fast Indexing Source"
+  defp map_worker_to_task_name("MediaDownloadWorker"), do: "Downloading Media"
+  defp map_worker_to_task_name("MediaCollectionIndexingWorker"), do: "Indexing Source"
+  defp map_worker_to_task_name("MediaQualityUpgradeWorker"), do: "Upgrading Media Quality"
+  defp map_worker_to_task_name("SourceMetadataStorageWorker"), do: "Fetching Source Metadata"
+  defp map_worker_to_task_name(other), do: other <> " (Report to Devs)"
+
+  defp task_to_record_name(%Task{} = task) do
+    case task do
+      %Task{source: source} when source != nil -> source.custom_name
+      %Task{media_item: mi} when mi != nil -> mi.title
+      _ -> "Unknown Record"
+    end
+  end
+
+  defp task_to_link(%Task{} = task) do
+    case task do
+      %Task{source: source} when source != nil -> ~p"/sources/#{source.id}"
+      %Task{media_item: mi} when mi != nil -> ~p"/sources/#{mi.source_id}/media/#{mi}"
+      _ -> "#"
+    end
+  end
+
+  defp format_datetime(nil), do: ""
+
+  defp format_datetime(datetime) do
+    TextComponents.datetime_in_zone(%{datetime: datetime, format: "%Y-%m-%d %H:%M"})
+  end
+end
@@ -26,8 +26,7 @@ defmodule PinchflatWeb.Podcasts.PodcastController do
    # if the source doesn't have any usable images
    media_items =
      MediaQuery.new()
-      |> MediaQuery.for_source(source)
-      |> MediaQuery.with_media_filepath()
+      |> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.downloaded()))
      |> Repo.maybe_limit(1)
      |> Repo.all()

@@ -10,19 +10,13 @@
      <%= if match?([_|_], @search_results) do %>
        <.table rows={@search_results} table_class="text-black dark:text-white">
          <:col :let={result} label="Title">
-            <%= result.title %>
+            <.subtle_link href={~p"/sources/#{result.source_id}/media/#{result.id}"}>
+              <%= StringUtils.truncate(result.title, 35) %>
+            </.subtle_link>
          </:col>
          <:col :let={result} label="Excerpt">
            <.highlight_search_terms text={result.matching_search_term} />
          </:col>
-          <:col :let={result} label="" class="flex place-content-evenly">
-            <.link
-              href={~p"/sources/#{result.source_id}/media/#{result.id}"}
-              class="hover:text-secondary duration-200 ease-in-out mx-0.5"
-            >
-              <.icon name="hero-eye" />
-            </.link>
-          </:col>
        </.table>
      <% else %>
        <p class="font-bold text-lg text-center text-black dark:text-white">No results found</p>
@@ -14,18 +14,27 @@ defmodule PinchflatWeb.Settings.SettingHTML do

  def apprise_server_help do
    url = "https://github.com/caronc/apprise/wiki/URLBasics"
-    classes = "underline decoration-bodydark decoration-1 hover:decoration-white"

-    ~s(Server endpoint for Apprise notifications when new media is found. See <a href="#{url}" class="#{classes}" target="_blank">Apprise docs</a> for more information)
+    ~s(Server endpoint for Apprise notifications when new media is found. See <a href="#{url}" class="#{help_link_classes()}" target="_blank">Apprise docs</a> for more information)
+  end
+
+  def youtube_api_help do
+    url = "https://github.com/kieraneglin/pinchflat/wiki/Generating-a-YouTube-API-key"
+
+    ~s(API key for YouTube Data API v3. Greatly improves the accuracy of Fast Indexing. See <a href="#{url}" class="#{help_link_classes()}" target="_blank">here</a> for details on generating an API key)
  end

  def diagnostic_info_string do
    """
-    App Version: #{Application.spec(:pinchflat)[:vsn]}
-    yt-dlp Version: #{Settings.get!(:yt_dlp_version)}
-    Apprise Version: #{Settings.get!(:apprise_version)}
-    System Architecture: #{to_string(:erlang.system_info(:system_architecture))}
-    Timezone: #{Application.get_env(:pinchflat, :timezone)}
+    - App Version: #{Application.spec(:pinchflat)[:vsn]}
+    - yt-dlp Version: #{Settings.get!(:yt_dlp_version)}
+    - Apprise Version: #{Settings.get!(:apprise_version)}
+    - System Architecture: #{to_string(:erlang.system_info(:system_architecture))}
+    - Timezone: #{Application.get_env(:pinchflat, :timezone)}
    """
  end
+
+  defp help_link_classes do
+    "underline decoration-bodydark decoration-1 hover:decoration-white"
+  end
 end
@@ -14,7 +14,7 @@
          () => copied = false
        )
      "}>
-      Copy Diagnotic Info
+      Copy Diagnostic Info
      <span x-show="copied" x-transition.duration.150ms><.icon name="hero-check" class="ml-2 h-4 w-4" /></span>
    </.button>
    <.link href={~p"/download_logs"}>
@@ -0,0 +1,17 @@
+<aside>
+  <h2 class="text-xl font-bold mb-2">Codec Preference</h2>
+  <section class="ml-2 md:ml-4 mb-2 max-w-prose">
+    <p>
+      Some users may want to specify the audio codec that <code class="text-sm">yt-dlp</code>
+      searches for when downloading content. For these users, you can specify one audio and video
+      codec to prefer above all others.
+    </p>
+    <p class="mt-4">
+      NOTE: this is a <em>soft</em>
+      preference. If the codec you specify is not available, <code class="text-sm">yt-dlp</code>
+      will download the next best available codec. This is expected and intended behavior.
+      <.inline_link href="https://github.com/yt-dlp/yt-dlp#sorting-formats">See here</.inline_link>
+      for more information and a list of available codecs.
+    </p>
+  </section>
+</aside>
@@ -1,13 +1,24 @@
-<.simple_form :let={f} for={@changeset} action={@action}>
+<.simple_form
+  :let={f}
+  for={@changeset}
+  action={@action}
+  x-data="{ advancedMode: !!JSON.parse(localStorage.getItem('advancedMode')) }"
+  x-init="$watch('advancedMode', value => localStorage.setItem('advancedMode', JSON.stringify(value)))"
+>
  <.error :if={@changeset.action}>
    Oops, something went wrong! Please check the errors below.
  </.error>

-  <h3 class="mt-2 md:mt-8 text-2xl text-black dark:text-white">
+  <section>
+    <section class="flex justify-between items-center mt-4">
+      <h3 class="text-2xl text-black dark:text-white">
        Notification Settings
      </h3>
+      <span class="cursor-pointer hover:underline" x-on:click="advancedMode = !advancedMode">
+        Editing Mode: <span x-text="advancedMode ? 'Advanced' : 'Standard'"></span>
+      </span>
+    </section>

-  <section>
    <%= live_render(
      @conn,
      Pinchflat.Settings.AppriseServerLive,
@@ -15,5 +26,57 @@
    ) %>
  </section>

+  <section class="mt-8">
+    <section>
+      <h3 class="text-2xl text-black dark:text-white">
+        Indexing Settings
+      </h3>
+
+      <.input
+        field={f[:youtube_api_key]}
+        placeholder="ABC123"
+        type="text"
+        label="YouTube API Key"
+        help={youtube_api_help()}
+        html_help={true}
+        inputclass="font-mono text-sm mr-4"
+      />
+    </section>
+  </section>
+
+  <section class="mt-8" x-show="advancedMode">
+    <section>
+      <h3 class="text-2xl text-black dark:text-white">
+        Codec Options
+      </h3>
+
+      <p class="text-sm mt-2 max-w-prose">
+        The best available codec will be used if your preferred codecs are not found
+      </p>
+
+      <.input
+        field={f[:video_codec_preference]}
+        placeholder="avc"
+        type="text"
+        label="Video Codec Preference"
+        help="Video codec preference. Will be remuxed into an MP4 container. See below for more details"
+        inputclass="font-mono text-sm mr-4"
+      />
+
+      <.input
+        field={f[:audio_codec_preference]}
+        placeholder="m4a"
+        type="text"
+        label="Audio Codec Preference"
+        help="Audio codec preference. See below for more details"
+        inputclass="font-mono text-sm mr-4"
+      />
+    </section>
+
+    <div class="rounded-sm dark:bg-meta-4 p-4 md:p-6 mt-5">
+      <.codec_settings_help />
+    </div>
+  </section>
+
  <.button class="mt-10 mb-4 sm:mb-8 w-full sm:w-auto" rounding="rounded-lg">Save Settings</.button>
 </.simple_form>
@@ -1,34 +1,73 @@
 defmodule PinchflatWeb.Sources.SourceController do
  use PinchflatWeb, :controller
-
-  import Ecto.Query, warn: false
+  use Pinchflat.Media.MediaQuery

  alias Pinchflat.Repo
  alias Pinchflat.Tasks
  alias Pinchflat.Sources
  alias Pinchflat.Sources.Source
+  alias Pinchflat.Media.MediaItem
  alias Pinchflat.Profiles.MediaProfile
+  alias Pinchflat.Sources.SourceDeletionWorker
  alias Pinchflat.Downloading.DownloadingHelpers
  alias Pinchflat.SlowIndexing.SlowIndexingHelpers
  alias Pinchflat.Metadata.SourceMetadataStorageWorker

  def index(conn, _params) do
-    sources =
-      Source
-      |> order_by(asc: :custom_name)
-      |> Repo.all()
-      |> Repo.preload(:media_profile)
+    source_query =
+      from s in Source,
+        as: :source,
+        inner_join: mp in assoc(s, :media_profile),
+        where: is_nil(s.marked_for_deletion_at) and is_nil(mp.marked_for_deletion_at),
+        preload: [media_profile: mp],
+        order_by: [asc: s.custom_name],
+        select: map(s, ^Source.__schema__(:fields)),
+        select_merge: %{
+          downloaded_count:
+            subquery(
+              from m in MediaItem,
+                where: m.source_id == parent_as(:source).id,
+                where: ^MediaQuery.downloaded(),
+                select: count(m.id)
+            ),
+          pending_count:
+            subquery(
+              from m in MediaItem,
+                join: s in assoc(m, :source),
+                where: m.source_id == parent_as(:source).id,
+                where: ^MediaQuery.pending(),
+                select: count(m.id)
+            )
+        }

-    render(conn, :index, sources: sources)
+    render(conn, :index, sources: Repo.all(source_query))
  end

-  def new(conn, _params) do
-    changeset = Sources.change_source(%Source{})
+  def new(conn, params) do
+    # This lets me preload the settings from another source for more efficient creation
+    cs_struct =
+      case to_string(params["template_id"]) do
+        "" -> %Source{}
+        template_id -> Repo.get(Source, template_id) || %Source{}
+      end

    render(conn, :new,
-      changeset: changeset,
      media_profiles: media_profiles(),
-      layout: get_onboarding_layout()
+      layout: get_onboarding_layout(),
+      # Most of these don't actually _need_ to be nullified at this point,
+      # but if I don't do it now I know it'll bite me
+      changeset:
+        Sources.change_source(%Source{
+          cs_struct
+          | id: nil,
+            uuid: nil,
+            custom_name: nil,
+            description: nil,
+            collection_name: nil,
+            collection_id: nil,
+            collection_type: nil,
+            original_url: nil
+        })
    )
  end

@@ -88,19 +127,15 @@ defmodule PinchflatWeb.Sources.SourceController do
  end

  def delete(conn, %{"id" => id} = params) do
-    delete_files = Map.get(params, "delete_files", false)
+    # This awkward comparison converts the string to a boolean
+    delete_files = Map.get(params, "delete_files", "") == "true"
    source = Sources.get_source!(id)
-    {:ok, _source} = Sources.delete_source(source, delete_files: delete_files)

-    flash_message =
-      if delete_files do
-        "Source and files deleted successfully."
-      else
-        "Source deleted successfully. Files were not deleted."
-      end
+    {:ok, _} = Sources.update_source(source, %{marked_for_deletion_at: DateTime.utc_now()})
+    SourceDeletionWorker.kickoff(source, %{delete_files: delete_files})

    conn
-    |> put_flash(:info, flash_message)
+    |> put_flash(:info, "Source deletion started. This may take a while to complete.")
    |> redirect(to: ~p"/sources")
  end

@@ -9,12 +9,14 @@ defmodule PinchflatWeb.Sources.SourceHTML do
  attr :changeset, Ecto.Changeset, required: true
  attr :action, :string, required: true
  attr :media_profiles, :list, required: true
+  attr :method, :string, required: true

  def source_form(assigns)

  def friendly_index_frequencies do
    [
      {"Only once when first created", -1},
+      {"30 minutes", 30},
      {"1 Hour", 60},
      {"3 Hours", 3 * 60},
      {"6 Hours", 6 * 60},
@@ -26,6 +26,11 @@
      <span x-show="copied" x-transition.duration.150ms><.icon name="hero-check" class="ml-2 h-4 w-4" /></span>
    </span>
  </:option>
+  <:option>
+    <.link href={~p"/sources/new?template_id=#{@source}"} method="get">
+      Use as Template
+    </.link>
+  </:option>
  <:option>
    <div class="h-px w-full bg-bodydark2"></div>
  </:option>
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Kieran Eglin	7fb6fa3af4	Bumped version	2024-09-12 14:39:07 -07:00
Kieran	bb8788f03d	[Dev] Version updates 2024-09-12 (#388 ) * Updated some deps * Updated phoenix	2024-09-12 13:04:49 -07:00
Kieran	ed4434512b	[Dev] Speed up CI (#385 ) * Moved around copy steps * no-op change * Reverted the no-op change * Testing if docker-compose volume is needed * Nope, it's needed * Testing alternate driver * Add containerd * Bump version * Revert	2024-09-12 12:38:12 -07:00
Kieran	f6344d43d2	[Bugfix] Disallow concurrent slow-indexing runs for the same source (#384 ) * Updated deps in case that does anything * Kills running slow-indexes when a new slow-index is enqueued * Revert deps upgrade (saving that for its own PR)	2024-09-10 14:16:10 -07:00
Kieran	e0745bdfbe	[Enhancement] Allow configuration of media container format (#383 ) * Adds media container to media_profiles and updates option builder * Added media container to media profile form	2024-09-10 11:44:57 -07:00
Kieran Eglin	a0a02cb4be	Updated docker-build-push version in docker release file as well	2024-09-10 11:42:10 -07:00
Kieran Eglin	7866f7258a	Updated docker-build-push version	2024-09-10 11:36:54 -07:00
Kieran	5cd82f9dd5	Updated the aspect ratio threshold for short detection (#382 )	2024-09-10 08:49:30 -07:00
Kieran Eglin	c85e9cb5f1	Retry with a non-yanked version of ffmpeg I guess?	2024-08-28 09:58:53 -07:00
Kieran Eglin	4837dd44c9	Bumped version	2024-08-28 08:47:13 -07:00
Kieran Eglin	e3f42c28d0	Opps - reversed the IPv6 logic	2024-08-27 20:25:53 -07:00
Kieran	4a11d47692	Locked ffmpeg to last known-good version (#373 )	2024-08-27 20:20:53 -07:00
Kieran	612c760755	[Enhancement] Allow IPv6 (#372 ) * Added option to enable IPv6 in runtime config * Updated README	2024-08-27 20:08:53 -07:00
Kieran	14b8ecbe44	[Enhancement] Show error messages in-app for easier triage (#365 ) * Added an improved error message screen with actionable details * Added a basic 404 page * fixed tests	2024-08-23 13:33:27 -07:00
Kieran	a6c61ccd0d	[Bugfix] Improve livestream detection (#362 ) * Livestream status is now based off live_status instead of was_live * Updated tests	2024-08-20 09:37:42 -07:00
Kieran Eglin	80209240d2	Updated logger to add dates in logfiles rather than just console output	2024-08-20 08:58:30 -07:00
Kieran Eglin	1d5b62ccf9	Bumped version	2024-08-14 14:48:58 -07:00
Kieran	af8235c6b3	[Enhancement] Filter media based on min and/or max duration (#356 ) * Added duration limits to source model * Added duration limits to source form * Added validation for min/max amounts * Added duration checks to pending query * Moved min/max filters up in source form * Removed debugger	2024-08-14 14:20:50 -07:00
Kieran	8e9f02c807	[Enhancement] Support specifying count of `yt-dlp` workers per-queue (#355 ) * Updated certain Oban queues to have a configurable limit * Updated README	2024-08-14 12:37:24 -07:00
Kieran	fd20ac5d84	[Enhancement] Use cookies on a per-source basis (#354 ) * Added 'use_cookies' column to source * Added cookies to the source form * Updated every command to optionally include cookies * LOTS of tests * Made YT cookies off by default	2024-08-14 12:00:18 -07:00
Kieran	23492961d5	[Dev] Adds `.devcontainer` manifest to help promote contributions (#342 ) * Added a devcontainer file * linting * Added link to README	2024-08-07 15:38:06 -07:00
Kieran Eglin	2326775d78	Bump for new yt-dlp version	2024-08-06 14:17:14 -07:00
Kieran Eglin	e0a84a521b	Bumped version	2024-07-30 08:23:03 -07:00
Kieran Eglin	d217ad5744	[Dev] Added ZSH to dev dockerfile	2024-07-30 08:20:55 -07:00
Kieran	d5ae41cdab	[Performance] Asyncronously delete sources and media profiles (#277 ) * [WIP] started on source deletion * Removed unneeded test blocks * Added marked_for_deletion_at to sources and media profiles * Hooked up async deletion to media profiles as well	2024-07-22 12:26:22 -07:00
Kieran	7a01db05dd	[Enhancement] Ignore media based on user scripts (#330 ) * Integrated pre-download callback for media items * Refactored existing tests * Docs, etc	2024-07-22 10:47:49 -07:00
Kieran	d392fc3818	[Enhancement] Media from before a source's cutoff date is now automatically deleted (#329 ) * Changed the purpose of the 'culled_at' flag. renamed some methods * implemented new culling behaviour for source cutoff dates * Removed culled_at flag if a media item gets redownloaded * Updated source form * Removed unused method	2024-07-19 16:39:30 -07:00
Kieran Eglin	47596d5f72	Added a script to help with version bumping	2024-07-18 15:59:31 -07:00
Kieran Eglin	dbc25e5730	changed version number because I'm a coward	2024-07-18 11:13:03 -07:00
Kieran Eglin	a3ebf59b29	Bumped version	2024-07-18 11:09:28 -07:00
Kieran	8dd8e49051	[Dev] Fix CI caching (#325 ) * Consolidated dockerfile commands * Added teardown step * FIxed typo * Got rid of old caching action * Added buildx * Added steps for compiling test ENV to dockerfile * Changed the order of things * Added env vars * more env vars * Removed step * No compile flag * fetch quest * Removed other thing * volumes * Rolling back some other changes * more consolidation * updated checkout runner	2024-07-17 14:21:01 -07:00
Kieran	029981b7ee	[Enhancement] Support auto-generated subs when embedding subs in media file (#324 ) * Allowed using auto subs when embedding * Updated profile form	2024-07-17 10:33:50 -07:00
Kieran	d423e4486f	Added padding to playlist_index output template (#321 )	2024-07-16 15:30:27 -07:00
Kieran	4c7a63c970	[Bugfix] Determine NFO season and episode from filepath (#320 ) * Updated NFO generator to determine S/E from filepath * Tests	2024-07-16 15:16:24 -07:00
Kieran	4822127e3f	Added date-based outtmpl helper for static playlists (#319 )	2024-07-16 14:06:54 -07:00
Kieran Eglin	4f25878f2f	Bumped version	2024-07-16 10:00:09 -07:00
Kieran	aec56e3a03	Added some improved output template helpers (#318 )	2024-07-16 09:41:43 -07:00
Kieran	e06e050f95	[Enhancement] Capture `playlist_index` for use with output templates (#315 ) * Started capturing playlist_index on indexing pass * Added playlist_index as a media item field * Added playlist index to output variable templates * Improved the way playlist_indexes are rejected on update * Updated docs * Undid unneeded changes	2024-07-15 16:06:13 -07:00
Kieran	5a10015634	[Enhancement] Download image when using playlists with media center apps (#313 ) * [WIP] started adding calls for downloading posters for playlists * Updated source image parser to work with playlists	2024-07-15 12:04:57 -07:00
Dan	0d5a41fd35	[Enhancement] Render URL attributes as a link (#299 ) * Render URLs as links when listing items from a map * Applies suggestions from MR feedback	2024-07-15 08:59:02 -07:00
Kieran	8f91c4e6a2	[Enhancement] Improve flow for adding many sources at once (#306 ) * Added mechamism for using existing sources as a reference for adding new sources * Added test	2024-07-12 08:53:26 -07:00
Kieran	7f1daf90ca	Add date to 'prod' logger (#305 )	2024-07-11 10:19:14 -07:00
Kieran	bb0e104b7e	Runs yarn install at project level on app boot (#304 )	2024-07-11 09:29:53 -07:00
Kieran Eglin	c0b8dec520	no-op change to force a rebuild with new yt-dlp version	2024-07-11 08:30:33 -07:00
Kieran Eglin	b8da973a81	Bumped version	2024-06-25 08:19:06 -07:00
Kieran	a46cfd1888	[Bugfix] Check for regex errors before saving a source (#296 ) * Added error if provided regex is invalid * improved test	2024-06-20 12:48:52 -07:00
Kieran Eglin	f08768b887	Updated formatter	2024-06-20 11:33:49 -07:00
Kieran Eglin	199d0065e1	Updated screenshots	2024-06-14 09:23:40 -07:00
Kieran	781061b483	[Dev] Reduce config file clutter (#290 ) * Added local tooling config files to the tooling/ directory * Moved as many docker-related files as possible * Updated dockerfile path * Updated the correct dockerfile instead how about * Off day - jeez louise * Juggled around some more dockerfiles * more dockerfile * More docker tests * it's docker time * One more shot	2024-06-13 16:21:09 -07:00
Kieran	132f8e8da6	[Dev] Upgrade to Elixir 1.17.0 (#289 ) * upgraded to Elixir 1.17.0 * Addressed deprecation warnings * Updated Credo	2024-06-13 10:05:03 -07:00
Kieran Eglin	9daef8b3b8	Bumped version	2024-06-10 11:51:15 -07:00
Kieran	f6708a327c	[Enhancement] Optionally use the YouTube API for improved fast indexing (#282 ) * Started adding youtube API for fast indexing * Hooked youtube API into fast indexing * Added youtube_api_key to settings * Added youtube api key to settings UI * Added tests * Refactored the youtube api module * More refactor * Changed editing mode name from basic to standard * [WIP] started on copy changes * Updated copy	2024-06-10 11:45:41 -07:00
Kieran	582eb53698	Add regex for validating source URL (#285 )	2024-06-06 11:54:39 -07:00
Kieran	a20d06072f	Fixed upload_date_index callback from running unnecessarily (#284 )	2024-06-05 17:04:49 -07:00
Kieran Eglin	6d18130351	Updated docker release YAML	2024-06-05 09:14:07 -07:00
Kieran	af86ca1e0e	Switched thumbnail downloading to use yt-dlp (#281 )	2024-06-04 10:39:54 -07:00
Kieran	4994e70652	[Bugfix] Partially revert custom codec selection (#279 ) * Removed and re-added codec preference columns * Removed custom codec work from download options builder * Updated settings UI * Made codec preferences non-optional fields	2024-06-04 09:19:37 -07:00
Kieran Eglin	1c9f8f6abc	Bumped version	2024-05-30 15:16:33 -07:00
Kieran Eglin	2ae1ee1c99	Updated portainer instructions to use GHCR	2024-05-30 15:16:15 -07:00
Kieran	b73a9a4a67	[Enhancement] Added support for subdirectory-style reverse proxies (#276 ) * Updated routes and other fun stuff to support reverse proxies * Updated README * Linting	2024-05-30 12:58:51 -07:00
Kieran	3dd901ff3e	Prevented error when media item doesn't have description (#274 )	2024-05-29 13:52:43 -07:00
Kieran	e751c65424	[Enhancement] Improve handling of large media collections (#272 ) * Added compound index for updated media item table for performance * Improved large number display on homepage * Improved UI around large numbers * Centered the content of homepage data cards * Renamed migration	2024-05-29 12:38:03 -07:00
Kieran	22439e0273	[Housekeeping] Store the upload event as a datetime rather than a date (#269 ) * [WIP] renamed and migrated upload_date column * Refactored yt-dlp media module * Refactored parse_upload_date * Refactored media item upload_date_index * Got media tests running * Refactored media item table live * Cleaned up the stragglers * Fixed old oversight re: original_url	2024-05-28 13:01:40 -07:00
Kieran Eglin	7603ba017f	Improved wording re: livestreams on media profile form	2024-05-27 11:15:11 -07:00
Kieran	10880e9699	[Enhancement] Added link to local stream on media item show page (#265 ) * Adds link for opening local stream * Put conditional on correct element	2024-05-27 11:07:23 -07:00
Kieran	b2e16d50cd	Added library size to homepage (#264 )	2024-05-27 10:43:50 -07:00
Kieran	3a5c06fe64	Added upload date (#260 )	2024-05-24 10:04:37 -07:00
Kieran Eglin	99fc2eb8f8	Bumped version	2024-05-24 09:38:06 -07:00
Kieran Eglin	e824b1a9f5	Fixed header height bug	2024-05-23 15:25:57 -07:00
Kieran	95a0c29358	[Enhancement] Added search to source forms (#259 ) * Changed sqlite FTS to use a trigram tokenizer * Added search UI to source tables * Fix bug with special chars in search form * improved main search results form * Improved centering for media table header elements	2024-05-23 15:09:49 -07:00
Kieran Eglin	8b0b41186a	Added code comments	2024-05-23 10:28:08 -07:00
Kieran	d2f91a8253	[Enhancement] Added "other" tab to see media that's not set for download (#258 ) * Added tab for other (non-downloaded and non-pending) media * Added column for tracking if media was manually ignored	2024-05-23 10:24:52 -07:00
Kieran Eglin	776b84c585	Switched back to mainline sqleton [skip-ci]	2024-05-23 10:15:35 -07:00
Kieran Eglin	29803c9b26	added bullet points to diag info string	2024-05-22 16:52:32 -07:00
Kieran Eglin	85be8744d4	Linting	2024-05-22 16:51:08 -07:00
Kieran Eglin	1cbc62dc27	[Bugfix] removed layout flash for config dropdown	2024-05-22 14:45:44 -07:00
Kieran	5af22d3a2f	Update issue templates	2024-05-22 14:27:04 -07:00
Kieran Eglin	bf917febe8	bumped version	2024-05-22 12:33:26 -07:00
Kieran Eglin	8189dcce71	Added link to env vars section	2024-05-22 12:18:20 -07:00
Kieran Eglin	4e4092c792	Add list of env vars to README	2024-05-22 12:17:09 -07:00
Kieran	0cd6ac797b	[Dev] Add ERD for tracking database state (#256 ) * Added ERD triggered on migration * Added alias	2024-05-22 10:40:30 -07:00
Kieran	81d5efd4c1	[Enhancement] Allow setting preference for video/audio codec (#255 ) * [WIP] started on codec parser * Added codec preferences to settings model * [WIP] added basic codec preference fields to settings form * Hooked up the backend portion of the codec preference work * Added codec settings to frontend * Ensured you can remove codec data	2024-05-21 16:28:19 -07:00
Kieran	dec3938780	Added 8k support (#254 )	2024-05-21 09:59:05 -07:00
Kieran	a2a0f48065	[Enhancement] Add media count to sources page (#251 ) * [WIP] Got query kinda working, now to refactor other queries * Refactored all query methods to use dynamic snippets * Refactored tab layout to grab tabs by name * Removed standalone show buttons from in-app tables * Removed unneeded comment	2024-05-21 08:53:22 -07:00
Kieran	94c0cf9970	[Enhancement] Show currently executing jobs on homepage (#250 ) * [WIP] Added PoC job table * [WIP] hooked up tasks table with real data * Added reload on websocket command * Hooked it all up to a websocket	2024-05-17 10:55:27 -07:00
Kieran Eglin	bbdc56c656	Removed old file	2024-05-16 10:15:58 -07:00
Kieran Eglin	7bf74a04e7	Bumped version	2024-05-16 09:54:20 -07:00
Andrew Shaffer	33040bba34	Fix typo on app info page (#248 )	2024-05-15 18:31:30 -07:00
Kieran	b1a6a6a81f	[Bugfix] Fix yt-dlp illegal write issue that causes failure to fetch source details (#247 ) * Bandaid fix for yt-dlp write issue * Ensured tempfile directory on app boot	2024-05-15 17:32:56 -07:00